Hadoop – Hbase Cluster with Docker on AWS 1. Problems of Hadoop1 – data streaming – map process and reduce process are seperated – job tracker manage all jobs alone (too busy) . cannot manage resource(cpu, memory) effectively – SPOF weakness (name node dies whole system dies) 2. Solution of Hadoop2 – job tracker is […]
About Map Reduce Code 1.Ordering with Map Reduce (A) Binary Search we are going to make a map reduce program which return N numbers of keywords from the top rank (ordered by number of appears) Hadoop support beautiful sorting Library which is called PriorityQueue and by calling peek you can get keyword on the […]
Build & Run Example Code 1. download maven – download maven build tool from site using apt-get sudo apt-get install maven 2. get test source code using wget wget https://s3.amazonaws.com/hadoopkr/source.tar.gz 3. build source with mvn cd /home/<user>/source/<where pom.xml> mvn compile 5. upload local file to hadoop hadoop fs -copyFromLocal README.txt / 6. execute on hadoop […]
Get Ubuntu Docker – docker pull ubuntu Start Container docker run -i -p 22 -p 8000:80 -m /data:/data -t <ubuntu> /bin/bash Install Jdk sudo add-apt-repository ppa:openjdk-r/ppa sudo apt-get update sudo apt-get install openjdk-7-jre .bashrc export JAVA_HOME=/usr/lib/jvm/… export CLASSPATH=$JAVA_HOME/lib/*:. export PATH=$PATH:$JAVA_HOME/bin HADOOP 1.2.1 install download hadoop and unpack root@4aa2cda88fcc:/home/kim# wget http://apache.mirror.cdnetworks.com/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz root@4aa2cda88fcc:/home/kim# mv ./hadoop-1.2.1.tar.gz /home/user root@4aa2cda88fcc:/home/kim# […]
– rvm (ruby version manager 설치 ) #curl -L https://get.rvm.io | bash -s stable –ruby #vi / etc/bashrc ==> source /root/.rvm/scripts/rvm #source /etc/bashrc – ruby 버전 변경 # rvm reinstall 2.2.2 # ruby -v
1. IPTABLE iptables -A INPUT -p tcp -m tcp –sport 80 -j ACCEPT iptables -A OUTPUT -p tcp -m tcp –dport 80 -j ACCEPT 2. Fire Wall sudo firewall-cmd –permanent –add-port=2181/tcp firewall-cmd –reload firewall-cmd –zone=public –list-all
1. 설치 STEP 1. install : sudo apt-get install vsftpd 2. EC2 inbound : 1024 ~ 1048 , 21 , 20 3. config : vi /etc/vsftpd/vsftpd.conf 4. User Ban List : /etc/vsftpd/user_list 5. Start : sudo service vsftpd start 2. vsfpd.conf 변경 #vsftpd.conf anonymous enable = NO pasv_enable=YES pasv_min_port=1024 pasv_max_port=1048 pasv_address=public ip (not dns) #chroot_local_user=YES […]
집에서 서버를 구축해서 사용을 해보면 SK, KT 등 통신사를 통하기 때문에 고정 IP 를 사용할 수 없다던지, Proxy 서버에 차단된다던지 하는 어려움도 존재하고 전기비도 생각하는 것보다 엄청 많이 나오는 관계로 AWS사용을 적극 검토하게 되었다. 오늘 작성하려는 내용은 AWS 에 EC2 서비스를 사용 Ubuntu 를 설치하고 Terminal 접속만 되는 문제를 해결하고자 XRDP 를 설치하여 접속하는 것 […]
General Router Info – SK Broad Band (Mercury) Router (MMC) . IP : http://192.168.25.1 . ID : admin . PASS : last six digit of MAC Address + _admin – IPTime Router (DVW) . IP : http://192.168.0.1 . ID/PASS : All yours Find you Server PC IP – Terminal >> Type “ip addr show” Login […]
가. Spark 설치 (1) http://spark.apache.org/downloads.html 접속 (2) 기 구축된 Hadoop 환경이 있는 것이 아니라면 Hadoop Pre-Build 선택 (3) download Spark (4) 압축 해제 tar -zxvf spark-1.6.1-bin-hadoop.2.6.tgz 나. Spark 실행 [커맨드 모드] (1) spark-1.6.1-bin-hadoop2.6/bin$ ./pyspark (2) Spark 모니터링 16/06/01 22:03:46 INFO SparkUI: Started SparkUI at http://192.168.0.3:4040 [Master Node] /sbin/start-master.sh ※ 아래 Page 기본 접속 포트는 8080 […]