基于Docker快速搭建多节点Hadoop集群
项目 镜像大小 问题 sequenceiq/hadoop-docker:latest 1.491GB 镜像太大,只支持单个节点 sequenceiq/hadoop-docker:2.7.0 1.76 GB sequenceiq/hadoop-docker:2.60 1.624GB sequenceiq/ambari:latest 1.782GB 镜像太大,使用太慢,使用复杂 sequenceiq/ambari:2.0.0 4.804GB sequenceiq/ambari:latest:1.70 4.761GB alvinhenrick/hadoop-mutinode 4.331GB 镜像太大,构建太慢,增加节点麻烦,有bug
镜像名称 构建时间 镜像层数 镜像大小 alvinhenrick/serf 258.213s 21 239.4MB alvinhenrick/hadoop-base 2236.055s 58 4.328GB alvinhenrick/hadoop-dn 51.959s 74 4.331GB alvinhenrick/hadoop-nn-dn 49.548s 84 4.331GB
镜像名称 构建时间 镜像层数 镜像大小 kiwenlau/serf-dnsmasq 509.46s 8 206.6 MB kiwenlau/hadoop-base 400.29s 7 775.4 MB kiwenlau/hadoop-master 5.41s 9 775.4 MB kiwenlau/hadoop-slave 2.41s 8 775.4 MB
sudo docker images
的运行结果:REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE index.alauda.cn/kiwenlau/hadoop-slave 0.1.0 d63869855c03 17 hours ago 777.4 MB index.alauda.cn/kiwenlau/hadoop-master 0.1.0 7c9d32ede450 17 hours ago 777.4 MB index.alauda.cn/kiwenlau/hadoop-base 0.1.0 5571bd5de58e 17 hours ago 777.4 MB index.alauda.cn/kiwenlau/serf-dnsmasq 0.1.0 09ed89c24ee8 17 hours ago 206.7 MB ubuntu 15.04 bd94ae587483 3 weeks ago 131.3 MB
sudo docker history index.alauda.cn/kiwenlau/hadoop-base:0.1.0
的部分运行结果IMAGE CREATED CREATED BY SIZE 2039b9b81146 44 hours ago /bin/sh -c #(nop) ADD multi:a93c971a49514e787 158.5 MB cdb620312f30 44 hours ago /bin/sh -c apt-get install -y openjdk-7-jdk 324.6 MB da7d10c790c1 44 hours ago /bin/sh -c apt-get install -y openssh-server 87.58 MB c65cb568defc 44 hours ago /bin/sh -c curl -Lso serf.zip https://dl.bint 14.46 MB 3e22b3d72e33 44 hours ago /bin/sh -c apt-get update && apt-get install 60.89 MB b68f8c8d2140 3 weeks ago /bin/sh -c #(nop) ADD file:d90f7467c470bfa9a3 131.3 MB
sudo docker pull index.alauda.cn/kiwenlau/hadoop-master:0.1.0 sudo docker pull index.alauda.cn/kiwenlau/hadoop-slave:0.1.0 sudo docker pull index.alauda.cn/kiwenlau/hadoop-base:0.1.0 sudo docker pull index.alauda.cn/kiwenlau/serf-dnsmasq:0.1.0
sudo docker pull kiwenlau/hadoop-master:0.1.0 sudo docker pull kiwenlau/hadoop-slave:0.1.0 sudo docker pull kiwenlau/hadoop-base:0.1.0 sudo docker pull kiwenlau/serf-dnsmasq:0.1.0
sudo docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE index.alauda.cn/kiwenlau/hadoop-slave 0.1.0 d63869855c03 17 hours ago 777.4 MB index.alauda.cn/kiwenlau/hadoop-master 0.1.0 7c9d32ede450 17 hours ago 777.4 MB index.alauda.cn/kiwenlau/hadoop-base 0.1.0 5571bd5de58e 17 hours ago 777.4 MB index.alauda.cn/kiwenlau/serf-dnsmasq 0.1.0 09ed89c24ee8 17 hours ago 206.7 MB
sudo docker tag d63869855c03 kiwenlau/hadoop-slave:0.1.0 sudo docker tag 7c9d32ede450 kiwenlau/hadoop-master:0.1.0 sudo docker tag 5571bd5de58e kiwenlau/hadoop-base:0.1.0 sudo docker tag 09ed89c24ee8 kiwenlau/serf-dnsmasq:0.1.0
sudo docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE index.alauda.cn/kiwenlau/hadoop-slave 0.1.0 d63869855c03 17 hours ago 777.4 MB kiwenlau/hadoop-slave 0.1.0 d63869855c03 17 hours ago 777.4 MB index.alauda.cn/kiwenlau/hadoop-master 0.1.0 7c9d32ede450 17 hours ago 777.4 MB kiwenlau/hadoop-master 0.1.0 7c9d32ede450 17 hours ago 777.4 MB kiwenlau/hadoop-base 0.1.0 5571bd5de58e 17 hours ago 777.4 MB index.alauda.cn/kiwenlau/hadoop-base 0.1.0 5571bd5de58e 17 hours ago 777.4 MB kiwenlau/serf-dnsmasq 0.1.0 09ed89c24ee8 17 hours ago 206.7 MB index.alauda.cn/kiwenlau/serf-dnsmasq 0.1.0 09ed89c24ee8 17 hours ago 206.7 MB
git clone https://github.com/kiwenlau/hadoop-cluster-docker
git clone http://git.oschina.net/kiwenlau/hadoop-cluster-docker
cd hadoop-cluster-docker ./start-container.sh
start master container... start slave1 container... start slave2 container... root@master:~#
ls
hdfs run-wordcount.sh serf_log start-hadoop.sh start-ssh-serf.sh
serf members
master.kiwenlau.com 172.17.0.65:7946 alive slave1.kiwenlau.com 172.17.0.66:7946 alive slave2.kiwenlau.com 172.17.0.67:7946 alive
ssh slave2.kiwenlau.com
Warning: Permanently added 'slave2.kiwenlau.com,172.17.0.67' (ECDSA) to the list of known hosts. Welcome to Ubuntu 15.04 (GNU/Linux 3.13.0-53-generic x86_64) * Documentation: https://help.ubuntu.com/ The programs included with the Ubuntu system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. root@slave2:~#
exit
logout Connection to slave2.kiwenlau.com closed.
./start-hadoop.sh
./run-wordcount.sh
input file1.txt: Hello Hadoop input file2.txt: Hello Docker wordcount output: Docker 1 Hadoop 1 Hello 2
./resize-cluster.sh 5
./start-container.sh 5
微信公众号【黄小斜】大厂程序员,互联网行业新知,终身学习践行者。关注后回复「Java」、「Python」、「C++」、「大数据」、「机器学习」、「算法」、「AI」、「Android」、「前端」、「iOS」、「考研」、「BAT」、「校招」、「笔试」、「面试」、「面经」、「计算机基础」、「LeetCode」 等关键字可以获取对应的免费学习资料。
网友评论