换下风格^_^

Hadoop 2.x编译

Hadoop admin 610℃ 0评论

前言

从官网下载的Apache 发行版本的hadoop默认是在32位下编译的。对于大都是64位系统的集群来说我们需要自己编译64位的hadoop版本,特别是native本地链接库。如果native与你所使用的64位系统不符,那么会影响集群的性能。

那么如何判断你使用的hadoop不是64位的呢?一般我们可以通过file命令来查看:

file  $HADOOP_HOME/lib/native/libhdfs.so.0.0.0

返回信息如下:

libhdfs.so.0.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped

说明你使用的hadoop是32的。

还有一个方法是,在你hadoop本地库安装正确的情况下使用hadoop fs -ls /  会出现如下警告:

[@tc-162-20 ~/soft/hadoop]$ hadoop fs -ls /
15/01/30 10:49:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

好了,问题描述完了,我们开始编译。

编译

1.准备

在编译hadoop前我们需要一些工具和依赖。

工具:

  • jdk
  • maven
  • ant

依赖:

  • C compiler (e.g. GNU C Compiler)
  • GNU Autools Chain: autoconf, automake, libtool
  • zlib-development package (stable version >= 1.2.0)
  • openssl-development package(e.g. libssl-dev)

2.步骤

2.1 、安装jdk、gcc、gcc-c++ 、make、cmake、openssl-devel、ncurses-devel

yum install gcc  gcc-c++  make cmake  openssl-devel ncurses-devel

2.2、安装maven

下载:http://mirror.bit.edu.cn/apache/maven/maven-3/3.2.1/binaries/apache-maven-3.1.1-bin.zip

tar zxf apache-maven-3.2.1-bin.tar.gz

mv apache-maven-3.2.1 /usr/local/lib/

安装在:

2.3、安装ant

http://mirrors.cnnic.cn/apache//ant/binaries/apache-ant-1.9.4-bin.tar.gz

tar xvf apache-ant-1.9.4-bin.tar.gz

mv apache-ant-1.9.4 /usr/local/lib/

2.4 、安装protobuf

下载: https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz

tar xvf protobuf-2.5.0.tar.gz

cd protobuf-2.5.0

./configure

make

make install

ldconfig

2.5 添加环境变量

export ANT_HOME=/usr/local/lib/apache-ant-1.9.4
export PATH=$PATH:$ANT_HOME/bin

export MVN_HOME=/usr/local/lib/apache-maven-3.2.1
export PATH=$PATH:$MVN_HOME/bin

 

3.编译

进入源码文件夹

cd /hadoop/hadoop-2.6.0-src

mvn clean

mvn package -Pdist,native -DskipTests -Dtar

 

 

如果没有错误等一段时间(我用了15分钟)后就可以看到下面结果:

[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ………………………….. SUCCESS [ 2.331 s]
[INFO] Apache Hadoop Project POM ……………………. SUCCESS [ 1.660 s]
[INFO] Apache Hadoop Annotations ……………………. SUCCESS [ 2.962 s]
[INFO] Apache Hadoop Assemblies …………………….. SUCCESS [ 0.255 s]
[INFO] Apache Hadoop Project Dist POM ……………….. SUCCESS [ 2.184 s]
[INFO] Apache Hadoop Maven Plugins ………………….. SUCCESS [ 6.201 s]
[INFO] Apache Hadoop MiniKDC ……………………….. SUCCESS [ 2.622 s]
[INFO] Apache Hadoop Auth ………………………….. SUCCESS [ 3.369 s]
[INFO] Apache Hadoop Auth Examples ………………….. SUCCESS [ 3.350 s]
[INFO] Apache Hadoop Common ………………………… SUCCESS [01:42 min]
[INFO] Apache Hadoop NFS …………………………… SUCCESS [ 9.511 s]
[INFO] Apache Hadoop KMS …………………………… SUCCESS [ 19.845 s]
[INFO] Apache Hadoop Common Project …………………. SUCCESS [ 0.030 s]
[INFO] Apache Hadoop HDFS ………………………….. SUCCESS [02:24 min]
[INFO] Apache Hadoop HttpFS ………………………… SUCCESS [ 23.309 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal …………. SUCCESS [ 14.419 s]
[INFO] Apache Hadoop HDFS-NFS ………………………. SUCCESS [ 4.119 s]
[INFO] Apache Hadoop HDFS Project …………………… SUCCESS [ 0.033 s]
[INFO] hadoop-yarn ………………………………… SUCCESS [ 0.050 s]
[INFO] hadoop-yarn-api …………………………….. SUCCESS [01:19 min]
[INFO] hadoop-yarn-common ………………………….. SUCCESS [ 27.037 s]
[INFO] hadoop-yarn-server ………………………….. SUCCESS [ 0.071 s]
[INFO] hadoop-yarn-server-common ……………………. SUCCESS [ 8.420 s]
[INFO] hadoop-yarn-server-nodemanager ……………….. SUCCESS [ 14.496 s]
[INFO] hadoop-yarn-server-web-proxy …………………. SUCCESS [ 2.303 s]
[INFO] hadoop-yarn-server-applicationhistoryservice …… SUCCESS [ 5.159 s]
[INFO] hadoop-yarn-server-resourcemanager ……………. SUCCESS [ 14.684 s]
[INFO] hadoop-yarn-server-tests …………………….. SUCCESS [ 3.705 s]
[INFO] hadoop-yarn-client ………………………….. SUCCESS [ 5.057 s]
[INFO] hadoop-yarn-applications …………………….. SUCCESS [ 0.030 s]
[INFO] hadoop-yarn-applications-distributedshell ……… SUCCESS [ 2.073 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher …. SUCCESS [ 1.478 s]
[INFO] hadoop-yarn-site ……………………………. SUCCESS [ 0.027 s]
[INFO] hadoop-yarn-registry ………………………… SUCCESS [ 3.902 s]
[INFO] hadoop-yarn-project …………………………. SUCCESS [ 5.796 s]
[INFO] hadoop-mapreduce-client ……………………… SUCCESS [ 0.050 s]
[INFO] hadoop-mapreduce-client-core …………………. SUCCESS [ 15.708 s]
[INFO] hadoop-mapreduce-client-common ……………….. SUCCESS [ 13.255 s]
[INFO] hadoop-mapreduce-client-shuffle ………………. SUCCESS [ 3.066 s]
[INFO] hadoop-mapreduce-client-app ………………….. SUCCESS [ 9.691 s]
[INFO] hadoop-mapreduce-client-hs …………………… SUCCESS [ 5.959 s]
[INFO] hadoop-mapreduce-client-jobclient …………….. SUCCESS [ 9.923 s]
[INFO] hadoop-mapreduce-client-hs-plugins ……………. SUCCESS [ 1.432 s]
[INFO] Apache Hadoop MapReduce Examples ……………… SUCCESS [ 4.431 s]
[INFO] hadoop-mapreduce ……………………………. SUCCESS [ 4.747 s]
[INFO] Apache Hadoop MapReduce Streaming …………….. SUCCESS [ 7.749 s]
[INFO] Apache Hadoop Distributed Copy ……………….. SUCCESS [ 10.764 s]
[INFO] Apache Hadoop Archives ………………………. SUCCESS [ 1.727 s]
[INFO] Apache Hadoop Rumen …………………………. SUCCESS [ 5.115 s]
[INFO] Apache Hadoop Gridmix ……………………….. SUCCESS [ 3.853 s]
[INFO] Apache Hadoop Data Join ……………………… SUCCESS [ 2.141 s]
[INFO] Apache Hadoop Ant Tasks ……………………… SUCCESS [ 1.836 s]
[INFO] Apache Hadoop Extras ………………………… SUCCESS [ 2.728 s]
[INFO] Apache Hadoop Pipes …………………………. SUCCESS [ 7.900 s]
[INFO] Apache Hadoop OpenStack support ………………. SUCCESS [ 4.414 s]
[INFO] Apache Hadoop Amazon Web Services support ……… SUCCESS [ 19.087 s]
[INFO] Apache Hadoop Client ………………………… SUCCESS [ 7.145 s]
[INFO] Apache Hadoop Mini-Cluster …………………… SUCCESS [ 0.101 s]
[INFO] Apache Hadoop Scheduler Load Simulator ………… SUCCESS [ 3.463 s]
[INFO] Apache Hadoop Tools Dist …………………….. SUCCESS [ 9.400 s]
[INFO] Apache Hadoop Tools …………………………. SUCCESS [ 0.032 s]
[INFO] Apache Hadoop Distribution …………………… SUCCESS [ 35.405 s]
[INFO] ————————————————————————
[INFO] BUILD SUCCESS
[INFO] ————————————————————————
[INFO] Total time: 11:51 min
[INFO] Finished at: 2015-01-30T10:10:10+08:00
[INFO] Final Memory: 185M/488M
[INFO] ————————————————————————

4.结果

可以在hadoop-2.6.0-src/hadoop-dist/target目录下找到编译好的hadoop

hadoop-2.6.0-src/hadoop-dist/target/hadoop-2.6.0.tar.gz

 

好了。完工

下载

这里给一个我编译好的下载:

64位Hadoop下载:hadoop-2.6.0.tar.gz

链接: http://pan.baidu.com/s/1dbNLO 密码: p34t

google protobuf下载

由于有些人不能上google。所以也把protobuf贡献出来

链接: http://pan.baidu.com/s/1pJLq90v 密码: y0gq

 

转载请注明:极豆技术博客 » Hadoop 2.x编译

喜欢 (0)
捐助本站极豆博客全站无广告。如果您觉得本博客的内容对您小有帮助,可以对我小额赞助,您的赞助将用于维持博客运营。

极豆博客

发表我的评论
取消评论
表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址