换下风格^_^

Spark1.2.0编译

Spark admin 509℃ 0评论
如果你还需要编译hadoop,请参考:《Hadoop 2.x编译

1. 准备

工具:

  • jdk
  • maven
  • ant

依赖:

  • C compiler (e.g. GNU C Compiler)
  • GNU Autools Chain: autoconf, automake, libtool
  • zlib-development package (stable version >= 1.2.0)
  • openssl-development package(e.g. libssl-dev)

详细可以参看《Hadoop 2.x编译

2、编译

先下载spark1.2.0 源码包:http://mirror.bit.edu.cn/apache/spark/spark-1.2.0/spark-1.2.0.tgz
配置Maven的内存使用,参数MAVEN_OPTS
export MAVEN_OPTS=”-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m”

如果不设置上面的参数会出现如下错误
[INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-2.10/classes…
[ERROR] PermGen space -> [Help 1]

在Java 8中不会出现上述问题。

mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-0.13.1 -Phive-thriftserver -DskipTests clean package

打包编译

make-distribution.sh  –tgz  –nam=hadoop-2.6.0     mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-0.13.1 -Phive-thriftserver -DskipTests

编译完成

Finished in 0 ms
[INFO] ————————————————————————
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM …………………….. SUCCESS [ 7.376 s]
[INFO] Spark Project Networking …………………….. SUCCESS [ 8.786 s]
[INFO] Spark Project Shuffle Streaming Service ……….. SUCCESS [ 5.549 s]
[INFO] Spark Project Core ………………………….. SUCCESS [03:35 min]
[INFO] Spark Project Bagel …………………………. SUCCESS [ 19.636 s]
[INFO] Spark Project GraphX ………………………… SUCCESS [ 51.508 s]
[INFO] Spark Project Streaming ……………………… SUCCESS [01:17 min]
[INFO] Spark Project Catalyst ………………………. SUCCESS [01:22 min]
[INFO] Spark Project SQL …………………………… SUCCESS [01:28 min]
[INFO] Spark Project ML Library …………………….. SUCCESS [01:36 min]
[INFO] Spark Project Tools …………………………. SUCCESS [ 12.052 s]
[INFO] Spark Project Hive ………………………….. SUCCESS [01:17 min]
[INFO] Spark Project REPL ………………………….. SUCCESS [ 34.480 s]
[INFO] Spark Project YARN Parent POM ………………… SUCCESS [ 3.353 s]
[INFO] Spark Project YARN Stable API ………………… SUCCESS [ 40.027 s]
[INFO] Spark Project Hive Thrift Server ……………… SUCCESS [ 23.705 s]
[INFO] Spark Project Assembly ………………………. SUCCESS [ 56.640 s]
[INFO] Spark Project External Twitter ……………….. SUCCESS [ 20.593 s]
[INFO] Spark Project External Flume Sink …………….. SUCCESS [ 18.497 s]
[INFO] Spark Project External Flume …………………. SUCCESS [ 24.978 s]
[INFO] Spark Project External MQTT ………………….. SUCCESS [ 14.521 s]
[INFO] Spark Project External ZeroMQ ………………… SUCCESS [ 16.708 s]
[INFO] Spark Project External Kafka …………………. SUCCESS [ 24.375 s]
[INFO] Spark Project Examples ………………………. SUCCESS [01:13 min]
[INFO] Spark Project YARN Shuffle Service ……………. SUCCESS [ 3.395 s]
[INFO] ————————————————————————
[INFO] BUILD SUCCESS
[INFO] ————————————————————————
[INFO] Total time: 18:18 min

下载

下载 链接: http://pan.baidu.com/s/1jGxKsZW 密码: gwrn

转载请注明:极豆技术博客 » Spark1.2.0编译

喜欢 (0)
捐助本站极豆博客全站无广告。如果您觉得本博客的内容对您小有帮助,可以对我小额赞助,您的赞助将用于维持博客运营。

极豆博客

发表我的评论
取消评论
表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址