【spark源码修改】hive3.1.3 spark3.5.4编译，需要修改源码，最终编译成功

---

1. 准备安装包与maven编译环境

1.1 安装环境准备

安装包到官网找， hive.apache.org, spark.apache.org
maven环境准备： 初始化只改了settings的ali源

1.2 修改pom

pom版本修改前：

    <spark.version>2.3.0</spark.version>
    <scala.binary.version>2.11</scala.binary.version>
    <scala.version>2.11.8</scala.version>

修改后：

    <spark.version>3.5.4</spark.version>
    <scala.binary.version>2.13</scala.binary.version>
    <scala.version>2.13.8</scala.version>

1.3 打包命令

mvn clean package -Pdist -DskipTests -Dmaven.javadoc.skip=true

2. 编译与问题解决

2.1 开始编译 失败， 缺包pentaho-aggdesigner-algorithm:pom:5.1.5-jhyde

pentaho-aggdesigner-algorithm:pom:5.1.5-jhyde 找不到

Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.2/hadoop-mapreduce-client-core-2.7.2.pom (3.5 kB at 12 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.6.4/hadoop-common-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.6.4/hadoop-common-2.6.4.pom (31 kB at 77 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-project-dist/2.6.4/hadoop-project-dist-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-project-dist/2.6.4/hadoop-project-dist-2.6.4.pom (18 kB at 48 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-project/2.6.4/hadoop-project-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-project/2.6.4/hadoop-project-2.6.4.pom (43 kB at 106 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-main/2.6.4/hadoop-main-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-main/2.6.4/hadoop-main-2.6.4.pom (18 kB at 47 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-annotations/2.6.4/hadoop-annotations-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-annotations/2.6.4/hadoop-annotations-2.6.4.pom (2.4 kB at 8.0 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-auth/2.6.4/hadoop-auth-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-auth/2.6.4/hadoop-auth-2.6.4.pom (7.7 kB at 20 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hive/hive-storage-api/2.2.1/hive-storage-api-2.2.1.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hive/hive-storage-api/2.2.1/hive-storage-api-2.2.1.pom (5.4 kB at 12 kB/s)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Hive 3.1.3:
[INFO] 
[INFO] Hive Upgrade Acid .................................. FAILURE [02:30 min]
[INFO] Hive ............................................... SKIPPED
[INFO] Hive Classifications ............................... SKIPPED
[INFO] Hive Shims Common .................................. SKIPPED
[INFO] Hive Shims 0.23 .................................... SKIPPED
[INFO] Hive Shims Scheduler ............................... SKIPPED
[INFO] Hive Shims ......................................... SKIPPED
[INFO] Hive Common ........................................ SKIPPED
[INFO] Hive Service RPC ................................... SKIPPED
[INFO] Hive Serde ......................................... SKIPPED
[INFO] Hive Standalone Metastore .......................... SKIPPED
[INFO] Hive Metastore ..................................... SKIPPED
[INFO] Hive Vector-Code-Gen Utilities ..................... SKIPPED
[INFO] Hive Llap Common ................................... SKIPPED
[INFO] Hive Llap Client ................................... SKIPPED
[INFO] Hive Llap Tez ...................................... SKIPPED
[INFO] Hive Spark Remote Client ........................... SKIPPED
[INFO] Hive Query Language ................................ SKIPPED
[INFO] Hive Llap Server ................................... SKIPPED
[INFO] Hive Service ....................................... SKIPPED
[INFO] Hive Accumulo Handler .............................. SKIPPED
[INFO] Hive JDBC .......................................... SKIPPED
[INFO] Hive Beeline ....................................... SKIPPED
[INFO] Hive CLI ........................................... SKIPPED
[INFO] Hive Contrib ....................................... SKIPPED
[INFO] Hive Druid Handler ................................. SKIPPED
[INFO] Hive HBase Handler ................................. SKIPPED
[INFO] Hive JDBC Handler .................................. SKIPPED
[INFO] Hive HCatalog ...................................... SKIPPED
[INFO] Hive HCatalog Core ................................. SKIPPED
[INFO] Hive HCatalog Pig Adapter .......................... SKIPPED
[INFO] Hive HCatalog Server Extensions .................... SKIPPED
[INFO] Hive HCatalog Webhcat Java Client .................. SKIPPED
[INFO] Hive HCatalog Webhcat .............................. SKIPPED
[INFO] Hive HCatalog Streaming ............................ SKIPPED
[INFO] Hive HPL/SQL ....................................... SKIPPED
[INFO] Hive Streaming ..................................... SKIPPED
[INFO] Hive Llap External Client .......................... SKIPPED
[INFO] Hive Shims Aggregator .....................

原文地址：https://blog.csdn.net/spark_dev/article/details/145054281

免责声明：本站文章内容转载自网络资源，如本站内容侵犯了原著者的合法权益，可联系本站删除。更多内容请关注自学内容网（zxcms.com）！