p13709645
/
jydata
forked from jiuyuan/jydata
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

update

This commit is contained in:
nezharen 2024-04-10 16:56:43 +08:00
parent 27d5499f8a
commit a25e778837
1 changed files with 17 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
ProjectDeploy.md
View File

@ -255,7 +255,21 @@ InstalledDir: /home/jydata/jydata/local/llvm-16.0.4/bin
jydata@node01$ for i in {2..6}; do ssh node0$i "mkdir -p ~/jydata/local/llvm-16.0.4/lib" && scp -r ~/jydata/local/llvm-16.0.4/lib/libomp.so node0$i:~/jydata/local/llvm-16.0.4/lib/; done jydata@node01$ for i in {2..6}; do ssh node0$i "mkdir -p ~/jydata/local/llvm-16.0.4/lib" && scp -r ~/jydata/local/llvm-16.0.4/lib/libomp.so node0$i:~/jydata/local/llvm-16.0.4/lib/; done
``` ```
#### 3.4 下载JNA jar包 #### 3.4 安装Scala工具链
x86_64环境：
```shell
jydata@node01$ curl -fL https://github.com/coursier/coursier/releases/latest/download/cs-x86_64-pc-linux.gz | gzip -d > cs && chmod +x cs && ./cs setup
```
aarch64环境：
```shell
jydata@node01$ curl -fL https://github.com/VirtusLab/coursier-m1/releases/latest/download/cs-aarch64-pc-linux.gz | gzip -d > cs && chmod +x cs && ./cs setup
```
#### 3.5 下载JNA jar包
```shell ```shell
jydata@node01$ cd ~/jydata/third-party jydata@node01$ cd ~/jydata/third-party
@ -387,7 +401,7 @@ jydata@node01$ git clone https://github.com/databricks/tpcds-kit.git
jydata@node01$ mv ./tpcds-kit ~/jydata/local/DSGen jydata@node01$ mv ./tpcds-kit ~/jydata/local/DSGen
jydata@node01$ cd ~/jydata/local/DSGen/tools jydata@node01$ cd ~/jydata/local/DSGen/tools
jydata@node01$ patch -p0 < ~/jydata/third-party/patches/tpcds_makefile.patch jydata@node01$ patch -p0 < ~/jydata/third-party/patches/tpcds_makefile.patch
jydata@node01$ make # 您可能需要自行安装GCC编译器 jydata@node01$ make
``` ```
最后将编译出的TPC-DS工具包同步到其它节点的相同路径下： 最后将编译出的TPC-DS工具包同步到其它节点的相同路径下：
@ -403,7 +417,7 @@ jydata@node01$ source ~/jydata/setenv.sh
jydata@node01$ git clone https://github.com/databricks/spark-sql-perf.git jydata@node01$ git clone https://github.com/databricks/spark-sql-perf.git
jydata@node01$ mv spark-sql-perf ~/jydata/local/ jydata@node01$ mv spark-sql-perf ~/jydata/local/
jydata@node01$ cd ~/jydata/local/spark-sql-perf jydata@node01$ cd ~/jydata/local/spark-sql-perf
jydata@node01$ sbt package # 您可能需要自行安装sbt等Scala编译器 jydata@node01$ sbt package
jydata@node01$ cd ~/jydata/sources/CMR/sbin/ jydata@node01$ cd ~/jydata/sources/CMR/sbin/
jydata@node01$ sed -i 's@$HOME@'"$HOME"'@g' tpcds_datagen_parquet.scala jydata@node01$ sed -i 's@$HOME@'"$HOME"'@g' tpcds_datagen_parquet.scala
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -mkdir -p /user/hive/tpc-ds-10T/decimaldata jydata@node01$ $HADOOP_HOME/bin/hadoop fs -mkdir -p /user/hive/tpc-ds-10T/decimaldata