62 KiB

Raw Blame History

大数据智能分析系统部署文档20240410

大数据智能分析系统介绍

大数据智能分析系统在Apache Spark的基础上构建了一套以性能为中心的高效数据处理系统。
一方面实现了一套基于C++的RDD编程接口和高性能大数据算子库以支持高性能大数据应用的编写；
另一方面提出了动静态结合的计算图使得在容错能力上等同Apache Spark。
同时，在数据预处理、图计算、机器学习等领域典型应用上进行了测试，取得了多个指标的性能优势。
该系统目前已支持TPC-DS中所有99个查询，且在相同环境下相比Apache Spark SQL达到了3.0倍以上的加速比。
该系统可以降低大数据处理的整体延迟、并可节省大数据处理的成本；
同时能够无缝部署在现有Apache Spark环境中，部署简单。因此，智能大数据分析系统具有显著的应用前景。

验证环境

此文档在6节点集群环境下验证，每台节点的环境如下：

条目	配置
服务器型号	浪潮NF5280M6
处理器	Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz * 2
内存	3200 MT/s 64 GB DDR4 内存 * 16
硬盘	NVMe 7TB 固态硬盘 * 8
网卡	Mellanox Technologies MT28908 Family
操作系统	Ubuntu 22.04.2 LTS
内核	Linux 5.15.0-60
C++编译器	Clang 16.0.4, GCC 10.2.0
Java	Oracle JDK 11.0.17 & 8u202
Spark	Apache Spark 3.1.2
HDFS & Yarn	Apache Hadoop 3.3.6
Hive	Apache Hive 3.1.3

如何部署和运行大数据智能分析系统

此文档主要提供了搭建大数据智能分析系统jydata运行环境、搭建TPC-DS测试环境、基于大数据智能分析系统运行TPC-DS查询并进行性能测试等相关内容。
以下为本文档的默认目录结构：
源代码目录：~/jydata/sources
项目的环境变量设置脚本：~/jydata/setenv.sh
依赖软件安装目录：~/jydata/local
第三方软件包目录：~/jydata/third-party
可执行脚本目录：~/jydata/sources/CMR/sbin
本文档默认集群包含node01，node02，……，node06共6个节点，其中node01为大数据智能分析系统运行的Driver所在节点。
如您的实际集群环境不同，请参考本文档根据您的集群情况作适当调整。

首先将本项目压缩包在集群所有节点解压缩到~目录下：

x86_64环境：

jydata@node01~node06$ tar -zxvf jydata_release_YYYY_MM_DD_x64.tar.gz -C ~
jydata@node01~node06$ mv ~/jydata_release_YYYY_MM_DD_x64 ~/jydata

aarch64环境：

jydata@node01~node06$ tar -zxvf jydata_release_YYYY_MM_DD_aarch64.tar.gz -C ~
jydata@node01~node06$ mv ~/jydata_release_YYYY_MM_DD_aarch64 ~/jydata

环境变量

# 将如下内容写入到你的 bashrc 文件并 source 该文件 
export SPARK_HOME=$HOME/jydata/local/spark-3.1.2
export PATH="$SPARK_HOME/bin:$PATH"
export JAVA_HOME=$HOME/jydata/local/jdk-11.0.17
export PATH="$JAVA_HOME/bin:$PATH"
export GCC_10_HOME=$HOME/jydata/local/gcc-10.2.0
export PATH="$GCC_10_HOME/bin:$PATH"
export PATH="$HOME/jydata/local/cmake-3.26.5/bin:$PATH"
export LLVM_16_HOME=$HOME/jydata/local/llvm-16.0.4
export CC="$LLVM_16_HOME/bin/clang"
export CXX="$LLVM_16_HOME/bin/clang++"
export PATH="$LLVM_16_HOME/bin:$PATH"
export HADOOP_HOME="$HOME/jydata/local/hadoop-3.3.6"
export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop"
export PATH="$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH"
export HIVE_HOME="$HOME/jydata/local/apache-hive-3.1.3-bin"
export PATH="$HIVE_HOME/bin:$PATH"

# 如您不方便将上述环境变量配置到 bashrc 文件中进行长期使用，你也可以在我们提供的 setenv.sh 文件中进行正确配置并激活，作为临时使用
jydata@node01$ source ~/jydata/setenv.sh

运行环境搭建部署

1. Spark部署

下载并解压Spark 3.1.2到~/jydata/local目录下。

jydata@node01$ tar -xvf spark-3.1.2-bin-hadoop3.2.tgz -C ~/jydata/local
jydata@node01$ cd ~/jydata/local
jydata@node01$ mv spark-3.1.2-bin-hadoop3.2 spark-3.1.2

~/jydata/setenv.sh为项目的环境变量设置脚本。接下来设置并应用Spark相关的环境变量：

jydata@node01$ echo 'export SPARK_HOME=$HOME/jydata/local/spark-3.1.2' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export PATH="$SPARK_HOME/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh

然后在node01 ~ node06每个节点下载并安装Spark和jydata的依赖，Java JDK 11.0.17。

x86_64环境：

jydata@node01~node06$ tar -zxvf jdk-11.0.17_linux-x64_bin.tar.gz -C ~/jydata/local

aarch64环境：

jydata@node01~node06$ tar -zxvf jdk-11.0.17_linux-aarch64_bin.tar.gz -C ~/jydata/local

在node01设置并应用JDK相关的环境变量：

jydata@node01$ echo 'export JAVA_HOME=$HOME/jydata/local/jdk-11.0.17' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export PATH="$JAVA_HOME/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh
# 查看JDK版本
jydata@node01$ java -version
# 样例输出
java version "11.0.17" 2022-10-18 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.17+10-LTS-269)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.17+10-LTS-269, mixed mode)

2. Lancher

在~/jydata/sources/CMR/cmr-spark-adapter/target/scala-2.12目录下，已提供编译好的cmr-spark-adapter_2.12-0.5.1.jar文件。

3. 下载并安装依赖

3.1 GCC编译安装10.2.0版本

jydata@node01$ mkdir ~/jydata/local/gcc-source
jydata@node01$ tar -zxvf gcc-10.2.0.tar.gz -C ~/jydata/local/gcc-source/
# 设置GCC的编译依赖gmp、mpfr、mpc
jydata@node01$ tar -xvf gmp-6.1.2.tar -C ~/jydata/local/gcc-source/gcc-10.2.0
jydata@node01$ ln -s ~/jydata/local/gcc-source/gcc-10.2.0/gmp-6.1.2 ~/jydata/local/gcc-source/gcc-10.2.0/gmp
jydata@node01$ tar -zxvf mpfr-4.2.0.tar.gz -C ~/jydata/local/gcc-source/gcc-10.2.0/
jydata@node01$ ln -s ~/jydata/local/gcc-source/gcc-10.2.0/mpfr-4.2.0 ~/jydata/local/gcc-source/gcc-10.2.0/mpfr
jydata@node01$ tar -xvf mpc-1.3.1.tar.gz -C ~/jydata/local/gcc-source/gcc-10.2.0/
jydata@node01$ ln -s ~/jydata/local/gcc-source/gcc-10.2.0/mpc-1.3.1 ~/jydata/local/gcc-source/gcc-10.2.0/mpc
# 编译安装GCC
jydata@node01$ mkdir ~/jydata/local/gcc-10.2.0
jydata@node01$ cd ~/jydata/local/gcc-source/gcc-10.2.0
jydata@node01$ patch -p0 < ~/jydata/third-party/patches/libstdc++-v3_Makefile.in.patch # 修复编译可能出现的问题
jydata@node01$ patch -p0 < ~/jydata/third-party/patches/libstdc++-v3_Makefile.am.patch # 修复编译可能出现的问题
jydata@node01$ mkdir build && cd build
jydata@node01$ ../configure --prefix=$HOME/jydata/local/gcc-10.2.0 --enable-checking=release --enable-languages=c,c++ --disable-multilib
jydata@node01$ make -j 32 # make并发数可根据实际情况调整
jydata@node01$ make install
# 添加相关环境变量
jydata@node01$ echo 'export GCC_10_HOME=$HOME/jydata/local/gcc-10.2.0' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export PATH="$GCC_10_HOME/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh
# 查看GCC版本
jydata@node01$ gcc --version
# 样例输出
gcc (GCC) 10.2.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

3.2 cmake 安装3.25以上版本

x86_64环境：

jydata@node01$ tar -zxvf cmake-3.26.5-linux-x86_64.tar.gz -C ~/jydata/local
jydata@node01$ cd ~/jydata/local
jydata@node01$ mv cmake-3.26.5-linux-x86_64 cmake-3.26.5

aarch64环境：

jydata@node01$ tar -zxvf cmake-3.26.5-linux-aarch64.tar.gz -C ~/jydata/local
jydata@node01$ cd ~/jydata/local
jydata@node01$ mv cmake-3.26.5-linux-aarch64 cmake-3.26.5

设置并应用cmake相关的环境变量：

jydata@node01$ echo 'export PATH="$HOME/jydata/local/cmake-3.26.5/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh
# 查看cmake版本
jydata@node01$ cmake --version
# 样例输出
cmake version 3.26.5

CMake suite maintained and supported by Kitware (kitware.com/cmake).

3.3 Clang编译安装16.0.4版本

jydata@node01$ mkdir ~/jydata/local/llvm-source
jydata@node01$ tar -xvf llvm-project-16.0.4.src.tar.xz -C ~/jydata/local/llvm-source/
jydata@node01$ mkdir ~/jydata/local/llvm-16.0.4
jydata@node01$ cd ~/jydata/local/llvm-source/llvm-project-16.0.4.src
jydata@node01$ mkdir build && cd build

先看一下系统自带的GCC版本

jydata@node01$ /usr/bin/gcc --version

如果系统自带的GCC版本高于10.2.0，则执行以下命令，使用系统自带GCC编译Clang：

jydata@node01$ CC=/usr/bin/gcc CXX=/usr/bin/g++ cmake -G "Unix Makefiles" -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_TARGETS_TO_BUILD="host" -DCMAKE_BUILD_TYPE="Release" -DLLVM_INCLUDE_TESTS=OFF -DCMAKE_INSTALL_PREFIX="$HOME/jydata/local/llvm-16.0.4" ../llvm

如果系统自带的GCC版本低于10.2.0，则执行以下命令，使用之前安装的GCC 10.2.0编译Clang：

jydata@node01$ CC=$GCC_10_HOME/bin/gcc CXX=$GCC_10_HOME/bin/g++ cmake -G "Unix Makefiles" -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_TARGETS_TO_BUILD="host" -DCMAKE_BUILD_TYPE="Release" -DLLVM_INCLUDE_TESTS=OFF -DCMAKE_INSTALL_PREFIX="$HOME/jydata/local/llvm-16.0.4" -DGCC_INSTALL_PREFIX=$GCC_10_HOME -DCMAKE_CXX_LINK_FLAGS="-Wl,-rpath,$GCC_10_HOME/lib64 -L$GCC_10_HOME/lib64" ../llvm

然后开始编译：

jydata@node01$ make -j 32 # make并发数可根据实际情况调整
jydata@node01$ make install
# 添加相关环境变量
jydata@node01$ echo 'export LLVM_16_HOME=$HOME/jydata/local/llvm-16.0.4' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export CC="$LLVM_16_HOME/bin/clang"' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export CXX="$LLVM_16_HOME/bin/clang++"' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export PATH="$LLVM_16_HOME/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh
# 查看Clang版本
jydata@node01$ clang --version
# 样例输出
clang version 16.0.4
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/jydata/jydata/local/llvm-16.0.4/bin

最后将编译出的Clang libomp库同步到其它节点的相同路径下：

jydata@node01$ for i in {2..6}; do ssh node0$i "mkdir -p ~/jydata/local/llvm-16.0.4/lib" && scp -r ~/jydata/local/llvm-16.0.4/lib/libomp.so node0$i:~/jydata/local/llvm-16.0.4/lib/; done

3.4 安装Scala工具链

x86_64环境：

jydata@node01$ curl -fL https://github.com/coursier/coursier/releases/latest/download/cs-x86_64-pc-linux.gz | gzip -d > cs && chmod +x cs && ./cs setup

aarch64环境：

jydata@node01$ curl -fL https://github.com/VirtusLab/coursier-m1/releases/latest/download/cs-aarch64-pc-linux.gz | gzip -d > cs && chmod +x cs && ./cs setup

3.5 下载JNA jar包

jydata@node01$ cd ~/jydata/third-party
jydata@node01$ wget https://repo1.maven.org/maven2/net/java/dev/jna/jna/5.6.0/jna-5.6.0.jar
jydata@node01$ mv jna-5.6.0.jar net.java.dev.jna_jna-5.6.0.jar

TPC-DS测试环境搭建

1. TPC-DS测试所需软件部署

1.1 HDFS及Yarn集群部署

在node01 ~ node06节点部署Apache Hadoop HDFS和Yarn集群，需要Apache Hadoop 3.2及以上版本。

首先在node01 ~ node06下载并安装Apache Hadoop和Hive的依赖，Java JDK 8u202。
注意这里的JDK版本与jydata使用的JDK版本不同，原因是Hive 3.1.3不支持JDK 11版本。

x86_64环境：

jydata@node01~node06$ tar -zxvf jdk-8u202-linux-x64.tar.gz -C ~/jydata/local

aarch64环境：

jydata@node01~node06$ tar -zxvf jdk-8u202-linux-arm64-vfp-hflt.tar.gz -C ~/jydata/local

然后在node01 ~ node06每个节点运行以下命令，安装Apache Hadoop：

x86_64环境：

jydata@node01~node06$ tar -zxvf hadoop-3.3.6.tar.gz -C ~/jydata/local

aarch64环境：

jydata@node01~node06$ tar -zxvf hadoop-3.3.6-aarch64.tar.gz -C ~/jydata/local

然后在各集群节点配置HDFS和Yarn。请根据您的集群实际情况进行配置。
~/jydata/third-party/conf/hadoop/下提供了配置文件样例供参考。

jydata@node01~node06$ cd ~/jydata/local/hadoop-3.3.6/etc/hadoop
jydata@node01~node06$ echo "export JAVA_HOME=$HOME/jydata/local/jdk1.8.0_202" >> hadoop-env.sh
jydata@node01~node06$ echo "export HADOOP_HOME=$HOME/jydata/local/hadoop-3.3.6" >> hadoop-env.sh
jydata@node01~node06$ echo "export HADOOP_CONF_DIR=$HOME/jydata/local/hadoop-3.3.6/etc/hadoop" >> hadoop-env.sh
jydata@node01~node06$ vi hadoop-env.sh # 请参考样例根据实际情况作适当配置
jydata@node01~node06$ vi core-site.xml # 请参考样例根据实际情况作适当配置
jydata@node01~node06$ vi hdfs-site.xml # 请参考样例根据实际情况作适当配置
jydata@node01~node06$ vi yarn-site.xml # 请参考样例根据实际情况作适当配置
jydata@node01~node06$ vi workers # 请参考样例根据实际情况作适当配置
jydata@node01~node06$ vi ~/.ssh/authorized_keys # 配置SSH免密码登录，请参考样例根据实际情况作适当配置
# 添加相关环境变量
jydata@node01~node06$ echo 'export HADOOP_HOME="$HOME/jydata/local/hadoop-3.3.6"' >> ~/jydata/setenv.sh
jydata@node01~node06$ echo 'export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop"' >> ~/jydata/setenv.sh
jydata@node01~node06$ echo 'export PATH="$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01~node06$ source ~/jydata/setenv.sh

最后启动HDFS和Yarn集群：

jydata@node01$ for i in {1..6}; do scp $SPARK_HOME/yarn/spark-3.1.2-yarn-shuffle.jar node0$i:$HADOOP_HOME/share/hadoop/yarn/lib/; done
jydata@node01$ $HADOOP_HOME/bin/hdfs namenode -format
jydata@node01$ $HADOOP_HOME/sbin/start-dfs.sh
jydata@node01$ $HADOOP_HOME/sbin/start-yarn.sh
# 检查HDFS和Yarn集群启动是否成功
jydata@node01$ $HADOOP_HOME/bin/hadoop dfsadmin -report
jydata@node01$ $HADOOP_HOME/bin/yarn node -list -showDetails

1.2 Hive部署

大数据智能分析系统使用Apache Hive的MetaStore作为元数据存储。接下来需要在node01节点下载并安装Hive：

jydata@node01$ tar -zxvf apache-hive-3.1.3-bin.tar.gz -C ~/jydata/local

然后配置Hive。请根据您的集群实际情况进行配置。~/jydata/third-party/conf/hive/下提供了配置文件样例供参考。

jydata@node01$ cd ~/jydata/local/apache-hive-3.1.3-bin/conf
jydata@node01$ cp hive-env.sh.template hive-env.sh
jydata@node01$ cp hive-exec-log4j2.properties.template hive-exec-log4j2.properties
jydata@node01$ cp hive-log4j2.properties.template hive-log4j2.properties
jydata@node01$ echo "export HADOOP_HOME=$HOME/jydata/local/hadoop-3.3.6" >> hive-env.sh
jydata@node01$ echo "export HIVE_CONF_DIR=$HOME/jydata/local/apache-hive-3.1.3-bin/conf" >> hive-env.sh
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -mkdir -p /user/hive/warehouse
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -mkdir -p /tmp/hive
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -chmod -R 777 /user/hive/warehouse
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -chmod -R 777 /tmp/hive
jydata@node01$ cp ~/jydata/third-party/conf/hive/hive-site.xml ./hive-site.xml # 若有必要请根据实际情况作适当配置
# 添加相关环境变量
jydata@node01$ echo 'export HIVE_HOME="$HOME/jydata/local/apache-hive-3.1.3-bin"' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export PATH="$HIVE_HOME/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh

最后启动Hive的MetaStore服务。执行以下命令，验证Hive MetaStore服务可启动成功且保持运行：

jydata@node01$ $HIVE_HOME/bin/schematool -dbType derby -initSchema
jydata@node01$ $HIVE_HOME/bin/hive --service metastore

若要使Hive MetaStore服务在后台运行，可使用nohup：

jydata@node01$ nohup $HIVE_HOME/bin/hive --service metastore > /dev/null 2>&1 &

2. TPC-DS测试数据生成

2.1 TPC-DS工具包编译

TPC-DS工具包包含了生成测试数据的程序，需要先对其进行编译：

jydata@node01$ source ~/jydata/setenv.sh
jydata@node01$ git clone https://github.com/databricks/tpcds-kit.git
jydata@node01$ mv ./tpcds-kit ~/jydata/local/DSGen
jydata@node01$ cd ~/jydata/local/DSGen/tools
jydata@node01$ patch -p0 < ~/jydata/third-party/patches/tpcds_makefile.patch
jydata@node01$ make

最后将编译出的TPC-DS工具包同步到其它节点的相同路径下：

jydata@node01$ for i in {2..6}; do scp -r ~/jydata/local/DSGen node0$i:~/jydata/local/; done

2.2 生成TPC-DS 10T测试数据

jydata@node01$ source ~/jydata/setenv.sh
jydata@node01$ git clone https://github.com/databricks/spark-sql-perf.git
jydata@node01$ mv spark-sql-perf ~/jydata/local/
jydata@node01$ cd ~/jydata/local/spark-sql-perf
jydata@node01$ sbt package
jydata@node01$ cd ~/jydata/sources/CMR/sbin/
jydata@node01$ sed -i 's@$HOME@'"$HOME"'@g' tpcds_datagen_parquet.scala
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -mkdir -p /user/hive/tpc-ds-10T/decimaldata
jydata@node01$ ./tpcds_datagen_parquet.sh

# 验证TPC-DS 10T测试数据表生成成功
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -ls /user/hive/tpc-ds-10T/decimaldata
#样例输出
Found 24 items
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/call_center
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/catalog_page
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:25 /user/hive/tpc-ds-10T/decimaldata/catalog_returns
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:23 /user/hive/tpc-ds-10T/decimaldata/catalog_sales
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/customer
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/customer_address
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/customer_demographics
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/date_dim
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/household_demographics
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/income_band
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:25 /user/hive/tpc-ds-10T/decimaldata/inventory
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/item
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/promotion
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/reason
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/ship_mode
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/store
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:30 /user/hive/tpc-ds-10T/decimaldata/store_returns
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:28 /user/hive/tpc-ds-10T/decimaldata/store_sales
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/time_dim
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/warehouse
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/web_page
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/web_returns
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:32 /user/hive/tpc-ds-10T/decimaldata/web_sales
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/web_site

TPC-DS测试运行

运行如下命令，开始运行TPC-DS测试：

jydata@node01$ source ~/jydata/setenv.sh
jydata@node01$ tar -zxvf ~/jydata/third-party/tpcds_3_2.tar.gz -C ~/jydata/local/spark-sql-perf/src/main/resources
jydata@node01$ cd ~/jydata/sources/CMR/sbin/
jydata@node01$ source ../dev/vcpkg/env.sh
jydata@node01$ ./tpcds_test.sh

# tpcds_test.sh的核心命令是通过spark-submit提交任务。其主要参数介绍如下：
spark-submit \
    --master yarn \ #提交任务到Yarn集群
    --deploy-mode client \
    --num-executors 24 \ #executors数量
    --executor-cores 7 \ #每个executor分配的核数
    --executor-memory 30g \ #每个executor分配的内存
    --driver-memory 40g \ #driver分配的内存
    --conf ... \ #各种Spark配置项
    --jars $HOME/jydata/third-party/net.java.dev.jna_jna-5.6.0.jar \ #额外的依赖库，Java JNA
    --class org.pacman.cmr.applications.LaunchSQLTpcdsPerformance \ #程序入口类，通过其中的main函数开始执行
    $HOME/jydata/sources/CMR/cmr-spark-adapter/target/scala-2.12/cmr-spark-adapter_2.12-0.5.1.jar \ #jydata Laucher的jar包
    $HOME/jydata/sources/CMR \ #jydata参数1，jydata项目源代码的根目录
    $HOME/jydata/sources/CMR/build_tpcds \ #jydata参数2，jydata项目的编译目录
    $HOME/jydata/local/spark-sql-perf/src/main/resources/tpcds_3_2 \ #jydata参数3，TPC-DS查询的SQL文件所在目录
    thrift://localhost:9083 \ #jydata参数4，Hive MetaStore服务URI
    ${QID} \ #jydata参数5，运行第几个TPC-DS查询。0~102对应TPC-DS的q1.sql~q99.sql
    ${NUM_REPEATS} \ #jydata参数6，重复运行次数，脚本默认3次
    ${DATABASE_NAME} \ #jydata参数7，Hive元数据库名称，在本文档中是tpcds10t
    2>&1 | tee rel_base_output_sf${SF}_query${QID}.log #运行log的保存文件

~/jydata/sources/CMR/sbin/tpcds_test.sh脚本会依次执行TPC-DS的每个查询。
每个查询分别在jydata和原生Spark中执行3次，并记录在jydata和原生Spark中的执行时间。

执行log保存在~/jydata/sources/CMR/sbin/rel_base_output_sf1_queryXX.log文件。

例如，对TPC-DS的q1.sql查询，查看jydata运行结果：

jydata@node01$ grep 'cmr_ans' rel_base_output_sf1_query0.log
# 样例输出
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 0: AAAAAAAAAAAAAABA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 1: AAAAAAAAAAAAADBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 2: AAAAAAAAAAAAAEDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 3: AAAAAAAAAAAAAOBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 4: AAAAAAAAAAAABEAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 5: AAAAAAAAAAAABNDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 6: AAAAAAAAAAAABOCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 7: AAAAAAAAAAAACCCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 8: AAAAAAAAAAAACDAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 9: AAAAAAAAAAAACEDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 10: AAAAAAAAAAAACGCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 11: AAAAAAAAAAAACLCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 12: AAAAAAAAAAAADLBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 13: AAAAAAAAAAAAEEDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 14: AAAAAAAAAAAAEHDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 15: AAAAAAAAAAAAEOAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 16: AAAAAAAAAAAAFFAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 17: AAAAAAAAAAAAFKAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 18: AAAAAAAAAAAAFMDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 19: AAAAAAAAAAAAGABA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 20: AAAAAAAAAAAAGDBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 21: AAAAAAAAAAAAGDCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 22: AAAAAAAAAAAAGKAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 23: AAAAAAAAAAAAGNBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 24: AAAAAAAAAAAAGPBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 25: AAAAAAAAAAAAHIAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 26: AAAAAAAAAAAAHIDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 27: AAAAAAAAAAAAHLAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 28: AAAAAAAAAAAAICCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 29: AAAAAAAAAAAAIHDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 30: AAAAAAAAAAAAIIAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 31: AAAAAAAAAAAAIJAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 32: AAAAAAAAAAAAIKBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 33: AAAAAAAAAAAAJIBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 34: AAAAAAAAAAAAJLAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 35: AAAAAAAAAAAAKCDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 36: AAAAAAAAAAAAKFAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 37: AAAAAAAAAAAALIBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 38: AAAAAAAAAAAALJAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 39: AAAAAAAAAAAAMGDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 40: AAAAAAAAAAAAMJDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 41: AAAAAAAAAAAANIBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 42: AAAAAAAAAAAAOEDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 43: AAAAAAAAAAAAOHCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 44: AAAAAAAAAAAAOJCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 45: AAAAAAAAAAAAOKAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 46: AAAAAAAAAAAAOOAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 47: AAAAAAAAAAAAPDDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 48: AAAAAAAAAAAAPJDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 49: AAAAAAAAAAAAPNBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 50: AAAAAAAAAAAAPNCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 51: AAAAAAAAAAAAPPBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 52: AAAAAAAAAAABACCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 53: AAAAAAAAAAABADDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 54: AAAAAAAAAAABAIBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 55: AAAAAAAAAAABAMAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 56: AAAAAAAAAAABANBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 57: AAAAAAAAAAABBCDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 58: AAAAAAAAAAABBCDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 59: AAAAAAAAAAABBFDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 60: AAAAAAAAAAABBLDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 61: AAAAAAAAAAABCDCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 62: AAAAAAAAAAABCHCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 63: AAAAAAAAAAABCHCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 64: AAAAAAAAAAABCKBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 65: AAAAAAAAAAABCMCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 66: AAAAAAAAAAABCOAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 67: AAAAAAAAAAABDCCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 68: AAAAAAAAAAABDFDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 69: AAAAAAAAAAABDLDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 70: AAAAAAAAAAABEDAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 71: AAAAAAAAAAABEFCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 72: AAAAAAAAAAABFNBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 73: AAAAAAAAAAABGJAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 74: AAAAAAAAAAABGNCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 75: AAAAAAAAAAABHIDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 76: AAAAAAAAAAABIAAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 77: AAAAAAAAAAABIGCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 78: AAAAAAAAAAABIJAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 79: AAAAAAAAAAABILBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 80: AAAAAAAAAAABJJBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 81: AAAAAAAAAAABKABA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 82: AAAAAAAAAAABKEBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 83: AAAAAAAAAAABKFAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 84: AAAAAAAAAAABKGBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 85: AAAAAAAAAAABKKCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 86: AAAAAAAAAAABKKCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 87: AAAAAAAAAAABLABA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 88: AAAAAAAAAAABLBBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 89: AAAAAAAAAAABLFDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 90: AAAAAAAAAAABLHCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 91: AAAAAAAAAAABLMDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 92: AAAAAAAAAAABLOAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 93: AAAAAAAAAAABLOBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 94: AAAAAAAAAAABMACA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 95: AAAAAAAAAAABMCDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 96: AAAAAAAAAAABMKCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 97: AAAAAAAAAAABMKDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 98: AAAAAAAAAAABNBDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 99: AAAAAAAAAAABNECA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1442] cmr_ans_size: 100
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1444] cmr_ans_display_time: 0.000261 sec
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 0: AAAAAAAAAAAAAABA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 1: AAAAAAAAAAAAADBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 2: AAAAAAAAAAAAAEDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 3: AAAAAAAAAAAAAOBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 4: AAAAAAAAAAAABEAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 5: AAAAAAAAAAAABNDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 6: AAAAAAAAAAAABOCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 7: AAAAAAAAAAAACCCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 8: AAAAAAAAAAAACDAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 9: AAAAAAAAAAAACEDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 10: AAAAAAAAAAAACGCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 11: AAAAAAAAAAAACLCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 12: AAAAAAAAAAAADLBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 13: AAAAAAAAAAAAEEDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 14: AAAAAAAAAAAAEHDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 15: AAAAAAAAAAAAEOAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 16: AAAAAAAAAAAAFFAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 17: AAAAAAAAAAAAFKAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 18: AAAAAAAAAAAAFMDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 19: AAAAAAAAAAAAGABA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 20: AAAAAAAAAAAAGDBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 21: AAAAAAAAAAAAGDCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 22: AAAAAAAAAAAAGKAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 23: AAAAAAAAAAAAGNBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 24: AAAAAAAAAAAAGPBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 25: AAAAAAAAAAAAHIAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 26: AAAAAAAAAAAAHIDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 27: AAAAAAAAAAAAHLAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 28: AAAAAAAAAAAAICCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 29: AAAAAAAAAAAAIHDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 30: AAAAAAAAAAAAIIAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 31: AAAAAAAAAAAAIJAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 32: AAAAAAAAAAAAIKBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 33: AAAAAAAAAAAAJIBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 34: AAAAAAAAAAAAJLAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 35: AAAAAAAAAAAAKCDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 36: AAAAAAAAAAAAKFAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 37: AAAAAAAAAAAALIBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 38: AAAAAAAAAAAALJAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 39: AAAAAAAAAAAAMGDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 40: AAAAAAAAAAAAMJDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 41: AAAAAAAAAAAANIBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 42: AAAAAAAAAAAAOEDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 43: AAAAAAAAAAAAOHCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 44: AAAAAAAAAAAAOJCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 45: AAAAAAAAAAAAOKAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 46: AAAAAAAAAAAAOOAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 47: AAAAAAAAAAAAPDDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 48: AAAAAAAAAAAAPJDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 49: AAAAAAAAAAAAPNBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 50: AAAAAAAAAAAAPNCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 51: AAAAAAAAAAAAPPBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 52: AAAAAAAAAAABACCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 53: AAAAAAAAAAABADDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 54: AAAAAAAAAAABAIBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 55: AAAAAAAAAAABAMAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 56: AAAAAAAAAAABANBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 57: AAAAAAAAAAABBCDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 58: AAAAAAAAAAABBCDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 59: AAAAAAAAAAABBFDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 60: AAAAAAAAAAABBLDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 61: AAAAAAAAAAABCDCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 62: AAAAAAAAAAABCHCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 63: AAAAAAAAAAABCHCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 64: AAAAAAAAAAABCKBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 65: AAAAAAAAAAABCMCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 66: AAAAAAAAAAABCOAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 67: AAAAAAAAAAABDCCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 68: AAAAAAAAAAABDFDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 69: AAAAAAAAAAABDLDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 70: AAAAAAAAAAABEDAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 71: AAAAAAAAAAABEFCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 72: AAAAAAAAAAABFNBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 73: AAAAAAAAAAABGJAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 74: AAAAAAAAAAABGNCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 75: AAAAAAAAAAABHIDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 76: AAAAAAAAAAABIAAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 77: AAAAAAAAAAABIGCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 78: AAAAAAAAAAABIJAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 79: AAAAAAAAAAABILBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 80: AAAAAAAAAAABJJBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 81: AAAAAAAAAAABKABA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 82: AAAAAAAAAAABKEBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 83: AAAAAAAAAAABKFAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 84: AAAAAAAAAAABKGBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 85: AAAAAAAAAAABKKCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 86: AAAAAAAAAAABKKCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 87: AAAAAAAAAAABLABA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 88: AAAAAAAAAAABLBBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 89: AAAAAAAAAAABLFDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 90: AAAAAAAAAAABLHCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 91: AAAAAAAAAAABLMDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 92: AAAAAAAAAAABLOAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 93: AAAAAAAAAAABLOBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 94: AAAAAAAAAAABMACA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 95: AAAAAAAAAAABMCDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 96: AAAAAAAAAAABMKCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 97: AAAAAAAAAAABMKDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 98: AAAAAAAAAAABNBDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 99: AAAAAAAAAAABNECA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1442] cmr_ans_size: 100
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1444] cmr_ans_display_time: 0.000137 sec
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 0: AAAAAAAAAAAAAABA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 1: AAAAAAAAAAAAADBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 2: AAAAAAAAAAAAAEDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 3: AAAAAAAAAAAAAOBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 4: AAAAAAAAAAAABEAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 5: AAAAAAAAAAAABNDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 6: AAAAAAAAAAAABOCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 7: AAAAAAAAAAAACCCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 8: AAAAAAAAAAAACDAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 9: AAAAAAAAAAAACEDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 10: AAAAAAAAAAAACGCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 11: AAAAAAAAAAAACLCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 12: AAAAAAAAAAAADLBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 13: AAAAAAAAAAAAEEDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 14: AAAAAAAAAAAAEHDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 15: AAAAAAAAAAAAEOAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 16: AAAAAAAAAAAAFFAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 17: AAAAAAAAAAAAFKAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 18: AAAAAAAAAAAAFMDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 19: AAAAAAAAAAAAGABA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 20: AAAAAAAAAAAAGDBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 21: AAAAAAAAAAAAGDCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 22: AAAAAAAAAAAAGKAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 23: AAAAAAAAAAAAGNBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 24: AAAAAAAAAAAAGPBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 25: AAAAAAAAAAAAHIAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 26: AAAAAAAAAAAAHIDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 27: AAAAAAAAAAAAHLAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 28: AAAAAAAAAAAAICCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 29: AAAAAAAAAAAAIHDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 30: AAAAAAAAAAAAIIAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 31: AAAAAAAAAAAAIJAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 32: AAAAAAAAAAAAIKBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 33: AAAAAAAAAAAAJIBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 34: AAAAAAAAAAAAJLAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 35: AAAAAAAAAAAAKCDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 36: AAAAAAAAAAAAKFAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 37: AAAAAAAAAAAALIBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 38: AAAAAAAAAAAALJAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 39: AAAAAAAAAAAAMGDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 40: AAAAAAAAAAAAMJDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 41: AAAAAAAAAAAANIBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 42: AAAAAAAAAAAAOEDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 43: AAAAAAAAAAAAOHCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 44: AAAAAAAAAAAAOJCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 45: AAAAAAAAAAAAOKAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 46: AAAAAAAAAAAAOOAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 47: AAAAAAAAAAAAPDDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 48: AAAAAAAAAAAAPJDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 49: AAAAAAAAAAAAPNBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 50: AAAAAAAAAAAAPNCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 51: AAAAAAAAAAAAPPBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 52: AAAAAAAAAAABACCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 53: AAAAAAAAAAABADDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 54: AAAAAAAAAAABAIBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 55: AAAAAAAAAAABAMAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 56: AAAAAAAAAAABANBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 57: AAAAAAAAAAABBCDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 58: AAAAAAAAAAABBCDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 59: AAAAAAAAAAABBFDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 60: AAAAAAAAAAABBLDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 61: AAAAAAAAAAABCDCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 62: AAAAAAAAAAABCHCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 63: AAAAAAAAAAABCHCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 64: AAAAAAAAAAABCKBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 65: AAAAAAAAAAABCMCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 66: AAAAAAAAAAABCOAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 67: AAAAAAAAAAABDCCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 68: AAAAAAAAAAABDFDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 69: AAAAAAAAAAABDLDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 70: AAAAAAAAAAABEDAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 71: AAAAAAAAAAABEFCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 72: AAAAAAAAAAABFNBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 73: AAAAAAAAAAABGJAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 74: AAAAAAAAAAABGNCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 75: AAAAAAAAAAABHIDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 76: AAAAAAAAAAABIAAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 77: AAAAAAAAAAABIGCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 78: AAAAAAAAAAABIJAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 79: AAAAAAAAAAABILBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 80: AAAAAAAAAAABJJBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 81: AAAAAAAAAAABKABA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 82: AAAAAAAAAAABKEBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 83: AAAAAAAAAAABKFAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 84: AAAAAAAAAAABKGBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 85: AAAAAAAAAAABKKCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 86: AAAAAAAAAAABKKCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 87: AAAAAAAAAAABLABA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 88: AAAAAAAAAAABLBBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 89: AAAAAAAAAAABLFDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 90: AAAAAAAAAAABLHCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 91: AAAAAAAAAAABLMDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 92: AAAAAAAAAAABLOAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 93: AAAAAAAAAAABLOBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 94: AAAAAAAAAAABMACA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 95: AAAAAAAAAAABMCDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 96: AAAAAAAAAAABMKCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 97: AAAAAAAAAAABMKDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 98: AAAAAAAAAAABNBDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 99: AAAAAAAAAAABNECA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1442] cmr_ans_size: 100
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1444] cmr_ans_display_time: 7.2e-05 sec

查看原生Spark运行结果：

jydata@node01$ grep 'spark_ans' rel_base_output_sf1_query0.log
# 样例输出
spark_ans 0 0: AAAAAAAAAAAAAABA |
spark_ans 0 1: AAAAAAAAAAAAADBA |
spark_ans 0 2: AAAAAAAAAAAAAEDA |
spark_ans 0 3: AAAAAAAAAAAAAOBA |
spark_ans 0 4: AAAAAAAAAAAABEAA |
spark_ans 0 5: AAAAAAAAAAAABNDA |
spark_ans 0 6: AAAAAAAAAAAABOCA |
spark_ans 0 7: AAAAAAAAAAAACCCA |
spark_ans 0 8: AAAAAAAAAAAACDAA |
spark_ans 0 9: AAAAAAAAAAAACEDA |
spark_ans 0 10: AAAAAAAAAAAACGCA |
spark_ans 0 11: AAAAAAAAAAAACLCA |
spark_ans 0 12: AAAAAAAAAAAADLBA |
spark_ans 0 13: AAAAAAAAAAAAEEDA |
spark_ans 0 14: AAAAAAAAAAAAEHDA |
spark_ans 0 15: AAAAAAAAAAAAEOAA |
spark_ans 0 16: AAAAAAAAAAAAFFAA |
spark_ans 0 17: AAAAAAAAAAAAFKAA |
spark_ans 0 18: AAAAAAAAAAAAFMDA |
spark_ans 0 19: AAAAAAAAAAAAGABA |
spark_ans 0 20: AAAAAAAAAAAAGDBA |
spark_ans 0 21: AAAAAAAAAAAAGDCA |
spark_ans 0 22: AAAAAAAAAAAAGKAA |
spark_ans 0 23: AAAAAAAAAAAAGNBA |
spark_ans 0 24: AAAAAAAAAAAAGPBA |
spark_ans 0 25: AAAAAAAAAAAAHIAA |
spark_ans 0 26: AAAAAAAAAAAAHIDA |
spark_ans 0 27: AAAAAAAAAAAAHLAA |
spark_ans 0 28: AAAAAAAAAAAAICCA |
spark_ans 0 29: AAAAAAAAAAAAIHDA |
spark_ans 0 30: AAAAAAAAAAAAIIAA |
spark_ans 0 31: AAAAAAAAAAAAIJAA |
spark_ans 0 32: AAAAAAAAAAAAIKBA |
spark_ans 0 33: AAAAAAAAAAAAJIBA |
spark_ans 0 34: AAAAAAAAAAAAJLAA |
spark_ans 0 35: AAAAAAAAAAAAKCDA |
spark_ans 0 36: AAAAAAAAAAAAKFAA |
spark_ans 0 37: AAAAAAAAAAAALIBA |
spark_ans 0 38: AAAAAAAAAAAALJAA |
spark_ans 0 39: AAAAAAAAAAAAMGDA |
spark_ans 0 40: AAAAAAAAAAAAMJDA |
spark_ans 0 41: AAAAAAAAAAAANIBA |
spark_ans 0 42: AAAAAAAAAAAAOEDA |
spark_ans 0 43: AAAAAAAAAAAAOHCA |
spark_ans 0 44: AAAAAAAAAAAAOJCA |
spark_ans 0 45: AAAAAAAAAAAAOKAA |
spark_ans 0 46: AAAAAAAAAAAAOOAA |
spark_ans 0 47: AAAAAAAAAAAAPDDA |
spark_ans 0 48: AAAAAAAAAAAAPJDA |
spark_ans 0 49: AAAAAAAAAAAAPNBA |
spark_ans 0 50: AAAAAAAAAAAAPNCA |
spark_ans 0 51: AAAAAAAAAAAAPPBA |
spark_ans 0 52: AAAAAAAAAAABACCA |
spark_ans 0 53: AAAAAAAAAAABADDA |
spark_ans 0 54: AAAAAAAAAAABAIBA |
spark_ans 0 55: AAAAAAAAAAABAMAA |
spark_ans 0 56: AAAAAAAAAAABANBA |
spark_ans 0 57: AAAAAAAAAAABBCDA |
spark_ans 0 58: AAAAAAAAAAABBCDA |
spark_ans 0 59: AAAAAAAAAAABBFDA |
spark_ans 0 60: AAAAAAAAAAABBLDA |
spark_ans 0 61: AAAAAAAAAAABCDCA |
spark_ans 0 62: AAAAAAAAAAABCHCA |
spark_ans 0 63: AAAAAAAAAAABCHCA |
spark_ans 0 64: AAAAAAAAAAABCKBA |
spark_ans 0 65: AAAAAAAAAAABCMCA |
spark_ans 0 66: AAAAAAAAAAABCOAA |
spark_ans 0 67: AAAAAAAAAAABDCCA |
spark_ans 0 68: AAAAAAAAAAABDFDA |
spark_ans 0 69: AAAAAAAAAAABDLDA |
spark_ans 0 70: AAAAAAAAAAABEDAA |
spark_ans 0 71: AAAAAAAAAAABEFCA |
spark_ans 0 72: AAAAAAAAAAABFNBA |
spark_ans 0 73: AAAAAAAAAAABGJAA |
spark_ans 0 74: AAAAAAAAAAABGNCA |
spark_ans 0 75: AAAAAAAAAAABHIDA |
spark_ans 0 76: AAAAAAAAAAABIAAA |
spark_ans 0 77: AAAAAAAAAAABIGCA |
spark_ans 0 78: AAAAAAAAAAABIJAA |
spark_ans 0 79: AAAAAAAAAAABILBA |
spark_ans 0 80: AAAAAAAAAAABJJBA |
spark_ans 0 81: AAAAAAAAAAABKABA |
spark_ans 0 82: AAAAAAAAAAABKEBA |
spark_ans 0 83: AAAAAAAAAAABKFAA |
spark_ans 0 84: AAAAAAAAAAABKGBA |
spark_ans 0 85: AAAAAAAAAAABKKCA |
spark_ans 0 86: AAAAAAAAAAABKKCA |
spark_ans 0 87: AAAAAAAAAAABLABA |
spark_ans 0 88: AAAAAAAAAAABLBBA |
spark_ans 0 89: AAAAAAAAAAABLFDA |
spark_ans 0 90: AAAAAAAAAAABLHCA |
spark_ans 0 91: AAAAAAAAAAABLMDA |
spark_ans 0 92: AAAAAAAAAAABLOAA |
spark_ans 0 93: AAAAAAAAAAABLOBA |
spark_ans 0 94: AAAAAAAAAAABMACA |
spark_ans 0 95: AAAAAAAAAAABMCDA |
spark_ans 0 96: AAAAAAAAAAABMKCA |
spark_ans 0 97: AAAAAAAAAAABMKDA |
spark_ans 0 98: AAAAAAAAAAABNBDA |
spark_ans 0 99: AAAAAAAAAAABNECA |
spark_ans_size: 100 [correct]
spark_ans 0 0: AAAAAAAAAAAAAABA |
spark_ans 0 1: AAAAAAAAAAAAADBA |
spark_ans 0 2: AAAAAAAAAAAAAEDA |
spark_ans 0 3: AAAAAAAAAAAAAOBA |
spark_ans 0 4: AAAAAAAAAAAABEAA |
spark_ans 0 5: AAAAAAAAAAAABNDA |
spark_ans 0 6: AAAAAAAAAAAABOCA |
spark_ans 0 7: AAAAAAAAAAAACCCA |
spark_ans 0 8: AAAAAAAAAAAACDAA |
spark_ans 0 9: AAAAAAAAAAAACEDA |
spark_ans 0 10: AAAAAAAAAAAACGCA |
spark_ans 0 11: AAAAAAAAAAAACLCA |
spark_ans 0 12: AAAAAAAAAAAADLBA |
spark_ans 0 13: AAAAAAAAAAAAEEDA |
spark_ans 0 14: AAAAAAAAAAAAEHDA |
spark_ans 0 15: AAAAAAAAAAAAEOAA |
spark_ans 0 16: AAAAAAAAAAAAFFAA |
spark_ans 0 17: AAAAAAAAAAAAFKAA |
spark_ans 0 18: AAAAAAAAAAAAFMDA |
spark_ans 0 19: AAAAAAAAAAAAGABA |
spark_ans 0 20: AAAAAAAAAAAAGDBA |
spark_ans 0 21: AAAAAAAAAAAAGDCA |
spark_ans 0 22: AAAAAAAAAAAAGKAA |
spark_ans 0 23: AAAAAAAAAAAAGNBA |
spark_ans 0 24: AAAAAAAAAAAAGPBA |
spark_ans 0 25: AAAAAAAAAAAAHIAA |
spark_ans 0 26: AAAAAAAAAAAAHIDA |
spark_ans 0 27: AAAAAAAAAAAAHLAA |
spark_ans 0 28: AAAAAAAAAAAAICCA |
spark_ans 0 29: AAAAAAAAAAAAIHDA |
spark_ans 0 30: AAAAAAAAAAAAIIAA |
spark_ans 0 31: AAAAAAAAAAAAIJAA |
spark_ans 0 32: AAAAAAAAAAAAIKBA |
spark_ans 0 33: AAAAAAAAAAAAJIBA |
spark_ans 0 34: AAAAAAAAAAAAJLAA |
spark_ans 0 35: AAAAAAAAAAAAKCDA |
spark_ans 0 36: AAAAAAAAAAAAKFAA |
spark_ans 0 37: AAAAAAAAAAAALIBA |
spark_ans 0 38: AAAAAAAAAAAALJAA |
spark_ans 0 39: AAAAAAAAAAAAMGDA |
spark_ans 0 40: AAAAAAAAAAAAMJDA |
spark_ans 0 41: AAAAAAAAAAAANIBA |
spark_ans 0 42: AAAAAAAAAAAAOEDA |
spark_ans 0 43: AAAAAAAAAAAAOHCA |
spark_ans 0 44: AAAAAAAAAAAAOJCA |
spark_ans 0 45: AAAAAAAAAAAAOKAA |
spark_ans 0 46: AAAAAAAAAAAAOOAA |
spark_ans 0 47: AAAAAAAAAAAAPDDA |
spark_ans 0 48: AAAAAAAAAAAAPJDA |
spark_ans 0 49: AAAAAAAAAAAAPNBA |
spark_ans 0 50: AAAAAAAAAAAAPNCA |
spark_ans 0 51: AAAAAAAAAAAAPPBA |
spark_ans 0 52: AAAAAAAAAAABACCA |
spark_ans 0 53: AAAAAAAAAAABADDA |
spark_ans 0 54: AAAAAAAAAAABAIBA |
spark_ans 0 55: AAAAAAAAAAABAMAA |
spark_ans 0 56: AAAAAAAAAAABANBA |
spark_ans 0 57: AAAAAAAAAAABBCDA |
spark_ans 0 58: AAAAAAAAAAABBCDA |
spark_ans 0 59: AAAAAAAAAAABBFDA |
spark_ans 0 60: AAAAAAAAAAABBLDA |
spark_ans 0 61: AAAAAAAAAAABCDCA |
spark_ans 0 62: AAAAAAAAAAABCHCA |
spark_ans 0 63: AAAAAAAAAAABCHCA |
spark_ans 0 64: AAAAAAAAAAABCKBA |
spark_ans 0 65: AAAAAAAAAAABCMCA |
spark_ans 0 66: AAAAAAAAAAABCOAA |
spark_ans 0 67: AAAAAAAAAAABDCCA |
spark_ans 0 68: AAAAAAAAAAABDFDA |
spark_ans 0 69: AAAAAAAAAAABDLDA |
spark_ans 0 70: AAAAAAAAAAABEDAA |
spark_ans 0 71: AAAAAAAAAAABEFCA |
spark_ans 0 72: AAAAAAAAAAABFNBA |
spark_ans 0 73: AAAAAAAAAAABGJAA |
spark_ans 0 74: AAAAAAAAAAABGNCA |
spark_ans 0 75: AAAAAAAAAAABHIDA |
spark_ans 0 76: AAAAAAAAAAABIAAA |
spark_ans 0 77: AAAAAAAAAAABIGCA |
spark_ans 0 78: AAAAAAAAAAABIJAA |
spark_ans 0 79: AAAAAAAAAAABILBA |
spark_ans 0 80: AAAAAAAAAAABJJBA |
spark_ans 0 81: AAAAAAAAAAABKABA |
spark_ans 0 82: AAAAAAAAAAABKEBA |
spark_ans 0 83: AAAAAAAAAAABKFAA |
spark_ans 0 84: AAAAAAAAAAABKGBA |
spark_ans 0 85: AAAAAAAAAAABKKCA |
spark_ans 0 86: AAAAAAAAAAABKKCA |
spark_ans 0 87: AAAAAAAAAAABLABA |
spark_ans 0 88: AAAAAAAAAAABLBBA |
spark_ans 0 89: AAAAAAAAAAABLFDA |
spark_ans 0 90: AAAAAAAAAAABLHCA |
spark_ans 0 91: AAAAAAAAAAABLMDA |
spark_ans 0 92: AAAAAAAAAAABLOAA |
spark_ans 0 93: AAAAAAAAAAABLOBA |
spark_ans 0 94: AAAAAAAAAAABMACA |
spark_ans 0 95: AAAAAAAAAAABMCDA |
spark_ans 0 96: AAAAAAAAAAABMKCA |
spark_ans 0 97: AAAAAAAAAAABMKDA |
spark_ans 0 98: AAAAAAAAAAABNBDA |
spark_ans 0 99: AAAAAAAAAAABNECA |
spark_ans_size: 100 [correct]
spark_ans 0 0: AAAAAAAAAAAAAABA |
spark_ans 0 1: AAAAAAAAAAAAADBA |
spark_ans 0 2: AAAAAAAAAAAAAEDA |
spark_ans 0 3: AAAAAAAAAAAAAOBA |
spark_ans 0 4: AAAAAAAAAAAABEAA |
spark_ans 0 5: AAAAAAAAAAAABNDA |
spark_ans 0 6: AAAAAAAAAAAABOCA |
spark_ans 0 7: AAAAAAAAAAAACCCA |
spark_ans 0 8: AAAAAAAAAAAACDAA |
spark_ans 0 9: AAAAAAAAAAAACEDA |
spark_ans 0 10: AAAAAAAAAAAACGCA |
spark_ans 0 11: AAAAAAAAAAAACLCA |
spark_ans 0 12: AAAAAAAAAAAADLBA |
spark_ans 0 13: AAAAAAAAAAAAEEDA |
spark_ans 0 14: AAAAAAAAAAAAEHDA |
spark_ans 0 15: AAAAAAAAAAAAEOAA |
spark_ans 0 16: AAAAAAAAAAAAFFAA |
spark_ans 0 17: AAAAAAAAAAAAFKAA |
spark_ans 0 18: AAAAAAAAAAAAFMDA |
spark_ans 0 19: AAAAAAAAAAAAGABA |
spark_ans 0 20: AAAAAAAAAAAAGDBA |
spark_ans 0 21: AAAAAAAAAAAAGDCA |
spark_ans 0 22: AAAAAAAAAAAAGKAA |
spark_ans 0 23: AAAAAAAAAAAAGNBA |
spark_ans 0 24: AAAAAAAAAAAAGPBA |
spark_ans 0 25: AAAAAAAAAAAAHIAA |
spark_ans 0 26: AAAAAAAAAAAAHIDA |
spark_ans 0 27: AAAAAAAAAAAAHLAA |
spark_ans 0 28: AAAAAAAAAAAAICCA |
spark_ans 0 29: AAAAAAAAAAAAIHDA |
spark_ans 0 30: AAAAAAAAAAAAIIAA |
spark_ans 0 31: AAAAAAAAAAAAIJAA |
spark_ans 0 32: AAAAAAAAAAAAIKBA |
spark_ans 0 33: AAAAAAAAAAAAJIBA |
spark_ans 0 34: AAAAAAAAAAAAJLAA |
spark_ans 0 35: AAAAAAAAAAAAKCDA |
spark_ans 0 36: AAAAAAAAAAAAKFAA |
spark_ans 0 37: AAAAAAAAAAAALIBA |
spark_ans 0 38: AAAAAAAAAAAALJAA |
spark_ans 0 39: AAAAAAAAAAAAMGDA |
spark_ans 0 40: AAAAAAAAAAAAMJDA |
spark_ans 0 41: AAAAAAAAAAAANIBA |
spark_ans 0 42: AAAAAAAAAAAAOEDA |
spark_ans 0 43: AAAAAAAAAAAAOHCA |
spark_ans 0 44: AAAAAAAAAAAAOJCA |
spark_ans 0 45: AAAAAAAAAAAAOKAA |
spark_ans 0 46: AAAAAAAAAAAAOOAA |
spark_ans 0 47: AAAAAAAAAAAAPDDA |
spark_ans 0 48: AAAAAAAAAAAAPJDA |
spark_ans 0 49: AAAAAAAAAAAAPNBA |
spark_ans 0 50: AAAAAAAAAAAAPNCA |
spark_ans 0 51: AAAAAAAAAAAAPPBA |
spark_ans 0 52: AAAAAAAAAAABACCA |
spark_ans 0 53: AAAAAAAAAAABADDA |
spark_ans 0 54: AAAAAAAAAAABAIBA |
spark_ans 0 55: AAAAAAAAAAABAMAA |
spark_ans 0 56: AAAAAAAAAAABANBA |
spark_ans 0 57: AAAAAAAAAAABBCDA |
spark_ans 0 58: AAAAAAAAAAABBCDA |
spark_ans 0 59: AAAAAAAAAAABBFDA |
spark_ans 0 60: AAAAAAAAAAABBLDA |
spark_ans 0 61: AAAAAAAAAAABCDCA |
spark_ans 0 62: AAAAAAAAAAABCHCA |
spark_ans 0 63: AAAAAAAAAAABCHCA |
spark_ans 0 64: AAAAAAAAAAABCKBA |
spark_ans 0 65: AAAAAAAAAAABCMCA |
spark_ans 0 66: AAAAAAAAAAABCOAA |
spark_ans 0 67: AAAAAAAAAAABDCCA |
spark_ans 0 68: AAAAAAAAAAABDFDA |
spark_ans 0 69: AAAAAAAAAAABDLDA |
spark_ans 0 70: AAAAAAAAAAABEDAA |
spark_ans 0 71: AAAAAAAAAAABEFCA |
spark_ans 0 72: AAAAAAAAAAABFNBA |
spark_ans 0 73: AAAAAAAAAAABGJAA |
spark_ans 0 74: AAAAAAAAAAABGNCA |
spark_ans 0 75: AAAAAAAAAAABHIDA |
spark_ans 0 76: AAAAAAAAAAABIAAA |
spark_ans 0 77: AAAAAAAAAAABIGCA |
spark_ans 0 78: AAAAAAAAAAABIJAA |
spark_ans 0 79: AAAAAAAAAAABILBA |
spark_ans 0 80: AAAAAAAAAAABJJBA |
spark_ans 0 81: AAAAAAAAAAABKABA |
spark_ans 0 82: AAAAAAAAAAABKEBA |
spark_ans 0 83: AAAAAAAAAAABKFAA |
spark_ans 0 84: AAAAAAAAAAABKGBA |
spark_ans 0 85: AAAAAAAAAAABKKCA |
spark_ans 0 86: AAAAAAAAAAABKKCA |
spark_ans 0 87: AAAAAAAAAAABLABA |
spark_ans 0 88: AAAAAAAAAAABLBBA |
spark_ans 0 89: AAAAAAAAAAABLFDA |
spark_ans 0 90: AAAAAAAAAAABLHCA |
spark_ans 0 91: AAAAAAAAAAABLMDA |
spark_ans 0 92: AAAAAAAAAAABLOAA |
spark_ans 0 93: AAAAAAAAAAABLOBA |
spark_ans 0 94: AAAAAAAAAAABMACA |
spark_ans 0 95: AAAAAAAAAAABMCDA |
spark_ans 0 96: AAAAAAAAAAABMKCA |
spark_ans 0 97: AAAAAAAAAAABMKDA |
spark_ans 0 98: AAAAAAAAAAABNBDA |
spark_ans 0 99: AAAAAAAAAAABNECA |
spark_ans_size: 100 [correct]

jydata与Spark的运行结果一致，证明jydata运行正确。

查看jydata运行时间：

jydata@node01$ grep 'cmr_time' rel_base_output_sf1_query0.log
# 样例输出
cmr_time 0 10.472 sec
cmr_time 1 5.4510000000000005 sec
cmr_time 2 5.309 sec

查看原生Spark运行时间：

jydata@node01$ grep 'spark_time' rel_base_output_sf1_query0.log
# 样例输出
spark_time 0 16.096 sec ( + 1.932 sec for analysis)
spark_time 1 11.684000000000001 sec ( + 1.359 sec for analysis)
spark_time 2 10.949 sec ( + 1.424 sec for analysis)

取后两次运行时间的平均值作为查询的运行时间，则jydata对q1.sql查询的运行时间为5.38s，原生Spark的运行时间为11.3165s。

62 KiB Raw Blame History Unescape Escape

大数据智能分析系统部署文档20240410

大数据智能分析系统介绍

验证环境

如何部署和运行大数据智能分析系统

环境变量

运行环境搭建部署

1. Spark部署

2. Lancher

3. 下载并安装依赖

3.1 GCC编译安装10.2.0版本

3.2 cmake 安装3.25以上版本

3.3 Clang编译安装16.0.4版本

3.4 安装Scala工具链

3.5 下载JNA jar包

TPC-DS测试环境搭建

1. TPC-DS测试所需软件部署

1.1 HDFS及Yarn集群部署

1.2 Hive部署

2. TPC-DS测试数据生成

2.1 TPC-DS工具包编译

2.2 生成TPC-DS 10T测试数据

TPC-DS测试运行

62 KiB

Raw Blame History