p60419572
/
jydata
forked from jiuyuan/jydata
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
jydata/ProjectDeploy.md

62 KiB
Raw Permalink Blame History

大数据智能分析系统部署文档20240410

大数据智能分析系统介绍

大数据智能分析系统在Apache Spark的基础上构建了一套以性能为中心的高效数据处理系统。
一方面实现了一套基于C++的RDD编程接口和高性能大数据算子库以支持高性能大数据应用的编写；
另一方面提出了动静态结合的计算图使得在容错能力上等同Apache Spark。
同时，在数据预处理、图计算、机器学习等领域典型应用上进行了测试，取得了多个指标的性能优势。
该系统目前已支持TPC-DS中所有99个查询，且在相同环境下相比Apache Spark SQL达到了3.0倍以上的加速比。
该系统可以降低大数据处理的整体延迟、并可节省大数据处理的成本；
同时能够无缝部署在现有Apache Spark环境中，部署简单。因此，智能大数据分析系统具有显著的应用前景。

验证环境

此文档在6节点集群环境下验证，每台节点的环境如下：

条目 配置
服务器型号 浪潮NF5280M6
处理器 Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz * 2
内存 3200 MT/s 64 GB DDR4 内存 * 16
硬盘 NVMe 7TB 固态硬盘 * 8
网卡 Mellanox Technologies MT28908 Family
操作系统 Ubuntu 22.04.2 LTS
内核 Linux 5.15.0-60
C++编译器 Clang 16.0.4, GCC 10.2.0
Java Oracle JDK 11.0.17 & 8u202
Spark Apache Spark 3.1.2
HDFS & Yarn Apache Hadoop 3.3.6
Hive Apache Hive 3.1.3

如何部署和运行大数据智能分析系统

此文档主要提供了搭建大数据智能分析系统jydata运行环境、搭建TPC-DS测试环境、基于大数据智能分析系统运行TPC-DS查询并进行性能测试等相关内容。
以下为本文档的默认目录结构：
源代码目录：~/jydata/sources
项目的环境变量设置脚本：~/jydata/setenv.sh
依赖软件安装目录：~/jydata/local
第三方软件包目录：~/jydata/third-party
可执行脚本目录：~/jydata/sources/CMR/sbin
本文档默认集群包含node01，node02，……，node06共6个节点，其中node01为大数据智能分析系统运行的Driver所在节点。
如您的实际集群环境不同，请参考本文档根据您的集群情况作适当调整。

首先将本项目压缩包在集群所有节点解压缩到~目录下：

x86_64环境：

jydata@node01~node06$ tar -zxvf jydata_release_YYYY_MM_DD_x64.tar.gz -C ~
jydata@node01~node06$ mv ~/jydata_release_YYYY_MM_DD_x64 ~/jydata

aarch64环境：

jydata@node01~node06$ tar -zxvf jydata_release_YYYY_MM_DD_aarch64.tar.gz -C ~
jydata@node01~node06$ mv ~/jydata_release_YYYY_MM_DD_aarch64 ~/jydata

环境变量

# 将如下内容写入到你的 bashrc 文件并 source 该文件 
export SPARK_HOME=$HOME/jydata/local/spark-3.1.2
export PATH="$SPARK_HOME/bin:$PATH"
export JAVA_HOME=$HOME/jydata/local/jdk-11.0.17
export PATH="$JAVA_HOME/bin:$PATH"
export GCC_10_HOME=$HOME/jydata/local/gcc-10.2.0
export PATH="$GCC_10_HOME/bin:$PATH"
export PATH="$HOME/jydata/local/cmake-3.26.5/bin:$PATH"
export LLVM_16_HOME=$HOME/jydata/local/llvm-16.0.4
export CC="$LLVM_16_HOME/bin/clang"
export CXX="$LLVM_16_HOME/bin/clang++"
export PATH="$LLVM_16_HOME/bin:$PATH"
export HADOOP_HOME="$HOME/jydata/local/hadoop-3.3.6"
export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop"
export PATH="$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH"
export HIVE_HOME="$HOME/jydata/local/apache-hive-3.1.3-bin"
export PATH="$HIVE_HOME/bin:$PATH"

# 如您不方便将上述环境变量配置到 bashrc 文件中进行长期使用，你也可以在我们提供的 setenv.sh 文件中进行正确配置并激活，作为临时使用
jydata@node01$ source ~/jydata/setenv.sh

运行环境搭建部署

1. Spark部署

下载并解压Spark 3.1.2到~/jydata/local目录下。

jydata@node01$ tar -xvf spark-3.1.2-bin-hadoop3.2.tgz -C ~/jydata/local
jydata@node01$ cd ~/jydata/local
jydata@node01$ mv spark-3.1.2-bin-hadoop3.2 spark-3.1.2

~/jydata/setenv.sh为项目的环境变量设置脚本。接下来设置并应用Spark相关的环境变量：

jydata@node01$ echo 'export SPARK_HOME=$HOME/jydata/local/spark-3.1.2' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export PATH="$SPARK_HOME/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh

然后在node01 ~ node06每个节点下载并安装Spark和jydata的依赖，Java JDK 11.0.17。

x86_64环境：

jydata@node01~node06$ tar -zxvf jdk-11.0.17_linux-x64_bin.tar.gz -C ~/jydata/local

aarch64环境：

jydata@node01~node06$ tar -zxvf jdk-11.0.17_linux-aarch64_bin.tar.gz -C ~/jydata/local

在node01设置并应用JDK相关的环境变量：

jydata@node01$ echo 'export JAVA_HOME=$HOME/jydata/local/jdk-11.0.17' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export PATH="$JAVA_HOME/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh
# 查看JDK版本
jydata@node01$ java -version
# 样例输出
java version "11.0.17" 2022-10-18 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.17+10-LTS-269)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.17+10-LTS-269, mixed mode)

2. Lancher

在~/jydata/sources/CMR/cmr-spark-adapter/target/scala-2.12目录下，已提供编译好的cmr-spark-adapter_2.12-0.5.1.jar文件。

3. 下载并安装依赖

3.1 GCC编译安装10.2.0版本

jydata@node01$ mkdir ~/jydata/local/gcc-source
jydata@node01$ tar -zxvf gcc-10.2.0.tar.gz -C ~/jydata/local/gcc-source/
# 设置GCC的编译依赖gmp、mpfr、mpc
jydata@node01$ tar -xvf gmp-6.1.2.tar -C ~/jydata/local/gcc-source/gcc-10.2.0
jydata@node01$ ln -s ~/jydata/local/gcc-source/gcc-10.2.0/gmp-6.1.2 ~/jydata/local/gcc-source/gcc-10.2.0/gmp
jydata@node01$ tar -zxvf mpfr-4.2.0.tar.gz -C ~/jydata/local/gcc-source/gcc-10.2.0/
jydata@node01$ ln -s ~/jydata/local/gcc-source/gcc-10.2.0/mpfr-4.2.0 ~/jydata/local/gcc-source/gcc-10.2.0/mpfr
jydata@node01$ tar -xvf mpc-1.3.1.tar.gz -C ~/jydata/local/gcc-source/gcc-10.2.0/
jydata@node01$ ln -s ~/jydata/local/gcc-source/gcc-10.2.0/mpc-1.3.1 ~/jydata/local/gcc-source/gcc-10.2.0/mpc
# 编译安装GCC
jydata@node01$ mkdir ~/jydata/local/gcc-10.2.0
jydata@node01$ cd ~/jydata/local/gcc-source/gcc-10.2.0
jydata@node01$ patch -p0 < ~/jydata/third-party/patches/libstdc++-v3_Makefile.in.patch # 修复编译可能出现的问题
jydata@node01$ patch -p0 < ~/jydata/third-party/patches/libstdc++-v3_Makefile.am.patch # 修复编译可能出现的问题
jydata@node01$ mkdir build && cd build
jydata@node01$ ../configure --prefix=$HOME/jydata/local/gcc-10.2.0 --enable-checking=release --enable-languages=c,c++ --disable-multilib
jydata@node01$ make -j 32 # make并发数可根据实际情况调整
jydata@node01$ make install
# 添加相关环境变量
jydata@node01$ echo 'export GCC_10_HOME=$HOME/jydata/local/gcc-10.2.0' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export PATH="$GCC_10_HOME/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh
# 查看GCC版本
jydata@node01$ gcc --version
# 样例输出
gcc (GCC) 10.2.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

3.2 cmake 安装3.25以上版本

x86_64环境：

jydata@node01$ tar -zxvf cmake-3.26.5-linux-x86_64.tar.gz -C ~/jydata/local
jydata@node01$ cd ~/jydata/local
jydata@node01$ mv cmake-3.26.5-linux-x86_64 cmake-3.26.5

aarch64环境：

jydata@node01$ tar -zxvf cmake-3.26.5-linux-aarch64.tar.gz -C ~/jydata/local
jydata@node01$ cd ~/jydata/local
jydata@node01$ mv cmake-3.26.5-linux-aarch64 cmake-3.26.5

设置并应用cmake相关的环境变量：

jydata@node01$ echo 'export PATH="$HOME/jydata/local/cmake-3.26.5/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh
# 查看cmake版本
jydata@node01$ cmake --version
# 样例输出
cmake version 3.26.5

CMake suite maintained and supported by Kitware (kitware.com/cmake).

3.3 Clang编译安装16.0.4版本

jydata@node01$ mkdir ~/jydata/local/llvm-source
jydata@node01$ tar -xvf llvm-project-16.0.4.src.tar.xz -C ~/jydata/local/llvm-source/
jydata@node01$ mkdir ~/jydata/local/llvm-16.0.4
jydata@node01$ cd ~/jydata/local/llvm-source/llvm-project-16.0.4.src
jydata@node01$ mkdir build && cd build

先看一下系统自带的GCC版本

jydata@node01$ /usr/bin/gcc --version

如果系统自带的GCC版本高于10.2.0，则执行以下命令，使用系统自带GCC编译Clang：

jydata@node01$ CC=/usr/bin/gcc CXX=/usr/bin/g++ cmake -G "Unix Makefiles" -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_TARGETS_TO_BUILD="host" -DCMAKE_BUILD_TYPE="Release" -DLLVM_INCLUDE_TESTS=OFF -DCMAKE_INSTALL_PREFIX="$HOME/jydata/local/llvm-16.0.4" ../llvm

如果系统自带的GCC版本低于10.2.0，则执行以下命令，使用之前安装的GCC 10.2.0编译Clang：

jydata@node01$ CC=$GCC_10_HOME/bin/gcc CXX=$GCC_10_HOME/bin/g++ cmake -G "Unix Makefiles" -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_TARGETS_TO_BUILD="host" -DCMAKE_BUILD_TYPE="Release" -DLLVM_INCLUDE_TESTS=OFF -DCMAKE_INSTALL_PREFIX="$HOME/jydata/local/llvm-16.0.4" -DGCC_INSTALL_PREFIX=$GCC_10_HOME -DCMAKE_CXX_LINK_FLAGS="-Wl,-rpath,$GCC_10_HOME/lib64 -L$GCC_10_HOME/lib64" ../llvm

然后开始编译：

jydata@node01$ make -j 32 # make并发数可根据实际情况调整
jydata@node01$ make install
# 添加相关环境变量
jydata@node01$ echo 'export LLVM_16_HOME=$HOME/jydata/local/llvm-16.0.4' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export CC="$LLVM_16_HOME/bin/clang"' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export CXX="$LLVM_16_HOME/bin/clang++"' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export PATH="$LLVM_16_HOME/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh
# 查看Clang版本
jydata@node01$ clang --version
# 样例输出
clang version 16.0.4
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/jydata/jydata/local/llvm-16.0.4/bin

最后将编译出的Clang libomp库同步到其它节点的相同路径下：

jydata@node01$ for i in {2..6}; do ssh node0$i "mkdir -p ~/jydata/local/llvm-16.0.4/lib" && scp -r ~/jydata/local/llvm-16.0.4/lib/libomp.so node0$i:~/jydata/local/llvm-16.0.4/lib/; done

3.4 安装Scala工具链

x86_64环境：

jydata@node01$ curl -fL https://github.com/coursier/coursier/releases/latest/download/cs-x86_64-pc-linux.gz | gzip -d > cs && chmod +x cs && ./cs setup

aarch64环境：

jydata@node01$ curl -fL https://github.com/VirtusLab/coursier-m1/releases/latest/download/cs-aarch64-pc-linux.gz | gzip -d > cs && chmod +x cs && ./cs setup

3.5 下载JNA jar包

jydata@node01$ cd ~/jydata/third-party
jydata@node01$ wget https://repo1.maven.org/maven2/net/java/dev/jna/jna/5.6.0/jna-5.6.0.jar
jydata@node01$ mv jna-5.6.0.jar net.java.dev.jna_jna-5.6.0.jar

TPC-DS测试环境搭建

1. TPC-DS测试所需软件部署

1.1 HDFS及Yarn集群部署

在node01 ~ node06节点部署Apache Hadoop HDFS和Yarn集群，需要Apache Hadoop 3.2及以上版本。

首先在node01 ~ node06下载并安装Apache Hadoop和Hive的依赖，Java JDK 8u202。
注意这里的JDK版本与jydata使用的JDK版本不同，原因是Hive 3.1.3不支持JDK 11版本。

x86_64环境：

jydata@node01~node06$ tar -zxvf jdk-8u202-linux-x64.tar.gz -C ~/jydata/local

aarch64环境：

jydata@node01~node06$ tar -zxvf jdk-8u202-linux-arm64-vfp-hflt.tar.gz -C ~/jydata/local

然后在node01 ~ node06每个节点运行以下命令，安装Apache Hadoop：

x86_64环境：

jydata@node01~node06$ tar -zxvf hadoop-3.3.6.tar.gz -C ~/jydata/local

aarch64环境：

jydata@node01~node06$ tar -zxvf hadoop-3.3.6-aarch64.tar.gz -C ~/jydata/local

然后在各集群节点配置HDFS和Yarn。请根据您的集群实际情况进行配置。
~/jydata/third-party/conf/hadoop/下提供了配置文件样例供参考。

jydata@node01~node06$ cd ~/jydata/local/hadoop-3.3.6/etc/hadoop
jydata@node01~node06$ echo "export JAVA_HOME=$HOME/jydata/local/jdk1.8.0_202" >> hadoop-env.sh
jydata@node01~node06$ echo "export HADOOP_HOME=$HOME/jydata/local/hadoop-3.3.6" >> hadoop-env.sh
jydata@node01~node06$ echo "export HADOOP_CONF_DIR=$HOME/jydata/local/hadoop-3.3.6/etc/hadoop" >> hadoop-env.sh
jydata@node01~node06$ vi hadoop-env.sh # 请参考样例根据实际情况作适当配置
jydata@node01~node06$ vi core-site.xml # 请参考样例根据实际情况作适当配置
jydata@node01~node06$ vi hdfs-site.xml # 请参考样例根据实际情况作适当配置
jydata@node01~node06$ vi yarn-site.xml # 请参考样例根据实际情况作适当配置
jydata@node01~node06$ vi workers # 请参考样例根据实际情况作适当配置
jydata@node01~node06$ vi ~/.ssh/authorized_keys # 配置SSH免密码登录，请参考样例根据实际情况作适当配置
# 添加相关环境变量
jydata@node01~node06$ echo 'export HADOOP_HOME="$HOME/jydata/local/hadoop-3.3.6"' >> ~/jydata/setenv.sh
jydata@node01~node06$ echo 'export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop"' >> ~/jydata/setenv.sh
jydata@node01~node06$ echo 'export PATH="$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01~node06$ source ~/jydata/setenv.sh

最后启动HDFS和Yarn集群：

jydata@node01$ for i in {1..6}; do scp $SPARK_HOME/yarn/spark-3.1.2-yarn-shuffle.jar node0$i:$HADOOP_HOME/share/hadoop/yarn/lib/; done
jydata@node01$ $HADOOP_HOME/bin/hdfs namenode -format
jydata@node01$ $HADOOP_HOME/sbin/start-dfs.sh
jydata@node01$ $HADOOP_HOME/sbin/start-yarn.sh
# 检查HDFS和Yarn集群启动是否成功
jydata@node01$ $HADOOP_HOME/bin/hadoop dfsadmin -report
jydata@node01$ $HADOOP_HOME/bin/yarn node -list -showDetails

1.2 Hive部署

大数据智能分析系统使用Apache Hive的MetaStore作为元数据存储。 接下来需要在node01节点下载并安装Hive：

jydata@node01$ tar -zxvf apache-hive-3.1.3-bin.tar.gz -C ~/jydata/local

然后配置Hive。请根据您的集群实际情况进行配置。~/jydata/third-party/conf/hive/下提供了配置文件样例供参考。

jydata@node01$ cd ~/jydata/local/apache-hive-3.1.3-bin/conf
jydata@node01$ cp hive-env.sh.template hive-env.sh
jydata@node01$ cp hive-exec-log4j2.properties.template hive-exec-log4j2.properties
jydata@node01$ cp hive-log4j2.properties.template hive-log4j2.properties
jydata@node01$ echo "export HADOOP_HOME=$HOME/jydata/local/hadoop-3.3.6" >> hive-env.sh
jydata@node01$ echo "export HIVE_CONF_DIR=$HOME/jydata/local/apache-hive-3.1.3-bin/conf" >> hive-env.sh
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -mkdir -p /user/hive/warehouse
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -mkdir -p /tmp/hive
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -chmod -R 777 /user/hive/warehouse
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -chmod -R 777 /tmp/hive
jydata@node01$ cp ~/jydata/third-party/conf/hive/hive-site.xml ./hive-site.xml # 若有必要请根据实际情况作适当配置
# 添加相关环境变量
jydata@node01$ echo 'export HIVE_HOME="$HOME/jydata/local/apache-hive-3.1.3-bin"' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export PATH="$HIVE_HOME/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh

最后启动Hive的MetaStore服务。执行以下命令，验证Hive MetaStore服务可启动成功且保持运行：

jydata@node01$ $HIVE_HOME/bin/schematool -dbType derby -initSchema
jydata@node01$ $HIVE_HOME/bin/hive --service metastore

若要使Hive MetaStore服务在后台运行，可使用nohup：

jydata@node01$ nohup $HIVE_HOME/bin/hive --service metastore > /dev/null 2>&1 &

2. TPC-DS测试数据生成

2.1 TPC-DS工具包编译

TPC-DS工具包包含了生成测试数据的程序，需要先对其进行编译：

jydata@node01$ source ~/jydata/setenv.sh
jydata@node01$ git clone https://github.com/databricks/tpcds-kit.git
jydata@node01$ mv ./tpcds-kit ~/jydata/local/DSGen
jydata@node01$ cd ~/jydata/local/DSGen/tools
jydata@node01$ patch -p0 < ~/jydata/third-party/patches/tpcds_makefile.patch
jydata@node01$ make

最后将编译出的TPC-DS工具包同步到其它节点的相同路径下：

jydata@node01$ for i in {2..6}; do scp -r ~/jydata/local/DSGen node0$i:~/jydata/local/; done

2.2 生成TPC-DS 10T测试数据

jydata@node01$ source ~/jydata/setenv.sh
jydata@node01$ git clone https://github.com/databricks/spark-sql-perf.git
jydata@node01$ mv spark-sql-perf ~/jydata/local/
jydata@node01$ cd ~/jydata/local/spark-sql-perf
jydata@node01$ sbt package
jydata@node01$ cd ~/jydata/sources/CMR/sbin/
jydata@node01$ sed -i 's@$HOME@'"$HOME"'@g' tpcds_datagen_parquet.scala
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -mkdir -p /user/hive/tpc-ds-10T/decimaldata
jydata@node01$ ./tpcds_datagen_parquet.sh

# 验证TPC-DS 10T测试数据表生成成功
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -ls /user/hive/tpc-ds-10T/decimaldata
#样例输出
Found 24 items
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/call_center
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/catalog_page
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:25 /user/hive/tpc-ds-10T/decimaldata/catalog_returns
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:23 /user/hive/tpc-ds-10T/decimaldata/catalog_sales
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/customer
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/customer_address
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/customer_demographics
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/date_dim
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/household_demographics
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/income_band
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:25 /user/hive/tpc-ds-10T/decimaldata/inventory
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/item
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/promotion
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/reason
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/ship_mode
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/store
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:30 /user/hive/tpc-ds-10T/decimaldata/store_returns
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:28 /user/hive/tpc-ds-10T/decimaldata/store_sales
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/time_dim
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/warehouse
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/web_page
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/web_returns
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:32 /user/hive/tpc-ds-10T/decimaldata/web_sales
drwxr-xr-x   - jydata supergroup          0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/web_site

TPC-DS测试运行

运行如下命令，开始运行TPC-DS测试：

jydata@node01$ source ~/jydata/setenv.sh
jydata@node01$ tar -zxvf ~/jydata/third-party/tpcds_3_2.tar.gz -C ~/jydata/local/spark-sql-perf/src/main/resources
jydata@node01$ cd ~/jydata/sources/CMR/sbin/
jydata@node01$ source ../dev/vcpkg/env.sh
jydata@node01$ ./tpcds_test.sh

# tpcds_test.sh的核心命令是通过spark-submit提交任务。其主要参数介绍如下：
spark-submit \
    --master yarn \ #提交任务到Yarn集群
    --deploy-mode client \
    --num-executors 24 \ #executors数量
    --executor-cores 7 \ #每个executor分配的核数
    --executor-memory 30g \ #每个executor分配的内存
    --driver-memory 40g \ #driver分配的内存
    --conf ... \ #各种Spark配置项
    --jars $HOME/jydata/third-party/net.java.dev.jna_jna-5.6.0.jar \ #额外的依赖库，Java JNA
    --class org.pacman.cmr.applications.LaunchSQLTpcdsPerformance \ #程序入口类，通过其中的main函数开始执行
    $HOME/jydata/sources/CMR/cmr-spark-adapter/target/scala-2.12/cmr-spark-adapter_2.12-0.5.1.jar \ #jydata Laucher的jar包
    $HOME/jydata/sources/CMR \ #jydata参数1，jydata项目源代码的根目录
    $HOME/jydata/sources/CMR/build_tpcds \ #jydata参数2，jydata项目的编译目录
    $HOME/jydata/local/spark-sql-perf/src/main/resources/tpcds_3_2 \ #jydata参数3，TPC-DS查询的SQL文件所在目录
    thrift://localhost:9083 \ #jydata参数4，Hive MetaStore服务URI
    ${QID} \ #jydata参数5，运行第几个TPC-DS查询。0~102对应TPC-DS的q1.sql~q99.sql
    ${NUM_REPEATS} \ #jydata参数6，重复运行次数，脚本默认3次
    ${DATABASE_NAME} \ #jydata参数7，Hive元数据库名称，在本文档中是tpcds10t
    2>&1 | tee rel_base_output_sf${SF}_query${QID}.log #运行log的保存文件

~/jydata/sources/CMR/sbin/tpcds_test.sh脚本会依次执行TPC-DS的每个查询。
每个查询分别在jydata和原生Spark中执行3次，并记录在jydata和原生Spark中的执行时间。

执行log保存在~/jydata/sources/CMR/sbin/rel_base_output_sf1_queryXX.log文件。

例如，对TPC-DS的q1.sql查询，查看jydata运行结果：

jydata@node01$ grep 'cmr_ans' rel_base_output_sf1_query0.log
# 样例输出
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 0: AAAAAAAAAAAAAABA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 1: AAAAAAAAAAAAADBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 2: AAAAAAAAAAAAAEDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 3: AAAAAAAAAAAAAOBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 4: AAAAAAAAAAAABEAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 5: AAAAAAAAAAAABNDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 6: AAAAAAAAAAAABOCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 7: AAAAAAAAAAAACCCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 8: AAAAAAAAAAAACDAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 9: AAAAAAAAAAAACEDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 10: AAAAAAAAAAAACGCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 11: AAAAAAAAAAAACLCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 12: AAAAAAAAAAAADLBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 13: AAAAAAAAAAAAEEDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 14: AAAAAAAAAAAAEHDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 15: AAAAAAAAAAAAEOAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 16: AAAAAAAAAAAAFFAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 17: AAAAAAAAAAAAFKAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 18: AAAAAAAAAAAAFMDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 19: AAAAAAAAAAAAGABA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 20: AAAAAAAAAAAAGDBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 21: AAAAAAAAAAAAGDCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 22: AAAAAAAAAAAAGKAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 23: AAAAAAAAAAAAGNBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 24: AAAAAAAAAAAAGPBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 25: AAAAAAAAAAAAHIAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 26: AAAAAAAAAAAAHIDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 27: AAAAAAAAAAAAHLAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 28: AAAAAAAAAAAAICCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 29: AAAAAAAAAAAAIHDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 30: AAAAAAAAAAAAIIAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 31: AAAAAAAAAAAAIJAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 32: AAAAAAAAAAAAIKBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 33: AAAAAAAAAAAAJIBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 34: AAAAAAAAAAAAJLAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 35: AAAAAAAAAAAAKCDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 36: AAAAAAAAAAAAKFAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 37: AAAAAAAAAAAALIBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 38: AAAAAAAAAAAALJAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 39: AAAAAAAAAAAAMGDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 40: AAAAAAAAAAAAMJDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 41: AAAAAAAAAAAANIBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 42: AAAAAAAAAAAAOEDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 43: AAAAAAAAAAAAOHCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 44: AAAAAAAAAAAAOJCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 45: AAAAAAAAAAAAOKAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 46: AAAAAAAAAAAAOOAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 47: AAAAAAAAAAAAPDDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 48: AAAAAAAAAAAAPJDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 49: AAAAAAAAAAAAPNBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 50: AAAAAAAAAAAAPNCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 51: AAAAAAAAAAAAPPBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 52: AAAAAAAAAAABACCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 53: AAAAAAAAAAABADDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 54: AAAAAAAAAAABAIBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 55: AAAAAAAAAAABAMAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 56: AAAAAAAAAAABANBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 57: AAAAAAAAAAABBCDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 58: AAAAAAAAAAABBCDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 59: AAAAAAAAAAABBFDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 60: AAAAAAAAAAABBLDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 61: AAAAAAAAAAABCDCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 62: AAAAAAAAAAABCHCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 63: AAAAAAAAAAABCHCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 64: AAAAAAAAAAABCKBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 65: AAAAAAAAAAABCMCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 66: AAAAAAAAAAABCOAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 67: AAAAAAAAAAABDCCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 68: AAAAAAAAAAABDFDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 69: AAAAAAAAAAABDLDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 70: AAAAAAAAAAABEDAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 71: AAAAAAAAAAABEFCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 72: AAAAAAAAAAABFNBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 73: AAAAAAAAAAABGJAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 74: AAAAAAAAAAABGNCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 75: AAAAAAAAAAABHIDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 76: AAAAAAAAAAABIAAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 77: AAAAAAAAAAABIGCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 78: AAAAAAAAAAABIJAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 79: AAAAAAAAAAABILBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 80: AAAAAAAAAAABJJBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 81: AAAAAAAAAAABKABA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 82: AAAAAAAAAAABKEBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 83: AAAAAAAAAAABKFAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 84: AAAAAAAAAAABKGBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 85: AAAAAAAAAAABKKCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 86: AAAAAAAAAAABKKCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 87: AAAAAAAAAAABLABA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 88: AAAAAAAAAAABLBBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 89: AAAAAAAAAAABLFDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 90: AAAAAAAAAAABLHCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 91: AAAAAAAAAAABLMDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 92: AAAAAAAAAAABLOAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 93: AAAAAAAAAAABLOBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 94: AAAAAAAAAAABMACA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 95: AAAAAAAAAAABMCDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 96: AAAAAAAAAAABMKCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 97: AAAAAAAAAAABMKDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 98: AAAAAAAAAAABNBDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 99: AAAAAAAAAAABNECA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1442] cmr_ans_size: 100
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1444] cmr_ans_display_time: 0.000261 sec
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 0: AAAAAAAAAAAAAABA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 1: AAAAAAAAAAAAADBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 2: AAAAAAAAAAAAAEDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 3: AAAAAAAAAAAAAOBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 4: AAAAAAAAAAAABEAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 5: AAAAAAAAAAAABNDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 6: AAAAAAAAAAAABOCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 7: AAAAAAAAAAAACCCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 8: AAAAAAAAAAAACDAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 9: AAAAAAAAAAAACEDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 10: AAAAAAAAAAAACGCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 11: AAAAAAAAAAAACLCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 12: AAAAAAAAAAAADLBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 13: AAAAAAAAAAAAEEDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 14: AAAAAAAAAAAAEHDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 15: AAAAAAAAAAAAEOAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 16: AAAAAAAAAAAAFFAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 17: AAAAAAAAAAAAFKAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 18: AAAAAAAAAAAAFMDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 19: AAAAAAAAAAAAGABA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 20: AAAAAAAAAAAAGDBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 21: AAAAAAAAAAAAGDCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 22: AAAAAAAAAAAAGKAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 23: AAAAAAAAAAAAGNBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 24: AAAAAAAAAAAAGPBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 25: AAAAAAAAAAAAHIAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 26: AAAAAAAAAAAAHIDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 27: AAAAAAAAAAAAHLAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 28: AAAAAAAAAAAAICCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 29: AAAAAAAAAAAAIHDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 30: AAAAAAAAAAAAIIAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 31: AAAAAAAAAAAAIJAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 32: AAAAAAAAAAAAIKBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 33: AAAAAAAAAAAAJIBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 34: AAAAAAAAAAAAJLAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 35: AAAAAAAAAAAAKCDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 36: AAAAAAAAAAAAKFAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 37: AAAAAAAAAAAALIBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 38: AAAAAAAAAAAALJAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 39: AAAAAAAAAAAAMGDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 40: AAAAAAAAAAAAMJDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 41: AAAAAAAAAAAANIBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 42: AAAAAAAAAAAAOEDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 43: AAAAAAAAAAAAOHCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 44: AAAAAAAAAAAAOJCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 45: AAAAAAAAAAAAOKAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 46: AAAAAAAAAAAAOOAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 47: AAAAAAAAAAAAPDDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 48: AAAAAAAAAAAAPJDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 49: AAAAAAAAAAAAPNBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 50: AAAAAAAAAAAAPNCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 51: AAAAAAAAAAAAPPBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 52: AAAAAAAAAAABACCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 53: AAAAAAAAAAABADDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 54: AAAAAAAAAAABAIBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 55: AAAAAAAAAAABAMAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 56: AAAAAAAAAAABANBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 57: AAAAAAAAAAABBCDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 58: AAAAAAAAAAABBCDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 59: AAAAAAAAAAABBFDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 60: AAAAAAAAAAABBLDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 61: AAAAAAAAAAABCDCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 62: AAAAAAAAAAABCHCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 63: AAAAAAAAAAABCHCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 64: AAAAAAAAAAABCKBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 65: AAAAAAAAAAABCMCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 66: AAAAAAAAAAABCOAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 67: AAAAAAAAAAABDCCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 68: AAAAAAAAAAABDFDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 69: AAAAAAAAAAABDLDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 70: AAAAAAAAAAABEDAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 71: AAAAAAAAAAABEFCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 72: AAAAAAAAAAABFNBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 73: AAAAAAAAAAABGJAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 74: AAAAAAAAAAABGNCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 75: AAAAAAAAAAABHIDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 76: AAAAAAAAAAABIAAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 77: AAAAAAAAAAABIGCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 78: AAAAAAAAAAABIJAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 79: AAAAAAAAAAABILBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 80: AAAAAAAAAAABJJBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 81: AAAAAAAAAAABKABA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 82: AAAAAAAAAAABKEBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 83: AAAAAAAAAAABKFAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 84: AAAAAAAAAAABKGBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 85: AAAAAAAAAAABKKCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 86: AAAAAAAAAAABKKCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 87: AAAAAAAAAAABLABA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 88: AAAAAAAAAAABLBBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 89: AAAAAAAAAAABLFDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 90: AAAAAAAAAAABLHCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 91: AAAAAAAAAAABLMDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 92: AAAAAAAAAAABLOAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 93: AAAAAAAAAAABLOBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 94: AAAAAAAAAAABMACA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 95: AAAAAAAAAAABMCDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 96: AAAAAAAAAAABMKCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 97: AAAAAAAAAAABMKDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 98: AAAAAAAAAAABNBDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 99: AAAAAAAAAAABNECA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1442] cmr_ans_size: 100
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1444] cmr_ans_display_time: 0.000137 sec
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 0: AAAAAAAAAAAAAABA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 1: AAAAAAAAAAAAADBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 2: AAAAAAAAAAAAAEDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 3: AAAAAAAAAAAAAOBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 4: AAAAAAAAAAAABEAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 5: AAAAAAAAAAAABNDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 6: AAAAAAAAAAAABOCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 7: AAAAAAAAAAAACCCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 8: AAAAAAAAAAAACDAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 9: AAAAAAAAAAAACEDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 10: AAAAAAAAAAAACGCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 11: AAAAAAAAAAAACLCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 12: AAAAAAAAAAAADLBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 13: AAAAAAAAAAAAEEDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 14: AAAAAAAAAAAAEHDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 15: AAAAAAAAAAAAEOAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 16: AAAAAAAAAAAAFFAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 17: AAAAAAAAAAAAFKAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 18: AAAAAAAAAAAAFMDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 19: AAAAAAAAAAAAGABA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 20: AAAAAAAAAAAAGDBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 21: AAAAAAAAAAAAGDCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 22: AAAAAAAAAAAAGKAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 23: AAAAAAAAAAAAGNBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 24: AAAAAAAAAAAAGPBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 25: AAAAAAAAAAAAHIAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 26: AAAAAAAAAAAAHIDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 27: AAAAAAAAAAAAHLAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 28: AAAAAAAAAAAAICCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 29: AAAAAAAAAAAAIHDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 30: AAAAAAAAAAAAIIAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 31: AAAAAAAAAAAAIJAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 32: AAAAAAAAAAAAIKBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 33: AAAAAAAAAAAAJIBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 34: AAAAAAAAAAAAJLAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 35: AAAAAAAAAAAAKCDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 36: AAAAAAAAAAAAKFAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 37: AAAAAAAAAAAALIBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 38: AAAAAAAAAAAALJAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 39: AAAAAAAAAAAAMGDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 40: AAAAAAAAAAAAMJDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 41: AAAAAAAAAAAANIBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 42: AAAAAAAAAAAAOEDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 43: AAAAAAAAAAAAOHCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 44: AAAAAAAAAAAAOJCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 45: AAAAAAAAAAAAOKAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 46: AAAAAAAAAAAAOOAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 47: AAAAAAAAAAAAPDDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 48: AAAAAAAAAAAAPJDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 49: AAAAAAAAAAAAPNBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 50: AAAAAAAAAAAAPNCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 51: AAAAAAAAAAAAPPBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 52: AAAAAAAAAAABACCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 53: AAAAAAAAAAABADDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 54: AAAAAAAAAAABAIBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 55: AAAAAAAAAAABAMAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 56: AAAAAAAAAAABANBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 57: AAAAAAAAAAABBCDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 58: AAAAAAAAAAABBCDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 59: AAAAAAAAAAABBFDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 60: AAAAAAAAAAABBLDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 61: AAAAAAAAAAABCDCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 62: AAAAAAAAAAABCHCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 63: AAAAAAAAAAABCHCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 64: AAAAAAAAAAABCKBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 65: AAAAAAAAAAABCMCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 66: AAAAAAAAAAABCOAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 67: AAAAAAAAAAABDCCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 68: AAAAAAAAAAABDFDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 69: AAAAAAAAAAABDLDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 70: AAAAAAAAAAABEDAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 71: AAAAAAAAAAABEFCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 72: AAAAAAAAAAABFNBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 73: AAAAAAAAAAABGJAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 74: AAAAAAAAAAABGNCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 75: AAAAAAAAAAABHIDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 76: AAAAAAAAAAABIAAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 77: AAAAAAAAAAABIGCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 78: AAAAAAAAAAABIJAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 79: AAAAAAAAAAABILBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 80: AAAAAAAAAAABJJBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 81: AAAAAAAAAAABKABA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 82: AAAAAAAAAAABKEBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 83: AAAAAAAAAAABKFAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 84: AAAAAAAAAAABKGBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 85: AAAAAAAAAAABKKCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 86: AAAAAAAAAAABKKCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 87: AAAAAAAAAAABLABA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 88: AAAAAAAAAAABLBBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 89: AAAAAAAAAAABLFDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 90: AAAAAAAAAAABLHCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 91: AAAAAAAAAAABLMDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 92: AAAAAAAAAAABLOAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 93: AAAAAAAAAAABLOBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 94: AAAAAAAAAAABMACA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 95: AAAAAAAAAAABMCDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 96: AAAAAAAAAAABMKCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 97: AAAAAAAAAAABMKDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 98: AAAAAAAAAAABNBDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 99: AAAAAAAAAAABNECA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1442] cmr_ans_size: 100
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1444] cmr_ans_display_time: 7.2e-05 sec

查看原生Spark运行结果：

jydata@node01$ grep 'spark_ans' rel_base_output_sf1_query0.log
# 样例输出
spark_ans 0 0: AAAAAAAAAAAAAABA |
spark_ans 0 1: AAAAAAAAAAAAADBA |
spark_ans 0 2: AAAAAAAAAAAAAEDA |
spark_ans 0 3: AAAAAAAAAAAAAOBA |
spark_ans 0 4: AAAAAAAAAAAABEAA |
spark_ans 0 5: AAAAAAAAAAAABNDA |
spark_ans 0 6: AAAAAAAAAAAABOCA |
spark_ans 0 7: AAAAAAAAAAAACCCA |
spark_ans 0 8: AAAAAAAAAAAACDAA |
spark_ans 0 9: AAAAAAAAAAAACEDA |
spark_ans 0 10: AAAAAAAAAAAACGCA |
spark_ans 0 11: AAAAAAAAAAAACLCA |
spark_ans 0 12: AAAAAAAAAAAADLBA |
spark_ans 0 13: AAAAAAAAAAAAEEDA |
spark_ans 0 14: AAAAAAAAAAAAEHDA |
spark_ans 0 15: AAAAAAAAAAAAEOAA |
spark_ans 0 16: AAAAAAAAAAAAFFAA |
spark_ans 0 17: AAAAAAAAAAAAFKAA |
spark_ans 0 18: AAAAAAAAAAAAFMDA |
spark_ans 0 19: AAAAAAAAAAAAGABA |
spark_ans 0 20: AAAAAAAAAAAAGDBA |
spark_ans 0 21: AAAAAAAAAAAAGDCA |
spark_ans 0 22: AAAAAAAAAAAAGKAA |
spark_ans 0 23: AAAAAAAAAAAAGNBA |
spark_ans 0 24: AAAAAAAAAAAAGPBA |
spark_ans 0 25: AAAAAAAAAAAAHIAA |
spark_ans 0 26: AAAAAAAAAAAAHIDA |
spark_ans 0 27: AAAAAAAAAAAAHLAA |
spark_ans 0 28: AAAAAAAAAAAAICCA |
spark_ans 0 29: AAAAAAAAAAAAIHDA |
spark_ans 0 30: AAAAAAAAAAAAIIAA |
spark_ans 0 31: AAAAAAAAAAAAIJAA |
spark_ans 0 32: AAAAAAAAAAAAIKBA |
spark_ans 0 33: AAAAAAAAAAAAJIBA |
spark_ans 0 34: AAAAAAAAAAAAJLAA |
spark_ans 0 35: AAAAAAAAAAAAKCDA |
spark_ans 0 36: AAAAAAAAAAAAKFAA |
spark_ans 0 37: AAAAAAAAAAAALIBA |
spark_ans 0 38: AAAAAAAAAAAALJAA |
spark_ans 0 39: AAAAAAAAAAAAMGDA |
spark_ans 0 40: AAAAAAAAAAAAMJDA |
spark_ans 0 41: AAAAAAAAAAAANIBA |
spark_ans 0 42: AAAAAAAAAAAAOEDA |
spark_ans 0 43: AAAAAAAAAAAAOHCA |
spark_ans 0 44: AAAAAAAAAAAAOJCA |
spark_ans 0 45: AAAAAAAAAAAAOKAA |
spark_ans 0 46: AAAAAAAAAAAAOOAA |
spark_ans 0 47: AAAAAAAAAAAAPDDA |
spark_ans 0 48: AAAAAAAAAAAAPJDA |
spark_ans 0 49: AAAAAAAAAAAAPNBA |
spark_ans 0 50: AAAAAAAAAAAAPNCA |
spark_ans 0 51: AAAAAAAAAAAAPPBA |
spark_ans 0 52: AAAAAAAAAAABACCA |
spark_ans 0 53: AAAAAAAAAAABADDA |
spark_ans 0 54: AAAAAAAAAAABAIBA |
spark_ans 0 55: AAAAAAAAAAABAMAA |
spark_ans 0 56: AAAAAAAAAAABANBA |
spark_ans 0 57: AAAAAAAAAAABBCDA |
spark_ans 0 58: AAAAAAAAAAABBCDA |
spark_ans 0 59: AAAAAAAAAAABBFDA |
spark_ans 0 60: AAAAAAAAAAABBLDA |
spark_ans 0 61: AAAAAAAAAAABCDCA |
spark_ans 0 62: AAAAAAAAAAABCHCA |
spark_ans 0 63: AAAAAAAAAAABCHCA |
spark_ans 0 64: AAAAAAAAAAABCKBA |
spark_ans 0 65: AAAAAAAAAAABCMCA |
spark_ans 0 66: AAAAAAAAAAABCOAA |
spark_ans 0 67: AAAAAAAAAAABDCCA |
spark_ans 0 68: AAAAAAAAAAABDFDA |
spark_ans 0 69: AAAAAAAAAAABDLDA |
spark_ans 0 70: AAAAAAAAAAABEDAA |
spark_ans 0 71: AAAAAAAAAAABEFCA |
spark_ans 0 72: AAAAAAAAAAABFNBA |
spark_ans 0 73: AAAAAAAAAAABGJAA |
spark_ans 0 74: AAAAAAAAAAABGNCA |
spark_ans 0 75: AAAAAAAAAAABHIDA |
spark_ans 0 76: AAAAAAAAAAABIAAA |
spark_ans 0 77: AAAAAAAAAAABIGCA |
spark_ans 0 78: AAAAAAAAAAABIJAA |
spark_ans 0 79: AAAAAAAAAAABILBA |
spark_ans 0 80: AAAAAAAAAAABJJBA |
spark_ans 0 81: AAAAAAAAAAABKABA |
spark_ans 0 82: AAAAAAAAAAABKEBA |
spark_ans 0 83: AAAAAAAAAAABKFAA |
spark_ans 0 84: AAAAAAAAAAABKGBA |
spark_ans 0 85: AAAAAAAAAAABKKCA |
spark_ans 0 86: AAAAAAAAAAABKKCA |
spark_ans 0 87: AAAAAAAAAAABLABA |
spark_ans 0 88: AAAAAAAAAAABLBBA |
spark_ans 0 89: AAAAAAAAAAABLFDA |
spark_ans 0 90: AAAAAAAAAAABLHCA |
spark_ans 0 91: AAAAAAAAAAABLMDA |
spark_ans 0 92: AAAAAAAAAAABLOAA |
spark_ans 0 93: AAAAAAAAAAABLOBA |
spark_ans 0 94: AAAAAAAAAAABMACA |
spark_ans 0 95: AAAAAAAAAAABMCDA |
spark_ans 0 96: AAAAAAAAAAABMKCA |
spark_ans 0 97: AAAAAAAAAAABMKDA |
spark_ans 0 98: AAAAAAAAAAABNBDA |
spark_ans 0 99: AAAAAAAAAAABNECA |
spark_ans_size: 100 [correct]
spark_ans 0 0: AAAAAAAAAAAAAABA |
spark_ans 0 1: AAAAAAAAAAAAADBA |
spark_ans 0 2: AAAAAAAAAAAAAEDA |
spark_ans 0 3: AAAAAAAAAAAAAOBA |
spark_ans 0 4: AAAAAAAAAAAABEAA |
spark_ans 0 5: AAAAAAAAAAAABNDA |
spark_ans 0 6: AAAAAAAAAAAABOCA |
spark_ans 0 7: AAAAAAAAAAAACCCA |
spark_ans 0 8: AAAAAAAAAAAACDAA |
spark_ans 0 9: AAAAAAAAAAAACEDA |
spark_ans 0 10: AAAAAAAAAAAACGCA |
spark_ans 0 11: AAAAAAAAAAAACLCA |
spark_ans 0 12: AAAAAAAAAAAADLBA |
spark_ans 0 13: AAAAAAAAAAAAEEDA |
spark_ans 0 14: AAAAAAAAAAAAEHDA |
spark_ans 0 15: AAAAAAAAAAAAEOAA |
spark_ans 0 16: AAAAAAAAAAAAFFAA |
spark_ans 0 17: AAAAAAAAAAAAFKAA |
spark_ans 0 18: AAAAAAAAAAAAFMDA |
spark_ans 0 19: AAAAAAAAAAAAGABA |
spark_ans 0 20: AAAAAAAAAAAAGDBA |
spark_ans 0 21: AAAAAAAAAAAAGDCA |
spark_ans 0 22: AAAAAAAAAAAAGKAA |
spark_ans 0 23: AAAAAAAAAAAAGNBA |
spark_ans 0 24: AAAAAAAAAAAAGPBA |
spark_ans 0 25: AAAAAAAAAAAAHIAA |
spark_ans 0 26: AAAAAAAAAAAAHIDA |
spark_ans 0 27: AAAAAAAAAAAAHLAA |
spark_ans 0 28: AAAAAAAAAAAAICCA |
spark_ans 0 29: AAAAAAAAAAAAIHDA |
spark_ans 0 30: AAAAAAAAAAAAIIAA |
spark_ans 0 31: AAAAAAAAAAAAIJAA |
spark_ans 0 32: AAAAAAAAAAAAIKBA |
spark_ans 0 33: AAAAAAAAAAAAJIBA |
spark_ans 0 34: AAAAAAAAAAAAJLAA |
spark_ans 0 35: AAAAAAAAAAAAKCDA |
spark_ans 0 36: AAAAAAAAAAAAKFAA |
spark_ans 0 37: AAAAAAAAAAAALIBA |
spark_ans 0 38: AAAAAAAAAAAALJAA |
spark_ans 0 39: AAAAAAAAAAAAMGDA |
spark_ans 0 40: AAAAAAAAAAAAMJDA |
spark_ans 0 41: AAAAAAAAAAAANIBA |
spark_ans 0 42: AAAAAAAAAAAAOEDA |
spark_ans 0 43: AAAAAAAAAAAAOHCA |
spark_ans 0 44: AAAAAAAAAAAAOJCA |
spark_ans 0 45: AAAAAAAAAAAAOKAA |
spark_ans 0 46: AAAAAAAAAAAAOOAA |
spark_ans 0 47: AAAAAAAAAAAAPDDA |
spark_ans 0 48: AAAAAAAAAAAAPJDA |
spark_ans 0 49: AAAAAAAAAAAAPNBA |
spark_ans 0 50: AAAAAAAAAAAAPNCA |
spark_ans 0 51: AAAAAAAAAAAAPPBA |
spark_ans 0 52: AAAAAAAAAAABACCA |
spark_ans 0 53: AAAAAAAAAAABADDA |
spark_ans 0 54: AAAAAAAAAAABAIBA |
spark_ans 0 55: AAAAAAAAAAABAMAA |
spark_ans 0 56: AAAAAAAAAAABANBA |
spark_ans 0 57: AAAAAAAAAAABBCDA |
spark_ans 0 58: AAAAAAAAAAABBCDA |
spark_ans 0 59: AAAAAAAAAAABBFDA |
spark_ans 0 60: AAAAAAAAAAABBLDA |
spark_ans 0 61: AAAAAAAAAAABCDCA |
spark_ans 0 62: AAAAAAAAAAABCHCA |
spark_ans 0 63: AAAAAAAAAAABCHCA |
spark_ans 0 64: AAAAAAAAAAABCKBA |
spark_ans 0 65: AAAAAAAAAAABCMCA |
spark_ans 0 66: AAAAAAAAAAABCOAA |
spark_ans 0 67: AAAAAAAAAAABDCCA |
spark_ans 0 68: AAAAAAAAAAABDFDA |
spark_ans 0 69: AAAAAAAAAAABDLDA |
spark_ans 0 70: AAAAAAAAAAABEDAA |
spark_ans 0 71: AAAAAAAAAAABEFCA |
spark_ans 0 72: AAAAAAAAAAABFNBA |
spark_ans 0 73: AAAAAAAAAAABGJAA |
spark_ans 0 74: AAAAAAAAAAABGNCA |
spark_ans 0 75: AAAAAAAAAAABHIDA |
spark_ans 0 76: AAAAAAAAAAABIAAA |
spark_ans 0 77: AAAAAAAAAAABIGCA |
spark_ans 0 78: AAAAAAAAAAABIJAA |
spark_ans 0 79: AAAAAAAAAAABILBA |
spark_ans 0 80: AAAAAAAAAAABJJBA |
spark_ans 0 81: AAAAAAAAAAABKABA |
spark_ans 0 82: AAAAAAAAAAABKEBA |
spark_ans 0 83: AAAAAAAAAAABKFAA |
spark_ans 0 84: AAAAAAAAAAABKGBA |
spark_ans 0 85: AAAAAAAAAAABKKCA |
spark_ans 0 86: AAAAAAAAAAABKKCA |
spark_ans 0 87: AAAAAAAAAAABLABA |
spark_ans 0 88: AAAAAAAAAAABLBBA |
spark_ans 0 89: AAAAAAAAAAABLFDA |
spark_ans 0 90: AAAAAAAAAAABLHCA |
spark_ans 0 91: AAAAAAAAAAABLMDA |
spark_ans 0 92: AAAAAAAAAAABLOAA |
spark_ans 0 93: AAAAAAAAAAABLOBA |
spark_ans 0 94: AAAAAAAAAAABMACA |
spark_ans 0 95: AAAAAAAAAAABMCDA |
spark_ans 0 96: AAAAAAAAAAABMKCA |
spark_ans 0 97: AAAAAAAAAAABMKDA |
spark_ans 0 98: AAAAAAAAAAABNBDA |
spark_ans 0 99: AAAAAAAAAAABNECA |
spark_ans_size: 100 [correct]
spark_ans 0 0: AAAAAAAAAAAAAABA |
spark_ans 0 1: AAAAAAAAAAAAADBA |
spark_ans 0 2: AAAAAAAAAAAAAEDA |
spark_ans 0 3: AAAAAAAAAAAAAOBA |
spark_ans 0 4: AAAAAAAAAAAABEAA |
spark_ans 0 5: AAAAAAAAAAAABNDA |
spark_ans 0 6: AAAAAAAAAAAABOCA |
spark_ans 0 7: AAAAAAAAAAAACCCA |
spark_ans 0 8: AAAAAAAAAAAACDAA |
spark_ans 0 9: AAAAAAAAAAAACEDA |
spark_ans 0 10: AAAAAAAAAAAACGCA |
spark_ans 0 11: AAAAAAAAAAAACLCA |
spark_ans 0 12: AAAAAAAAAAAADLBA |
spark_ans 0 13: AAAAAAAAAAAAEEDA |
spark_ans 0 14: AAAAAAAAAAAAEHDA |
spark_ans 0 15: AAAAAAAAAAAAEOAA |
spark_ans 0 16: AAAAAAAAAAAAFFAA |
spark_ans 0 17: AAAAAAAAAAAAFKAA |
spark_ans 0 18: AAAAAAAAAAAAFMDA |
spark_ans 0 19: AAAAAAAAAAAAGABA |
spark_ans 0 20: AAAAAAAAAAAAGDBA |
spark_ans 0 21: AAAAAAAAAAAAGDCA |
spark_ans 0 22: AAAAAAAAAAAAGKAA |
spark_ans 0 23: AAAAAAAAAAAAGNBA |
spark_ans 0 24: AAAAAAAAAAAAGPBA |
spark_ans 0 25: AAAAAAAAAAAAHIAA |
spark_ans 0 26: AAAAAAAAAAAAHIDA |
spark_ans 0 27: AAAAAAAAAAAAHLAA |
spark_ans 0 28: AAAAAAAAAAAAICCA |
spark_ans 0 29: AAAAAAAAAAAAIHDA |
spark_ans 0 30: AAAAAAAAAAAAIIAA |
spark_ans 0 31: AAAAAAAAAAAAIJAA |
spark_ans 0 32: AAAAAAAAAAAAIKBA |
spark_ans 0 33: AAAAAAAAAAAAJIBA |
spark_ans 0 34: AAAAAAAAAAAAJLAA |
spark_ans 0 35: AAAAAAAAAAAAKCDA |
spark_ans 0 36: AAAAAAAAAAAAKFAA |
spark_ans 0 37: AAAAAAAAAAAALIBA |
spark_ans 0 38: AAAAAAAAAAAALJAA |
spark_ans 0 39: AAAAAAAAAAAAMGDA |
spark_ans 0 40: AAAAAAAAAAAAMJDA |
spark_ans 0 41: AAAAAAAAAAAANIBA |
spark_ans 0 42: AAAAAAAAAAAAOEDA |
spark_ans 0 43: AAAAAAAAAAAAOHCA |
spark_ans 0 44: AAAAAAAAAAAAOJCA |
spark_ans 0 45: AAAAAAAAAAAAOKAA |
spark_ans 0 46: AAAAAAAAAAAAOOAA |
spark_ans 0 47: AAAAAAAAAAAAPDDA |
spark_ans 0 48: AAAAAAAAAAAAPJDA |
spark_ans 0 49: AAAAAAAAAAAAPNBA |
spark_ans 0 50: AAAAAAAAAAAAPNCA |
spark_ans 0 51: AAAAAAAAAAAAPPBA |
spark_ans 0 52: AAAAAAAAAAABACCA |
spark_ans 0 53: AAAAAAAAAAABADDA |
spark_ans 0 54: AAAAAAAAAAABAIBA |
spark_ans 0 55: AAAAAAAAAAABAMAA |
spark_ans 0 56: AAAAAAAAAAABANBA |
spark_ans 0 57: AAAAAAAAAAABBCDA |
spark_ans 0 58: AAAAAAAAAAABBCDA |
spark_ans 0 59: AAAAAAAAAAABBFDA |
spark_ans 0 60: AAAAAAAAAAABBLDA |
spark_ans 0 61: AAAAAAAAAAABCDCA |
spark_ans 0 62: AAAAAAAAAAABCHCA |
spark_ans 0 63: AAAAAAAAAAABCHCA |
spark_ans 0 64: AAAAAAAAAAABCKBA |
spark_ans 0 65: AAAAAAAAAAABCMCA |
spark_ans 0 66: AAAAAAAAAAABCOAA |
spark_ans 0 67: AAAAAAAAAAABDCCA |
spark_ans 0 68: AAAAAAAAAAABDFDA |
spark_ans 0 69: AAAAAAAAAAABDLDA |
spark_ans 0 70: AAAAAAAAAAABEDAA |
spark_ans 0 71: AAAAAAAAAAABEFCA |
spark_ans 0 72: AAAAAAAAAAABFNBA |
spark_ans 0 73: AAAAAAAAAAABGJAA |
spark_ans 0 74: AAAAAAAAAAABGNCA |
spark_ans 0 75: AAAAAAAAAAABHIDA |
spark_ans 0 76: AAAAAAAAAAABIAAA |
spark_ans 0 77: AAAAAAAAAAABIGCA |
spark_ans 0 78: AAAAAAAAAAABIJAA |
spark_ans 0 79: AAAAAAAAAAABILBA |
spark_ans 0 80: AAAAAAAAAAABJJBA |
spark_ans 0 81: AAAAAAAAAAABKABA |
spark_ans 0 82: AAAAAAAAAAABKEBA |
spark_ans 0 83: AAAAAAAAAAABKFAA |
spark_ans 0 84: AAAAAAAAAAABKGBA |
spark_ans 0 85: AAAAAAAAAAABKKCA |
spark_ans 0 86: AAAAAAAAAAABKKCA |
spark_ans 0 87: AAAAAAAAAAABLABA |
spark_ans 0 88: AAAAAAAAAAABLBBA |
spark_ans 0 89: AAAAAAAAAAABLFDA |
spark_ans 0 90: AAAAAAAAAAABLHCA |
spark_ans 0 91: AAAAAAAAAAABLMDA |
spark_ans 0 92: AAAAAAAAAAABLOAA |
spark_ans 0 93: AAAAAAAAAAABLOBA |
spark_ans 0 94: AAAAAAAAAAABMACA |
spark_ans 0 95: AAAAAAAAAAABMCDA |
spark_ans 0 96: AAAAAAAAAAABMKCA |
spark_ans 0 97: AAAAAAAAAAABMKDA |
spark_ans 0 98: AAAAAAAAAAABNBDA |
spark_ans 0 99: AAAAAAAAAAABNECA |
spark_ans_size: 100 [correct]

jydata与Spark的运行结果一致，证明jydata运行正确。

查看jydata运行时间：

jydata@node01$ grep 'cmr_time' rel_base_output_sf1_query0.log
# 样例输出
cmr_time 0 10.472 sec
cmr_time 1 5.4510000000000005 sec
cmr_time 2 5.309 sec

查看原生Spark运行时间：

jydata@node01$ grep 'spark_time' rel_base_output_sf1_query0.log
# 样例输出
spark_time 0 16.096 sec ( + 1.932 sec for analysis)
spark_time 1 11.684000000000001 sec ( + 1.359 sec for analysis)
spark_time 2 10.949 sec ( + 1.424 sec for analysis)

取后两次运行时间的平均值作为查询的运行时间，则jydata对q1.sql查询的运行时间为5.38s，原生Spark的运行时间为11.3165s。