jiuyuan
/
jydata
3
0
Fork 16
Code Issues Pull Requests Projects Releases Wiki Activity
jydata/ProjectDeploy.md

1126 lines
62 KiB
Markdown
Raw Normal View History

deploy doc
2024-04-09 17:25:12 +08:00
update
2024-04-10 10:10:11 +08:00
# 大数据智能分析系统部署文档20240410
deploy doc
2024-04-09 17:25:12 +08:00
## 大数据智能分析系统介绍
大数据智能分析系统在Apache Spark的基础上构建了一套以性能为中心的高效数据处理系统。
一方面实现了一套基于C++的RDD编程接口和高性能大数据算子库以支持高性能大数据应用的编写；
另一方面提出了动静态结合的计算图使得在容错能力上等同Apache Spark。
同时，在数据预处理、图计算、机器学习等领域典型应用上进行了测试，取得了多个指标的性能优势。
update
2024-04-10 10:51:59 +08:00
该系统目前已支持TPC-DS中所有99个查询，且在相同环境下相比Apache Spark SQL达到了3.0倍以上的加速比。
deploy doc
2024-04-09 17:25:12 +08:00
该系统可以降低大数据处理的整体延迟、并可节省大数据处理的成本；
同时能够无缝部署在现有Apache Spark环境中，部署简单。因此，智能大数据分析系统具有显著的应用前景。
## 验证环境
此文档在6节点集群环境下验证，每台节点的环境如下：
| 条目 | 配置 |
| ------ |--------------------------------------------------|
| 服务器型号 | 浪潮NF5280M6 |
| 处理器 | Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz * 2 |
| 内存 | 3200 MT/s 64 GB DDR4 内存 * 16 |
| 硬盘 | NVMe 7TB 固态硬盘 * 8 |
| 网卡 | Mellanox Technologies MT28908 Family |
| 操作系统 | Ubuntu 22.04.2 LTS |
| 内核 | Linux 5.15.0-60 |
| C++编译器 | Clang 16.0.4, GCC 10.2.0 |
| Java | Oracle JDK 11.0.17 & 8u202 |
| Spark | Apache Spark 3.1.2 |
| HDFS & Yarn | Apache Hadoop 3.3.6 |
| Hive | Apache Hive 3.1.3 |
## 如何部署和运行大数据智能分析系统
此文档主要提供了搭建大数据智能分析系统CMR运行环境、搭建TPC-DS测试环境、基于大数据智能分析系统运行TPC-DS查询并进行性能测试等相关内容。
以下为本文档的默认目录结构：
源代码目录：```~/jydata/sources```
项目的环境变量设置脚本：``~/jydata/setenv.sh``
依赖软件安装目录：```~/jydata/local```
第三方软件包目录：``~/jydata/third-party``
可执行脚本目录：``~/jydata/sources/CMR/sbin``
本文档默认集群包含``node01``，``node02``，……，``node06``共6个节点，其中``node01``为大数据智能分析系统运行的Driver所在节点。
如您的实际集群环境不同，请参考本文档根据您的集群情况作适当调整。
首先将本项目压缩包在集群所有节点解压缩到``~``目录下：
x86_64环境：
```shell
jydata@node01~node06$ tar -zxvf jydata_release_YYYY_MM_DD_x64.tar.gz -C ~
jydata@node01~node06$ mv ~/jydata_release_YYYY_MM_DD_x64 ~/jydata
```
aarch64环境：
```shell
jydata@node01~node06$ tar -zxvf jydata_release_YYYY_MM_DD_aarch64.tar.gz -C ~
jydata@node01~node06$ mv ~/jydata_release_YYYY_MM_DD_aarch64 ~/jydata
```
## 环境变量
```shell
# 将如下内容写入到你的 bashrc 文件并 source 该文件
export SPARK_HOME=$HOME/jydata/local/spark-3.1.2
export PATH="$SPARK_HOME/bin:$PATH"
export JAVA_HOME=$HOME/jydata/local/jdk-11.0.17
export PATH="$JAVA_HOME/bin:$PATH"
update
2024-04-10 10:10:11 +08:00
export GCC_10_HOME=$HOME/jydata/local/gcc-10.2.0
export PATH="$GCC_10_HOME/bin:$PATH"
export PATH="$HOME/jydata/local/cmake-3.26.5/bin:$PATH"
export LLVM_16_HOME=$HOME/jydata/local/llvm-16.0.4
export CC="$LLVM_16_HOME/bin/clang"
export CXX="$LLVM_16_HOME/bin/clang++"
export PATH="$LLVM_16_HOME/bin:$PATH"
deploy doc
2024-04-09 17:25:12 +08:00
export HADOOP_HOME="$HOME/jydata/local/hadoop-3.3.6"
export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop"
export PATH="$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH"
export HIVE_HOME="$HOME/jydata/local/apache-hive-3.1.3-bin"
export PATH="$HIVE_HOME/bin:$PATH"
# 如您不方便将上述环境变量配置到 bashrc 文件中进行长期使用，你也可以在我们提供的 setenv.sh 文件中进行正确配置并激活，作为临时使用
jydata@node01$ source ~/jydata/setenv.sh
```
## 运行环境搭建部署
### 1. Spark部署
下载并解压Spark 3.1.2到``~/jydata/local``目录下。
```shell
jydata@node01$ tar -xvf spark-3.1.2-bin-hadoop3.2.tgz -C ~/jydata/local
jydata@node01$ cd ~/jydata/local
jydata@node01$ mv spark-3.1.2-bin-hadoop3.2 spark-3.1.2
```
``~/jydata/setenv.sh``为项目的环境变量设置脚本。接下来设置并应用Spark相关的环境变量：
```shell
jydata@node01$ echo 'export SPARK_HOME=$HOME/jydata/local/spark-3.1.2' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export PATH="$SPARK_HOME/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh
```
update
2024-04-10 10:51:59 +08:00
然后在``node01`` ~ ``node06``每个节点下载并安装Spark和CMR的依赖，Java JDK 11.0.17。
deploy doc
2024-04-09 17:25:12 +08:00
x86_64环境：
```shell
jydata@node01~node06$ tar -zxvf jdk-11.0.17_linux-x64_bin.tar.gz -C ~/jydata/local
```
aarch64环境：
```shell
jydata@node01~node06$ tar -zxvf jdk-11.0.17_linux-aarch64_bin.tar.gz -C ~/jydata/local
```
在``node01``设置并应用JDK相关的环境变量：
```shell
jydata@node01$ echo 'export JAVA_HOME=$HOME/jydata/local/jdk-11.0.17' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export PATH="$JAVA_HOME/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh
# 查看JDK版本
jydata@node01$ java -version
# 样例输出
java version "11.0.17" 2022-10-18 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.17+10-LTS-269)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.17+10-LTS-269, mixed mode)
```
### 2. Lancher
在``~/jydata/sources/CMR/cmr-spark-adapter/target/scala-2.12``目录下，已提供编译好的``cmr-spark-adapter_2.12-0.5.1.jar``文件。
### 3. 下载并安装依赖
update
2024-04-10 10:10:11 +08:00
#### 3.1 GCC编译安装10.2.0版本
```shell
jydata@node01$ mkdir ~/jydata/local/gcc-source
jydata@node01$ tar -zxvf gcc-10.2.0.tar.gz -C ~/jydata/local/gcc-source/
# 设置GCC的编译依赖gmp、mpfr、mpc
jydata@node01$ tar -xvf gmp-6.1.2.tar -C ~/jydata/local/gcc-source/gcc-10.2.0
jydata@node01$ ln -s ~/jydata/local/gcc-source/gcc-10.2.0/gmp-6.1.2 ~/jydata/local/gcc-source/gcc-10.2.0/gmp
jydata@node01$ tar -zxvf mpfr-4.2.0.tar.gz -C ~/jydata/local/gcc-source/gcc-10.2.0/
jydata@node01$ ln -s ~/jydata/local/gcc-source/gcc-10.2.0/mpfr-4.2.0 ~/jydata/local/gcc-source/gcc-10.2.0/mpfr
jydata@node01$ tar -xvf mpc-1.3.1.tar.gz -C ~/jydata/local/gcc-source/gcc-10.2.0/
jydata@node01$ ln -s ~/jydata/local/gcc-source/gcc-10.2.0/mpc-1.3.1 ~/jydata/local/gcc-source/gcc-10.2.0/mpc
# 编译安装GCC
jydata@node01$ mkdir ~/jydata/local/gcc-10.2.0
jydata@node01$ cd ~/jydata/local/gcc-source/gcc-10.2.0
jydata@node01$ patch -p0 < ~/jydata/third-party/patches/libstdc++-v3_Makefile.in.patch # 修复编译可能出现的问题
jydata@node01$ patch -p0 < ~/jydata/third-party/patches/libstdc++-v3_Makefile.am.patch # 修复编译可能出现的问题
jydata@node01$ mkdir build && cd build
jydata@node01$ ../configure --prefix=$HOME/jydata/local/gcc-10.2.0 --enable-checking=release --enable-languages=c,c++ --disable-multilib
jydata@node01$ make -j 32 # make并发数可根据实际情况调整
jydata@node01$ make install
# 添加相关环境变量
jydata@node01$ echo 'export GCC_10_HOME=$HOME/jydata/local/gcc-10.2.0' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export PATH="$GCC_10_HOME/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh
# 查看GCC版本
jydata@node01$ gcc --version
# 样例输出
gcc (GCC) 10.2.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
```
#### 3.2 cmake 安装3.25以上版本
x86_64环境：
```shell
jydata@node01$ tar -zxvf cmake-3.26.5-linux-x86_64.tar.gz -C ~/jydata/local
jydata@node01$ cd ~/jydata/local
jydata@node01$ mv cmake-3.26.5-linux-x86_64 cmake-3.26.5
```
aarch64环境：
```shell
jydata@node01$ tar -zxvf cmake-3.26.5-linux-aarch64.tar.gz -C ~/jydata/local
jydata@node01$ cd ~/jydata/local
jydata@node01$ mv cmake-3.26.5-linux-aarch64 cmake-3.26.5
```
设置并应用cmake相关的环境变量：
```shell
jydata@node01$ echo 'export PATH="$HOME/jydata/local/cmake-3.26.5/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh
# 查看cmake版本
jydata@node01$ cmake --version
# 样例输出
cmake version 3.26.5
CMake suite maintained and supported by Kitware (kitware.com/cmake).
```
#### 3.3 Clang编译安装16.0.4版本
```shell
jydata@node01$ mkdir ~/jydata/local/llvm-source
jydata@node01$ tar -xvf llvm-project-16.0.4.src.tar.xz -C ~/jydata/local/llvm-source/
jydata@node01$ mkdir ~/jydata/local/llvm-16.0.4
jydata@node01$ cd ~/jydata/local/llvm-source/llvm-project-16.0.4.src
jydata@node01$ mkdir build && cd build
```
先看一下系统自带的GCC版本
```shell
jydata@node01$ /usr/bin/gcc --version
```
如果系统自带的GCC版本高于10.2.0，则执行以下命令，使用系统自带GCC编译Clang：
```shell
jydata@node01$ CC=/usr/bin/gcc CXX=/usr/bin/g++ cmake -G "Unix Makefiles" -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_TARGETS_TO_BUILD="host" -DCMAKE_BUILD_TYPE="Release" -DLLVM_INCLUDE_TESTS=OFF -DCMAKE_INSTALL_PREFIX="$HOME/jydata/local/llvm-16.0.4" ../llvm
```
如果系统自带的GCC版本低于10.2.0，则执行以下命令，使用之前安装的GCC 10.2.0编译Clang：
```shell
jydata@node01$ CC=$GCC_10_HOME/bin/gcc CXX=$GCC_10_HOME/bin/g++ cmake -G "Unix Makefiles" -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_TARGETS_TO_BUILD="host" -DCMAKE_BUILD_TYPE="Release" -DLLVM_INCLUDE_TESTS=OFF -DCMAKE_INSTALL_PREFIX="$HOME/jydata/local/llvm-16.0.4" -DGCC_INSTALL_PREFIX=$GCC_10_HOME -DCMAKE_CXX_LINK_FLAGS="-Wl,-rpath,$GCC_10_HOME/lib64 -L$GCC_10_HOME/lib64" ../llvm
```
然后开始编译：
```shell
jydata@node01$ make -j 32 # make并发数可根据实际情况调整
jydata@node01$ make install
# 添加相关环境变量
jydata@node01$ echo 'export LLVM_16_HOME=$HOME/jydata/local/llvm-16.0.4' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export CC="$LLVM_16_HOME/bin/clang"' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export CXX="$LLVM_16_HOME/bin/clang++"' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export PATH="$LLVM_16_HOME/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh
# 查看Clang版本
jydata@node01$ clang --version
# 样例输出
clang version 16.0.4
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/jydata/jydata/local/llvm-16.0.4/bin
```
最后将编译出的Clang libomp库同步到其它节点的相同路径下：
```shell
jydata@node01$ for i in {2..6}; do ssh node0$i "mkdir -p ~/jydata/local/llvm-16.0.4/lib" && scp -r ~/jydata/local/llvm-16.0.4/lib/libomp.so node0$i:~/jydata/local/llvm-16.0.4/lib/; done
```
#### 3.4 下载JNA jar包
deploy doc
2024-04-09 17:25:12 +08:00
```shell
jydata@node01$ cd ~/jydata/third-party
jydata@node01$ wget https://repo1.maven.org/maven2/net/java/dev/jna/jna/5.6.0/jna-5.6.0.jar
jydata@node01$ mv jna-5.6.0.jar net.java.dev.jna_jna-5.6.0.jar
```
## TPC-DS测试环境搭建
### 1. TPC-DS测试所需软件部署
#### 1.1 HDFS及Yarn集群部署
update
2024-04-10 10:51:59 +08:00
在``node01`` ~ ``node06``节点部署Apache Hadoop HDFS和Yarn集群，需要Apache Hadoop 3.2及以上版本。
deploy doc
2024-04-09 17:25:12 +08:00
update
2024-04-10 10:51:59 +08:00
首先在``node01`` ~ ``node06``下载并安装Apache Hadoop和Hive的依赖，Java JDK 8u202。
deploy doc
2024-04-09 17:25:12 +08:00
注意这里的JDK版本与CMR使用的JDK版本不同，原因是Hive 3.1.3不支持JDK 11版本。
x86_64环境：
```shell
jydata@node01~node06$ tar -zxvf jdk-8u202-linux-x64.tar.gz -C ~/jydata/local
```
aarch64环境：
```shell
jydata@node01~node06$ tar -zxvf jdk-8u202-linux-arm64-vfp-hflt.tar.gz -C ~/jydata/local
```
update
2024-04-10 10:51:59 +08:00
然后在``node01`` ~ ``node06``每个节点运行以下命令，安装Apache Hadoop：
deploy doc
2024-04-09 17:25:12 +08:00
x86_64环境：
```shell
jydata@node01~node06$ tar -zxvf hadoop-3.3.6.tar.gz -C ~/jydata/local
```
aarch64环境：
```shell
jydata@node01~node06$ tar -zxvf hadoop-3.3.6-aarch64.tar.gz -C ~/jydata/local
```
然后在各集群节点配置HDFS和Yarn。请根据您的集群实际情况进行配置。
``~/jydata/third-party/conf/hadoop/``下提供了配置文件样例供参考。
```shell
jydata@node01~node06$ cd ~/jydata/local/hadoop-3.3.6/etc/hadoop
jydata@node01~node06$ echo "export JAVA_HOME=$HOME/jydata/local/jdk1.8.0_202" >> hadoop-env.sh
jydata@node01~node06$ echo "export HADOOP_HOME=$HOME/jydata/local/hadoop-3.3.6" >> hadoop-env.sh
jydata@node01~node06$ echo "export HADOOP_CONF_DIR=$HOME/jydata/local/hadoop-3.3.6/etc/hadoop" >> hadoop-env.sh
jydata@node01~node06$ vi hadoop-env.sh # 请参考样例根据实际情况作适当配置
jydata@node01~node06$ vi core-site.xml # 请参考样例根据实际情况作适当配置
jydata@node01~node06$ vi hdfs-site.xml # 请参考样例根据实际情况作适当配置
jydata@node01~node06$ vi yarn-site.xml # 请参考样例根据实际情况作适当配置
jydata@node01~node06$ vi workers # 请参考样例根据实际情况作适当配置
jydata@node01~node06$ vi ~/.ssh/authorized_keys # 配置SSH免密码登录，请参考样例根据实际情况作适当配置
# 添加相关环境变量
jydata@node01~node06$ echo 'export HADOOP_HOME="$HOME/jydata/local/hadoop-3.3.6"' >> ~/jydata/setenv.sh
jydata@node01~node06$ echo 'export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop"' >> ~/jydata/setenv.sh
jydata@node01~node06$ echo 'export PATH="$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01~node06$ source ~/jydata/setenv.sh
```
最后启动HDFS和Yarn集群：
```shell
jydata@node01$ for i in {1..6}; do scp $SPARK_HOME/yarn/spark-3.1.2-yarn-shuffle.jar node0$i:$HADOOP_HOME/share/hadoop/yarn/lib/; done
jydata@node01$ $HADOOP_HOME/bin/hdfs namenode -format
jydata@node01$ $HADOOP_HOME/sbin/start-dfs.sh
jydata@node01$ $HADOOP_HOME/sbin/start-yarn.sh
# 检查HDFS和Yarn集群启动是否成功
jydata@node01$ $HADOOP_HOME/bin/hadoop dfsadmin -report
jydata@node01$ $HADOOP_HOME/bin/yarn node -list -showDetails
```
#### 1.2 Hive部署
大数据智能分析系统使用Apache Hive的MetaStore作为元数据存储。
接下来需要在``node01``节点下载并安装Hive：
```shell
jydata@node01$ tar -zxvf apache-hive-3.1.3-bin.tar.gz -C ~/jydata/local
```
然后配置Hive。请根据您的集群实际情况进行配置。``~/jydata/third-party/conf/hive/``下提供了配置文件样例供参考。
```shell
jydata@node01$ cd ~/jydata/local/apache-hive-3.1.3-bin/conf
jydata@node01$ cp hive-env.sh.template hive-env.sh
jydata@node01$ cp hive-exec-log4j2.properties.template hive-exec-log4j2.properties
jydata@node01$ cp hive-log4j2.properties.template hive-log4j2.properties
jydata@node01$ echo "export HADOOP_HOME=$HOME/jydata/local/hadoop-3.3.6" >> hive-env.sh
jydata@node01$ echo "export HIVE_CONF_DIR=$HOME/jydata/local/apache-hive-3.1.3-bin/conf" >> hive-env.sh
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -mkdir -p /user/hive/warehouse
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -mkdir -p /tmp/hive
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -chmod -R 777 /user/hive/warehouse
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -chmod -R 777 /tmp/hive
jydata@node01$ cp ~/jydata/third-party/conf/hive/hive-site.xml ./hive-site.xml # 若有必要请根据实际情况作适当配置
# 添加相关环境变量
jydata@node01$ echo 'export HIVE_HOME="$HOME/jydata/local/apache-hive-3.1.3-bin"' >> ~/jydata/setenv.sh
jydata@node01$ echo 'export PATH="$HIVE_HOME/bin:$PATH"' >> ~/jydata/setenv.sh
jydata@node01$ source ~/jydata/setenv.sh
```
最后启动Hive的MetaStore服务。执行以下命令，验证Hive MetaStore服务可启动成功且保持运行：
```shell
jydata@node01$ $HIVE_HOME/bin/schematool -dbType derby -initSchema
jydata@node01$ $HIVE_HOME/bin/hive --service metastore
```
若要使Hive MetaStore服务在后台运行，可使用nohup：
```shell
jydata@node01$ nohup $HIVE_HOME/bin/hive --service metastore > /dev/null 2>&1 &
```
### 2. TPC-DS测试数据生成
#### 2.1 TPC-DS工具包编译
TPC-DS工具包包含了生成测试数据的程序，需要先对其进行编译：
```shell
jydata@node01$ source ~/jydata/setenv.sh
jydata@node01$ git clone https://github.com/databricks/tpcds-kit.git
jydata@node01$ mv ./tpcds-kit ~/jydata/local/DSGen
jydata@node01$ cd ~/jydata/local/DSGen/tools
jydata@node01$ patch -p0 < ~/jydata/third-party/patches/tpcds_makefile.patch
jydata@node01$ make # 您可能需要自行安装GCC编译器
```
最后将编译出的TPC-DS工具包同步到其它节点的相同路径下：
```shell
jydata@node01$ for i in {2..6}; do scp -r ~/jydata/local/DSGen node0$i:~/jydata/local/; done
```
#### 2.2 生成TPC-DS 10T测试数据
```shell
jydata@node01$ source ~/jydata/setenv.sh
jydata@node01$ git clone https://github.com/databricks/spark-sql-perf.git
jydata@node01$ mv spark-sql-perf ~/jydata/local/
jydata@node01$ cd ~/jydata/local/spark-sql-perf
jydata@node01$ sbt package # 您可能需要自行安装sbt等Scala编译器
jydata@node01$ cd ~/jydata/sources/CMR/sbin/
jydata@node01$ sed -i 's@$HOME@'"$HOME"'@g' tpcds_datagen_parquet.scala
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -mkdir -p /user/hive/tpc-ds-10T/decimaldata
jydata@node01$ ./tpcds_datagen_parquet.sh
# 验证TPC-DS 10T测试数据表生成成功
jydata@node01$ $HADOOP_HOME/bin/hadoop fs -ls /user/hive/tpc-ds-10T/decimaldata
#样例输出
Found 24 items
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/call_center
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/catalog_page
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:25 /user/hive/tpc-ds-10T/decimaldata/catalog_returns
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:23 /user/hive/tpc-ds-10T/decimaldata/catalog_sales
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/customer
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/customer_address
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/customer_demographics
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/date_dim
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/household_demographics
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/income_band
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:25 /user/hive/tpc-ds-10T/decimaldata/inventory
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/item
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/promotion
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/reason
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/ship_mode
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/store
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:30 /user/hive/tpc-ds-10T/decimaldata/store_returns
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:28 /user/hive/tpc-ds-10T/decimaldata/store_sales
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/time_dim
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/warehouse
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/web_page
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/web_returns
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:32 /user/hive/tpc-ds-10T/decimaldata/web_sales
drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/web_site
```
## TPC-DS测试运行
运行如下命令，开始运行TPC-DS测试：
```shell
jydata@node01$ source ~/jydata/setenv.sh
jydata@node01$ tar -zxvf ~/jydata/third-party/tpcds_3_2.tar.gz -C ~/jydata/local/spark-sql-perf/src/main/resources
jydata@node01$ cd ~/jydata/sources/CMR/sbin/
jydata@node01$ source ../dev/vcpkg/env.sh
jydata@node01$ ./tpcds_test.sh
# tpcds_test.sh的核心命令是通过spark-submit提交任务。其主要参数介绍如下：
spark-submit \
--master yarn \ #提交任务到Yarn集群
--deploy-mode client \
--num-executors 24 \ #executors数量
--executor-cores 7 \ #每个executor分配的核数
--executor-memory 30g \ #每个executor分配的内存
--driver-memory 40g \ #driver分配的内存
--conf ... \ #各种Spark配置项
--jars $HOME/jydata/third-party/net.java.dev.jna_jna-5.6.0.jar \ #额外的依赖库，Java JNA
--class org.pacman.cmr.applications.LaunchSQLTpcdsPerformance \ #程序入口类，通过其中的main函数开始执行
$HOME/jydata/sources/CMR/cmr-spark-adapter/target/scala-2.12/cmr-spark-adapter_2.12-0.5.1.jar \ #CMR Laucher的jar包
$HOME/jydata/sources/CMR \ #CMR参数1，CMR项目源代码的根目录
$HOME/jydata/sources/CMR/build_tpcds \ #CMR参数2，CMR项目的编译目录
$HOME/jydata/local/spark-sql-perf/src/main/resources/tpcds_3_2 \ #CMR参数3，TPC-DS查询的SQL文件所在目录
thrift://localhost:9083 \ #CMR参数4，Hive MetaStore服务URI
${QID} \ #CMR参数5，运行第几个TPC-DS查询。0~102对应TPC-DS的q1.sql~q99.sql
${NUM_REPEATS} \ #CMR参数6，重复运行次数，脚本默认3次
${DATABASE_NAME} \ #CMR参数7，Hive元数据库名称，在本文档中是tpcds10t
2>&1 | tee rel_base_output_sf${SF}_query${QID}.log #运行log的保存文件
```
``~/jydata/sources/CMR/sbin/tpcds_test.sh``脚本会依次执行TPC-DS的每个查询。
每个查询分别在CMR和原生Spark中执行3次，并记录在CMR和原生Spark中的执行时间。
执行log保存在``~/jydata/sources/CMR/sbin/rel_base_output_sf1_queryXX.log``文件。
例如，对TPC-DS的q1.sql查询，查看CMR运行结果：
```shell
jydata@node01$ grep 'cmr_ans' rel_base_output_sf1_query0.log
# 样例输出
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 0: AAAAAAAAAAAAAABA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 1: AAAAAAAAAAAAADBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 2: AAAAAAAAAAAAAEDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 3: AAAAAAAAAAAAAOBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 4: AAAAAAAAAAAABEAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 5: AAAAAAAAAAAABNDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 6: AAAAAAAAAAAABOCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 7: AAAAAAAAAAAACCCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 8: AAAAAAAAAAAACDAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 9: AAAAAAAAAAAACEDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 10: AAAAAAAAAAAACGCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 11: AAAAAAAAAAAACLCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 12: AAAAAAAAAAAADLBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 13: AAAAAAAAAAAAEEDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 14: AAAAAAAAAAAAEHDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 15: AAAAAAAAAAAAEOAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 16: AAAAAAAAAAAAFFAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 17: AAAAAAAAAAAAFKAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 18: AAAAAAAAAAAAFMDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 19: AAAAAAAAAAAAGABA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 20: AAAAAAAAAAAAGDBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 21: AAAAAAAAAAAAGDCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 22: AAAAAAAAAAAAGKAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 23: AAAAAAAAAAAAGNBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 24: AAAAAAAAAAAAGPBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 25: AAAAAAAAAAAAHIAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 26: AAAAAAAAAAAAHIDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 27: AAAAAAAAAAAAHLAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 28: AAAAAAAAAAAAICCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 29: AAAAAAAAAAAAIHDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 30: AAAAAAAAAAAAIIAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 31: AAAAAAAAAAAAIJAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 32: AAAAAAAAAAAAIKBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 33: AAAAAAAAAAAAJIBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 34: AAAAAAAAAAAAJLAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 35: AAAAAAAAAAAAKCDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 36: AAAAAAAAAAAAKFAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 37: AAAAAAAAAAAALIBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 38: AAAAAAAAAAAALJAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 39: AAAAAAAAAAAAMGDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 40: AAAAAAAAAAAAMJDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 41: AAAAAAAAAAAANIBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 42: AAAAAAAAAAAAOEDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 43: AAAAAAAAAAAAOHCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 44: AAAAAAAAAAAAOJCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 45: AAAAAAAAAAAAOKAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 46: AAAAAAAAAAAAOOAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 47: AAAAAAAAAAAAPDDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 48: AAAAAAAAAAAAPJDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 49: AAAAAAAAAAAAPNBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 50: AAAAAAAAAAAAPNCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 51: AAAAAAAAAAAAPPBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 52: AAAAAAAAAAABACCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 53: AAAAAAAAAAABADDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 54: AAAAAAAAAAABAIBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 55: AAAAAAAAAAABAMAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 56: AAAAAAAAAAABANBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 57: AAAAAAAAAAABBCDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 58: AAAAAAAAAAABBCDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 59: AAAAAAAAAAABBFDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 60: AAAAAAAAAAABBLDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 61: AAAAAAAAAAABCDCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 62: AAAAAAAAAAABCHCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 63: AAAAAAAAAAABCHCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 64: AAAAAAAAAAABCKBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 65: AAAAAAAAAAABCMCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 66: AAAAAAAAAAABCOAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 67: AAAAAAAAAAABDCCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 68: AAAAAAAAAAABDFDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 69: AAAAAAAAAAABDLDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 70: AAAAAAAAAAABEDAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 71: AAAAAAAAAAABEFCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 72: AAAAAAAAAAABFNBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 73: AAAAAAAAAAABGJAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 74: AAAAAAAAAAABGNCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 75: AAAAAAAAAAABHIDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 76: AAAAAAAAAAABIAAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 77: AAAAAAAAAAABIGCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 78: AAAAAAAAAAABIJAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 79: AAAAAAAAAAABILBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 80: AAAAAAAAAAABJJBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 81: AAAAAAAAAAABKABA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 82: AAAAAAAAAAABKEBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 83: AAAAAAAAAAABKFAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 84: AAAAAAAAAAABKGBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 85: AAAAAAAAAAABKKCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 86: AAAAAAAAAAABKKCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 87: AAAAAAAAAAABLABA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 88: AAAAAAAAAAABLBBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 89: AAAAAAAAAAABLFDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 90: AAAAAAAAAAABLHCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 91: AAAAAAAAAAABLMDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 92: AAAAAAAAAAABLOAA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 93: AAAAAAAAAAABLOBA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 94: AAAAAAAAAAABMACA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 95: AAAAAAAAAAABMCDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 96: AAAAAAAAAAABMKCA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 97: AAAAAAAAAAABMKDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 98: AAAAAAAAAAABNBDA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 99: AAAAAAAAAAABNECA |
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1442] cmr_ans_size: 100
[2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1444] cmr_ans_display_time: 0.000261 sec
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 0: AAAAAAAAAAAAAABA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 1: AAAAAAAAAAAAADBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 2: AAAAAAAAAAAAAEDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 3: AAAAAAAAAAAAAOBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 4: AAAAAAAAAAAABEAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 5: AAAAAAAAAAAABNDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 6: AAAAAAAAAAAABOCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 7: AAAAAAAAAAAACCCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 8: AAAAAAAAAAAACDAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 9: AAAAAAAAAAAACEDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 10: AAAAAAAAAAAACGCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 11: AAAAAAAAAAAACLCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 12: AAAAAAAAAAAADLBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 13: AAAAAAAAAAAAEEDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 14: AAAAAAAAAAAAEHDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 15: AAAAAAAAAAAAEOAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 16: AAAAAAAAAAAAFFAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 17: AAAAAAAAAAAAFKAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 18: AAAAAAAAAAAAFMDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 19: AAAAAAAAAAAAGABA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 20: AAAAAAAAAAAAGDBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 21: AAAAAAAAAAAAGDCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 22: AAAAAAAAAAAAGKAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 23: AAAAAAAAAAAAGNBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 24: AAAAAAAAAAAAGPBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 25: AAAAAAAAAAAAHIAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 26: AAAAAAAAAAAAHIDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 27: AAAAAAAAAAAAHLAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 28: AAAAAAAAAAAAICCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 29: AAAAAAAAAAAAIHDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 30: AAAAAAAAAAAAIIAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 31: AAAAAAAAAAAAIJAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 32: AAAAAAAAAAAAIKBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 33: AAAAAAAAAAAAJIBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 34: AAAAAAAAAAAAJLAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 35: AAAAAAAAAAAAKCDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 36: AAAAAAAAAAAAKFAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 37: AAAAAAAAAAAALIBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 38: AAAAAAAAAAAALJAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 39: AAAAAAAAAAAAMGDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 40: AAAAAAAAAAAAMJDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 41: AAAAAAAAAAAANIBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 42: AAAAAAAAAAAAOEDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 43: AAAAAAAAAAAAOHCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 44: AAAAAAAAAAAAOJCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 45: AAAAAAAAAAAAOKAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 46: AAAAAAAAAAAAOOAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 47: AAAAAAAAAAAAPDDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 48: AAAAAAAAAAAAPJDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 49: AAAAAAAAAAAAPNBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 50: AAAAAAAAAAAAPNCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 51: AAAAAAAAAAAAPPBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 52: AAAAAAAAAAABACCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 53: AAAAAAAAAAABADDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 54: AAAAAAAAAAABAIBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 55: AAAAAAAAAAABAMAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 56: AAAAAAAAAAABANBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 57: AAAAAAAAAAABBCDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 58: AAAAAAAAAAABBCDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 59: AAAAAAAAAAABBFDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 60: AAAAAAAAAAABBLDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 61: AAAAAAAAAAABCDCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 62: AAAAAAAAAAABCHCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 63: AAAAAAAAAAABCHCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 64: AAAAAAAAAAABCKBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 65: AAAAAAAAAAABCMCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 66: AAAAAAAAAAABCOAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 67: AAAAAAAAAAABDCCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 68: AAAAAAAAAAABDFDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 69: AAAAAAAAAAABDLDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 70: AAAAAAAAAAABEDAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 71: AAAAAAAAAAABEFCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 72: AAAAAAAAAAABFNBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 73: AAAAAAAAAAABGJAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 74: AAAAAAAAAAABGNCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 75: AAAAAAAAAAABHIDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 76: AAAAAAAAAAABIAAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 77: AAAAAAAAAAABIGCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 78: AAAAAAAAAAABIJAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 79: AAAAAAAAAAABILBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 80: AAAAAAAAAAABJJBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 81: AAAAAAAAAAABKABA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 82: AAAAAAAAAAABKEBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 83: AAAAAAAAAAABKFAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 84: AAAAAAAAAAABKGBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 85: AAAAAAAAAAABKKCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 86: AAAAAAAAAAABKKCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 87: AAAAAAAAAAABLABA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 88: AAAAAAAAAAABLBBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 89: AAAAAAAAAAABLFDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 90: AAAAAAAAAAABLHCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 91: AAAAAAAAAAABLMDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 92: AAAAAAAAAAABLOAA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 93: AAAAAAAAAAABLOBA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 94: AAAAAAAAAAABMACA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 95: AAAAAAAAAAABMCDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 96: AAAAAAAAAAABMKCA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 97: AAAAAAAAAAABMKDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 98: AAAAAAAAAAABNBDA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 99: AAAAAAAAAAABNECA |
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1442] cmr_ans_size: 100
[2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1444] cmr_ans_display_time: 0.000137 sec
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 0: AAAAAAAAAAAAAABA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 1: AAAAAAAAAAAAADBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 2: AAAAAAAAAAAAAEDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 3: AAAAAAAAAAAAAOBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 4: AAAAAAAAAAAABEAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 5: AAAAAAAAAAAABNDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 6: AAAAAAAAAAAABOCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 7: AAAAAAAAAAAACCCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 8: AAAAAAAAAAAACDAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 9: AAAAAAAAAAAACEDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 10: AAAAAAAAAAAACGCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 11: AAAAAAAAAAAACLCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 12: AAAAAAAAAAAADLBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 13: AAAAAAAAAAAAEEDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 14: AAAAAAAAAAAAEHDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 15: AAAAAAAAAAAAEOAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 16: AAAAAAAAAAAAFFAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 17: AAAAAAAAAAAAFKAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 18: AAAAAAAAAAAAFMDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 19: AAAAAAAAAAAAGABA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 20: AAAAAAAAAAAAGDBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 21: AAAAAAAAAAAAGDCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 22: AAAAAAAAAAAAGKAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 23: AAAAAAAAAAAAGNBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 24: AAAAAAAAAAAAGPBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 25: AAAAAAAAAAAAHIAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 26: AAAAAAAAAAAAHIDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 27: AAAAAAAAAAAAHLAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 28: AAAAAAAAAAAAICCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 29: AAAAAAAAAAAAIHDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 30: AAAAAAAAAAAAIIAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 31: AAAAAAAAAAAAIJAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 32: AAAAAAAAAAAAIKBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 33: AAAAAAAAAAAAJIBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 34: AAAAAAAAAAAAJLAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 35: AAAAAAAAAAAAKCDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 36: AAAAAAAAAAAAKFAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 37: AAAAAAAAAAAALIBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 38: AAAAAAAAAAAALJAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 39: AAAAAAAAAAAAMGDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 40: AAAAAAAAAAAAMJDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 41: AAAAAAAAAAAANIBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 42: AAAAAAAAAAAAOEDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 43: AAAAAAAAAAAAOHCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 44: AAAAAAAAAAAAOJCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 45: AAAAAAAAAAAAOKAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 46: AAAAAAAAAAAAOOAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 47: AAAAAAAAAAAAPDDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 48: AAAAAAAAAAAAPJDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 49: AAAAAAAAAAAAPNBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 50: AAAAAAAAAAAAPNCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 51: AAAAAAAAAAAAPPBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 52: AAAAAAAAAAABACCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 53: AAAAAAAAAAABADDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 54: AAAAAAAAAAABAIBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 55: AAAAAAAAAAABAMAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 56: AAAAAAAAAAABANBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 57: AAAAAAAAAAABBCDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 58: AAAAAAAAAAABBCDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 59: AAAAAAAAAAABBFDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 60: AAAAAAAAAAABBLDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 61: AAAAAAAAAAABCDCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 62: AAAAAAAAAAABCHCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 63: AAAAAAAAAAABCHCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 64: AAAAAAAAAAABCKBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 65: AAAAAAAAAAABCMCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 66: AAAAAAAAAAABCOAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 67: AAAAAAAAAAABDCCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 68: AAAAAAAAAAABDFDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 69: AAAAAAAAAAABDLDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 70: AAAAAAAAAAABEDAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 71: AAAAAAAAAAABEFCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 72: AAAAAAAAAAABFNBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 73: AAAAAAAAAAABGJAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 74: AAAAAAAAAAABGNCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 75: AAAAAAAAAAABHIDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 76: AAAAAAAAAAABIAAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 77: AAAAAAAAAAABIGCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 78: AAAAAAAAAAABIJAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 79: AAAAAAAAAAABILBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 80: AAAAAAAAAAABJJBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 81: AAAAAAAAAAABKABA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 82: AAAAAAAAAAABKEBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 83: AAAAAAAAAAABKFAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 84: AAAAAAAAAAABKGBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 85: AAAAAAAAAAABKKCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 86: AAAAAAAAAAABKKCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 87: AAAAAAAAAAABLABA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 88: AAAAAAAAAAABLBBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 89: AAAAAAAAAAABLFDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 90: AAAAAAAAAAABLHCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 91: AAAAAAAAAAABLMDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 92: AAAAAAAAAAABLOAA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 93: AAAAAAAAAAABLOBA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 94: AAAAAAAAAAABMACA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 95: AAAAAAAAAAABMCDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 96: AAAAAAAAAAABMKCA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 97: AAAAAAAAAAABMKDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 98: AAAAAAAAAAABNBDA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 99: AAAAAAAAAAABNECA |
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1442] cmr_ans_size: 100
[2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1444] cmr_ans_display_time: 7.2e-05 sec
```
查看原生Spark运行结果：
```shell
jydata@node01$ grep 'spark_ans' rel_base_output_sf1_query0.log
# 样例输出
spark_ans 0 0: AAAAAAAAAAAAAABA |
spark_ans 0 1: AAAAAAAAAAAAADBA |
spark_ans 0 2: AAAAAAAAAAAAAEDA |
spark_ans 0 3: AAAAAAAAAAAAAOBA |
spark_ans 0 4: AAAAAAAAAAAABEAA |
spark_ans 0 5: AAAAAAAAAAAABNDA |
spark_ans 0 6: AAAAAAAAAAAABOCA |
spark_ans 0 7: AAAAAAAAAAAACCCA |
spark_ans 0 8: AAAAAAAAAAAACDAA |
spark_ans 0 9: AAAAAAAAAAAACEDA |
spark_ans 0 10: AAAAAAAAAAAACGCA |
spark_ans 0 11: AAAAAAAAAAAACLCA |
spark_ans 0 12: AAAAAAAAAAAADLBA |
spark_ans 0 13: AAAAAAAAAAAAEEDA |
spark_ans 0 14: AAAAAAAAAAAAEHDA |
spark_ans 0 15: AAAAAAAAAAAAEOAA |
spark_ans 0 16: AAAAAAAAAAAAFFAA |
spark_ans 0 17: AAAAAAAAAAAAFKAA |
spark_ans 0 18: AAAAAAAAAAAAFMDA |
spark_ans 0 19: AAAAAAAAAAAAGABA |
spark_ans 0 20: AAAAAAAAAAAAGDBA |
spark_ans 0 21: AAAAAAAAAAAAGDCA |
spark_ans 0 22: AAAAAAAAAAAAGKAA |
spark_ans 0 23: AAAAAAAAAAAAGNBA |
spark_ans 0 24: AAAAAAAAAAAAGPBA |
spark_ans 0 25: AAAAAAAAAAAAHIAA |
spark_ans 0 26: AAAAAAAAAAAAHIDA |
spark_ans 0 27: AAAAAAAAAAAAHLAA |
spark_ans 0 28: AAAAAAAAAAAAICCA |
spark_ans 0 29: AAAAAAAAAAAAIHDA |
spark_ans 0 30: AAAAAAAAAAAAIIAA |
spark_ans 0 31: AAAAAAAAAAAAIJAA |
spark_ans 0 32: AAAAAAAAAAAAIKBA |
spark_ans 0 33: AAAAAAAAAAAAJIBA |
spark_ans 0 34: AAAAAAAAAAAAJLAA |
spark_ans 0 35: AAAAAAAAAAAAKCDA |
spark_ans 0 36: AAAAAAAAAAAAKFAA |
spark_ans 0 37: AAAAAAAAAAAALIBA |
spark_ans 0 38: AAAAAAAAAAAALJAA |
spark_ans 0 39: AAAAAAAAAAAAMGDA |
spark_ans 0 40: AAAAAAAAAAAAMJDA |
spark_ans 0 41: AAAAAAAAAAAANIBA |
spark_ans 0 42: AAAAAAAAAAAAOEDA |
spark_ans 0 43: AAAAAAAAAAAAOHCA |
spark_ans 0 44: AAAAAAAAAAAAOJCA |
spark_ans 0 45: AAAAAAAAAAAAOKAA |
spark_ans 0 46: AAAAAAAAAAAAOOAA |
spark_ans 0 47: AAAAAAAAAAAAPDDA |
spark_ans 0 48: AAAAAAAAAAAAPJDA |
spark_ans 0 49: AAAAAAAAAAAAPNBA |
spark_ans 0 50: AAAAAAAAAAAAPNCA |
spark_ans 0 51: AAAAAAAAAAAAPPBA |
spark_ans 0 52: AAAAAAAAAAABACCA |
spark_ans 0 53: AAAAAAAAAAABADDA |
spark_ans 0 54: AAAAAAAAAAABAIBA |
spark_ans 0 55: AAAAAAAAAAABAMAA |
spark_ans 0 56: AAAAAAAAAAABANBA |
spark_ans 0 57: AAAAAAAAAAABBCDA |
spark_ans 0 58: AAAAAAAAAAABBCDA |
spark_ans 0 59: AAAAAAAAAAABBFDA |
spark_ans 0 60: AAAAAAAAAAABBLDA |
spark_ans 0 61: AAAAAAAAAAABCDCA |
spark_ans 0 62: AAAAAAAAAAABCHCA |
spark_ans 0 63: AAAAAAAAAAABCHCA |
spark_ans 0 64: AAAAAAAAAAABCKBA |
spark_ans 0 65: AAAAAAAAAAABCMCA |
spark_ans 0 66: AAAAAAAAAAABCOAA |
spark_ans 0 67: AAAAAAAAAAABDCCA |
spark_ans 0 68: AAAAAAAAAAABDFDA |
spark_ans 0 69: AAAAAAAAAAABDLDA |
spark_ans 0 70: AAAAAAAAAAABEDAA |
spark_ans 0 71: AAAAAAAAAAABEFCA |
spark_ans 0 72: AAAAAAAAAAABFNBA |
spark_ans 0 73: AAAAAAAAAAABGJAA |
spark_ans 0 74: AAAAAAAAAAABGNCA |
spark_ans 0 75: AAAAAAAAAAABHIDA |
spark_ans 0 76: AAAAAAAAAAABIAAA |
spark_ans 0 77: AAAAAAAAAAABIGCA |
spark_ans 0 78: AAAAAAAAAAABIJAA |
spark_ans 0 79: AAAAAAAAAAABILBA |
spark_ans 0 80: AAAAAAAAAAABJJBA |
spark_ans 0 81: AAAAAAAAAAABKABA |
spark_ans 0 82: AAAAAAAAAAABKEBA |
spark_ans 0 83: AAAAAAAAAAABKFAA |
spark_ans 0 84: AAAAAAAAAAABKGBA |
spark_ans 0 85: AAAAAAAAAAABKKCA |
spark_ans 0 86: AAAAAAAAAAABKKCA |
spark_ans 0 87: AAAAAAAAAAABLABA |
spark_ans 0 88: AAAAAAAAAAABLBBA |
spark_ans 0 89: AAAAAAAAAAABLFDA |
spark_ans 0 90: AAAAAAAAAAABLHCA |
spark_ans 0 91: AAAAAAAAAAABLMDA |
spark_ans 0 92: AAAAAAAAAAABLOAA |
spark_ans 0 93: AAAAAAAAAAABLOBA |
spark_ans 0 94: AAAAAAAAAAABMACA |
spark_ans 0 95: AAAAAAAAAAABMCDA |
spark_ans 0 96: AAAAAAAAAAABMKCA |
spark_ans 0 97: AAAAAAAAAAABMKDA |
spark_ans 0 98: AAAAAAAAAAABNBDA |
spark_ans 0 99: AAAAAAAAAAABNECA |
spark_ans_size: 100 [correct]
spark_ans 0 0: AAAAAAAAAAAAAABA |
spark_ans 0 1: AAAAAAAAAAAAADBA |
spark_ans 0 2: AAAAAAAAAAAAAEDA |
spark_ans 0 3: AAAAAAAAAAAAAOBA |
spark_ans 0 4: AAAAAAAAAAAABEAA |
spark_ans 0 5: AAAAAAAAAAAABNDA |
spark_ans 0 6: AAAAAAAAAAAABOCA |
spark_ans 0 7: AAAAAAAAAAAACCCA |
spark_ans 0 8: AAAAAAAAAAAACDAA |
spark_ans 0 9: AAAAAAAAAAAACEDA |
spark_ans 0 10: AAAAAAAAAAAACGCA |
spark_ans 0 11: AAAAAAAAAAAACLCA |
spark_ans 0 12: AAAAAAAAAAAADLBA |
spark_ans 0 13: AAAAAAAAAAAAEEDA |
spark_ans 0 14: AAAAAAAAAAAAEHDA |
spark_ans 0 15: AAAAAAAAAAAAEOAA |
spark_ans 0 16: AAAAAAAAAAAAFFAA |
spark_ans 0 17: AAAAAAAAAAAAFKAA |
spark_ans 0 18: AAAAAAAAAAAAFMDA |
spark_ans 0 19: AAAAAAAAAAAAGABA |
spark_ans 0 20: AAAAAAAAAAAAGDBA |
spark_ans 0 21: AAAAAAAAAAAAGDCA |
spark_ans 0 22: AAAAAAAAAAAAGKAA |
spark_ans 0 23: AAAAAAAAAAAAGNBA |
spark_ans 0 24: AAAAAAAAAAAAGPBA |
spark_ans 0 25: AAAAAAAAAAAAHIAA |
spark_ans 0 26: AAAAAAAAAAAAHIDA |
spark_ans 0 27: AAAAAAAAAAAAHLAA |
spark_ans 0 28: AAAAAAAAAAAAICCA |
spark_ans 0 29: AAAAAAAAAAAAIHDA |
spark_ans 0 30: AAAAAAAAAAAAIIAA |
spark_ans 0 31: AAAAAAAAAAAAIJAA |
spark_ans 0 32: AAAAAAAAAAAAIKBA |
spark_ans 0 33: AAAAAAAAAAAAJIBA |
spark_ans 0 34: AAAAAAAAAAAAJLAA |
spark_ans 0 35: AAAAAAAAAAAAKCDA |
spark_ans 0 36: AAAAAAAAAAAAKFAA |
spark_ans 0 37: AAAAAAAAAAAALIBA |
spark_ans 0 38: AAAAAAAAAAAALJAA |
spark_ans 0 39: AAAAAAAAAAAAMGDA |
spark_ans 0 40: AAAAAAAAAAAAMJDA |
spark_ans 0 41: AAAAAAAAAAAANIBA |
spark_ans 0 42: AAAAAAAAAAAAOEDA |
spark_ans 0 43: AAAAAAAAAAAAOHCA |
spark_ans 0 44: AAAAAAAAAAAAOJCA |
spark_ans 0 45: AAAAAAAAAAAAOKAA |
spark_ans 0 46: AAAAAAAAAAAAOOAA |
spark_ans 0 47: AAAAAAAAAAAAPDDA |
spark_ans 0 48: AAAAAAAAAAAAPJDA |
spark_ans 0 49: AAAAAAAAAAAAPNBA |
spark_ans 0 50: AAAAAAAAAAAAPNCA |
spark_ans 0 51: AAAAAAAAAAAAPPBA |
spark_ans 0 52: AAAAAAAAAAABACCA |
spark_ans 0 53: AAAAAAAAAAABADDA |
spark_ans 0 54: AAAAAAAAAAABAIBA |
spark_ans 0 55: AAAAAAAAAAABAMAA |
spark_ans 0 56: AAAAAAAAAAABANBA |
spark_ans 0 57: AAAAAAAAAAABBCDA |
spark_ans 0 58: AAAAAAAAAAABBCDA |
spark_ans 0 59: AAAAAAAAAAABBFDA |
spark_ans 0 60: AAAAAAAAAAABBLDA |
spark_ans 0 61: AAAAAAAAAAABCDCA |
spark_ans 0 62: AAAAAAAAAAABCHCA |
spark_ans 0 63: AAAAAAAAAAABCHCA |
spark_ans 0 64: AAAAAAAAAAABCKBA |
spark_ans 0 65: AAAAAAAAAAABCMCA |
spark_ans 0 66: AAAAAAAAAAABCOAA |
spark_ans 0 67: AAAAAAAAAAABDCCA |
spark_ans 0 68: AAAAAAAAAAABDFDA |
spark_ans 0 69: AAAAAAAAAAABDLDA |
spark_ans 0 70: AAAAAAAAAAABEDAA |
spark_ans 0 71: AAAAAAAAAAABEFCA |
spark_ans 0 72: AAAAAAAAAAABFNBA |
spark_ans 0 73: AAAAAAAAAAABGJAA |
spark_ans 0 74: AAAAAAAAAAABGNCA |
spark_ans 0 75: AAAAAAAAAAABHIDA |
spark_ans 0 76: AAAAAAAAAAABIAAA |
spark_ans 0 77: AAAAAAAAAAABIGCA |
spark_ans 0 78: AAAAAAAAAAABIJAA |
spark_ans 0 79: AAAAAAAAAAABILBA |
spark_ans 0 80: AAAAAAAAAAABJJBA |
spark_ans 0 81: AAAAAAAAAAABKABA |
spark_ans 0 82: AAAAAAAAAAABKEBA |
spark_ans 0 83: AAAAAAAAAAABKFAA |
spark_ans 0 84: AAAAAAAAAAABKGBA |
spark_ans 0 85: AAAAAAAAAAABKKCA |
spark_ans 0 86: AAAAAAAAAAABKKCA |
spark_ans 0 87: AAAAAAAAAAABLABA |
spark_ans 0 88: AAAAAAAAAAABLBBA |
spark_ans 0 89: AAAAAAAAAAABLFDA |
spark_ans 0 90: AAAAAAAAAAABLHCA |
spark_ans 0 91: AAAAAAAAAAABLMDA |
spark_ans 0 92: AAAAAAAAAAABLOAA |
spark_ans 0 93: AAAAAAAAAAABLOBA |
spark_ans 0 94: AAAAAAAAAAABMACA |
spark_ans 0 95: AAAAAAAAAAABMCDA |
spark_ans 0 96: AAAAAAAAAAABMKCA |
spark_ans 0 97: AAAAAAAAAAABMKDA |
spark_ans 0 98: AAAAAAAAAAABNBDA |
spark_ans 0 99: AAAAAAAAAAABNECA |
spark_ans_size: 100 [correct]
spark_ans 0 0: AAAAAAAAAAAAAABA |
spark_ans 0 1: AAAAAAAAAAAAADBA |
spark_ans 0 2: AAAAAAAAAAAAAEDA |
spark_ans 0 3: AAAAAAAAAAAAAOBA |
spark_ans 0 4: AAAAAAAAAAAABEAA |
spark_ans 0 5: AAAAAAAAAAAABNDA |
spark_ans 0 6: AAAAAAAAAAAABOCA |
spark_ans 0 7: AAAAAAAAAAAACCCA |
spark_ans 0 8: AAAAAAAAAAAACDAA |
spark_ans 0 9: AAAAAAAAAAAACEDA |
spark_ans 0 10: AAAAAAAAAAAACGCA |
spark_ans 0 11: AAAAAAAAAAAACLCA |
spark_ans 0 12: AAAAAAAAAAAADLBA |
spark_ans 0 13: AAAAAAAAAAAAEEDA |
spark_ans 0 14: AAAAAAAAAAAAEHDA |
spark_ans 0 15: AAAAAAAAAAAAEOAA |
spark_ans 0 16: AAAAAAAAAAAAFFAA |
spark_ans 0 17: AAAAAAAAAAAAFKAA |
spark_ans 0 18: AAAAAAAAAAAAFMDA |
spark_ans 0 19: AAAAAAAAAAAAGABA |
spark_ans 0 20: AAAAAAAAAAAAGDBA |
spark_ans 0 21: AAAAAAAAAAAAGDCA |
spark_ans 0 22: AAAAAAAAAAAAGKAA |
spark_ans 0 23: AAAAAAAAAAAAGNBA |
spark_ans 0 24: AAAAAAAAAAAAGPBA |
spark_ans 0 25: AAAAAAAAAAAAHIAA |
spark_ans 0 26: AAAAAAAAAAAAHIDA |
spark_ans 0 27: AAAAAAAAAAAAHLAA |
spark_ans 0 28: AAAAAAAAAAAAICCA |
spark_ans 0 29: AAAAAAAAAAAAIHDA |
spark_ans 0 30: AAAAAAAAAAAAIIAA |
spark_ans 0 31: AAAAAAAAAAAAIJAA |
spark_ans 0 32: AAAAAAAAAAAAIKBA |
spark_ans 0 33: AAAAAAAAAAAAJIBA |
spark_ans 0 34: AAAAAAAAAAAAJLAA |
spark_ans 0 35: AAAAAAAAAAAAKCDA |
spark_ans 0 36: AAAAAAAAAAAAKFAA |
spark_ans 0 37: AAAAAAAAAAAALIBA |
spark_ans 0 38: AAAAAAAAAAAALJAA |
spark_ans 0 39: AAAAAAAAAAAAMGDA |
spark_ans 0 40: AAAAAAAAAAAAMJDA |
spark_ans 0 41: AAAAAAAAAAAANIBA |
spark_ans 0 42: AAAAAAAAAAAAOEDA |
spark_ans 0 43: AAAAAAAAAAAAOHCA |
spark_ans 0 44: AAAAAAAAAAAAOJCA |
spark_ans 0 45: AAAAAAAAAAAAOKAA |
spark_ans 0 46: AAAAAAAAAAAAOOAA |
spark_ans 0 47: AAAAAAAAAAAAPDDA |
spark_ans 0 48: AAAAAAAAAAAAPJDA |
spark_ans 0 49: AAAAAAAAAAAAPNBA |
spark_ans 0 50: AAAAAAAAAAAAPNCA |
spark_ans 0 51: AAAAAAAAAAAAPPBA |
spark_ans 0 52: AAAAAAAAAAABACCA |
spark_ans 0 53: AAAAAAAAAAABADDA |
spark_ans 0 54: AAAAAAAAAAABAIBA |
spark_ans 0 55: AAAAAAAAAAABAMAA |
spark_ans 0 56: AAAAAAAAAAABANBA |
spark_ans 0 57: AAAAAAAAAAABBCDA |
spark_ans 0 58: AAAAAAAAAAABBCDA |
spark_ans 0 59: AAAAAAAAAAABBFDA |
spark_ans 0 60: AAAAAAAAAAABBLDA |
spark_ans 0 61: AAAAAAAAAAABCDCA |
spark_ans 0 62: AAAAAAAAAAABCHCA |
spark_ans 0 63: AAAAAAAAAAABCHCA |
spark_ans 0 64: AAAAAAAAAAABCKBA |
spark_ans 0 65: AAAAAAAAAAABCMCA |
spark_ans 0 66: AAAAAAAAAAABCOAA |
spark_ans 0 67: AAAAAAAAAAABDCCA |
spark_ans 0 68: AAAAAAAAAAABDFDA |
spark_ans 0 69: AAAAAAAAAAABDLDA |
spark_ans 0 70: AAAAAAAAAAABEDAA |
spark_ans 0 71: AAAAAAAAAAABEFCA |
spark_ans 0 72: AAAAAAAAAAABFNBA |
spark_ans 0 73: AAAAAAAAAAABGJAA |
spark_ans 0 74: AAAAAAAAAAABGNCA |
spark_ans 0 75: AAAAAAAAAAABHIDA |
spark_ans 0 76: AAAAAAAAAAABIAAA |
spark_ans 0 77: AAAAAAAAAAABIGCA |
spark_ans 0 78: AAAAAAAAAAABIJAA |
spark_ans 0 79: AAAAAAAAAAABILBA |
spark_ans 0 80: AAAAAAAAAAABJJBA |
spark_ans 0 81: AAAAAAAAAAABKABA |
spark_ans 0 82: AAAAAAAAAAABKEBA |
spark_ans 0 83: AAAAAAAAAAABKFAA |
spark_ans 0 84: AAAAAAAAAAABKGBA |
spark_ans 0 85: AAAAAAAAAAABKKCA |
spark_ans 0 86: AAAAAAAAAAABKKCA |
spark_ans 0 87: AAAAAAAAAAABLABA |
spark_ans 0 88: AAAAAAAAAAABLBBA |
spark_ans 0 89: AAAAAAAAAAABLFDA |
spark_ans 0 90: AAAAAAAAAAABLHCA |
spark_ans 0 91: AAAAAAAAAAABLMDA |
spark_ans 0 92: AAAAAAAAAAABLOAA |
spark_ans 0 93: AAAAAAAAAAABLOBA |
spark_ans 0 94: AAAAAAAAAAABMACA |
spark_ans 0 95: AAAAAAAAAAABMCDA |
spark_ans 0 96: AAAAAAAAAAABMKCA |
spark_ans 0 97: AAAAAAAAAAABMKDA |
spark_ans 0 98: AAAAAAAAAAABNBDA |
spark_ans 0 99: AAAAAAAAAAABNECA |
spark_ans_size: 100 [correct]
```
CMR与Spark的运行结果一致，证明CMR运行正确。
查看CMR运行时间：
```shell
jydata@node01$ grep 'cmr_time' rel_base_output_sf1_query0.log
# 样例输出
cmr_time 0 10.472 sec
cmr_time 1 5.4510000000000005 sec
cmr_time 2 5.309 sec
```
查看原生Spark运行时间：
```shell
jydata@node01$ grep 'spark_time' rel_base_output_sf1_query0.log
# 样例输出
spark_time 0 16.096 sec ( + 1.932 sec for analysis)
spark_time 1 11.684000000000001 sec ( + 1.359 sec for analysis)
spark_time 2 10.949 sec ( + 1.424 sec for analysis)
```
取后两次运行时间的平均值作为查询的运行时间，则CMR对q1.sql查询的运行时间为5.38s，原生Spark的运行时间为11.3165s。