# 大数据智能分析系统部署文档20240410 ## 大数据智能分析系统介绍 大数据智能分析系统在Apache Spark的基础上构建了一套以性能为中心的高效数据处理系统。 一方面实现了一套基于C++的RDD编程接口和高性能大数据算子库以支持高性能大数据应用的编写； 另一方面提出了动静态结合的计算图使得在容错能力上等同Apache Spark。 同时，在数据预处理、图计算、机器学习等领域典型应用上进行了测试，取得了多个指标的性能优势。 该系统目前已支持TPC-DS中所有99个查询，且在相同环境下相比Apache Spark SQL达到了2.2倍以上的加速比。 该系统可以降低大数据处理的整体延迟、并可节省大数据处理的成本； 同时能够无缝部署在现有Apache Spark环境中，部署简单。因此，智能大数据分析系统具有显著的应用前景。 ## 验证环境 此文档在6节点集群环境下验证，每台节点的环境如下： | 条目 | 配置 | | ------ |--------------------------------------------------| | 服务器型号 | 浪潮NF5280M6 | | 处理器 | Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz * 2 | | 内存 | 3200 MT/s 64 GB DDR4 内存 * 16 | | 硬盘 | NVMe 7TB 固态硬盘 * 8 | | 网卡 | Mellanox Technologies MT28908 Family | | 操作系统 | Ubuntu 22.04.2 LTS | | 内核 | Linux 5.15.0-60 | | C++编译器 | Clang 16.0.4, GCC 10.2.0 | | Java | Oracle JDK 11.0.17 & 8u202 | | Spark | Apache Spark 3.1.2 | | HDFS & Yarn | Apache Hadoop 3.3.6 | | Hive | Apache Hive 3.1.3 | ## 如何部署和运行大数据智能分析系统 此文档主要提供了搭建大数据智能分析系统CMR运行环境、搭建TPC-DS测试环境、基于大数据智能分析系统运行TPC-DS查询并进行性能测试等相关内容。 以下为本文档的默认目录结构： 源代码目录：```~/jydata/sources``` 项目的环境变量设置脚本：``~/jydata/setenv.sh`` 依赖软件安装目录：```~/jydata/local``` 第三方软件包目录：``~/jydata/third-party`` 可执行脚本目录：``~/jydata/sources/CMR/sbin`` 本文档默认集群包含``node01``，``node02``，……，``node06``共6个节点，其中``node01``为大数据智能分析系统运行的Driver所在节点。 如您的实际集群环境不同，请参考本文档根据您的集群情况作适当调整。 首先将本项目压缩包在集群所有节点解压缩到``~``目录下： x86_64环境： ```shell jydata@node01~node06$ tar -zxvf jydata_release_YYYY_MM_DD_x64.tar.gz -C ~ jydata@node01~node06$ mv ~/jydata_release_YYYY_MM_DD_x64 ~/jydata ``` aarch64环境： ```shell jydata@node01~node06$ tar -zxvf jydata_release_YYYY_MM_DD_aarch64.tar.gz -C ~ jydata@node01~node06$ mv ~/jydata_release_YYYY_MM_DD_aarch64 ~/jydata ``` ## 环境变量 ```shell # 将如下内容写入到你的 bashrc 文件并 source 该文件 export SPARK_HOME=$HOME/jydata/local/spark-3.1.2 export PATH="$SPARK_HOME/bin:$PATH" export JAVA_HOME=$HOME/jydata/local/jdk-11.0.17 export PATH="$JAVA_HOME/bin:$PATH" export GCC_10_HOME=$HOME/jydata/local/gcc-10.2.0 export PATH="$GCC_10_HOME/bin:$PATH" export PATH="$HOME/jydata/local/cmake-3.26.5/bin:$PATH" export LLVM_16_HOME=$HOME/jydata/local/llvm-16.0.4 export CC="$LLVM_16_HOME/bin/clang" export CXX="$LLVM_16_HOME/bin/clang++" export PATH="$LLVM_16_HOME/bin:$PATH" export HADOOP_HOME="$HOME/jydata/local/hadoop-3.3.6" export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop" export PATH="$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH" export HIVE_HOME="$HOME/jydata/local/apache-hive-3.1.3-bin" export PATH="$HIVE_HOME/bin:$PATH" # 如您不方便将上述环境变量配置到 bashrc 文件中进行长期使用，你也可以在我们提供的 setenv.sh 文件中进行正确配置并激活，作为临时使用 jydata@node01$ source ~/jydata/setenv.sh ``` ## 运行环境搭建部署 ### 1. Spark部署 下载并解压Spark 3.1.2到``~/jydata/local``目录下。 ```shell jydata@node01$ tar -xvf spark-3.1.2-bin-hadoop3.2.tgz -C ~/jydata/local jydata@node01$ cd ~/jydata/local jydata@node01$ mv spark-3.1.2-bin-hadoop3.2 spark-3.1.2 ``` ``~/jydata/setenv.sh``为项目的环境变量设置脚本。接下来设置并应用Spark相关的环境变量： ```shell jydata@node01$ echo 'export SPARK_HOME=$HOME/jydata/local/spark-3.1.2' >> ~/jydata/setenv.sh jydata@node01$ echo 'export PATH="$SPARK_HOME/bin:$PATH"' >> ~/jydata/setenv.sh jydata@node01$ source ~/jydata/setenv.sh ``` 然后在``node01``~``node06``每个节点下载并安装Spark和CMR的依赖，Java JDK 11.0.17。 x86_64环境： ```shell jydata@node01~node06$ tar -zxvf jdk-11.0.17_linux-x64_bin.tar.gz -C ~/jydata/local ``` aarch64环境： ```shell jydata@node01~node06$ tar -zxvf jdk-11.0.17_linux-aarch64_bin.tar.gz -C ~/jydata/local ``` 在``node01``设置并应用JDK相关的环境变量： ```shell jydata@node01$ echo 'export JAVA_HOME=$HOME/jydata/local/jdk-11.0.17' >> ~/jydata/setenv.sh jydata@node01$ echo 'export PATH="$JAVA_HOME/bin:$PATH"' >> ~/jydata/setenv.sh jydata@node01$ source ~/jydata/setenv.sh # 查看JDK版本 jydata@node01$ java -version # 样例输出 java version "11.0.17" 2022-10-18 LTS Java(TM) SE Runtime Environment 18.9 (build 11.0.17+10-LTS-269) Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.17+10-LTS-269, mixed mode) ``` ### 2. Lancher 在``~/jydata/sources/CMR/cmr-spark-adapter/target/scala-2.12``目录下，已提供编译好的``cmr-spark-adapter_2.12-0.5.1.jar``文件。 ### 3. 下载并安装依赖 #### 3.1 GCC编译安装10.2.0版本 ```shell jydata@node01$ mkdir ~/jydata/local/gcc-source jydata@node01$ tar -zxvf gcc-10.2.0.tar.gz -C ~/jydata/local/gcc-source/ # 设置GCC的编译依赖gmp、mpfr、mpc jydata@node01$ tar -xvf gmp-6.1.2.tar -C ~/jydata/local/gcc-source/gcc-10.2.0 jydata@node01$ ln -s ~/jydata/local/gcc-source/gcc-10.2.0/gmp-6.1.2 ~/jydata/local/gcc-source/gcc-10.2.0/gmp jydata@node01$ tar -zxvf mpfr-4.2.0.tar.gz -C ~/jydata/local/gcc-source/gcc-10.2.0/ jydata@node01$ ln -s ~/jydata/local/gcc-source/gcc-10.2.0/mpfr-4.2.0 ~/jydata/local/gcc-source/gcc-10.2.0/mpfr jydata@node01$ tar -xvf mpc-1.3.1.tar.gz -C ~/jydata/local/gcc-source/gcc-10.2.0/ jydata@node01$ ln -s ~/jydata/local/gcc-source/gcc-10.2.0/mpc-1.3.1 ~/jydata/local/gcc-source/gcc-10.2.0/mpc # 编译安装GCC jydata@node01$ mkdir ~/jydata/local/gcc-10.2.0 jydata@node01$ cd ~/jydata/local/gcc-source/gcc-10.2.0 jydata@node01$ patch -p0 < ~/jydata/third-party/patches/libstdc++-v3_Makefile.in.patch # 修复编译可能出现的问题 jydata@node01$ patch -p0 < ~/jydata/third-party/patches/libstdc++-v3_Makefile.am.patch # 修复编译可能出现的问题 jydata@node01$ mkdir build && cd build jydata@node01$ ../configure --prefix=$HOME/jydata/local/gcc-10.2.0 --enable-checking=release --enable-languages=c,c++ --disable-multilib jydata@node01$ make -j 32 # make并发数可根据实际情况调整 jydata@node01$ make install # 添加相关环境变量 jydata@node01$ echo 'export GCC_10_HOME=$HOME/jydata/local/gcc-10.2.0' >> ~/jydata/setenv.sh jydata@node01$ echo 'export PATH="$GCC_10_HOME/bin:$PATH"' >> ~/jydata/setenv.sh jydata@node01$ source ~/jydata/setenv.sh # 查看GCC版本 jydata@node01$ gcc --version # 样例输出 gcc (GCC) 10.2.0 Copyright (C) 2020 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ``` #### 3.2 cmake 安装3.25以上版本 x86_64环境： ```shell jydata@node01$ tar -zxvf cmake-3.26.5-linux-x86_64.tar.gz -C ~/jydata/local jydata@node01$ cd ~/jydata/local jydata@node01$ mv cmake-3.26.5-linux-x86_64 cmake-3.26.5 ``` aarch64环境： ```shell jydata@node01$ tar -zxvf cmake-3.26.5-linux-aarch64.tar.gz -C ~/jydata/local jydata@node01$ cd ~/jydata/local jydata@node01$ mv cmake-3.26.5-linux-aarch64 cmake-3.26.5 ``` 设置并应用cmake相关的环境变量： ```shell jydata@node01$ echo 'export PATH="$HOME/jydata/local/cmake-3.26.5/bin:$PATH"' >> ~/jydata/setenv.sh jydata@node01$ source ~/jydata/setenv.sh # 查看cmake版本 jydata@node01$ cmake --version # 样例输出 cmake version 3.26.5 CMake suite maintained and supported by Kitware (kitware.com/cmake). ``` #### 3.3 Clang编译安装16.0.4版本 ```shell jydata@node01$ mkdir ~/jydata/local/llvm-source jydata@node01$ tar -xvf llvm-project-16.0.4.src.tar.xz -C ~/jydata/local/llvm-source/ jydata@node01$ mkdir ~/jydata/local/llvm-16.0.4 jydata@node01$ cd ~/jydata/local/llvm-source/llvm-project-16.0.4.src jydata@node01$ mkdir build && cd build ``` 先看一下系统自带的GCC版本 ```shell jydata@node01$ /usr/bin/gcc --version ``` 如果系统自带的GCC版本高于10.2.0，则执行以下命令，使用系统自带GCC编译Clang： ```shell jydata@node01$ CC=/usr/bin/gcc CXX=/usr/bin/g++ cmake -G "Unix Makefiles" -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_TARGETS_TO_BUILD="host" -DCMAKE_BUILD_TYPE="Release" -DLLVM_INCLUDE_TESTS=OFF -DCMAKE_INSTALL_PREFIX="$HOME/jydata/local/llvm-16.0.4" ../llvm ``` 如果系统自带的GCC版本低于10.2.0，则执行以下命令，使用之前安装的GCC 10.2.0编译Clang： ```shell jydata@node01$ CC=$GCC_10_HOME/bin/gcc CXX=$GCC_10_HOME/bin/g++ cmake -G "Unix Makefiles" -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_TARGETS_TO_BUILD="host" -DCMAKE_BUILD_TYPE="Release" -DLLVM_INCLUDE_TESTS=OFF -DCMAKE_INSTALL_PREFIX="$HOME/jydata/local/llvm-16.0.4" -DGCC_INSTALL_PREFIX=$GCC_10_HOME -DCMAKE_CXX_LINK_FLAGS="-Wl,-rpath,$GCC_10_HOME/lib64 -L$GCC_10_HOME/lib64" ../llvm ``` 然后开始编译： ```shell jydata@node01$ make -j 32 # make并发数可根据实际情况调整 jydata@node01$ make install # 添加相关环境变量 jydata@node01$ echo 'export LLVM_16_HOME=$HOME/jydata/local/llvm-16.0.4' >> ~/jydata/setenv.sh jydata@node01$ echo 'export CC="$LLVM_16_HOME/bin/clang"' >> ~/jydata/setenv.sh jydata@node01$ echo 'export CXX="$LLVM_16_HOME/bin/clang++"' >> ~/jydata/setenv.sh jydata@node01$ echo 'export PATH="$LLVM_16_HOME/bin:$PATH"' >> ~/jydata/setenv.sh jydata@node01$ source ~/jydata/setenv.sh # 查看Clang版本 jydata@node01$ clang --version # 样例输出 clang version 16.0.4 Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /home/jydata/jydata/local/llvm-16.0.4/bin ``` 最后将编译出的Clang libomp库同步到其它节点的相同路径下： ```shell jydata@node01$ for i in {2..6}; do ssh node0$i "mkdir -p ~/jydata/local/llvm-16.0.4/lib" && scp -r ~/jydata/local/llvm-16.0.4/lib/libomp.so node0$i:~/jydata/local/llvm-16.0.4/lib/; done ``` #### 3.4 下载JNA jar包 ```shell jydata@node01$ cd ~/jydata/third-party jydata@node01$ wget https://repo1.maven.org/maven2/net/java/dev/jna/jna/5.6.0/jna-5.6.0.jar jydata@node01$ mv jna-5.6.0.jar net.java.dev.jna_jna-5.6.0.jar ``` ## TPC-DS测试环境搭建 ### 1. TPC-DS测试所需软件部署 #### 1.1 HDFS及Yarn集群部署 在``node01``~``node06``节点部署Apache Hadoop HDFS和Yarn集群，需要Apache Hadoop 3.2及以上版本。 首先在``node01``~``node06``下载并安装Apache Hadoop和Hive的依赖，Java JDK 8u202。 注意这里的JDK版本与CMR使用的JDK版本不同，原因是Hive 3.1.3不支持JDK 11版本。 x86_64环境： ```shell jydata@node01~node06$ tar -zxvf jdk-8u202-linux-x64.tar.gz -C ~/jydata/local ``` aarch64环境： ```shell jydata@node01~node06$ tar -zxvf jdk-8u202-linux-arm64-vfp-hflt.tar.gz -C ~/jydata/local ``` 然后在``node01``~``node06``每个节点运行以下命令，安装Apache Hadoop： x86_64环境： ```shell jydata@node01~node06$ tar -zxvf hadoop-3.3.6.tar.gz -C ~/jydata/local ``` aarch64环境： ```shell jydata@node01~node06$ tar -zxvf hadoop-3.3.6-aarch64.tar.gz -C ~/jydata/local ``` 然后在各集群节点配置HDFS和Yarn。请根据您的集群实际情况进行配置。 ``~/jydata/third-party/conf/hadoop/``下提供了配置文件样例供参考。 ```shell jydata@node01~node06$ cd ~/jydata/local/hadoop-3.3.6/etc/hadoop jydata@node01~node06$ echo "export JAVA_HOME=$HOME/jydata/local/jdk1.8.0_202" >> hadoop-env.sh jydata@node01~node06$ echo "export HADOOP_HOME=$HOME/jydata/local/hadoop-3.3.6" >> hadoop-env.sh jydata@node01~node06$ echo "export HADOOP_CONF_DIR=$HOME/jydata/local/hadoop-3.3.6/etc/hadoop" >> hadoop-env.sh jydata@node01~node06$ vi hadoop-env.sh # 请参考样例根据实际情况作适当配置 jydata@node01~node06$ vi core-site.xml # 请参考样例根据实际情况作适当配置 jydata@node01~node06$ vi hdfs-site.xml # 请参考样例根据实际情况作适当配置 jydata@node01~node06$ vi yarn-site.xml # 请参考样例根据实际情况作适当配置 jydata@node01~node06$ vi workers # 请参考样例根据实际情况作适当配置 jydata@node01~node06$ vi ~/.ssh/authorized_keys # 配置SSH免密码登录，请参考样例根据实际情况作适当配置 # 添加相关环境变量 jydata@node01~node06$ echo 'export HADOOP_HOME="$HOME/jydata/local/hadoop-3.3.6"' >> ~/jydata/setenv.sh jydata@node01~node06$ echo 'export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop"' >> ~/jydata/setenv.sh jydata@node01~node06$ echo 'export PATH="$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH"' >> ~/jydata/setenv.sh jydata@node01~node06$ source ~/jydata/setenv.sh ``` 最后启动HDFS和Yarn集群： ```shell jydata@node01$ for i in {1..6}; do scp $SPARK_HOME/yarn/spark-3.1.2-yarn-shuffle.jar node0$i:$HADOOP_HOME/share/hadoop/yarn/lib/; done jydata@node01$ $HADOOP_HOME/bin/hdfs namenode -format jydata@node01$ $HADOOP_HOME/sbin/start-dfs.sh jydata@node01$ $HADOOP_HOME/sbin/start-yarn.sh # 检查HDFS和Yarn集群启动是否成功 jydata@node01$ $HADOOP_HOME/bin/hadoop dfsadmin -report jydata@node01$ $HADOOP_HOME/bin/yarn node -list -showDetails ``` #### 1.2 Hive部署 大数据智能分析系统使用Apache Hive的MetaStore作为元数据存储。 接下来需要在``node01``节点下载并安装Hive： ```shell jydata@node01$ tar -zxvf apache-hive-3.1.3-bin.tar.gz -C ~/jydata/local ``` 然后配置Hive。请根据您的集群实际情况进行配置。``~/jydata/third-party/conf/hive/``下提供了配置文件样例供参考。 ```shell jydata@node01$ cd ~/jydata/local/apache-hive-3.1.3-bin/conf jydata@node01$ cp hive-env.sh.template hive-env.sh jydata@node01$ cp hive-exec-log4j2.properties.template hive-exec-log4j2.properties jydata@node01$ cp hive-log4j2.properties.template hive-log4j2.properties jydata@node01$ echo "export HADOOP_HOME=$HOME/jydata/local/hadoop-3.3.6" >> hive-env.sh jydata@node01$ echo "export HIVE_CONF_DIR=$HOME/jydata/local/apache-hive-3.1.3-bin/conf" >> hive-env.sh jydata@node01$ $HADOOP_HOME/bin/hadoop fs -mkdir -p /user/hive/warehouse jydata@node01$ $HADOOP_HOME/bin/hadoop fs -mkdir -p /tmp/hive jydata@node01$ $HADOOP_HOME/bin/hadoop fs -chmod -R 777 /user/hive/warehouse jydata@node01$ $HADOOP_HOME/bin/hadoop fs -chmod -R 777 /tmp/hive jydata@node01$ cp ~/jydata/third-party/conf/hive/hive-site.xml ./hive-site.xml # 若有必要请根据实际情况作适当配置 # 添加相关环境变量 jydata@node01$ echo 'export HIVE_HOME="$HOME/jydata/local/apache-hive-3.1.3-bin"' >> ~/jydata/setenv.sh jydata@node01$ echo 'export PATH="$HIVE_HOME/bin:$PATH"' >> ~/jydata/setenv.sh jydata@node01$ source ~/jydata/setenv.sh ``` 最后启动Hive的MetaStore服务。执行以下命令，验证Hive MetaStore服务可启动成功且保持运行： ```shell jydata@node01$ $HIVE_HOME/bin/schematool -dbType derby -initSchema jydata@node01$ $HIVE_HOME/bin/hive --service metastore ``` 若要使Hive MetaStore服务在后台运行，可使用nohup： ```shell jydata@node01$ nohup $HIVE_HOME/bin/hive --service metastore > /dev/null 2>&1 & ``` ### 2. TPC-DS测试数据生成 #### 2.1 TPC-DS工具包编译 TPC-DS工具包包含了生成测试数据的程序，需要先对其进行编译： ```shell jydata@node01$ source ~/jydata/setenv.sh jydata@node01$ git clone https://github.com/databricks/tpcds-kit.git jydata@node01$ mv ./tpcds-kit ~/jydata/local/DSGen jydata@node01$ cd ~/jydata/local/DSGen/tools jydata@node01$ patch -p0 < ~/jydata/third-party/patches/tpcds_makefile.patch jydata@node01$ make # 您可能需要自行安装GCC编译器 ``` 最后将编译出的TPC-DS工具包同步到其它节点的相同路径下： ```shell jydata@node01$ for i in {2..6}; do scp -r ~/jydata/local/DSGen node0$i:~/jydata/local/; done ``` #### 2.2 生成TPC-DS 10T测试数据 ```shell jydata@node01$ source ~/jydata/setenv.sh jydata@node01$ git clone https://github.com/databricks/spark-sql-perf.git jydata@node01$ mv spark-sql-perf ~/jydata/local/ jydata@node01$ cd ~/jydata/local/spark-sql-perf jydata@node01$ sbt package # 您可能需要自行安装sbt等Scala编译器 jydata@node01$ cd ~/jydata/sources/CMR/sbin/ jydata@node01$ sed -i 's@$HOME@'"$HOME"'@g' tpcds_datagen_parquet.scala jydata@node01$ $HADOOP_HOME/bin/hadoop fs -mkdir -p /user/hive/tpc-ds-10T/decimaldata jydata@node01$ ./tpcds_datagen_parquet.sh # 验证TPC-DS 10T测试数据表生成成功 jydata@node01$ $HADOOP_HOME/bin/hadoop fs -ls /user/hive/tpc-ds-10T/decimaldata #样例输出 Found 24 items drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/call_center drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/catalog_page drwxr-xr-x - jydata supergroup 0 2023-09-15 17:25 /user/hive/tpc-ds-10T/decimaldata/catalog_returns drwxr-xr-x - jydata supergroup 0 2023-09-15 17:23 /user/hive/tpc-ds-10T/decimaldata/catalog_sales drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/customer drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/customer_address drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/customer_demographics drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/date_dim drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/household_demographics drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/income_band drwxr-xr-x - jydata supergroup 0 2023-09-15 17:25 /user/hive/tpc-ds-10T/decimaldata/inventory drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/item drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/promotion drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/reason drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/ship_mode drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/store drwxr-xr-x - jydata supergroup 0 2023-09-15 17:30 /user/hive/tpc-ds-10T/decimaldata/store_returns drwxr-xr-x - jydata supergroup 0 2023-09-15 17:28 /user/hive/tpc-ds-10T/decimaldata/store_sales drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/time_dim drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/warehouse drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/web_page drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/web_returns drwxr-xr-x - jydata supergroup 0 2023-09-15 17:32 /user/hive/tpc-ds-10T/decimaldata/web_sales drwxr-xr-x - jydata supergroup 0 2023-09-15 17:34 /user/hive/tpc-ds-10T/decimaldata/web_site ``` ## TPC-DS测试运行 运行如下命令，开始运行TPC-DS测试： ```shell jydata@node01$ source ~/jydata/setenv.sh jydata@node01$ tar -zxvf ~/jydata/third-party/tpcds_3_2.tar.gz -C ~/jydata/local/spark-sql-perf/src/main/resources jydata@node01$ cd ~/jydata/sources/CMR/sbin/ jydata@node01$ source ../dev/vcpkg/env.sh jydata@node01$ ./tpcds_test.sh # tpcds_test.sh的核心命令是通过spark-submit提交任务。其主要参数介绍如下： spark-submit \ --master yarn \ #提交任务到Yarn集群 --deploy-mode client \ --num-executors 24 \ #executors数量 --executor-cores 7 \ #每个executor分配的核数 --executor-memory 30g \ #每个executor分配的内存 --driver-memory 40g \ #driver分配的内存 --conf ... \ #各种Spark配置项 --jars $HOME/jydata/third-party/net.java.dev.jna_jna-5.6.0.jar \ #额外的依赖库，Java JNA --class org.pacman.cmr.applications.LaunchSQLTpcdsPerformance \ #程序入口类，通过其中的main函数开始执行 $HOME/jydata/sources/CMR/cmr-spark-adapter/target/scala-2.12/cmr-spark-adapter_2.12-0.5.1.jar \ #CMR Laucher的jar包 $HOME/jydata/sources/CMR \ #CMR参数1，CMR项目源代码的根目录 $HOME/jydata/sources/CMR/build_tpcds \ #CMR参数2，CMR项目的编译目录 $HOME/jydata/local/spark-sql-perf/src/main/resources/tpcds_3_2 \ #CMR参数3，TPC-DS查询的SQL文件所在目录 thrift://localhost:9083 \ #CMR参数4，Hive MetaStore服务URI ${QID} \ #CMR参数5，运行第几个TPC-DS查询。0~102对应TPC-DS的q1.sql~q99.sql ${NUM_REPEATS} \ #CMR参数6，重复运行次数，脚本默认3次 ${DATABASE_NAME} \ #CMR参数7，Hive元数据库名称，在本文档中是tpcds10t 2>&1 | tee rel_base_output_sf${SF}_query${QID}.log #运行log的保存文件 ``` ``~/jydata/sources/CMR/sbin/tpcds_test.sh``脚本会依次执行TPC-DS的每个查询。 每个查询分别在CMR和原生Spark中执行3次，并记录在CMR和原生Spark中的执行时间。 执行log保存在``~/jydata/sources/CMR/sbin/rel_base_output_sf1_queryXX.log``文件。 例如，对TPC-DS的q1.sql查询，查看CMR运行结果： ```shell jydata@node01$ grep 'cmr_ans' rel_base_output_sf1_query0.log # 样例输出 [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 0: AAAAAAAAAAAAAABA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 1: AAAAAAAAAAAAADBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 2: AAAAAAAAAAAAAEDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 3: AAAAAAAAAAAAAOBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 4: AAAAAAAAAAAABEAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 5: AAAAAAAAAAAABNDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 6: AAAAAAAAAAAABOCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 7: AAAAAAAAAAAACCCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 8: AAAAAAAAAAAACDAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 9: AAAAAAAAAAAACEDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 10: AAAAAAAAAAAACGCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 11: AAAAAAAAAAAACLCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 12: AAAAAAAAAAAADLBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 13: AAAAAAAAAAAAEEDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 14: AAAAAAAAAAAAEHDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 15: AAAAAAAAAAAAEOAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 16: AAAAAAAAAAAAFFAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 17: AAAAAAAAAAAAFKAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 18: AAAAAAAAAAAAFMDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 19: AAAAAAAAAAAAGABA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 20: AAAAAAAAAAAAGDBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 21: AAAAAAAAAAAAGDCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 22: AAAAAAAAAAAAGKAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 23: AAAAAAAAAAAAGNBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 24: AAAAAAAAAAAAGPBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 25: AAAAAAAAAAAAHIAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 26: AAAAAAAAAAAAHIDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 27: AAAAAAAAAAAAHLAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 28: AAAAAAAAAAAAICCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 29: AAAAAAAAAAAAIHDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 30: AAAAAAAAAAAAIIAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 31: AAAAAAAAAAAAIJAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 32: AAAAAAAAAAAAIKBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 33: AAAAAAAAAAAAJIBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 34: AAAAAAAAAAAAJLAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 35: AAAAAAAAAAAAKCDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 36: AAAAAAAAAAAAKFAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 37: AAAAAAAAAAAALIBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 38: AAAAAAAAAAAALJAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 39: AAAAAAAAAAAAMGDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 40: AAAAAAAAAAAAMJDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 41: AAAAAAAAAAAANIBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 42: AAAAAAAAAAAAOEDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 43: AAAAAAAAAAAAOHCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 44: AAAAAAAAAAAAOJCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 45: AAAAAAAAAAAAOKAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 46: AAAAAAAAAAAAOOAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 47: AAAAAAAAAAAAPDDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 48: AAAAAAAAAAAAPJDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 49: AAAAAAAAAAAAPNBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 50: AAAAAAAAAAAAPNCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 51: AAAAAAAAAAAAPPBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 52: AAAAAAAAAAABACCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 53: AAAAAAAAAAABADDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 54: AAAAAAAAAAABAIBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 55: AAAAAAAAAAABAMAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 56: AAAAAAAAAAABANBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 57: AAAAAAAAAAABBCDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 58: AAAAAAAAAAABBCDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 59: AAAAAAAAAAABBFDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 60: AAAAAAAAAAABBLDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 61: AAAAAAAAAAABCDCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 62: AAAAAAAAAAABCHCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 63: AAAAAAAAAAABCHCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 64: AAAAAAAAAAABCKBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 65: AAAAAAAAAAABCMCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 66: AAAAAAAAAAABCOAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 67: AAAAAAAAAAABDCCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 68: AAAAAAAAAAABDFDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 69: AAAAAAAAAAABDLDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 70: AAAAAAAAAAABEDAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 71: AAAAAAAAAAABEFCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 72: AAAAAAAAAAABFNBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 73: AAAAAAAAAAABGJAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 74: AAAAAAAAAAABGNCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 75: AAAAAAAAAAABHIDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 76: AAAAAAAAAAABIAAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 77: AAAAAAAAAAABIGCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 78: AAAAAAAAAAABIJAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 79: AAAAAAAAAAABILBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 80: AAAAAAAAAAABJJBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 81: AAAAAAAAAAABKABA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 82: AAAAAAAAAAABKEBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 83: AAAAAAAAAAABKFAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 84: AAAAAAAAAAABKGBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 85: AAAAAAAAAAABKKCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 86: AAAAAAAAAAABKKCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 87: AAAAAAAAAAABLABA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 88: AAAAAAAAAAABLBBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 89: AAAAAAAAAAABLFDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 90: AAAAAAAAAAABLHCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 91: AAAAAAAAAAABLMDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 92: AAAAAAAAAAABLOAA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 93: AAAAAAAAAAABLOBA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 94: AAAAAAAAAAABMACA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 95: AAAAAAAAAAABMCDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 96: AAAAAAAAAAABMKCA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 97: AAAAAAAAAAABMKDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 98: AAAAAAAAAAABNBDA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 99: AAAAAAAAAAABNECA | [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1442] cmr_ans_size: 100 [2023-09-19 09:09:39.048] [stderr] [info] [SqlOperator.h:1444] cmr_ans_display_time: 0.000261 sec [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 0: AAAAAAAAAAAAAABA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 1: AAAAAAAAAAAAADBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 2: AAAAAAAAAAAAAEDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 3: AAAAAAAAAAAAAOBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 4: AAAAAAAAAAAABEAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 5: AAAAAAAAAAAABNDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 6: AAAAAAAAAAAABOCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 7: AAAAAAAAAAAACCCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 8: AAAAAAAAAAAACDAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 9: AAAAAAAAAAAACEDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 10: AAAAAAAAAAAACGCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 11: AAAAAAAAAAAACLCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 12: AAAAAAAAAAAADLBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 13: AAAAAAAAAAAAEEDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 14: AAAAAAAAAAAAEHDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 15: AAAAAAAAAAAAEOAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 16: AAAAAAAAAAAAFFAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 17: AAAAAAAAAAAAFKAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 18: AAAAAAAAAAAAFMDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 19: AAAAAAAAAAAAGABA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 20: AAAAAAAAAAAAGDBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 21: AAAAAAAAAAAAGDCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 22: AAAAAAAAAAAAGKAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 23: AAAAAAAAAAAAGNBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 24: AAAAAAAAAAAAGPBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 25: AAAAAAAAAAAAHIAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 26: AAAAAAAAAAAAHIDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 27: AAAAAAAAAAAAHLAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 28: AAAAAAAAAAAAICCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 29: AAAAAAAAAAAAIHDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 30: AAAAAAAAAAAAIIAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 31: AAAAAAAAAAAAIJAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 32: AAAAAAAAAAAAIKBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 33: AAAAAAAAAAAAJIBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 34: AAAAAAAAAAAAJLAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 35: AAAAAAAAAAAAKCDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 36: AAAAAAAAAAAAKFAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 37: AAAAAAAAAAAALIBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 38: AAAAAAAAAAAALJAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 39: AAAAAAAAAAAAMGDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 40: AAAAAAAAAAAAMJDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 41: AAAAAAAAAAAANIBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 42: AAAAAAAAAAAAOEDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 43: AAAAAAAAAAAAOHCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 44: AAAAAAAAAAAAOJCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 45: AAAAAAAAAAAAOKAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 46: AAAAAAAAAAAAOOAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 47: AAAAAAAAAAAAPDDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 48: AAAAAAAAAAAAPJDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 49: AAAAAAAAAAAAPNBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 50: AAAAAAAAAAAAPNCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 51: AAAAAAAAAAAAPPBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 52: AAAAAAAAAAABACCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 53: AAAAAAAAAAABADDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 54: AAAAAAAAAAABAIBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 55: AAAAAAAAAAABAMAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 56: AAAAAAAAAAABANBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 57: AAAAAAAAAAABBCDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 58: AAAAAAAAAAABBCDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 59: AAAAAAAAAAABBFDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 60: AAAAAAAAAAABBLDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 61: AAAAAAAAAAABCDCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 62: AAAAAAAAAAABCHCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 63: AAAAAAAAAAABCHCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 64: AAAAAAAAAAABCKBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 65: AAAAAAAAAAABCMCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 66: AAAAAAAAAAABCOAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 67: AAAAAAAAAAABDCCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 68: AAAAAAAAAAABDFDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 69: AAAAAAAAAAABDLDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 70: AAAAAAAAAAABEDAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 71: AAAAAAAAAAABEFCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 72: AAAAAAAAAAABFNBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 73: AAAAAAAAAAABGJAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 74: AAAAAAAAAAABGNCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 75: AAAAAAAAAAABHIDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 76: AAAAAAAAAAABIAAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 77: AAAAAAAAAAABIGCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 78: AAAAAAAAAAABIJAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 79: AAAAAAAAAAABILBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 80: AAAAAAAAAAABJJBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 81: AAAAAAAAAAABKABA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 82: AAAAAAAAAAABKEBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 83: AAAAAAAAAAABKFAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 84: AAAAAAAAAAABKGBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 85: AAAAAAAAAAABKKCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 86: AAAAAAAAAAABKKCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 87: AAAAAAAAAAABLABA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 88: AAAAAAAAAAABLBBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 89: AAAAAAAAAAABLFDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 90: AAAAAAAAAAABLHCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 91: AAAAAAAAAAABLMDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 92: AAAAAAAAAAABLOAA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 93: AAAAAAAAAAABLOBA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 94: AAAAAAAAAAABMACA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 95: AAAAAAAAAAABMCDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 96: AAAAAAAAAAABMKCA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 97: AAAAAAAAAAABMKDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 98: AAAAAAAAAAABNBDA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 99: AAAAAAAAAAABNECA | [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1442] cmr_ans_size: 100 [2023-09-19 09:10:10.990] [stderr] [info] [SqlOperator.h:1444] cmr_ans_display_time: 0.000137 sec [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 0: AAAAAAAAAAAAAABA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 1: AAAAAAAAAAAAADBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 2: AAAAAAAAAAAAAEDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 3: AAAAAAAAAAAAAOBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 4: AAAAAAAAAAAABEAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 5: AAAAAAAAAAAABNDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 6: AAAAAAAAAAAABOCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 7: AAAAAAAAAAAACCCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 8: AAAAAAAAAAAACDAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 9: AAAAAAAAAAAACEDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 10: AAAAAAAAAAAACGCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 11: AAAAAAAAAAAACLCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 12: AAAAAAAAAAAADLBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 13: AAAAAAAAAAAAEEDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 14: AAAAAAAAAAAAEHDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 15: AAAAAAAAAAAAEOAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 16: AAAAAAAAAAAAFFAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 17: AAAAAAAAAAAAFKAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 18: AAAAAAAAAAAAFMDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 19: AAAAAAAAAAAAGABA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 20: AAAAAAAAAAAAGDBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 21: AAAAAAAAAAAAGDCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 22: AAAAAAAAAAAAGKAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 23: AAAAAAAAAAAAGNBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 24: AAAAAAAAAAAAGPBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 25: AAAAAAAAAAAAHIAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 26: AAAAAAAAAAAAHIDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 27: AAAAAAAAAAAAHLAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 28: AAAAAAAAAAAAICCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 29: AAAAAAAAAAAAIHDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 30: AAAAAAAAAAAAIIAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 31: AAAAAAAAAAAAIJAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 32: AAAAAAAAAAAAIKBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 33: AAAAAAAAAAAAJIBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 34: AAAAAAAAAAAAJLAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 35: AAAAAAAAAAAAKCDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 36: AAAAAAAAAAAAKFAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 37: AAAAAAAAAAAALIBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 38: AAAAAAAAAAAALJAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 39: AAAAAAAAAAAAMGDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 40: AAAAAAAAAAAAMJDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 41: AAAAAAAAAAAANIBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 42: AAAAAAAAAAAAOEDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 43: AAAAAAAAAAAAOHCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 44: AAAAAAAAAAAAOJCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 45: AAAAAAAAAAAAOKAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 46: AAAAAAAAAAAAOOAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 47: AAAAAAAAAAAAPDDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 48: AAAAAAAAAAAAPJDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 49: AAAAAAAAAAAAPNBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 50: AAAAAAAAAAAAPNCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 51: AAAAAAAAAAAAPPBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 52: AAAAAAAAAAABACCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 53: AAAAAAAAAAABADDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 54: AAAAAAAAAAABAIBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 55: AAAAAAAAAAABAMAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 56: AAAAAAAAAAABANBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 57: AAAAAAAAAAABBCDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 58: AAAAAAAAAAABBCDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 59: AAAAAAAAAAABBFDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 60: AAAAAAAAAAABBLDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 61: AAAAAAAAAAABCDCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 62: AAAAAAAAAAABCHCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 63: AAAAAAAAAAABCHCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 64: AAAAAAAAAAABCKBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 65: AAAAAAAAAAABCMCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 66: AAAAAAAAAAABCOAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 67: AAAAAAAAAAABDCCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 68: AAAAAAAAAAABDFDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 69: AAAAAAAAAAABDLDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 70: AAAAAAAAAAABEDAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 71: AAAAAAAAAAABEFCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 72: AAAAAAAAAAABFNBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 73: AAAAAAAAAAABGJAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 74: AAAAAAAAAAABGNCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 75: AAAAAAAAAAABHIDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 76: AAAAAAAAAAABIAAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 77: AAAAAAAAAAABIGCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 78: AAAAAAAAAAABIJAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 79: AAAAAAAAAAABILBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 80: AAAAAAAAAAABJJBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 81: AAAAAAAAAAABKABA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 82: AAAAAAAAAAABKEBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 83: AAAAAAAAAAABKFAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 84: AAAAAAAAAAABKGBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 85: AAAAAAAAAAABKKCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 86: AAAAAAAAAAABKKCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 87: AAAAAAAAAAABLABA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 88: AAAAAAAAAAABLBBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 89: AAAAAAAAAAABLFDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 90: AAAAAAAAAAABLHCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 91: AAAAAAAAAAABLMDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 92: AAAAAAAAAAABLOAA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 93: AAAAAAAAAAABLOBA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 94: AAAAAAAAAAABMACA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 95: AAAAAAAAAAABMCDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 96: AAAAAAAAAAABMKCA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 97: AAAAAAAAAAABMKDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 98: AAAAAAAAAAABNBDA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1439] cmr_ans 0 99: AAAAAAAAAAABNECA | [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1442] cmr_ans_size: 100 [2023-09-19 09:10:37.844] [stderr] [info] [SqlOperator.h:1444] cmr_ans_display_time: 7.2e-05 sec ``` 查看原生Spark运行结果： ```shell jydata@node01$ grep 'spark_ans' rel_base_output_sf1_query0.log # 样例输出 spark_ans 0 0: AAAAAAAAAAAAAABA | spark_ans 0 1: AAAAAAAAAAAAADBA | spark_ans 0 2: AAAAAAAAAAAAAEDA | spark_ans 0 3: AAAAAAAAAAAAAOBA | spark_ans 0 4: AAAAAAAAAAAABEAA | spark_ans 0 5: AAAAAAAAAAAABNDA | spark_ans 0 6: AAAAAAAAAAAABOCA | spark_ans 0 7: AAAAAAAAAAAACCCA | spark_ans 0 8: AAAAAAAAAAAACDAA | spark_ans 0 9: AAAAAAAAAAAACEDA | spark_ans 0 10: AAAAAAAAAAAACGCA | spark_ans 0 11: AAAAAAAAAAAACLCA | spark_ans 0 12: AAAAAAAAAAAADLBA | spark_ans 0 13: AAAAAAAAAAAAEEDA | spark_ans 0 14: AAAAAAAAAAAAEHDA | spark_ans 0 15: AAAAAAAAAAAAEOAA | spark_ans 0 16: AAAAAAAAAAAAFFAA | spark_ans 0 17: AAAAAAAAAAAAFKAA | spark_ans 0 18: AAAAAAAAAAAAFMDA | spark_ans 0 19: AAAAAAAAAAAAGABA | spark_ans 0 20: AAAAAAAAAAAAGDBA | spark_ans 0 21: AAAAAAAAAAAAGDCA | spark_ans 0 22: AAAAAAAAAAAAGKAA | spark_ans 0 23: AAAAAAAAAAAAGNBA | spark_ans 0 24: AAAAAAAAAAAAGPBA | spark_ans 0 25: AAAAAAAAAAAAHIAA | spark_ans 0 26: AAAAAAAAAAAAHIDA | spark_ans 0 27: AAAAAAAAAAAAHLAA | spark_ans 0 28: AAAAAAAAAAAAICCA | spark_ans 0 29: AAAAAAAAAAAAIHDA | spark_ans 0 30: AAAAAAAAAAAAIIAA | spark_ans 0 31: AAAAAAAAAAAAIJAA | spark_ans 0 32: AAAAAAAAAAAAIKBA | spark_ans 0 33: AAAAAAAAAAAAJIBA | spark_ans 0 34: AAAAAAAAAAAAJLAA | spark_ans 0 35: AAAAAAAAAAAAKCDA | spark_ans 0 36: AAAAAAAAAAAAKFAA | spark_ans 0 37: AAAAAAAAAAAALIBA | spark_ans 0 38: AAAAAAAAAAAALJAA | spark_ans 0 39: AAAAAAAAAAAAMGDA | spark_ans 0 40: AAAAAAAAAAAAMJDA | spark_ans 0 41: AAAAAAAAAAAANIBA | spark_ans 0 42: AAAAAAAAAAAAOEDA | spark_ans 0 43: AAAAAAAAAAAAOHCA | spark_ans 0 44: AAAAAAAAAAAAOJCA | spark_ans 0 45: AAAAAAAAAAAAOKAA | spark_ans 0 46: AAAAAAAAAAAAOOAA | spark_ans 0 47: AAAAAAAAAAAAPDDA | spark_ans 0 48: AAAAAAAAAAAAPJDA | spark_ans 0 49: AAAAAAAAAAAAPNBA | spark_ans 0 50: AAAAAAAAAAAAPNCA | spark_ans 0 51: AAAAAAAAAAAAPPBA | spark_ans 0 52: AAAAAAAAAAABACCA | spark_ans 0 53: AAAAAAAAAAABADDA | spark_ans 0 54: AAAAAAAAAAABAIBA | spark_ans 0 55: AAAAAAAAAAABAMAA | spark_ans 0 56: AAAAAAAAAAABANBA | spark_ans 0 57: AAAAAAAAAAABBCDA | spark_ans 0 58: AAAAAAAAAAABBCDA | spark_ans 0 59: AAAAAAAAAAABBFDA | spark_ans 0 60: AAAAAAAAAAABBLDA | spark_ans 0 61: AAAAAAAAAAABCDCA | spark_ans 0 62: AAAAAAAAAAABCHCA | spark_ans 0 63: AAAAAAAAAAABCHCA | spark_ans 0 64: AAAAAAAAAAABCKBA | spark_ans 0 65: AAAAAAAAAAABCMCA | spark_ans 0 66: AAAAAAAAAAABCOAA | spark_ans 0 67: AAAAAAAAAAABDCCA | spark_ans 0 68: AAAAAAAAAAABDFDA | spark_ans 0 69: AAAAAAAAAAABDLDA | spark_ans 0 70: AAAAAAAAAAABEDAA | spark_ans 0 71: AAAAAAAAAAABEFCA | spark_ans 0 72: AAAAAAAAAAABFNBA | spark_ans 0 73: AAAAAAAAAAABGJAA | spark_ans 0 74: AAAAAAAAAAABGNCA | spark_ans 0 75: AAAAAAAAAAABHIDA | spark_ans 0 76: AAAAAAAAAAABIAAA | spark_ans 0 77: AAAAAAAAAAABIGCA | spark_ans 0 78: AAAAAAAAAAABIJAA | spark_ans 0 79: AAAAAAAAAAABILBA | spark_ans 0 80: AAAAAAAAAAABJJBA | spark_ans 0 81: AAAAAAAAAAABKABA | spark_ans 0 82: AAAAAAAAAAABKEBA | spark_ans 0 83: AAAAAAAAAAABKFAA | spark_ans 0 84: AAAAAAAAAAABKGBA | spark_ans 0 85: AAAAAAAAAAABKKCA | spark_ans 0 86: AAAAAAAAAAABKKCA | spark_ans 0 87: AAAAAAAAAAABLABA | spark_ans 0 88: AAAAAAAAAAABLBBA | spark_ans 0 89: AAAAAAAAAAABLFDA | spark_ans 0 90: AAAAAAAAAAABLHCA | spark_ans 0 91: AAAAAAAAAAABLMDA | spark_ans 0 92: AAAAAAAAAAABLOAA | spark_ans 0 93: AAAAAAAAAAABLOBA | spark_ans 0 94: AAAAAAAAAAABMACA | spark_ans 0 95: AAAAAAAAAAABMCDA | spark_ans 0 96: AAAAAAAAAAABMKCA | spark_ans 0 97: AAAAAAAAAAABMKDA | spark_ans 0 98: AAAAAAAAAAABNBDA | spark_ans 0 99: AAAAAAAAAAABNECA | spark_ans_size: 100 [correct] spark_ans 0 0: AAAAAAAAAAAAAABA | spark_ans 0 1: AAAAAAAAAAAAADBA | spark_ans 0 2: AAAAAAAAAAAAAEDA | spark_ans 0 3: AAAAAAAAAAAAAOBA | spark_ans 0 4: AAAAAAAAAAAABEAA | spark_ans 0 5: AAAAAAAAAAAABNDA | spark_ans 0 6: AAAAAAAAAAAABOCA | spark_ans 0 7: AAAAAAAAAAAACCCA | spark_ans 0 8: AAAAAAAAAAAACDAA | spark_ans 0 9: AAAAAAAAAAAACEDA | spark_ans 0 10: AAAAAAAAAAAACGCA | spark_ans 0 11: AAAAAAAAAAAACLCA | spark_ans 0 12: AAAAAAAAAAAADLBA | spark_ans 0 13: AAAAAAAAAAAAEEDA | spark_ans 0 14: AAAAAAAAAAAAEHDA | spark_ans 0 15: AAAAAAAAAAAAEOAA | spark_ans 0 16: AAAAAAAAAAAAFFAA | spark_ans 0 17: AAAAAAAAAAAAFKAA | spark_ans 0 18: AAAAAAAAAAAAFMDA | spark_ans 0 19: AAAAAAAAAAAAGABA | spark_ans 0 20: AAAAAAAAAAAAGDBA | spark_ans 0 21: AAAAAAAAAAAAGDCA | spark_ans 0 22: AAAAAAAAAAAAGKAA | spark_ans 0 23: AAAAAAAAAAAAGNBA | spark_ans 0 24: AAAAAAAAAAAAGPBA | spark_ans 0 25: AAAAAAAAAAAAHIAA | spark_ans 0 26: AAAAAAAAAAAAHIDA | spark_ans 0 27: AAAAAAAAAAAAHLAA | spark_ans 0 28: AAAAAAAAAAAAICCA | spark_ans 0 29: AAAAAAAAAAAAIHDA | spark_ans 0 30: AAAAAAAAAAAAIIAA | spark_ans 0 31: AAAAAAAAAAAAIJAA | spark_ans 0 32: AAAAAAAAAAAAIKBA | spark_ans 0 33: AAAAAAAAAAAAJIBA | spark_ans 0 34: AAAAAAAAAAAAJLAA | spark_ans 0 35: AAAAAAAAAAAAKCDA | spark_ans 0 36: AAAAAAAAAAAAKFAA | spark_ans 0 37: AAAAAAAAAAAALIBA | spark_ans 0 38: AAAAAAAAAAAALJAA | spark_ans 0 39: AAAAAAAAAAAAMGDA | spark_ans 0 40: AAAAAAAAAAAAMJDA | spark_ans 0 41: AAAAAAAAAAAANIBA | spark_ans 0 42: AAAAAAAAAAAAOEDA | spark_ans 0 43: AAAAAAAAAAAAOHCA | spark_ans 0 44: AAAAAAAAAAAAOJCA | spark_ans 0 45: AAAAAAAAAAAAOKAA | spark_ans 0 46: AAAAAAAAAAAAOOAA | spark_ans 0 47: AAAAAAAAAAAAPDDA | spark_ans 0 48: AAAAAAAAAAAAPJDA | spark_ans 0 49: AAAAAAAAAAAAPNBA | spark_ans 0 50: AAAAAAAAAAAAPNCA | spark_ans 0 51: AAAAAAAAAAAAPPBA | spark_ans 0 52: AAAAAAAAAAABACCA | spark_ans 0 53: AAAAAAAAAAABADDA | spark_ans 0 54: AAAAAAAAAAABAIBA | spark_ans 0 55: AAAAAAAAAAABAMAA | spark_ans 0 56: AAAAAAAAAAABANBA | spark_ans 0 57: AAAAAAAAAAABBCDA | spark_ans 0 58: AAAAAAAAAAABBCDA | spark_ans 0 59: AAAAAAAAAAABBFDA | spark_ans 0 60: AAAAAAAAAAABBLDA | spark_ans 0 61: AAAAAAAAAAABCDCA | spark_ans 0 62: AAAAAAAAAAABCHCA | spark_ans 0 63: AAAAAAAAAAABCHCA | spark_ans 0 64: AAAAAAAAAAABCKBA | spark_ans 0 65: AAAAAAAAAAABCMCA | spark_ans 0 66: AAAAAAAAAAABCOAA | spark_ans 0 67: AAAAAAAAAAABDCCA | spark_ans 0 68: AAAAAAAAAAABDFDA | spark_ans 0 69: AAAAAAAAAAABDLDA | spark_ans 0 70: AAAAAAAAAAABEDAA | spark_ans 0 71: AAAAAAAAAAABEFCA | spark_ans 0 72: AAAAAAAAAAABFNBA | spark_ans 0 73: AAAAAAAAAAABGJAA | spark_ans 0 74: AAAAAAAAAAABGNCA | spark_ans 0 75: AAAAAAAAAAABHIDA | spark_ans 0 76: AAAAAAAAAAABIAAA | spark_ans 0 77: AAAAAAAAAAABIGCA | spark_ans 0 78: AAAAAAAAAAABIJAA | spark_ans 0 79: AAAAAAAAAAABILBA | spark_ans 0 80: AAAAAAAAAAABJJBA | spark_ans 0 81: AAAAAAAAAAABKABA | spark_ans 0 82: AAAAAAAAAAABKEBA | spark_ans 0 83: AAAAAAAAAAABKFAA | spark_ans 0 84: AAAAAAAAAAABKGBA | spark_ans 0 85: AAAAAAAAAAABKKCA | spark_ans 0 86: AAAAAAAAAAABKKCA | spark_ans 0 87: AAAAAAAAAAABLABA | spark_ans 0 88: AAAAAAAAAAABLBBA | spark_ans 0 89: AAAAAAAAAAABLFDA | spark_ans 0 90: AAAAAAAAAAABLHCA | spark_ans 0 91: AAAAAAAAAAABLMDA | spark_ans 0 92: AAAAAAAAAAABLOAA | spark_ans 0 93: AAAAAAAAAAABLOBA | spark_ans 0 94: AAAAAAAAAAABMACA | spark_ans 0 95: AAAAAAAAAAABMCDA | spark_ans 0 96: AAAAAAAAAAABMKCA | spark_ans 0 97: AAAAAAAAAAABMKDA | spark_ans 0 98: AAAAAAAAAAABNBDA | spark_ans 0 99: AAAAAAAAAAABNECA | spark_ans_size: 100 [correct] spark_ans 0 0: AAAAAAAAAAAAAABA | spark_ans 0 1: AAAAAAAAAAAAADBA | spark_ans 0 2: AAAAAAAAAAAAAEDA | spark_ans 0 3: AAAAAAAAAAAAAOBA | spark_ans 0 4: AAAAAAAAAAAABEAA | spark_ans 0 5: AAAAAAAAAAAABNDA | spark_ans 0 6: AAAAAAAAAAAABOCA | spark_ans 0 7: AAAAAAAAAAAACCCA | spark_ans 0 8: AAAAAAAAAAAACDAA | spark_ans 0 9: AAAAAAAAAAAACEDA | spark_ans 0 10: AAAAAAAAAAAACGCA | spark_ans 0 11: AAAAAAAAAAAACLCA | spark_ans 0 12: AAAAAAAAAAAADLBA | spark_ans 0 13: AAAAAAAAAAAAEEDA | spark_ans 0 14: AAAAAAAAAAAAEHDA | spark_ans 0 15: AAAAAAAAAAAAEOAA | spark_ans 0 16: AAAAAAAAAAAAFFAA | spark_ans 0 17: AAAAAAAAAAAAFKAA | spark_ans 0 18: AAAAAAAAAAAAFMDA | spark_ans 0 19: AAAAAAAAAAAAGABA | spark_ans 0 20: AAAAAAAAAAAAGDBA | spark_ans 0 21: AAAAAAAAAAAAGDCA | spark_ans 0 22: AAAAAAAAAAAAGKAA | spark_ans 0 23: AAAAAAAAAAAAGNBA | spark_ans 0 24: AAAAAAAAAAAAGPBA | spark_ans 0 25: AAAAAAAAAAAAHIAA | spark_ans 0 26: AAAAAAAAAAAAHIDA | spark_ans 0 27: AAAAAAAAAAAAHLAA | spark_ans 0 28: AAAAAAAAAAAAICCA | spark_ans 0 29: AAAAAAAAAAAAIHDA | spark_ans 0 30: AAAAAAAAAAAAIIAA | spark_ans 0 31: AAAAAAAAAAAAIJAA | spark_ans 0 32: AAAAAAAAAAAAIKBA | spark_ans 0 33: AAAAAAAAAAAAJIBA | spark_ans 0 34: AAAAAAAAAAAAJLAA | spark_ans 0 35: AAAAAAAAAAAAKCDA | spark_ans 0 36: AAAAAAAAAAAAKFAA | spark_ans 0 37: AAAAAAAAAAAALIBA | spark_ans 0 38: AAAAAAAAAAAALJAA | spark_ans 0 39: AAAAAAAAAAAAMGDA | spark_ans 0 40: AAAAAAAAAAAAMJDA | spark_ans 0 41: AAAAAAAAAAAANIBA | spark_ans 0 42: AAAAAAAAAAAAOEDA | spark_ans 0 43: AAAAAAAAAAAAOHCA | spark_ans 0 44: AAAAAAAAAAAAOJCA | spark_ans 0 45: AAAAAAAAAAAAOKAA | spark_ans 0 46: AAAAAAAAAAAAOOAA | spark_ans 0 47: AAAAAAAAAAAAPDDA | spark_ans 0 48: AAAAAAAAAAAAPJDA | spark_ans 0 49: AAAAAAAAAAAAPNBA | spark_ans 0 50: AAAAAAAAAAAAPNCA | spark_ans 0 51: AAAAAAAAAAAAPPBA | spark_ans 0 52: AAAAAAAAAAABACCA | spark_ans 0 53: AAAAAAAAAAABADDA | spark_ans 0 54: AAAAAAAAAAABAIBA | spark_ans 0 55: AAAAAAAAAAABAMAA | spark_ans 0 56: AAAAAAAAAAABANBA | spark_ans 0 57: AAAAAAAAAAABBCDA | spark_ans 0 58: AAAAAAAAAAABBCDA | spark_ans 0 59: AAAAAAAAAAABBFDA | spark_ans 0 60: AAAAAAAAAAABBLDA | spark_ans 0 61: AAAAAAAAAAABCDCA | spark_ans 0 62: AAAAAAAAAAABCHCA | spark_ans 0 63: AAAAAAAAAAABCHCA | spark_ans 0 64: AAAAAAAAAAABCKBA | spark_ans 0 65: AAAAAAAAAAABCMCA | spark_ans 0 66: AAAAAAAAAAABCOAA | spark_ans 0 67: AAAAAAAAAAABDCCA | spark_ans 0 68: AAAAAAAAAAABDFDA | spark_ans 0 69: AAAAAAAAAAABDLDA | spark_ans 0 70: AAAAAAAAAAABEDAA | spark_ans 0 71: AAAAAAAAAAABEFCA | spark_ans 0 72: AAAAAAAAAAABFNBA | spark_ans 0 73: AAAAAAAAAAABGJAA | spark_ans 0 74: AAAAAAAAAAABGNCA | spark_ans 0 75: AAAAAAAAAAABHIDA | spark_ans 0 76: AAAAAAAAAAABIAAA | spark_ans 0 77: AAAAAAAAAAABIGCA | spark_ans 0 78: AAAAAAAAAAABIJAA | spark_ans 0 79: AAAAAAAAAAABILBA | spark_ans 0 80: AAAAAAAAAAABJJBA | spark_ans 0 81: AAAAAAAAAAABKABA | spark_ans 0 82: AAAAAAAAAAABKEBA | spark_ans 0 83: AAAAAAAAAAABKFAA | spark_ans 0 84: AAAAAAAAAAABKGBA | spark_ans 0 85: AAAAAAAAAAABKKCA | spark_ans 0 86: AAAAAAAAAAABKKCA | spark_ans 0 87: AAAAAAAAAAABLABA | spark_ans 0 88: AAAAAAAAAAABLBBA | spark_ans 0 89: AAAAAAAAAAABLFDA | spark_ans 0 90: AAAAAAAAAAABLHCA | spark_ans 0 91: AAAAAAAAAAABLMDA | spark_ans 0 92: AAAAAAAAAAABLOAA | spark_ans 0 93: AAAAAAAAAAABLOBA | spark_ans 0 94: AAAAAAAAAAABMACA | spark_ans 0 95: AAAAAAAAAAABMCDA | spark_ans 0 96: AAAAAAAAAAABMKCA | spark_ans 0 97: AAAAAAAAAAABMKDA | spark_ans 0 98: AAAAAAAAAAABNBDA | spark_ans 0 99: AAAAAAAAAAABNECA | spark_ans_size: 100 [correct] ``` CMR与Spark的运行结果一致，证明CMR运行正确。 查看CMR运行时间： ```shell jydata@node01$ grep 'cmr_time' rel_base_output_sf1_query0.log # 样例输出 cmr_time 0 10.472 sec cmr_time 1 5.4510000000000005 sec cmr_time 2 5.309 sec ``` 查看原生Spark运行时间： ```shell jydata@node01$ grep 'spark_time' rel_base_output_sf1_query0.log # 样例输出 spark_time 0 16.096 sec ( + 1.932 sec for analysis) spark_time 1 11.684000000000001 sec ( + 1.359 sec for analysis) spark_time 2 10.949 sec ( + 1.424 sec for analysis) ``` 取后两次运行时间的平均值作为查询的运行时间，则CMR对q1.sql查询的运行时间为5.38s，原生Spark的运行时间为11.3165s。