优化Trino 以支持ES中的脏数据 – 补丁/编译/重打包

Posted on by admin

1. 背景

在使用Trino对ES中数据进行检索时，存在一些检索产生随机报错，重试几次之后可以正常检索数据，报错内容关键字 `null value in entry`。

2. Patch

通过查询数据源，猜测和ES中文档的部分字段缺失有关，通过此patch可修复此问题

diff --git a/core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java b/core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java
index 227fc54876..4729b73691 100644
--- a/core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java
+++ b/core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java
@@ -2547,9 +2547,14 @@ class StatementAnalyzer
                         Optional.of(tableName),
                         Optional.of(column.getName()),
                         false);
-                fields.add(field);
+                
                 ColumnHandle columnHandle = columnHandles.get(column.getName());
-                checkArgument(columnHandle != null, "Unknown field %s", field);
+                if (columnHandle != null) {
+                    fields.add(field);
+                    analysis.setColumn(field, columnHandle);
+                    analysis.addSourceColumns(field, ImmutableSet.of(new SourceColumn(tableName, column.getName())));
+                }
+
                 analysis.setColumn(field, columnHandle);
                 analysis.addSourceColumns(field, ImmutableSet.of(new SourceColumn(tableName, column.getName())));
             }
diff --git a/core/trino-main/src/main/java/io/trino/sql/planner/RelationPlanner.java b/core/trino-main/src/main/java/io/trino/sql/planner/RelationPlanner.java
index ae15275994..7d990cfc82 100644
--- a/core/trino-main/src/main/java/io/trino/sql/planner/RelationPlanner.java
+++ b/core/trino-main/src/main/java/io/trino/sql/planner/RelationPlanner.java
@@ -238,7 +238,11 @@ class RelationPlanner
                 Symbol symbol = symbolAllocator.newSymbol(field);
 
                 outputSymbolsBuilder.add(symbol);
-                columns.put(symbol, analysis.getColumn(field));
+                try {
+                    columns.put(symbol, analysis.getColumn(field));
+                } catch (NullPointerException e) {
+                }
+
             }
 
             List<Symbol> outputSymbols = outputSymbolsBuilder.build();

3.编译&重打包

1. Apply Patch

git clone https://github.com/trinodb/trino/ && git apply allow_missing_field.patch
2. 重新编译打包trino
./mvnw clean install  -DskipTests -e -Dcheckstyle.skip=true  -Dair.check.skip-checkstyle=true -T 4

3. 打包生成docker镜像

cd core/docker && ./build.sh -a amd64

发表评论