Support COPY TO TsFile for exporting query results in table model by shuwenwei · Pull Request #17372 · apache/iotdb

shuwenwei · 2026-03-27T06:32:10Z

Description

This PR adds support for exporting query results (TsBlock) into Table Model TsFile through the SQL interface.

Export full table (explicit SELECT)

COPY (SELECT * FROM t1) TO '1.tsfile';

Export full table (simplified syntax)

COPY t1 TO '1.tsfile';

Complete example

COPY (
  SELECT region, device, time, temperature, humidity
  FROM sensors
  WHERE temperature > 20
)
TO 'sensors.tsfile'
WITH (
  TABLE target_table,
  TIME time,
  TAGS (region, device),
  MEMORY_THRESHOLD 1000000
);

Copilot

Pull request overview

Adds support for COPY ... TO ... in the table model to export query results into TsFile format, integrating the feature across SQL grammar, AST/planner nodes, execution operators, and integration tests.

Changes:

Extend relational SQL grammar + parser/AST to support COPY TO with options (format/table/time/tags/memory threshold).
Add planner/execution support (CopyToNode + TableCopyToOperator) that writes query TsBlocks into a TsFile and returns a summary result row.
Add end-to-end integration tests covering multiple COPY TO scenarios (table copy, query copy, tags/time overrides, auto-generated time column, joins).

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
pom.xml	Bumps `tsfile.version` snapshot used by the build.
iotdb-core/relational-grammar/src/main/antlr4/org/apache/iotdb/db/relational/grammar/sql/RelationalSql.g4	Adds `COPY TO` statement + options to the relational SQL grammar and tokens.
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/table/TsTable.java	Adds helper to find a table’s time column name.
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/column/ColumnHeaderConstant.java	Adds COPY result headers/constants.
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/conf/IoTDBConstant.java	Introduces `copy_to` folder name constant.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/rescon/disk/TierManager.java	Adds folder selection support for `copy_to` export targets.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/util/SqlFormatter.java	Adds formatting support for `CopyTo` AST.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/parser/AstBuilder.java	Builds `CopyTo` AST and parses COPY options.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/ast/DefaultTraversalVisitor.java	Traversal support for `CopyTo`.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/ast/CopyTo.java	New AST node representing `COPY TO`.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/ast/AstVisitor.java	Visitor hook for `CopyTo`.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/optimizations/UnaliasSymbolReferences.java	Adds unalias rewrite logic for `CopyToNode`.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/optimizations/LogicalOptimizeFactory.java	Registers column-pruning rule for `CopyToNode`.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/node/Patterns.java	Adds pattern for matching `CopyToNode`.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/node/CopyToNode.java	New plan node for `COPY TO` execution.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/iterative/rule/PruneCopyToColumns.java	New rule to restrict child outputs for COPY.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/distribute/TableModelTypeProviderExtractor.java	Adds visitor handling for `CopyToNode`.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/distribute/TableDistributedPlanGenerator.java	Ensures COPY node participates in distributed plan generation.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/TableLogicalPlanner.java	Plans `COPY TO` into `CopyToNode` and changes output header logic.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/StatementAnalyzer.java	Adds analysis/scoping for `CopyTo`.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanVisitor.java	Adds PlanVisitor method for `CopyToNode`.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/TableOperatorGenerator.java	Generates `TableCopyToOperator` for `CopyToNode`.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/copyto/tsfile/TsFileFormatCopyToWriter.java	New writer that converts table TsBlocks into TsFile output + result row.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/copyto/tsfile/CopyToTsFileOptions.java	New TSFILE option inference + validation for COPY.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/copyto/TableCopyToOperator.java	New operator that writes query results to a target file and returns summary.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/copyto/IFormatCopyToWriter.java	New writer interface abstraction.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/copyto/CopyToOptions.java	New options interface + builder for COPY formats.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/common/header/DatasetHeader.java	Adds overload to map output columns to TsBlock indexes for COPY.
integration-test/src/test/java/org/apache/iotdb/relational/it/query/recent/copyto/IoTDBCopyToTsFileIT.java	New IT coverage for COPY TO TsFile in table model.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-27T08:00:03Z

...a/org/apache/iotdb/db/queryengine/execution/operator/process/copyto/TableCopyToOperator.java

+  private File createTargetFile(String path) throws Exception {
+    File file = new File(path);
+    if (file.getParent() == null) {
+      String dir = TierManager.getInstance().getNextFolderForCopyToTargetFile();
+      file = new File(dir, path);
+    }
+    File parent = file.getParentFile();
+    if (parent != null && !parent.exists() && !parent.mkdirs()) {
+      throw new IOException("Failed to create directories: " + parent);
+    }
+    if (!file.createNewFile()) {
+      throw new IOException("Target file already exists: " + file.getAbsolutePath());
+    }


createTargetFile() will create a new file at an arbitrary user-supplied path (including absolute paths and paths with ..) and will also create parent directories. Since the path comes from a SQL statement, this effectively allows remote clients to write files anywhere the server user has permissions. Please restrict COPY TO targets to the dedicated copy_to directory (or another explicit allowlist/configured base dir), and reject absolute paths / path traversal.

...e/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/ast/CopyTo.java

Copilot · 2026-03-27T08:00:04Z

...ache/iotdb/db/queryengine/plan/relational/planner/optimizations/UnaliasSymbolReferences.java

+    public PlanAndMappings visitCopyTo(CopyToNode node, UnaliasContext context) {
+      PlanAndMappings rewrittenSource = node.getChild().accept(this, context);
+      Map<Symbol, Symbol> mapping = new HashMap<>(rewrittenSource.getMappings());
+      SymbolMapper mapper = symbolMapper(mapping);
+      List<Symbol> newChildPermittedOutputs = mapper.map(node.getChildPermittedOutputs());
+      return new PlanAndMappings(
+          new CopyToNode(
+              node.getPlanNodeId(),
+              rewrittenSource.getRoot(),
+              node.getTargetFilePath(),
+              node.getCopyToOptions(),
+              newChildPermittedOutputs,
+              node.getInnerQueryDatasetHeader(),
+              node.getInnerQueryOutputNode()),


In visitCopyTo, the child plan is rewritten and symbols may be remapped, but innerQueryOutputNode is passed through unchanged. TableCopyToOperator later calls datasetHeader.setTableColumnToTsBlockIndexMap(innerQueryOutputNode, childNode) and relies on innerQueryOutputNode.getOutputSymbols() matching the (possibly rewritten) child’s output symbols; if they don’t, outputSymbolsIndexMap.get(...) can return null and fail at runtime. Please remap innerQueryOutputNode’s output symbols using the same SymbolMapper (similar to childPermittedOutputs), or store a symbol-independent mapping instead of the original OutputNode.

...b-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/rescon/disk/TierManager.java

...ode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/util/SqlFormatter.java

...pache/iotdb/db/queryengine/execution/operator/process/copyto/tsfile/CopyToTsFileOptions.java

codecov · 2026-03-27T10:04:26Z

Codecov Report

❌ Patch coverage is 7.78689% with 450 lines in your changes missing coverage. Please review.
✅ Project coverage is 39.72%. Comparing base (402abdd) to head (20352f7).
⚠️ Report is 21 commits behind head on master.

Files with missing lines	Patch %	Lines
...tor/process/copyto/tsfile/CopyToTsFileOptions.java	0.00%	92 Missing ⚠️
...rocess/copyto/tsfile/TsFileFormatCopyToWriter.java	0.00%	86 Missing ⚠️
...n/operator/process/copyto/TableCopyToOperator.java	0.00%	65 Missing ⚠️
...yengine/plan/relational/sql/parser/AstBuilder.java	13.88%	31 Missing ⚠️
...db/queryengine/plan/relational/sql/ast/CopyTo.java	0.00%	26 Missing ⚠️
...e/plan/relational/planner/TableLogicalPlanner.java	32.43%	25 Missing ⚠️
...ngine/plan/relational/planner/node/CopyToNode.java	0.00%	25 Missing ⚠️
...ecution/operator/process/copyto/CopyToOptions.java	0.00%	23 Missing ⚠️
...eryengine/plan/planner/TableOperatorGenerator.java	0.00%	15 Missing ⚠️
...elational/planner/distribute/AddExchangeNodes.java	0.00%	13 Missing ⚠️
... and 11 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #17372      +/-   ##
============================================
- Coverage     39.74%   39.72%   -0.02%     
  Complexity      312      312              
============================================
  Files          5126     5134       +8     
  Lines        346228   346840     +612     
  Branches      44083    44181      +98     
============================================
+ Hits         137608   137797     +189     
- Misses       208620   209043     +423

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

sonarqubecloud · 2026-03-27T11:03:35Z

Quality Gate passed

Issues
22 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
3.5% Duplication on New Code

See analysis details on SonarQube Cloud

shuwenwei added 4 commits March 27, 2026 14:30

copy to tsfile

0fa94cd

fix

a342878

fix

ecdca52

fix

8be92b5

JackieTien97 requested a review from Copilot March 27, 2026 07:52

Copilot started reviewing on behalf of JackieTien97 March 27, 2026 07:52 View session

Copilot AI reviewed Mar 27, 2026

View reviewed changes

shuwenwei added 2 commits March 27, 2026 16:12

fix copilot review comments

807005a

fix it

500131a

add it

0701d5e

shuwenwei marked this pull request as ready for review March 27, 2026 10:20

fix it

20352f7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support COPY TO TsFile for exporting query results in table model#17372

Support COPY TO TsFile for exporting query results in table model#17372
shuwenwei wants to merge 8 commits intomasterfrom
copy_to

shuwenwei commented Mar 27, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 27, 2026

Uh oh!

Uh oh!

Copilot AI Mar 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Mar 27, 2026 •

edited

Loading

Uh oh!

sonarqubecloud bot commented Mar 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

shuwenwei commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sonarqubecloud bot commented Mar 27, 2026

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shuwenwei commented Mar 27, 2026 •

edited

Loading

codecov bot commented Mar 27, 2026 •

edited

Loading