Repository navigation
Conversation
Rename the closing thank-you from "Sling" to "sling-cli" so the contributin guidance names this repository rather than the broader ecosystem.
- New HanaConn on github.com/SAP/go-hdb. The DSN is rebuilt from only the query keys go-hdb accepts, sparing users failures on stray URL params; TLS turns on via TLSServerName set from the original host so HANA Cloud still connects through an SSH tunnel. - go-hdb yields LOB locators and sql decomposer decimals instead of plain values. Scan NCLOB/CLOB/BLOB into NullLob, convert unbindable values on insert, and turn Decompose() decimals into big.Rat in CastVal so precision survives instead of erroring. - Load with one single-row insert reused across rows (go-hdb bulk exec), force default tx isolation since HANA rejects others, and keep `do begin ... end` as one statement so drop blocks parse. - Register type, templates, and connection properties; update README and LLM resources; add unit tests plus CLI cases 812/813 covering a 100K-row CSV and a repeat incremental upsert.
…FIER
- Full-refresh and truncate custom SQL with a range (chunking) must
contain {incremental_where_cond} or {incremental_value}. Before, each
chunk read the same rows and the run showed success.
- BCP: add dashes to 32-hex values for uniqueidentifier columns, and
load empty strings as NULL.
- BCP: shorten the error text. Remove progress lines, duplicate lines
and row values, and cap the line count.
Refs SLING-98
Refs SLING-98
HarnessProviderChoice passes the model of the key it finds to opencode. google/gemini-2.0-flash was shut down on 2026-06-01, and anthropic/claude-sonnet-4-5 is retired on 2026-11-30. They are now anthropic/claude-sonnet-5-5 and google/gemini-3.8-flash. Both ids are in the opencode model catalog (models.dev), and only the model string is sent, so no request parameters change. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ult-models Update the harness default models of the Anthropic and Google keys
- connection_local: validate the resolved conn.Name rather than the raw name argument, so requesting an unknown connection actually errors instead of falling through to discovery on an empty connection. - iop/ssh: unwrap *url.Error before wrapping the SSH_TUNNEL parse failure, since url.Error embeds the raw URL and would leak the password in logs and error output. - Add regression tests covering both the unknown-connection error message and the absence of credentials in the tunnel parse error.
The server picked a free port, released it, and bound it again in a goroutine after a fixed 100ms sleep. A bind error was not logged. Now the listener is bound at once on 127.0.0.1, and a serve error is logged. SLING-109
duckdb with MotherDuck can read the source of one query again after the first read: a second HEAD, and sometimes a second GET. The batch was served once and the server closed after the first GET, so the query failed with "Could not establish connection error for HTTP HEAD". Now the first GET also writes the batch to a spool file, and each later GET gets the spool. The producer keeps the batch and the server until the consumer calls the new HttpStreamPart.Done after the query returns. SLING-109
fix(s3): support virtual-hosted addressing for custom endpoints
A check with on_failure: break stopped the step loop. The steps after it sent no status. The platform kept their tasks pending and marked them stalled after 5 minutes. A forward goto had the same problem for the steps that it jumped over. The loop now continues and runs these steps with skip=true, so each step sends a "skipped" status. Test 814 captures the status posts.
- Add dbase and firebolt connection templates, plus Athena's required data_location property. - Rename properties to match what the code reads: motherduck_token, prometheus http_url, and s3 url_style (replacing force_path_style). - List ADBC, DynamoDB, Fabric, Firebolt, ODBC, and ScyllaDB in the supported connector docs and update the per-category counts. - Fix the Oracle docs URL and reverse the credential precedence order to match loader behavior. - Correct assistant skill docs: split_part is 0-based, hmac and sha512 exist, gs is the filesystem type, and batch_limit, `sling validate --parse-only`, and change_capture retry options replace the stale flags.
- StarRocks: the stream load PUT ends 10s before the SLING_TIMEOUT deadline. Before, its deadline was the timeout prop + 30s (10h for timeout=36000), so SLING_TIMEOUT killed the exec and the load error was lost. - Threads: after one child cannot connect, the streams that did not start yet fail at once with "target unreachable" (shared code in sling). - Add TestStarRocksStreamLoadTimeout and CLI test 815.
Compile now checks the source and target names first. One error names all the missing connections. A Google Drive folder or file that is not found now tells the user to share it with the service account email.
The Postgres ADBC driver gives numeric as text with no typmod. The lane used the default decimal(24,6), so it cut values to 6 decimals. - Give the source catalog columns to the lane read. Use their precision and scale for numeric fields. - Stage 2 declines a decimal with no precision, or a precision above 38. - The ClickHouse columns template returns the precision and scale. The row path to ClickHouse ADBC used decimal(24,6) before. - ColumnsToArrowSchema keeps a scale of 0 when the precision is set, and uses decimal256 above 38 digits. - AppendToBuilder uses the scale of the builder type, and cuts toward zero (-1e-10 became -0.000001 before).
…129) With SLING_DIRECT_INSERT and chunk_count, each chunk created the final table. On ClickHouse Cloud, the chunks that lost the race failed with TABLE_ALREADY_EXISTS. - createTableIfNotExists holds a lock for each table in the process. A chunk that gets "already exists" after a chunk of the same process created the table continues. - A chunk in a different process (SLING_THREADS > 1) continues when the create fails with "already exists". A stream without chunks keeps the error, because it can show overlapping schedules. - The Arrow lane gate and the write path read SLING_DIRECT_INSERT, not only the direct_insert option. Incremental chunks by chunk_expr lost rows with direct insert. Each chunk read the target max at its start, after other chunks wrote rows. ProcessChunks now reads the target max once and gives it to each chunk.
- Open Postgres connections with the pgx stdlib driver. Reads use the
binary protocol and take less CPU (8-15% in the benchmark).
- Give timestamptz values in the session time zone, as lib/pq did.
- Do not use named prepared statements, for PgBouncer in transaction mode.
- Load with pgx COPY. Use a text COPY when a column type has no pgx
binary codec (timetz, extension types).
- Keep the lib/pq type names and the timetz format ("08:30:45+02:00").
- Add CLI test 816 for timestamptz in the session time zone.
The row path sent each row through an unbuffered channel. Each row was a goroutine handoff, and the Go scheduler used most of the CPU. - Batch: the producer sends chunks of 256 rows. A timer sends a chunk that is not full after 50ms, if the consumer waits. - Batch.Rows and Datastream.Rows are now iterators (iter.Seq). - Transforms and column padding run on the consumer side. - Batch.Close gives all pushed rows to the consumer. It does not give rows back to the producer. - ds.Count counts the rows that the consumer takes. A new producer counter drives the limit checks. - The bytes-written goroutine gets one chunk at a time. - Datastream.Close keeps the ready signal, because a stream can now close before a consumer starts. - NewCsvReader uses the latest batch when BatchChan is closed. - NewColumns does not copy whole columns (data race on the checksum). - SQL Server: the rows in transit after Close go into the same file. - Remove the unused Datastream.MapParallel. - Tests 622 and 623 read 5M rows, so that the duckdb sidecar lives long enough to see. PG to stdout, 1M x 75: wall 26.8s to 18.3s, CPU 44.9s to 34.7s. PG to ClickHouse ADBC: CPU 46.1s to 33.2s.
Databricks reports ShortType and ByteType as short and byte, which had no native_type_map entry and fell back to text. The Zerobus Arrow schema then used LargeUtf8 for a column the table creates as SMALLINT. Signed-off-by: Alex Castillo <alex.castillog33@gmail.com>
A child process got its task config as inline JSON in SLING_TASK_CONFIG. Linux limits one env value to 128 KiB, so a large SQL gave E2BIG. - Config.UnmarshalTaskEnv accepts config text, a file path, or a file:// path. - It parses the file content as config text, so env vars expand as with inline JSON. - Add a unit test (test-core) and CLI test 817 (inline 150 KB SQL, retries on).
…-153) A late StateSet from the 5 s ticker read the Env of each task while runTask wrote SLING_PROJECT_ID for the next stream. This caused "concurrent map iteration and map write" after a successful stream. - Execute stops the ticker and waits for it to exit before it returns. - Config.SetEnv and the Prepare env loop take the state lock. - GetFormatMap holds the state read lock. RuntimeState uses the internal variant, because it holds the lock already. - initRuntimeState updates the state fields under the lock. - GetStore returns a copy. MergeStore merges under the lock. - The API connection gets a copy of the store. A deferred merge writes it back on all exit paths.
…G-134) The stream processor gives an empty binary value as a string. pgx encodes only []byte into bytea, so the COPY failed. copyRows now changes a string in a bytea column to []byte. Test 244 now has an empty bytea row.
The bytes-written processor read the same row slices that the consumer gets. The consumer changes the values in place (for example, the native ClickHouse writer converts decimals and integers), so the processor read changed or torn values. The byte count was then much too large (48 GB for 1.3 GB of data). The producer now copies each chunk for the processor before the consumer can get it. SLING_PROCESS_BW=false skips the copy.
…ING-130) On the row path, a source with use_adbc reads with its native driver. Only the arrow lane reads with ADBC. Log an info line with the reason when the lane is off for such a source. DuckDB and adbc connections always read with ADBC, so they get no line.
…LING-126) A local .arrow source (file:///path/x.arrow) failed with "unsupported arrow file format:". The file listing did not know the arrow extension, so the lane got an empty format. - The lane gate gets the format from the format option, then from the stream path, then from the connection URL. - A format option of parquet or arrow keeps the lane when it agrees with the file extension, or when the file has no known extension. - FileNodes.InferFormat knows the arrow extension. - Add gate tests and CLI test 738.
…SLING-127) The lane wrote .arrow files in the IPC stream format, and the row path wrote the IPC file format. The footer reader read only the file format, so Sling could not read the files that the lane wrote. - The lane file writer uses the IPC file writer. It needs only a writer, not a seek, so the pipe stays as it is. - ArrowIPCFileSchema and ConsumeArrowReaderSeeker check the magic bytes and also read the stream format (older lane output, conns exec --output arrow). - Add iop and file set tests, and CLI test 739.
… Parquet The row path gave an unconstrained numeric the precision of the first rows, for example decimal(24,15). A later value with more integer digits overflowed, and pqarrow wrote a value that was not valid. - Columns.FileDecimals gives a decimal that is not from the source DDL the max precision (38). - The Arrow writers check each decimal value against the precision and scale. A value that does not fit stops the run with a clear error. - The DuckDB file path shows the row error, not the short read of DuckDB. - Add unit tests and CLI test 837.
ExecuteReplicationHook set the status to error but not te.Err. The platform then booked the failed hook run with no error. Item: SLING-196
CleanHeaderRow lowercases every header but tracked duplicates with the original casing, so a header like "ID,id" became "id,id". Key the duplicate check on the lowercased name so the second column becomes "id1". Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sling set client_charset=UTF8 on each Oracle connection. Then go-ora sends UTF-8 bind bytes, but the session announces the server charset. On a non-UTF-8 target (for example WE8ISO8859P1), the server stores the bytes without conversion, and the non-bulk insert path corrupts non-ASCII text. Without the default, go-ora encodes binds in the negotiated server charset. Add suite test 843: a self-contained Docker Oracle with a WE8ISO8859P1 PDB, use_bulk false, VARCHAR2, NVARCHAR2 and CLOB.
…SLING-207) On a database charset such as WE8ISO8859P1, the non-bulk path lost the characters that the charset cannot hold, also for N-type target columns (for example U+0142 became U+00FD). - Bind string values for NVARCHAR2, NCHAR and NCLOB target columns as go_ora.NVarChar, so go-ora sends them in the national charset. - Give the temp table the N-types of an existing target (not for full-refresh). Before, a VARCHAR2/CLOB temp column lost the text. - OracleConn.GetNativeType maps the DbType NVARCHAR2 to nvarchar2(n) (nclob above 2000 chars) and NCLOB to nclob. - CastColumnForSelect does not cast a CLOB to varchar2(4000) for an NCLOB target. Before, a value above 4000 chars into an existing NCLOB column failed with ORA-22835. Extend suite test 843 with an NVARCHAR2/NCLOB target, through the temp table and with direct_insert.
- Oracle reports data_length=4000 for CLOB, NCLOB, BLOB and LONG columns - the sqlldr ctl took that length, so values over 4000 chars failed - a LOB column now uses the text(N) precision, else the 400000 default - add test 844 (p.61) to load a 10000-char value into a CLOB Ref: #446
fix(iop): dedupe CSV headers case-insensitively
NewExcelDataset ignored the error from GetRows when no cell range was given, so a misspelled sheet name silently produced an empty dataset.
When internal_stage names an external stage (S3, GCS, Azure), the read failed: Snowflake does not allow GET on an external stage. Now sling reads the stage location with DESC STAGE. For an external stage, it reads the unloaded files from the cloud location, with the credentials of the Snowflake connection, and then deletes them. No local download occurs. A load through an external stage now fails with a clear error, because Snowflake does not allow PUT on an external stage. Fixes #716
Add model.update_key and model.unique_key to the hook state. Remove omitempty from the model fields, so that an empty value renders as an empty string. Add CLI test 462 for a ClickHouse start hook. Refs SLING-209
fix(excel): return an error when the requested sheet does not exist
…nking The chunking decision and StreamRowsChunked parsed "now-2M" / "now-7d" with time.ParseDuration, which rejects d/w/M. The error was ignored in the decision (so the query was never chunked) and returned as an error in StreamRowsChunked. Share the existing toHourDuration conversion through a package-level helper and use it at every duration parse site. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tive-duration fix(prometheus): accept d/w/M suffixes in relative start/end when chunking
Regression test for SLING-210. A 429 then a 200 over HTTP/2 must get the same body on each attempt.
…ta race Chunked range queries had these problems: - The chunk size was always 1h. Now it is step x max(60, ceil(1h/step)), so each call returns about 60 points for each series. The size is a multiple of step, so the chunks keep the evaluation times of one full query. - Prometheus includes both ends of a range. Each chunk ended at the start of the next chunk, so each boundary point came back twice. Each chunk now ends one step before the next chunk. - step did not accept d, w and M. It now uses toHourDuration. - A range shorter than one chunk went to StreamRowsContext without the query options. One chunk now reads it. - The producer goroutine set ds.Columns and ds.Ready while the consumer read them. The first rows are now read before the goroutine starts. promChunkReader owns the chunk window and the row conversion. Datastream.Collect now clears the buffer under the context lock, because Close also clears it. Refs SLING-211
With the default 1m step, now-2M is 1,488 chunks. Each test took about 2-3 minutes. With step 1h, it is 25 chunks and each test takes about 4s. Refs SLING-211
Collect gave its dataset the ds.Columns slice, and NewColumns wrote into it while the producer goroutine read it. Collect now reads ds.Columns only after the rows loop ends, and gives the dataset a copy. The producer used the StringType that Collect wrote into a column without a type. Start now sets that type before the goroutines start. The Prometheus tests use ds.Collect again, so they check for this race.
Change-capture now uses primary_key, else the PRIMARY KEY of the source table, else _id for MongoDB. With no key, the run stops before it reads data, in place of a bad WHERE () IN (SELECT ...) merge on the second run. - primary_keys templates (mysql, mariadb, postgres, sqlserver) return only PRIMARY KEY columns, not UNIQUE keys. - mysql/mariadb change_capture merges do not use ON DUPLICATE KEY, because the target table that Sling makes has no unique key. The soft merge UPDATE no longer refers to aliases that the ON clause does not use. - Remove the Iceberg check that needed primary_key in the config. - Tests: p.62 (key from source, MariaDB and DuckDB targets), r.145 (fail fast), p.31 runs on MySQL and MariaDB with a soft merge pass.
- primary_keys reads pg_constraint. information_schema hides the constraints from a role that has only SELECT on the table. - New replica_identity_keys template: the columns of the index set by REPLICA IDENTITY USING INDEX. - Suite test 849: Postgres source as a REPLICATION + SELECT role, a key with two columns, a replica identity index with and without a primary key, and the fail-fast check on a table with no key.
Add the unique_keys template for MySQL, MariaDB, PostgreSQL, SQL Server and Oracle. It finds the unique index with the fewest columns, where all key columns are NOT NULL. It does not use filtered, partial or expression indexes. SQL Server first uses the @index_name of the capture instance. The PostgreSQL replica identity query now reads only key columns, not INCLUDE columns. Test 847 now runs three streams: one with a unique NOT NULL id, one with no key, and one with a nullable unique id. The run must fail and name the two streams with no key.
Provides a single entry point for the pre-release test set (cli, connections, dbio, core, python) without running the slower cdc and eval suites that `test-all` includes. Fails fast via `set -e` and prints a pass marker.
Since 214782a, the chunked reader called SetReady before it pushed rows. The ready buffer was empty, so a file target wrote 0 rows ("Nothing to do"). The transforms also changed the column types after the dataflow took the columns, so a cast to string had no effect on the schema. Now the chunks feed an iterator and ds.Start reads the first rows through the transforms before SetReady. A cast transform sets ds.Inferred, as the non-chunked path does. CLI test 80 passes again.
The archive went to ~/<asset> for all processes. Two processes that
download DuckDB at the same time overwrote or removed the same file
("unexpected EOF", "no such file"). CLI tests 794 and 803 failed this way.
- Test 38: r.07 now has 19 streams, because tests/files has test_clob_large.csv. - Tests 84/85 and 845/846: put each pair in a group. Each pair uses the same table, and the parallel runs dropped the table of the other test. - Test 760: comment out. The DATABRICKS workspace is inactive since 2026-10-13. TestArrowLane_Gate covers the zerobus decline.
…error A killed sidecar can stop the stdout scanner before its exit is recorded. procDeathErr then reported "stdout scanner stopped", which IsDuckDbProcDeath does not match, so the task did not retry. Under load, CLI test 622 failed this way. Now procDeathErr waits up to 2s for the exit after a scanner error. If the process exited, the error is a process death.
The chunked Prometheus stream now applies column_casing (normalize), as the non-chunked path does. Snowflake columns are upper case, so the quoted lower-case names in the end hook did not resolve.
Step 5 read all tables of sling_test. Other tests create and drop tables in that dataset at the same time, so the step failed when a table was dropped during the read. Now the wildcard is sling_test.xdb_*, and the step creates xdb_extra for it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New Features
primary_key, then the Postgres replica identity index, then the table primary key, then a unique index on NOT NULL columns (_idfor MongoDB). Fail before any read when streams have no key, and name all of them in one error.parallel_scansource option to read a Postgres table with parallelctidranges on the Arrow lane.insert_concurrencytarget option for ADBC and native ClickHouse targets.proxy_url.filter()function, for examplefilter(array, "value > 1").pgxdriver instead oflib/pqfor Postgres. Reads use the binary protocol and take less CPU.big.Rat) and the row path (rows pass between goroutines in chunks)..arrowfiles).use_adbcdoes not change the read.claude-sonnet-5-5,gemini-3.8-flash).mcp-goto v1.5.0 for the current MCP protocol.dbaseandfirebolt).Bug Fixes
{incremental_where_cond}or{incremental_value}placeholder. Before, each chunk read the same rows.TABLE_ALREADY_EXISTSon ClickHouse Cloud).proxy_urltunnels for Postgres (verify-full), MySQL, SQL Server and ClickHouse.pgx) for empty binary values and for integer or text values in bool columns.role), and Snowflake one-line PEM keys from a secret manager.client_charset, keep NVARCHAR2/NCLOB text on non-Unicode databases, and fix LOB column length forsqlldr.UNIQUEIDENTIFIERvalues.tar.gzasset, bind the HTTP import listener early, and let DuckDB read an HTTP import batch more than once (MotherDuck).shortandbytenative types (map tosmallint).true/falsefor1/0.d/w/Msuffixes, use step-based chunk size, and remove boundary duplicates.{model.*}variables insling buildmodel hooks, and runKILLand similar statements withExecon ClickHouse.breakorgoto: send the skipped status.SLING_TASK_CONFIGfor large configs: read it from a file path (E2BIGon Linux).Env, state store and shared replication state.Datastream.Collect, and a 0-row file write for small streams.explodeand alias docs.