Skip to content

chore: standardize the repository like tree-sitter-kotlin - #6

Merged
exKAZUu merged 8 commits into
mainfrom
standardize
Sep 29, 2026
Merged

exKAZUu merged 8 commits into
mainfrom
standardize

Conversation

@exKAZUu

@exKAZUu exKAZUu commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Why

This repository was imported with upstream's setup: npm with ESLint, the C/Go/Node.js/Python/Swift bindings, and upstream's CI, fuzz, and lint workflows, with no release pipeline. WillBooster/code-gauge needs a Rust crate of this grammar that it can pin to a release, as it does with WillBooster/tree-sitter-kotlin. That requires the same tests and release pipeline as that fork.

Requirements

  • required: fork the grammars that need fixes as WillBooster/tree-sitter-kotlin is forked, following its format.
  • required: do not be impolite to upstream or cause it trouble.
  • required: keep grammar.js in the npm package, since WillBooster/tree-sitter-typescript will build on it.
  • required: fix defects that the stricter checks find at their root cause in this PR instead of exempting them from the checks.
  • required: leave the JSX text and attribute strings containing a bare & and the tagged templates with invalid escapes (e.g. String.raw) unfixed here; they get their own issues.
  • required: leave the misparse of a line starting with [ after an arrow function with a block body (examples/cli/lib/base-cmd.js in script/known-failures.txt) unfixed here; the requester files its issue.
  • chosen: end declarations in a for_statement initializer only with ;, because ECMAScript inserts no semicolon inside a for header and accepting one there made for (let x\n of y) ambiguous, which broke incremental reparsing.
  • chosen: credit upstream in the README (one sentence, "not an official release") and keep the upstream copyright line in LICENSE, as tree-sitter-kotlin does, without naming the upstream authors as authors of this fork's packages.
  • chosen: publish only under WillBooster names (@willbooster/tree-sitter-javascript, the crate willbooster-tree-sitter-javascript), so nothing competes with upstream's tree-sitter-javascript packages.
  • chosen: ship only the Wasm build and the Rust binding (exporting upstream's LANGUAGE and NODE_TYPES), the two forms the Kotlin fork ships and its consumers use.
  • chosen: parse the repository upstream's CI cloned (npm/cli), pinned to a commit, plus the checked-in examples/, against script/known-failures.txt, so the list of failures changes only with the grammar.
  • chosen: keep upstream's queries/ in the repository, untested and not referenced from tree-sitter.json, because the test layout allows only corpus and unit tests and neither package ships it, so the shipped tree-sitter.json must not name query files absent from the npm package and the crate (the alternative, shipping untested queries, was rejected); the Kotlin fork's metadata names no queries either.

Customer Summary

Nothing is released yet. After this merges, the first feat/fix commit releases @willbooster/tree-sitter-javascript to npm (with tree-sitter-javascript.wasm and grammar.js) and the willbooster-tree-sitter-javascript crate to crates.io, as version 1.0.0.

Technical Summary

  1. Copied from WillBooster/tree-sitter-kotlin and renamed: agent instructions, issue and PR templates, lefthook, bun, mise, oxlint/oxfmt, Renovate, .releaserc.json, the test/test-rust/robustness/release/semantic-pr workflows, script/, test/unit/, bindings/rust/, Cargo.toml, Cargo.lock, and package.json.
  2. Removed: package-lock.json, eslint.config.mjs, the non-Rust bindings and their build files, upstream's FUNDING.yml, issue templates, dependabot config, and workflows, and the highlight and tag tests in test/highlight and test/tags.
  3. tree-sitter.json keeps upstream's grammar entry without the query paths, since neither package ships queries/, names WillBooster Inc. as the author, and enables only the Rust binding.
  4. grammar.js is reformatted by oxfmt and made lint-clean: () => for unused $, \u0000-\u001F instead of \x00-\x1F, an unnecessary \- removed, and the u flag on the regexes using \p{…}. At the first commit, the regenerated src/parser.c is byte-identical to the one generated from upstream's grammar.js, and src/grammar.json differs only in those regex spellings and flags. What to check hardest: the \[ escapes in the regex-pattern and identifier character classes stay with lint suppressions, because the autofix removed them and tree-sitter's regex parser then rejects the pattern.
  5. The Robustness test job fails when the committed src/ differs from what grammar.js generates, since both packages ship the committed src/. test/unit/performance.test.ts also rejects a Wasm build compiled from a src/parser.c older than grammar.js. Renovate updates the fuzz job's tree-sitter-version together with tree-sitter-cli through a regex custom manager in renovate.jsonc. oxlint and oxfmt ignore examples/, whose JavaScript is test input: the autofix had rewritten it.
  6. The second commit fixes an incremental-parsing defect that the fuzz check found in the corpus case "For statements". After let j inside for (, both a declaration ended by an automatic semicolon and a for-in header ended by an optional automatic semicolon were live, so for (let j\n of k) stayed ambiguous until of. After an edit that turned of k into something else and its undo, the reparse reused the lexical_declaration built when the for-in path had died, and yielded an ERROR. ECMAScript never inserts a semicolon inside a for header, so the declarations in a for_statement initializer now end only with ; (for_lexical_declaration/for_variable_declaration rules used only through aliases to the public node names, so the node types are unchanged; they are visible rules because a hidden one would make for_statement inherit their kind field), and _for_header no longer accepts an automatic semicolon. The ambiguity and its GLR conflict are gone, and for (var j\n of k) now parses, where it used to fail; the corpus case covers it. What to check hardest: that the 1,640 example files parse to the same trees as before the fix (checked), and that rejecting for (let i = 0\n i < n; i++), which is not valid JavaScript, loses nothing real-world code needs.
  7. src/ is regenerated with tree-sitter-cli 0.27 (upstream used 0.25).

Testing

  • bun run verify, bun wb test (corpus 116/116 with the native and the Wasm build, incremental-parsing fuzz, performance, and examples), and cargo test --locked pass locally.
  • tree-sitter fuzz passes on all corpus cases with seeds 1, 2, 3, 4, 5, 42, and 1234 (1,000 iterations, 10 edits) and with seeds 7 and 99 (20 edits).
  • The parse trees of all 1,640 example files are identical before and after the fix.
  • script/parse-examples: 1 of 1,640 files fails, examples/cli/lib/base-cmd.js, which is valid JavaScript: a line starting with [ after an arrow function with a block body is parsed as a subscript of that arrow function instead of a new statement.
  • Not run: the release workflow (it runs only on main), the macOS Intel and Linux arm64 jobs, and libFuzzer (Apple clang has no libFuzzer).

Notes

  • The highlight and tag tests were removed because the test layout allows only corpus and unit tests; queries/ is kept but no longer tested (neither package ships it).
  • semantic-pr / semantic-pr runs on pull_request_target, which uses the workflow files on main, so it cannot run for this PR.
  • Publishing uses trusted publishing on npm and crates.io, which has to be configured for the new package names before the first release. crates.io lets a trusted publisher be configured only for an existing crate, so the first crate version must be published manually (as the Kotlin fork's was) before the first automated release; otherwise that release publishes the npm package and fails at the crate.

exKAZUu and others added 8 commits September 28, 2026 22:49
Co-authored-by: WillBooster (Claude Code) <agent@willbooster.com>
ECMAScript never inserts a semicolon inside a for header, but the grammar
accepted an automatic semicolon after its declarations. That made
`for (let x\n of y)` ambiguous until `of`, so an incremental reparse could
reuse a declaration built when the ambiguity had been resolved the other way,
and `for (var x\n of y)` failed to parse.

Co-authored-by: WillBooster (Claude Code) <agent@willbooster.com>
Co-authored-by: WillBooster (Claude Code) <agent@willbooster.com>
…e CLI

Co-authored-by: WillBooster (Claude Code) <agent@willbooster.com>
Co-authored-by: WillBooster (Claude Code) <agent@willbooster.com>
…kind field

Co-authored-by: WillBooster (Claude Code) <agent@willbooster.com>
Co-authored-by: WillBooster (Claude Code) <agent@willbooster.com>
Co-authored-by: WillBooster (Claude Code) <agent@willbooster.com>
@exKAZUu
exKAZUu merged commit 0c5b3db into main Sep 29, 2026
9 checks passed
@exKAZUu
exKAZUu deleted the standardize branch September 29, 2026 00:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant