Skip to content

Formalize the ABI template, and generate language bindings outside ts-gen - #207

Draft
guybedford wants to merge 6 commits into
gbedford/zena-targetfrom
gbedford/abi-format
Draft

guybedford wants to merge 6 commits into
gbedford/zena-targetfrom
gbedford/abi-format

Conversation

@guybedford

Copy link
Copy Markdown
Contributor

This makes the Wasm ABI template a formal, versioned format, so language bindings generators can live outside ts-gen. It moves the Zena generator out onto that format alone, and adds a Kotlin/Wasm worker whose generator is written in Kotlin.

--target wat already wrote the ABI as a WAT import template, but a generator could only consume the ABI through ts-gen's Rust model. That meant every language target had to live in this repository. The template now carries the whole model as (@abi.*) annotations. These are the WebAssembly annotations proposal syntax, so any WAT tool still assembles the file, and a generator in any language can read it with a small S-expression reader.

(@abi.class "Request" (extends "Body"))
(import "./bindings.wasm" "Request.url"
  (func $Request.url (@abi.get "url") (@abi.owner "Request")
    (@abi.ts "readonly url: string")
    (param $this externref (@abi.ref "Request"))
    (result externref (@abi.ref string))))
  • ABI.md specifies the format. It covers:
    • the version
    • owner records, with kind, module, aliases, extends and docs
    • one operation per import: get, set, call, new and the sys operations
    • reference heads and empties, aliases, variadics and handlers
    • a grammar of the forms used
  • The ts-gen-abi crate holds the model, the writer and a reader, and has no dependencies. Every fixture template, workers-types included, reads back and rewrites byte for byte. The snapshot test checks this on every run.
  • Owner references in heads and extends are absolute: "Path" for a global owner, "Path" (module "spec") for a module's. Previously a head inside declare module resolved to the module's owner of that path first. That made a module unable to reference a global owner its own owner shadows. This changes heads in the ambient-module-merge, constructor-vars and member-scopes fixtures.
  • Owner records now carry extends, since inherited members are not repeated on derived owners and a generator needs to follow them.
  • ts-gen embed --abi <template> builds from a template without the .d.ts.
    • --bindings-module <name> links only ABI-named imports of that module and leaves the rest to the host. Kotlin's @JsFun imports all come from js_code.
    • --imported-string-constants <name> sets the string-constants module the bindings use.
    • Validation accepts legacy exception handling, which Kotlin emits by default.
  • A JsValue.undefined sys function gives modules an undefined value.
  • The Zena generator moves to generators/zena as ts-gen-zena, a CLI and crate that depends only on ts-gen-abi. --target zena is removed. It reproduces all Zena snapshots from the templates alone, apart from the head fixes above.
  • examples/kotlin-worker is a Kotlin/Wasm worker.
    • Its generator is Kotlin/JVM with its own reader, and has no dependency on ts-gen.
    • It emits @JsFun externals named so their imports are the ABI's names, typed owner interfaces over them, and a @JsExport entry for each handler.
    • run.sh serve builds it through embed --abi --bindings-module js_code and serves it in workerd.
    • check-templates.sh generates and compiles bindings for every fixture template.

Tests cover the format round trip, reader rejections, absolute owner references, embedding from a template, bindings-module linking and legacy exceptions. The Zena compile and worker tests need a Zena CLI with value imports, so they run only when ZENA is set and were not run for this change. The Kotlin example is not in CI because it needs a JDK and Gradle.

Open questions surfaced by the Kotlin generator:

  • Function results that are Promise<T> carry only the Promise head, so a generator cannot type what they resolve to. Type arguments on heads, such as (@abi.ref "Promise" (of string)), would fix that for Promise and other generics.
  • Kotlin turns every undefined an external returns into null, and cannot pass undefined either. A position that admits undefined but not null could have embed map a Wasm null to undefined.
  • A module class collapsed from a global constructor var still types its new result as the module class, while its statics now return the global interface, as in TypeScript.

The `--target wat` template listed the ABI's imports, but a bindings
generator also needs what only the model held: each import's owner and
JS operation, what each reference position names, and which owners
inherit from which. So `--target zena` had to live inside ts-gen, and
`embed` had to re-derive the ABI from the `.d.ts`.

The template now carries the whole model as `@abi.*` annotations, which
WAT tools ignore, so the file still assembles. ABI.md specifies the
format. The model moves to a `ts-gen-abi` crate with no dependencies,
holding the reference writer and reader.

* Owners are first-class: `(@abi.class "Request" (extends "Body"))`,
  with their kind, module, aliases, heritage and docs. Inherited members
  are only on the base, so a generator needs `extends` to present them.
* Each import names its operation (`(@abi.call "get")`, `(@abi.get "x"
  (on "WebAssembly"))`, `(@abi.new "Headers")`), its owner, its TS
  declaration and its JSDoc.
* Each reference position says what it names and which empties it
  admits: `(@abi.ref string null)`. Scalar heads (`number`, `boolean`,
  `bigint`) mark a nullable scalar that crosses as a reference.
* A `JsValue.undefined` sys function gives `undefined` to a language
  that cannot import globals.

The snapshot harness checks that every fixture's template reads back
and writes the same text.
…le names

`embed --abi <template>` reads the ABI from the file a generator wrote
bindings from, instead of re-deriving it from the `.d.ts`, so the two
cannot disagree.

Two options let a language link the ABI without naming its imports.

* `--bindings-module <name>`: Kotlin's `@JsFun` always imports from
  `js_code`, under the function's qualified name, which a generator can
  make the ABI name. Imports from that module that are ABI functions
  link to the bindings; its other imports are left to the host.
* `--imported-string-constants <namespace>`: the bindings import their
  string constants from it, so the merged module compiles with the one
  `importedStringConstants` the language module already uses.

`embed` also implements the new `JsValue.undefined`.
`--target zena` needed the ABI model ts-gen held in memory. With the
model in the template, a generator can read the file instead, so Zena's
generator becomes `ts-gen-zena`, a crate depending only on `ts-gen-abi`,
which can live in Zena's own tree.

* `ts-gen-zena <template.wat> -o bindings.zena` writes the bindings, and
  `main.zena` and an `app.zena` scaffold for a template with handlers.
  `--target zena` is removed.
* Its tests generate from the templates ts-gen's snapshot harness
  commits for every fixture, so the generator is tested the way any
  consumer of the format runs it. Every snapshot is unchanged from
  `--target zena`, read from the file alone.
* The worker example builds through the template: `ts-gen --target
  wat`, `ts-gen-zena`, `zena build`, `ts-gen embed --abi`.
* The Zena mapping moves from CONVENTIONS to the crate's README.
A head of "W" inside declare module "m" was resolved to m's W when it
had one, else the global W, so a module could not reference a global
owner its own owner shadows, as a module's constructor var typed by a
global interface does. Owner references in heads and extends now carry
the module, as owner records already do, and the Zena generator looks
classes up by that identity.
… and owner-less heads

Kotlin/Wasm emits the legacy try/catch by default. ABI.md now says an
import must carry an operation of its version, since new operations
change the version, and that a head may name a type with no owner
record.
The generator reads the template with its own reader and no dependency
on ts-gen. Its @JSFun externals import from js_code under the ABI's
names, which embed --bindings-module js_code links to the bindings while
leaving Kotlin's own snippets to its import object.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant