Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

![Schema — architectural pieces sharing one design](images/schema.png)

Schema is a data language where you define your constants, enums, flags, types and tables for your game, then from this one definition, it generates code in multiple languages: C, C++, C#, Dart, Elixir, Go, Java, JavaScript, Lua and Rust.
Schema is a data language where you define your constants, enums, flags, types and tables for your game, then from this one definition, it generates code in multiple languages: C, C++, C#, Dart, Elixir, Go, Java, JavaScript, Lua, Python and Rust.

Schema is currently under active development. See **[ROADMAP.md](ROADMAP.md)** for details.

Expand Down Expand Up @@ -87,7 +87,7 @@ Cost to serialize a representative game packet, relative to generated C++ at 100
```
make # builds the compiler at bin/schema
bin/schema check <dir of .schema files>
bin/schema generate --lang c|cpp|cs|dart|elixir|go|java|js|lua|rust --out <outdir> <dir>
bin/schema generate --lang c|cpp|cs|dart|elixir|go|java|js|lua|python|rust --out <outdir> <dir>
bin/schema fmt <dir of .schema files>
```

Expand Down
18 changes: 11 additions & 7 deletions docs/PORT-PYTHON.md
Original file line number Diff line number Diff line change
Expand Up @@ -291,9 +291,10 @@ generated class per table and per map entry, each with its descriptor
(SPEC-TABLES §8.1), and one runtime under `<package>/_schema/`.

- **One module per field kind.** `_schema/kinds.py` is the dispatch table;
each `_schema/kind_<kind>.py` beside it registers a kind this cut carries
(today `kind_bool.py`, `kind_int.py`, `kind_map.py`, `kind_optional.py`,
`kind_table.py` and `kind_pointer.py`; the directory is the list). A later
each `_schema/kind_<kind>.py` beside it registers a kind the leg carries
(`kind_bool.py`, `kind_int.py`, `kind_map.py`, `kind_optional.py`,
`kind_table.py`, `kind_pointer.py` and the modules later cards added
beside them; the directory is the list). A later
card adds its module and one registration line in it, never an edit to the
body walk; `kinds.py` imports every `kind_*.py` itself, so no other file
names a new module. A kind that wraps another names the wrapped codec in the
Expand Down Expand Up @@ -417,7 +418,7 @@ generated class per table and per map entry, each with its descriptor
- `map[K]V` for the base key and value kinds: carried
([the table maps](port-python/table-maps.md))

- `?map[K]V`: refused
- `?map[K]V`: carried ([the table optional maps](port-python/table-optional-map.md))

- unions with table arms, and arrays of them: carried
([the table unions](port-python/table-unions.md))
Expand Down Expand Up @@ -554,9 +555,12 @@ out as lowercase hex lines.**

## THE REGISTER

The `python` column of [PORTING.md](PORTING.md) holds every technique ❌,
citing #381 (the language list) and the `port/python` layer that carries it,
or `—` with the platform's reason:
The `python` column of [PORTING.md](PORTING.md) marks a technique ✅ where
the leg carries it, naming the tests and targets that prove it; ❌ citing #381
(the language list) and the `port/python` layer that carries it; or `—` with
the platform's reason. M17's reason (every blob is one `bytes` or `str` of
its own length, so there is no slab to outgrow) is in its cell; the others
are:

- **M2 and M3**: every CPython `int` and `float` is a heap object, so there is
no unboxed form to keep.
Expand Down
26 changes: 13 additions & 13 deletions docs/PORTING.md

Large diffs are not rendered by default.

32 changes: 22 additions & 10 deletions docs/SPEC-TABLES.md
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,9 @@ C-like dialect of `serialize.h`, with library calls behind hooks (§13.9).
C++ and C carry both storage classes. C# carries both classes and the current
id-table file and bitpacked message forms, JSON, runtime cook writing, block
construction, native regions, builders, retain-unknown and UnitView. Go carries
these surfaces too. Rust, Elixir and Java carry both table wires, form
these surfaces too. Python carries both classes, form 1 and form 3, JSON,
retain-unknown, the block form's read and build, and the cook's read and
write (docs/PORT-PYTHON.md). Rust, Elixir and Java carry both table wires, form
1 and form 3 (§3.4), beside the block and cook read halves. **Dart and
JavaScript carry both wires too**, form 1 and form 3 (§3.4): form 3 is
the FIXED form — the write template, the plan-driven read and the LAYOUT /
Expand All @@ -289,8 +291,9 @@ language has a table backend; refusal is scoped to a construct, never the table

**WIRE FORM STATUS.** Section 3's id-table form is carried by the C++
reference, the compiler engine (`internal/tablewire`), C, C#, Go, Dart,
JavaScript, Rust, Elixir and Java. Section 3.4's FIXED form is carried by the C++ reference, by Go,
by Rust, by Elixir, by Java, by Dart, and by JavaScript, whose
JavaScript, Rust, Elixir, Java and Python. Section 3.4's FIXED form is carried by the C++ reference, by Go,
by Rust, by Elixir, by Java, by Python (`fixed_load`, `fixed_save`;
docs/port-python/table-fixed-form.md), by Dart, and by JavaScript, whose
layout/block, hash and record bytes are held against the reference's own over the
paired bench's corpus, the seven reference files (`tables-dart-fixed-form`), and over an
OPTIONAL corpus the reference writes at every place a present byte can ride —
Expand Down Expand Up @@ -1748,6 +1751,8 @@ reads.
**Backend status: the C++ REFERENCE and the TOOL carry it; C carries its file
wire, region and text forms; Go carries `*bytes` and `*string`** with the
blob-node runtime, the region and cook views and the text form (§8, §16.8).
Python carries both on its wire, cook and text form
(docs/port-python/table-blobs.md).
Every writer of the wire, the message form and the text refuses a blob named
from two slots by name, `blob_shared`, and the strict read refuses it too
(§6.5); the remaining ports refuse a unit that declares one, by name (§11), and
Expand Down Expand Up @@ -5405,8 +5410,8 @@ pointer, therefore no node table, therefore exactly the bytes §3 already
describes.

**Backend status for this section: the TOOL, the C++ REFERENCE, C and GO
write and read it; the other backends refuse a pointered unit's wire by name
(§11).**
write and read it, and so does PYTHON; the other backends refuse a pointered
unit's wire by name (§11).**
`schema pack`'s engine — the compiler-side encoder and decoder this repo runs
as tooling (§17.1) — and the generated C++ codecs are two implementations of
this section, written from it rather than from each other, and each reads what
Expand Down Expand Up @@ -10914,7 +10919,10 @@ the conformance `retain` rows. The reference emits
`TableRetain`, the three verbs on every variable-class root, the refusal on
every fixed-class one, `LoadRetainMessages` and `SaveRetainMessages` among
them (§3.3), and a second family of body functions beside the three the wire
already had, so `Load`, `Measure` and `Save` are unchanged. What is still owed
already had, so `Load`, `Measure` and `Save` are unchanged. Python emits
`TableRetain` and `load_retain`, `measure_retain` and `save_retain` on every
variable-class root and refusals by name on every fixed-class one, and refuses
message-form retention by name (docs/port-python/table-retain.md). What is still owed
is the other seven ports.

## 7. The cooked form
Expand Down Expand Up @@ -10945,8 +10953,8 @@ const Scene * scene = SceneOpen( bytes, length ); // point at it, or NULL

`schema cook` writes the file; the runtime only ever points at one.

**Backend status: the TOOL, EVERY READ SIDE, and the C, C++ and Go WRITE sides
for BOTH classes are built (schema#251).** `schema cook`, `schema cook-check` and
**Backend status: the TOOL, EVERY READ SIDE, and the C, C++, Go and Python
WRITE sides for BOTH classes are built (schema#251).** `schema cook`, `schema cook-check` and
`schema uncook` produce, validate and read back the form below, in both byte
orders, over the same IR the emitters consume — and `wire → cook → wire` is
byte-identical over the corpus, which is what proves the accelerator loses no
Expand Down Expand Up @@ -15872,11 +15880,15 @@ SceneFromJson( builder, text, text_bytes, &report );

**Backend status for this section: the FIXED class in every table backend —
C++, C, C#, Dart, Elixir, Go, Goose, Java, JavaScript, Lobster, Lua, Luau, MOO,
Rust and Squirrel — and the VARIABLE class in C and C++ (§16.7).** A pointered unit's text form reads through the builder. In C#, Dart,
Rust and Squirrel, and Python beside them — and the VARIABLE class in C, C++
and Python (§16.7).** In C and C++ a pointered unit's text form reads through the builder. In C#, Dart,
Elixir, Go, Java, JavaScript and Rust the absence is already made one level
up: a
pointered unit gets no table source at all (§11), so it has no text form for
the same reason it has no wire codec.
the same reason it has no wire codec. Python carries the text form in both
classes, a pointered unit's included: `T.from_json(text)` reads into the
generated classes and `value.to_json()` writes a node two pointers name once,
labelled `"&node"`, and names it by label after.

**The FLOAT SPELLING is C's `%.*g`, byte for byte, in every port**, and each
says how it gets there rather than reaching for its runtime's default. C++
Expand Down
15 changes: 9 additions & 6 deletions docs/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ document disagree, one of them is a bug (see VERSIONING.md).

**schema** is a small language for describing bitpacked network data, and a
compiler — written in Go — that translates `*.schema` files into generated C,
C++, C#, Dart, Elixir, Go, Java, JavaScript, Rust, Lobster, Lua, Luau and
Squirrel source code. Ten of the thirteen targets read and write through a
C++, C#, Dart, Elixir, Go, Java, JavaScript, Rust, Lobster, Lua, Luau, Python and
Squirrel source code. Eleven of the fourteen targets read and write through a
serialize-family runtime library; the other three carry the bit reader and
writer in the generated code itself and
need nothing but their own toolchain:
Expand All @@ -28,6 +28,7 @@ need nothing but their own toolchain:
| Lobster | [serialize.lobster](https://github.com/mas-bandwidth/serialize.lobster) |
| Lua | [serialize.lua](https://github.com/mas-bandwidth/serialize.lua) |
| Luau | [serialize.luau](https://github.com/mas-bandwidth/serialize.luau) |
| Python | serialize.py, staged in this repository at `third_party/serialize.py/` and shipped inside each generated package |
| Squirrel | [serialize.nut](https://github.com/mas-bandwidth/serialize.nut) |

The runtimes are bit-for-bit wire compatible, pinned in CI with shared golden
Expand Down Expand Up @@ -843,7 +844,7 @@ FloatExpr = float expression over float literals, int literals and const names
language — every packet target emits it in its own spelling:
`CapsCount` in C++, C#, Go, JavaScript, Lua, Luau, Squirrel, MOO,
Goose and Lobster; `CAPS_COUNT` in C, Rust and Java; `capsCount` in Dart;
`caps_count/0` in Elixir.
`caps_count/0` in Elixir; `Caps.Count`, a class attribute, in Python.
**`flags` is a
contextual keyword, not reserved** — it introduces a declaration at file
scope only, so a *field* named `flags` stays fully legal; declarations at
Expand Down Expand Up @@ -1249,7 +1250,9 @@ it (§4.6).
a write traps naming `non_finite_float`, a read refuses with it. Luau
(serialize.luau v2.1.3) enforces it through its vendored serialize
runtime: a write traps naming the finite rule, a read refuses with
`non_finite_float`. Every packet target enforces it. A
`non_finite_float`. Python's generated writer refuses one with
`InvalidValue` naming the field, and its vendored serialize.py refuses one
on read with `non_finite_float`. Every packet target enforces it. A
*quantized* `float32` is a different field: its wire is the step index and
not the pattern, per its row below. The table forms take the same rule
(SPEC-TABLES.md §3.3; 2026-10-27, `docs/rulings.json`, `tables-finite-only`).
Expand Down Expand Up @@ -1685,7 +1688,7 @@ a buffer of N + 1.

**Every packet target uses the same framing** — length over [0, N], then
align, then raw bytes. MOO emits a runtime `serializeString` call on both write
and read; C, C++, C#, Rust, Java, JavaScript, Dart, Elixir, Go, Lua, Luau,
and read, and Python a runtime `serialize_string` call on both; C, C++, C#, Rust, Java, JavaScript, Dart, Elixir, Go, Lua, Luau,
Squirrel, Goose and Lobster compose the framing from primitives. This is a
generated-code consequence, stated so no backend discovers it late: the
runtimes' `serialize_string` is the **wire oracle** for `string(N)`, not
Expand Down Expand Up @@ -2244,7 +2247,7 @@ face untrusted bytes; writers do not.
The packet `Write` contract includes row 8 (§5.1): a zero byte or used bytes
that are not well-formed UTF-8. Each leg checks writes by its own mechanism.
C, C++, C#, Rust, Java and Dart use a debug-only assert; Go, JavaScript,
Elixir, Lua, Luau, Squirrel, MOO and Goose use a refusal in every build.
Elixir, Lua, Luau, Squirrel, MOO, Goose and Python use a refusal in every build.
The reader independently refuses the same ill-formed text in every build
(§4.7). Ranges are trusted inputs everywhere: generated code never feeds
attacker-influenced values as min/max.
Expand Down
97 changes: 94 additions & 3 deletions docs/USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ and that this run did not write. With `--surface` nothing is removed; only

A declaration's name and its fields' names are checked against the reserved
words of EVERY supported language (SPEC §4.6) — C, C++, C#, Go, Rust, Java,
JavaScript, Dart, Elixir, Lua, Luau, Squirrel, Goose and Lobster — whatever
JavaScript, Dart, Elixir, Lua, Luau, Squirrel, Goose, Lobster and Python — whatever
target is generated, one diagnostic per name listing every language it
collides with: a schema compiled only for C++ today must still compile for
Elixir tomorrow. Go is the exception (SPEC §4.6, the Go
Expand Down Expand Up @@ -5122,8 +5122,99 @@ while node:
```
<!-- /lobster-table-usage -->

All nine are generated from the same IR and compared against each other in CI
on every push. The wire is bit-packed, so the property being checked is
**Python** — CPython 3.11 or later, standard library only, no dependencies
(the leg is tested on CPython 3.11.18 and 3.15.2; [PORT-PYTHON.md](PORT-PYTHON.md)
is its design). `schema generate --lang python --out <dir> <unit>` writes one
package per unit, named for the unit's `package`, with its runtime in the
package's own `_schema/` directory, so there is nothing to install: put
`<dir>` on `sys.path` and import the package. A value is an instance of a
generated class with `__slots__`, every field born at its declared default.
Integers of every width are `int`, an enum is a class of plain `int`
constants (`ShipType.Corvette`; the implicit zero is spelled `None_`), and a
flags set is an `int` mask (`ShipFlags.Boosting | ShipFlags.Aiming`). A
packet type reads with `T.read(data)` and writes with `value.write()`; a
table loads with `T.load(data, report)` and saves with `value.save()`, and
has `T.from_json(text)` and `value.to_json()` for its text form. A read takes
`bytes`, `bytearray` or a byte-format `memoryview` and returns the whole value or raises
`Refused`, whose `reason` names the rule; a write checks the whole value
before it writes a byte and raises `InvalidValue`, whose `path` and `rule`
name the field and what it broke. Both are `ValueError` subclasses. A `*T`
holds the instance or `None`, and two slots holding one instance are one node
on the wire and one object again after the load. These two fragments are run
against the `examples` and `tables/pointers` units under both pinned
interpreters by `make tables-python-usage`:

<!-- python-usage examples -->
```python
from example import InvalidValue, Refused, ShipCreate, ShipFlags, ShipType

ship = ShipCreate() # every field at its declared default
ship.ship_type = ShipType.Corvette # an enum is a plain int
ship.has_flags = True
ship.flags = ShipFlags.Boosting | ShipFlags.Aiming
ship.health = 750

data = ship.write() # the packed bits, as bytes
back = ShipCreate.read(data) # the whole value, or raises Refused
if back != ship:
raise SystemExit("the ship did not read back")
try:
ShipCreate.read(data[:1]) # a truncated packet
except Refused as refusal:
print("refused:", refusal.reason) # past_end
else:
raise SystemExit("a truncated packet was read")

ship.health = -1 # outside its declared [0, MaxHealth]
try:
ship.write()
except InvalidValue as refusal:
print("refused:", refusal.path, refusal.rule)
else:
raise SystemExit("an out-of-range health was written")
```
<!-- /python-usage -->

<!-- python-usage tables/pointers -->
```python
from graphdemo import ListNode, LoadReport, Refused, Scene

# a two-node list, its tail also named by `alias`: one node on the wire
tail = ListNode()
tail.value = 2
head = ListNode()
head.value = 1
head.next = tail
scene = Scene() # every field at its declared default
scene.name = "intro"
scene.head = head
scene.alias = tail

data = scene.save() # the id-table wire, or raises InvalidValue
report = LoadReport() # its counters change only on success
back = Scene.load(data, report) # the whole value, or raises Refused
values = []
node = back.head
while node is not None:
values.append(node.value)
node = node.next # None is the null reference
if values != [1, 2] or back.alias is not back.head.next:
raise SystemExit("the scene did not load back")

text = back.to_json() # the JSON text form
if Scene.from_json(text) != back:
raise SystemExit("the JSON did not read back")
try:
Scene.load(data[:-1]) # a truncated file
except Refused as refusal:
print("refused:", refusal.reason)
else:
raise SystemExit("a truncated file was loaded")
```
<!-- /python-usage -->

All of them are generated from the same IR and held to the same golden bytes
in CI. The wire is bit-packed, so the property being checked is
bit-identity — if they ever disagree by one bit, the build fails.

---
Expand Down
Loading
Loading