1brc

The One Billion Row Challenge in Go: aggregate min/mean/max temperature per weather station from a 1,000,000,000-row text file, as fast as the machine allows.

Input is <station>;<temp>\n with one decimal digit. Output is the reference format:

{Abha=-23.0/18.0/59.2, Abidjan=-16.2/26.0/67.3, ...}

Every attempt is a self-contained cmd/vN/main.go, so any two can be diffed to see exactly what one idea cost or bought. The results table below is the point of the repo.

Requirements

• Go 1.26+
• Python 3.14 for the dataset generator and the benchmark harness (brew install python@3.14). make builds a .venv from requirements.txt on first use. Tho 3.13 will work just fine, feel free to downgrade.
• ~14 GB of free disk for the full dataset, and enough free RAM to hold it in the page cache — timings assume the file is already resident (see Benchmarking).
• A 64-bit little-endian POSIX host. The hot loops do unaligned little-endian word loads through unsafe and map the input with syscall.Mmap; there is no big-endian or Windows path. Developed on macOS/arm64.

Datasets

make small     # data/small.txt      10M rows, ~138 MB   — the edit/measure loop
make data      # data/measurements.txt  1B rows, ~13.8 GB — the real thing
make stress    # data/stress.txt    100M rows, 10k stations — hash map pressure

Benchmarking

make bench-small                        # every attempt against small.txt
make bench                              # every attempt against the full 1B rows
make bench ARGS="v3 v4 -n 10"           # only these two, ten timed runs each
make bench ARGS="--cold"                # purge the page cache between runs (needs sudo)
.venv/bin/python tools/bench.py --help

Each attempt is built to bin/<name>, run once to warm the page cache, then timed -n times. Every run is checked against data/expected-<dataset>.txt, so an attempt that is only intermittently wrong is caught. For an externally produced dataset, pass --expect TRUSTED_FILE. Generated references are tied to the exact input file and rejected if the dataset is regenerated without its reference. Results append to .bench/history.jsonl (kept by make clean, removed by make clean-bench), and bench exits non-zero on a failed build, a crash or a mismatch.

Timings are warm-cache by design with the file already in RAM the number reflects compute, not the SSD. --cold measures the I/O path instead.

Reporting

make report                             # full dataset
make report ARGS="-i data/small.txt"
make report ARGS="--cold"               # latest compatible cold-cache runs

Takes the latest complete, verified run of each attempt under the newest compatible dataset and cache mode, then rewrites the chart and the table below. Attempts benchmarked with different -n values are kept and ranked by median time, avoiding the low-outlier bias of comparing minima from unequal sample counts. The footer says which run counts went in.

Profiling

make prof V=v5                            # full dataset -> .profs/v5-measurements.prof
make prof-list V=v5                       # per line of the worker closure
make prof-list V=v5 F=main.parseTemp
make prof-disasm V=v5 F=IndexByteString   # per instruction

Results

#	attempt	best	median	rows/s	cpu	peak rss	vs leader	what changed
1	v12	528 ms	532 ms	1879.8M	0.0x	4.9 MB	1.00x	fork trick: hand the 13.8 GB teardown to a child and exit off the clock. legitimate: used by official 1brc leader
2	v11	573 ms	576 ms	1736.3M	11.5x	13.8 GB	1.08x	inline accumulate's fast path: a hot loop with zero calls
3	v10	660 ms	665 ms	1504.8M	11.5x	13.8 GB	1.25x	kill memequal, unsafe loads
4	v9	1.01 s	1.01 s	987.9M	12.1x	13.8 GB	1.90x	ILP, OoOE
5	v8	1.25 s	1.26 s	792.3M	12.6x	13.8 GB	2.37x	Branchless temperature parse, and a branchless 16-byte ';' scan fused into the hash. fixes 50% mispredict chance
6	v7	1.26 s	1.27 s	788.9M	12.5x	13.8 GB	2.38x	SWAR: branchless temperature parse, and a ';' scan fused into the hash
7	v6	1.59 s	1.60 s	623.3M	12.8x	13.8 GB	3.02x	delete the direct newline search, refactor loop
8	v5	1.76 s	1.79 s	558.3M	12.9x	13.8 GB	3.37x	faster hashing technique, use slot array
9	v4	2.38 s	2.41 s	415.7M	13.1x	13.8 GB	4.52x	float parsing -> int specialized parser
10	v3	3.57 s	3.65 s	273.7M	13.1x	13.8 GB	6.87x	add parallelism
11	v2	37.62 s	37.72 s	26.5M	1.0x	13.8 GB	70.91x	mmap instead of scanner. value map costs 2 hashes + a key alloc per row, so -> pointer map
12	v1	55.41 s	55.59 s	18.0M	1.0x	12.6 MB	104.51x	Baseline: scanner with a map of station statistics

_{measurements.txt · 13.8 GB · 1,000,000,000 rows · warm page cache · 2–5 timed runs per result · ranked by median · updated 2026-08-11T20:13:21+00:00}

_{measured on Apple M4 Max · 10P + 4E cores · 38.7 GB RAM · Darwin 27.0.0 · go1.27rc2}

v12 uses the fork trick from the official repo: the real work happens in a re-executed child, and the parent exits as soon as it has relayed the answer, so the kernel's teardown of 13.8 GB of mappings lands after the clock stops. It makes nothing faster — it moves ~50 ms off the measured window. It also means the cpu and peak rss columns describe the parent, which does nothing, and so read 0.0x and a few MB for that row.

Credits

• gunnarmorling/1brc — the original challenge, its rules, and the reference output format. data/stations_413.csv is the station table from its CreateMeasurements.java (Apache-2.0), with means from Wikipedia.
• data/weather_stations.csv is adapted from simplemaps world cities, licensed CC BY 4.0.

This repo is MIT-licensed; see LICENSE. The two data files keep their own licenses above.