Skip to content

zsh history parser: formats, unmetafy, multiline, caps #10

Description

@Saber5656

Title

zsh history parser: formats, unmetafy, multiline, caps

Summary

Implement the pure parsing layer for zsh history files: unmetafication,
simple + extended format detection, multiline entry joining, and the
size/line caps — exactly per docs/research/zsh-history-format.md.

Context

This is parsing only (bytes → entries); provider glue (date filtering,
config, warnings) is issue 10. The reference machine's history is
simple-format (no timestamps), and oh-my-zsh machines are extended-format;
both must parse. Metafied bytes (0x83 escapes) corrupt Japanese commands if
skipped — this is the most commonly missed zsh detail.

Scope

  • internal/source/zsh/parser.go + tests + testdata/ fixture files.

Detailed Requirements

  1. type Entry struct { Start time.Time; Duration time.Duration; Command string; HasTimestamp bool }.
  2. Unmetafy([]byte) []byte — exact algorithm from the research doc
    (0x83 marker, next byte XOR 0x20; trailing lone 0x83 dropped). Operates
    on raw bytes BEFORE any string conversion.
  3. ParseReader(r io.Reader, caps Caps) (entries []Entry, stats Stats):
    • Caps{MaxLineBytes int (default 64*1024), MaxFileBytes int64 (default 128*1024*1024)}; Stats{Lines, Oversized, Malformed int; Truncated bool; ExtendedCount, SimpleCount int}.
    • Streaming scan (bufio.Reader, manual line assembly — bufio.Scanner
      token limits must not abort the file: an oversized line is consumed,
      counted in Oversized, and skipped).
    • Per physical line: unmetafy → UTF-8 repair (sanitize.Clean is NOT
      applied here; only strings.ToValidUTF8 — commands keep their tabs
      etc.; display hygiene happens downstream) → multiline joining: while
      the line ends with an odd number of \, strip the final \, append
      \n, read next physical line (a line starting with ": " while in
      extended mode terminates a malformed continuation; count Malformed).
    • Entry header: ^: (\d{10,}):(\d+);(.*)$ → extended entry
      (HasTimestamp=true, Start from epoch, Duration seconds); otherwise
      the whole line is a simple entry (HasTimestamp=false).
    • Stop reading at MaxFileBytes (set Truncated).
    • Empty lines are skipped (not Malformed).
  4. No file I/O in this file beyond the io.Reader (testability); no
    goroutines.

Acceptance Criteria

  • Fixture extended.hist: ≥ 5 extended entries incl. one with
    duration > 0; timestamps parse to the exact epochs.
  • Fixture simple.hist: plain commands; all HasTimestamp=false.
  • Fixture metafied.hist: contains a Japanese command stored with real
    0x83 metafication (fixture built by a small Go generator in
    testdata/gen/main.go, committed with its output); parsed Command
    equals the original Japanese string.
  • Fixture multiline.hist: an extended entry whose command spans 3
    physical lines via trailing backslashes → one Entry with two \n.
  • Fixture torn.hist: file ending mid-entry (no trailing newline,
    dangling continuation) → parsed without error; tail counted Malformed
    or yielded as-is (documented choice: count Malformed, drop).
  • Oversized line fixture (>64 KiB) → skipped, Oversized==1, following
    entries still parse.
  • Mixed file (simple + extended lines) → both counted in Stats.
  • FuzzParseReader fuzz target (seeded with all fixtures) runs 30s
    locally with no panics and no invalid-UTF-8 Command output.
  • ≥ 90% coverage.

Validation

go test -race -cover ./internal/source/zsh/ and 30s fuzz run output in the
PR. Reviewer check: Unmetafy table includes the byte pairs
0x83 0xC3 → 0xE3 (start of Japanese UTF-8) and lone-trailing-0x83.

Dependencies

01, 06 (only for shared constants if any; parser itself uses
strings.ToValidUTF8 — keep sanitize out of the parse path).

Non-goals

Date filtering, HISTFILE resolution, warnings, Events (all issue 10);
bash/fish formats (v2).

Design References

  • docs/research/zsh-history-format.md (normative format spec)
  • docs/DESIGN.md §7.1

Source of truth: docs/issues/09-zsh-history-parser.md (PR #1, branch docs/v1-design). If this issue and the repo docs disagree, the docs win. Execution order and dependencies: docs/ISSUE_PLAN.md (this is issue 09 of 33).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions