Skip to content

ABI Mapper Improvements#67

Merged
ikskuh merged 37 commits into
masterfrom
work/abi-mapper-improvements
Jun 16, 2026
Merged

ABI Mapper Improvements#67
ikskuh merged 37 commits into
masterfrom
work/abi-mapper-improvements

Conversation

@ikskuh

@ikskuh ikskuh commented Mar 2, 2026

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • New Features

    • Added structured documentation comment support with cross-references and inline markup in ABI definitions
    • Implemented stable UID database for consistent ABI identifier mapping
    • Added support for underscore separators in numeric literals
    • Enabled named parameters in function pointer types
  • Bug Fixes

    • Fixed optional type handling in syscall parameters
    • Improved bitstruct field packing validation
    • Enhanced syscall output parameter constraints
  • Documentation

    • Created comprehensive ABI documentation format specification
    • Added extensive test coverage for parser and semantic analysis
  • Improvements

    • Enhanced website rendering with improved cross-references and anchoring
    • Refined HTML styling for documentation sections and callouts
    • Optimized build caching strategy

Felix "xq" Queißner added 2 commits March 2, 2026 15:50
@coderabbitai

coderabbitai Bot commented Mar 2, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Replaces the flat DocString model with a structured DocComment (Section/Block/Inline) throughout the ABI mapper, adds a UidDatabase for stable FQN-to-UID mapping with JSON persistence, extends syntax parsing to support digit separators and named fnptr parameters, overhauls semantic analysis error reporting and constraint validation, rewires the website HTML generator to render rich doc sections, migrates ashet.abi doc comments to @-reference markup, adds an eleven-module test suite, and switches CI cache keys to a Python script-derived SHA-256 hash.

Changes

ABI Mapper Rework

Layer / File(s) Summary
DocComment model and JSON serialization
src/tools/abi-mapper/src/model.zig
Removes DocString; introduces DocComment with nested Section/Block/Inline sum types plus jsonStringify/jsonParse. Adds FunctionPointerParam. Updates is_c_abi_compatible to resolve alias/typedef. Migrates all docs fields across 14 public structs.
UID database for stable FQN mapping
src/tools/abi-mapper/src/uid_db.zig, src/abi/db/abi-id-db.json
Adds UidDatabase with init/deinit/get_or_assign/load/save and atomic JSON persistence. Populates abi-id-db.json with 2448 lines of FQN-to-UID entries.
Doc-comment parser, spec, and findings
src/tools/abi-mapper/src/doc_comment.zig, src/tools/abi-mapper/rework/abi-doc-format.md, src/tools/abi-mapper/rework/findings.md
Implements block/inline parser with ParseError set and describe_parse_error. Adds normative IDL doc-comment specification (EBNF, JSON schema, examples, migration guide) and nine-finding stress-test findings document.
Syntax parser: digit separators and fnptr named params
src/tools/abi-mapper/src/syntax.zig
Adds _-tolerant digit helpers for hex/binary/decimal, strips separators before parseInt, extends fnptr parsing for optional name:type syntax, introduces FnPtrParam struct.
Semantic analysis overhaul
src/tools/abi-mapper/src/sema.zig
Changes analyze to accept optional uid_database and errors_out; introduces AnalysisError and fail_if_errors. Replaces DocString logic with map_doc_comment pipeline and scoped reference resolution. Overhauls bitstruct packing, native-call ABI shaping, constraint validation, fnptr equality, constant ordering, and syscall out-param count checks.
CLI, build wiring, and ConversionMode
src/tools/abi-mapper/src/abi-parser.zig, src/tools/abi-mapper/build.zig, build.zig, build.zig.zon, src/abi/build.zig
Exports doc_comment; adds --id-db CLI option with load/save. Makes Converter.get_json_dump id_database optional. Wires testsuite into abi-mapper build. Registers abi_mapper in root manifests. Extracts ConversionMode enum and adds escaping ABI test loop.
Zig code generator: fmt_sym_name and DocComment adaptation
src/abi/utility/render_zig_code.zig
Adds ZigRenderer.fmt_sym_name replacing ad-hoc fmt_fqn+separator patterns at all kernel/userland call sites. Simplifies fmt_fqn to dot-only. Adapts render_docs for DocComment (no-op). Updates fnptr param formatting to include optional names.
ashet.abi doc-comment reformatting and escaping fixture
src/abi/src/ashet.abi, src/abi/tests/escaping.abi
Converts ~100 doc comment lines from backtick/@ref style to @...`` inline reference markup. Adds escaping.abi fixture with `suspend`/`resume` syscalls and an `embedded` namespace to validate Zig keyword escaping.
Comprehensive test suite
src/tools/abi-mapper/tests/*
Adds testsuite.zig and eleven test modules covering doc-comment parsing, JSON roundtrip, FQN reference resolution, digit separators, constant ordering, nonstandard backing types, optional type handling, unknown named types, bitstruct array fields, syscall output counts, and fnptr named parameters, plus three ABI fixture files.
Website HTML generator and CSS
src/website/src/syscalls-gen.zig, src/website/www/theme.css, src/website/build.zig
Replaces fmt_docs/format_docs with DocFmt struct rendering DocComment sections/blocks/inlines as HTML. Adds find_declaration(), LqnFmt, anchored div IDs for fields. Updates CSS for basic-list, dl>div:target highlight, and admonition ::before labels (NOTE/WARNING/LORE/EXAMPLE/DEPRECATED/DECISION/LEARN).
Miscellaneous build updates
src/os/build.zig
Configures dimmer dependency with .release = true.

CI Package Cache Key Automation

Layer / File(s) Summary
Python cache key script
scripts/zig_package_cache_key.py
Adds script extracting .url/.hash entries from all tracked build.zig.zon files, aggregating unique pairs, and hashing via SHA-256 to output key=<hexdigest>.
CI workflow updates and git attributes
.github/workflows/build.yml, .github/workflows/pages.yml, .github/workflows/smoketest.yml, .gitattributes
Switches all three workflows from hashFiles('**/build.zig.zon') to the script-derived cache key. Adds libgtk-3-dev to smoketest QEMU install. Adds *.abi LF line-ending rule.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Possibly related PRs

Suggested labels

codex

🐇 A rabbit in the warren declared,
"Our docs were just strings — how impaired!
Now Sections and Blocks,
Inline refs in the mix,
With UIDs stable, the ABI is prepared!" 🎉

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch work/abi-mapper-improvements

Felix "xq" Queißner added 27 commits March 2, 2026 21:06
…de ashet.abi, adds broken links for code references.
…pes are now also rendered as locally qualified names to simplify display.
…to get from the unpatched abi file to a working solution
@ikskuh ikskuh marked this pull request as ready for review June 15, 2026 20:21

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 14

🧹 Nitpick comments (4)
src/tools/abi-mapper/tests/testsuite.zig (1)

1-13: ⚡ Quick win

Add a UID database invariant test module to this test aggregator.

Since src/abi/db/abi-id-db.json is now a committed stability contract, include one test module here that validates at least uniqueness of fqn and uid (and optionally deterministic load/save roundtrip). This catches accidental mapping drift early in CI.

src/tools/abi-mapper/tests/doc_ref_emission.zig (1)

59-62: ⚡ Quick win

Fail fast when semantic diagnostics are emitted on successful analysis.

errors is collected but never asserted on success, so tests may pass while analyzer diagnostics are still produced.

Suggested change
-    return abi_parser.sema.analyze(allocator, ast_document, null, &errors);
+    const analyzed = try abi_parser.sema.analyze(allocator, ast_document, null, &errors);
+    try std.testing.expectEqual(`@as`(usize, 0), errors.items.len);
+    return analyzed;
src/tools/abi-mapper/tests/optional_type_handling.zig (1)

30-33: ⚡ Quick win

Assert optional type preservation explicitly in the positive-path test.

This test currently validates only the native input name. Add a type-level assertion so it fails if ?Callback is flattened or rewritten incorrectly.

src/tools/abi-mapper/tests/syscall_output_count.zig (1)

4-14: ⚡ Quick win

Extract parse_and_analyze into a shared test utility.

This helper is duplicated across the new test modules, which increases drift risk when parser/analyzer call signatures change.

♻️ Proposed refactor
+// src/tools/abi-mapper/tests/test_utils.zig
+const std = `@import`("std");
+const abi_parser = `@import`("abi-parser");
+
+pub fn parse_and_analyze(allocator: std.mem.Allocator, source: []const u8) !abi_parser.model.Document {
+    var tokenizer: abi_parser.syntax.Tokenizer = .init(source, "test");
+    var parser: abi_parser.syntax.Parser = .{
+        .allocator = allocator,
+        .core = .init(&tokenizer),
+    };
+    const ast = try parser.accept_document();
+    var errors: std.ArrayList(abi_parser.sema.AnalysisError) = .empty;
+    defer errors.deinit(allocator);
+    return abi_parser.sema.analyze(allocator, ast, null, &errors);
+}
-const abi_parser = `@import`("abi-parser");
+const abi_parser = `@import`("abi-parser");
+const test_utils = `@import`("test_utils.zig");

-fn parse_and_analyze(allocator: std.mem.Allocator, source: []const u8) !abi_parser.model.Document {
-    ...
-}
-
-const doc = try parse_and_analyze(allocator, source);
+const doc = try test_utils.parse_and_analyze(allocator, source);

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 55966225-af50-47c2-8051-621b74724b09

📥 Commits

Reviewing files that changed from the base of the PR and between 946fa68 and a45b431.

📒 Files selected for processing (35)
  • build.zig
  • build.zig.zon
  • src/abi/build.zig
  • src/abi/db/abi-id-db.json
  • src/abi/src/ashet.abi
  • src/abi/tests/escaping.abi
  • src/abi/utility/render_zig_code.zig
  • src/tools/abi-mapper/build.zig
  • src/tools/abi-mapper/rework/abi-doc-format.md
  • src/tools/abi-mapper/rework/findings.md
  • src/tools/abi-mapper/src/abi-parser.zig
  • src/tools/abi-mapper/src/doc_comment.zig
  • src/tools/abi-mapper/src/model.zig
  • src/tools/abi-mapper/src/sema.zig
  • src/tools/abi-mapper/src/syntax.zig
  • src/tools/abi-mapper/src/uid_db.zig
  • src/tools/abi-mapper/tests/bitstruct_array_field.zig
  • src/tools/abi-mapper/tests/constant_ordering.zig
  • src/tools/abi-mapper/tests/digit_separator.zig
  • src/tools/abi-mapper/tests/doc_parser.zig
  • src/tools/abi-mapper/tests/doc_ref_emission.abi
  • src/tools/abi-mapper/tests/doc_ref_emission.zig
  • src/tools/abi-mapper/tests/doc_ref_resolution.abi
  • src/tools/abi-mapper/tests/doc_ref_resolution.zig
  • src/tools/abi-mapper/tests/fnptr_named_params.zig
  • src/tools/abi-mapper/tests/nonstandard_backing_type.zig
  • src/tools/abi-mapper/tests/optional_type_handling.zig
  • src/tools/abi-mapper/tests/stress/ashet-1.0.abi
  • src/tools/abi-mapper/tests/stress/ashet-1.0.abi.patch
  • src/tools/abi-mapper/tests/syscall_output_count.zig
  • src/tools/abi-mapper/tests/testsuite.zig
  • src/tools/abi-mapper/tests/unknown_named_type.zig
  • src/website/build.zig
  • src/website/src/syscalls-gen.zig
  • src/website/www/theme.css

Comment thread src/abi/utility/render_zig_code.zig
Comment thread src/tools/abi-mapper/rework/abi-doc-format.md
Comment thread src/tools/abi-mapper/src/abi-parser.zig
Comment thread src/tools/abi-mapper/src/doc_comment.zig
Comment thread src/tools/abi-mapper/src/doc_comment.zig
Comment thread src/tools/abi-mapper/src/syntax.zig
Comment thread src/tools/abi-mapper/src/syntax.zig
Comment thread src/tools/abi-mapper/src/uid_db.zig
Comment thread src/tools/abi-mapper/tests/syscall_output_count.zig
Comment thread src/website/src/syscalls-gen.zig

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/tools/abi-mapper/tests/doc_parser.zig (1)

283-304: ⚠️ Potential issue | 🟡 Minor

Add LEARN to the admonition kinds test cases.

The Section.Kind enum in model.zig includes learn as a valid variant, and parse_admonition in doc_comment.zig already handles it. The test loop should cover all admonition kinds for completeness. Add .{ .tag = "LEARN", .kind = .learn } to the cases array.


ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 2840e6cf-ebf5-4e18-85d8-58c5698e9e54

📥 Commits

Reviewing files that changed from the base of the PR and between a45b431 and a56ddd3.

📒 Files selected for processing (9)
  • src/os/build.zig
  • src/tools/abi-mapper/src/doc_comment.zig
  • src/tools/abi-mapper/src/sema.zig
  • src/tools/abi-mapper/src/syntax.zig
  • src/tools/abi-mapper/tests/digit_separator.zig
  • src/tools/abi-mapper/tests/doc_parser.zig
  • src/tools/abi-mapper/tests/doc_ref_resolution.zig
  • src/tools/abi-mapper/tests/stress/ashet-1.0.abi
  • src/website/src/syscalls-gen.zig
🚧 Files skipped from review as they are similar to previous changes (4)
  • src/tools/abi-mapper/src/syntax.zig
  • src/tools/abi-mapper/src/doc_comment.zig
  • src/website/src/syscalls-gen.zig
  • src/tools/abi-mapper/src/sema.zig

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2


ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 927e7989-e470-4059-a999-417dff21d138

📥 Commits

Reviewing files that changed from the base of the PR and between a56ddd3 and eb4cc10.

📒 Files selected for processing (4)
  • .github/workflows/build.yml
  • .github/workflows/pages.yml
  • .github/workflows/smoketest.yml
  • scripts/zig_package_cache_key.py

Comment thread .github/workflows/build.yml
Comment thread scripts/zig_package_cache_key.py
@ikskuh ikskuh force-pushed the work/abi-mapper-improvements branch from eb4cc10 to 11e553b Compare June 16, 2026 19:01

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
scripts/zig_package_cache_key.py (1)

41-43: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Inline trailing comments on .url/.hash entries still hard-fail parsing.

At Line 42, assert value.endswith('",') rejects valid Zig like .hash = "...", // comment, which can break CI cache-key generation.

Proposed fix
 import hashlib
 import pathlib
+import re
 import subprocess
 import sys
 
+ENTRY_RE = re.compile(r'^(\.url|\.hash)\s*=\s*"(.*)"\s*,')
+
@@
-        for key in (".url", ".hash"):
-            prefix = f"{key} ="
-            if not stripped.startswith(prefix):
-                continue
-
-            _, value = stripped.split("=", 1)
-            value = value.strip()
-
-            assert value.startswith('"'), f"{path}: malformed {key} entry: {line!r}"
-            assert value.endswith('",'), f"{path}: malformed {key} entry: {line!r}"
-
-            entries.add(f"{key}={value}")
+        m = ENTRY_RE.match(stripped)
+        if not m:
+            continue
+        key, raw_value = m.groups()
+        entries.add(f'{key}="{raw_value}",')

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 86d2545e-76e9-4f37-a4fc-5be2a0677cfd

📥 Commits

Reviewing files that changed from the base of the PR and between eb4cc10 and c01b8d8.

📒 Files selected for processing (5)
  • .gitattributes
  • .github/workflows/build.yml
  • .github/workflows/pages.yml
  • .github/workflows/smoketest.yml
  • scripts/zig_package_cache_key.py
✅ Files skipped from review due to trivial changes (1)
  • .gitattributes

@ikskuh ikskuh force-pushed the work/abi-mapper-improvements branch from 3b2a23e to 8fb6c61 Compare June 16, 2026 20:15
@ikskuh ikskuh force-pushed the work/abi-mapper-improvements branch from 8fb6c61 to d765ff7 Compare June 16, 2026 20:27
@ikskuh ikskuh merged commit cd63828 into master Jun 16, 2026
24 of 50 checks passed
@ikskuh ikskuh deleted the work/abi-mapper-improvements branch June 16, 2026 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant