Skip to content

test: add class-width compile-time benchmark for issue 559 - #3118

Open
carloscasellas wants to merge 1 commit into
glideapps:masterfrom
carloscasellas:agent/issue-559
Open

carloscasellas wants to merge 1 commit into
glideapps:masterfrom
carloscasellas:agent/issue-559

Conversation

@carloscasellas

Copy link
Copy Markdown

Merging this PR is not necessary, I wanted to provide the scripts in case anyone looking at issue 559 wanted to look at them.

Description

Adds a reproducible benchmark (script/benchmark-class-width/) measuring whether narrowing the widest generated Rust struct — as issue #559 proposes — actually improves compile time. It doesn't add or change any generated-code behavior; it's a standalone investigation with scripts, a manual (README.md), and a results write-up (RESULTS.md).

Related Issue

Investigates #559. Recommends closing it as obsolete based on the measurements below.

Motivation and Context

#559 claims blns-object.json, keywords.json, and keyword-unions.schema have objects large enough to slow down compilation, "particularly Rust," citing profiling from #516 (cargo-expand + -Ztime-passes on rustc ~1.24, showing 30k lines concentrated in one visit_map function from serde's derive).

That's an eight-year-old measurement. Before doing the work of splitting these fixtures, I wanted to confirm the underlying cost still exists on a current toolchain.

Previous Behaviour / Output

No benchmark existed for this — the issue's justification was 2018-era profiling with no way to re-verify it against today's rustc.

New Behaviour / Output

script/benchmark-class-width/reshape-blns.py and reshape-keyword-unions.py regroup the fixtures' keys into progressively narrower classes while holding the total field count constant, so any compile-time change is attributable to width rather than data volume. bench-rust.sh times cargo build against each candidate with serde prebuilt in a shared CARGO_TARGET_DIR, isolating the compile of the generated module itself.

Result on rustc/cargo 1.97.1 (median of 10 builds per candidate):

widest class blns compile keyword-unions compile
276 / 230 (today) 0.73s 1.98s
138 / 116 0.74s 1.97s
56 / 59 0.78s 2.00s
28 / 30 0.82s 2.02s

keyword-unions is flat within noise. blns gets monotonically slower as classes narrow (+12%), tracking the extra code from more, smaller types. Neither supports the issue's premise on current rustc.

Separately, ~91% of a Rust sample's cost in the real fixture harness turns out to be rebuilding serde from scratch per sample (no setupCommand, no shared CARGO_TARGET_DIR in test/languages.ts/test/fixtures.ts) — noted in RESULTS.md as a lead worth its own follow-up spike, not a verified fix.

Full write-up, including the keyword-unions.schema → Python 255-argument limit finding and the keywords.json generator-drift bug found along the way, is in script/benchmark-class-width/RESULTS.md.

How Has This Been Tested?

This PR is the test — a benchmark, not a code change. To reproduce:

npm run build
export PATH="$HOME/.cargo/bin:$PATH"   # if using rustup
export BENCH=/tmp/bench
mkdir -p $BENCH/inputs
B=script/benchmark-class-width

for n in 1 2 4 8;  do python3 $B/reshape-blns.py $n           > $BENCH/inputs/blns-$n.json
for n in 1 2 5 10; do python3 $B/reshape-keyword-unions.py $n > $BENCH/inputs/ku-$n.schema; done

REPS=10 bash $B/bench-rust.sh --src-lang json   $BENCH/inputs/blns-{1,2,4,8}.json
REPS=10 bash $B/bench-rust.sh --src-lang schema $BENCH/inputs/ku-{1,2,5,10}.schema

Full instructions, including reference output and troubleshooting, are in script/benchmarkm run test:unit and npm run lint are unaffected since this only adds new files underscript/.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant