fix(i18n): rescue unparseable translated MDX before attribute repair - #153
Open
hxy91819 wants to merge 1 commit into
Open
fix(i18n): rescue unparseable translated MDX before attribute repair#153hxy91819 wants to merge 1 commit into
hxy91819 wants to merge 1 commit into
Conversation
Full-translation shards keep failing at packaging when translated pages contain JSX-looking damage (fabricated <id> elements, stray closers, bad attribute characters) that the strict MDX parsers in the repair chain reject while check-docs-mdx passes .md pages as plain Markdown. Add a parser-guided syntax repair stage that runs before the existing protected-attribute repair: - acceptance equals the downstream chain's tolerant parser, so valid Markdown constructs (HTML comments, prose less-than) pass untouched - fabricated elements absent from the source are removed with prose kept; missing or stray closing tags are resynced; void elements are self-closed; unquoted attribute values are quoted; unterminated comments are closed - deterministic bounded loop (64 patches) keyed to the parser's own diagnostics; unresolvable damage still fails the shard fail-closed Also validate --locale as a single safe path segment in both repair writers (the Python-side manifest validation runs only after the Node process has already written), and preserve UTF-16 length when masking comments so astral characters cannot shift diagnostic offsets. Artifacts record mdx_syntax_repair_outcome; failed shards keep the existing empty-artifact contract for the weekly reconciliation.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Recent
Translate Fullruns kept failing at shard packaging even after the translation itself succeeded (e.g. run 33169524293:Docs MDX check passed (759 files)thenmdx protected attribute repair failed). Root cause: the packaging-time repair chain parses every page with strictformat: "mdx", whilecheck-docs-mdxcompiles pages by file extension (.mdas plain Markdown). Translated pages with JSX-looking damage — fabricated<id>elements, stray closers, malformed attribute characters — therefore pass the workflow gate and only explode inside the repair chain, discarding otherwise complete (and expensive) translation shards.Fix
Add a parser-guided syntax repair stage (
.github/scripts/i18n/repair_mdx_syntax.mjs) that runs before the existing protected-attribute repair inpackage_artifact.py:parseMdx), and diagnosis reuses the same tolerant masking with an offset map (parseMdxForOffsets, now exported), so valid Markdown constructs — HTML comments, prose less-than — are never diagnosed or rewritten./>delimiter); unterminated single-line comments are closed. Multi-line unterminated comments fail closed instead of guessing what stays hidden.--localeas a single safe path segment before any write (the Python-side manifest validation previously ran only after Node had already written).mdx_syntax_repair_outcome; failed shards keep the existing empty-artifact contract for the weekly reconciliation. Docs updated indocs/.i18n/translation-workflow.md.Testing
python3 -m pytest .github/scripts/i18n/tests/test_i18n_scripts.py— 106 passed; 2 pre-existing failures unrelated to this change (test_translation_workflows_pin_latest_codex_and_tier_effortexpects codex@0.146.0 while the workflow pins 0.146.1; a canary git-rebase test that also fails on pristinemainin sandboxed environments), verified failing on unmodified HEAD./>-aware value quoting).