Repository navigation
Conversation
|
@vidigoat is attempting to deploy a commit to the MarkedJS Team on Vercel. A member of the Team first needs to authorize it. |
| // would interrupt one. Only a non-empty list starting with a bullet or 1 can do that | ||
| const lastTokenIsText = tokens.at(-1)?.type === 'text'; | ||
| if ((!lastTokenIsText || this.options.pedantic || this.tokenizer.rules.other.listInterruptsParagraph.test(src)) | ||
| && (token = this.tokenizer.list(src))) { |
There was a problem hiding this comment.
This logic should move into the list tokenizer
There was a problem hiding this comment.
Makes sense. The list tokenizer only gets src, though, and the check depends on the token before it (a text line inside a list item), which only the lexer has right now. Would passing the tokens be OK, e.g. this.tokenizer.list(src, tokens) with an optional second parameter so existing list(src) overrides keep working, and the listInterruptsParagraph check at the top of list()? Or did you have another way in mind for the tokenizer to know it is continuing a paragraph?
There was a problem hiding this comment.
Moved it into the list tokenizer in 63b081c: the lexer passes its tokens as an optional second argument, list(src, tokens), and list() returns early when the last token is text and the list can't interrupt a paragraph. list(src) overrides keep working (an override returning false falls back with the tokens passed through), and USING_PRO.md lists the new parameter. Happy to change it if you had another way in mind. I also rebased on master and added demo links to the description.
There was a problem hiding this comment.
Could we add tokens to the lexer state instead of passing it in? So in the tokenizer you can use this.lexer.state.tokens to access it?
There was a problem hiding this comment.
Done in 36869b4. blockTokens sets this.lexer.state.tokens to the tokens it is lexing and restores the outer ones before returning, since list items and block quotes lex their contents with a nested call (without the restore, 8 existing specs fail). list(src) keeps its signature, so the USING_PRO.md change is gone. I also added a case to the spec for a list starting with 14 after a list item ending in text.
There was a problem hiding this comment.
Sorry for all the change requests but I'm trying to fit this in without causing possible breaking changes.
I think we should just save the last token in the lexer state instead of all block tokens. There shouldn't be any reason we need other tokens.
There was a problem hiding this comment.
It's completely fine. All good. Updated in a9a51d6: blockTokens now keeps only the last block token in this.lexer.state.lastToken (set before each tokenizer runs, restored when a nested call returns), and the list tokenizer checks this.lexer.state.lastToken?.type === 'text'. The whole token list is no longer in the state.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…in a list item Inside a list item, paragraph lines are lexed as text tokens, and the list tokenizer ran before the text tokenizer without checking whether the list may interrupt the paragraph, so `14.` or an empty `*` on the next line started a nested list. Apply the same rule as the paragraph interrupt: only a non-empty list starting with a bullet or 1 can interrupt a paragraph. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lexer passes its tokens to `list(src, tokens)`, and the tokenizer checks the last one, so `list(src)` overrides keep working. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
705defc to
63b081c
Compare
…izer `blockTokens` sets `this.lexer.state.tokens` to the tokens it is lexing and restores the outer ones when it returns, so `list(src)` keeps its signature and checks the token before it from the state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`blockTokens` records the last token of the run it is lexing in `this.lexer.state.lastToken` before each tokenizer runs, and restores the outer one when a nested call returns, so the list tokenizer checks `state.lastToken` instead of the whole token list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Marked version: 18.1.1 (4b7b5b2)
Markdown flavor: CommonMark|GitHub Flavored Markdown
Description
CommonMark only lets a list interrupt a paragraph when the list starts with a bullet or with 1, and its first item is not empty (spec example 304, and 285 for the empty item). marked follows this for top-level paragraphs, but not for a paragraph inside a list item, so a line like the one in example 304 starts a nested list when it's in a list item.
Expectation
Same for an empty item, e.g.
- foofollowed by*, which should stay as the textfoo\n*. This is what commonmark.js produces.Result
and
- foo/*gives an empty nested<ul><li></li></ul>.Demo
What was attempted
Found while diffing marked (
gfm: false) against commonmark.js on the CommonMark spec examples placed inside list items and block quotes. Spec examples 304, 285 and 367 all differ inside a list item (bullet or ordered) and match at the top level and in a block quote.Inside a list item the paragraph lines are lexed as
texttokens, and the list tokenizer runs before the text tokenizer without the paragraph's interrupt check.blockTokensnow keeps the tokens it is lexing inthis.lexer.state.tokens, and restores the outer ones when it returns (list items and block quotes lex their contents with a nested call).list()returns early when the last of those tokens istextand the list doesn't match the same rule as the paragraph interrupt (listInterruptsParagraph, a non-empty item starting with a bullet or1./1)). Its signature is unchanged. Pedantic mode is unchanged. After a blank line, or with a bullet or1., nested lists start as before.Contributor
test/specs/new/list_item_paragraph_interrupt.md(expected HTML from commonmark.js), which also checks that a list starting with 14 after a list item ending in text still starts at the top level. It fails onmaster, andnpm testpasses. Comparing with commonmark.js (gfm: false) on the 652 spec examples at the top level, in a bullet item, an ordered item, a block quote and a list item in a block quote, plus 19 hand-written nesting cases, 17 outputs change: all now match commonmark.js, and none that matched onmasterchanged. Without restoring the outer tokens, 8 existing specs fail. A block extension that lexes its contents withthis.lexer.blockTokens(), and alisttokenizer override returningfalse, work as before. Parse time is unchanged within noise.AI disclosure: this PR (code, tests and description) was written with Claude Code (Claude Opus 5.5); the verification above is what was run.
🤖 Generated with Claude Code