Pulldown-cmark (CommonMark in Rust)

[vitaly]

@raph

It would be extremely helpful if the efficient parsing algorithm were documented somewhere (and this is even more true for the state machine @vitaly describes in #7, if he hopes other implementors to adopt it).

IMHO it’s impossible at current stage (useless waste of time). Problem of markdown is that syntax can not be described in BNF or similar notation. That cause empyrics in code. In theory, spec can be changed in such way, that it will fuckup implementation logic and require serious rewrite (at least in my case). I think, any kinds of algorythm freeze possible only after spec stabilization (read: not soon).

Also, implementations are different, because authors have different priorities. For example:

• reference implementation by @jgm is available to many languages and has stackless parser, allowing infinite nesting.
• “alternative” js implementation by me & Alex is focused on web & end user’s needs, and expects to generate safe html by default + allow syntax plugins. For example, we just limited nesting level instead of fighting with performance, because users don’t care. Parser flexibility is much more important for them.

Since parser logic can’t be formalized now in one best way, development is iterative process, and each iteration needs a lot of resources for experimenting and checking ideas. For example, we did 1 big rewrite of markdown-it, and i expect 1-2 more (for sourcemaps and plugins improvement). Right now i’m a bit tied of this, and decided to postpone next attempt for 0.5-1 years, when spec become more complete.

I beleive, in far future, someone will invent ideal parser algorythm, that will satisfy all requirements. But it’s not available now. It will be great, if someone will decide to reuse existing experience and move forward. There are serious achievements - @jgm demostrated, that nested tags can be parsed with linear complecity, and me & Alex proven that syntax plugins are possible.

@raph

Regarding extensions, my ideas are still bubbling, but I do not think that a plugin architecture, by itself, solves the problem of divergence of extensions.

IMHO we can’t completely stop divergence, but extensions divergence is much better than parsers divergence - less splits in developper’s community. There are many people (for example me) who need more than described in CM spec, but don’t see alternatives to CommonMark in general. It’s a waiste of human resource if we send such people to hell