Fenced Block Types, Generic Extension/Webcomponents, and fallback handling


#1

So what fenced containers could we possibly adopt? Say these?

``` python filename ```
...code...
```````````````````````

::: warning custom containers :::
...content...
:::::::::::::::::::::::::::::::::

^^^ spoiler hidden container ^^^
...snape dies...
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

And if we take the logic these defines the fallback states of generic extensions/web-components, and that the trigger character is ! . (And adopt the consistent attribute syntax) Then:

``` !dot UML Diagram ``` #id .class {key=value}
...UML dot syntax...
```````````````````````

:::::: !gallery cat pics :::: #id .class {key=value}
...commonmark content...
:::::::::::::::::::::::::::::

^^^ !pgp My Pub Key ^^^ #id .class {key=value}
...pgp key here...
^^^^^^^^^^^^^^^^^^^^^^^

Purpose

Think of these, as best practices for preprocessors to avoid colliding with other preprocessors due to syntax collision.

Standardizing the extension metadata in infotext (characters right after ```), will also serve to make it easier for viewers to know what settings is being passed to the plugin/extension/preprocessor, without having to read the extensions instructions.

Also if there is no handler, having a known way for commonmark to handle unknown content will aid in portability, and futureproofing. Especially if we treat the info text after the trigger word as alt definitions, which would be good for disabled users as well.

To aid in this, we could include libraries that people can include in their C, python, etc… to safely handle these fenced syntax, and extract these extra metadata in the infotext.

In practice:

code mode

```mermaid
graph TD;
  A-->B;
  A-->D;
  B-->C;
  D-->C;
```

acts as normal like:

graph TD;
  A-->B;
  A-->D;
  B-->C;
  D-->C;

render mode

```!mermaid
graph TD;
  A-->B;
  A-->D;
  B-->C;
  D-->C;
```

may render like:

else fallback to

graph TD;
  A-->B;
  A-->D;
  B-->C;
  D-->C;

web components AST intercept

With the above examples if a web component handler exist and we want to render it as one then it make look like:

<dot id=id class=class key=value alt="UML Diagram" > ...UML dot syntax... </dot>
<gallery id=id class=class key=value alt="cat pics"> ...commonmark content... </gallery>
<pgp id=id class=class key=value alt="My Pub Key"> ...pgp key here... </pgp>

How does generic directives factor into this? Well it can intercept the AST before it falls back into web components.

How does fenced containers above factor in? Well if no web-component handler is available, then you can default to either displaying as code (for code blocks), content (for custom containers) or as hidden (using the spoiler box).

Reference:


The minimal option

``` !triggerName info text
...content here...
```````````````````````

If we are seeking the most minimal option. We could just set the best practice as the ! in front of a trigger word like !triggerName, and that the setting is the characters just after the trigger word (which is most likely going to be used as settings). You lose the ability to have alt text to be displayed as a fallback behaviour, since it would be filled with settings gibberish, rather than Diagram of a Cat discription. Also you lose the ability to style the fenced code/content/spoiler fallback behaviour.


Mermaid - Generation of diagrams and flowcharts from text in a similar manner as markdown
#2

I can’t quite understand the terminology you use here: what do you mean by “web-components”? What is a “generic extension”? An extension of the CommonMark syntax (specification)?

It is a bit unfortunate to have two or even more discussions running in parallel about the same topic, which I would state as:

How could one implement “special processing” of specially-labeled code blocks (and code spans), in a generic, flexible, and robust way, in accord with the CommonMark specification as well as existing Markdown practice?

I have put forward arguments over there why I think that it would be neither useful nor required to distinguish (by prefixing the label with “!” for example, or in any other way) between

  • “what settings is being passed to the plugin/extension/preprocessor” (I think you mean here: raw text contents of blocks, not settings are passed to the “external” processor, or did you mean that the label specifies which processor will have the these raw block contents passed to?), in other words: processing the raw block content using an “external” tool, and which; and

  • “regular” CommonMark semantics for labeled code blocks, which as a matter of fact does not specify anything how these labels ought to be interpreted, and would them process just the same, disregarding the label alltogether. (I guess that you mean something like this by the words “if we take the logic these defines the fallback states of generic extensions/web-components”.)

Doing otherwise—forcing the CommonMark author to make a distinction between those “labels” (which really mean: labels for syntaxes, raw text classes, or what you want to call it) that can and should be processed and those that can not by a specific implementation (say a web site)—well, doing otherwise would fly in the face of portability of CommonMark documents in my view.

I really would want to gain (or retain?) the ability for an author to write, for example:

~~~~C
int foo = bar;
~~~~

and having a good chance that a CommonMark (or Markdown for that matter) implementation either

  • does know what to do with “C” labeled code blocks, and does interpret their content as C programmin language source code, in the implementation’s own way—probably producing syntax-highlighted HTML rendering (whether by using an external processor or by any other means is irrelevant for an author, or the CommonMark syntax); or else

  • doesn’t know what to do with this “C” label, and just falls back to the default behaviour, the one cmark shows right now: treat it and render it like any other, ordinary code block, and simply present the content as mono-spaced text with hard line-breaks and no other interpretation or processing.

You would have to come up with a really good argument to convince me otherwise, that is: that distinguishing in the author’s marked-up typescript between “!C” labels (meaning “do process”), and “C” labels (meaning “do not process”) would actually achieve something valuable, and for whom.

As far as I can see, it wouldn’t—not for the author, and even less so for the implementor, and even lesser so for the maintainers and adherents of a stable, simple, and portable CommonMark syntax specification.


#3
~~~~C
int foo = bar;
~~~~

the above would be treated as normal code, with syntax highlighting (via preprocessor if that’s what you want as well.). But basically it says, “I am just code, of this language, to be displayed verbatim, with syntax highlighting if possible”.

But the example below would also, be treated as normal code block if no preprocessor captures it. However it may also turn into something totally different, e.g. rending a UML diagram. It basically says “I am a source code/setting to a render that would display non code content if functioning. But if no-one knows how to handle me, then just display my source code to the user verbatim, with syntax highlighting if you want. Oh and I am a ‘C’ source code, so highlight me like that.”

~~~~!C
int foo = bar;
~~~~

#4

Two questions:

  1. If the first example means “do syntax highlighting”—who does the highlighting? Certainly not cmark or any Markdown parser I know. I therefore guess that another component does it (be it a separate process in a pipeline, or some kind of plug-in for the parser).

  2. How is the first example treated by a run-off-the-mill parser (like cmark, or discount, or sundown etc)? No syntax highlighting for sure, in fact the label is completely ignored (if the Markdown syntax variant allows it in the first place).

So I can’t really see what difference between the two examples you consider so important that it needs to be represented in the author’s written text.

Wouldn’t an author much more likely just want to signal that “this code block contains C source code”, and let the processor do what it can with it? Much like we all are used to file name extensions: if it’s C source code, “mark” the file with a .c ending, no matter to which processor you’re going to feed it (a compiler, a pretty-printer, lint, etc).

I still don’t quite get what the author of your examples wants to express when in one case he is using “C”, and in the other using “!C” for a label. It seems to me the difference is in fact which processor gets to process which block—one for the syntax-highlighter, one for generating UML diagrams from C source code fragments [who does that? ;-)].

I guess the difference in perspectives between yours and mine is that:

  • I would consider the label on a code block rather like a MIME type (or a file name extension), specifying the kind of content (rsp content syntax); while

  • you would see the label rather as a specific processing instruction (like: do highlighting for C source code; or: generate UML diagrams from C source code).

Is that fair to say?

Note also that your (supposed) intent—targeting a processor by specifying a label—would be encompassed quite easily by simply (inside cmark) mapping labels to processors, and ignoring non-mapped labels. Which is what I proposed.

For your examples, a mapping to post-processors (assuming those are Windows programs, and further assuming that cmark would invoke them, which is not what I proposed!) would look like this:

"C" --> "path\to\do_syntax_highlighting.exe"
"!C" --> "path\to\somewhere\else\create_UML_diagrams.exe"

Is there something in your examples that could not be mapped into this set-up?


#5

For question 1:

That is handled the way that cmark is already handling it. Don’t care. Can be internal or external. But the intention is that the code is verbatim and if possible then to syntax highlight with the correct language (as specified in infotext). Basically I’m just saying that without !, it just acts how it already acts by default.

For question 2:

According to dingus it at least leaves a class="language-c" marker, allowing for an postprocessor (xml postprocessor, or a javascript library modifying the DOM) to locate and add syntax highlighting. In most cases, you don’t see highlighting, because most users dont bother adding the syntax highlighting extension, but the capability is there and thus portability of the intention of syntax highlighting.

```c
print()
```
<pre><code data-sourcepos="1:1-3:7" class="language-c">print()</code></pre>

On the notion of perspective

And yea we are speaking on different pages. My intention is that without the !, the intention of the author is that it be more like a MIME type block that wants to be displayed as verbatim code (e.g. content syntax).

While if it has a !, the intention of the author is that is it a specific processing instruction. But not to do highlighting, since that is a verbatim code instruction already supported by default cmark. But rather, to do stuff like UML diagram.

Again. To illustrate the point.

code mode

```mermaid
graph TD;
  A-->B;
  A-->D;
  B-->C;
  D-->C;
```

acts as normal like:

graph TD;
  A-->B;
  A-->D;
  B-->C;
  D-->C;

render mode

```!mermaid
graph TD;
  A-->B;
  A-->D;
  B-->C;
  D-->C;
```

may render like:

else fallback to

graph TD;
  A-->B;
  A-->D;
  B-->C;
  D-->C;

with maybe an optional notice, or some marking indicating that this was meant to be rendered. (e.g. shrinking the box if its too long, with a “missing render” icon or something like that)


I think the confusion between us, is that you are thinking of “specific processing steps”, while I am thinking “writer’s intent”. The writer does not care what happens backstage or front stage, only that it happens.


#6

I see better now where you stand. Could I rephrase your idea like this?:

  • Mark with simple “C” (or whatever label) if the code block content should just be rendered as text, beautifyed by syntax highlighting or by whatever means available or not; but

  • mark with the added exclamation sign “!C” if the author wants the code block content to be “processed with bigger tools”, like in your Mermaid examples, render the output diagram resulting from interpreting the Mermaid code.

In short: your difference between “!” and not is the difference between interpreting text and solely rendering it, right?

Maybe it does make sense to denote this difference with different labels, after all!

But I still don’t see why you would want to set the different labels in stone: exactly two for each “MIME type”, one for rendering, one for interpreting the syntax. Recurring to my .c file example: we don’t mark the difference between compiling it or linting it or editing it (with syntax highlighting) in the file name extension, but by separate means—passing the file with it’s extension to a compiler or lint or an editor, which then figures out if and how they can process the “type” of content indicated by the file name extension in the intended way.

If one transfers the analogy to our code block discourse, I’d rather suggest labels like

````C
int foo(void) { puts("baz"); }
````

for the “just render that thing, don’t try to understand what it means” case, and something like

````C:lint
int foo(void) { puts("baz"); }
````

for “use this fancy feature of the Markdown processor that can actually lint C source now!”

While the label syntax I invented here is completely arbitrary, the important point to carry over from my file name extension analogy is: use two independent elements for specifying

  1. the “type” of code block content (similar to a MIME type), useful for rendering alone, without interpreting it;

  2. the intended interpretation rsp processing of the code block content, if any.

Your distinction between exactly two labels for each “MIME type” would be mapped into this model by having a “default interpreter” for each such type, denoted—if you want—by an exclamation sign, or something like that (an asterisk “*” would do nicely too, but that’s a matter of taste). You then get back your two alternatives:

````C
int foo();
````

and

````C:!
int foo();
````

or even just “C!”, or by reverting the two elements: “!C” again ;-). But you’d still keep the door open for distinguishing different kinds of interpretations in the label, like in my lint example, which could then use the label “lint!C” and be compatible with your exclamation-sign notation so far.

The preliminary syntax for labels so far would then be: [[interpretation] "!"] type, where a ! with a missing interpretation would mean the default interpretation for the given type: which is an easy choice for Mermaid (“make a diagram out of the text!”), but not so easy for C source (“compile-link-execute!” is off the table…).

Could we come together on this perspecitve?


#7

Eh, I’m whatever will allow for the “difference between interpreting text and solely rendering it”, as long as the specification have a clear recommendation. I just don’t want adhoc methods of triggering render mode, as is happening currently (which I think will cause issues in the near future).


Oh btw, if you are trying to allow for compatibility with existing code highlighting, you really should add a space between the format name and the trigger symbol. (e.g. C ! not C!. Else it would capture as class=language-C!

```C ! 
int foo(void) { puts("baz"); }
```

Looks bit weird to me however. But I’ll leave that distinction up to the community.


#8

On the matter of implementation: An “enhanced” CommonMark parser could and probably would still use a table to map labels to post-processors (or transformation functions, or plug-ins, or attributes in “transport elements”), using the complete label (like “lint!C”) as the table key.

With one exception, made possible by the proposed two-element structure of labels: If it can’t map a “processing” label (containing a “!” in the preliminary syntax), the parser could still use the part after the “!” to at least try a decent text rendering for the given code block.

Regarding the class attribute put into the <code> element: I would prefer to have the whole label placed into that attribute, because a post-processor can then see if the required interpretation/processing is of the kind this post-processor is capable to perform, and could check as well that the “type” is the expected one. That would mean that the labels are to be written in one piece, without intervening spaces.

However, due to the mapping table, no <code> element would be sent out to post-processing for which no post-processor is applicable.

[I would vote for a different cmark output mode where this stuff is sent to post-processors not in <PRE><CODE> nested elements, but simply in “transport elements” for the dedicated consumption by post-processors, but that’s not a difference in principle, but in practice.]

I would hope that the difference between “rendering” and “processing” a code block’s content can be expressed clearly in a specification. Behind the scenes however, there really would be no such difference, except that there is the “fallback rule” mentioned above.

So the problem is rather how to communicate to authors what the two pieces in a label mean; and one could still just be moot about the fact that there can be two parts, or that there may be something written in front of the exclamation mark :wink: and just try your explanation of the difference between “Mermaid” and “!Mermaid” …


I wasn’t even aware of the mechanism that puts a code block label into the class attribute in the output. Playing with it on dingus, it seems that a label without space in it will always be placed into a class attribute, but always with a language- prefix—which is a bit silly IMO, but not a problem either.

If the label has space in it, like you proposed, the part after the space will not be represented in the output. And that’s what I want to avoid, because the output is intended for a post-processor in our case, and having the complete label information in the class (or whatever) attribute reach the post-processor is not only useful, but I would say essential.

[I really should do a concept demo implementation of this stuff …]


#9

Nothing more to add. You got the core idea. Just need to iron out on argument passing and “alt text” for blind/hearing impaired.

A concept demo implementation would certainly help in stress testing it.


#10

Rather than using code blocks for rendering (which are intended for source code), I suggest we use the concept of “hybrid” documents containing special sections that the parser skips over but may be recognised by other applications. That way we’re not introducing yet another syntax and concept to learn. Refer to @jgm’s comment here:

http://talk.commonmark.org/t/metadata-in-documents/721/53


#11

@mofosyne: I have put together a plan for a project which would also encompass the “syntax extensions” discussed in this thread.

See my “Announcement/Request for Comments” post over there in the Extensions category.