Djot: A light markup language by @jgm

Continuing the discussion from Beyond Markdown because I think @jgm’s jdot deserves its own forum topic.

Link to a Playground created by @dtinth: https://djot-playground-poc.vercel.app

2 Likes

My thoughts…

Lot’s of things to like:

  1. Simpler and richer inline syntax. The precedence rules.

  2. Nixing indented code blocks. Yea! :tada:

  3. Footnotes. Yea! :tada:

  4. No magic whitespace thresholds. :tada::tada:

    In djot, you don’t have to count spaces anywhere. You don’t need to count spaces to make sure successive lines of a list item are treated as such. You won’t be the surprised by dramatic difference between indenting a line 3 versus 4 spaces, nor have to notice the very subtle difference between 4 and 5 leading spaces, because unlike CommonMark, there is no difference.

    I don’t consider CommonMark’s indentation rules for list item continuation “magic thresholds” since the idea of vertical alignment make intuitive sense and is easy to learn, but jdot’s approach works too. It would work even better if it did away with manual line wrapping, which I’ll go into in the last section of this reply.

    I found one flaw with regard to whitespace treatment, at least in my opinion. It is easy to rectify.

  5. Leading and trailing pipes in Pipe Tables are required.

    Allowing them to be optional, especially allowing occurance to vary within the same table leads to the following ambiguity:

    column a | column b 
    -------- | -------- 
    in a     | in b      
    in a     |           
             | in b? No, GFM puts me in a.
    
  6. Official support for custom containers / divs :rocket:

The things I’m not so sure of, feels to much like writing code:

  • The elimination of shortcut link syntax. I think this makes things harder for humans just to make them easier for machines. I thought the point of machines was the reverse. I guess am not in favor of Rationales #1 and #2.
  • The {_ and _} syntax. But maybe the use cases for this are so corner that it’s ok that it’s not very natural.

I don’t yet grok the impact of Rationale #6

Rationale #6:

Parsers should not be forced to recognize unicode character classes, HTML tags, or entities, or perform unicode case folding. That adds a lot of complexity.

I suspect the case-sensitivity will be counterintuitive to most people and will result in a lot of broken links. My understanding is that djot does this purely for performance reasons. This to me is another case of favoring the machine over the human.

Manual line wrapping: The decision I question most, for Markdown, djot and others

:triangular_flag_on_post: To be clear, this criticism is NOT djot specific. But if we are going to talk about a replacement for Markdown, it must include a reevaluation of any assumptions or decisions that might have been a mistake.

The unintended consequences to a syntax precipitated by support for manual line wrapping are far ranging and unintuitive, consequences I believe resulting in the well-known non-programmer dislike of Markdown and other “writer-oriented” markup syntaxes.

You might think that manually wrapping lines is entirely optional. Yes it is, but paying the costs of that syntax support is not.

Rationale #7:

The syntax should be friendly to hard-wrapping: hard-wrapping a paragraph should not lead to different interpretations.

Before I get into the costs, I want to question the justification:

why hard-wrap at all? Answer: so that your document is readable just as it is, without conversion to HTML and without special editor modes that soft-wrap long lines

Is soft wrapping truly a “special editor mode”? If it is, it seems like another case where we’re forgetting that machines should be serving humans. There are plenty of editors that soft wrap, and many of them soft wrap intelligently, maintaining the indentation of the preceding line. And for any editor that is going to support syntax highlighting, it can just as easily support smart soft wrapping and indentation that reflects the syntax, e.g. dynamically indenting to align list item content past the list marker.

Some of the many costs (to humans) imposed by manual line wrapping support

  • You need to use a special character to force a line break, e.g. placing a \ at the end of each line of verse.

  • Writing block quotes or list items with manual line wrapping is a real chore. No one should have to enter so many > chars or have to manually indent each wrapped line of a list item. That should be the machine’s job, and in fact many editors do automatic indentation of soft-wrapped lines.

    Even when using a dumb editor that does “dumb” soft-wrapping of block quotes and list items, ditching manual line wrapping and relying on soft wrapping looks no worse than “lazy continuation”. In fact, it is even lazier while having better behavior. It dynamically rewraps as you edit the content or change the viewport size:

    > This is a manually wrapped block
    quote with lazy continuation. Because
    it is hard wrapped, it does not adjust to
    changes in viewport size or to user preference. Even with the
    lazy continuation, it also difficult to edit
    unless you decide to not care about the
    line widths anymore. I needed to insert
    "or to user preference" above.
    
    > This is a soft-wrapped block quote that
    rewraps itself as you edit it. It looks no
    worse than lazy (yet manual) continuation,
    while working much better. It is not subject
    to the weird behavior I describe in a later
    bullet point. A different user can change
    the desired column width, or can edit it
    on a device with a narrower screen, and
    it will rewrap accordingly.
    
    1. This is a soft-wrapped list item. Again,
    it looks just like lazy continuation, but
    behaves way better.
    
        - This is a nested list item that even
    when soft-wrapped by a dumb editor as
    you see here is perfectly legible as an
    nested list item. It will look great with
    the typical *slightly* smarter editor that
    indents soft-wrapped lines.
    
  • Editing existing manually wrapped text becomes a chore. Inserting even a word requires manually rewrapping all of a paragraph. It’s even worse for block quotes, as you have to
    remove and reinsert > as you manually hard wrap. Sure, you can choose not to rewrap, but then why the manual wrapping in the first place?

    :triangular_flag_on_post: You can see this in the many plain text examples below. They became irregular in line width as I edited them. It would have been much worse if the examples weren’t so trivial.

  • Supporting manual line wrapping results in syntax complications all over the place. For example, you are forced to choose between CommonMark’s complex rules for starting a list item after a paragraph without a blank line, or always requiring a blank line as djot does. Forcing blank lines seems the simple solution, but it’s very natural for many people to not use them, even feeling “wrong” to be forced to use them when unnecessary:

    Before leaving:
    - eat all the perishables
    - water the plants
    - turn out the lights
    

    It feels even more unnatural for nested lists:

    - Two weeks before
    
      - Find someone to feed the cat
      - Start eating all the perishables
    
    - Before leaving:
    
      - eat all the perishables
      - water the plants
      - turn out the lights
    
    
  • Complex and unnatural behavior for block continuation:

    > Inside a block quote.
    Still inside the block quote.
    > The second `>` preceding "This" below is treated as
    > regular text if you remove the following blank line.
    >
    > > This is inside a nested block quote.
    > Still inside a nested block quote! 
    
    
    1. In a list item.
    Still in the list item.
    
    Not in the list item.
    
    2. Another list item.
    
        - Nested list item.
       Still in the nested list item.
     Still in the nested list item.
    
        Not in the nested list item.
     But in the outer list item.
    Still in the outer list item.
    
    Not in the list item.
    

    If we instead always treat newline characters as significant, here is what the above would look like with all the manual hard wraps removed and an editor soft-wrapping at 60 chars:

    > Inside a block quote. Still inside the block quote. The
    second `>` preceding "This" below is NOT treated as
    regular text if you remove the following blank line.
    > > This is inside a nested block quote. Still inside a
    nested block quote! 
    
    1. In a list item. Still in the list item.
    
    Not in the list item.
    
    2. Another list item.
        - Nested list item. Still in the nested list item.
    Still in the nested list item.
    
      Not in the nested list item. But in the outer list item.
    Still in the outer list item.
    
    Not in the list item.
    

    That to me is much easier for the human eye to read and correctly parse. It gets even better if the editor maintains indentation when it soft-wraps, as many editors do:

    > Inside a block quote. Still inside the block quote. The
    second `>` preceding "This" below is NOT treated as
    regular text if you remove the following blank line.
    > > This is inside a nested block quote. Still inside a
    nested block quote! 
    
    1. In a list item. Still in the list item.
    
    Not in the list item.
    
    2. Another list item.
        - Nested list item. Still in the nested list item.
        Still in the nested list item.
    
      Not in the nested list item. But in the outer list item.
      Still in the outer list item.
    
    Not in the list item.
    

    And even better yet if the editor is truly smart. Here it also puts in soft > chars when it soft wraps the block quote, and left aligns list item content to the right of the list item marker:

    > Inside a block quote. Still inside the block quote. The
    > second `>` preceding "This" below is NOT treated as
    > regular text if you remove the following blank line.
    > > This is inside a nested block quote. Still inside a
    > > nested block quote! 
    
    1. In a list item. Still in the list item.
    
    Not in the list item.
    
    2. Another list item.
        - Nested list item. Still in the nested list item.
          Still in the nested list item.
    
       Not in the nested list item. But in the outer list item.
       Still in the outer list item.
    
    Not in the list item.
    

    :triangular_flag_on_post: Taking advantage of smart plain text editors doesn’t undermine one of the cardinal benefits of plain text markup, that you don’t need a format-specific editor to edit the content, you just need a d̶u̶m̶b̶ generic editor. All but the last of the above progression of smarter plain text editors are entirely generic and already commonplace. And even the last doesn’t prevent the editor from also working as a generic editor, with its extra-smarts triggered only by filename extension or mimetype.

  • Alway using blank lines isn’t natural or often desirable, as explained within this example:

    ## Examples
    I don't think any human will see this as part of
    the heading.
    
    ### Example 1
    It's natural and often desirable to not use blank
    lines between blocks when they are not needed. 
    
    ### Example 2
    There are many reasons people will choose not to put
    blank lines between blocks, such as:
    - compactness
    - grouping
    - not diluting the visual impact of white space that you
      need elsewhere.
    
    ### Example 3
    
    Whitespace is a powerful tool for readability. But,
    
    - when you have to use it even when not needed
      for the human eye,
    
       - it dilutes that power.
    
         - The above two blank lines are unnecessary.
       - Also, the resulting visual grouping is
         counterintuitive, confusing, reducing readability.
    
         - You can fix this confusing grouping by putting
           whitespace between every list item.
    
         - But would you want to?
    

There other costs. The ones above are off the top of my head. But in essence:

:hot_pepper: Manual line breaks and its unintended consequences to me are Markdown’s greatest Achilles heal. It unnecessarily took us backward in time, back to dumb typewriters, where you have to manual slide the print head at the end of each line, where editing becomes a nightmare. Just because we want to ditch the proprietary or human-unfriendly storage formats of word processors doesn’t mean we have to go back to typewriters.

1 Like

I fixed some significant typos and added clarifications above. The most important being:

My obviously strong and likely controversial thoughts on manual line wrapping reflects a lot time and work I’ve put into a project aimed at making plain text more expressive and flexible while making it more natural at the same time. I’ve examined many, many paths backwards and forwards, not just in my head but in countless use-cases and working code. In doing so, I discovered how much complexity and unnaturalness derive simply from the manual line wrapping fork in the road. This fork occurred so far back that I think most of us, myself included, don’t even realize it was a fork, that we could have made a different choice, and that there is a whole other realm of possibility on the other side of the mountain.

I’m not done with my exploration. I’ve only recently decided to backtrack all the way to that primordial fork and will be exhaustively examining the other world. Maybe I’ll discover I was wrong / rediscover the reason all(?) lightweight markup formats made the choice they did. I wasn’t going to write about this yet, but it was djot doubling down on manual line wrapping that spurred me to write. I’m hoping this leads to a fruitful discussion.

1 Like

Thanks for these interesting comments. The issue of hard-wrapping was one we talked about in the discussions leading up to commonmark.

I have to say that I don’t really agree with your readability judgments; it seems to me that the extra whitespace is desirable (and certainly not undesirable) in these cases.

A couple additional points:

First, if your proposal is that a hard line break should be interpreted as a hard break (i.e. <br>), then there would still be an ambiguity with

She counted (all the way from 1 to
9) and then went in.

Should this be parsed as a single paragraph containing a hard line break and the literal 9), or a paragraph followed by an ordered list? Here you’d need something like commonmark’s current kludge (making a special exception for the number 1 for starting a list without a line break), or you’d need to forbid hard breaks before potential list markers (an expressive blind spot).

Second, genuine hard line breaks inside a paragraph are extremely rare in good typography; it seems a shame to assign this rare thing to the newline character. A somewhat different proposal would make newlines create paragraph breaks. But this isn’t good for readability; it will be too unclear where a paragraph ends and the next one begins.

Third, paragraphs consisting of extremely long lines can cause problems unrelated to readability. For example, lines of text in emails are not supposed to exceed 998 characters (RFC 5322 - Internet Message Format). Coding standards also sometimes impose a maximum line length; if the syntax is used in code comments, it would be affected by this. Diffability is another important consideration: if you have extremely long lines, it is harder to see where changes have occurred inside a paragraph.

Djot does not require hard-wrapping, but it allows it, and I think that’s still important.

1 Like

Sure. I intended to write a short summary that would fit into here but it quickly got long.

New topic: My Experiences Developing a Personal “Beyond Markdown” markdown - Touch Lightweight Markup Language

Depends on use case, but not everything people write is destined for print with “good typography”. Most of it stays online, in casual discussions.
Now here is another “fork”: Some people, I have no idea how many, routinely break lines after sentences or short groups of sentences, with the intent of making the structure of their writing clearer to the reader. I do. We embrace the structure of plain text to have 2 levels of grouping: long lines that are effectively “sub-paragraphs”, and “real paragraphs” separated by blank lines.
(In markdown, I find myself adding double spaces after most line breaks. Rarely do I hard-wrap for myself without wanting the reader to also see the break.)

It’s rare in print and definitely violates writing style guides (unless you write poetry). Possibly because paper pages has a cost and a weight? But online I’ve already seen it in some people’s emails in the 90s (can’t speak to before my time :person_shrugging:) and AFAICT it’s not going away.

This consideration goes both ways…

  • All good diff tools like github/gitlab/meld/delta do 2-level diffing where the whole touched line is highlighted, but then the specific touched words are highlighted stronger within it. So writing each paragraph as one long line is not catastrophic to modern diffing. :neutral_face:
  • Automated hard-wrapping around a fixed width leads to the worst diffs! When words slide from one line to another, they add tons of noise to the diff that’s hard to filter out. :-1:
  • OTOH the best advice for minimizing diff noise is indeed “add newline after every sentence”, when working with formats like LaTeX, markdown etc. that’ll still render them as single paragraph. :+1:
1 Like

It’s a good oportunity to mention the little-known text/plain; format=flowed RFC 3676: The Text/Plain Format and DelSp Parameters. A brilliant ASCII hack IMHO, and I regret it hasn’t caught on among unix terminal tools — or even among mail readers where it was intended :frowning_face:

Hard-wrapping is a lossy action — it’s hard to recognize which newlines are “original” semantic breaks and which were inserted “soft breaks” and safe to remove.
The idea of format=flowed was that by default line breaks still mean a hard break and you opt-out by leaving a single trailing space before the newline. Pairs of “Space Newline” are generally removable, or at least the Newline is (a 2nd param DelSp controlled whether the space is kept, which fits e.g. English, or dropped, which fits languages like Chinese that don’t use spaces between words).
That’s the opposite of markdown, where single source newlines by default are ignored as soft breaks, and 2 trailing spaces opt-in to a hard break.
This choice was made for compatibility with existing ambiguous plain text (and with email clients ignorant of format=flawed), where you can’t be sure which breaks were soft. It’s safer to preserve all breaks than munge all lines together.

=> The particular choice of “Space Newline” is debatable, but consider the idea of opt-in vs opt-out.
Whether a markup format defaults to ignoring newlines or treating them as semantic line breaks, either way it can allow hard-wrapping without requiring it!

1 Like

@jgm, I wanted to hear other points of view before I chimed in again, but since it’s been a couple of weeks…

I spoke of manual line wrapping being a primordial fork in the road. The real primordial fork is this choice:

Should machines be made to understand humans or should humans be made to make themselves understandable to machines?

It is this question that drives my analysis.

machine                                          human
oriented                                        oriented
   <------------------------------------------------> 
       SGML/HTML      Asciidoc   Markdown

Where does djot fall? For some of its choices, a little to the right of Markdown. But in doubling down on manual line wrapping, to the left, which the examples in my above reply try to demonstrate.

The notion of “tight lists” tells us that extra whitespace can be undesirable. More importantly, requiring it has only one purpose: making humans make themselves understandable to the machine. We would not be allowed to write the following even though it is both unambiguous and natural for us:

Before leaving:
- eat all the perishables
- water the plants
- turn out the lights

And we would only know that we were not allowed to write it this way because we’d have to learn a set of rules designed to make ourselves understandable to machines.

I always thought CommonMark’s "kludge’ was a step in the right direction (and defended it more than once on this forum): Make the machine figure out the human. I’d want to go further, adding a heuristic such as: Is there an 8) list item somewhere before this? Is there an 10) item that follows? If not, this is unlikely to be a list item.

If you eliminate manual line wrapping (treating newlines as soft breaks), then it gets considerably better. First, a large source of ambiguity, what @cben calls “Hard-wrapping is a lossy action”, goes away. There simply would be no line break before 9). We can come up with other examples where ambiguity occurs, but they become increasingly contrived. In other words, incredibly rare in the real world. Second, you could modify the heuristic: If unsure, leave it as-is. Even if it were a list item, it will still be seen as such by the human reader even without an explicit list rendering (in HTML or whatever), because it was seen as such in the original.

On your second point: Without any rules, humans will naturally delineate paragraphs with blank lines where necessary.

On your third: Thank you for making me aware of RFC-5322. At some point we have to break the bonds out outmoded limitations (limitations I assume exist because at the time computing resources were so limited choices were made in favor of machines at the expense of humans). Even the RFC says “Receiving implementations would do well to handle an arbitrarily large number of characters in a line for robustness sake.” But more importantly, who writes in plain text markup in an email client that won’t render that Markdown into HTML? Finally, say someone wants/needs to email raw plain text markup. The software would insert newlines where necessary. The occasional > 998 character paragraph (this one is 780 chars), would get an odd line break in the middle. It’s not a big deal.

I’m with @cben on the diffability question.

Djot doesn’t require hard-wrapping, but even users who don’t use it pay its costs.

[I sent a reply by email, but for some reason it didn’t come through. So, apologies if this is a duplicate.]

On Jul 26, 2022, at 11:13 AM, vas via CommonMark Discussion <noreply@talk.commonmark.org> wrote:

If you eliminate manual line wrapping (treating newlines as soft breaks), then it gets considerably better. First, a large source of ambiguity, what @cben calls “Hard-wrapping is a lossy action”, goes away. There simply would be no line break before 9).

I don’t understand your response to my point. Most likely we are understanding some key terms, like “soft break,” differently. As I was using the term here, a “soft break” is a newline in the markdown source that is interpreted semantically as a space. Djot does “treat newlines as soft breaks” in this sense, so I assume you mean something different.

I understood you to be proposing that newlines in paragraphs would produce a hard line break in the rendered output. Assuming that’s right, then you face an ambiguity in

She counted (all the way from 1 to
9) and then went in.

Is this a single paragraph with a hard break,

<p>She counted (all the way from 1 to<br>
9) and then went in.</p>

or is it a paragraph followed by a list?

<p>She counted (all the way from 1 to</p>
<ol start=9><li>and then went in</li></ol>

That’s the ambiguity. It’s just the same as the ambiguity we face now with commonmark, and which we resolve with the unprincipled restriction on start numbers. I guess you like that way of resolving it, but I’ve never been happy with it. In any case, it comes up whether or not you allow hard wrapping (i.e., treat newlines in paragraphs as equivalent to spaces).

As for your main point: certainly, djot falls a bit to the left of markdown. With markdown and commonmark, the aim was to magically guess what humans intended, as far as possible. My mantra was always “favor what is intuitive to humans, and make the parser more complex if necessary.” That is why the parsing rules are so complex. The problem is, even with all this complexity, there are many cases where we don’t get the results people intuitively intend. So, my choice with djot is to give up trying. Performing the task well would require a high degree of general intelligence. Maybe in the future, some successor of GPT-3 will be used to parse our plain text documents, but for now, I’d rather have a simple set of rules that we can keep in our heads, so the output is predictable.

To elaborate: in my example above, any human can tell that the 9) on the second line is the end of the parenthesized phrase in the preceding line, and not the start of a list, whereas in

A further point is
9) blah blah blah

we have a list. But in doing this we’re relying on our grasp of the meaning of what is written; it’s very hard to predict human intentions in such cases with a set of syntactic rules. Consider the minimal pair:

A more interesting number is
6. This is a "perfect" number.

vs

A more interesting point is
6. This is a "perfect" number.

In the second case a list is probably intended (depending on preceding content), while in the first a list is not intended. We can figure that out because we know the difference between “point” and “number” and we have a grip on what the writer is trying to achieve. Our markdown parsers can’t learn this by being given more and more complex rules. They’d need to have a psychological model of the humans writing the text, and an understandig of the meanings of the words.

The idea behind djot is to keep things simple, uniform and predictable while still achieving most of the aims of markdown.

John

On diffability: I think the best approach for diffability is one sentence per line. However, that speaks in favor of djot’s current approach, on which newlines inside paragraphs render as spaces rather than hard breaks.

The bottom line, though, is that I don’t want to impose any one style on the user. Some people like one sentence per line. Some people like hard wrapping to a fixed width. Some people like one big line per paragraph. All of these styles work fine with djot. I’d be reluctant to change djot in a way that requires one of these styles and excludes the others.

The most obvious criticism I have is that

  1. It should be possible to attach arbitrary attributes to any element.

Is not a proper design goal. It’s a design decision made to achieve some design goal. A design goal is something like “It should be easy to implement the parser” or “It should be possible to add html blink tags.”

Now I’m very dubious on attaching arbitrary attributes as a design choice, but it would be a mistake for me to argue against it since there’s no actual design goal. I don’t know what you’re trying to accomplish. And I need to know to be able to suggest alternatives.

Part of what I’m thinking at the moment is that I don’t like the idea of sprinkling bits of markup (in the form of attribute specifications) all over the text, since this directly conflicts with markdown’s goal of being readable as text. An alternative might be to actually standardize the “Front Matter” convention you see in CMS’s that use markdown (e.g. Hugo), where you have

-----
title: My stupid clickbait blog post
date: 2022-8-6
tags:
  - seo
  - clickbait
-----

Then maybe you could define one or more attributes to contain queries (maybe css selectors) to locate block/inline elements and specify attributes to be applied to them. In this way you contain all the ugliness to the front matter. Now this is probably a terrible idea that you shouldn’t use, but without a proper design goal it’s hard to actually demonstrate this. For example, this approach would probably make the parser easier to implement (assuming parsing/evaluating the css selectors is handled by an existing library), and it satisfies the goal of allowing us to apply arbitrary attributes to elements.

Or to make this even more pathological, we specify a series of sed programs to be run on the finished html, so we don’t have to worry about having a css library. This appears to satisfy the design goals even better. I just threw up in my mouth a little.

Let’s switch gears and see if I can come up with some suggestions that are actually worthwhile. One of the changes you made was to cut out redundant syntax elements. Personally, I have a strong preference for the setext headings and would miss them if they were gone, but let’s leave that aside briefly. One could argue that if you’re going to start trimming down the language, there’s no reason to include the inline link definition ([link text](uri)) when the reference link definition ([link text]...[link text]: uri) is obviously so much better.

Assuming for the sake of argument that we agree with me about reference links, we could apply the same reasoning to attributes. Instead of writing

The word *atelier*{weight="600"} is French.

(Yuck!) We write e.g.

The word *atelier*{fw} is French.

{fw}: weight="600"

In both cases, we satisfy the goal of being able to add non-text stuff (links, arbitrary html attributes) to our text without substantially compromising the readability of the text. And by banning the obviously-inferior inline versions, we simplify the parser. Also, we only define fw once, so we can re-use the same 4-char reference several times in case we add more french words to our document.

The disadvantage of this is that we can’t straightforwardly compose the two definitions, as in

[Link text](url){title="Click me!"}

But we could still presumably compose the invocations and write

[Link text]{cmt}

{cmt}: title="Click Me"
[Link text]: url

in re the setext headings: I want the way I write the h1 elt to look like a big title, and the way I write the h2 elt to look like a substantially-document-dividing subtitle. I don’t think that the atx headings qualify. Ignoring post-title hashes does at least let me write

# Title ##################################

intro

## Subsection 0 #########

text

## Subsection 1 ##########

more text

Which isn’t bad, but I’m not a huge fan. This maybe isn’t a critical issue, but I still don’t get the same warm fuzzy feelings I get from markdown setext headings.

An alternative might be: If the heading contains a trailing #, The line immediately after the heading is skipped. Then I write

# Title #
=============================

intro

## Sub 0 #
------------------------------------------------

text

## Sub 1 #
-------------------------------------------------

text

Now this is almost as nice in plain text as markdown (the extra hashes aren’t a big deal), and we don’t actually have to make the parser understand the setext headings, since they’re basically just comments. In fact, I can use $$$$$$$$$$$$$$$$$$$$$$$$$$$ as my h3 separator if I want.

Finally, I agree with @jgm on the hard-wrap support thing, but I really don’t feel like saying anything substantial about it.

The primary goal is to provide a flexible way to make the markup extensible.

Suppose you are writing a document that needs to index certain terms. What you want is a filter: a program that reads the abstract produced by the parser and enhances it by adding an index. (If you’re not familiar with this in practice, go look at some examples of pandoc filters.) But in order for this to work, the filter needs to know which terms are to be put into the index, so this information needs to be represented in the AST. If we can add attributes to bits of text, we can do that using attributes. e.g., [cat]{.index}, or [cat]{.index see="feline"}.

That’s just one application. Instead of marking individual words and phrases, you might want to mark groups of paragraphs. Then you attach the attributes to a fenced div. Or you might want to mark a code block as content that should be turned into a diagram via mermaid, using certain DPI settings or whatever. Again, you can do this with attributes.

So, attributes provide hooks through which external filters can interact with the parsed AST.

That’s the main reason I want them, anyway. They’re also useful for things like defining anchors for cross-references.

Okay, that makes sense.

Looking at your example makes me wonder if I want to augment my attribute-ref proposal to make the attribute specs composable, as in

[cat]{cat} and [dog]{dog}

{idx}: .index
{cat}: {idx} see="feline"
{dog}: {idx} see="canine"

Or whether that introduces too much complexity.