Djot: A light markup language by @jgm

vas · July 12, 2022, 1:15am

My thoughts…

Lot’s of things to like:

Simpler and richer inline syntax. The precedence rules.
Nixing indented code blocks. Yea!
Footnotes. Yea!
No magic whitespace thresholds.

In djot, you don’t have to count spaces anywhere. You don’t need to count spaces to make sure successive lines of a list item are treated as such. You won’t be the surprised by dramatic difference between indenting a line 3 versus 4 spaces, nor have to notice the very subtle difference between 4 and 5 leading spaces, because unlike CommonMark, there is no difference.

I don’t consider CommonMark’s indentation rules for list item continuation “magic thresholds” since the idea of vertical alignment make intuitive sense and is easy to learn, but jdot’s approach works too. It would work even better if it did away with manual line wrapping, which I’ll go into in the last section of this reply.

I found one flaw with regard to whitespace treatment, at least in my opinion. It is easy to rectify.
Leading and trailing pipes in Pipe Tables are required.

Allowing them to be optional, especially allowing occurance to vary within the same table leads to the following ambiguity:
```
column a | column b 
-------- | -------- 
in a     | in b      
in a     |           
         | in b? No, GFM puts me in a.
```
Official support for custom containers / divs

The things I’m not so sure of, feels to much like writing code:

The elimination of shortcut link syntax. I think this makes things harder for humans just to make them easier for machines. I thought the point of machines was the reverse. I guess am not in favor of Rationales #1 and #2.
The {_ and _} syntax. But maybe the use cases for this are so corner that it’s ok that it’s not very natural.

I don’t yet grok the impact of Rationale #6

Rationale #6:

Parsers should not be forced to recognize unicode character classes, HTML tags, or entities, or perform unicode case folding. That adds a lot of complexity.

I suspect the case-sensitivity will be counterintuitive to most people and will result in a lot of broken links. My understanding is that djot does this purely for performance reasons. This to me is another case of favoring the machine over the human.

Manual line wrapping: The decision I question most, for Markdown, djot and others

To be clear, this criticism is NOT djot specific. But if we are going to talk about a replacement for Markdown, it must include a reevaluation of any assumptions or decisions that might have been a mistake.

The unintended consequences to a syntax precipitated by support for manual line wrapping are far ranging and unintuitive, consequences I believe resulting in the well-known non-programmer dislike of Markdown and other “writer-oriented” markup syntaxes.

You might think that manually wrapping lines is entirely optional. Yes it is, but paying the costs of that syntax support is not.

Rationale #7:

The syntax should be friendly to hard-wrapping: hard-wrapping a paragraph should not lead to different interpretations.

Before I get into the costs, I want to question the justification:

why hard-wrap at all? Answer: so that your document is readable just as it is, without conversion to HTML and without special editor modes that soft-wrap long lines

Is soft wrapping truly a “special editor mode”? If it is, it seems like another case where we’re forgetting that machines should be serving humans. There are plenty of editors that soft wrap, and many of them soft wrap intelligently, maintaining the indentation of the preceding line. And for any editor that is going to support syntax highlighting, it can just as easily support smart soft wrapping and indentation that reflects the syntax, e.g. dynamically indenting to align list item content past the list marker.

Some of the many costs (to humans) imposed by manual line wrapping support

You need to use a special character to force a line break, e.g. placing a \ at the end of each line of verse.

Writing block quotes or list items with manual line wrapping is a real chore. No one should have to enter so many > chars or have to manually indent each wrapped line of a list item. That should be the machine’s job, and in fact many editors do automatic indentation of soft-wrapped lines.

Even when using a dumb editor that does “dumb” soft-wrapping of block quotes and list items, ditching manual line wrapping and relying on soft wrapping looks no worse than “lazy continuation”. In fact, it is even lazier while having better behavior. It dynamically rewraps as you edit the content or change the viewport size:

> This is a manually wrapped block
quote with lazy continuation. Because
it is hard wrapped, it does not adjust to
changes in viewport size or to user preference. Even with the
lazy continuation, it also difficult to edit
unless you decide to not care about the
line widths anymore. I needed to insert
"or to user preference" above.

> This is a soft-wrapped block quote that
rewraps itself as you edit it. It looks no
worse than lazy (yet manual) continuation,
while working much better. It is not subject
to the weird behavior I describe in a later
bullet point. A different user can change
the desired column width, or can edit it
on a device with a narrower screen, and
it will rewrap accordingly.

1. This is a soft-wrapped list item. Again,
it looks just like lazy continuation, but
behaves way better.

    - This is a nested list item that even
when soft-wrapped by a dumb editor as
you see here is perfectly legible as an
nested list item. It will look great with
the typical *slightly* smarter editor that
indents soft-wrapped lines.

Editing existing manually wrapped text becomes a chore. Inserting even a word requires manually rewrapping all of a paragraph. It’s even worse for block quotes, as you have to
remove and reinsert > as you manually hard wrap. Sure, you can choose not to rewrap, but then why the manual wrapping in the first place?

You can see this in the many plain text examples below. They became irregular in line width as I edited them. It would have been much worse if the examples weren’t so trivial.
Supporting manual line wrapping results in syntax complications all over the place. For example, you are forced to choose between CommonMark’s complex rules for starting a list item after a paragraph without a blank line, or always requiring a blank line as djot does. Forcing blank lines seems the simple solution, but it’s very natural for many people to not use them, even feeling “wrong” to be forced to use them when unnecessary:
```
Before leaving:
- eat all the perishables
- water the plants
- turn out the lights
```
It feels even more unnatural for nested lists:
```
- Two weeks before

  - Find someone to feed the cat
  - Start eating all the perishables

- Before leaving:

  - eat all the perishables
  - water the plants
  - turn out the lights
```

Complex and unnatural behavior for block continuation:

> Inside a block quote.
Still inside the block quote.
> The second `>` preceding "This" below is treated as
> regular text if you remove the following blank line.
>
> > This is inside a nested block quote.
> Still inside a nested block quote! 


1. In a list item.
Still in the list item.

Not in the list item.

2. Another list item.

    - Nested list item.
   Still in the nested list item.
 Still in the nested list item.

    Not in the nested list item.
 But in the outer list item.
Still in the outer list item.

Not in the list item.

If we instead always treat newline characters as significant, here is what the above would look like with all the manual hard wraps removed and an editor soft-wrapping at 60 chars:

> Inside a block quote. Still inside the block quote. The
second `>` preceding "This" below is NOT treated as
regular text if you remove the following blank line.
> > This is inside a nested block quote. Still inside a
nested block quote! 

1. In a list item. Still in the list item.

Not in the list item.

2. Another list item.
    - Nested list item. Still in the nested list item.
Still in the nested list item.

  Not in the nested list item. But in the outer list item.
Still in the outer list item.

Not in the list item.

That to me is much easier for the human eye to read and correctly parse. It gets even better if the editor maintains indentation when it soft-wraps, as many editors do:

> Inside a block quote. Still inside the block quote. The
second `>` preceding "This" below is NOT treated as
regular text if you remove the following blank line.
> > This is inside a nested block quote. Still inside a
nested block quote! 

1. In a list item. Still in the list item.

Not in the list item.

2. Another list item.
    - Nested list item. Still in the nested list item.
    Still in the nested list item.

  Not in the nested list item. But in the outer list item.
  Still in the outer list item.

Not in the list item.

And even better yet if the editor is truly smart. Here it also puts in soft > chars when it soft wraps the block quote, and left aligns list item content to the right of the list item marker:

> Inside a block quote. Still inside the block quote. The
> second `>` preceding "This" below is NOT treated as
> regular text if you remove the following blank line.
> > This is inside a nested block quote. Still inside a
> > nested block quote! 

1. In a list item. Still in the list item.

Not in the list item.

2. Another list item.
    - Nested list item. Still in the nested list item.
      Still in the nested list item.

   Not in the nested list item. But in the outer list item.
   Still in the outer list item.

Not in the list item.

Taking advantage of smart plain text editors doesn’t undermine one of the cardinal benefits of plain text markup, that you don’t need a format-specific editor to edit the content, you just need a d̶u̶m̶b̶ generic editor. All but the last of the above progression of smarter plain text editors are entirely generic and already commonplace. And even the last doesn’t prevent the editor from also working as a generic editor, with its extra-smarts triggered only by filename extension or mimetype.

Alway using blank lines isn’t natural or often desirable, as explained within this example:

## Examples
I don't think any human will see this as part of
the heading.

### Example 1
It's natural and often desirable to not use blank
lines between blocks when they are not needed. 

### Example 2
There are many reasons people will choose not to put
blank lines between blocks, such as:
- compactness
- grouping
- not diluting the visual impact of white space that you
  need elsewhere.

### Example 3

Whitespace is a powerful tool for readability. But,

- when you have to use it even when not needed
  for the human eye,

   - it dilutes that power.

     - The above two blank lines are unnecessary.
   - Also, the resulting visual grouping is
     counterintuitive, confusing, reducing readability.

     - You can fix this confusing grouping by putting
       whitespace between every list item.

     - But would you want to?

There other costs. The ones above are off the top of my head. But in essence:

Manual line breaks and its unintended consequences to me are Markdown’s greatest Achilles heal. It unnecessarily took us backward in time, back to dumb typewriters, where you have to manual slide the print head at the end of each line, where editing becomes a nightmare. Just because we want to ditch the proprietary or human-unfriendly storage formats of word processors doesn’t mean we have to go back to typewriters.