My Experiences Developing a Personal “Beyond Markdown” markdown - Touch Lightweight Markup Language

1

Originally posted in Djot: A light markup language by @jgm.

Hello,

I have been interested and have been following CommonMark and Beyond Markdown for a couple of years now. I love that you unified the different markdown flavors by writing a specification and that you are constantly trying to improve it.

I used to write tons of documentation in markdown, CommonMark, and, finally, Pandoc’s Markdown. With time I needed more HTML-like elements and features, that’s why I transitioned to pandoc which has the largest feature set. While pandoc is an awesome tool, I found it’s markdown’s syntax “hacky” (lacking a better word) and verbose, thus harder to read.

There was another thing that was bothering me. As written in Beyond Markdown, because CommonMark tries to stay faithful to markdown, it has complex rules which make the language harder to understand for humans and computers alike.

For these reasons (and because I really wanted to write a parser), I made my own “Beyond Markdown” markdown, called Touch (GitHub - touchmarine/to: 📜 Touch Lightweight Markup Language; Familiar, Extendable, Auto-Formattable). I have made it well over a year ago now and I have been using it daily since. As you are freshly starting to develop djot (at least publicy) and getting the ball rolling on Beyond Markdown, I would like to share some of my findings from developing my personal version of Beyond Markdown.

My Experiences Developing a Personal “Beyond Markdown” markdown

This is a brain dump, it is not necessarily written in a meaningful order.

I will use markdown as an all-encompassing term for markdown-like languages.

Feature Set and Progressive Enhancement

There are roughly 3 levels of markdown usage:

  1. plain text with annotations (em, strong)
  2. structured content (headings, code blocks)
  3. advanced markup (tables, table of contents, HTML elements, latex)

While CommonMark serves well the levels 1 and 2, things get complicated in level 3. If you switch to GFM, which is widely known and supported, you get tables, but other features are still out of reach.

You can go a long way with the features CommonMark/GFM offer you, but when you need more, you will have to drastically change your workflow. Either you switch to another, more powerful, and lesser-known markdown flavour or you switch to another (non-lightweight) language completely and use markdown inside it.

Goal

The language should be progressive and able to scale with you and your team.

My Solution

The language should come with a default set of common elements (like a protocol that everyone speaks and can interchange). But, it should also be extendable so you and your team can add the elements you need without having to change the whole workflow.

Instead of hardcoding elements, Touch knows how to parse “signatures” of elements. All the different types of element syntaxes are abstraced into “signature” types like indented and fenced.

Explicit, Simple, and Obvious

I like Go’s syntax and was inspired by it; I have a preference for orthogonal building blocks that are easily composable and as little to no magic as possible.

Goals

My goals for the syntax were/are:

  • familiar (to markdown and all other commonly used markup languages)
  • no complicated rules
  • orthogonal (changing a does not change b)
  • canonical (one true way to do things)
  • don’t use characters with widely-known uses as delimiters (e.g. #, @, {{, {#)

My Solution

Straightforward rules
  • UTF-8
  • no SmartyPants or entity/numeric references—use Unicode which is readable in plain text form
  • only obvious whitespace—no double space line breaks
  • a list is nested if it is indented more than the delimiter of the parent list
  • no indented code blocks
  • no HTML (you can add custom elements instead)
Inline elements use 2 character delimiters

Pros:

  • more explicit:
    • easier to differentiate from normal punctuation (* vs **)
    • easier to parse
    • improved scannability (and I believe unhurt readability)
  • closing delimiter is not required (the element spans until the end of the current block):
    • closing delimiter is added automatically by auto-formatter
    • when typing in environment with live preview, **a can immediately bold a

Cons:

  • more verbose
Block elements use 1 character delimiters

We don’t normally use punctuation characters (which usually serve as delimiters) at the start of paragraphs. As such, a single character is enough to know that * in * a is a delimiter. Because we know that inline delimiters are 2 characters and blocks 1 character long, we can easily differentiate between the two even if they use the same delimiter character:

*  // block element
** // inline element

Orthogonal and Composable

Elements should do one job and that job only, no matter where they are placed. Following this principle makes for obvious syntax and enables composability. Touch utilises composability to enable the use of complex elements that are hard to express otherwise.

Sticky Elements

Sticky elements are elements that stick to other elements. They form a compound element with the element they stick to (called the target element). In this compound element, the sticky element acts as an auxiliary that provides additional info to the target element. Let’s look at a couple of sticky elements.

Named Link

Named Link is formed if a Link is placed after a Group.

[[a]]      // Group
((b))      // Link
[[a]]((b)) // Named Link
Sticky Subtitle

Sticky Subtitle is formed if a Subtitle is placed after any other block element.

= Report // Title (or other block element)
_ Q2     // Subtitle

In HTML, the Sticky Subtitle in the example above can be represented in a <header>:

<header>
  <h1>Report</h1>
  <p>Subtitle</p>
</header>
Sticky Attributes

Sticky Attributes is formed if an Attributes is placed before any other block element.

! id="heading2" class="display" // Attributes
== Heading 2                    // Heading (or other block element)

Auto-Formattable

I like style guides and I love code formatters. They relieve me of an unnecessary burden and make collaboration easier. No more style-related pull requests and flame wars.

That is why, Touch comes with a formatter that automatically formats your code into the normalized form. It also supports hard wrapping at a custom line length (which solves the pain point of manual line wrapping that @vas pointed out).

Post-Processing (Transformers and Aggregators)

Touch utilisies a lot of post-processing which has the following advantages:

  • leaner and simpler parser
  • enables greater level of composition
  • modular design

Touch splits post-processors into transformers and aggregators.

Transformers

Transformers traverse the node tree and add new elements to it. Their job is to group elements by looking for simple patterns. They are used to add paragraphs, lists, and sticky elements.

The patterns are simple for humans and computers:

  • paragraphs group leaf elements that have a sibling
  • lists group contiguous sequences of the same sibling elements
  • sticky elements group elements that have a sticky element placed before/after them

Aggregators

Aggregators aggregate (collect) data we are interested in, such as the headings needed to generate a table of contents. They traverse the node tree after the transformers.

Tables

Tables are hard. You need another dimension to represent them which is almost impossible in plain text if you need readability. I have tried numerous notations for tables and none felt right. GFM tables and Djot pipe tables provide enough functionality, are easy to read and write, and are useful. Nonetheless, they make parsing and tooling more complex and we should minimize complexity if possible.

I think that tables are important and some basic functionality should be available in lightweight markup languages. Without them, we quickly get to the point of breaking our workflow and needing to change our tools, again. I currently use Unicode tables designed in online editors, but while the readability is great, editing is painful as you need another tool.

One of the major challenges with tables is what is allowed inside them. Can you place blockquotes in them? Can you write inline comments? Can you use backslash escapes? Tables have their own context and the rules inside them are not the same as outside. They break our orthogonality rule. There is no easy solution as they do not naturally fit into the linear format of documents.

The most pragmatic solution (to me, right now) seems to be Djot-like pipe tables with auto-formatting help. Or maybe, we could have a small purpose-built markup language for tables that would be integrated like code blocks? After all, would it not be more correct and obvious that tables would live in their own blocks?

Configuration

This section is specific to Touch and it’s configuration.

Touch’s configuration is pretty messy and difficult to use right now. You need to use my “Extended JSON” (which is just pre-processed JSON with raw multiline strings) which is rubbish and the templates are hard to read and contain too much logic.

I am currently testing a new configuration which makes things much easier and templates basically logic-less. If you are evaluating whether you would use a configurable markup language, make sure to checkout the new configuration. You can compare the default configurations to see the difference:

2 Likes
2

I was excited to read about Touch, because I, like you, decided that what the structured plain text world really needs is an easy-to-use declarative meta language or meta solution. Mine is called Plain Text Style Sheets, and combined with Textplain, my rendering engine, it has some striking similarities to yours:

  • recognizing common lightweight markup conventions (what you call “signatures”) and factoring them out into a set of generic semantic and structural elements that form the basis of a declarative, meta markup language. Custom user syntax is built up by declaring specific configurations of these meta elements (e.g. Touch’s leaf, walled, fenced, hanging, uniform types)

  • declarative customization of element rendering via element-level templates. No need to code an AST walker/renderer, which is not only harder, but only works with one parser (e.g. CommonMark.js or Markdown-it) and only in one language (e.g. Javascript). Also supports templates for different output formats (e.g. HTML, LaTeX, PDF).

  • composability / compound elements

  • a processing pipeline (e.g. Touch’s transformers and aggregators)

This is very cool. While Touch and PTSS/Textplain share these principles/approaches, they also differ in other principles, approaches, and, for example, in the visual styles of the plain text element types supported. The resulting supported syntaxes will be significantly different. They won’t be interchangeable – which is a good thing, for me at least. You haven’t made my project obsolete even before I released it!

I won’t be ready to release PTSS and Textplain into the wild for a while yet, even though they already fully support CommonMark, GFM and other syntaxes, because there are other ideas I am incorporating into my POC beyond PTSS and Textplain. Hopefully by end of year, depending on how much time I can spare it. (The time spend looking at Touch and writing this is also in service of my project :).

some questions

  1. How would Touch support something like CommonMark’s Link Ref Defs? Does the Aggregates feature support that?
  2. In the case of NamedLink, you have a very specific composition of two specific elements, Link and Group. But you define Subtitle as something that can go after anything, from what I can tell what it really does it turn whatever it is attached to into an HTML5 <header>, with itself as paragraph text within, more like a caption within the header. Doesn’t it make more sense to change its definition to only be attachable to Headings? I ask because I want to know if there is reason you did it that way that has something to do with how Touch works or a limitation or something. Especially since you have Subtitle in the default config.
  3. If you did attach a Subtitle to a heading, the ideal behavior would be for it to render as a heading with a level one lower that the heading to which it is attached, e.g. an <h3> attached to an <h2> wrapped in a <header>. Can/will Touch support something dynamic like that?
  4. In the playground you state “This config overrides (shallow merges) the default config.” Is there a way to control whether a config inherits from the default config, from nothing, or extends some other arbitrary config?
  5. The processing pipeline is fixed, right? Is there a way for users to insert their own transformers or aggregators or an arbitrary AST processor?
  6. You can’t define a configuration that would parse Markdown, right? I’m not saying you should, just want to confirm what I see.

suggestions

  • I couldn’t find any really good demonstrations with extended, real-world examples showing off the power of all the default syntax elements and how easy it is to extend them, both by adding new elements and via composition. I’d recommend you find some good README’s and perhaps an textbook chapter or dissertation with complex formatting and show how you would compose them in Touch.

  • You have overloaded “group” with two meanings. Suggest you pick another term for one of the meanings. E.g. Span may be a better term for the element you defined as type: uniform, delimiter: '['.

  • Yes, the YAML is much better than the JSON. You could also consider your own DSL, but looking at the JSON I don’t see a need for anything more sophisticated. You get only one first impression with each person who looks at Touch, so you want to make sure they can grok it quickly. The YAML is much better for that. For example when looking at the playground, it’s really hard to read the config, and also very hard to edit it without breaking the JSON. You should switch as quickly as possible and update all your examples accordingly.

  • The playground should let you modify the default config as well, if you really want to allow visitors to understand Touch.

  • In your blog post https://touchlabs.io/blog/the-go-flag-package-strange-but-good you have some quotes followed by a citation. Isn’t this the perfect opportunity to define a custom sticky for block quotes that result in an attached citation, rendering to an HTML5 <cite>? If you demonstrate that in a before and after example, it would show potential users how easy it is to defined new syntax and map it to new HTML elements.

  • Since you built it in Go, I recommend you fork Hugo and add it as an optional parser/renderer, at least for demonstration for how well it would integrate into a static site generator like Hugo. When Touch is ready for primetime, you could submit your fork as a PR, though I must warn you Hugo’s maintainer is really hard to work with.

  • Also your mapping of element definitions to rendering templates clearly supports outputs other than HTML. It might be powerful to show a LaTex or even PDF rendering.

Great work!

3

Whoa, thanks for taking the time to explore Touch and writing this up! I’m excited about PTSS/Textplain and would love to try them when they’re ready. It’ll be nice to see a “path not taken” and to learn from them.

Answers to Questions

  1. I don’t think that Link reference definitions provide enough value for the complexity and unclarity they bring. That’s why I wouldn’t include them by default. However, you could add them, as you mentioned, using aggregates. (More on aggregates and transfomers below.)

  2. Yes, it would probably make more sense for Subtitle to only be attachable to Headings. I haven’t done so, mainly, because I’m always testing different possible “combinations” and syntaxes. I didn’t want to limit myself to a preconceived notion of Subtitle being limited to Headings.

    Maybe I can join Subtitle and Caption into a single generalized element? After all, they are used in a similar manner and exclusively of each other. Joining them would mean users would need to know and choose between one less element.

    Another, less important reason, is that Touch comes with Heading and Numbered Heading elements and the Target option for sticky elements doesn’t support multiple elements. I would need to add support for multiple elements (of which I’m not sure if it’s needed) or create two different Sticky Subtitles (one for each heading).

  3. I haven’t thought of something like that yet actually—this is why communication is great.

    And, it’s already possible—you can pass information between child elements of a sticky element, and the heading level is accessible as .Data.rank (rank=level of ranked elements).

  4. To make configuration as transparent as possible I decided that:

    • you can only add configs; and
    • the default config is always present.

    This combined should make for easier debugging.

    You can modify the current config by overriding (shallow merging) it. To remove an element you add an element with the same name and an entry "Disabled": true (see How to remove default elements - Tour of Touch).

    For convenience, I plan on adding a config that would reset the default config, getting you to “nothing”.

  5. This the section with more about aggregates and transformers.

    I realized that trying to make transformers and aggregators configurable (in the way they are configurable today) was a mistake. Overall, I shoved too much things into the config making it needlessly complicated. See the Table of Contents config (toc.extjson) for an example. Writing and reading this config, especially the template, is much more difficult than simply delegating the complex logic to plugins.

    With the new YAML configuration this will change. Config will be as simple as possible so anyone with a little YAML and HTML knowledge will be able use it. All the complex logic like aggregates and transformers will move into plugins (Go plugins at first and later also JS). In the config you will define which aggregates and transformers to run and their arguments alongside them. Even templates for aggregates/transformers will be passed as arguments.

    And to answer your question: there is currently no way to add your own post-processing (which is stupid).

  6. Touch has simplified and different rules that aren’t able to parse markdown. I would love, however, to make a pandoc writer/reader for Touch so it could be converted from/into a myriad of other markups.

Suggestions

The documentation and examples are without doubt lacking at present. I recognize that it’s hard for newcomers to get started with Touch as there is not enough support material. But, this will change as I’m steadily working on Touch.

Regarding configuration, I’m very happy with going with YAML. It has it’s quirks but it has all the necesarry features and most imporantly—it’s widely understood and has implementations in all major languages.

As with the documentation, integrations are also on the roadmap. They are very important to me and key to overall usability of Touch.

Finally, multiple outputs are a bit like the aggregates/transformers fiasco. The idea of them is great, but at this stage they’re only a burden. I should’ve started working on them only around now or preferably even later. (But I do use them to convert Touch to markdown before commiting to GitHub and it gets me most of the way.)

1 Like