What is necessary to get long-standing discussions actually formalized into CommonMark?

Meta
#1

We’ve recently adopted CommonMark on our platform after a long period of time using a general Markdown input that was mostly CommonMark but not quite. Even now, we’ve needed to add additional support for “homegrown” Markdown for things that haven’t been formalized into CM. That said, one of the things I notice is that there are many long-standing discussions about how CM could implement one feature or another - for example, spoilers.

That discussion was started in 2014 and still doesn’t have a solution for what seems like (on the surface) to be a relatively simple format (as opposed to something like tables). In the time since, other platforms that use CM have created diverging solutions to how to create spoilers leaving the CM implementation solution muddier because, rather than 1-2 potential solutions to choose from, there are now several - which means, if CM chooses a standard, those variants will have to adjust and potentially annoy their users. There are similar discussions for things like tables, too, so this isn’t specifically about spoilers - and I’d guess there are some I’m not aware of.

So, my questions

  • At what point does CM decide that a discussion has been going on long enough and opt to actually choose a standard for implementation and document it?
  • Is there something I can do to help push a standard to be implemented (regardless of what that standard is)?
  • Alternatively, is there a point where one should assume that there are too many diverging solutions that CM won’t create a standard?
#2

My take is that extending Markdown to do what everyone wants ends up violating Markdown’s philosophy and the very reason it succeeded so well. Shoehorning everything one can do in HTML into Markdown’s syntax changes it from a format natural and intuitive for humans into machine code. Markdown syntax was never meant to be a sort of “Turing complete” source language for HTML. It was meant to cover the vast majority of content. For the rest, one can use the loophole (Markdown’s support for embedded HTML), or use another format.

I believe the reason there has been little urgency on the part of CommonMark’s stewards to add extensions is that they understand this (Please correct me if I’m wrong @jgm and @codinghorror). I think any extensions that might be added would be for those very common needs, such as tables.

Now if there were a way to make Markdown extensible “in the Markdown way”…

1 Like
#3

I can certainly understand wanting to only include the universal sorts of things and that’s completely a valid option for any of the various discussions about what should/should not be included - the reason I’m asking here is because, at some point, there should be something that comes from the CommonMark Team/Community/Whoever that says - “we’ve reviewed this idea and discussed it and we don’t think it’s universal enough to include in our spec.”

And then there can be a sort of … end to the discussion.

If people have spent six years throwing around ideas for spoilers, eventually we should probably just say, “Yeah, we don’t think spoilers are universal enough to make a standard.” (For example) And then, of course, we can start discussions about why they should be. :wink:

If there’s no indication that something’s not happening and people like @codinghorror are saying things like -

… it feels like it’s actually going somewhere (for both tables and spoilers), so you’re just … waiting for something to happen. I understand completely if this is not a priority - I’m walking in off the street and don’t actually know what the priorities are right now or if anything’s actually moving forward with CM… but I’d love to at least resolve some of these requests.

1 Like
#4

There hasn’t been a movement for including additional features in the core CommonMark spec - tables, spoilers, etc are categorised under “Extensions” on this forum for a reason. So these features, if covered, would probably be seperate extension specs.

There’s some discussion of how to formalise extensions in this topic:

1 Like
#5

I appreciate the pointer to that discussion - it’s helpful to see. I also find it (somewhat) amusing that the question was asked by you relatively generally and the responses seem focused on an offshoot of “how can we solve specific problems with tables” rather than addressing the crux of your question, essentially, “Should we start with something simpler than tables?”

We’ve also recently added tables to our platform and we’re running into many of the same issues I see discussed there and some that are probably discussed elsewhere - tables, particularly considering headers (horizontal and vertical) merged/spanning cells, using pipes in cells, using other MD in cells (lists, etc), and all other sorts of things does greatly complicate that as an extension, particularly if you’re looking for a proof of concept that could be simpler and not have so many caveats.

I can also understand why these features would be extensions rather than default - they are complex and not always needed, so having them be add-ons rather than defaults makes sense. Or maybe I’m not clear on the difference between an extension and being part of the core spec. jgm’s last post there seems to state that there may be cases where tables need to be part of the core spec rather than an extension.

Either way, I do like your idea of choosing a simpler place to start with an extension though, other than tables and spoilers, I don’t know what candidates there are.

#6

I think this is an important question. There are countless appeals for things to be added to the standard, and it would be good that the line between what is added to the core and what is left for extensions is based on a consistently applied principle.

I think the benefits of putting something in the core is that writers can depend on it being available on any CommonMark supporting platform. But it also has a corresponding cost: every tool/platform/site has to support it. There are other cost: making the core standard less simple for writers and more complex for implementers, taking up some of the already very limited design space for human-oriented plain text (there are only so many symbols in the ASCII space available to use).

In the case of spoilers I’m not so sure about the benefit and even less sure about justifying the costs:

  • The benefit is that a user can post something with spoilers without looking up a site specific spoiler syntax. But I think that, even if @codinghorror is correct that spoilers are not uncommon, this benefit is limited to a rather narrow subset of people and websites. Obviously it has no use for books, research papers, or anything that is not interactive such as LaTeX, PDF or even HTML that must work without Javascript. It has no use for the vast majority of writers whose writing doesn’t contain spoilers.

  • The cost is rather large because even in an interactive context such as the web, it is not natively supported by HTML. CommonMark would have to define an HTML class attribute used to tag spoilers (choosing a name that avoids collisions), and then every CommonMark supporting website would have to add CSS/Javascript for that class to properly hide and show it upon request.

Is the cost of asking users on Reddit or a Discourse movie forum to learn the site specific forum syntax (e.g. [spoiler] [/spoiler]) that big a deal?

1 Like
#7

The feature set of CommonMark (core) reflects the feature set of the Markdown spec. So, for example, “code blocks”, including the additional GFM syntax for this feature, are part of the spec because code blocks are part of Markdown. Tables and spoilers were never supported in the original Markdown implementation (outside of raw HTML) so have been classified under “extensions”.

1 Like
#8

That seems like a clear enough separation - but the statement that jgm wrote implies there might be cases where it’s necessary to include what might otherwise be an extension as part of core, or am I misunderstanding. Here’s the quote:

That seems like it only applies in an edge case but that edge case might also be an approach that is considered.

I do like the suggestion that @vas makes about opting for something to be an extension vs core depending on whether it’s a feature that should be universal everywhere that uses CommonMark or if it’s something that a platform can opt to add on but, that may also not work if there are cases where it’s necessary for it to be part of the core for the functionality to work fully.

2 Likes
#9

I think maybe we have an oddball usage - unlike other platforms, spoiler syntax isn’t immediately available in an editor and never has been. I’m not actually sure why but it means that people who want to use spoilers have to really dig to find the syntax - now, that’s our own “fault” if you will but I do think there’s value in having a standard for it. If you’re a frequent user of a platform (and use their syntax regularly), you’re likely able to remember the syntax but if you’re more infrequently on the platform or using that syntax, you rely on it being standardized across several platforms or being able to find the syntax on the site.

There’s just something cool about being able to go between three platforms that all use CommonMark and … just having it work. We absolutely want that for stuff everyone uses all the time and the core set is that - it’s unlikely someone’s using tables or spoilers or page anchors frequently. But that infrequency might be an argument itself for having a single standard. :slight_smile:

That said, there are other options - I personally love the way block level hidden text works here on Discourse - the ability to collapse and expand the sections and give them a visible text summary is something I really like a lot and I don’t see a lot of platforms using.

1 Like
#10

The counter example is fenced code blocks, which are not supported by the original Markdown.

1 Like
#11

Fenced code blocks are an alternative version of the feature of code blocks. Whereas tables are a completely new feature. If the extensions are being defined by functionality (not syntax), the two variants of code blocks could be considered a single feature.

This was an observation based on what is already in the spec. If the spec is going to be expanded to include other commonly used features such as tables, a new line would need to be drawn to avoid feature creep.

1 Like
#12

I think we are in violent agreement :slight_smile:

1 Like
#13

I do like the ideas of having a hard and fast rule that we can apply to the main spec.

From where I sit, a given feature needs to be universally needed for it to be in the main spec. If it isn’t universal, we should consider having “commonMark-spoiler” or something as an extension. We should have the same rigor as with the main specification, but be clear on the scope and how it applies.

I.e. As the author of the linter that I am working on, I MUST include everything in the base spec that is not an “extension”, and can then choose which extensions to include.

The existing extensions in the main specification, in my opinion, meet that bar. Spoilers do not. Specifying extra information to influence the CSS when rendered does not.

Does that help?

1 Like
#14

Well, tables is next on our agenda from my perspective. How are you feeling about things these days @jgm – what can we do to help?

Could there be a release of CommonMark where we move tables to the core, and start digging into the corner conditions?