Requiring a blank line between a paragraph and list

#6

Can you please edit your posts to spell “Babel” correctly? There’s a reason I posted the wikipedia article above.

#7

Supply-side stats like those in the Babelmark table are, of course, easier to obtain, but the demand-side figures would be more useful in the design of high quality tools for users.

It would be interesting to know:

  • What proportion of edit events involve insertion of a line first time ?
  • and what proportion involve subsequent editing to add a further line ?

A graphical model weighted with that kind of data would allow more intelligent behaviour than a crude normative rule-base.

1 Like
#8

Speaking just as a user, and having posted this as a bug in Discourse, I can say it is really annoying to require a blank line in this context. Its very clear what the user intention is if they write:

These three things:
* one
* two
* three

Having that produce anything other than a list is a bug IMNVHO. It also is substantially ugly in the text to add a blank line there, and since the text may be being displayed as plain text in other formats (eg plain text email), this is definitely a problem.

The fact that nineteen implementations require it most likely says more about the testing of those implementations than any real intent.

2 Likes
#9

Hmmmmmm… wonder if : at the end of a line always signify ‘next paragraph’ or list. And if so, can there ever be a situation where it is still a new paragraph/list/etc… without : at end of line.

#10

Yep, this is the most common mistake I see people making who are new to markdown as well.

I have never noticed it implemented any other way in practice, so I’ve gotten used to it.

The topic linked above has a number of legitimate reasons not to consider line breaks as new paragraphs by default in general, so its worth a read.

If lists could be special without any major pitfalls I think it would better meet user expectations.

especially when the "treat line breaks as paragraphs option is used"
like
this.

#11

Thanks Peter, I absolutely understand this, and value a user point of view. But in doing language design, I think it’s important we establish a clear priority between two potential goals for CommonMark, that may frequently be in conflict moving forward if we do not address them now:

  • The goal to be “as compatible as possible”

  • The goal to resolve minor annoyances and/or frequent mistakes.

I believe that both these goals can be accommodated, but only thorough formalization of the concept of flavors and versions. These concepts may be a little fuzzy because they are not yet well defined in the CommonMark context, but I don’t think they will be onerous to implement or use.

But right now, without flavors and versions I believe we should put design-for-compatibility above design-by-annoyance, or risk CommonMark becoming just another YAMF and making the real problem that lead to CommonMark’s creation (the lack of user and document portability between platforms) worse.

Today, CommonMark is not final, and getting from here to there may look bit like making sausage. To people who fall closer to the user role than the implementer role, some of this process needed to refine the specification may seem unnecessarily ugly, but I hope you’ll trust me when I say that “clear intent” becomes anything but clear when attempting to parse natural languages in production quality code.

1 Like
#12

When the domain of discourse is this constrained (a few elements of document structure), and the set of tokens this small (newlines, prefixes, inline emphases etc), the challenge is really not great, especially if some data on user habits has been collected.

1 Like
#13

It is true that most implementations require the blank line. (And, although the requirement is not in John Gruber’s syntax description, this issue was explicitly discussed; see the links here for the relevant history.)

There is a serious issue here about how heavily backwards compatibility should weigh. I must say, it would weigh much more heavily for me if there were a chance of avoiding fragmentation. But John Gruber has made it pretty clear that there is not. (I did ask him whether he could get behind this effort if we removed divergences like this, but there was no uptake.) If we are to be working on CommonMark instead of Markdown, then perhaps it is a good opportunity to rethink some things that may have been ill advised in the original.

The first observation to make is that, no matter which way we go on this, there will be inputs that will not behave as their writers intended. However, I think these will be much more common if we do require the blank line than if we do not. Burt_Harris’s original example is not as ordinary a piece of text as, for example:

TODO:
- fix bugs
- write documentation

Either way we go, there will be some inputs where manual escaping is necessary to get the intended interpretation.

Now, it seems to me that there is a strong reason for NOT requiring the blank lines. Even in Gruber’s Markdown, blank lines are not required inside a list item. So, this gives you a list item containing a paragraph followed by a bullet list:

1.  Foo
    - Bar
    - Baz

CommonMark uses the principle that the meaning of a block of text should not change when it is put into a list item. It follows, then, that this should be a paragraph followed by a bullet list:

Foo
- Bar
- Baz

(Indeed, the way the spec for list items is written requires that this property hold.) I think this is the least surprising behavior for users. Hence, I conclude, either we should require the blank line between a paragraph and a list everywhere – even in sublist contexts (which is what reStructuredText does) – or we should require it nowhere.

5 Likes
#14

Based on this discussion, and the cited examples/data, I am persuaded that we should support requiring a blank line nowhere.

That is, I think

para

  • foo
  • bar

should work as is (e.g. that should be a <ul> list, above)

6 Likes
#15

I’m not sure I follow your logic here, @jgm. I’m also not sure you’ve followed my reasoning on the other thread. I’ll summarize it here:

Yet, even after the renaming, the web site continues to prominently display the former phrase, despite the fact it’s a clear violation of the third bulleted clause of the Markdown license. In my mind, it is quite possible that this issue, one of lack of respect for the terms of the license, and not issues of compatibility or fragmentation that have blocked John Gruber’s “getting behind” this effort, and continue to do so.

But in my mind, the chance of avoiding (further) fragmentation has nothing whatsoever with getting John Gruber’s support or agreement. Gruber has held the Mardown.js implementation’s syntax stable for a long time, thus he’s not the one causing fragmentation.

The fragmentation problem is also independent of naming. It is caused by people deciding to do things differently in their own implementations, or lacking the skill and discipline to get it right. I trust it’s not the later with you.

Yes, I admit that. So lets use your example or @codinghorror’s example and view them into Babelmark 2’s Preview view (rather than worrying about the exact HTML generated.) This much simpler example generates only two visual representations:

Markdown.pl (the only Markdown implementation), and 17-other YAMF implementations generate the one that looks like this:

TODO: - fix bugs - write documentation

While only 4 implementations display it as a bulleted list, including stmd 0.1 which I suggest shouldn’t be counted. So if you really like that 18:3 minority opinion, I’d suggest renaming this whole effort YAMF as others have suggested.

The problem with oversimplifying test cases

I understand why you simplified my test case, but I think you went too far removing the non-blank line before the start of the when you started to talk about nested lists. What if you restore it…

Is the following a list or part of this paragraph?
1.  Foo
    - Bar
    - Baz

Plugging that into Babelmark 2, the range of variations is greater, but most of the implementations I would consider high-quality (including Markdown.pl and Pandoc) display results that don’t format any of it list items (nested or not). stmd is (again) almost unique in its interpretation,

So now trying to following your logic about the “principal” of CommonMark, I remove the first four characters of all but the first line, and get back to something that behaves just like the TODO list one.

1 Like
#16

About the test case, my point is that the behavior you find objectionable in stmd in reproduced in all markdown implementations in list contexts. Try this in BabelMark2:

1.  *ax*<sup>2</sup>
    + *bx*
    + *c*
    = 0

stmd is more consistent, in that the content is interpreted the same way in a list context and outside of one. That seems to me an important feature, from the point of view of user surprise. It also permits a clean description of list item syntax.

Note also that Markdown.pl does allow block quotes to start without an intervening blank line. Why are blockquotes treated differently from lists? Try this in BabelMark2:

*ax*<sup>2</sup>
+ *bx*
+ *c*
> 0 - *ax*

It’s very similar to your original example, but you won’t find Markdown.pl doing what you intended here.

The fact is that cases like this, where ordinary text lines begin with + or - or >, are fairly rare. Making sure they behave the same way as with Markdown.pl is not, in my view, worth the cost in simplicity and coherence – not to mention the myriad realistic cases like the “TODO” example where stmd will render according to the writer’s intent and Markdown.pl will not.

And yes, Gruber has caused fragmentation. For a decade now, implementers have come to his markdown-discuss list asking for guidance on corner cases and bigger issues, and he has responded with silence. That is a major reason why there are divergences of the sort displayed by BabelMark2. If he had exhibited leadership, saying, in response to our queries, “here’s the precise rule for indented list nesting,” and clarifying his syntax description to make it clear, then there would have been no need for this effort.

2 Likes
#17

Perhaps the philosophy of commonmark should be less about what every other parser is doing.

And more about, what would be the Common Expectation of users when writing markdown.

In other words. The aim should be more about a Lightweight Markup Language of No SurprisesTM .

This is because the biggest attraction that made iphone beat most other more feature rich competitors, is that it ‘just works’ from the average user experience.

tl;dr: Be common in user expectations. Not in the Status Quo

2 Likes
CommonMark should have a philosophy
#18

Well said. I wish that that could be true. Unfortunately, surprises have too much to do with context and previous experience. For example, with people saying things like:

For those who’s context includes using (and accepting) Markdown, changing the choices that Gruber made is surprising. For others, (perhaps newbies or those with a YAML background) the interpretation of Markdown is surprising. (Personally, I have no problem with Markdown’s interpretation this because I keep in mind that YAML Ain’t Markup Language.)

#19

Unless Gruber is the author or principal actor of Genesis 11, that seems a remarkably implausible hypothesis to me.

Divergence in requirements and applications is enough to account for the most significant divergences of idiom, and more social processes of distancing and identification explain much of the rest.

The three light markup applications which I use most would all be broken by the adoption of the current CommonMark. Not because the Ur-spec was too loose, but because they have been shaped by more specialised needs, like nested tables (MMD) and outlining (FoldingText).

Spanish is gaining demographic ground on English now in North America. Not, I suspect, because English grammar lacks a formal specification … (or do you think the tide could be turned by a better grammar book ?)

Babel will always be with us. Technical specialisation, and in-group out-group identification and distancing are the perennial drivers of divergence. Throwing verbal stones towards Gruber (and deprecating features of his dialect) is the very essence of the Babel story at work …

#20

A (now-fixed) bug report I just filed on the spec makes me wonder whether you’ve come down on the wrong side here. I think the choice of examples on this post are weighted towards the decision you made, but should this really be interpreted as a list inside a paragraph:

[Hexadecimal entities](@hexadecimal-entities)
consist of `&#` + either `X` or `x` + a string of 1-8 hexadecimal digits
+ `;`. They will also be parsed and turned into the corresponding
unicode codepoints in the AST.

Perhaps a special case needs to be made for ‘list item(s)’ inside a paragraph that don’t look like lists, eg don’t have any further indentation and aren’t followed by a blank line, but on the other hand, perhaps that adds too much complexity. Either way I think it may be worth reconsidering.

#21

As I said in the posts I linked to on your issue, the principle of uniformity is the main thing that sways me towards allowing lists to interrupt paragraphs. The meaning of a text should not change when you put it in a list item.

So, if this is not a list:

Words:
+ bar
+ baz

then it should not be a list when you put it in a list item:

1.  Words:
    + bar
    + baz

Many implementations break this principle of uniformity.
But it’s important to keep it. Indeed, it’s required by the way the spec for list items is written.

I’d be even more happy to require the blank line in both contexts – a top-level paragraph and a paragraph in a list – so that you needed to write, in the second example:

1.  Words:

    + bar
    + baz

reStructuredText imposes a requirement like this. But it would be a pretty major backwards-compatibility breakage, so I don’t think anyone considered it seriously.

2 Likes
#22

There’s no question that lists should interrupt paragraphs, so the final resolution of this topic is IMHO correct.

This pattern is one I see users typing, almost literally every single day in Discourse:

Stuff
- item
- item

And they absolutely (and reasonably) expect that to be a paragraph followed by a list. Now with CommonMark it will be!

#23

I see your reasoning, and don’t doubt that this pattern is indeed “common practice”—even thought I wouldn’t use or recommend it, if only for the simple reason that it is highly “non-portable”.

However, allowing any line that “looks like” a list item to interrupt any old paragraph in the CommonMark syntax, for the sole purpose that:

Now with CommonMark it will be!

seems a bit far-reaching in my point of view: in particular it introduces a whole range of possibilities to “inadvertently” have list items recognized inside a paragraph where none are intended—I think this is a well-known problem in cases like:

Stuff that I
- nor anyone else -
would never write in the
1. place.

Situations like this (admittedly contrived) example do occur in practice when paragraph text gets re-formatted. I for one have turned to using NBSP to make it clear (for the formatter, and for CommonMark) that “-~” and “1.~” are “just text” and in no way list item markers [the tilde “~” here stands for my use of NBSP]. But I don’t think this is a common way of writing CommonMark, nor that it should be.


Is there any chance to “tighten” the syntax rules in a way that

  1. your example is still recognized as a list, but
  2. my example is not?

Maybe by recognizing that your example matches the pattern

  • the first line in the block is a “vanilla line”,
  • each subsequent line is a “list item” line,

while mine does not?


Or would it be too “extreme” to require additional markup for your example to work, like (off he top of my head and for illustration only):

:Stuff
- item
- item

or, as had been mentioned elsewhere, maybe detect a COLON at the end of the first line in a block, and use this to “enable” list items in your example:

Stuff:
- item
- item

This would obviously much simpler to implement (and to explain, IMO!) that the “pattern-matching” convention above.

And a “COLON at the end of the first line in a block” is, I think, unlikely to be left there inadvertently as a result of re-flowing a paragraph (though not impossible, but in any case easier to see and to fix than having to eyeball every line start in the re-flowed paragraph …).


Any thoughts on this (old topic!)—or is it simply “just too late”?

Blank lines before lists, revisited
#24

Sorry, you are arriving extremely late to this party. Perhaps you can focus these walls of text on the things that have not been definitively decided? :wink:

#25

Sorry. I thought, based on the wild variation that Babelmark shows in this case, that this matter was not definitively decided—at least not in the wider Markdown realm (the “compatibility argument”).