Unicode Character 'BULLET' (U+2022)

I personally like the idea - it is more simple than the original list syntax, but it doesn’t seem to be convenient to me to insert unicode symbols.
Maybe • could be substituted by + sign?
So you could write:
+Hello
+World

Or even by * sign:

*Hello
*World

Looks more readable to me.

The current spec allows +, - and *. This works already with pretty much all implementations. My point was to add the bullet character to this list, not to replace.

In my opinion, it is not relevant for a standard, if it is inconvenient for some people. For many it is.

And even if it was inconvenient for all, that would be still irrelevant. More important is the following question:

What happens when this is not implemented?

Answer:

Markdown destroys a plain text “bulleted” (literally) list by placing the whole list in one line.

2 Likes

Never mind, my proposition was to remove space between + (-, * - whaterver) and an item. So that you write +Item,not + Item. I’m not so sure about that now, though. It’s a little easier to parse IMO, but maybe less readable.

Wikipedia – Compose key has more information, for others who didn’t know about it.

Some people have 21st century keyboards, and they will become more prevalent.

The original point remains unaddressed; for those who do use a bullet, the CommonMark output mangles the input.

Even if so, what’s wrong with making CommonMark work well for them?

1 Like

The point of markdown is also to use common characters available on the vast majority of keyboards to derive it’s syntax from, making it ‘easy’ to do formatting and such. I understand that your keyboard supports it easily enough, but most people’s don’t…

So one reason for not having it in the spec as a list marker is because people who can’t easily input the bullet (which is a lot of people at least right now) might be presented a situation where the markdown document they have received uses bullets. So they’d either have to change all the bullets to what they can easily use, or difficultly input bullets if they want to edit it, making it inconveniant, hard and therefore against the philosophy of markdown for them (which = most people). They’d need to do that because different list markers indicate a new list, so they can’t just mix your bullet with their preferred list marker, be it -, + or *, between different list items.

I thought the same with emacs and grid tables. Where emacs relates to your keyboard, and grid tables relate to the bullet character.

What’s a 21st century keyboard?

3 Likes

I didn’t say anything about my keyboard. The discussion is about a standard.

I doubt that I really have to explain what I meant by that.

I acknowledge that your accessibility/lowest common denominator argument is a good one that I hadn’t considered. I tried to come up with a rebuttal but couldn’t … kudos to you.

At the same time almost all new programming languages accept Unicode. More and more people will be receiving programs and other documents containing Unicode characters and will have to deal with them. This will eventually lead to everyone having easy ways to input them. When that happens, this issue can be revisited. But I think you give a compelling argument for why now is not the time.

Sorry if it came off in the wrong tone, it was a genuine question! I was thinking it was some type of new keyboard which allowed easier unicode character input or something. :smiley:

I had (wrongly) assumed your references to a 21st century keyboard was referring to your keybaord too. Sorry.

I do not think that U+2022 should be added to the list of special characters for bulleted list interpretations.

U+2022 is a Unicode character. Markdown, however, is agnostic to character set: neither Gruber’s original specification nor CommonMark say anything about which character set Markdown content is to be authored in. Control characters are all drawn from the POSIX Portable Character Set, i.e., US-ASCII (without assuming particular scalar values for particular characters). Gruber’s Perl script also does not assume particular encodings–what you put in is what you get out. It should stay that way.

1 Like

I wish that would work. But it doesn’t. When I enter a bulleted list using the unicode character, markdown turns it to

<p>• Hello
• World!</p>

Which is rendered by the browser to

• Hello • World!

That means input is not equal to output. Therefore, it can not stay that way as you also required and the bullet should be added to the list of markers.

Additionally, adding U+2022 to the list of list markers to the common mark spec does not harm those, who do not use any unicode.

If I copy and paste bullet list from Microsoft Word into my plain text editor, I also get the Unicode bullets.

So, to summarize. The U+2022 Unicode bullet character:

  • is standardized by Unicode;
  • is used by some users to create a correct plain text list;
  • is easily typed by OS X users;
  • is pasted when a list is copied from Word (and possibly other text editors and browsers).

Advantages of adding the U+2022 Unicode bullet character support to Markdown:

  • Markdown no longer messes up seemingly correct lists that use this character.

Disadvantages:

  • Markdown has to add support for a non-POSIX Portable Character;
  • therefore implementations have to recognize Unicode;
  • support is currently not present in other variants.

In my opinion the disadvantages (that I could come up with) don’t outweigh the advantages. Adding a standardized character from Unicode (the most comprehensive, most standardized, most widely used character ‘set’) is not a downside to me. Users seeing their list getting messed up when copied from Word or typed with the standardized bullet character is a much bigger issue.

So, I support adding the U+2022 Unicode bullet character to CommonMark.

7 Likes

Those may not be the only ones. Implementations would need to:

  • support Unicode parsing, which, depending on the tools you use, can be cumbersome (e.g. javascript in the browser)
  • know (or infer, but that may not be acceptable) the encoding of the input text, whereas it can currently just deal with ASCII chars and ignore others with a lot of very common encodings (e.g. UTF-8, ISO-1559-*…)

As for the rendering of a list bulleted with , you can add two spaces at the end of the lines so that it would still get a line break. While I get that it is not what you want, you can get your “output ≥ input” with only a minor inconvenience.

3 Likes

Wouldn’t implementations need to understand the encoding to be able to parse a document at all? For example, UTF-16 is completely different from UTF-8. Even with single-byte character encodings I’m quite sure byte values below 128 (thus US-ASCII) are also used as part of byte sequences that are valid code points in other encodings. So, ignoring all encoding and treating all text as US-ASCII wouldn’t work.

As implementations have to determine which encoding they’re reading, you might as well expect them to support UTF-8 (of all possible encodings), and therefore Unicode.

1 Like

Let’s compare this statement with this one:

I would say, a parser, which chooses not to support unicode, can simply ignore the unicode characters.

I also think it is not wise for a parser not to support unicode. All modern operating systems, libraries etc. support it. I can’t imagine why someone would want to write software that doesn’t support unicode.

Not supporting unicode means excluding all those who write something in any other language then English. I don’t think a spec like this can call itself “standard” or “common” by excluding the majority of text in the world.

No. This is not a minor inconvinance. This is a major annoyance. One should not force the user to do anything at all just to prevent markdown from messing the text up as I have shown in my original example.

In addition, I agree to the below quote:

I see two options to ensure this:

  1. Adding U+2022 to the list of list markers
  2. Getting rid of the two spaces. See the discussion about line breaks.

No, I mean that the parser can skip the non-ASCII characters, since it would be a given that those are not significant in the syntax. Those characters would still be outputted, of course.

I’m talking about the significant, parsed bits. If you use, say, regexes to parse them, then you need proper Unicode support in your regex engine. That might not be the case, even though your parser still read your intput correctly.

1 Like

Not necessarily. A simple parser that would match significant markers on ASCII chars and blindly output all other bytes would sill work fine with one-byte encodings like ISO-1559-* and one-byte-or-non-ASCII encodings like UTF-8.

Admittedly it won’t work with UTF-16, though. I’m just saying that including Unicode markers raises the bar for implementations.

Because text is not HTML, you have to make concessions, like requiring two spaces at the end of a line to get a line break, otherwise you cannot have a well-formatted text file OR you get line breaks every other sentence in the output. Markdown already works like this in almost all implementations, except for some that are not destined to be used with text files (e.g. embedded in an app or website).

So when you say “it doesn’t [work]”, you mean that, like most markdown implementations today, it doesn’t work for this quite specific case of both using the bullet and not wanting to append those two spaces.

Consider the alternatives:

  1. Loose the encoding-agnosticism – backward-incompatible in a major way plus lots of drawbacks
  2. always breaking lines – backward-incompatible in a major way, loosing the ability to have clean text files

I think not supporting as a list item marker it is the lesser important annoyance.

2 Likes

Like instanceofme said:

CommonMark (as I understand it) is supposed to smooth out differences between all of the implementations out there, and smooth out the ambiguities in Gruber’s writeup. Its purpose is not necessarily to add new features. Regardless of the character set issue, it looks and smells like a new feature.

4 Likes

Fenced code blocks are old and list items are new?

1 Like

No.

With • (and the Unicode standard), yes.