After giving some more thought on this, here are my conclusions…
The way fenced code block are handled is not practical in a live context like an editor because we can force reparsing files of possibles tens of thousands of lines unlike any programming languages where a file hardly ever goes beyond 2000 lines (in which case it’s a good indication there is too much there, but that’s another subject…)
So the use case for Markdown is to be able to handle big files fast (novels, technical documentation, etc…) As anybody can read in the README of the C reference implementation, it can parse War And Peace in 127 milliseconds. It’s fast, but not enough, and the War And Peace file is basically just text, there is no Markdown formatting in there, which, I suppose, would surely slows things down.
So, to come back to your options (John MacFarlane), option A could not be an option because of the speed required to compute properly in a live environment, and since computer chips does not seem to improve a lot in terms of speed, I think going forward with option A would severely limit editors applications.
Option B, is more interesting but it has also severe limitations, one of which is that since there is no specific syntax for open (with the exception when it’s added language) and close fence, any fence can match with any other. It adds a lot of complexity to the parsing strategies and a lot of state have to be kept to handle things properly. So I wouldn’t go that road either.
The only option left is my last proposal but I think it may be too restrictive. It also remove the possibility to use library like highlight.js to parse markdown code inside code block. So I would suggest to allow markdown inside a fenced code block if the open fence is specifically marked with markdown. Otherwise, I would close the fenced code block like in the previous proposal: end any fenced code block when encountering a Markdown valid explicit block (excluding paragraph), like a header, a blockquote, or anything that makes sense, basically, to be defined.
So, the only case left is when we don’t close a fence specifically marked with markdown language. I would handle this case as the reference implementation does right now. And I think applications developers can live with that.
So both code below would be parsed as fenced code blocks:
``` markdown
# header 1
```
parsed as:
# header 1
and:
```
func isEven(number: Int) {
...
}
# header
would be parsed as:
func isEven(number: Int) {
...
}