I have started work on making libcmark extensible, see https://github.com/jgm/cmark/issues/100 for the (pretty long) discussion. My test case / use case for this is tables, it seems it’s something a lot of other people want too, and I was made aware of that escaping problem through Parsing strategy for tables? .
My humble opinion on that is that as it seems accepted that block level rules should take precedence over inline rules, the correct way to approach the issue is to match lines with table row rules by ignoring all backslash-escaped pipes. If a line matches and a table row block is created, the backslash should be removed before parsing inlines. Thus:
| A cell `\|` with a pipe | another cell |
should be interpreted as a table row with two columns, the content of the first cell being translated to:
A cell `|` with a pipe
before inlines in it are parsed.
That’s the behaviour my test extension now implements at https://github.com/MathieuDuponchelle/cmark/commits/extensions_draft_3 .