This sounds like an implementation issue, but strangely, most (or even all?) CommonMark parsers seem to agree on that: http://johnmacfarlane.net/babelmark2/?normalize=1&text=[a](<b)c>.
To me, it looks like the question is if
<b is a valid link destination.
Spec 0.27 says:
A link destination consists of either
a sequence of zero or more characters between an opening
<and a closing
>that contains no spaces, line breaks, or unescaped
a nonempty sequence of characters that does not include ASCII space or control characters, and includes parentheses only if (a) they are backslash-escaped or (b) they are part of a balanced pair of unescaped parentheses that is not itself inside a balanced pair of unescaped parentheses.
… and I would say that it totally fits the description of the second point, doesn’t it?
So why does no CommonMark implementation in babelmark2 recognize it as such?