Spec 0.16 release

I’ve tagged 0.16 in the repository and catalogued the changes to the spec and reference implementations:

The spec changes are fairly minor, but there are many improvements to the implementations.

The JavaScript implementation is now much faster. The AST has been changed to use a doubly linked list structure, rather than JavaScript arrays. This has allowed us to eliminate recursive algorithms, so that deeply nested structures can be rendered without stack overflows. We also now provide a “tree walking” iterator that makes it easier to modify the AST between parsing and rendering.

Both the C and the JavaScript implementations now provide an XML renderer, which produces an XML representation of the CommonMark AST. (See CommonMark.dtd for a description of the format.)

A --sourcepos flag will insert source position information into HTML tags.

See the changelogs for a more exhaustive list of changes.

1 Like

Thanks!

Example (interact) looks broken?

Example (interact) looks broken?

Thanks for letting me know. Fixed now!

You’re talking about the C library, libcmark?

There were some recent changes in the iterator API, yes.
See https://github.com/jgm/CommonMark/pull/277

Does that help identify the problems you’re having?

+++ James Adam [Jan 16 15 02:37 ]:

Yea, think I’ve got it narrowed down to cmark_iter_next(). It seems to be causing the occasional SIGABRT.

Note: the recent PR I pointed you to changed the way iterators work. Previously the iterator advanced to the next node when cmark_iter_get_node was called; with this change, it advances earlier, when cmark_iter_get_node is called. So, make sure your calling code isn’t assuming the earlier behavior.

In any case it would be helpful to see an example of the code (even if it’s in go) that causes the problem. (But, better if you could give an example in pure C, or spot a problem in the C code.)

(@nwellnhof should also be brought in to the loop here.)

+++ James Adam [Jan 16 15 12:54 ]:

Wow, I’ve been troubleshooting this and I don’t think it has anything to
do with the iterators, or even libcmark. I think I’ve stumbled upon
some weirdness in the Go runtime/GC.

I create nodes like so:

//Creates a new node of the specified type
func NewCMarkNode(nt NodeType) *CMarkNode {
     n := &CMarkNode{
         node: C.cmark_node_new(C.cmark_node_type(nt)),
     }
     runtime.SetFinalizer(n, (*CMarkNode).Free)
     return n
}

And set a finalizer to give a hint to the Go runtime to call my Free
function when a Node falls out of scope (My free just does a null check
and then turns around and calls cmark_node_free). I’m getting random
errors / invalid pointer / etc. in cmark_node_free as my tests finish
and Go is stopping, caused by that finalizer firing (but only sometimes,
because the GC doesn’t necessarily act on hints if it doesn’t want to).
My Iterator unit test just happened to be last in line. So I can only
guess that something wonky is happening to memory as the Go runtime is
coming down.

As I recall, I also just upgraded to Go 1.4 recently (which, if memory
serves, involved substantial changes to the runtime), so I guess I
oughta bring this behavior up with the Go guys (if they care).

Anyway, thanks :slight_smile:

Interesting. Well, come on back here if the go developers tell you there’s a problem in libcmark. At this point it seems not!

Making the Go finalizer unconditionally free the C library’s node can’t work. This will delete a node from the tree whenever the last Go reference to this node goes away. I would suggest something like this

  • Add a pointer to the root Go node of a tree to every node. This makes sure that Go’s GC keeps the whole tree alive even if only references to child nodes remain.
  • Make sure to initialize this pointer whenever a a new Go node is created and to update this pointer whenever the tree structure is modified.
  • Only free the C node if the root node is finalized.

@nwellnhof – That makes sense, thank you for pointing it out. And it seems obvious now, of course :neutral_face: .

None of my unit tests would run into this problem because none of my nodes would fall out of scope before the test completes. I’ll have to write another test to specifically check for this problem.

I added that finalizer as a ‘just in case’ I forget to call Free() on a node, but if it’s going to be a hassle to deal with, it may be easier (i.e., less error-prone) to just do without it and make sure I remember to call Free() when I need to.

As a side note, the recently released Go 1.4.1 update seems to resolve my original issue.