Embedded audio and video

It’s worth noting that Maruka is using the .mp4 file extension together with the Markdown image syntax for video.

https://leanpub.com/markua/read#leanpub-auto-resource-type-image-code-math-text-audio-and-video

Markua Processors rely on file extension to determine the type of media and must not attempt to parse files to determine their type. Because of this, the choice of acceptable file extensions for the various media types is a subset of the total available, so that audio and video files can be distinguished solely by their file extension instead of by examining the file or by requiring authors to type some special metadata syntax.

and

.mp4
The file is treated as an MP4 video.

and when describing the audio file extensions:

.m4a, .aac
The file is treated as an MP4 AAC (Advanced Audio Coding) audio file. Note that .mp4 is not supported as a file extension for MP4 AAC audio, since that is the file extension used for MP4 video.

This is essentially the same as what I suggested for CommonMark.

While Maruka isn’t exactly Markdown (it’s not aiming for backward compatibility - it removes some syntax from Markdown and adds new syntax), I think it would be wise to aim for some level of syntax compatibility. Markdown and Maruka are close enough to allow copy/pasting text between the two (with some modification).

Maruka? More markdown specs? Will this ever end lol.

I am itching to do libmagic-like header detecting in javascript. Should i attempt that route? I’ve done header parsing for detecting file type in Python , i think it should work fine for javascript fileobject , except we do not have mmap in javascript.

with that we can easily detect mime via file contents.

The CMS that I primarily work with serves all media (images, video, audio, pdfs, you name it) from requests that use the .asax extension. This is similar to a .php file serving any sort of data. If this extension is going to be useful, there needs to be a way to mark what type of media you’re serving.

My initial reaction would be to mark the type of media via the document fragment (I blogged about leveraging this for styling images with markdown), but it could conflict with existing document fragments used in SVG.

As this is being proposed as an extension, maybe it makes sense to use different symbols for the different embeds.

Audio could use @[](), and video could use ^[](). That way it’s up to the author to specify what they’re embedding.

1 Like

.mp4 does mark that the file contains video - conventionally at least. It could be a used for audio as well, but this seems like an edge case. So I’m not sure that a different start symbol is needed in practice.

The convention is what people will think of when they’re writing Markdown, rather than what is strictly defined.

I can see why you’d want to group these together, but picture is going to be difficult to include. Most of the time you shouldn’t use picture, you should use img with srcset. (CSS Tricks has a good write up, and this post is quite blunt about it.) Switching ![]() to produce <picture> tags isn’t the right thing to do for the handful of times it might be right for the author. The principle of least surprise makes me think ![](abc.jpg) should always produce an <img> tag.

1 Like

I like this idea! I think it works quite well with some of the other discussion, too. How does this sound:

Syntax for embedded media is like the syntax for links, with one difference. Instead of link text, we have a media description. The rules for this are the same as for link text, except that (a) a media description starts with ![ or !mediatype[ rather than [ (where mediatype is one of image, audio or video) and (b) a media description may contain links. A media description has inline elements as its contents. When an image is rendered to HTML, this is standardly used as the image’s alt attribute.

  • If the mediatype is image then the media is rendered as an image. When rendered to HTML it should produce an <img> element.
  • If the mediatype is video then the media is rendered as embedded video. When rendering to HTML this may produce a <video> element.
  • If the mediatype is audio then the media is rendered as embedded audio. When rendering to HTML this may produce a <audio> element.
  • If the mediatype is not specified, or is unrecognised by the renderer, then the renderer may attempt to determine the type of media by file extension or other means. If it does not, it should default to producing an image.

So that means that existing markup like ![](image.jpg) will continue to work, as well as items like ![](image.php). Markup like ![](music.mp3) can be rendered as audio tags, as can !audio[](music.php).

I like that the syntax is extensible, too: if there are types of embedded media that we haven’t thought of (or haven’t been invented yet!) then it can be incorporated into the scheme without rewriting the spec.

3 Likes

We at the Diaspora* community are discussing the syntax to use for audio/video embed. As a major markdown user, it is important for us to be in conformity with the Markdown specifications, practices and ecosystem. So we are interested in having an approved official extension to the standard for media embedding.

5 Likes

As Founder of http://phwa.be , and Original Author of!()[mediafile.ext] Embedded audio and video for markdown-it’s audio + video implmentation . I would like to get an approved official extension for embeddable media types.

At http://phwa.be we are going to support many other embeddable documents ( any renderable document tru html5 ) . Tru the same syntax !()[doc.webm/mp4/ogv/ogg/pdf/**etc**]

1 Like

Yeah, or if we had generic attributes we could do:

![my title](music.php){type=audio}

There was also some discussion in the generic directives thread as to whether something like this should be rendered as a block or inline element. The tentative conclusion was that :audio[my title]{src=music.php} should be rendered inline (even if the markdown processor doesn’t know the audio directive/extension it would generate a <span> which postprocessor or js could pick up), and with two colons it should be rendered as a block (a <div> if extension unknown):

::audio[my title]{src=music.php}

Do you really want to make a http call each time to check the document type so you know whether you should render an <audio> element or embed the containing document? I think @zzzzBov really has a point that there might not always be a file extension present.

1 Like

Looks like it’s time for me to recommend additional specialized post-processors again.

I’m not saying that we can’t or shouldn’t have a format for embedding audio and video, but consider this a suggestion for possible alternatives:

<embed type="audio" src="path/to/audio.ext">
<embed type="video" src="path/to/video.ext">
<embed type="table" src="path/to/data.csv">

or if you need multiple sources:

<embed type="audio" src="path/to/audio.abc, path/to/audio.xyz">

The post processor could grab these embed elements and convert them into the appropriate markup. Authoring becomes easier, the specs become simpler, and the features are more atomic and reusable outside of the context of markdown.

Why would someone want to do that? We only doing that in client side, you can check what we did here : https://github.com/cmrd-senya/markdown-it-html5-embed

there is no server-side request there. We did that by looking at fileext and determining mime .

Yes @zzzzBov , there are problems with missing file extensions , and my idea of using lib-magic style header detection is strictly clientside (Uploading file type, Blobs etc) .

yeah, but even if you do it client side, I think it’s suboptimal, e.g. if the files are remote hosted (or currently not available). At least allow for an optional {type=audio} or similar.

Why not just use the alt?

![mp3](path/to/a/mp3/file)

![](path/to/filename.mp3)

![mp4](path/to/a/mp4/file =640x480)

How does the parser know the user wasn’t intending to embed an image with alt="mp3"?

![=mp3](path/to/file)
![mp3=](path/to/file)
![mp3:](path/to/file)

or similar signal character with resource type identifier.

![audio/mpeg;version=1;layer=2](path/to/file)

Note that I didn’t verify the actual syntax for MIME type parameters.

ASCII art could also be used (bad adhoc designs):

![o_o](path/to/video_file)
![.|`](path/to/audio_file)
![o"](path/to/image_file)

… or Unicode emoji:

  • :movie_camera: U+1F3A5 :movie_camera:
  • :camera: U+1F4F7 :camera:
  • :video_camera: U+1F4F9 :video_camera:
  • :clapper: U+1F3AC :clapper:
  • :vhs: U+1F4FC :vhs:
  • :tv: U+1F4FA :tv:
  • :dvd: U+1F4C0 :dvd:
  • :cinema: U+1F3A6 :cinema:
  • :radio: U+1F4FB :radio:
  • :musical_note: U+1F3B5 :musical_note: ♩♪♫♬
  • :notes: U+1F3B6 :notes:
  • :microphone: U+1F3A4 :microphone:
  • :musical_score: U+1F3BC :musical_score:
  • :art: U+1F3A8 :art:
    *:film_strip: U+1F39E :film_frames:
    *:film_projector: U+1F4FD :film_projector:
    *:framed_picture: U+1F5BC :gallery:
    *:paintbrush: U+1F58C :paintbrush:
1 Like

For missing extension name, use consistent attributes syntax

A more consistent approach to missing extension name ( missing .mp3) is to just use the currently discussed consistent attributes syntax like:

![ cat meowing ]( example.com/meow ){ mediatype=audio/mpeg }

http://talk.commonmark.org/t/consistent-attribute-syntax/272


Why/Purpose of type field?

Purpose of the type field. The purpose of the type field is to indicate to the renderer how to deal with missing content. E.g. If it cannot find meow.mp3, then display a missing audio icon or show ::warning: Audio File Missing:: . If no type is specified, it will try to infer by extention name or mediatype.

e.g.

![ cat meowing ]( meow.mp3 ){ type=audio }

or

!audio[ cat meowing ]( meow.mp3 )

And hey, if the viewer program is smart enough, it could have a button saying "Search online for similar audio files for cat meowing"

3 Likes

I agree with Consistent attrib syntax.

3 Likes

While I understand that assuming .mp4 = video/mp4 may occasionally produce unexpected results, the times when this assumption is not accurate appear to be an edge cases. So perhaps consistent attribute syntax could be used to cover just the edge cases. The problem with requiring consistent attribute syntax is that it is an advanced feature and more difficult to remember for general writers. By following the convention, .mp4 = video/mp4, we allow users to simply write ![](video.mp4) and get the results that, in the majority of cases, they would expect. For edge cases they could use consistent attribute syntax to specify the content type, or raw HTML if the consistent attribute syntax extension is not enabled.

As an addendum to the proposed syntax, I suggest using the full source type (e.g. “video/mp4”) for consistency with HTML. Perhaps the order could be significant if multiple video sources are defined. So the following Markdown:

![](video.mp4 video.webm){type=video/mp4,video/webm}

Would render as:

<video>
  <source type="video/mp4" src="video.mp4">
  <source type="video/webm" src="video.webm">
</video>

There was also mention of using the contents of the square brackets for representing the media type. I disagree with this because it goes against the priniciple of uniformity (or at least the philosophy behind it). I suggest that square brackets are also used for alternative text for audio and video. E.g.

![Your browser does not support the video element.](video.mp4 video.webm)

Renders as:

<video>
  <source type="video/mp4" src="video.mp4">
  <source type="video/webm" src="video.webm">
  <p>Your browser does not support the video element.</p>
</video>
2 Likes

btw I thought that the [] in ![]() is for alt description for blind users etc…?