After some hacking, I managed to demux and play video streams directly from .mp4 files. An updated demo is available at:
Warning, the video file needs to be completely downloaded before playback can begin, so it's a bit slow at first.
Moreover, Broadway.js now supports WebGL and background decoding in web workers. Decoding in a worker is still a bit slow due to the overhead of passing decoded video frames via postMessage, but hopefully that should go away once array buffer transfers land.