How to Stream Video to HTML5 Video Tag Using WebSocket: Node.js & Socket.IO Example with MediaSource

Table of Contents#

Prerequisites
Project Setup
Preparing the Video File
Building the Node.js Server
Client-Side Implementation with MediaSource API
Testing the Application
Troubleshooting Common Issues
References

Prerequisites#

Before diving in, ensure you have the following:

Node.js & npm: Install from nodejs.org.
Basic Knowledge: Familiarity with JavaScript, HTML, and Node.js.
Video File: A short video file (we’ll use MP4; see Preparing the Video File for details).
Tools: A code editor (e.g., VS Code) and a modern browser (Chrome, Firefox, or Edge—MediaSource API support is required).

Project Setup#

Let’s start by setting up the project structure and installing dependencies.

Step 1: Initialize the Project#

Create a new directory and initialize a Node.js project:

mkdir websocket-video-stream
cd websocket-video-stream
npm init -y

Step 2: Install Dependencies#

We’ll use:

express: To serve static files (HTML/JS) and create an HTTP server.
socket.io: To handle WebSocket communication.

Install them via npm:

npm install express socket.io

Step 3: Project Structure#

Organize your files as follows:

websocket-video-stream/
├── server.js          # Node.js server code  
├── public/            # Client-side files  
│   └── index.html     # HTML/JS for the video player  
└── videos/            # Video file to stream  
    └── input.mp4      # Fragmented MP4 (see next section)

Preparing the Video File#

The MediaSource API requires video files to be in a streamable format (e.g., fragmented MP4, or fMP4). Unlike regular MP4s (which store metadata at the end), fragmented MP4s split the video into small "fragments" with metadata interleaved, allowing the client to decode chunks as they arrive.

How to Fragment an MP4#

Use MP4Box (a tool for manipulating MP4 files) to fragment your video:

Install MP4Box:
- On macOS: brew install mp4box
- On Windows/Linux: Download from GPAC’s MP4Box page.
Fragment your video (replace input.mp4 with your file):
```
MP4Box -dash 1000 -frag 1000 -rap input.mp4 -out videos/streamable.mp4
```
- -dash 1000: Creates 1-second segments (for DASH, but also fragments the MP4).
- -frag 1000: Ensures fragments are 1 second long.
- -rap: Ensures each fragment starts with a random access point (keyframe), critical for streaming.

The output streamable.mp4 in the videos/ folder is now ready for streaming.

Building the Node.js Server#

The server will:

Serve the client-side HTML/JS.
Accept WebSocket connections via Socket.IO.
Stream the fragmented MP4 file to connected clients in chunks.

Step 1: Server Code (`server.js`)#

const express = require('express');
const http = require('http');
const { Server } = require('socket.io');
const fs = require('fs');
const path = require('path');
 
// Initialize Express and HTTP server
const app = express();
const server = http.createServer(app);
const io = new Server(server, {
  cors: {
    origin: "*", // Allow all origins (for development; restrict in production)
    methods: ["GET", "POST"]
  }
});
 
// Serve static files from the "public" directory
app.use(express.static(path.join(__dirname, 'public')));
 
// Video file path (use the fragmented MP4 we created)
const VIDEO_PATH = path.join(__dirname, 'videos', 'streamable.mp4');
 
// Handle WebSocket connections
io.on('connection', (socket) => {
  console.log('Client connected');
 
  // Create a read stream for the video file
  const readStream = fs.createReadStream(VIDEO_PATH, {
    highWaterMark: 64 * 1024, // 64KB chunks (adjust based on network)
    flags: 'r'
  });
 
  // Track if the client is ready to receive data (to handle backpressure)
  let clientReady = true;
 
  // Pause the stream initially; resume when client is ready
  readStream.pause();
 
  // When the client signals readiness, resume streaming
  socket.on('ready', () => {
    clientReady = true;
    if (!readStream.isReadable()) return;
    readStream.resume();
  });
 
  // Send video chunks to the client
  readStream.on('data', (chunk) => {
    if (!clientReady) {
      readStream.pause(); // Pause if client isn't ready
      return;
    }
 
    // Send chunk as binary data
    socket.emit('videoChunk', chunk);
    clientReady = false; // Wait for client to signal "ready" before next chunk
  });
 
  // Handle stream errors
  readStream.on('error', (err) => {
    console.error('Stream error:', err);
    socket.disconnect();
  });
 
  // Cleanup on stream end or client disconnect
  readStream.on('end', () => {
    console.log('Video stream ended');
    socket.emit('streamEnd');
  });
 
  socket.on('disconnect', () => {
    console.log('Client disconnected');
    readStream.destroy(); // Stop the stream if client leaves
  });
});
 
// Start the server
const PORT = process.env.PORT || 3000;
server.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
});

Key Server Features:#

Backpressure Handling: The server pauses the video stream if the client isn’t ready (via the ready event from the client), preventing buffer overflow.
Binary Streaming: Chunks are sent as raw binary data (Node.js Buffer), which the client converts to Uint8Array for the MediaSource API.
Cleanup: The stream is destroyed if the client disconnects, avoiding resource leaks.

Client-Side Implementation with MediaSource API#

The client will:

Connect to the WebSocket server via Socket.IO.
Use the MediaSource API to decode and play video chunks.
Signal the server when it’s ready for the next chunk.

Step 1: HTML/JS Client (`public/index.html`)#

<!DOCTYPE html>
<html>
<head>
  <title>WebSocket Video Stream</title>
  <style>
    video { width: 800px; border: 2px solid #333; }
  </style>
</head>
<body>
  <h1>WebSocket Video Stream</h1>
  <video id="videoPlayer" controls autoplay></video>
 
  <script src="/socket.io/socket.io.js"></script>
  <script>
    const video = document.getElementById('videoPlayer');
    const socket = io(); // Connect to the server
 
    // MediaSource setup
    const mediaSource = new MediaSource();
    video.src = URL.createObjectURL(mediaSource);
 
    let sourceBuffer;
    const MIME_TYPE = 'video/mp4; codecs="avc1.42E01E, mp4a.40.2"'; // Update with your video's codecs
 
    // Wait for MediaSource to be ready
    mediaSource.addEventListener('sourceopen', async () => {
      try {
        sourceBuffer = mediaSource.addSourceBuffer(MIME_TYPE);
        console.log('MediaSource ready. SourceBuffer created.');
        startReceivingChunks();
      } catch (e) {
        console.error('Error creating SourceBuffer:', e);
      }
    });
 
    // Handle video chunks from the server
    function startReceivingChunks() {
      socket.on('videoChunk', (chunk) => {
        // Convert Buffer to Uint8Array
        const uint8Chunk = new Uint8Array(chunk);
 
        // Append chunk to SourceBuffer (handle "updating" state)
        if (!sourceBuffer.updating) {
          appendToSourceBuffer(uint8Chunk);
        } else {
          // Wait for current update to finish
          sourceBuffer.addEventListener('updateend', () => {
            appendToSourceBuffer(uint8Chunk);
          }, { once: true });
        }
      });
 
      // Notify server we're ready for the first chunk
      socket.emit('ready');
    }
 
    // Append chunk to SourceBuffer and signal server for next chunk
    function appendToSourceBuffer(chunk) {
      try {
        sourceBuffer.appendBuffer(chunk);
        sourceBuffer.addEventListener('updateend', () => {
          socket.emit('ready'); // Signal server for next chunk
        }, { once: true });
      } catch (e) {
        console.error('Error appending buffer:', e);
      }
    }
 
    // Handle end of stream
    socket.on('streamEnd', () => {
      console.log('Stream ended.');
      if (!sourceBuffer.updating) {
        mediaSource.endOfStream();
      } else {
        sourceBuffer.addEventListener('updateend', () => {
          mediaSource.endOfStream();
        }, { once: true });
      }
    });
  </script>
</body>
</html>

Key Client Features:#

MediaSource API: Manages video decoding by appending chunks to a SourceBuffer.
Codec Support: The MIME_TYPE must match your video’s codecs (update this with your video’s codecs—see below).
Chunk Handling: Ensures chunks are appended only when the SourceBuffer is not updating, preventing errors.

Finding Your Video’s Codecs#

To get the correct MIME_TYPE, use ffprobe (part of FFmpeg) on your fragmented MP4:

ffprobe -v error -select_streams v:0 -show_entries stream=codec_name,codec_long_name -of default=noprint_wrappers=1:nokey=1 videos/streamable.mp4

For H.264 video and AAC audio, the MIME type is typically video/mp4; codecs="avc1.42E01E, mp4a.40.2".

Testing the Application#

Step 1: Place the Fragmented Video#

Ensure your fragmented MP4 is in videos/streamable.mp4 (or update VIDEO_PATH in server.js).

Step 2: Run the Server#

node server.js

Step 3: Open the Client#

Visit http://localhost:3000 in your browser. The video should start playing as chunks are received.

Troubleshooting Common Issues#

1. Video Not Playing#

Check Codecs: Ensure the MIME_TYPE in index.html matches your video’s codecs (use ffprobe to verify).
Fragmented MP4: Confirm your video is fragmented (use MP4Box as shown earlier). Regular MP4s won’t stream.

2. Choppy Playback#

Adjust Chunk Size: Modify highWaterMark in server.js (e.g., 32KB or 128KB) to optimize for your network.
Backpressure: Ensure the server pauses when the client isn’t ready (the ready event logic).

3. "Unsupported MIME Type" Error#

The browser may not support your video’s codecs. Use widely supported codecs (H.264 for video, AAC for audio).

References#

By following this guide, you’ve built a real-time video streaming application using WebSockets, Node.js, and the MediaSource API. This approach is ideal for low-latency scenarios like live monitoring or video chat. For production, consider adding authentication, adaptive bitrate streaming, and error recovery!