How to create and architecture an open-source and/or free video platform ?

This post is the result of a sort of challenge. The objective is to create an architecture to generate, manage and deliver HTTP streaming videos using free and/or open-source tools and/or applications. It need to answer to many questions : tools must be open source or free; reliability of the platform and the ability to scale up quickly. The architecture can be split in two parts : the content preparation and the delivery. I will expose an overview for each part of the architecture. Then I will list differents tools you can use. To finish, I will give you in details the architecture I will choose to deploy. But to realize this, few or some developments can be required, the language you will use, will be your choice. The operating system will be on Linux.

Content preparation

Here is the part to prepare the content. It will be separate in 3 differents items, I call them services :

  • Manager this service will orchestrate all the tasks. In this part, I choose the self-development but another option is possible (see below).
  • Bus the service bus will deliver all messages between others services
  • Worker each worker will be in charge of one specific task : encode, upload, download, package.

All of these is exposed in this figure :

how_to_open_arch_video_step1

The generated assets are transfered in the delivery zone storage.

Service : Manager

Here is the intelligence of the system. You have two choice :

  • create your own system. In this case, the manager need some developments, you can choose the language you want : Python, PHP, NodeJS, Java, Perl, etc…
  • using a system already developed. I think about Kaltura platform for example. This option need some integration and learning time if you want to use it.

I recommend to associate your code to a database. The database will save the metadata of your video assets and some configurations datas.

Service : Bus

The service bus will use the concept of Message queue. That is very interesting because it helps to manage more easily the communication between each application : you send the message and the library or the system make the rest. There are many tools in the open source world :

For each message queue solutions, you will find connectors for separates languages (C, C++, PHP, Python, Perl, …). Depends on the message queue solution you choose, you need to make more or less developments.

Service : Worker

Workers services are here to make specific tasks :

  • encode : this service is based on the most famous open source tools in the encoding’s world : ffmpeg and its differents libraries (libfaac, libx264, …). But you can use mencoder too.
  • package : the objective is not to encode the video asset, but to change the format. Some tools are available :
  • transferts : at the end of a process, you will need to transfert the asset to a destination. Transfert workers is able to use some protocols : FTP, SCP, HTTP, etc… So you can push the content to the delivery storage.

All of this services can be in the same physical server. If you have more assets to manage and generate, it is possible very easily to scale up. Workers can be deploy on dedicated server and the bus service is here to manage the network. You can add another service manager to add redundance. Thoses updates will have a impact on the reliability of the platform.

Here is one solution which include the most part of those precepts : Transcodem. You can fork this project on GitHub too.

Delivery

Ok, now your assets are ready to deliver and are in the delivery storage. In our architecture, we want to stream content based on HTTP protocol (HDS, HLS, Smooth Streaming). You can start with one physical server, but all tools and the architecture I expose can scale up easily to many physicals servers. We have many tools to help us :

And if your platform will support a high load, you can use proxy cache solution like Varnish. I gave in a previous post a solution to deliver content with Varnish and MistServer.

how_to_open_arch_video_step2

To finish, you can split the two part or merge them in one infrastructure. I am opened to talk about this architecture and which tools we can use to deploy it.

3 thoughts on “How to create and architecture an open-source and/or free video platform ?

  1. Doug Stevens

    Interesting idea. We spent a lot of time at my last company in the innards of ffmpeg, making custom changes to make this kind of workflow possible, but gradually moved away from ffmpeg because it tended to be a bit too unstable.

    What input formats do you plan to support? File, UDP, RTMP, …? MTS, MPEG4/Quicktime, MXF, GXF, grouped elementary streams, …? What kind of video and audio formats?

    What output formats do you plan to support? File, UDP, RTMP, HDS, HLS, Microsoft Smooth, MTS, MPEG4/Quicktime …?

    Reply
    1. Olivier Post author

      For ffmpeg, this tools is very stable for me. As I said, the architecture I exposed need some developments and integrations. And the big part is probably on ffmpeg side (understand all the parameters and find the best presets).

      The input format can be all the format you want to use. It depends on the usage of your platform.

      The output, I said it in the introduction, will be all the HTTP Adaptive Bit Rate formats (Smooth Streaming, Dynamic Streaming, Live Streaming, DASH). This platform can deliver too MP4 files via progressive download.

      Otherwise, I am very interested about your feedback on ffmpeg usage.

      Reply
  2. Pingback: How to create and architecture an open-source a...

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.