Globo.com is the Internet arm of Grupo Globo, the largest media conglomerate in Latin America. We have recently broadcasted the FIFA 2014 World Cup for more than 450 thousand concurrent users.
Our new player, called Clappr, supports HLS, a protocol that is also supported by the two major mobile platforms: Android and iOS. This is a huge advantage, because we can stream to web and mobile clients with a single protocol, keeping our architecture very simple and efficient. Basically, we use an encoder (mostly Elemental) to encode a video signal in H.264 and ingest the video stream into our segmenter (EvoStream) using the RTMP Protocol. Evostream creates the HLS video chunks and video playlists in disk and because HLS is HTTP based, we can use a couple of geographically distributed layers of Nginx servers to deliver and cache the video files.
One of the new features of our Live Video Streaming platform, developed for the World Cup, is the DVR, or Digital Video Recording. We keep the last couple of hours of the video stream recorded, so the user can pause or seek back to watch that goal he missed. For the World Cup, we only had two simultaneous matches, so we could easily keep the streams in memory. To add this feature, we created a Python application that moves the HLS video segments into Redis and then scripted Nginx with Lua to get the segments from Redis and to generate HLS playlists dynamically.
The main problem with this solution is that it does not have high availability. If the Redis server failed (which is rare, but happens), we would switch to another Redis instance, but we would lose all the video recorded. Thankfully, this did not happen during any World Cup match (although we would not mind deleting those 7 goals from Germany). The other issue is that this solution did not scale well for more streams. Since each stream requires around 10GB per recorded hour, we could not record much more than two or three streams with a single Redis instance.
Brazil recently had presidential and governor elections and we wanted to stream all 27 of the governor debates that would occur simultaneously, and with the DVR functionality. Since our solution with Redis would not scale, we decided to give Cassandra a try.
It turned out to be relatively easy to modify our Python software to post the files to Cassandra instead of Redis using the DataStax Python driver. The hardest problem was to handle the huge ammount of file events that this application had to handle. We ended up creating multiple Python processes and using async execution. Another issue is that we did not know how we would extract the blobs from Cassandra and deliver them using Nginx, since we couldn’t find Cassandra driver’s available for Lua or Nginx. We thought about developing a Python application, but we are huge fans of Lua and Nginx, so we decided to go ahead and develop our own driver for Lua: lua-resty-cassandra; yes, it’s open source! When it was ready, it became pretty easy to port our Lua scripts from Redis to Cassandra.
Our solution worked, but Cassandra response time increased a lot after a few hours. After a certain point, the clients would start to timeout, and the video playback stopped. It took a couple of weeks for us to realize that we had implemented a known Cassandra anti-pattern: Queues and queue-like datasets.
Fortunately, we could fix the problem with a few simple changes:
We denormalized our data, using different tables for chunks (since each video chunk can have up to 2MB) and chunk indexes (that are stored as rows and can contain a few thousand columns).
Finally, but most importantly, since we knew the maximum size a playlist could have, we could specify the start column (filtering with id > minTimeuuid(now – playlist_duration)), as suggested on DataStax’s blog. This really mitigated the effect of tombstones for reads.
After these changes, we were able to achieve a latency in the order of 10ms for our 99% percentile. The debates were held and we were able to stream all of the 27 streams with 2 hours of recording with no major problems. We are really happy with Cassandra now.
We are using only 4 nodes at the moment. Each node has 24 cores, 64GB of RAM, 6 disks and we are running Cassandra 2.1.0 on RHEL 6.5. We have about 60GB of stored data on each node (the RF is 2). The network flow rate is about 65 Mbits/sec. The CPU load avg is less than 1. The 99% read latency is 10ms and the 99% write latency is 30ms.