Disclaimer: I recently attended VMworld 2014 – SF.  My flights and accommodation were paid for by myself, however VMware provided me with a free pass to the conference and various bits of swag. There is no requirement for me to blog about any of the content presented and I am not compensated in any way for my time at the event.  Some materials presented were discussed under NDA and don’t form part of my blog posts, but could influence future discussions.

 

VAPP2457 – Hitting It Out Of The Park: vSphere and MLB Network

This session has been one of my highlights so far, simply because I’m fascinated by sports broadcasting and the infrastructure that goes into media production at a broadcast level in general. Again, I’ll try not to post a transcript, but just some of the bits I found interesting. Which means it will be a transcript, because this stuff rocks in my opinion. But firstly, the proof of life photo.

 

This session was presented by Tab Butler – MLB Network Director – Media Management & Post Production. Tab knows his stuff, and I found this to be a particularly entertaining session for that reason alone. And these guys are everywhere, at least in America.

Tab starts out with a bit of background of the Network, and it sounds pretty intense.

The big challenge was working out how to bring up a back office infrastructure within a few weeks. vSphere was the key to the success here. To give you an idea of the timeframe they were working to, they gave their hardware vendors a PO on the Friday, and were hopeful that they could deliver on the Saturday. I simply can’t imagine that happening in AU.

For production they used Grass Valley Aurora on standard (Dell 9G) servers – about 400 of them. They have 80 recording channels bringing content in, and the data is stored on 2 SANs. Tab load balances across the two by having an American League SAN and a National League SAN. They also used Apple Final Cut Pro 7 (still a staple for performance editing). This is then fed into a file-based service called the Omneon system (I swear there’s more acquisitions going on in this industry than in the storage industry) sitting on more (13) file-based SANs. There wasn’t a media asset management system agile enough to meet their requirements so they built their own – the Diamond System. This is a big ole system for logging metadata to a database. Seems simple enough, but it still took a while to put it together. Their in-house development continues today.

Funnily enough, the SANs fill up quickly, and content needs to be moved within 8 hours to the archive, based on LTO-4 (StorageTek / Sun / Oracle SL8500).

MLB Productions (the keeper of the archives) also leverage this system. This is where the footage you see in movies and documentaries comes from. There’s an on-site and off-site copy of the library.

Today, they record 7 hours minimum of baseball for every hour played:

• A clean feed (no graphics) from the home team and from the away team
• With graphics from the home team and from the away team
• Dugout isolation camera
• Backup records, and other iso records that people want as well.

In 2009 / 10 it was all getting too much. So how about running virtual in the production space? They asked Grass Valley what they did to scale at trade shows, and the answer came back: VMware. With the help of an initially reluctant Grass Valley team, they put it on a pair of IBM servers running 8Gbs FC to NetApp E-series and it’s still running today.

The Diamond System has grown, and now runs on Simplivity Omnicube CN-3000s, while Omnicube CN-2000s run on the back-end for development.  Incidentally, if you’re at VMworld in SF this year and want to know more about what the Omnicube can do, go and talk to David at the Simplivity stand. He’s a good guy and knows his stuff.

There’s over 500,000 hours of content in the archive now. To put it in perspective, as of 2008 they had around 150,000 hours of content, and they’ve tripled that in the last 6. It’s not just MLB Productions or MLB Network using it, it’s also used for umpire review, and by the teams as a critical source of information and analysis.

Five years in though, they were running into issues with the post-production infrastructure, because a bunch of stuff was either approaching EOSL or well and truly there.

So they went for:

• Open architecture (vSphere, NetApp E-Series)
• Scalability and performance (UCS)
• Convergence (Simplivity)

All with a CIO-mandated 7-year ROI. Eeep! And issues with power, cooling and space. And 25% infrastructure growth.

They also introduced optical routers (and a lot of glass). And seriously responsive KVMs for editors (all on one fibre). And there’s 192 cores of fibre between sites as well. This has really changed the game for their editors in terms of efficiency. Between the two sites they have 320Gbps as a starting point for bandwidth, with a lot of room to scale up if required. 10Gbps to every server in the racks. Cisco 5596s delivering 8Gbps FC to servers as required. One other cool thing they have is monitors on top of the extra-high racks to keep on eye on what’s happening on the servers. They now have 136 recording channels (increased from 80 previously).  They’re using an SDCAM HD50 codec – 50Mbps (16 audio tracks). The SANs currently hold 4500hrs of content, but Tab expects this to scale 10x in the next 7 years. Everything sits on a StorNext 5.1 file system.

Matching the end-user experience (read expectations) and the technology available today has been a key design goal of the build. Sometimes they need bare-metal, but UCS works fine for that too. They don’t have a lot of people looking after this either, so they need to use everything VMware offers to keep it running smoothly. There’s also some AWS in the mix.

The challenge in metadata management is sticking to a taxonomy and not changing it (they haven’t changed it since 2011). The tighter they are with the metadata, the easier it is to find what they need.

And that’s about it. Sorry I got a bit carried away, but I found this session fascinating. 5 stars.