Lessons from Live Streaming’s Front Lines: Hosting a High-Stakes Global Concert Event

For many organizations, streaming live video for a high-stakes event can be a white-knuckle affair. A lot can go wrong, and often does.

Add the pressure of a global audience, simultaneous streams to four different video platforms, embedded video players on the websites of some of the top names in music, and an organization depending on real-time donations to meet its pandemic-induced budget shortfall, and you have a production that would set any seasoned pro on edge.

That’s the situation that OpenExchange faced on December 15, 2020, as it pulled off one of the most extraordinary live-streamed video events of the year, including top-flight talent such as Rod Stewart, Neil Young, Nile Rogers, Florence Welch, Tony Bennett, Saving Grace, Robbie Williams, and nearly a dozen other top-flight singers and readers.

The event was simultaneously livecast to YouTube, Facebook Live, London Stock Exchange’s Spark Live service, and OpenExchange.tv, as well as to fan pages for many of the performers. It was viewed live by an audience of nearly 200,000, with many more viewing it on-demand.

This is the story of how OpenExchange put together the live stream for this event, and the lessons it offers for similar high-stakes live streams.

The Concert

During a holiday season looking much different from past years, OpenExchange was proud to sponsor and produce the live stream for Nordoff Robbins Music Therapy’s “The Stars Come out to Sing at Christmas” charity concert.

Nordoff Robbins is the United Kingdom’s largest music therapy organization that has supported those with life-limiting illnesses, isolation, or disability for over 50 years. Nordoff Robbins has worked with over 270 organizations across the UK to deliver quality music therapy to over 11,000 individuals in need. In addition to servicing communities across the UK, Nordoff Robbins also conducts extensive research to fuel the development of music therapy practices across the globe.

For the past 25 years its annual holiday Carol Service has been a critical source of charitable funding to help Nordoff Robbins to carry out its therapeutic work through music. As the Covid-19 pandemic swept across the UK and the rest of the world, it became obvious that the usual production in St. Luke’s Cathedral in Chelsea could not be held as usual.

Like so many events, its organizers had to figure out how to “go virtual”. And as in so many mission-critical events needing to virtualize in 2020, its organizers turned to OpenExchange to help pull it off.

Going Virtual

Nile Rodgers provided the continuity between music segments, acting as program host.

With time short and the pressure high, a cross-organization team was formed to marshal the resources necessary to execute the event. Legendary music agent Nick Stewart, music promoter Andrew Wilkinson, and music license entrepreneur Merck Mercuriadis of Hipgnosis Music Fund would assemble the performers and talent, a team from ModernEnglish Digital would set up donation pages and coordinate with social media sites, and OpenExchange and London Stock Exchange would manage the live stream and outreach to the corporate community — an effort all coordinated by OpenExchange’s Glenna Lynch out of the company’s London office.

A critical early decision was to make this a “live-on-tape” or “simulated live” virtual experience. That is, it would be live-streamed at a specific time — December 15th at 7pm GMT — but all of the elements will have been pre-recorded. Since the objective would be fundraising, it would be possible to live-switch fundraising total overlays in real time to show progress, but the musical elements would be pre-recorded and played in sequence at the appointed hour.

The decision to present the program as a “simulated live” experience had three benefits:

• It reduced the risk inherent in coordinating live feeds from more than a dozen different locations across Europe and the US;
• It increased the opportunity for each of the artists to exercise maximum possible creativity in their individual productions; and
• By playing it at a specific hour, it encouraged the idea of shared simultaneous experience to hundreds of thousands of viewers.

Veteran video show producer Tom Shaw (Shorty Productions, UK) was engaged to manage the production itself.

Managing the Live Stream

In its role as producer of the live stream, OpenExchange focused on three critical factors:

• Video stream quality
• Multi-platform distribution
• Live-title overlays
• Risk reduction and fallback strategies

The first factor — video stream quality — is core to the OpenExchange mission. The key to a high quality video stream is paying uncompromising attention to every step in the distribution process:

• The entire production would also be done in full high-definition (1920×1080 frame) quality, although one of the planned platforms — Facebook Live — has a maximum streaming quality of 720p. In a musical performance, audio would be critical, so we specified that the audio be encoded at a minimum of 192kbps.
• All original video submissions from artists would be requested in high-definition (1080p) quality — using broadcast-quality video cameras and professional audio setups wherever possible.
• While compression of video and audio is necessary when streaming video (the original ProRes source video file would be 120gb, which would be impossible to stream on even the biggest of broadband transmission pipes), we specified that the source file be subjected to the minimum possible compression before we received it — we settled on a minimally-compressed MP4 file that was about 20gb for about 2 hours of material.
• We would originate the streams from the OpenExchange Boston Livecast Center (OEBLC), which sports state-of-the-art twin NewTek Tricaster Elite 2 broadcast-quality switchers, each of which could encode two streams at 7200kbps for video and 192kpbs for audio.
• Another important aspect of OEBLC is that it’s in a location with both an external fiber-optic gigabit internet connection and a gigabit Ethernet backbone. This enabled a high-speed link between our two TriCaster units, and ensured that there would be no weak links in the transmission chain to the internet.
• We would use OpenExchange’s Knovio video platform for live-streaming the production to OpenExchange.tv and London Stock Exchange’s Spark Live pages, which would give us great quality and scalability for our live streams to corporate attendees and sponsors. Knovio, in turn, transcodes the live video feed into multiple bandwidth feeds (so-called “adaptive bitrate streaming”), which are automatically selected based on each viewer’s bandwidth and device type.
• We also set up the OEBLC TriCaster to feed Facebook Live and YouTube Music channels, also at 7200kbps for video and 192kbps for audio, assuring the best possible result for viewers.

Taken together, all of these steps assured the best possible feed out of our studios to each streaming service.

Calls to Action

Since this was a fundraising event, we wanted to make sure that it would be easy for viewers to donate at the click of a button. Because this event would be streamed across four different platforms, we needed to be thoughtful about how to support these calls-to-action.

For the streams over our own Knovio platform, the answer was easy: Knovio’s video player has interactivity built in. We created a Nordoff Robbins “Donate” watermark that, when clicked, would take viewers directly to a donations page on the organization’s website. Tracking codes would identify the source of the donation.

Facebook and YouTube were a bit trickier. While Facebook supports a call-to-action button, it had to be deployed manually and only is displayed for about 10 seconds. And YouTube didn’t offer one at all. So instead, links were placed on the page adjacent to each video, and there were also “text to donate” notifications set up, embedded in the live stream.

Co-led by Nordoff Robbins’ Matt Rigby and OpenExchange’s Glenna Lynch, the production team met by Zoom three times a week down the stretch.

Mitigating Risk

One of the reasons a livecast tends to turn knuckles white is that there are a whole lot of things that could go wrong: power failures, network outages, computer crashes, sunspots, and even people tripping over Ethernet cables. That’s why it’s important to engage an experienced hand like OpenExchange when the stakes are high: we’re experts at managing and reducing risk. With a quarter-million live viewers expected to view the live stream simultaneously and an entire organization depending on the funds that would be raised in a two-hour period, we wanted to apply every lesson we’d ever learned to ensuring that this stream went off flawlessly, no matter what came our way.

To do that, we set up in two different locations outside Boston, each on totally different electric power grids. We took full advantage of the fact that Knovio, Facebook Live, and YouTube Music all offer backup stream addresses in case one connection goes out, so we made duplicate copies of the video files, placed them into each of the two locations, and initiated feeds to the network from each location.

We made sure that our primary locations were connected to uninterruptible power supplies, which would give us 30 minutes of power continuity in the event of a power failure, so that we could execute a back-end switchover smoothly and our viewers would barely notice.

In addition, we prepared “just in case” instructions referring viewers to a different streaming service if one of the services we were using failed altogether.

And two days prior to the livecast, we performed a complete run-through of the program on all of the streaming channels — an exercise that gave comfort to the event organizers as much as to all of us behind the scenes.

In short, we constantly asked ourselves about each link in the chain, “What could possibly go wrong here?” and made sure we had an answer for that problem.

Livecast Day

In short, the livecast came off flawlessly. The trans-Atlantic back-channel Zoom call that we set up among the execution team was remarkably quiet, as we simply enjoyed the varied musical performances and periodically flashed donation totals using the Livecast Center’s titling capabilities.

Not only was it an artistic success (the Daily Telegraph likened it to “socially distanced Live Aid and just as moving”), but it raised more than £300,000 on the livecast day, and its on-demand version continues to raise money with more than 8 million views since the concert day!

Most importantly none of our fallback plans needed to be executed, though it was comforting to have them in place.

Lessons Learned

Aside from the learning that naturally came from thinking through and creating contingency plans for everything that could go wrong, there were several significant lessons that we took away.

The first had to do with musical clearances for top global performers. To protect the intellectual property and performance copyrights of artists, Facebook and YouTube have both implemented automated mechanisms that detect and block the broadcast of copyrighted music. This is largely designed to trap either intentional or unintentional use of music without being properly licensed. Of course, in our instance we not only had permission of the artists — they were creating performances specifically for our program! Our team members coordinated with Facebook and YouTube’s license clearance teams to ensure that our streams would not be interrupted, and we event ran our “dress rehearsal” through both platforms to ensure that their automation would not block our streams (they did, allowing us to tackle issues before the live broadcast). Yet as hard as we worked on this, we still ran afoul of Facebook’s copyright tripwire late in the broadcast; Facebook didn’t terminate the feed, but it flashed warnings and prevented new viewers from signing on to see the feed. This is definitely a consideration for anyone wishing to use copyrighted music on Facebook or YouTube.

Another lesson came in the form of the number of different video formats that came in from various performers. Screen resolutions were all over the map, and even the video frame rates were divided between the US standard (30 frames per second) and the European standard (25 frames per second) — not to mention the popular 24 frames per second approach that many producers use to achieve the elusive “film look” for their videos. Fortunately, we were able to use post-production magic to get all the video into a common standard, 1080p at 25 fps, and our Livecast studio was able to handle the conversion seamlessly, even though it’s set up to the US standard.

Finally, the quality of video submissions from the various performers varied widely — not so much in the audio (these are, after all, professional musicians who either have professional recording setups at home or have access to studios like the famed Abbey Road in London) as in video. Specifically, many of the submissions were shot with handheld cameras and used flashing lights and lots of motion — anathema to live streaming compression. Our compression algorithms strained but never broke, but in retrospect we might have suggested somewhat more livestream-friendly shooting techniques.

Most of all, though, we learned what a tremendous amount of work is involved in pulling off an event like this during a pandemic, but at the same time how satisfying the result can be when an event like this is placed in the hands of a highly skilled group of professionals.

Viewers can enjoy this virtual holiday concert experience throughout the Christmas season until January 5th, 2021 on OpenExchange’s own streaming channel, OpenExchange.tv.

OpenExchange’s Colin Hickey contributed to the reporting on this article.