Hybrid Events – The New Normal

Due to the COVID-19 lockdowns, people were forced to get comfortable with the online meeting platforms such as Google Meet, Zoom, and Microsoft Teams. Now that people are used to those technologies, it has changed the way many participate in meetings and as we return to in person events, there is now an expectation that not everyone needs to attend, or even present in venue.

This is the new normal…Hybrid Events.

A Hybrid Event is a traditional in-person event at a venue, but with an online component bolted on such as presenters delivering their presentations remotely, and of course delegates viewing remotely. And by the way, at every event I work on nowadays, there is always a group of presenters who have just tested positive to COVID and therefore cannot attend in person. They’ll need to present remotely.

It is crucial that the online side of an event is rock solid reliable. I’d argue just as, if not more important than the AV facilities at the venue itself. Remote contributors have come to completely rely on remote technologies. I have done numerous events where prime ministers, business titans, and high-profile celebrities have called in remotely, and the success of the total event relies on these links working well. Similarly, remote delegates expect to be able to participate just as if they were in the room.

To achieve a good result, you must have a good, hardwired, enterprise grade internet connection. In my opinion, any reputable venue should have this now, especially after COVID. Relying on Wi-Fi is a recipe for disaster. Wi-Fi is capable of high speeds, but is not capable of delivering that consistently. It is susceptible to interference, especially in a venue where there will be a lot of users, which leaves you exposed to dropouts.

A backup connection is always wise. The Netgear Nighthawk range of mobile routers offer amazing value as they are a Category 16 Modem, and they have an Ethernet connection. Combine one of those with your house internet connection by using a Ubiquiti Edgerouter X Router, and you have a fully automatic load balanced redundant internet connection. Ubiquiti Edgerouter Xs are an enterprise grade load balancing router that offer great specifications for little money (if you can get one due to the worldwide chip shortages).

Note that I said load balanced. Load balancing (often referred to as dual-WAN routing or multihoming) is the ability to balance traffic across two or more WAN (Internet) links without using complex routing protocols. So if one of the connections were to fail, it can take time for the router to recognise the failure and route the traffic to the remaining internet connection.

This is not a perfect solution, but if you can tolerate a rare failure of say up to 1 minute, it provides cost-effective peace of mind.

A superior option is a Bonded connection where multiple internet connections are combined into one single pipe of data through a VPN tunnel to a server (usually housed in a data centre) that takes the data from the multiple connections, recombines it back into single stream, and forwards it on to its final destination. Bonded solutions have a huge advantage in that if a connection fails, the data is duplicated on the remaining connections so that no packets are lost. Peplink are the kings of bonded data solutions.

I have built a custom Peplink Pepwave system which combines 4, 4G and 5G cellular modems into a single data stream to a server that I host in Sydney. If one or 2 modems drop out (and they sometimes do), it doesn’t matter. The remaining modems take up the slack and I have a completely reliable data feed.

The Pepwave system is a great mobile solution but it does have a couple of drawbacks. Firstly you need to be in an area with good, high speed mobile overage. Secondly the system does add some latency due to the extra processing and wireless connection, about 30-80 milliseconds which can become noticeable on a two-way video call. And lastly, because it is wireless, jitter will be greater. With video calls, dropouts and instability are often a result of high jitter. However, with Pepwave, I can tune a lot of this out by increasing the latency of my VPN tunnel.

I use the Pepwave gear when I cannot get a good hardwired connection from the venue. I push hard for a venue connection but sometimes it is not available due to information security reasons. For example, I do events for one of the big four international consulting firms where their client’s information could be worth billions of dollars. They are quite rightly not going to allow some streaming dude access into their network.

By now you have probably realised that I am focussed on reliability. As long as the speed is adequate, I am happy.

So what is an acceptable speed? I like to see double what I actually expect to use. As stream to YouTube should be about 4.5 Mbit upload. Combine that with some remote callers on Zoom, say 2 Mbit each way, plus some download for monitoring purposes you are looking at 10 Mbit up and down. Therefore, 20/20 Mbit is the minimum I would accept.

Be aware though, that if you are on 4G and your speeds are low, that generally is a sign that your connection is poor too. In those case, I’d be looking to get the modems at least near a window and in some cases outside. Some modern buildings have metallic film on their windows which acts as a Faraday cage (a Faraday cage is a metal enclosure used to block electromagnetic fields). I even have one client whose premises was deliberately built as a Faraday cage so as to ensure wireless technology was blocked and their secrets could not be stolen! When we do events for there, we locate my Pepwave gear outside the building and a hire a guard to watch the gear and to ensure the entrance is secure.

But with remote callers, their bandwidth can be a factor too. Before COVID, 1/3 of Australians had signed up to NBN’s slowest plan. 12 Mbits down, and 0.8 Mbit up. 0.8 Mbit is right on the edge of an acceptable Zoom call. As soon as someone else in the household starts using the internet, the call will likely suffer. Fortunately this is less of a problem now as most NBN users have upped their plans with 75% now on at least 50 Mbit down, and 12 Mbit up.

With remote callers, internet bandwidth is only the first consideration. They will almost definitely be using Wi-Fi and not a hardwired connection. This is where you need to balance production expectations, versus how much the remote contributor is prepared, or able to do to improve production values.

You can always do more to improve the production values, but presenters are not there for production values alone. They are there because they have something to communicate. As an online producer, it is up to us to deliver in the most faithful way, but not the point where the production overloads the presenters.

Firstly, I push hard for a quick tech check prior to the event for each presenter. It provides an opportunity to introduce myself to them, and importantly, we can deal with technical issues without the time pressure of the event itself. During the tech check call, you need to be ready to guide the caller on where to make changes. That means if you are on Zoom or Teams, you need to know those applications really well.

First will be their connection quality. Most people will be on Wi-Fi and hardwired ethernet won’t be an option. If it is poor, ask them to move location within their building where hopefully the Wi-Fi connection will be better. As well as this, get them to ensure no-one else is using the Wi-Fi at the same time. If it is still poor, encourage them to try another location completely. This happens regularly and presenters are often receptive to moving because they want it to work well too.

Sound is next. I really encourage the use of hardwired headsets. Presenters often push back and say something to the effect of “it is normally ok”. We want better than just OK. I explain that we get better results with a headset because the echo cancellation technology does not have to work as hard. Consequently, you’ll achieve a more free-flowing conversation combined with much better audio due to the microphone being closer to their mouth.

That being said, I care more about the caller getting headphones or earbuds working so their laptop speaker is not required. The earlier generation Apple Airpods had a crappy microphone so often I’ll get the presenter to use the microphone in their laptop, whilst listening by using the Airpods.

Bluetooth headsets, and especially Bluetooth Airpods are a world of pain. I have observed countless occasions where the caller has problems getting their own Bluetooth headsets working, usually followed by the phrase “it normally works fine!”. It is hard to guide them to a solution in these cases because you don’t know their setup. One thing to be aware of though is that Airpods are promiscuous. They’ll connect automatically to the user’s phone, which breaks the connection to their laptop so the presenter must ensure that Bluetooth is turned off on their phone.

Whilst I am talking about audio, avoid lapel microphones in the venue where a PA system is used. I always use handhelds, or even better headsets. A carefully positioned omni headset capsule is about three centimetres from the mouth, whereas an omni lapel microphone is roughly thirty centimetres. That is, the lapel is ten times further way than the headset. That means to achieve the same signal level going to the remote speaker, the gain has to be increased by twenty decibels on the lapel when compared to the headset. Consequently, room noise including the PA system is brought up by twenty decibels as well! Therefore, the remote presenter is much more likely to hear an echo of themselves coming back through the venue microphones.

Returning to our remote presenter, I coach them on the best angle to have their camera. Ideally, the camera needs to be at eye height so often that means putting the laptop on a stand, or books and of course we don’t want them too high or low in the frame. I literally say “we want to see more of you, and not so much of your roof”.

Then we discuss lighting. Good lighting is incredibly important and you can get remarkably good results with poor cameras as long as the subject is lit well. This is especially important if they use a synthetic background. That means the light from windows is usually too strong and needs to be controlled. And then depending on the lighting in the room, some front light may be needed. Often this is just something as simple as a desk lamp.

For high profile events, I have shipped microphone and ring lights to the remote presenter. The client pays for these, and gifts them to the presenters. This might sound expensive, but a Rode NT-USB mini is about $130, and a decent ring light can be bought for about $70. By doing this, we achieve consistent results.

As the online producer at the venue, what vision do you send back to the remote presenters? You want the remote presenter to feel as much as possible that they are in the room, and you don’t want them distracted. Therefore a camera pointing at the people on stage is crucial, but the projection screen should not be in that frame.

You want to avoid the remote presenter seeing themselves, especially on projection screens in the venue. It is a major distraction. Ideally you want them to see whoever they are talking with, and that includes the audience. In my return vision I typically have a wide shot of the stage (with no projection screens seen), a close-up of the presenter in the room, and a camera pointed at the audience in the room and a countdown timer. Once the event has started, I do not send the camera image of myself as that can also be distracting.

When planning the event, encourage the organisers to schedule the remote presenters at the start of the event, or immediately after a break. That way, the remote presenter can call in during the break and has an opportunity to get comfortable with what they are seeing and hearing from the venue. I encourage the MC to have a chat with the remote presenter through the PA system so both of them can get a feel for what to expect.

Once the event starts, an audio back channel to the remote presenter is essential. They are feeling isolated, not really sure what is going on in the venue and wonder if they can be heard clearly and so on. With an audio back channel, you can jump in and talk with them if required. I set up an arrangement where I can hear them without their audio going to the PA, and a microphone with a push to talk button. By having a push to talk button, I can’t accidentally feed my audio back to the caller; I need to deliberately push a button to be heard.

A lot can go wrong with hybrid events. You are working with lots of variables so for those reasons, plenty of testing is crucial for a successful event. For that reason, I insist that at least a full hour is available to test all systems. Test, then test, and then do some tests. It is so important reveal issues before the event rather than during it. There will be teething issues…

I try to set up the day before for that reason. That way I have plenty of time to identify any issues and I usually leave the equipment on overnight. If things have remained stable overnight, it is usually a good sign that you’ll have a good show!

By the way, avoid system updates leading up to the show. Microsoft have become increasingly aggressive with their updates lately. Even if you turn them off, they still seem to get pushed through. I recently noticed many of my USB audio interfaces would inexplicably stop working on most of my Windows machines on a really random basis. It turned out that Microsoft had pushed an update that reenabled the USB Selective suspend setting that I had previously disabled. My USB ports randomly stopped working!

In these uncertain times, you need to be flexible. You will have unplanned situations due to someone getting the bug or other reason. That being the case, have some contingencies in place to accommodate “the speaker who won’t be attending person now”. It is normal now.

You also should have a plan if the remote caller’s connection fails, or you cannot get your feed out. It does happen, usually for reasons out of your control. Have an honest chat with the organisers prior to the event so they too can have contingencies in place if there is a failure.

Lastly, record every single event, even if you are not asked to. That way, if there were a major stream failure, you have a copy available to play at a later time.

I had a recent event where the online platform provider screwed up their registration system which meant 40% of our audience could not view the event live. Far from ideal! But as I had a recording, we were able to make that available to the viewers who missed out.

Online events can be surprisingly complex. As you’d expect, the key to delivering a successful online experience is preparation, combined with managing the client’s and presenter’s expectations, and plenty of testing.