I flew a helicopter, and then the helicopter flew me

I’m sitting in a secure beige conference room at the headquarters of Lockheed Martin-owned aircraft manufacturer Sikorsky in Stratford, Connecticut. In a few hours, I’ll be in the cockpit of a Sikorsky S-76 helicopter outfitted with the company’s six-year-old Matrix autonomous flight technology, which was born out of a DARPA program, letting a computer take me on a guided tour of the Housatonic River from about 3,000 feet in the air. The tech has come so far in that time frame that the Federal Aviation Administration has given Sikorsky permission to let a total newbie try it out. So, in the afternoon, I’ll become one of the first non-pilots to do just that.

Before that, Chris Van Buiten, vice president of Sikorsky Innovations, the company’s future-solving division, explains how much work that system will be doing, and he tells me something that rattles around in my skull for the rest of the day.

“We talked about a way to have you fly the helicopter with the [autonomous] system off… and we couldn’t do it safely because you would die, and it would be on the order of 60 seconds,” Van Buiten says with a smile, leaning back in his chair. “It would be so shocking to you how out of control you would be.”

This is not because of some personal failing. Helicopters are really that hard to fly. For reference, Van Buiten tells me how, after about four hours of flight training in a helicopter years ago, he finally felt comfortable holding the aircraft steady in a simple hover. But when his instructor asked him to call the airport tower on the radio, he started to lose control. Flying a helicopter requires such an intense mental load, he says, that even small things like pushing the “talk” button can put a novice pilot in peril.

“Eventually, the computational firepower between your ears runs out,” he says.

This is one of the problems Sikorsky is trying to solve by developing autonomous technology for helicopters. An obvious, if still very distant, end game is that the tech could help spark the rise of autonomous “flying taxis” that revolutionize how we get around big cities. (In fact, this week, Sikorsky dreamed up such a situation in a commercial it made with The Spaceship Company and Otis Elevators.) But long before that happens, the company can use the tech to help human pilots who are already in the air.

Commercial airplanes have had autopilot functions for years, so you may think that adding autonomy to helicopters is a no-brainer. But helicopters fly much closer to the ground, which inevitably means there more obstacles to avoid. For reference, Van Buiten imagines a tiny plane flying through a conference room. It’d be about five feet in the air, with nothing blocking its path. A helicopter, meanwhile, would be no more than a half-inch or so off the floor where there’s a whole bunch of stuff in the way.

Add in all of the complicated situations that helicopter pilots experience, like dropping onto the landing pad of an oil rig during a storm in the North Sea or trying to airlift someone off of the side of a highway at night surrounded by trees and power lines. That complexity in these kinds of situations is what breeds accidents, Van Buiten says.

“It’s not, you know, ‘Jimmy fell asleep.’ It’s not that,” he says. “They’re incredible crews doing an incredibly difficult task. It’s that half inch of the floor problem.”

In other words, even the best pilots in the world could use a little help. A little autonomy would go a long way.

What Sikorsky has developed is more than just a little autonomy, though. The innovation team has built a fascinating, complex sort of hybrid system that makes flying a helicopter so simple that they let me try it after barely 45 minutes on a simulator. What shocks me the most once I’m up in the air is how easy everything is. Sikorsky has put enough work into this tech — on both the flight control and the user interface side — that it was easier to fly than a DJI drone.

My journey to flying in an autonomous helicopter starts in the back of a Winnebago. The Sikorsky Innovations team has taken an RV and turned it into a “mobile ground station” for the autonomous helicopter, which is called SARA (Sikorsky Autonomy Research Aircraft). Up front are computers and giant screens where Sikorsky Innovations employees can make changes to the helicopter’s software on the fly. They can monitor and control SARA from here in real time, though the team is careful to point out that the helicopter doesn’t need the RV to operate.

SARA is a self-contained research vehicle, chief pilot and development lead Mark Ward explains. It’s wrapped in a gradient of dark blue (in the front) to light blue (in the back), and it’s also splashed with computer circuitry graphics, which provide the only hint from the outside that something is different about this helicopter. At 43 feet long, it can normally fit 12 passengers, and it’s often used to fly CEOs. Behind the cockpit, this version has been gutted and outfitted with the brains that run the Matrix tech.

SARA uses LIDAR sensors and computer vision to measure the ground beneath it or detect nearby objects that present a threat. But SARA doesn’t rely as much on “high-order” functions like AI since those are harder to certify. (“To be FAA certifiable, you need a certain level of determinism in the outcomes,” Van Buiten says.)

In the back of the Winnebago is the flight simulator. It consists of a giant flat panel television, a chair, a tablet, and two “inceptors”: a joystick on my right, and a lever to my left, which are both 3D-printed. Running on Linux, the custom-built software shows me a map (both on the screen and on the tablet) of the surrounding area and a digital representation of SARA.

Ward, who will play my copilot in the real helicopter, walks me through the user interface the team has developed for the autonomy system. I’m told I’ll use the tablet to direct SARA to autonomously take off and land and also to load a 15-minute flight plan that the team has already cleared with air traffic control. I also learn that I’ll use those inceptors to manipulate the autonomous flight and, at a few different points, to fly the helicopter myself.

During the simulation, Ward runs me through how to use the inceptors. In a normal helicopter, you control flight three ways: the “cyclic,” or the stick, is used to move forward, backward, or side to side; the “collective,” a bar on the side of the pilot’s seat, controls altitude; and pedals at the feet rotate the helicopter.

The inceptors don’t directly adjust any physical controls. Instead, they’re taking my inputs and telling the Matrix software where I want to go, and the computer translates that to the flight controls. Push right on the joystick inceptor, and the helicopter will drift to the right at whatever speed I choose. Pull back on the lever inceptor, and we’ll climb until I let go. Roll the thumb wheel on the lever inceptor, spin SARA.

When the helicopter is in the middle of autonomously following a flight plan, I’ll also be able to use the inceptors to briefly move off the path. When I let go, the autonomous system will readjust and guide us back.

It’s all meant to be so simple that I should be able to figure it out without much instruction, which is essentially the opposite of what it takes to learn to fly a regular helicopter. So when Ward asks me how I think I should change my airspeed, and I say “push forward on the joystick inceptor,” his face lights up and he claps his hands together, letting out a long “yesss.”

What’s clear almost immediately is that while making a helicopter that can fly itself is understandably hard, the task of developing this “human-machine interface,” or HMI as Ward calls it, seems equally daunting.

Ward’s team built the HMI from the ground up. He loves this because it gives the team free rein to design it the way they like. If he doesn’t like the size of a particular UI element, for example, the team can change it in a matter of minutes. Or if they want to make a change to the inceptors, they can do that, too. (Ward says they’ve tried “some weird stuff,” including trying to fly using a drafting mouse.) “If we were working with a third party, anything like that would take weeks,” he says.

There are many obvious choices Ward and the Sikorsky Innovations team had to make when developing this HMI from scratch. He explains a few. They decided not to allow pinch-to-zoom interactions on the tablet like in Google Maps, he says, because that could lead to “spurious inputs,” which is official speak for careless taps. The main way to direct the helicopter to fly somewhere autonomously is to pull up a widget — “the daddy of all widgets,” Ward calls it — and drag it to the point on the map where you want to go.

This widget lets you set ground speed, altitude, or simply command the helicopter to take off and settle into a hover. Once you set all of the desired parameters, you tap a box at the top of the tablet screen that says “READY TO EXECUTE.” You can use the widget to set a “fly to” point on the map, and the helicopter will take the simplest, safest route there, constantly scanning its surroundings to make sure nothing gets in the way. Or you can load pre-calculated flight plans.

Each of these corners of the UI represents dozens of decisions made by Ward and his team. He says the FAA sets guidelines for HMIs, right down to font size and color, so they do have some constraints. But otherwise, the interface represents a world of opportunity. Sikorsky is not only trying to make helicopters that can fly themselves, but it’s also thinking a lot about the humans who will still be in the cockpit for many years to come.

“All the fancy autonomy algorithms happening in the background, they’re very, very hard to do, don’t get me wrong,” Ward says. “But the HMI is really hard.”

It’s because of all this work that I won’t have to touch the cyclic, the collective, or the pedals. I won’t have to do any hard work. This is why I never wind up running into the situation Van Buiten, the vice president, warned me about a few hours before. The one where I die.

It’s finally time to fly the helicopter. Ward runs through a preflight checklist, where I learn we won’t bother with parachutes. We walk out onto the tarmac under a bright, clear blue sky, one day removed from gale-force winds. I get a typical safety briefing, where Ward demonstrates how to operate locks on the helicopter doors that are straight from a 1970s-era Ford van. I should be nervous, but I’m not. The day-long pitch about the work that Sikorsky’s done to get this tech to its current state has been effective. It also helps that at the same moment, a replica Marine One is taking off across the airfield. Reliability is sort of the company’s whole thing.

I hop into the cockpit, and Ward starts tapping buttons and flipping switches. We’re in a helicopter that’s about to fly itself, but it’s clear that the preflight preparations haven’t been automated yet. But before I know it, we’ve taxied out to the center of the company’s private airfield, and the little box at the top of the tablet strapped to my leg says “READY TO EXECUTE.”

Tap.

The S-76, a workhorse of Sikorsky’s fleet that weighs nearly 3 tons and cost millions of dollars to build, lifts gently off the asphalt. The helicopter climbs up into the blue sky to 60 feet and stops; it’s now holding steady in a one-foot by one-foot by one-foot box we told the computer to find. It jostles a bit — keeping a dynamic aircraft like this in one spot is tough, even for an advanced computer — but we’re hovering calmly above the runway.

Ward gives me a minute to get used to the sensitivity of the inceptors. He tells me to move the helicopter a few hundred feet to our right over the runway number and to point the nose to the south. I tilt the joystick a few degrees, roll the wheel on the lever to my left, and we glide over the pavement. In one fluid movement — and on my first try — I pull off a move that would have taken countless hours to learn and a lot more nerve if I were using the helicopter’s usual controls.

We play around over the airfield a bit longer before Ward has me plug in our planned autonomous flight. He slips in a Star Trek joke (“make it so”) as I tap the tablet again to send us on our trip. The helicopter turns, slowly, and then starts to bring us up over the Housatonic River toward the first waypoint, which is to our east.

The second waypoint is a few hundred feet higher and a few thousand feet farther north along the river, so the helicopter banks to the left and continues to climb. Ward asks me if I think the tuning of the helicopter’s turns is a bit aggressive; I say yes, but then again, you always seem to notice the dynamics of a vehicle more when you’re not in control.

The helicopter keeps bringing us north, and Ward has me try using the inceptors to break off the path a bit. I bank us to the left a few hundred feet, then ease off the joystick. The helicopter rights itself, then points its nose back toward our final destination. Like that, it’s already recalculated the best way to get there.

While all of this is happening, dozens of blue dots — some big, some small — pop up and then disappear on the map on the tablet. Ward is obviously along for the ride in case anything goes wrong, but if something goes really wrong, the helicopter is always finding safe places to land. Each blue dot on the map represents a spot where the computer has decided we can land in an emergency. I flash back to earlier in the morning, when Van Buiten told me that Sikorsky has tested how the autonomous system handles emergency landings and how they found it gets them right more consistently than even the best pilots.

Once we reach the final waypoint, Ward picks out another spot in the distance (a teal building poking out through the trees) and asks me to fly us to it. Again, I’m able to pull off the maneuver as if I’ve been doing this for years. It’s like flying a consumer drone you can sit in without having to worry about small wind gusts or battery life.

I’ve taken some wild rides in my time with The Verge, but this one is almost placid — though I’m struck by how similar it feels to the final scene of Willy Wonka & the Chocolate Factory, where Charlie Bucket and Wonka ride the Wonkavator through the roof and up into a big blue sky. (Maybe this is why Sikorsky made that commercial with Otis.) By the time I punch in the command that tells the helicopter to take us back to the airfield, my only thought is that I wish I could have gone rogue and taken us on a much longer joy ride.

The helicopter lands itself without a hitch. The broad strokes of the technology are very obviously in place. Now comes the hard part.

There are a few obvious challenges ahead. One, Sikorsky now has to validate this tech to the point that it satisfies a wide number of potential customers and regulatory bodies. The good news there is that Sikorsky has decades of experience doing just that.

“We are willing to do the un-fun, mind-numbing work of making [Matrix] highly reliable, and robust and redundant and able to sustain failures — multiple failures — and still work,” Van Buiten says. He never mentions any of the competitors in this space by name, even though there are quite a few. Bell, Boeing-owned Aurora Flight Sciences, SkyRyse, and even Uber are among the companies exploring autonomous flight. “I think there’s some flash out there that is not pursuing the same level of rigor,” he says. “We have the appetite for that rigor.”

Another challenge is finding customers, though Sikorsky already has some in mind. Van Buiten says the tech has “a lot of potential in Black Hawk,” the helicopter Sikorsky makes for the US Army. In fact, he says, the company is going to perform autonomous Black Hawk test flights this year. Military settings might make a lot of sense for this technology, whether it’s used to safely shuttle cargo or for taking the strain off of pilots who are flying high-stress missions.

Van Buiten says he also sees the tech spreading to more of Sikorsky’s commercial fleet beyond the S-76, and that it could even translate to other types of aircraft. “The technology is similar, and the helicopter problem is harder,” he says.

It’s easy to imagine Sikorsky’s tech becoming the backbone of an autonomous taxi service someday, but, if anything, my ride with SARA reminded me of how far away a use case like that might actually be. Sikorsky should be able to validate the tech to the point that it can consistently operate in highly specific settings, like in the military, or as an assist to highly trained pilots in more commercial settings.

But proving it out to the point that we feel comfortable putting our families in it for a short ride from a city center to the airport or wherever else flying taxis might go, requires a whole other level of certainty.

“The burden of responsibility — the same burden we’ve carried for decades with people who fly in our products — is now going to go up by, it’s almost two orders of magnitude,” Van Buiten says before my flight. “We’ve got to create a completely new exceptional standard for safety, and we think the autonomy that you’re going to fly today gets us there.”