All processes mature over time, and while aircraft inspection has been honed in many ways over the last few decades, it’s now marking a rather large maturation milestone. 

It’s been about a decade since drones were first used to examine critical infrastructure like bridges, wind turbines and electricity transmission lines—an eye in the sky, if you will—making inspections both faster and safer. 

Now, advancements in automation, camera resolution and machine learning are enabling engineers to take inspection by drones, in combination with ground-based robots, to unprecedented levels. And in this case, into the military realm.

To understand how this all came together, let’s catch up with the team at Asylon, based in Norristown, Pa. The U.S. Department of War recently awarded Asylon a multi-million-dollar contract to develop a system called MARIA (Multi-modal Autonomous Robotics for Inspection of Aircraft). Due to the excellence of Asylon engineers and technicians (more on the short project timeline later), MARIA is already being used for U.S. Air Force aircraft inspections at the Warner Robins Air Logistics Complex in Georgia. 

READ MORE: The Ups and Downs of Drones

On the road to success, Asylon engineers worked out thorny issues related to visual data capture by both ground robots (“dogs”) and drones, integrating the data to provide thorough automated examination of the exterior of aircraft such as C-5 Galaxy and C-17 Globemaster jets, along with C-130 Hercules turboprops. 

To do this, Asylon drew on experience gained from the 400,000+ inspection and security missions already under its belt for clients across infrastructure, logistics, government and commercial sectors. 

But Asylon’s aircraft inspections are completely automated. That is, when an inspection is initiated through previous scheduling or manual start, the dog leaves its “doghouse” and drone its “nest.” Together they walk/fly to the specified aircraft, avoiding any potential obstacles along the way. They then proceed to gather all the visual inspection data required, returning as needed for recharging or battery swap until the inspection tasks for the day are complete. 

Overview: Building a Fully Autonomous Aircraft Inspection System

To provide context for this achievement, Asylon Chief Technology Officer and Co-Founder Adam Mohamed, an MIT graduate in aerospace, aeronautical and astronautical engineering, first notes that robotic inspection in other industries has come a long way. 

“Inspections of bridges, buildings, things like that, robots are doing great job of those today,” he says. “We are now taking it to the next level with a multimodal and fully autonomous remote methodology.  We’re flying and walking next to aircraft worth $100+ million, achieving the equivalent detail of human eyes three feet away. 

“And while we could not do this without advancements in camera resolution, our engineers had to design the hardware, addressing several unique changes so that the dog and drone could carry the required sensors, efficiently hit the target camera angles, vibration is controlled, and so on.”

On the software side, he continues, “the LIDAR/point-cloud data has to achieve potential obstacle avoidance and correct localization of the drone or dog so that their cameras are at the right distances to the aircraft. That and the telemetry and metadata, it all has to all be captured and integrated into a holistic dataset and we create a true digital twin from the sensor fusion of this collected data. All of this is controlled through our software stack, DroneIQ, that’s a ‘single pane of glass’ for both the operator and maintainers.”

READ MORE: Perfecting Humanoids is an Endurance Challenge, Not a Test of Speed

The entire process also needs to function at high temperatures to boot (think of an aircraft on the tarmac on a hot, sunny day)—and completely reliable.

The system had to offer repeatable and consistent outstanding performance, generating point-cloud and imagery data that can be mapped onto each other, with high-enough resolution that an aircraft maintenance engineer can review the inspection data and actually use it to decide what components need to be fixed or replaced and which don’t, at that point in time. 

Trade-offs: Balancing Payload, Performance and Flexibility

As is the norm in any engineering project, trade-offs had to be negotiated by the team—but that came after they gained an understanding of how to intelligently break up the data collection between the drone and the dog. The drone would obviously gather visual data from a top-down view, looking down on areas typically very difficult for an aircraft technician to get to. The dog would meanwhile inspect the sides of the aircraft, the landing gear, underneath the wings and so on. 

“We needed to put the optimal package on board the drone to capture the data we need at range from the target, while also maximizing flight time,” says Mohamed. “There’s uniqueness in that the camera needs a specialized gimbal to allow for the accuracy required for the image capture. We took a stock security drone and had to put on a new payload system. We had to work out how it could carry this much weight on the nose of our drone, how do we supply additional power, how do we make it all work while still maintaining the required flight performance.” 

There were also interesting trade-off challenges, he says, in adding more wire runs, stabilization and more. “Transposing all of this onto a drone is a straightforward but non-trivial task in a lot of ways,” he explains. “The same thing for the quadruped. We had to basically retool our security ‘pup pack,’ rip off the front third of it to put on a new LIDAR system and a camera that is three times the size of our pup pack onto the back of the unit.

“And it had to be done in a way that allows us to encase and protect the camera during operation but also enable the camera to be able to inspect the aircraft at various angles and incidences. It was actually fairly challenging. Sometimes the camera needs to be at 0 deg. from vertical to inspect the bottom of a wing or landing gear and sometimes it’s at 90 deg. Understanding those trade-offs and mechanical optimizations while still protecting the camera was a great exercise for our engineers to go chase after and solve.”

Beyond the AI capabilities, another unique design characteristic sets the system apart. Mohamed says that these robots were designed to be modular from the outset, with the expectation that their use cases would expand over time. “A lot of other systems are designed and optimized for a single payload or camera,” says Mohammed, “which limits the ability to rapidly modify equipment to chase new mission sets and requirements.” 

Holding Steady: Achieving Precision Imaging on Moving Robots

In both the air and ground units, the entire suite of sensors also had to be aligned in such a way that they could be synchronized. Mohammed explains that “there’s the software aspect where they have to adjust and calibrate and all sensors relative to one another, but there’s also just the straightforward mechanical aspects to enable consistent image captures, with the same level of precision required for both the drone and the dog. There were the mechanical constraints to make sure that we hold those tolerances and hold that entire structure in alignment while still in motion.” 

Indeed, the superhigh-resolution camera system had to be isolated from vibration and sway in the drone, as well as from motion-induced disturbances in the robotic dog. While the quadruped moves remarkably smoothly, Mohammed notes that, at its most basic level, it is still “basically punching the ground with its feet.” That reality created a significant challenge for a vision system that depended on image stability. 

READ MORE: A Robot Dog Demonstrates How Mobile Manipulation Affects Automation

“As a result, we had to add a comprehensive vibration isolation system in to enable for the camera to hold steady,” he says. “There were a whole series of experimentations and tests completed to compensate for this environmental condition.”

Still, overall, the drone platform was more challenging than the dog to stabilize, due to its motor vibration and the additional mass. 

But the dog won out in the challenges surrounding another factor. 

Cooling: Managing Heat and Thermal Loads in Autonomous Inspection Robots

LIDAR systems, present in both the dog and the drone, create quite a bit of heat.

Obviously in a drone, you have airflow, but while Mohammed explains that “we knew that quite a bit of cooling would be needed in the dog,” he shares that their initial models “didn't reflect reality.” 

“The dog is doing inspections on tarmac, in potentially very hot weather,” he says. “Ambient weather conditions added an additional requirement and the thermal loads skyrocketed. Initially we were using a passive cooling system and we ended up having to go active, adding fans in addition to traditional heat sinks. The fans were critical.”

An Integrated Team: Fast Iteration Through Tight Collaboration

Robotics projects of this complexity depend on deep integration across teams and technologies. Given the system’s performance, it is little surprise that collaboration was a defining characteristic of the development effort.

In building MARIA, the Asylon electrical, software, mechanical engineers and their support staff continually iterated as one. They gave feedback to each other and together worked through constraints and potential solutions. This tight feedback loop enabled rapid iterations and updates through the prototyping and validation stages of design development.

All told, it was a little under a year to finish development of MARIA. But Mohamed explains that “hardware was probably closer to six months to finish and flight test the new platform.”

“Then it was off to other parts of our team to get the system nominally operational, with feedback and changes made by all team members after that point, eventually making the workflows function seamlessly,” he says. “It’s been a very interesting project and we look forward iterating on this system and also working on other exciting challenges in the future.”

More content from Takeover Week: Automation & Robotics.