Much like Skittles, augmented reality (AR) comes in many different varieties, and mixing those “flavors” together can sometimes lead to very fruitful outcomes. At Pack Expo International's 2020 virtual conference, Alexander Ouellet, innovation engineer at Harpak-Ulma described these varieties and their use spectrum in packaging facilities.
The first step in implementating AR is defining it and deciding where it fits in relation to other reality technologies on the market. The Milgrum Extended Reality Spectrum is a base framework to understanding what these technologies are and where they line up with reality.
“At a high level, extended reality—or XR—refers to all the real and virtual environments generated by computer technologies and wearables,” Ouellet explained. XR includes virtual reality (VR), augmented reality (AR) and mixed reality (MR). “It’s important to understand that the defining trait of these experiences is not necessarily the content, but how the user is going to experience it.”
He described the next step into the XR landscape after VR is augmented: virtual reality (aVR). With this technology, the developer takes content from the physical world and adds to a virtual experience to give the end-user a more immersive feel. Examples that Oullet gave were 360-deg. videos and photos, as well as pre-recorded audio.
Assisted reality, Oullet pointed out, is on the closest end of the spectrum to reality. Notable examples of this technology are Google’s Glass and RealWear headsets. “Assisted reality refers to projecting additional information into a user’s field of vision—importantly, hands-free,” he said. “It differs from augmented reality in that it doesn’t change what the user is seeing, nor is it aware of the geometry of the user’s environment.”
AR is the next step on the spectrum, which is defined as an overlay of computer-generated content on the real world that can superficially interact with the environment in real time.
Mixed reality (MR), an overlay of synthetic content anchored to and interacting with real-world objects in real time, is in the middle of the spectrum. Oullet explained that in MR, computer-generated objects are visibly obscured by objects in the physical environment.
“When implemented well, MR should be a perfect union of digital and physical content interacting seamlessly,” he said.
So, what can this technology do?
Oullet said that AR (referring to all technologies above) comes in three flavors. The one most familiar to people is video see-through technology (VST), in which users can view through a smartphone or tablet rather than a headset. An industrial example of this is PTC’s Vuforia Chalk, in which the view on a tablet allows users to view graphics over what they were seeing.
Another flavor is optical see-through technology (OST). These lens-based technologies, like the Microsoft HoloLens, project AR information through the lens; the users’ brains then snip the information together to create an augmented view of reality.
The last flavor Oullet detailed was projection AR, where projected images mapped onto a specially designed work surface creates a direct overlay on the part where the user is working.
“The object’s end-users can move around in the environment, but the AR experience is limited to the fields of view—both of the fixed projector and the supporting camera for tracking,” he said.
Now we have varieties and flavors. How can the packaging industry mix-and-match this technology to use it to its full potential?
“Staff can be guided through how to get a machine back up-and-running, perform a frontline changeover or be alerted to a performance issue using AR technology,” Ouellet noted. “We see the AR value proposition as delivering significant—and I mean substantial—productivity gains in human labor.”
Depending on the application, he said AR technology can result in a 30 to 60% increase in human productivity, which is being tested through Beta customers. He also referenced an Iowa State University study that showed performance improvements when a worker completed complex tasks using AR.
Stumbling blocks remain, however: Many companies still don’t know where they should start with AR, and as with many innovative technologies, there are pain points. “One of the biggest challenges we see in packaging is infrastructure,” Ouellet said. “The devices that I believe best enable AR are relatively expensive, so there needs to be a plan around how you manage those and how you deploy them.”
Another challenge OEMs face is the high-quality 3D data requirement for AR systems. Ouellet explained that good engineering data lacks texture and visual characteristics, contains too much extraneous data and may unintentionally expose IP. He noted the only data needed are the elements an OEM wants to augment.
Integrating AR with machine data is also a challenge because it allows the AR experience to be scalable enough for a facility, but it can still relate specifically to an individual machine.
Ouellet said the biggest challenge of all is building out different AR experiences. “Building an AR experience is unlike any typical engineering development pipeline most industrial organizations are used to,” he said. “It’s more similar to a combination of software development and video game development, where engineers build functionality and infrastructure while developers utilize a game development engine like Unity.”
A mixture of AR technologies to complete traditional industrial tasks can leverage the technology to help productivity and reduce downtime in a number of ways. Ouellet highlighted some use cases:
- Augmented reality for remote service. If a machine breaks, a technician can guide a worker via a mobile device or headset through a repair.
- Factory acceptance tests (FATs). Virtual FATs create documentation and capture it in AR. Technicians wear headsets that detect data while the machine is running, including HMI output and customized validation criteria.
- Augmented reality service work instructions. AR is used to provide real-time worker guidance during operation.
- IoT service diagnostics. Leveraging AR for diagnostics allows staff to view performance data as the machine runs.
“Making this happen takes a village, as they say,” Ouellet said in closing. “Making all of this real required us to evolve into smart, connected packaging platforms first and, second, to employ application tools designed specifically for the industrial AR experience.”