��������

Engineers at the University of Waterloo have developed a new system that will use augmented reality (AR) and virtual reality (VR) to help extend the lifespan of Canada’s critical infrastructure.

The cutting-edge system, called the Smart Infrastructure Metaverse, enables inspectors who are onsite and others who are off-site to use the technology to view a real structure and 3D-scanned replica at the same time, enabling them to collaborate more effectively on possible fixes.

“The Smart Infrastructure Metaverse is all about making it easier for onsite and off-site inspectors to work together on structural inspections,” says Chul Min Yeum, a professor in the university’s department of civil and environmental engineering who worked on the system.

“We’re using AR/VR headsets to give them a shared view so they can see exactly where they are and what they’re looking at during the inspection, no matter where they’re located.”

Yeum worked on the project with Zaid Abbas Al-Sabbag, a professor in the same department, and Sriram Narasimhan, of the department of civil and environmental engineering at the University of California Los Angeles.

Onsite inspectors are typically limited in what they can observe when inspecting bridges, roads, towers and pipelines because they can’t always see or detect every potential problem within large infrastructure.

Visual inspections have constraints when dealing with large, complex structures that are difficult to access, and training inspectors is costly and time-consuming.

However, the new system produces more rapid, comprehensive and complete results than traditional onsite visual examinations and a more complete context for structural issues. Advances in computer vision and AI will enable the inspectors to automatically assess the condition of structures from the data captured using optical sensors.

Headsets equipped with a holographic display allow onsite inspectors to visualize digitized information overlaid on the structure. The scene of the structure is viewed through 360-degree panoramic images aligned with a pre-built map using a multi-shot localization algorithm.

The headsets are used with artificial intelligence to track the positions of inspectors and enable them to analyze structural images in real time. They enable users to see and interact with virtual content overlaid and blended on the real scene as holograms.

Recent advancements in head-mounted display resolution, positional tracking, hand tracking and on-board computation have allowed the headsets to be used more widely in research applications. Lately, the concept of multiple users interacting synchronously within a 3D virtual environment, often referred to as the Metaverse, has become more commonplace.

The technology allows for a more robust inspection when dealing with challenging environmental conditions or unforeseen defect types.

Since 2020, the project has been funded as part of the Rogers-University of Waterloo Partnership, identifying it as a compelling use case for 5G technology and mobile edge computing.

The research team, consisting of both onsite and remote engineers, tested their system on an in-service, reinforced concrete railway bridge in Kitchener, 20 metres long and 3.4 metres high, that had two large damage regions of half-a-metre each located at the deck and pier of the structure.

An off-site engineer wearing a VR headset explored the 3D model of the bridge, similar to the way someone would navigate a video game. While this was happening, onsite inspectors wearing AR headsets could, through a holographic display, see the actual bridge, the VR user, and additional information within a digital map.

The team created a three-dimensional model of the structure using 3D scanners and a panorama camera. The model allowed accurate tracking of the location and head position of both the onsite and off-site inspectors on the digital map. The off-site inspector was able to see the exact locations of the onsite users along with the areas of the structure they were inspecting.

The setup meant the remote inspector could ensure the data collected was accurate. To back up the human inspectors, the research team used artificial intelligence to analyze the images sent from the AR headsets of the onsite inspectors to further identify any structural damage.

With much of Canada’s infrastructure having been built in the mid-20th century and nearing the end of its lifespan, the team’s solution addresses an urgent public safety risk.

Public infrastructure in Canada, excluding social and affordable housing, had a total replacement value of $2.1 trillion at the end of 2020, according to Canada’s Core Public Infrastructure Survey.

The cost to replace all assets rated in poor or very poor condition was estimated at slightly more than 10 per cent of the replacement value – at $264.7 billion or $16,252 per private dwelling.

Roads accounted for most of the infrastructure in need of rehabilitation or replacement at 48.1 per cent, followed by wastewater infrastructure at 13.9 per cent and potable water infrastructure at 11.1 per cent.