AR/VR Technologies Adopted for First Responders and Emergency Cases

The increase in the amount and type of data related to emergency cases pose continuous challenges to first responders, who mainly consist of firefighters, emergency medical services, and law enforcement agencies. As equipped with specialized skills and qualifications, they are the groups of people, services and organizations whose duty is to arrive first to the emergency zone, carry out rescue operations, and perform crisis management in natural or human-made disasters. Usually, the characteristics and nature of such emergency cases are quite complicated. On the other hand, as the existing information communication technology suffers from fragility and unstable connectivity, the adaptation of advancement of latest computer technologies in the emergency-related scenarios are far from the satisfactory level.

Recently, there emerged some recent achievements focusing on adopting the latest technologies of augmented reality (AR) and virtual reality (VR) to serve first responders. Wang et al. proposed an AR system designed to facilitate the control of rescue robot and explore unknown environments for urban search and save operations¹. Based on simultaneous localization and mapping (SLAM) technology with RGB-D camera, their system first runs to get the position and posture of the rescue robot. Subsequently, they used a deep learning-based algorithm to obtain the target location, place an AR marker in the global coordinate and display it on operator screen to indicate the target even when it is out of camera view. Park et al. presented an AR-based emergency management system for fire safety route guidance². Their system can acquire visibility and grasp occupants in case of fire disasters in buildings, and provide visualization information and optimal guide for quick initial response using smart element AR-based disaster management service through linkage of physical virtual domain in the building. Hu et al. developed a high scene-rendering frame rate to achieve better immersion and prevent users from feeling dizzy in disaster scene simulation³. In their study, they designed a plugin-free browser/sever (B/S) architecture for 3D disaster scene construction and visualization based in mobile VR, and focused on the construction and optimization of a 3D disaster scene to satisfy the high framerate requirements for the rendering of 3D disaster scenes in mobile VR.

Meanwhile, AR/VR serves as an effective approach to reproduce a disaster scene such that information can be extracted from the virtual environment for safety assessment. For this regard, safety training based on AR/VR is another significant application in this field. Virtual simulations of complex situations can enable trainees to grasp a comprehensive understanding of safety issues⁴. Gong et al. designed an earthquake evacuation education system and conducted a virtual dormitory earthquake⁵. Boulos et al. studied the possibility of applying a VR geographic information system (GIS) in emergency training⁶. Pham et al. used VR interactive safety education and building anatomy modeling to instruct students safety operations knowledge⁷. Lovreglio et at. presented a comparison of effectiveness of fire extinguisher trained with VR and video, and concluded that people trained with VR obtained better score in knowledge acquisition and self-efficacy⁸.

On the other hand, AR/VR-based applications gained tremendous attention in the emergency response research area during the last several years. Meng et al. discovered that a virtual environment with smoke and virtual fire can induce participants to experience higher physiological and psychological stress. Their study focused on how to improve the emergency simulation quality in a VR experiment, and better simulate a real fire situation⁹. Bourhim et al. proposed a holistic evaluation system to measure the efficacy of VR simulation of fire, and concluded that VR simulation could be both realistic and engaging¹⁰. Goldiez et al. evaluated rescue navigation training using an AR map. Their results indicated that compared with a traditional paper map or a compass, an AR map can promote wayfinding performance in search and rescue¹¹. Xu et al. used a VR-based fire training simulator for smoke hazards in a fire¹². One of their training scenarios was fire rescue in a primary school, and they utilized VR to train firefighters to choose a safer path to rescue a trapped pupil.

There are also some studies focusing on employing AR/VR technology for post-emergency recovery, which mainly includes damage detection and building reconstruction, to take measures to assess the disaster damage and rebuild the buildings in the affected areas. In such cases, AR/VR can help better understand the condition of a building and plan the way to rebuild it. Dai et al.¹³ investigated the application of AR visualization to inter-story drift measurement through photogrammetry to extract the information from an image and measure the inter-story drift through the generated data. Dong et al.¹⁴ carried out inter-story drift measurement based on AR technology. They used an AR algorithm to superimpose the baseline on a structure and detect building edges to avoid pre-installation of infrastructure. Authors in¹⁵ employed mobile AR for rapid status assessment of a building after earthquakes due to its advantages when measuring structural integrity and specific damage status.

At the same time, some other studies focused on utilizing AR/VR technology in the emergency management field. Visualization enhancement can improve the learning ability of users, transfer the complex knowledge conveniently, and demonstrate abstract concepts to them¹⁶. Such advantages are beneficial to applications in emergency management, such as safety training and hazard identification, because they help participants better understand what my happen and what should be done in emergencies. Together with visualization, AR/VR can also be used as bidirectional interactive tools between humans and computers. Arias et al.¹⁷ used VR to simulate a fire scene in a hotel and provided several items that the participants could interact with, such as chairs that can be picked up and moved, pillows and towels that can be wetted and used to block smoke vents, telephones that produce dial tones, and windows that open and close. Li et al.¹⁸ designed a virtual escape experiment involving people and obstacles when investigating escape route finding behavior. Another important aspect in AR/VR-based bidirectional interaction is observing how the virtual environment influences participant’s behavioral modes and mental states in emergencies¹⁹. Emergencies simulated in a virtual environment is proven to influence participants’ behavioral modes. The basic collection method for behavioral modes includes recording the time, distance, and route in the virtual environment. In addition, the mental state in a virtual emergency scene is also significant for researching the interaction effect.

Besides, there also exist some state-sponsored projects focusing on employing AR/VR technologies in emergency cases. In April 2021, the Research Triangle Institute (RTI) was awarded a $750K to develop a user-centered persistent first responder AR (FRAR) test bed²⁰; they aim to create and support a market for public safety user interfaces (UIs) through improvements of existing AR technologies for firefighters, emergency medical services (EMS), and law enforcement operations and tasks. There are also some commercial-level companies that focus on using AR/VR for first responder training²¹, or search and rescue operations²².

About Sabancı University - BAVLAB

At the national level, SU has received the highest number of EU funded projects per faculty member, taking part in 20 FP6 projects, 53 FP7 projects including 37 Marie Curie Grants, Cooperation Projects and 4 Capacities Projects. In H2020, SU has been involved in 13 funded projects. SU also has a commendable performance in the EU funded education programs, with 5 Jean Monnet European Modules, 3 Jean Monnet Chairs Ad Personam and 1 Jean Monnet Centre of Excellence. These European grants constitute about 18% of the SU’s total budget for research.

Sabanci University’s Behavioral Analytics & Visualization Lab (BAVLAB) will be representing Sabanci University in this call. The main goal of BAVLAB is to conduct research on Big Data Analytics for understanding human behavior and data analytics as well as visualizing Big Data in many diverse settings. Since 2015 the lab has got funded by research grants, corporate funding and sponsorship. The researchers of the lab pursue collaborations or partnerships with companies in different sectors, such as Telecom, Finance, Retailing, Energy and Healthcare, where large datasets or Big Data are involved. The lab currently hosts 2 Faculty Members, 2 Visiting Researchers, 2 PhD students, 6 Master’s students at Sabancı University. In 2020 BAVLAB presented research results at more than 15 international conferences and journals. In this project, BAVLAB with the lead of Prof. Balcisoy will contribute in designing and developing visual analytics systems for large spatio-temporal datasets and trajectories where Prof. Balcisoy and his team has an extensive track record of publications and enterprise level projects.

SU in TeamAware

Based on HMI displaying refined, filtered, and managebale common situational awareness picture, Sabancı University will take part in the development of AR-enhanced user interfaces (UI) to present redefined information to the first responders during the TeamAware project.

Besides, together with other partners, SU will also make contribution to the validation of TeamAware platform in terms of defined performance metrics (correct display of data received from sensors in the field, successful fusion of collected data, resultant Common Situational Awareness Picture approved by first responders) with respect to related requirements.