Timothy Mills

DHS Firefighter Augmented Reality Heads-Up Display

Cockpit Screenshot

The challenge from DHS was to create a heads-up display for firefighters that would aggregate a number of current and future data feeds to improve the situational awareness of emergency responders working inside a fire.

Contextual inquiry and ethnographic research

A typical firefighter SCBA mask.

Firefighter demonstrating standard personal protective equipment (PPE). Firefighters demonstrating hazmat PPE. The view through a thermal imaging camera.

We started the project with a two day research visit to the Champaign, IL Fire Department (CFD) to conduct some contextual inquiry and ethnographic research. During our visit, we spoke with and observed a number of firefighters with varying years of experience, from new recruits all the way up to the Assistant Chief and Chief. We covered everything from basic firefighting tactics, to detailed examinations of various personal protective equipment (PPE), to proper techniques for using thermal imaging cameras, to the perceived aversion to technology in the fire service, to the basics of HAZMAT response. We supplemented our visit to the CFD with phone and video interviews with firefighters from Arizona, California, and South Carolina along with visits to the Illinois Fire Service Institute and the Chicago Office of Emergency Management and Communication.

Augmented reality mask interface

After synthesizing everything we’d learned from our research with firefighters along with reviewing existing solutions, we started work concepting a next-generation heads-up display that would use augmented reality. Our hypothesis was that using augmented reality would allow us to create a much more immersive, contextual solution to increase the situational awareness of a firefighter. We created a data categorization system to help ensure the right information was displayed at the right time. Constant data included things like air remaining, ambient temperature, and egress points. On-Demand data included elements like thermal imaging or the current radio channel. Alerts were reserved for data that a firefighter would need to act on immediately, like an evacuation call.

Augmented reality glove interface


DEXTAR-2-Menu DEXTAR-3-Selection DEXTAR-4-Detail

In our discussions with firefighters we heard repeatedly that they do not have the time or patience for complex interfaces. Additionally, the personal protective equipment (PPE) that firefighters wear greatly reduces their dexterity, making it difficult for them to operate any interface with small buttons or controls. With in mind, we wanted to create a much more natural and easy way to interact with the heads-up display that would not interfere with the job a firefighter is asked to do. Our solution was to display the user interface on the firefighter’s glove through augmented reality, relying on a small set of intuitive gestures to control the UI.

Concept video

The project ended with a final report to DHS outlining our vision along with this video we created to help bring that vision to life. This video starts with a loud fire engine siren.