DualMode Outward-facing Communication

2018 - 2021

Strategy to instinctively communicate an AGV's condition and expected user action via lights and sound to humans in the environment.

Project Overview

Crown AGVs are designed to co-exist with human workers in the environment. It needs to be able to communicate its condition to the surrounding in order to prompt the desired response from people. This is a goal that the team believes can be best achieved via non-verbal means such lights & sound on the AGV.

Role & Team

I worked as part of a design team consisting of myself and 2 other designers with varying degrees of involvement throughout the project, and also in collaboration with the wider vehicle engineering teams. I was responsible for defining the foundation for the communication strategy, that is later tested and refined before eventually being handed off for implementation.

The problem

Crown's line of 'DualMode' AGVs are designed to operate in a shared environment with human workers. Despite being able to function safely and autonomously around people, the lack of a human operator on the vehicle takes away the ability for people in the surrounding to predict the AGVs intent, condition, and consequently what is expected of them in response to what the AGV is doing. We believe the AGV should pro-actively offer the missing communications to the surrounding in order to compensate.

Announcing AGV presence, condition, and intent to the surrounding

Expected response from people in the surrounding may range from leaving the AGV as is, approaching the AGV to help, or staying away as the AGV needs to room to move or maneuver. These expected responses need to be effectively communicated by the AGV to people in the surrounding; including those who may not be within distance to see the on-truck display and those without direct line of sight to the AGV.

Lights and sound communication on the AGV is perceived to be an ideal way to prompt such responses from people; as the message can still be conveyed without direct line of sight to the vehicle, and can be instinctively understood due to immersion with machines and technology in our daily life.

The effort

I worked as part of a design team across New Zealand and Germany in defining the communication strategy, and later on with the human factors team in North America in testing the strategy with vehicle operators. As a design team we also had to work very closely with the vehicle development teams in order to understand different AGV states, and the risk factors involved which determine the necessary communication to people in the surrounding.

I was responsible in laying down the overall foundation for the outward-facing communication strategy, but my involvement tapered down to a supporting role towards the user testing phase to make room for other projects.

Parts of the project that I led:

  • Initial research

  • Auditing AGV workflows & physical vehicle behaviours

  • Identifying desired user response in relation to vehicle behaviour

  • Translating desired response into lights & sound specifications

  • Interfacing with stakeholders & cross disciplinary teams

I supported in the following aspects:

  • Prototyping lights & sound communication in a simulation (test material)

  • Testing lights & sound strategy with vehicle operators

How should the AGV communicate its condition to the surrounding?

There was a philosophical decision that we had to make early in the effort, which may sound trivial but is important in guiding the decisions we make moving forward- what communication paradigm should we adopt in communicating AGV condition to people in the surrounding?

AGV as an 'equipment', or another 'co-worker'?

Will an autonomous vehicle be perceived as a moving piece of equipment, or ‘humanized’ and seen as another co-worker by people in the environment? This was an important question to answer as the expectation translates to how dynamic and ‘alive’ the communication needs to be, and subsequently the choice of hardware to implement it.

We mounted an action camera on a prototype AGV during a test run at an internal Crown manufacturing plant to gauge people’s reaction and very quickly learned that it was treated differently to a human-operated vehicle. It was seen as an unfamiliar piece of equipment, and given a wide berth from people in the surrounding.

GoPro footage from prototype AGV at Crown manufacturing plant

Precedence from the industry

Once it became clear that the AGV is perceived as a moving piece of equipment, we then looked into precedence in communication from machinery and technology in warehouse and manufacturing environments. This a a 'language' that users in the AGV's eventual habitat are very familiar with, and we decided to leverage them as much of them as possible.

Communicating AGV Condition

We used precedence set by stack lights commonly used in industrial machineries to communicate the AGV's condition. This is a language that the end users are familiar with, and also an internationally recognized industry standard.

Green means safe, machine operating normally.

Yellow means warning, something different is about to happen or has happened.

Red means fault or emergency, machine needs help.

Communicating movement intent

This is a case where we shouldn’t re-invent the wheel. We adopted the precedence set by the automotive industry in communicating a vehicle’s movement intent- particularly around turning and hazards.

What to communicate, and why?

The diagram below illustrates the typical sequence in an AGV workflow. In addition to movements such as travelling and turning, there are other actions that are not typical for passenger vehicles such as waiting for server-assigned order and forks movement when engaging with a pallet. These are all actions and conditions that the AGV needs to communicate to the surrounding at some point in the workflow.

We then performed a full audit of the AGV’s workflow sequence and micro-actions within them to identify any associated risk to people in the surrounding (as the AGV performs the sequence) and as a result, the expected user action/response to it.

AGV Workflow & Sequence Audit Result

A preliminary strategy

Result from the audit became guiding principle for the preliminary outward-facing communication strategy. This was our best guess at what the AGV needs to communicate to the surrounding and the ideal means to do so.

Preliminary Outward-facing Communication Strategy

Capturing the essence of ‘urgency’

From the audit we also learned that certain sequence actions carry greater risk than others. The message that the AGV needs to convey may range in urgency from “pay attention” to “please stay away from me”. We figured that the urgency of a message can be conveyed through a combination of colour, and firing interval (frequency) of both lights & tones.

Bespoke tones for AGV conditions

We commissioned an audio design agency in North America to create bespoke tones representing each one of three AGV conditions: Travel (good), Caution, and Error (bad). These tones are then paired with each individual AGV communication identified from the audit.

Click on the thumbnails below to preview the AGV tones.

Travel Tone

Caution Tone

Error Tone

Translating the strategy into physical design

The preliminary outward-facing communication strategy were then presented to the Design Team in Germany, who then proceeded to incorporate them in the AGV design.

Some notable decisions made on the physical AGV design directly informed by our strategy:

  • Separating status & indicator lights to create hierarchy

  • All-around visibility for travel indicator lights

  • Projector to indicate floor allowance for AGV

Testing the preliminary strategy

We conducted remote user testing sessions to evaluate the intuitiveness and comprehension of our strategy with 27 participants (mix of truck operators and floor personnel) from Crown's manufacturing plants in Germany & North America.

Each 30 minute session consists of video simulations of the AGV performing various actions with the corresponding lights & sound, followed by an interview to understand the participant's interpretation of AGV communication.

This effort was led by the Human Factors Team in North America, with support from myself and the NZ Design Team.

General Comprehension 1

General Comprehension 2

Colour Interpretation

Perimeter Confusion

What we learned from testing..

From testing we learned that users were able to correctly interpret the message conveyed by the simulated AGV lights & sound for the most part, except for the 'AGV Event' state communication where the AGV tries to communicate an error that it is experiencing and therefore asking for help.

Turning off the projected perimeter light on the floor during the 'AGV Event' state was an attempt to 'invite' people towards the AGV, but was unexpectedly interpreted as the vehicle 'powering down' therefore causing confusion for some participants.

This informed us to keep the perimeter light on in communicating the 'AGV Event' state, as it is seen as a primary indicator of the AGV's operation mode.

The DualMode Outward-facing Communication Strategy.

By collaborating with cross-regional design and engineering teams we were able to form, test, and refine an outward-facing communication strategy for the AGV to convey its condition and intent through lights, sound, and motion combined.

These non-verbal means of communication are designed to be instinctive- leaving little room for misinterpretation, and causing minimal cognitive load in prompting the desired response from people (users) in the environment .

Outward-facing Communication Strategy 'Spec Sheet'- lights & sound to prompt desired user response.

Anatomy of on-truck hardware responsible for outward-facing communication

AGV in stationary position and silently indicating that it is waiting for an order.

AGV warning its surrounding to keep a safe distance as it is about to begin automated travel.

AGV announcing its presence to the surrounding as it travels autonomously.

AGV calling out for help to remove an object in the detection field.

To sum it up..

The DualMode 'L' model AGV is still in development, with a few prototype models running around for testing at the New Zealand facility. We have handed off the outward-facing communication strategy to the vehicle development team who are implementing the strategy one piece at a time, in keeping with their development timeline.

I find this to be one of the more enjoyable projects in my time at RTC, as it allows me work up close and personal with AGVs that the team have done a very great job in developing. The term 'UX' is synonymous with experience when interacting with a digital user interface, and I feel very fortunate to I have what feels like a rare opportunity to work on the physical, non-UI side of the experience. This sits very well with me, having coming from a physical product background.

This project gave me an appreciation to the amount of care and consideration that needs to be put into designing (and engineering) the interaction between a human and robot, both from a user experience and legal perspective. I look forward to the opportunity of being involved in similar projects in the future.

Have a peek at my other work!