Fleet Management System

Lead Product Designer

Robotic automation for challenges in the logistics industry

Most of the consumer products we use today require long term storage - an essential step of the product supply chain. Large amounts of boxes with consumer goods are arranged on pallets designed for easy picking and placing by forklift trucks.

Recently, the warehouse industry has experienced labor shortages and increasing operational costs leading logistics companies to explore opportunities to automate.

Robot Fleet Management System

I joined Third Wave Automation (TWA) to help the Product team bring to market an autonomous forklift service. The objective of this service was to address the labor challenges and the increased operating costs by reducing the reliance on manual forklift drivers and making warehouse operations more efficient.

The solution I designed was a Robot Fleet Management System which we called Armada - a user interface where one or more operators can supervise and teleoperate multiple (up to 10) autonomous forklifts.

Human in the loop

As it often happens with automation, the autonomous agents run into unexpected situations they cannot handle on their own. With Armada, we put a human in the loop - an operator can remotely address any issues that can’t be resolved autonomously by the robot forklift.

Fleet Management View

A key attribute of the product was the Fleet view module – the main touch point between the robot forklift and the operator. This view provided operators with the ability to supervise, manage, and assist multiple autonomous forklifts. It provided up-to-date information about each truck’s position and its status in relation to the warehouse.

Context, control, and visibility

Three main design goals emerged for Armada after I conducted contextual inquiries and worked through design iterations. It became clear we need to provide operators with:

1. sufficient context,
2. control, and
3. visibility over their fleet.

The majority of operators sit far from the actual warehouse floor, lacking the spatial awareness, visual, and auditory cues a truck typically provides to a manual forklift operator. Knowing where the autonomous forklifts are, where they are going, their status, and the reasons behind that status became crucial for the design to address in order to enable effective fleet management.

A key feature of the Fleet view that I designed was the warehouse map and the digital representation of the forklift, visually representing the actual truck’s telemetry data within that map. This representation provided operators with at-a-glance information about position, orientation, and status.

The status was indicated by a colored display to show four main states: autonomous, assisted, ready for receiving tasks, and stopped. It closely related to the halo lights that we installed on the forklifts to denote each of those states for staff that may be in physical proximity to a truck.

Each forklift was also represented in a List view panel within the Fleet view. Each item within the list communicated a truck’s status very quickly through the use of a colored status pill component.

I also designed a sliding detail panel for each item that provided additional details and actions for each truck. The panel facilitates design scalability - additional details and actions were easily added in subsequent product iterations.

I designed the map and the List view to be interconnected – selecting a truck from the List view would smoothly reposition the map’s view to that truck’s location and encapsulate its current location, planned path, and destination. This interface behavior provided the context and visibility required by our operators. The design pattern also provided a quick way to navigate from one vehicle to another - a necessity when an operator has to oversee multiple trucks.

Situational awareness with Uber-style pickup patterns

To further improve the operator’s situational awareness, i.e., the ability to answer the question “What are my trucks doing now” in real-time, I designed a fleet map experience akin to an “Uber pickup”. Ridesharing apps like Uber and Lyft give you real-time updates on where your ride is in relation to your pickup location. The vehicle’s current location and ETA are conveyed visually by gradually reducing the dimensions of the map view as the vehicle approaches your location - a very intuitive and informative design pattern.

Using the “Uber pickup” pattern, I designed an experience where operators can effortlessly track at any moment the location and destination of each truck. In earlier versions, our operators often complained and demonstrated it was hard to track the forklifts. They had to constantly click on a truck, drag, and zoom the map to get an overview of its movement. Implementing this new pattern resolved this limitation.

Uber-style pickup pattern: detailed design specs and prototyping

Applying the “Uber pickup” pattern for multiple vehicles in an industrial setting required a lot of exploration and very detailed animated prototypes in order to convey the desired interface behavior and micro interactions. I designed, documented, and collaborated with Engineering on several different map behaviors covering key truck status cases:

1. when trucks are traveling at a significant distance from one another,
2. when they are in close proximity,
3. when a truck has requested intervention (assist request).

Assists and Remote Control (Teleoperation)

Although the goal of any automation system is to remove the need for human intervention (also known as Lights-out Automation), there continue to be cases, especially with less mature autonomy systems, where a robot cannot handle certain situations. When such situations arose with Armada, operators were signaled by receiving an assist. Assists could be resolved with a single click for simple confirmations or by entering Remote Control in more complex situations.

Assist Panel

A key challenge with Remote Control was that it decreased operators’ visibility to the entire fleet. So, although operators could always access it to intervene, we (the product team) wanted to reduce the amount of time spent in that mode. This prompted my design of the Assist Panel - a panel displayed within Fleet view that provided contextual information and relevant quick actions for resolving assists without losing visibility of the full fleet.

To keep the Assist Panel in a more compact form and reduce the cognitive load of operators, I designed the Assist Detail Panel that expands on demand. This pattern was also tightly coupled with the map to provide additional context to the operator - selecting an assist from the list not only expanded the Detail Panel but also centered the map’s view over the truck which raised the assist. This interface behavior provided additional context to operators—they could see the truck’s location, orientation, and current stage of task execution (e.g., navigation, lining up to pick or place).

Entering Remote Control

When an assist can’t be resolved quickly from the Assist Panel, operators can get full control over the robot using teleoperation, which we called Remote Control.

Visual prioritization of camera feeds

Remote Control (RC for short) provided multiple camera feeds - feeds for navigation and feeds for fork control. With earlier versions of RC, I quickly realized that dealing simultaneously with so many cameras created a cognitive overload for the operator. Additionally, the operator station’s ultra wide monitor was causing ergonomic issues.

During field studies, I witnessed how operators kept twisting their heads to the right and straining their eyes, in order to observe the auxiliary feeds situated on the side of the interface. This was most pronounced when they performed pick or place operations. I addressed the ergonomic issues by designing a more dynamic RC view that adjusts the visual priority of the feeds based on the task stage or at the press of a button by the operator.

Prioritizing for pick-and-place

For RC pick-and-place operations, I designed the layout to prioritize the camera feeds most useful for forks visibility and manipulation. I used visual hierarchy through scale and positioning to aid the operators’ focus. The perspective forks camera—the primary camera in this view—provides significant detail on the forks’ positioning relative to the pallet’s pockets, enabling greater precision. Operators also gain better visibility of the pallet’s clearance from the bottom and sides of the rack, ensuring safer extraction or placement procedures.

Collision Shield - collision avoidance system

Safety was one of the primary pillars of the autonomous forklift service. One of the features in RC dedicated to this effort was a collision avoidance system I branded as Collision Shield.

Any time an operator got too close to an obstacle or a structure, Collision Shield engaged the brakes. To convey this within RC, I designed an experience consisting of the toast notification and control-disabling patterns, which were already familiar to our operators from previous releases. The toast appeared as soon as the truck was stopped and controls were disabled. This created a strong connection between the truck’s behavior and the interface feedback.

The toast explained the reason for the truck’s behavior and presented an option to disable the protection. Disabling the protection was an undesirable behavior as operators were no longer shielded from manipulating the forklift unsafely. Therefore, I designed extra friction - the operator was forced to use the cursor on a small target to select that option and explicitly confirm their intention.

Forks Assistant

After field studies and contextual inquiries following the initial release of the RC feature, it quickly became apparent that users were struggling with visibility and spatial awareness of the forks - they were often unsure which rack level the forks had reached. Additionally, they relied solely on visual approximation from the camera feeds to determine forks and pallet clearances from the rack, which was not considered a safe method.

To address these teleoperation challenges, I conceptualized and designed the Forks Assistant. The Forks Assistant displayed a minimalist interface of the truck’s mast and its forks. As the operator raised the forks, the assistant appeared in place of the warehouse map to show the precise vertical position of the forks in relation to the rack level. Extending the forks into the rack led to a display of the location’s upper and lower boundaries with additional visual aids to enable swift and safe pick-and-place procedures.

The Forks Assistant provided the needed visibility and precision for our operators to feel confident when performing pick-and-place operations in RC. The feature successfully reduced safety accidents and made the RC experience significantly better.

Conclusion

Robot fleet management systems are very complex and require close collaboration between roboticists, systems engineers, interface engineers, and product designers. The autonomous forklift service introduced a lot of design challenges related to operator context, control, visibility, and safety. Using field studies, contextual inquiries, usability testing, and design iterations, I created an interface that enabled our operators to effectively use robotics automation to complete real world warehousing tasks.