The problem
An early build was close to shipping, and this was the chance to test it with real pilots first.
The app let pilots view, update and file flight routes, and provided real-time weather updates, time-sensitive safety notifications and Temporary Flight Restrictions for pre-flight checks and route adjustments.
The goal was to learn how pilots actually worked through tasks in the app, find what blocked them and decide what to fix first.
My role and the team
I led the research with three colleagues: a designer, the leader of our customer service center and a product owner who was a trained pilot and former dispatcher.
I wrote the facilitator script and participant guide, organized the on-site session, designed the two-group approach, led the usability testing and brought the prioritization framework.
Three colleagues came with me. The leader of our Honeywell customer service center got us access to the pilots and worked with the designer on the open feedback sessions. I had been working with her team on the flight planning website redesign. The product owner joined the usability sessions.
I ran the research on March 14, 2017, at a corporate aviation fixed-base operator (FBO), a private terminal that serves business aircraft, connected to Vancouver International Airport. It was a working environment where pilots came and went between their actual flight duties. Each pilot came in for a slot of about 45 minutes around their flight schedule. We had planned two days and had to adjust the plan to fit everything into one.
Step 1: Design the study
Two groups and two methods, with controlled access to the build, gave us precision and untainted first reactions.
Eleven pilots participated across the day, six in usability testing and five giving feedback after using the early build. I deliberately split them into two groups, making sure each had a cross-section of newer and more seasoned pilots based on years of flying, years in corporate aviation and aircraft type.
-
Group 1 (usability testing)
I led structured usability testing, observing pilots as they attempted specific tasks with the app and gathering feedback on workflow, design and potential new features. My script covered seven task areas: sign-in, flight planning, trip kits, flight schedule, weather charts, settings and in-flight mode with overlays. This group only received access to the early build after completing their session, so their reactions were uninfluenced by prior use.
-
Group 2 (general feedback)
The designer, working with the customer service center lead, led open-ended conversations with pilots who had already been using the early build, exploring overall impressions of usability and functionality.
Running both methods in parallel let us triangulate: the structured task failures from Group 1 gave us precision, while the open feedback from Group 2 gave us broader context.
Step 2: Run the sessions
Watching 6 pilots attempt real tasks surfaced bugs earlier testing missed and a cruise mode no pilot found unaided.
I invited the product owner to collaborate with us during the sessions. She was a trained pilot and former dispatcher, which meant she understood the domain at a level most product owners don't. Watching the sessions shaped how she prioritized what came next.
Reading a findings report is one thing; watching a pilot you've just met fail to complete a task you thought was straightforward is another.
Pilots worked on an iPad while I recorded the session, with a mobile testing rig capturing their touches, then rated ease and satisfaction. The testing surfaced workflow issues, design problems and bugs that hadn't been caught in prior testing rounds: 25 in all. The clip at the top of this page shows one moment that was hard to ignore: a pilot attempting a straightforward task and not being able to complete it.
Beyond the bugs, we heard consistent feedback about specific flows that weren't matching how pilots actually thought about their work. The designer's open conversations added texture: where the usability tests showed us where pilots got stuck, the open conversations told us why.
Cruise mode surfaced a different kind of problem. Selecting it required a swipe gesture, and no pilot found it on their own: one asked whether it was possible, and the rest had to be shown. The clip below shows the pattern we saw repeatedly.
Session clip: the cruise mode swipe
Pilots also liked what was working, including the snapshot feature and the Temporary Flight Restrictions display, and one called the whole system a one-stop shop for flight planning.
Step 3: Debrief and prioritize
A frequency-by-impact framework gave the product owner a defensible fix order, including a reasoned "not yet."
After the sessions, the product owner, the designer, the customer service center lead and I debriefed on findings from both groups. I brought a prioritization framework to the conversation: each issue was rated by how often pilots hit it and how much it would hurt usability if left unresolved. High frequency plus high impact went to the top of the list. That gave the product owner a defensible basis for deciding what the team would address first, and the designer reworked the flows and screens behind each fix as we worked through the list.
- Bugs first. Those were features that should have simply worked, so they came before any usability fix.
- Two quick wins. Several pilots didn't recognize the trip kit icon until they were prompted, so we labeled it with the words trip kit. We also moved the weather overlay control to the lower right of the screen, where pilots expected to find it.
- The trip kit that did not save. This was the one thing every pilot struggled with. Pilots would add documents, move on and come back to find their additions hadn't saved, and several told us the old app had been easier because it asked what they wanted included. The fix was a confirmation step before generation, showing exactly what was in the trip kit and letting pilots add to it from the same screen.
- Cruise mode without the swipe. The fix wasn't to explain the gesture better. It was to stop relying on it. We introduced a small visual peek, a sliver of the next panel at the edge of the screen, so pilots could see there was more content.
- Not yet. One pilot used a third-party app to annotate flight plans with typed text and asked for the same in the trip kit. It was a real idea, but it had come from a single pilot, and I wanted to see more interest before committing the team's time.
The framework's value wasn't just deciding what to fix; it was giving the team a defensible reason to say not yet to something real.
Outcome
Every prioritized fix shipped, including bugs earlier testing had missed.
From planning the study to the fixes shipping took about three months.
Task-based testing had one limit. The app's tutorial tripped up most pilots, but they had been asked to complete a task, so it was hard to judge whether the tutorial itself worked.
Honeywell discontinued the platform that included the app in 2025, so this is a record of the research and the decisions it shaped, not of a product still in use.
What I learned
Put the decision maker in the room, run two methods in parallel and control who has seen the build.
-
Get the product owner into the session
It is worth the effort every time. Her aviation background meant she could connect each failure she saw to how it would play out in a real cockpit, which made the prioritization conversation afterward faster and more grounded.
-
Run two methods in parallel
Neither the task testing nor the open feedback alone would have given us the complete picture we came home with.
-
Control who has seen the build
Keeping the usability group away from the early build until after their session was a small decision that mattered. Pilots who had used the app came in with formed opinions; the usability group reacted fresh. Mixing them would have muddied both.
-
What I would do differently
I would put more structure into the open feedback sessions. I shared my script with the designer so he could cover similar ground, but the debrief would have gone more smoothly with a shared guide. We had planned two days so we could sync at the end of the first, and we lost that when the plan became one day.