When the Problem Wasn’t Where the Client Thought It Was
Summary
Challenge: Support desk staff could not find policies on the website quickly enough to answer caller questions.
Role: UX Research Lead / Sole Researcher - I selected the research methods, designed the studies, identified where users were getting lost or confused, synthesized the findings, made decisions that led a second round of testing and contributed to final recommendations.
Decisions: I recommended eliminating the section of the website the client originally wanted us to fix. Then we redistributed its information across the site. Content was moved to the places where the research showed users were already looking for it. I collaborated with a content lead and lead designer to develop a new information architecture for both the main menu and internal pages.
Method: Both rounds of testing included tree testing, first-click testing, and card sorting to identify where users were getting lost or taking indirect paths.
Outcome: After restructuring the navigation and shortening user journeys, we validated the changes with real staff. Task success for the support desk audience increased from 36% to 60%.
Research
The Problem Was Bigger Than One Section of the Website
The client serviced a variety of people but even internal office workers were having an issue finding policies to answer their website visitor questions. Due to all the services provided, they also had thousands of PDFs housed on their site.
Questions to answer:
What kept website visitors from completing tasks without calling the office?
Where were internal office workers struggling?
How confident were both sets of users in getting things done using the website?
Initially, the client wanted to know what could be done about the Rules & Policies section of the website, but I zoomed out further to include all of the site to see if the Rules & Policies section might be a symptom with a larger cause.
This diagram has been simplified and anonymized to protect client privacy.
My Role and Responsibilities
Leading the Research From Discovery Through Validation
I served as Lead UX Researcher guiding the full discovery-to-validation process by:
Planning and conducting generative and evaluative research.
Running tree testing exercises to find-out where users were getting lost.
Running card sorting to refine the navigation through the users language.
Analyzing heat map and first click data to examine behavior.
Partnering with our designer to wireframe and implement design changes.
Leading usability retesting to confirm improvements and measure success.
Presenting research finding and recommendations to agency stakeholders.
Methods & Process
Choosing Methods to Find Where Users Were Getting Lost
Heuristic Evaluation: I started here to identify any low-hanging fruit that could be fixed right away.
Analytics Review: This was to determine where research might begin
Talking with Stakeholders: The client’s site serviced three different audiences: General citizens who pay taxes and own cars, business owners and the clients internal employees. Although the focus was on why support desk users were not finding niche information, I didn’t want any site re-organization to negatively affect the other audiences
Recruiting: It relied on a mix of internal users given to use by the client, and purchasing a test group of general residents within the state of Georgia and business owners within the state of Georgia.
Vetting Purchased Recruits: Whenever an individual tester’s results showed a combination of unusually short answer times plus incorrect answers, I marked those as “junk data” and were given replacement testers from Optimal Workshop.
Testing (Tool: Optimal Workshop): I used a variety of tests (card sort, tree test, first click / heat maps) with a focus on information architecture and page layouts. Because internal users relied on the site daily and were familiar with it, I focused on identifying recurring patterns in where they became lost or took indirect paths.
Re-testing: I recruited new participants from each identified audience segment, including car owners, business owners, and internal support desk users. Then I conducted additional information architecture testing to evaluate the revised structure.
Round One: Key Takeaways
Users Were Looking Somewhere We Didn’t Expect
One of the most fascinating discoveries to me was that we were originally asked to only focus on one navigation section. And it turned out, everyone was skipping over that section completely.
Through testing, users also told us they look for “details”, but their test behavior showed us they actually clicked on “general” labels.
So I tested again with a new menu information architecture catering to the three audiences.
I collaborated with my team’s Lead Content Strategist who used my initial test results and created a new proposed menu structure pictured below.
This diagram has been simplified and anonymized to protect client privacy.
Reducing the Number of Steps Between Users and Their Goal:
New page layout suggestions were developed by our Lead UX Designer.
Our UX designer structured pages so it took two steps to get them off the home page to the next step in their journey.
These were the Category and Subject pages.
And then two steps to continue on to complete a task.
These were the Service and Details pages.
This diagram has been simplified and anonymized to protect client privacy.
Round Two: Did Changes Make a Difference? Yes!
Removing a Dead-End Section Changed User Behavior
By eliminating the Rules & Policies section that users were not visiting and redistributing its bulletins into the categories where users expected to find them, users stopped taking detours. This contributed to a significant 24% increase in users who took a direct path to complete tasks successfully.
The change is visible in the green path images below. Both images show results for the same task. In Round One, users followed numerous incorrect branches, and deeper analysis revealed frequent backtracking. In Round Two, substantially more users followed the correct path to the destination, although there was still room for improvement near the very end of the journey.
This diagram has been simplified and anonymized to protect client privacy.
Success Jumped From 36% to 60%
The light blue bars represent the average success scores for round 1.
And, the dark blue bars represent the average success scores for round 2.
Notice in the second set of bars, there is a dramatic 24% increase between the round 1 and round 2 success scores for the Internal audience. This boosted an extremely low 36% success rate to 60%. The internal audience is the same audience that triggered the usability testing and the results had a negligible impact on the other two audiences.
Turning Research Findings Into a Simpler IA
Eliminate the Rules & Regulations Page
Re-distribute information to the places where users are actually looking for them, instead of where the agency assume they will look.
Shorten the Main Menu from 8 Items to 6
The new main menu items: Taxes, Motor Vehicles, Alcohol & Tobacco, Unclaimed Property, Local Services, About The “Client”
Simplify User Journeys
Shorten the number of clicks required to find a desired page.
Examine New Content Before Creating New Pages
Is this task related?
If yes, can it be added to existing instructions or should it be a new content page?
Did this question come directly from a user?
Can you start with content that exists and find out what’s keeping it from answering the user’s need satisfactorily?
Organize Pages in a Common Flow
Categories guide the user to several appropriate subjects
Subject pages can point the user to several services
Service pages can help a user complete a task or direct the user to more details
Detail pages can help a user complete a task or read the most specific information on a topic
Shorten Pages Where You Can
Users found things more easily on shorter pages.
Emphasize the most important links with an icon or button and place them near the top of the page.
Use graphs to shorten content if feasible.
Complex content, like step-by-step instructions, can be shortened with the judicial use of accordions.
Results & Impact
A 24% Increase in Successful Task Completion
The task success rate increased from 36% to 60% in the target group with minimal impact on the additional user groups who also use the website.
Reducing top-level navigation choices and re-distributing content, immediately improved task-completion speed.
Reflection
I’m Proud of Our Team
I’m proud of the research I led in collaboration with our Lead Designer and Lead Content Strategist. We reached our goal of making rules and policies easier for the appropriate audience to find. And we gave site editors a solid framework to use in the future when adding and evaluating content.