Boosting a Success Rate 24%
Summary
Overview
Challenge: Support desk could not find policies on the website to answer caller questions
Role: UX Research Lead / Sole Researcher - I selected study types, created studies, to determine where callers were getting lost or confused. I synthesized insights to make decisions for a second round of testing.
Decisions: I tested again after eliminating a section of the website and re-distributing the section’s information across the website. This time information was placed where data told us, users were looking for it. This included collaborating with a content lead and UX designer to design a new information architecture for both the menu and internal pages.
Method: Both rounds of testing included tree testing, first click testing, and card sorts to track where users were getting lost.
Outcome: After restructuring the menu and shortening page clicks, the changes were validated with real staff. Support desk call success rates jumped from 36% to 60%.
Research
What Was the Issue to Be Solved?
The client serviced a variety of people but even internal office workers were having an issue finding policies to answer their website visitor questions. Due to all the services provided, they also had thousands of PDFs housed on their site.
Questions to answer:
What kept website visitors from completing tasks without calling the office?
Where were internal office workers struggling?
How confident were both sets of users in getting things done using the website?
Initially, the client wanted to know what could be done about the Rules & Policies section of the website, but I zoomed out further to include all of the site to see if the Rules & Policies section might be a symptom with a larger cause.
My Role and Responsibilities
Key Responsibilities
I served as Lead UX Researcher guiding the full discovery-to-validation process by:
Planning and conducting generative and evaluative research.
Running tree testing exercises to find-out where users were getting lost.
Running card sorting to refine the navigation through the users language.
Analyzing heat map and first click data to examine behavior.
Partnering with our designer to wireframe and implement design changes.
Leading usability retesting to confirm improvements and measure success.
Presenting research finding and recommendations to agency stakeholders.
Methods & Process
Process Overview
Heuristic Evaluation: I started here to identify any low-hanging fruit that could be fixed right away.
Analytics Review: This was to determine where research might begin
Talking with Stakeholders: The client’s site serviced three different audiences: General citizens who pay taxes and own cars, business owners and the clients internal employees. Although the focus was on why support desk users were not finding niche information, I didn’t want any site re-organization to negatively affect the other audiences
Recruiting: It relied on a mix of internal users given to use by the client, and purchasing a test group of general residents within the state of Georgia and business owners within the state of Georgia.
Vetting Purchased Recruits: Whenever an individual tester’s results showed a combination of unusually short answer times plus incorrect answers, I marked those as “junk data” and were given replacement testers from Optimal Workshop.
Testing (Tool: Optimal Workshop): I used a variety of tests (card sort, tree test, first click / heat maps) with a focus on information architecture and page layouts. Because internal users relied on the site daily and were familiar with it, I focused on identifying recurring patterns in where they became lost or took indirect paths.
Re-testing: I recruited new participants from each identified audience segment, including car owners, business owners, and internal support desk users. Then I conducted additional information architecture testing to evaluate the revised structure.
Round One: Key Takeaways
Main Points to Address in Round Two Testing
One of the most fascinating discoveries to me was that we were originally asked to only focus on one navigation section. And it turned out, everyone was skipping over that section completely.
Through testing, users also told us they look for “details”, but their test behavior showed us they actually clicked on “general” labels.
So I tested again with a new menu information architecture catering to the three audiences.
I collaborated with my team’s Lead Content Strategist who used my initial test results and created a new proposed menu structure pictured below:
New Page Layout Suggestions:
The new page layout suggestions were conceived by our Lead UX Designer.
Our UX designer structured pages so it took two steps to get them off the home page to the next step in their journey.
These were the Category and Subject pages.
And then two steps to continue on to complete a task.
These were the Service and Details pages.
Round Two: Did Changes Make a Difference? Yes!
Suggestions That Worked Best for the Internal Group (Our Target Group)
By eliminating the Rules & Policies section that users were not visiting and redistributing its bulletins into the categories where users expected to find them, users stopped taking detours. This contributed to a significant 24% increase in users who took a direct path to complete tasks successfully.
The change is visible in the green path images below. Both images show results for the same task. In Round One, users followed numerous incorrect branches, and deeper analysis revealed frequent backtracking. In Round Two, substantially more users followed the correct path to the destination, although there was still room for improvement near the end of the journey.
Bar Chart Comparison
The light blue bars represent the average success scores for round 1.
And, the dark blue bars represent the average success scores for round 2.
Notice in the second set of bars, there is a dramatic 24% increase between the round 1 and round 2 success scores for the Internal audience.This boosted an extremely low 36% success rate to 60%.The internal audience is the same audience that triggered the usability testing and the results had a negligible impact on the other two audiences.
Evidence-based Recommendations
Eliminate the Rules & Regulations Page
Re-distribute information to the places where users are actually looking for them, instead of where the agency assume they will look.
Shorten the Main Menu from 8 Items to 6
The new main menu items: Taxes, Motor Vehicles, Alcohol & Tobacco, Unclaimed Property, Local Services, About The “Client”
Simplify User Journeys
Shorten the number of clicks required to find a desired page.
Examine New Content Before Creating New Pages
Is this task related?
If yes, can it be added to existing instructions or should it be a new content page?
Did this question come directly from a user?
Can you start with content that exists and find out what’s keeping it from answering the user’s need satisfactorily?
Organize Pages in a Common Flow
Categories guide the user to several appropriate subjects
Subject pages can point the user to several services
Service pages can help a user complete a task or direct the user to more details
Detail pages can help a user complete a task or read the most specific information on a topic
Shorten Pages Where You Can
Users found things more easily on shorter pages.
Emphasize the most important links with an icon or button and place them near the top of the page.
Use graphs to shorten content if feasible.
Complex content, like step-by-step instructions, can be shortened with the judicial use of accordions.
Results & Impact
Success Rates Increased, and Tasks Were Completed Faster
The task success rate increased from 36% to 60% in the target group with minimal impact on the additional user groups who also use the website.
Reducing top-level navigation choices and re-distributing content, immediately improved task-completion speed.
Reflection
I’m Proud of Our Team
I’m proud of the research I led in collaboration with a team of content editors and designers. We reached our goal of making rules and policies easier for the appropriate audience to find. And we gave site editors a solid framework to use in the future when adding and evaluating content.