Overview of UX Testing

How I discovered UX Testing

I had been building websites for over 10 years before I got to observe another person using one of them. I was working at a company that decided to embrace a User Centered Design strategy, and I was sent to Texas Tech University to become certified in UX testing.

During the certification course, the first test that we conducted was on a website that allowed teachers to manage their class resources. It was a project that I had worked on for almost a year. It was absolutely shocking, and gut-wrenching, to watch as our test users struggled to figure out how to use the site. They were confused by things that never occurred to me as difficult when I was programming the site. Very few of them could complete even the simplest tasks without some difficulty. It was a terrible feeling for me to consider that in spite of all my experience as a web developer, I wasn't sure if I had ever built a useful site. So this is how my journey into UX testing began. Here is a summary of what I have learned about it.


UX Testing

  • Interesting historical note on UX testing

    Like many other advancements, UX testing can be traced to military necessity. Before WWII, heavy machinery was built with the premise that it would require specialized training to operate. In the U.S., during the rush to ramp up the military before entering the second world war, there was an unfortunate trend in which pilots were dying, not because of equipment failure, but because they didn't know how to use the equipment. It was recognized that there wasn't enough time to select and train people to use complicated machinery, so the machinery had to become easier to use. The problem: How to make a tank so that an Iowa farm boy could use and maintain it in short order.

  • Usefulness = Utility + Usability

    Utility = what your product is capable of doing
    Usability = how easy it is to utilize the product

    There's a bit of a balancing act here, because in many cases ease of use increases as utility decreases, and vice versa. So if we are focusing on the usability aspect of a product, we often strive for simplification. This requires us remove any features that are not necessary and may actually complicate the usabaility of a product.

  • What UX Testing is NOT

    UX Testing does not solve usabililty problems, it exposes them.

  • The Process
    • Find representative users to test
    • Ask them to do representative tasks
    • Observe what they say, but more importantly, observe what they do (because you can't always trust what users say; more on that below).
    • Identify problem areas and make adjustments
    • Repeat
  • UX Testing Commandments
    • Know thy users - When running tests, it is critically important to include real users. Asking your cousin, or friend to be a subject out of convenience will render your tests worthless.
    • Know that thou are NOT thy users - don't even begin to pretend that you know what will be easy or hard for users.
      Avoid ego centric design, which is what happens when you design to satisfy your own personal assumptions rather than the needs of the people what will use your product. 
      Side note: here's what happens when you let programmers design UI
    • Let users be users - don't try to steer users in any direction when conducting tests. If they are struggling to complete a task during a test, and they ask you 'what should I do now?', your response should be something like 'what do you think you should have to do know?'. You'll learn more by observing them struggle. A very helpful tactic is to ask test users to 'think out loud' and listen to their comments as they try to complete the tasks that you assign during the test. 
  • Users Lie

    They won't intentially lie, but you can't always trust what they say. During testing, when you ask them if something was easy or hard, they might say it was easy because they're worried about feeling stupid (after all they know they're being tested, and nobody wants to fail a test). You have to keep this in mind when conducting tests. You can't solely trust verbal feedback from users, you have to observe their actions to determine if a task was easy or difficult for them.

  • What makes a product 'usable'
    • Memorable
    • Efficient
    • Error free*
    • Learnable
    • Satisfaction

    *Note: UX errors are different than functional errors. A UX error occurs, not when a bug appears, but when the user takes an action is not part of the most efficient path to acheiving a task.

  • There's little use in UX testing at the end of the product development cycle.

    The cost of fixing a problem when the project is already complete is very high. UX Testing should be a part of the design process, problems should be exposed as early as possible.

  • How do you test something on users before it's completed?
    • Low Fidelity Prototypes
    • High Fidelity Prototypes

    When dealing with software development, a low fidelity prototype can be as simple as a set of paper diagrams that can be presented to your test users. These paper prototypes are very cheap and quick to create, so there's not a lot of up front investment. When you show them to users you can catch problems very early in the development process. Labeling systems (for example, the text on buttons or in links) are a good example of the most basic UX features that can be tested with paper prototypes.

    As your design process continues, you create higer fidelity prototypes, such as wire-framed web pages (which include minimal graphic design and may be without some features that will be in the final design).