Recently, I spent a week preparing for and moderating concept tests for one of my largest clients. Concept testing—which is different from usability testing—puts two different website design treatments in front of representative users to see if one concept is perceived to be more usable or engaging than the other. Once a design is chosen and client-approved, the build phase can begin.
Users’ opinions about graphic design are obviously very subjective. Just as people have different tastes in music, food and home décor, so too do their preferences differ online. When asked to compare two designs side by side, participants may have a difficult time articulating what it is that they like about a design. Even if they can find the words, people’s rationale for their tastes is likely not related to the business or brand goals at the core of a design concept.
The fact that someone dislikes bright colors doesn’t help us as researchers to assess the emotional impact of an interface or how it aligns with intended brand attributes. Therefore, we must create tests that try to structure participant feedback, and remove some of the subjectivity that accompanies their varying tastes.
First, begin with the participant recruit. As in other user research, it’s important to make sure you’re testing with a representative user population. If possible, recruit from a customer list or develop a screener to filter out participants who are wildly dissimilar from your typical user base.
Next, draft a discussion guide. Your discussion guide sets the objectives for your test and then outlines the activities and questions that the moderator will ask the participant during the course of the research. Having a script will ensure that each research session stays on track and that each participant experiences roughly the same stimulus.
I generally like to break the discussion guide down into components so that each piece feels like a separate activity or game, keeping participants engaged and cooperative throughout the session. This is particularly important in concept testing where all questions are repeated for both concepts. (To control for bias, don’t forget to alternate which concept is shown first to each participant.)
Next, we move forward with a 15-second test where I show the participant a mocked-up homepage design (on screen) before turning off the monitor. Then I ask the participant what he remembers about the page, what stood out and what types of tasks could be accomplished given the page’s functionality. This exercise helps confirm whether or not users are seeing and internalizing your page’s main messages and top calls to action. Because participants have limited interaction with the design, they are providing you with a glimpse into their first impressions.
After turning the monitor back on, the participant and I talk further about the homepage design—things he particularly likes or dislikes, content that doesn’t make sense or seems misplaced, font sizes and readability, and modules that should be added or removed. By asking if anything should be added or removed, you’re also asking about the amount of content on the page. If it feels too busy, participants will let you know.