Thursday, June 22, 2006
Card Sorting as Cognitive Method: Categories, Structure, or Mental Model?
A recent email list discussion debated the use and merits of card sorting, whether and where it was effective, and where it might be misleading to adopt. As typical, I probably overstated my case as email list discourse tends toward fast responses and exposing rough and initial ideas. But consider how well the email list works as a genre for collaborative discourse. Where the blog is more of a mini-article or opinion piece, the email discussion allows exchange of rough and not-yet-ready ideas, getting some quick feedback and response (or not).
When we choose any method, it should fit an overall research plan consistent with the method and its collected data. But card sorting is often adopted a a kind of panacea - Product managers like the apparent quantification of user information models (like it distances the researcher from the interpretation). Designers like the concrete view of user categories a card sort offers. It is not a miracle method, it should not be used in every information architecture situation. But card sorting has powerful leverage if used correctly and inventively. Also, as Don Norman suggests, it can lead us astray.
Don has recently made the point that it guides design toward a path of the user's information structure, and does not capture their activities. He makes a good point that card sorts (if used to design traditional menu systems in cell phones, for example) may indeed locate functions together logically. But, will this organization of functions support the actual activities people want to do? Menus are a poor way to organize vehicle features, as in the BMW i-Drive example.
We need to select card sorting for its best applications. When using it, we should ask of this or any other method:
What are we actually analyzing? Category preferences, mental models, or an inherent organization within a domain?
Does our sample of users really allow us capture and validate these dimensions?
How well does it compare with other methods? Is it faster (and just as precise) to just compare A-B-C versions of information structures and let participants choose the best-fit? Yes, we pre-construct these paper/visual prototypes, but it is a design process to determine the space of evaluation.
How are we then using the data collected? What does the card-sort tell us - must we further adapt and design from the sort data, or do we take the categories/labels/structures as given?
Sometimes we cannot yet identify whether information structures should be activity or object-oriented. But even if so, card sorting may not be the best way to identify information structures. From a designer's (mine) perspective, it has become over-used (a kind of method panacea) and can be an expensive solution for a simple enough design task. For example: For a corporate website with 5 markets and 5 user segments in each, would you card-sort 5 users from each segment to get a fair reading of the user's information space? That's 5x5x5 = 125 x your 20-230 minutes for each participant.
There are some places where it works very, very well. For example, rank-listing by comparing 20-30 features. And clustering and ranking feature groups can be useful, and done quickly. Ranking is very easy to score, and comparable to other ranking data. Such as interaction logs, survey preference data, and revenue by feature.
When we choose any method, it should fit an overall research plan consistent with the method and its collected data. But card sorting is often adopted a a kind of panacea - Product managers like the apparent quantification of user information models (like it distances the researcher from the interpretation). Designers like the concrete view of user categories a card sort offers. It is not a miracle method, it should not be used in every information architecture situation. But card sorting has powerful leverage if used correctly and inventively. Also, as Don Norman suggests, it can lead us astray.
Don has recently made the point that it guides design toward a path of the user's information structure, and does not capture their activities. He makes a good point that card sorts (if used to design traditional menu systems in cell phones, for example) may indeed locate functions together logically. But, will this organization of functions support the actual activities people want to do? Menus are a poor way to organize vehicle features, as in the BMW i-Drive example.
We need to select card sorting for its best applications. When using it, we should ask of this or any other method:
What are we actually analyzing? Category preferences, mental models, or an inherent organization within a domain?
Does our sample of users really allow us capture and validate these dimensions?
How well does it compare with other methods? Is it faster (and just as precise) to just compare A-B-C versions of information structures and let participants choose the best-fit? Yes, we pre-construct these paper/visual prototypes, but it is a design process to determine the space of evaluation.
How are we then using the data collected? What does the card-sort tell us - must we further adapt and design from the sort data, or do we take the categories/labels/structures as given?
Sometimes we cannot yet identify whether information structures should be activity or object-oriented. But even if so, card sorting may not be the best way to identify information structures. From a designer's (mine) perspective, it has become over-used (a kind of method panacea) and can be an expensive solution for a simple enough design task. For example: For a corporate website with 5 markets and 5 user segments in each, would you card-sort 5 users from each segment to get a fair reading of the user's information space? That's 5x5x5 = 125 x your 20-230 minutes for each participant.
There are some places where it works very, very well. For example, rank-listing by comparing 20-30 features. And clustering and ranking feature groups can be useful, and done quickly. Ranking is very easy to score, and comparable to other ranking data. Such as interaction logs, survey preference data, and revenue by feature.
Then, even if an "object-oriented" design is called for rather than activity, there are other problems with card sorting that I find in practice, and they relate back to theory.
1) Basically, domain experts and customers have different preferences for information structures - which do you privilege?
2) It takes a very large sample to normalize information structures among multiple observations across different samples of the user population. If segmenting the population is important, it can be a nightmare to resolve differences to achieve a common structure.
3) Card sort participants are demand-guided by the card sort options provided. Sorting and clustering are performed on pre-defined categories usually. This is OK if all the items you are sorting are specific and known features to the constituents. But if its a corporate website, "users" usually do not care about the predefined organization labels or elements. People have rival notions for many of the labels we may be sorting, and there's little time to really resolve ambiguity. You can ask people to create their own labels, but "users" are highly variable, and usually people generate weak ideas for new feature labels. OK, maybe one good one per 20 or 30 users. Existing feature, not so bad - its recognition vs. recall vs. formative ideation.
4) People are also bad at freelisting alternatives (as a recall task) - and then freelists end up with multiple labels that must be resolved within categories.
This is enough for now - my intention with this post was to clarify rapid-fire statements made in the CHI discussion list, where much of this argument was thrown together, and in retrospect, looked like a weak argument. Even worse, it looked like I personally disliked card sorts. And perhaps, as with any method, you get tired of the same stuff and want to try new things, and don't get a lot of leeway (time to experiment and develop) in well-defined client projects. So what do you think? How do we advance card sorts and make them interesting again?
1) Basically, domain experts and customers have different preferences for information structures - which do you privilege?
2) It takes a very large sample to normalize information structures among multiple observations across different samples of the user population. If segmenting the population is important, it can be a nightmare to resolve differences to achieve a common structure.
3) Card sort participants are demand-guided by the card sort options provided. Sorting and clustering are performed on pre-defined categories usually. This is OK if all the items you are sorting are specific and known features to the constituents. But if its a corporate website, "users" usually do not care about the predefined organization labels or elements. People have rival notions for many of the labels we may be sorting, and there's little time to really resolve ambiguity. You can ask people to create their own labels, but "users" are highly variable, and usually people generate weak ideas for new feature labels. OK, maybe one good one per 20 or 30 users. Existing feature, not so bad - its recognition vs. recall vs. formative ideation.
4) People are also bad at freelisting alternatives (as a recall task) - and then freelists end up with multiple labels that must be resolved within categories.
This is enough for now - my intention with this post was to clarify rapid-fire statements made in the CHI discussion list, where much of this argument was thrown together, and in retrospect, looked like a weak argument. Even worse, it looked like I personally disliked card sorts. And perhaps, as with any method, you get tired of the same stuff and want to try new things, and don't get a lot of leeway (time to experiment and develop) in well-defined client projects. So what do you think? How do we advance card sorts and make them interesting again?
