Using a card sort test to build a taxonomy

A few months ago I was asked to build a taxonomy for a new wiki portal for informal learning.  The portal was to be an authoritative source of information, knowledge and wisdom for the practioners of our flavor of the Agile software development process.

For content research, I had access to an existing wiki portal which had developed and grown organically. The existing portal was open, and anyone could add, delete or comment on any page. It included a preliminary glossary of Agile terms, consultant documents, team home pages, and much more. Our user research revealed that many learners hesitated to go to the existing portal for information because it was hard to navigate, inconsistent in nomenclature and definitions of terms, and because their feeds showed them that the portal information changed almost every day, every time someone added something new to it.  How can a portal be trusted as a source of truth when it keeps changing? some asked. However, it is the nature of the Agile process to be…well, Agile.

Nonetheless, I needed to find out what were the main buckets into which our learners tended to cognitively toss different aspects of their Agile work.

I began by creating my own high-level categories for the taxonomy, by analyzing what I had seen on the exiting portal. I used Google Analytics, ethnographic field studies (talking to people around the office), and my own thought processes, to come up with this first list of category names:

  • Key Agile concepts
  • Processes/phases of Agile
  • Agile tools
  • Roles and responsibilities
  • Job aids

Then, under each category name, I listed Agile-related terms used on the existing portal, and in the literature of Agile. Then I removed the buckets, and listed all of those terms alphabetically. I had about a hundred terms. The terms were evaluated by several subject matter experts. A few terms were removed as being unimportant to the process, and a few terms were added.

Our internal usability person, let’s call her Grace, conducted five in-person interviews with a people from different teams and with different levels of Agile expertise. She wanted to see what the gaps in knowledge were.

Next, Grace set up a twenty-minute test on WebSort, an online card sorting tool.

Grace emailed an encouraging email to selected respondents, and gave them a deadline. After the date had passed, I walked around the company and visited the people who had not yet completed the test, and invited them to do so.

The test instructions asked respondents to sort my list of 100 terms into categories, and give each category a name.

After the results were in, we analyzed them. Our respondents agreed strongly on certain items being grouped together. Some of their categories matched mine closely, while others didn’t match my groupings at all. The respondents were fuzzy, disagreed, or lacked knowledge around how to group some Agile terms, however. This was useful information to pass on to the people in charge of Agile learning in general.

Grace created a spreadsheet which conveyed our key findings, which we shared with the people who had asked me to do create the taxonomy. We hammered out definitions for some of the terms where our respondents had had disagreed, or just plain didn’t know what to say, so that we were clear about the meanings of most of the key terms.

After we had discussed what we saw, I locked myself in a small room with my laptop and a supply of Diet Cokes. Some time later, I emerged with an eight-page outline numbered taxonomy. This was basically a content outline for the portal, which also could serve as a framework for developing a training program for new developers. My main content buckets did not match what I had started with, nor did they match exactly what the card sort respondents had given me, but that had all been good fuel to create a taxonomy that I felt would be very useful.

While I was at it, I created a rough draft of a glossary which was an attempt to capture all the term definitions I had gained an understanding of over the past several weeks.  I also provided a sitemap and a wireframe which showed a homepage structure for the new portal. One of my top recommendations was that one specific person be presented in the role of “Agile authority” on the portal home page. Another recommendation was that some key pages on the portal be locked down, and that only the “authority” could make changes to them, with the caveat that as our Agile process evolved, Mr. Authority would have to update those pages in a timely way, or else risk losing credibility again.  Much of the existing portal content could be repurposed but some new content needed to be written, and so two tech writers were added to the project. One of the Agile implementation team members took over the task of actually building the portal, which is looking slick in our Confluence Wiki. They are all still at work, but they have done an awesome job of adhering to the spirit of my taxonomy, while adding their own enhancements.

Now, that’s Agile.

Conceptual Graph Structures: Part 4

Conceptual Graph Structure

Hand-drawn Conceptual Graph Structure by Scott Confer

The Secret Sauce

In past posts, I have outlined the components of the type of mental model called Conceptual Graph Structures. There are six kinds of nodes, connected by 18 types of arcs, which indicate semantic relationships between the nodes.  I have talked about the basic CGS substructures: taxonomies, goal heirarchies, causal networks, and spatial relationships. I have given you cheat sheets for the “legal combinations” of nodes and arcs for each substructure type.

But let’s say you are on a new project of some kind or another, and all you have to work from is your rough notes from a kickoff meeting, and an sketchy set of project requirements which your project manager dashed off last night between dinner, and his 9 pm appointment to read The Phantom Tollbooth to the kids before bed.

So where do you start?

Take your rough notes and the requirements document and lay them out on the dining room table. Read them both through once, identifying key pieces of information:

  • who are the learners, or people who are going to use your work?
  • who are the actors (the people who are going to do the work on this project)?
  • what is the context?
  • what transformation is needed?
  • can you pick out any taxonomies (like a navigation system, or a group of roles, for example)?
  • does a goal or two jump out at you?
  • can you see any cause and effect relationships?
  • are there any spatial relationships described?

Now you’ll need a big, clean piece of paper and a pencil. Sentence by sentence, identify nodes (concepts, states, styles, events, goals, and goal-actions) and how they could be connected, using any of the 18 types of arcs.  Draw these nodes and arcs on the piece of paper. You’ll wind up with something that looks sort of like the drawing above.

My colleague Scott Confer (who drew the CGS above) calls this method the “secret sauce” of making Conceptual Graph Structures (CGS). It’s a great way to kick-start an Agile project.

Other posts on the topic of Conceptual Graph Structures

Sallie Gordon-Becker, working with colleagues, developed the Conceptual Graph Structures (CGS) process. The arc structure is drawn from Arthur Graesser’s research into how people tell stories. The CGS guides, templates and instructions for the use of Conceptual Graph Structures were developed by myself and my colleague Scott Confer.The Visio stencil was developed by myself, Scott Confer and Andrew Rice.

Conceptual Graph Structures: Part 2

You can jumpstart Agile IxD processes with concept graphing

As an Agile interaction designer, you need to be able to quickly represent problems.

Using the lightweight concept graphing approach I described in yesterday’s post on Concept Graphing, you can rapidly make mental models. This gives Agile teams a metacognitive scaffolding for user-centric solution design.

Semantic relationships and built-in grammar make Conceptual Graph Structures unique. Nodes and arcs can only be connected in “legal” ways, an approach derived from cognitive task analysis and research in story telling.

As I described yesterday, six types of agency can be linked via eighteen interaction types to indicate relationships (e.g. “event” and “goal” connect via an “initiates” arc). Workflows, taxonomies, domains, goal hierarchies, causal relationships, and more are built by simply snapping arcs to nodes.

At a high level, understand what you’re going to build
You can use the process of building  revisits requirements throughout the lifecycle of a project.

It’s not a quick and easy, down and dirty approach. There is rigor to it, a formal process. It requires research with end-users, and analysis using cognitive tools that have to be learned. However, it is collaborative, and it works.

Download a presentation on how concept graphing fits into an overall requirements elicitation process.  I delivered this presentation with colleague Scott Confer at the Information Architecture Summit 2007 in Las Vegas, and then again in Barcelona at the EuroIA 2008

Other posts on the topic of Conceptual Graph Structures

Sallie Gordon-Becker, working with colleagues, developed the Conceptual Graph Structures (CGS) process. The arc structure is drawn from Arthur Graesser’s research into how people tell stories. The CGS guides, templates and instructions for the use of Conceptual Graph Structures were developed by myself and my colleague Scott Confer.The Visio stencil was developed by myself, Scott Confer and Andrew Rice.