Using a card sort test to build a taxonomy

A few months ago I was asked to build a taxonomy for a new wiki portal for informal learning.  The portal was to be an authoritative source of information, knowledge and wisdom for the practioners of our flavor of the Agile software development process.

For content research, I had access to an existing wiki portal which had developed and grown organically. The existing portal was open, and anyone could add, delete or comment on any page. It included a preliminary glossary of Agile terms, consultant documents, team home pages, and much more. Our user research revealed that many learners hesitated to go to the existing portal for information because it was hard to navigate, inconsistent in nomenclature and definitions of terms, and because their feeds showed them that the portal information changed almost every day, every time someone added something new to it.  How can a portal be trusted as a source of truth when it keeps changing? some asked. However, it is the nature of the Agile process to be…well, Agile.

Nonetheless, I needed to find out what were the main buckets into which our learners tended to cognitively toss different aspects of their Agile work.

I began by creating my own high-level categories for the taxonomy, by analyzing what I had seen on the exiting portal. I used Google Analytics, ethnographic field studies (talking to people around the office), and my own thought processes, to come up with this first list of category names:

  • Key Agile concepts
  • Processes/phases of Agile
  • Agile tools
  • Roles and responsibilities
  • Job aids

Then, under each category name, I listed Agile-related terms used on the existing portal, and in the literature of Agile. Then I removed the buckets, and listed all of those terms alphabetically. I had about a hundred terms. The terms were evaluated by several subject matter experts. A few terms were removed as being unimportant to the process, and a few terms were added.

Our internal usability person, let’s call her Grace, conducted five in-person interviews with a people from different teams and with different levels of Agile expertise. She wanted to see what the gaps in knowledge were.

Next, Grace set up a twenty-minute test on WebSort, an online card sorting tool.

Grace emailed an encouraging email to selected respondents, and gave them a deadline. After the date had passed, I walked around the company and visited the people who had not yet completed the test, and invited them to do so.

The test instructions asked respondents to sort my list of 100 terms into categories, and give each category a name.

After the results were in, we analyzed them. Our respondents agreed strongly on certain items being grouped together. Some of their categories matched mine closely, while others didn’t match my groupings at all. The respondents were fuzzy, disagreed, or lacked knowledge around how to group some Agile terms, however. This was useful information to pass on to the people in charge of Agile learning in general.

Grace created a spreadsheet which conveyed our key findings, which we shared with the people who had asked me to do create the taxonomy. We hammered out definitions for some of the terms where our respondents had had disagreed, or just plain didn’t know what to say, so that we were clear about the meanings of most of the key terms.

After we had discussed what we saw, I locked myself in a small room with my laptop and a supply of Diet Cokes. Some time later, I emerged with an eight-page outline numbered taxonomy. This was basically a content outline for the portal, which also could serve as a framework for developing a training program for new developers. My main content buckets did not match what I had started with, nor did they match exactly what the card sort respondents had given me, but that had all been good fuel to create a taxonomy that I felt would be very useful.

While I was at it, I created a rough draft of a glossary which was an attempt to capture all the term definitions I had gained an understanding of over the past several weeks.  I also provided a sitemap and a wireframe which showed a homepage structure for the new portal. One of my top recommendations was that one specific person be presented in the role of “Agile authority” on the portal home page. Another recommendation was that some key pages on the portal be locked down, and that only the “authority” could make changes to them, with the caveat that as our Agile process evolved, Mr. Authority would have to update those pages in a timely way, or else risk losing credibility again.  Much of the existing portal content could be repurposed but some new content needed to be written, and so two tech writers were added to the project. One of the Agile implementation team members took over the task of actually building the portal, which is looking slick in our Confluence Wiki. They are all still at work, but they have done an awesome job of adhering to the spirit of my taxonomy, while adding their own enhancements.

Now, that’s Agile.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s