This is a post about the limitations of one of my most-preferred research methods: the elite-level interview. In particular, I want to talk about the problems that can crop up when we construct illustrative case studies based solely on a few elite interviews. It’s a common practice – you can find it even in some of the finest academic books. But it’s a practice that often leads to hollow case studies.
I was thinking about this methodological issue last month, while I was reading Steven Schier’s 2000 book, By Invitation Only: The Rise of Exclusive Politics in the United States. Schier offers an engaging and provocative argument about the difference between mobilization and activation. For Schier, activation consists of “identifying and activating the small segments of citizens most likely to ‘get the message’ and vote or lobby government” (page 1). This definition of activation sounds an awful lot like present-day mobilization to me, and indeed this is his point. Political mobilization today involves far more targeting than mass mobilization by 19th century political parties. The tools and techniques of mobilization have become more fine-grained. In the process of developing more efficient techniques, we have lost some of the democratically-enriching value that comes from political participation. It’s an important argument, even more salient today than when it was written and Schier makes it well.
In chapter 5 of the book, titled “Interest Organizations and Government: Lobbying By Activation,” Schier provides some descriptive cases to show how interest groups are employing activation strategies in their work. These case studies were constructed on the basis of elite interviews with multiple senior staff members at each organization, conducted in 1997. And one of those cases was the Sierra Club.
In his five-paragraph description of the Sierra Club, I identified five (well, four and a half) errors.*
Some context: In 1997, I was entering my freshman year at Oberlin College and had just been elected Chair of the Sierra Student Coalition’s (SSC’s) national executive committee. I had already spent two years within the vast volunteer bureaucracy of the Sierra Club. I would go on to spend another 13 years collecting various titles within the organization, including 6 on the board of directors. I was also an avid Sierra Club history buff in college.
Suffice it to say, these are not errors that the average reader would pick up on. They are items where a quote from the interviewee is slightly misinterpreted. They are descriptions of committee names and governance processes that don’t quite fit. They are not errors that undermine his argument.
But therein lies the problem. If you can make as many errors as you have paragraphs without changing the contribution to your argument, then how important can the case example really be to your thesis? What’s more, when I think back to those years, a better case example jumps immediately to mind:
In January 1998, I attended my first “winter gathering” with the SSC executive committee. We spent much of that weekend retreat discussing the unfolding saga of “The Checkbox.”
In the late 1990s, most Sierra Club member recruitments and renewals occurred through direct mail. For a while, that mail included a “student” checkbox. Between 10,000 and 30,000 people would check that box in a given year, thus paying the (lower) student rate and ostensibly joining the SSC as members. But recent mail tests had showed that removing the checkbox increased direct mail response rates. Member Services removed it without telling the SSC. All of a sudden, our “membership” rolls crumbled.
The SSC also had a network of active groups at high school and college campuses. These were our “activists,” rather than our “members.” But our annual funding was linked to membership, not activists. So we either had to get the checkbox back or get the funding mechanism changed. This was a matter of organizational life-and-death. We spent years working on it. And it was all a byproduct of the very “activation” strategies that Schier is calling our attention to.
The Checkbox would have made an excellent illustrative example for Schier’s chapter. By 1997, membership in groups like Sierra had become a thin and transactional relationship. These groups also supported political activism, but they did so by cultivating a small core group of devoted participants. Yet the single sentence that Schier devotes to the SSC – “Sierra also maintains a ‘student coalition’ of some 10,000 members that are ‘somewhat active.” – conveys none of this information. Instead, it is barely recognizable to me as a former participant.
This is too common an occurrence in qualitative, case-based research. Brief case examples, based on a few elite interviews, fill out pages without telling us anything of much substance. The problem isn’t the interviews (Schier interviewed the right people). The problem is that it is just interviews.
If we want to understand how Sierra, or the NRA, or AARP, or any other organization engages their supporters, then we need to triangulate from multiple reference points. Read the minutes from board meetings. Analyze newsletter content. Read the magazines and listservs that organizational activists participate in. Then draw upon these reference points in your elite interviews. You can also share preliminary findings with them, to find out what you’re getting not-quite-right and dig deeper into key terms and concepts. This triangulation (a variant on what Richard Fenno called “soak and poke” research) helps us move from thin description to thick description.
Now, Schier probably wouldn’t have stumbled upon The Checkbox as his guiding example of activation in practice. We were a bunch of scrappy undergrads, and probably would’ve been intimidated if a political science professor had wanted to interview us anyway. But he would have found some other robust campaign or controversy. That controversy could serve to demonstrate the normative problems of “activation” strategies, while simultaneously rendering Sierra in more recognizable terms. Five paragraphs of surface description, touching on governance committees, ballot initiatives, membership levels and the SSC tells the reader barely anything at all. The interviews fill pages, but shed little light.
By Invitation Only is serving as my example here because Schier happened to study my organization, but also because of the considerable strength of his book. Thin illustrative case examples are everywhere in the literature, including dozens of lesser books and articles far more deserving of critique. But my point here is that even otherwise-excellent research often stumbles through this methodological pothole.
The measure of a descriptive case study should be just how much does it describe? Thin description, based on a few interviews, can rarely reach the complexity and nuance that we aim for in qualitative political communication research.
Elite interviews are a necessary tool for producing rich case studies. But they are hardly sufficient.
*The four and a half errors were:
(1) “A smaller set of 5,000 members, known as the “core group…”’ A staff member may have used this term, but it was not in common usage at that time.
(2) “One-fifth of the organization’s membership is in California, where the socially oriented Sierra Singles attracts many members.” The membership is 1/5th Californian because the organization was founded in California (by John Muir, who is memorialized on the California quarter) and grew in diasporic fashion in the 1960s, 70s, and 80s. Sierra Singles has never been a particularly popular program.
(3) “The governance committee of Sierra’s national board of directors has formal authority over the organization’s policy direction.” The Board delegated authority to six governance committees in 1997 (Conservation, Organizational Effectiveness, Training, Finance, Outdoor Activities, and Communication & Education). These committees passed recommendations up to the Board, which maintained formal authority and frequently challenged govcom recommendations.
(4) “If 2,000 members sign in favor of a referendum on an issue, a ballot goes out to all members.” A ballot goes to all members every year. The number of signatures required for a referendum fluctuates, depending on the number of ballots cast in the previous year’s election.
(1/2) “Sierra also maintains a ‘student coalition’ of some 10,000 members that are ‘somewhat active.’” See above.