Over the past 20 years of providing top-level trial consulting services, we have seen a number of changes to the business landscape. One of the more recent developments is what could be called the “commoditization” of trial consulting services, (i.e., the rise of limited, one-size-fits-all, inexpensive jury research exercises – anything from quick focus groups to online jury surveys.
While we certainly welcome and support making jury research services more widely available and affordable to clients who may have smaller cases that still could benefit from testing the waters before going to trial, flat-fee-style simplification of those services raises some “you get what you pay for” concerns. The reality is that if you want to be able to have faith in the results and recommendations for your case, how the research is designed, conducted and analyzed is critical. So before you sign off on cut-rate services, consider the following issues of methodology and qualification that drive what you pay (or don’t pay) for.
1. Methodology Matters. Most of us intuitively understand that getting a group of your friends together in your living room to bat around ideas about your case hardly generates valid and reliable data that can guide your case strategy. But why is that? It’s because how the research is designed, what stimulus materials are used, what and how questions are asked and how the test data is analyzed all affect the type, quality and accuracy of the information you receive. It’s also because “a group of your friends” is a skewed population that is unlikely to accurately represent your potential jurors. This means you can’t trust that what they say even applies to your case, much less rely on it to shape your case narrative and strategy. And so it is when you go outside your circle of friends and family to conduct jury research: the audience you’re presenting your case to, and the analysis and interpretation of their responses, are of utmost importance when you want answers you can rely on and confidently apply at trial.
Juror Costs: Quality of the Recruit Impacts Quality of the Research.
One of the most important considerations for whether you can have faith in the results of the research is how your consultant recruits mock jurors. There are many ways to recruit respondents for a jury research exercise. There are only a couple of methods, however, that will provide you with a scientifically sound, reliable sample of “jury eligibles.” While an ad on Craigslist, or a flyer posted at a church, or even a call to a temp agency will likely attract a good number of willing participants, the jury sample you will have in attendance is likely to be over-homogenous. Craigslist users, for example, are predominantly male and most are in the 25-44 age range. In addition, these potential jurors typically are not representative of the actual jury pool, a condition necessary to increase the external validity of a test. That is, recruiting respondents that are representative of your jury pool will help you to ensure you are not receiving a false positive and to feel comfortable generalizing from the results. For instance, respondents from temp agencies, for the most part, are not working and that demographic does not even come close to matching the venire you will encounter at trial.
What about online focus groups? Again, you’ll run into some of the same limitations presented by a Craigslist recruit; in addition, many online focus group outfits use the same respondents over and over. The result? Jurors are evaluating your case in comparison with other cases they’ve weighed in on that week (or that day), instead of independently gauging the merits of your case based only on the evidence they see online. Moreover, and more importantly, when repeat respondents are used, we tend to see a “response bias,” a phenomenon demonstrated in years of academic research. A response bias occurs when jurors, who understand how this research process works and assume that one party is sponsoring the study, will tell you what they think you want to hear, not what you need to hear. When this happens, the results are compromised and not reliable. So even with online research, methodology matters.
What, then, makes for a good recruit? Two things: 1) a broad cross-section of jurors that match the trial venue demographically in terms of age, education level and ethnic makeup; 2) jurors who come to the exercise “fresh,” with no prior jury research experience that might color their views of your case. Ultimately, this recruit is achieved through the use of either a random-digit-dial recruit in which market researchers call phone numbers provided to them via a computer-generated phone sample, or through the use of a tightly controlled database maintained by the recruiter. When using the database, it is critical that the respondents be pulled from a “virgin” database file. In other words, the respondents can be individuals who are in the database but who have not participated in legal research previously. Again, the aim is to ensure that the case is evaluated by “fresh” jurors who are not “seasoned” respondents telling you what they think you want to hear or who are comparing the case to the one they sat in on the week before.
Post-Test Analysis: Not Only the “What” But the “Why.”
A second critical reason that methodology matters is that how the information gathered in your research is analyzed and used to help shape the case going forward affects what you learn from your investment. It is important that you find out, not only “what happened,” but also what those results mean and what to do about them in order to better connect with the jurors going forward. It is simple to collect and relay quantitative data – demographics, diachronic juror leanings, verdict decisions – to the client. But what does the data mean? In the absence of supporting qualitative data, you may leave your research exercise with a decent what, but with no why. Qualitative data collection, analysis and synthesis that focuses on jurors’ reasoning behind their leanings and verdict decisions, and is grounded in what mock jurors have said, is essential to a valid and in-depth analysis of the case. This analysis should explore the strengths and vulnerabilities of the case as revealed by juror feedback, along with practical recommendations for themes, story structure, witness order and strategic guideposts that can be applied at trial. Without this qualitative piece, all you have is a pile of numbers.
2. What Is a Trial Consultant? Unlike the practice of law, which requires its practitioners to pass the bar, hold a license and adhere to continuing education requirements, the practice of trial consulting has no formal educational or certification requirements. With such a low barrier to entry, people from many related disciplines – court reporting, trial technology and yes, lawyers – are holding themselves out as trial consultants. While the “one-stop shop” setup may be enticing, the downside is that few of these folks have the training or experience in social science, quantitative and qualitative methods that would enable them to provide externally valid research designs, proper questionnaire design and reliable analyses. For instance, as many lawyers know, the way a question is worded can impact the response given. In jury research projects, asking jurors leading questions is imposing a confirmatory bias on the data received. Generalizations from this data are suspect at best. Court reporting and trial technology are valuable litigation support services; expertise in these areas, however, does not make for a qualified trial consultant.
We are all familiar with the saying, “You get what you pay for.” If cost is the sole driving factor in choosing jury research services, there is something out there to fit every budget. But from the standpoint of validity of that jury research, there is another saying that comes to mind: “Caveat emptor.”
By: Robert Gerchen, Senior Consultant