Recently, one of our clients attended a CLE where a trial consultant was demonstrating the use of dial testing technology as a mechanism to receive mock juror feedback. As some of you may be aware, during mock presentations when using this “clicker” technology, the participants in a jury research project individually turn their dials to reflect how they are leaning every few seconds. The most important advantage of this technology is that it can yield real-time feedback directly linked to presentation content and illustrate trends in jurors’ leanings that are visible via graphs and displays on large monitors in the client viewing room throughout the project. It also looks very impressive and very “high-tech.”
Not surprisingly, our client was immediately wowed by the technology and asked us about it. Our response was that there are certain types of research projects for which dial testing is appropriate and useful and other types of research projects for which the technology is less useful and thus, not appropriate. It all depends on what your research goals are – what you want to learn from your investment in jury research. While the feedback generated by the technology is impressive when viewing the graphs unfold in real time, the real issue is whether it answers the types of questions you have. The use of technology always should be driven by your research goals; the feedback has to be useful – to address what you want to know. Therefore, in this issue of Insights, we explore when dial testing technology is appropriate to use for jury research feedback, and when it is not.
There are advantages and disadvantages to using dial testing to gather data in your research, and there are certain test designs and types of cases where its use is more applicable than others.
Dial Testing in Inductive Focus Groups
Dial testing is best used during focus group research, when the research goals center on the inductive investigation of jurors’ evaluations of the evidence in a non-verdict driven test (a.k.a., a mock trial). In this type of research, the goal is to “cast a wide net” and learn about a broad range of potential responses.
- Beneficial to receive initial and specific feedback. Specifically, if the research goals are to discern what jurors think about certain key issues in a case or to help ascertain jurors’ comprehension, confusion and responses to initial themes and strategies, then dial testing is one way to gain this very specific feedback without interrupting the flow of presentations and to gauge the leaning of the room as a whole. When conducting a focus group, dial testing can reveal trends throughout the presentations and is most effective when it is paired with facilitated follow-up during moderator-guided question and answer sessions. The facilitated questioning is essential in order to ascertain the reasoning behind jurors’ real-time leanings; without this additional information, it is impossible to know why jurors’ views changed.
- Immediate feedback within short time frame. Dial testing also provides accelerated findings for clients. Sometimes the trial team needs global results immediately because trial is just around the corner and there is no time to do a full analysis of results. When global feedback (i.e., this strategy worked, that one didn’t) is needed more quickly, dial testing is a solution that can provide real-time results.
Dial Testing in Deductive, Verdict-Driven Mock Trials/Deliberation Groups
While dial testing can be well suited for inductive focus group research, it is less useful, and thus less appropriate, for deductive, verdict-driven research such as a mock trial or deliberation group. In these tests, one of the primary goals is to learn how jurors evaluate the evidence in a deliberative process. Careful consideration should be exercised to determine if the use of the dial will interfere with the quality of jurors’ feedback, as use of the technology has the potential to diminish that quality. We have identified from our analyses several potential pitfalls of using dial technology in this type of research, including:
- What does the click mean? When jurors are clicking we can see a colorful trend line, but what we are missing behind this line are answers to the most important questions – what does the trend line mean? Or what were the jurors thinking when the line went up or down? It’s not enough to know that jurors are favoring one side or the other, it’s most important to know why they feel that way. As noted above, in a facilitated focus group, we have an immediate opportunity to follow up on jurors’ clicks with the moderator-guided feedback sessions. However, interrupting presentations is not feasible (or desirable) in a mock trial setting where witnesses are testifying or presenters are giving their opening or closing statements. As a result, we have only a trend line and can only surmise why jurors dialed the clicker in a particular direction.
- For example, when delivering his opening argument in a mock trial about asbestos where a plaintiff is suffering from mesothelioma, plaintiff’s counsel divulges that his client is a long-term cigarette smoker, how do we know what jurors’ dialed responses mean? Based on the trend line from the graph, we know that this point has made the group swing in favor of the defense, but we don’t know why they think this way. Was it because juror #2 is a former smoker himself who believes that the plaintiff needs to take more responsibility for his own smoking? Was it because juror #20 lost her mother to a smoking-related illness and hates the tobacco industry? Is it because juror #15 stopped paying attention and is inadvertently moving his dial to the “left” because he is bored? There is more to the story than the quantitative data can show.
- If jurors are clicking, they are not always absorbing. While the human brain is complex in nature and can handle many functions simultaneously, it simply cannot focus on everything all of the time. Recent research on the fallacy of “multi-tasking” and the recent attention to the problem of “texting while driving” are some of the best examples of this phenomenon. When jurors are constantly attuned to dialing their leaning, they tend not to focus on assimilating all of the information being presented to them, which can severely affect the quality of deliberations. Moreover, if jurors are dialing, they are not taking notes. Consequently, it has been our experience that the quality and quantity of the jury deliberations have suffered. That is, because jurors were dialing, most of them had not taken notes or really paid attention to the story or details, which significantly affected the depth and quality of their discussions.
- Complex cases need more attention. In addition to lack of attention to detail, the type of case can have an exponential effect on this pattern of behavior. Complex cases such as patent cases, insurance coverage cases, etc., exacerbate the fact that jurors are focused on dialing and not the details of the case. Jurors’ deliberations after they had dialed during presentations were significantly more superficial than when jurors had been allowed to take notes.
- Response inaccuracy and bias. Occasionally, when using dial testing, jurors get confused and turn the dial in the wrong direction, which leads to inaccurate results. Additionally, some jurors get so wrapped up in a presentation that they forget the dial is there, and thus fail to provide feedback altogether. To prevent this problem, technicians who supply and run the equipment will often offer jurors monetary incentives to keep jurors dialing. This can lead to the opposite effect – jurors who dial frequently for no reason so they can earn the reward.
- Ecological validity of results. Jurors involved in an actual trial do not use dials to send results to the court; therefore, it further reduces the ecological validity of research results when mock jurors are consistently dialing their results to the viewing room. While all applied research contains the caveat of the loss of ecological validity for a variety of reasons, dial testing further complicates an already complicated process.
- Increased cost of research. The investment in jury research is significant; it is expensive to get good, high-quality feedback about your case. The cost of dial testing on top of existing expenses can add $5,000 to $7,000 to the cost of a research project.
At Litigation Insights we have used dial testing for focus groups when our client’s goals are consistent with its use. With our verdict-driven tests, we rely heavily on the use of interim questionnaires throughout research in order to give us both quantitative feedback (jurors tell us first where they are leaning), and qualitative feedback (jurors explain the reasoning behind their leaning). While this approach may not provide the immediate graph generating real-time on the screen or data points provided every few seconds throughout the day, it is this qualitative responses that makes the written questionnaire method for receiving feedback valuable. That is, jurors can offer on their questionnaires what their most salient takeaways were from the presentations, as well as, via their remaining questions, information they feel is still missing. In verdict-driven research, we have found these “lower-tech” questionnaires provide more useful feedback and reduce the need to “guess” about the reasons jurors are clicking the way they are leaning.
So as we have outlined here, there are times to “click,” and there are times not to “click.” It is important for the research team and the trial team to work together to determine the best feedback mechanisms for the project based on your project goals and test design.
By: Merrie Jo Pitera, Ph.D., CEO