• Jan 26, 2021

Fool’s Gold: How RCT Research Harms Communities Impacted by Criminal Punishment

On January 24th, Law360 published a story quoting our Staff Attorney, Katy Naples-Mitchell, entitled, “Studying Justice Or Hurting It: The Fight Over A2J Research.” Her quotes, while accurate, were significantly abbreviated and do not fully capture our concerns and critiques about the research, which is rather glowingly portrayed in the article. The reporter who wrote the piece had solicited comments by email in response to five questions. We have uploaded her full comments in response to the reporter’s questions here.

Interview about Randomized Controlled Trials in “Access to Justice” Research:

1. What are your objections to the use of randomized controlled trials in access to justice research? Is your objection that RCTs in this context are unethical or that they’re ineffective at addressing the root causes of injustice in the legal system? Or is there another reason?

All of the above: ethics, effectiveness, but also overall political framework. RCTs are targeted technocratic interventions chiefly concerned with efficiency, which fail to confront or redress the ways in which criminal punishment is built on a foundation of racism, fear, inequity, and disparate access to power. My ethical concerns are more fully explained below. I’d also mention that the criminal system is designed to express a community’s values; criminal law proceeds from moral and political questions. Its normative framework is somewhat inapposite to these kinds of technocratic evaluations. If we believe there is a right to counsel, randomizing access to lawyers is not and cannot be part of the solution. Ensuring that everyone has counsel is and should be – investing philanthropic funds that would otherwise go to support research at elite institutions would be a good start and a good example. And finally, no one required an evidence base to implement the policies that vastly expanded policing, jails, and prisons over the last fifty years. I am wary of calls from elites to now require causal evidence for interventions to begin to undo systems of harm which disparately target Black, Latinx, and/or poor people.

2. Is it ethical to randomize access to interventions like legal aid or bail money for those dealing with the criminal justice system?

One chief concern is the lack of equipoise in these kinds of studies. For example, abundant research already exists showing that pretrial incarceration causes harm to detained people and their loved ones, is criminogenic, results in worse case outcomes for accused people, destabilizes lives, interrupts social supports, and is extractive and costly. Denying a control group of people pretrial release, something we know will help them, and subjecting them to something we know hurts them in the interests of research is ethically insupportable. Instead of randomizing access to interventions for which there is already an evidence base in order to prove the causal mechanism with greater precision, we should be spending our efforts, and our funds, on implementing those policies as widely and quickly as possible.

And again to revert to the normative framework: if we believe that accused people are innocent until proven guilty beyond a reasonable doubt, a bedrock principle of our constitutional system, then pretrial detention is fundamentally inconsistent with and violates that principle. It is a legal fiction that pretrial detention is merely administrative, whereas post-conviction incarceration is punishment. In many states, including here in Massachusetts, people are sent to the exact same correctional institutions whether pretrial or post-conviction, subject to the same set of rules, restrictions on their liberty, banal brutality, and traumatizing indignities, including expired food, squalid conditions, deficient medical care, and invasive strip searches — and today, danger of COVID-19 and its effects. To know this, and then to watch researchers approach detained people about gaining their consent to participate in such a study raises its own subset of serious ethical concerns. To argue that a 50-50 chance of obtaining relief is informed consent is cynical at best, bordering on cruel.

3. What do you say to researchers who argue that since there are limited resources, such as help with bail or free legal services, it is more ethical to randomize who gets that help than to make those choices based on other factors?

The issue of “limited resources” is a scarcity created by human decision-making and political will. There never seem to be limited resources when it comes to making arrests, initiating prosecutions, or incarcerating more people. There is no such thing as a natural experiment in the realm of public policy–all public policy is mutable, and particularly so if reform efforts are backed by substantial investment. Instead of funding a study to determine whether–or, given existing research, by how much–access to counsel at arraignment improves system functioning and outcomes for the accused, why not just directly fund public defense? There is no virtue in randomizing an intervention that (1) we believe is morally required (we call it a “right” to counsel) and (2) we know based on existing empirical evidence will improve the defendant’s outcomes and the administration of justice. Our focus and financial backing should be devoted to efforts so that everyone can meaningfully vindicate that right. Finally, randomization is not necessarily the fairest possible distribution of any scarce good or service. For that to be true assumes a level playing field, which does not exist in a criminal legal system marred by structural and institutional racism.

4. Are there any contexts in which you think RCTs can be helpful in studying or improving access to justice?

My ethical concerns are most animated by field experiment RCTs, and less urgent in the context of RCTs of administrative data. But even there, I note that studies using existing, historical data risk developing solutions that will simply reproduce the hierarchal racist system that already exists.

5. Is there research you think we should be doing instead of conducting RCTs in the access to justice context?

At the Houston Institute, we absolutely support evidence-based criminal law reform, but many kinds of evidence have value. We are especially interested in public health, sociological, and qualitative and participatory action research with directly impacted people, who too often are excluded from public policymaking and yet have the most direct knowledge and expertise of how existing systems cause harm, fail to prevent harm, and fail to meet human needs.