In this podcast recorded at the Courts,

Community Engagement, and Innovative Practices in a Changing Landscape symposium held

in Anaheim in December 2015, Susan Turner, professor in the department of Criminology,

Law & Society at the University of California-Irvine, explains how risk assessment tools are developed and

discusses the strengths and limitations of risk assessment. 

RAPHAEL

POPE-SUSSMAN: This is Raphael Pope Sussman, with the Center for Court Innovation. This podcast is part of a series
of dispatches from the Court’s Community Engagement and Innovative Practices in a Changing Landscape Symposium
held in Anaheim in December 2015. The conference focused on justice reforms, including recent developments in California,
public safety realignment and Proposition 47. Public safety realignment refers to changes brought about by 2011 legislation
that shifted responsibility for certain populations of offenders from the State to the county level. Proposition
47, a ballot initiative passed by referendum in 2014 reclassified certain low level felonies as misdemeanors. I hope
you enjoy listening.

Hi, this is Raphael Pope Sussman, with the Center for Court Innovation, and

I’m here today at the Court’s Community Engagement and Innovative Practices in a Changing Landscape Conference
in Anaheim. Right now I’m sitting with Susan Turner, professor in the Department of Criminology, Law and Society
at University of California, Irvine. Susan, thank you for speaking with me today.

SUSAN TURNER:

Glad to be here.

POPE-SUSSMAN: What’s the process for developing a new risk assessment tool?

TURNER: Well, risk assessment tools can be developed by an organization. There are also ones that are available

that have been developed by others, so some of the first decisions that jurisdictions make is whether to do it home
grown or whether or not to buy something that’s been developed and is available for purchase, or is perhaps
free. If you’re interested in developing a risk tool for your jurisdiction, one of first things you need to
decide is what are you interested in predicting? Most jurisdictions in risk assessment are interested in predicting
some kind of recidivism behavior, such as arrest or incarceration or conviction. Then we generally use a set of variables
that are related to an offender’s prior record, age or gender, and then some other factors that may be related
to their drug use, mental health status, their association with criminal peers. You basically need to make sure that
you have the data for your left hand and your right hand side of the equation.

You gather the

information on an individual level, so for every person you’re interested in, you gather information on whatever
outcome and then their record, their variables such as their gender, their age, and their prior record. You want
to have a sample that’s large enough. You put them generally in a regression equation, where you try to figure
out whether or not these factors are predictive. You’re able to gather weights for the variables based on these
mocks. For example, if you’re predicting whether somebody would be convicted, you might have age and you’d
have a weight of 5, gender you’d have a weight of 3, different prior record variables would have various weights.
You can just imagine sort of scoring everybody up, and the higher the scores, generally the more likely someone will
recidivate. Once you do that on one half of your sample, you build your model. Then you do something called validation
on a separate set of data that you haven’t looked at before and that tells you whether or not what you’ve
developed on your first sample validates or is predictive on your second sample. That’s basically a kind of
simplistic view, but the overall gist is that you have data available, you develop your instrument on one half of
the sample and test it on the second.

POPE-SUSSMAN: What are the limitations of these tools?

TURNER: Well, there’s a lot of good things about the tools and there are, of course, always limitations

with any kind of tool. Risk assessment for recidivism is, like many other risk tools, we have risk tools in our regular
lives. You think of your car insurance premiums that you pay are based on risk assessment tools. Risk assessments
tools are only as good as the data that are behind them, number one. Another limitation to risk assessments tools
that we all need to be aware of is that they’re never 100% accurate, and probably will never be 100% accurate.
They generally are what’s referred to as a moderately good predictor, but because of the nature of human behavior
and variables that we don’t know why people do the things we do, we can never gain 100% accuracy, which sometimes
gets us into trouble with say politicians who have very high standards for whether or not you want to use an instrument.
Sometimes so high that they’re really not practical.

POPE-SUSSMAN: What are the great strengths?

TURNER: Great strengths are that we normally discuss risk assessments tools contrasted with what we refer

to clinical judgment, or some people call it gut. The research has shown that actuarial tools or risk prediction
instruments are more accurate than gut or clinical experience. Many people don’t like to believe that because
they’re often trained in a clinical background, or you can imagine a parole agent with 30 years of experience
is sometimes looking askew at risk assessments tool, but they’re more accurate. They can also result in decision
making that’s more consistent, because you can imagine the great variability if everyone follows their own gut.
You have lots of different ways the same person might be scored, so there’s consistency. There’s also,
with some risk assessments tools the advantage of time. Some of them are automated so for large numbers of people
they can be scored pretty quickly, allowing an agency to be able to get actuarial tool on a number of offenders without
the resources that may be required for, say, 45 or 50 minute instruments that are collected individually.

POPE-SUSSMAN: What roles do you see for risk assessments tools in the larger project of decarceration?

TURNER: One of the things we’ve talked about here today was the use of risk assessment and one of the

things I mentioned was that risk assessment’s been around for a long time. It’s got a resurgence nowadays,
I think partly because we’re trying to reduce the current populations that we have incarcerated and we don’t
have enough money to be able to keep everyone locked up. Money is driving a lot of decisions and one of them I think
actually may be risk assessment. We need to decide who are the highest risk people, the ones that we need to devote
our resources to, our public safety resources, and to not spend the resources on the ones who are the lower risk.

POPE-SUSSMAN: How do we ensure that risk tools don’t show bias against individuals and communities

that may have disproportionate rates of established risk factors, like housing or employment or disability?

TURNER: Very good question. I think there are a number of people who are concerned with risk assessments

tools based on dimensions that may impact certain groups unfairly. Many risk assessments tools use an offender’s
prior record as a major variable in their recidivism outcomes. Then there’s obviously the question, if prior
records are somehow biased in their very creation within communities of color, what does that mean in terms of risk
assessments tools. It’s a very valid question. One of the ways that we help address this is when we develop
tools, we take a look at and see how well the tools works with the different populations of interest.

For

example, when we developed a tool for the California Department of Corrections and Rehabilitation, we were particularly
concerned about whether the tool predicted for females as well as males, because there’s a lot of work suggesting
that some of the risk factors for women are different and for men. Also, we were concerned about whether the risk
tool worked equally well for blacks and Hispanics and whites and other groups. What we do with those is that we separate
the samples into the groups of interest and check to see whether the tool is as predictive in each of the groups.
If you find a tool that doesn’t work very well in one of the segments that you’re interested in, then you
need to go back to the drawing board.

POPE-SUSSMAN: I’ve heard that misdemeanor populations

are particularly difficult to assess. Why is that?

TURNER: Well, the discussion of misdemeanor

risk assessments has really come into the light here in California with proposition 47. Traditionally, most of us
deal with risk assessments in the felony world. All the risk assessments tools I’ve done are pretty much predicting
felony recidivism. There is a question, are these tools good for misdemeanors. There are tools out there that have
been developed for misdemeanors. There’s a tool for misdemeanor DUIs and some tools that are developed by the
University of Cincinnati team have established one that’s for misdemeanors. They basically work the same as
the felony tools. They’re predicting a different kind of an outcome, which is maybe a lower level of behavior.
What we find with our risk tools is that these tools can be very robust to different populations and different kind
of outcomes.

POPE-SUSSMAN: Do you have any big vision going forward, the next step for risk assessment?

TURNER: Well, one of the risk assessment has always been used pretrial for a long time. It’s been used

the last 20 or 30 years at the probation stage. I think more recently it’s being considered at the sentencing
stage, which I think in many ways has a little more ethical issues, whether or not in sentencing actually considering
risk as opposed to the sentence type.

POPE-SUSSMAN: What do you mean by ethical issues?

TURNER: I mean, what’s the purpose of sentencing and some people say the purpose of sentencing is that

it should be proportionate to the crime that was committed and that our statutes are based, sentences are based on
perceived severity of violent offences being more serious that property offenses and you reserve the highest sentence
lengths for the more violent. If you move to a risk based system and you say, well as we found in our research, people
convicted of violent crimes are not necessarily the most likely to re-offend, so you basically a conflict going with
what is the justice based, in terms of the offense, versus a risk based, which is about future behavior. That’s
sort of conflict that you’ve got to figure out how you’re going to deal with. In our own development of
the risk tool for the State, in our training and discussion of the tool, we’ve come across people who are very
perplexed by our findings that show that people who have convictions for violent offenses, there’s actually
what we refer to as negative weights on that, so that people with violent offenses are actually less likely to have
recidivism than those that don’t, which is very counter-intuitive.

POPE-SUSSMAN: I guess

that would also get to people’s ideas about what the justice system is supposed to do. Is it supposed to be
punitive? Is it supposed to be rehabilitative? Is it supposed to be utilitarian?

TURNER: Yeah,

I think, as I teach my students, a couple things you have to remember. If you remember a couple of things, you go
out after school it’ll be very helpful. One of the them is to remember always the different purposes of punishment.
One is deterrence, and that’s what we we think risk based decision making is based on. But there’s retribution,
incapacitation. Yeah, there’s many philosophies of what the goal of punishment is. I think risk prediction sets
you straight in the cross hairs of some of the other theories of punishment.

POPE-SUSSMAN: Well,

thank you. I really appreciate you taking the time to speak with me.

TURNER: Thank you very much

for the opportunity.

POPE-SUSSMAN: This is Raphael Pope Sussman, of the Center for Court Innovation,

and I’ve been speaking with Susan Turner, Professor in the Department of Criminology, Law and Society at the
University of California, Irvine. For more information on the Center for Court Innovation, visit www.courtinnovation.org