#28 Great therapists have 10x better outcomes. Or do they?

Interrogating the shaky claims of "supershrink" literature.

Girl At Sewing Machine - Edward Hopper (Thyssen-Bornemisza Museum, Madrid)

Hi friends,

Supershrink is a term first used in the literature by D.F. Ricks in 1974 to describe exceptionally effective therapists whose patients see dramatically better outcomes than the average. A handful of impressive-sounding papers indicate that this is a striking phenomenon; that in fact, great therapists can have client outcomes that are ten times better than their colleagues. As a therapist who cares about doing a good job, this was obviously of great interest to me. What are these master therapists doing differently?

I hope to eventually have something to share with you on that question, but I was derailed in my research by some troubling things I noticed about how the empirical data on this topic are used and referenced. I don’t think the supershrink research is special in this way, in the sense that it almost certainly reflects broader issues with the scientific method as it is currently used in psychology and the health sciences. I want to use a seminal paper on this topic to illuminate these issues, which lie on a spectrum of empirical limitations to gross misrepresentations of the truth. In psychology, claiming to offer evidence-based practice has come to be synonymous with being ethical, skilful, and safe. While practitioners relish this signifier, the evidence-base itself can perpetuate what I can only describe as a fake reality.

Waiting for Supershrink: An empirical analysis of therapist effects (Okiishi, Lambert, Nielsen, & Ogles, 2003)

Waiting for Supershrink is considered a seminal paper in the literature on this topic. Published in 2003, it has been cited 628 times according to Google Scholar, including in a handful of papers published this year (23 years later). The study examined data collected on 1,779 clients seen by 56 therapists over a 2.5-year period in a University Counselling Center. (A further 35 therapists who had fewer than 15 clients were excluded from the study; summaries of the paper elsewhere cite the original number of therapists and patients included.) Client outcomes were measured using the OQ-45, an outcome tracking measure that is now widely used and arguably synonymous with the contemporary evidence-based-practice movement.

While there are various other critiques one might make of the paper, I want to highlight the most catching and widely reported one, which I mentioned above. The authors reported that the clients of the best therapist in the study (therapist #1) improved at a rate 10 times faster than the average. The clients of the worst therapist in the study (therapist #56) actually got slightly worse over time. The authors described these results as dramatic, and it’s easy to see why. This information is what is typically reported about this study in blogs and other articles I read on this subject, as well as by several different LLMs I asked to summarise this paper (including Google AI, ChatGPT, and Claude, which I use frequently for this kind of research and with which I have pleaded for fastidious, academic analysis).

But wait.

Actually Reading This Paper

On the surface, this is actually a rigorous study: 2.5 years long, quite a large pool of participants using an empirically valid measure, peer-reviewed, and published in a well-respected journal. It looks very legitimate, and by the usual signifiers, it is.

When I saw the summary of it in another (peer-reviewed, academic journal) paper, I was almost content to take it at face value. The average, on-the-clock, busy copywriter or student would probably read ChatGPT’s summary of it, or maybe the abstract at best, and be content that they had learned some good science. For some reason though, on this particular day, I actually read the whole paper. And here’s what I learned:

Therapist #1’s average number of sessions per client = 0.8

(Wait, what? How can a client attend less than one session? Apparently because of the way the sessions were recorded, the intake session was counted as session 0.).

So, in reality, the source of a cascade of claims all throughout the literature about this drastic phenomenon of therapists with miracle results ten times better than the average, is actually an individual student therapist who had incredibly inflated scores due to the fact that most of their patients only attended two sessions, the minimum number of sessions possible. (!!!) When the length of treatment was accounted for in the analysis, therapist #1, the former supershrink, dropped to a ranking of 38 in the sample of 56. (!!!)

Here are some other notes about claims made in the study:

• In the study, the theoretical orientation, type of training, years of training, and sex of the therapist had no relationship to outcomes. The result is often reported in other papers, as well as in other places online. However, all the therapists in this study were students in training, meaning variability in their years of training was limited. It’s hard to see how any real conclusion about years of experience can be meaningful drawn from a group of student therapists. Further, their theoretical orientation would be in its infancy, so again, it’s unlikely that this data accurately reflects a general population of therapists practicing from different modalities.

• The study only included clients with at least two data points. While this makes sense as a statistical necessity (it’s impossible to create comparative data with nothing to compare to), the conclusions do not speak to the frequency of this occurrence. This means that a therapist who frequently fails to retain clients after a single session may still come out looking highly effective in the dataset. For example, the paper does not state how many of therapist #1’s clients were excluded from the study because they only attended one session.

• The average number of sessions for the whole sample was 4.47. There was a 14-session ‘limit’ at the centre where the study took place, although clients were allowed to be seen for more sessions if the therapist felt it was necessary. This makes the research of limited validity to any relational or long term therapy outcomes.

• Rather than acknowledge the quantity of unusually brief treatments as a limitation of the study, the authors suggested that these data indicated that exceptional therapists simply get the work done faster. (!!!) While they stated this just made “logical sense”, their interpretation of the data in this way reflects an unnamed ideological bias that left no room for consideration of other reasons why clients only stay in treatment for a couple of sessions, including the possibility that the therapist might be encouraging them to do so for any number of conscious or unconscious reasons.

Beloved Evidence Base

All research has these kinds of limitations. Some of the ones I’ve mentioned are explicitly acknowledged by the authors of this study. This does not stop the conclusions from being referenced and re-referenced until the nuance is so far corroded that ChatGPT says the study demonstrates that “Clients of the most effective therapists improved at rates approximately 10 times greater than the average therapist.” This plainly isn’t true.

I obviously don’t think academic research is inherently bad, or irrelevant, or that studies like this should not be conducted and talked about. I’m not even saying that some therapists don’t have 10x better outcomes than others. I suspect they probably do? But this study is not evidence of that, despite being referred to as such. And this study, for its strengths and weaknesses, is what we are talking about when we talk about the “evidence base”. This type of paper is the gold standard of what psychologists and lecturers and teachers refer to so smugly as an indicator of goodness, the thing they stake their personal brands and careers on. And while many other papers have come after it, this particular study is still referenced today.

The notion of the supershrink has also been used to inform an entire movement called Feedback Informed Treatment, championed by Dr. Scott D. Miller, which presupposes that the differences between therapists (like those the Okiishi et al. paper attempts to highlight) can be corrected with the use of therapy outcome measures like the OQ-45. I am personally not a great fan of this model and its factory-farm-ification of client feedback. I think constant interrogation of clients with iPad surveys, aside from just not being very chic, ignores transference and the unconscious entirely, supposes that the client knows and can speak clearly about what is wrong, and implies an overly clinical, reductionist idea of a therapeutic relationship. A critique of Miller’s work is beyond the scope of this particular piece, but I wanted to mention it as an example of how the paper examined above has real world implications by serving to support the ideology of Feedback Informed Treatment and position it as an evidence-based, and therefore legitimate, way of doing therapy.

This type of research is not the silver bullet it is touted to be. The reduction of its meaning to a few rhetorically appealing catch phrases creates a fake reality that leads us further and further away from knowing the truth about what actually happens in therapy, and by extension, what types of therapists, with what types of training, orientation, and disposition, have the potential to help real people. Despite what is taught in universities, peer-reviewed journal publication is not a default indicator of ultimate truth. If we want to practice with rigour and integrity, we cannot pretend that abstracts or media releases about academic papers (much less AI summaries) are going to show us what is real. We cannot abdicate responsibility for the ongoing quest to discover the truth, which plays out in every session, but also in the lifelong exploration of ideas we undertake in our private thinking, living, reading, loving, and wondering. I believe this, not prostration at the alter of performative scientism, forms the basis of our vocation.

In love, and the pursuit of a shared path to a greater truth,

Kate