Case Studies and Evidence.

OK. I’m going to stipulate up front that many of my readers know far, far more about this than I do. My educational background is in engineering, not medicine or epidemiology. I have, however, worked in direct collaboration with professional epidemiologists for many years, and with physicians for two decades, and I’ve taken a short course in epidemiology from the University of Michigan summer sessions. I say this not to argue from authority, but merely to say that I’m not talking entirely out of my ass. I have a minimal background. I do not claim to be an expert.

The course I took at UMich was “reading the medical literature”. Basically, it was a course for med students, residents, and other non-epidemiologists who work in medicine or medical research and need to be able to read and interpret medical papers. Can we critique papers for the quality of their evidence? What are the gold standards? How can we tell if a paper really supports its claims? What makes good a good basis for treatment?

I’m not going to rehash all of that today, but a couple of comments on yesterday’s post got me interested in reviewing the quality of evidence from case studies. Case studies are, by their very natures, anecdotes. There’s a big group of people out there who sort of rally around the cry, “the plural of anecdote is not evidence.” Well, certainly not always, but there is more to evidence than simply the peer-reviewed result of a double-blinded randomized controlled trial too. Evidence comes in many forms. I just learned, 20 minutes ago, of an aphorism (Thanks @amfeinman!) in statistics: “If you have a talking pig, you only need one.”

It is certainly true that the double-blinded randomized controlled trial is the gold standard of medical evidence. But there are other types of studies, like case control studies, which can provide crucial and important evidence. Like, you know, that smoking causes lung cancer [1]. But what kind of evidence do case studies provide?

In my course at UMich, the lecturer (a rather famous Canadian physician and epidemiologist) was adamant: Case Studies are not Evidence. His point was that you can’t generalize from case studies to general practice, and insofar as I’m qualified to argue the point, I agree. A case study is a single event. The information it provides is: “This thing has happened, and therefore can happen.” It does not provide any information about prevalence.

In logic, taking information from a single event and applying it generally is called “specious generalization”. Essentially, it means mixing up an “existence” statement and a “for all” statement. If we have successfully demonstrated that a particular element of a set has a property, it is a specious generalization to say that therefore all elements of the set have that property. While this seems obvious in plain language, it can be subtle in propositional calculus.

In disciplines that aren’t as rigorous – by necessity – as logic is, specious generalization may be required. After all, it is very difficult to prove that anything will be true in medicine for all people. We have to generalize and make assumptions, and apply specific knowledge generally. Also from my UMich course, the instructor said: “The only population that any study applies to for certain is the original study population.” And frankly, even then it may not be so certain. Confounders abound.

So case studies aren’t really evidence, unless they provide a counter example. If we believe that something is always true, and we have a case study where it isn’t, we have evidence that disproves our hypothesis. But that’s rarely the case in medicine. Case studies can also provide important safeguards. A case study of a complication may prove valuable in demonstrating that a technique is flawed, in surgery or anesthesia, for example.

If there are enough case studies that all say the same thing, then yes, they do become evidence of a sort. But generally, they become evidence that an event is worth research. Not necessarily evidence that we should immediately change practice or care delivery. So, those who claim that anything not properly randomized and controlled is nothing but useless anecdote are wrong. Anecdotes are not useless. Evidence comes in many flavors.

So, my take? Case studies are information. Information is valuable. Information can become evidence with further research. And dismissing case studies because they’re anecdotal is foolish.

***************

[1] Doll R, Hill AB, Smoking and Carcinoma of the Lung. Br Med J. 1950 September 30; 2(4682): 739–748.