We Only Know What We Can Measure
We rarely stop to appreciate how research is driven and constrained by what we can measure. It is the reason many findings we once thought of as true turn out to be false and why we continue to make new discoveries. We get better ways of measuring things.
Phrenology is one of my favorite examples. Around 1815, Franz Joseph Gall wrote “The Anatomy and Physiology of the Nervous System in General, and of the Brain in Particular, with Observations upon the possibility of ascertaining the several Intellectual and Moral Dispositions of Man and Animal, by the configuration of their Heads.”
His conceit was that the brain controlled our behavior and that measuring the brain would tell us about how people did and might behave. Unfortunately, the only way he had of measuring the brain was to measure the shape and size of the skull.
So eventually he and phrenology were laughed off. But his conceit was correct! He just had the measurement wrong.
Since the advent of functional Magnetic Resonance Imaging (fMRI), magnetoencephalography (MEG) and electroencephalography (EEG) we use new types of measurements to tell us which parts of the brain are active, how they are connected, and what that implies about the person with the brain.
Why COVID Answers Keep Changing
Sometimes our ability to study something is limited by what we can practically measure. Let’s say the question you want to ask is “Does wearing a mask prevent COVID infection?” The correct way to answer this question is to put a person with COVID in a room with 30 people all wearing masks. How many people become infected with COVID after an hour?
The study would be repeated with different masks, COVID patients at different infection levels, different lengths of time, different amounts of ventilation, different distances to the uninfected people, etc. The problem is that we can’t physically or ethically run these studies.
So what we do instead is something like this. We create a machine (above) that “breathes” and measure how different masks capture particles of different size breathed out of the machine. This is very helpful information. We learn that:
Tightly wound fabric is effective at capturing particles
Gaps around the edges cut effectiveness by two thirds
But this study thinks about movement of particles not infection, and it makes a lot of assumptions that we do not know are sound. The study is limited by the fact that we can’t see the virus particles so no one knows how they disperse or move in the air, how long they hang in the air versus settling to the ground, how long they are viable, and if breathing them is enough to actually get you sick.
Very few scientists admit our blindness to how viruses travel from one person to another. But you can easily imagine that if we could see it, we would suddenly fund research to stop transmission of viruses rather than to build vaccines to viruses.
Imagine spending all our money on terminating pregnancies rather than how pregnancy happens and what steps can be done to prevent unwanted pregnancies. It would be silly, right?
Discovering What was Already There
Sometimes, we don’t even realize that we are measuring the wrong thing.
Everyone agrees that environment & behavior plays as big a role, if not a bigger role than genetics in determining your risk of chronic disease. Today, chronic disease is the cause of over 70% of deaths. And most of these chronic diseases appeared in a single generation - far too fast to be explained by genetics. Yet if you look at research grants over the last 30 years, a huge proportion of it has gone to fund genetics research assuming DNA as the cause of these diseases.
Then something interesting happened in the last 10 years. We “discovered” the microbiome -- the billions of bacteria living on and in us. Except, we didn’t discover it. We’ve known about it for a really long time. We just couldn’t measure it, so we ignored it.
A decade ago, our ability to sequence genes increased in speed and accuracy and reduced in cost so as to allow significantly more samples to be run. With that technological change, scientists were able to name and quantify the billions of bacteria that live on us. They could see how people’s biomes differed from each other, how they changed over time, and how they interacted with our environment.
Now there is a seemingly endless amount of money pouring into research on how our bacteria-friends do critical functions like
Digest food
Protect our barriers
What’s the Point
We can’t “know” what we can’t study. But we can’t study what we can’t measure.
“There’s no data to support that” doesn’t mean something isn’t true. Conversely, data doesn’t mean we know the whole story. So rest assured that new “discoveries” will continue to be made about the nature of life, the way the human body & mind work, and how best to keep things safe. Don’t expect controversies in nutrition or child-rearing to end any time soon. And practice asking yourself “Does this measurement actually answer the question?” when someone send you a study. More on that next time...
