Episode post here. Transcription by Prexie Magallanes.
Matt Teichman:
Hello and welcome to Elucidations, an
unexpected philosophy podcast. I’m Matt Teichman, and with me today
is Witold Więcek, Consulting Director for Development Innovation
Lab at the University of Chicago, and he’s
here to discuss statistics and academic research. Witold Więcek,
welcome.
Witold Więcek:
Hi, thanks for coming.
Matt Teichman:
This is a topic that’s near and dear to my
heart. I have to say one thing that has been bothering me for the last
20-25 years in our popular media is: there seems to be sort of an arms
race in overconfident language. People will say something like: this
study shows XYZ, and it’ll be, like, one experiment. And you can’t
conclude that the thing is the case after one experiment.
This “shows” word typically is reserved for cases where the thing you’re saying is shown is actually true. It has what philosophers would call a /factive/ presupposition. And I feel like we see this all over the place. Instead of saying something like, “there was a study and the results are suggestive of XYZ,” people will be like: “well, we now /know/ XYZ,” where I don’t know we’re really in a position to conclude that with strong certainty yet. I wonder if is this something that you’ve also noticed in our media culture: that there’s a hesitancy to say that there’s anything we don’t know, and a pressure to act omniscient.
Witold Więcek:
I think there is a few layers to this
question, the first one being that generic point, going beyond any
media depiction of scientific facts, is that the language interacts
with statistical knowledge, which I guess we’ll be talking about here,
mostly in a quite imperfect way. The obverse of what you
describe—which is my particular pet peeve—is headlines where “no
evidence” makes an appearance. So in colloquial language where we’ll
say, “but currently there is no evidence for X,” which typically
people will take to mean that there is actually /negative evidence/.
Matt Teichman:
There’s evidence that it’s /not/ the case.
Witold Więcek:
Yeah, rather than absence of evidence. And the
way we discuss science is so good at conflating all kinds of
conflicting claims. Then what you describe is the degree of certainty,
which is also very hard to express in regular conversation. But I
think apart from everyday language, it’s interesting to think about
this from the perspective of practice of science and scientists. Or,
since we will be touching on economics in this conversation a lot, do
a supply-side analysis. Because if I’m thinking about demand side for
those media headlines, you would /a priori/ think, well, to say “study
shows” or “study suggests” is the same number of bits of
information. So why would we have a preference for one or the other?
If we start from the supply side of scientific facts, it seems like there is obviously a very competitive market out there to break through the noise, and it probably highlights the general trend in academia of there being more scientific facts and also more scientists competing against more noise.
Matt Teichman:
Like, if there was a case where we could be
really confident of the results, that would get more clicks, and so
there is pressure to act like we actually have gotten a result that’s
that robust.
Witold Więcek:
There is more value nowadays to creating an
atomized scientific fact, because there are outlets out there that are
specializing in creating that product. But to say that doesn’t really
answer why, culturally, we do have media outlets that specialize in
packaging atomized scientific facts into things that people will click
on. I don’t have a great take on this, but it’s interesting that you
could make an almost conservative or reactionary critique of this
entire process, where as science is becoming something commodified and
popular because of the assumption that a regular person is meant to be
interested in science.
It’s maybe a bit of a chicken and egg problem, as we assume that people care about scientific facts, or maybe we assume that people /should/ care about scientific facts. The role of science, or /scientists/, slowly becomes to be a supplier of those atomized facts which are packaged in the lowest common denominator kind of way, so that it’s as accessible as it can be to a regular consumer of media news. This probably differs from a model of science as it was in 1920s, even though the concept of popular science also existed back then.
Matt Teichman:
There was maybe less meddling from the public
in the process of the deliberation, maybe because people felt less
like it was their job to have an opinion about the stuff that was
being debated among scientists.
Witold Więcek:
I still think about this from a viewpoint of a
scientist as a /supplier/, rather than demand. Someone who’s trying to
second guess as to how to make their product most attractive.
Matt Teichman:
One thing that I found when I was doing my PhD
was there was tons of pressure put on especially junior
researchers—but also PhD students—to always be having a
breakthrough. And frankly, it was exhausting. Can you have a
breakthrough every three minutes? Can every single person in a
graduate student cohort be revolutionizing everything? I mean, it
seems to me no. But there was this feeling that we have to get the
best for every position, and we have to put out the best for our
latest journal issue, and the best equals radical breakthrough,
because those are the things that we’ve enjoyed in the past. What’s
the role of the breakthrough, versus the interesting paper that builds
on some previous stuff and might be useful later, with more modest
pretensions?
Witold Więcek:
Yeah. This is reminiscent of a hypothesis that
you and your listeners probably know by Ortega y
Gasset, a
Spanish philosopher from before the Second World War, about how
science is proceeding in a cumulative fashion through the accrual of
small discovering, or the addition of small facts, rather than through
Copernican revolutions every hundred years. I think people nowadays
are more in favor of this mode of thinking, because academia is just a
much larger universe than it was a hundred years ago, and the number
of PhDs per discovery is growing at a steady clip. It sounds flippant,
but these are things that people have actually graphed. Because the
inputs into any single discovery are getting more heterogeneous and
larger in volume.
So, you just need more people to produce those inputs. But it’s a funny philosophical tension here, because we still operate under the model of science as, aesthetically speaking, moving in breakthroughs and leaps and bounds. At the same time, not only the environment of doing science has changed, but also the mindset is of this Taylorization of production of knowledge. It is a thing which, in terms of its working practices, is getting more and more rigid and proscribed.
Matt Teichman:
Maybe to spell that out for people who haven’t
heard of Frederick Taylor, /Taylorist/ is a word for the kind of thing
that happened in the early 20th century, where a lot of individual
fabrication processes were mechanized and the assembly line was
invented, so that things get mass produced in factories. Is that
right? What you’re alluding to here is that academic research is more
like it’s like coming out of a factory now.
Witold Więcek:
Yeah, you can produce scientific papers,
especially in harder sciences, in a completely algorithmic
fashion. Science has been commodified as a product, and this product
has sort of created a number of efficiencies in producing those
papers. And we’ve all seen them—everyone who has interacted with
academia—those super successful professors who are producing 25 to
30 papers per year. There is, of course, a whole separate conversation
as to who’s actually producing those papers. But you wouldn’t really
imagine the old model of science in the early 20th century as any
scientist, no matter how brilliant, producing 30 to 40 pieces of work
every year. Tt’s a production line: that’s a short version of it.
Matt Teichman:
Do you think that there’s a parallel between
this phenomenon that we’re on about now, namely, that new researchers
feel pressure to be transforming everything with every single paper
they publish, and kind of the issue we started with, which is that
people reporting on current science often feel pressure to exaggerate
the level of confidence we should place in the results. Does that come
from the same place?
Witold Więcek:
We spoke a little bit about the supply side of
science and scientific products. To answer this, maybe it’s good to
briefly go back to the demand side. As we said before, it seems like
if I’m on the receiving end of some scientific information, it’s not
/a priori/ obvious that I should be more drawn to more certain
information than less certain information. It’s an imperfect analogy,
but humans, as a species, are designed to crave uncertainty. If we
think about how dopamine works, people are uncertainty consumers.
Matt Teichman:
That’s right. The first thing I thought was
skydiving. You want to not know if you’re going to live for sure, and
have an adventure. It’s this impulse to adventure.
Witold Więcek:
And on the sad end of the spectrum are slot
machines, and why people get into gambling.
Matt Teichman:
Right—I could know that I’m going to hold
onto my money, but no, it’s /more fun/ to not know if I’m going to
hold onto my money.
Witold Więcek:
It’s slightly perverse to ask this, but why
does this principle not hold when it comes to us experiencing
scientific knowledge? In terms of us as people in academia, trying to
find things out or as consumers of facts, trying to find out
information. Do we actually crave certainty? Do we hate ambiguity? I
feel like psychologically, that’s a slightly unresolved question.
Matt Teichman:
It seems like a different context. Sometimes
we crave uncertainty, and in other contexts, we flee in terror from
uncertainty. Figuring out which of those things happens when is maybe
a little bit tricky.
Witold Więcek:
Therefore, I think a really productive
critique of this starts from the supply side, as I called it: the
institutional side. What is making us act this way? And what I find
curious about this—here maybe we are now segueing from very general
points about science to talking about statistics—is that in sciences
that utilize statistical reasoning and statistical analysis of
experiments, for example, there are many initiatives for fixing
science, but interestingly, not so much for fixing /scientists/. What
I mean to say by this is we put a lot of focus on things such as
replication of studies, or practices that will make people publish
less attractive results. And this is great. This is institutional
fixes to improve the standards of science as a whole. But
interestingly, this still means that the scientific field is
essentially divided into policemen and research bandits.
Matt Teichman:
That’s pretty funny.
Witold Więcek:
I don’t have a great take on this, but it’s
interesting to ask whether there is some innate difference between
different temperaments in academia, where we will always have this
divide, or is there some institutional incentive which is turning
regular people who could sit down with uncertainty in normal
circumstances into peddlers of certainty and scientific half-truths?
I don’t know. Actually, what do you think?
Matt Teichman:
I’m not sure, at a global scale. I will say
that anecdotally, in terms of what I’ve experienced, these social
conventions can vary from discipline to discipline. To take an
example, when I was studying philosophy, I think there was a strong
expectation that any speaker giving a paper would be really, really
confident about the argument they were making. You see it all over the
place. At conferences, you see people basically saying: everyone’s
gotten it wrong before me, for the past 2,500 years. I finally figured
it out. We’re done now because I figured it out.
But lately, as I’ve been getting more and more interested in computer science, at least in programming language theory talks, people are way more tentative. They’re like, oh, you know: I have this new idea for a construction, and here’s how it works. Actually, last night, I think I mathematically proved that it fails in every case, but maybe you guys could help me fix it. I’m not sure what the takeaway of that is from those two contexts, but at least suggests to me that there’s huge potential for variation.
Witold Więcek:
I think there is an interesting takeaway here,
in that this issue can be solved by institutions and is very
culturally-specific to different domains. I think the thing to say
here, which comes true in your example already, is that you might
initially think that this maps onto hardness of science. So, I studied
mathematics. And in mathematics, it’s quite normal for someone to come
in with an attitude of subverting existing knowledge. And this is done
without any amount of peacocking. Either some facts are true and some
facts are false, and you have to present them with absolute
confidence.
But then something that’s in the middle of the spectrum of hardness, which I have been interacting with a lot in last few years, which is economics, has conferences, which to me, when people describe them to me, sound like a duel to death to present your paper. There’s definitely this motive of having to show that you’re absolutely right about everything, and you thought about every possible logical objection to your paper. And we’re not talking about a particularly hard science here. We’re talking about something which is often completely assumption and model-driven. So rightness and wrongness doesn’t really enter into it.
Matt Teichman:
And where it’s hard to get repeatable
experiments, laboratory conditions, and all that. That’s another real
difficulty, I guess, in economics.
Witold Więcek:
I would say that the presentation doesn’t map
onto the degree of confidence, in that you have to present any fact,
no matter how uncertain you are, as ultimately true, to break through
the noise. And from my experience working in bioinformatics and
medical sciences as a statistician, those conversations, they don’t
have to be so intense.
This part of the conversation reminds me of a fascinating example, which I know a little but about from my work with Michael Kremer at Development Innovation Lab. Michael, as some of you might know, is a Nobel Prize winner in economics in 2019. He’s a super accomplished development economist. And interestingly, some years back, prior to his Nobel, he had an episode of one of his major findings being found to be wrong. Or to be more precise, one of his papers was found to contain a number of mistakes.
What’s very interesting about this case, especially because it happened in a discipline like economics, where, as we said, all of the claims of certainty have to be made in this completely unwavering way, is that Michael very graciously produced the update to his paper. He redid his analysis. He explained why there were mistakes in his data, he refreshed his analysis, and he collaborated with people who were criticizing his research by giving them full access to his data. And lo and behold, he came out of this scientific disagreement with his reputation /enhanced/, precisely because it’s so uncommon for people to engage in dialogue with people who are trying to pick their paper apart.
To me, this story is a bit like an analogy to something we hear in the media. It’s going to be a weird analogy, but I just can’t help but think of it. There is a lot of talk in international relations about giving people off-ramps to de-escalate conflicts. So when there is an intense disagreement, the way to resolve to sit down together and have talks, rather than escalate the conflict, is apparent to both sides at all times. And it’s a weird analogy, but it seems like in academia, people are completely unaware of existence of an off-ramp. You see this over and over, where when people’s research findings get criticized, there are data problems, or sometimes they find to have fabricated the results. Sometimes it’s something more innocent. But invariably, they go for the nuclear option, which is really scary, in this analogy.
Matt Teichman:
They try to personally attack the person that
identified the mistake and claim it’s definitely not a mistake, etc.
Witold Więcek:
Yeah. It seems like in our academic
institutions, people need more practical cases of people making
mistakes and then doing the right thing.
Matt Teichman:
Really, if somebody points out a mistake in
your research, the first thing you should do is thank them, in my
opinion, because I don’t want to be making mistakes in my
research. But what happens is if your goal in the back of your mind
when you’re doing research gets twisted from what it should be (which
is to learn the truth) to getting lots of acclaim for your important
work, then the incentives can be aligned such that you feel like you
can’t thank the person for noting the mistake in your work.
We were talking about what happens in the good case: the case where everyone’s acting honestly with good intentions. Somebody published research that contains a genuine mistake. Another person pointed out the mistake. And then what they do is collaborate together on a follow-up study that rectifies the error. What would be an example of research where everyone /isn’t/ such a good Samaritan, and the original author digs their heels in when the error is pointed out? Or maybe the data are straight up falsified?
Witold Więcek:
The first and most obvious is fabrication of
data. This doesn’t really occur that often when it comes to
statistical analysis, but you’d be surprised still that with people
having all of the more complicated tools to fabricate claims
available, they will go for this lowest hanging fruit.
A really cool example from last year was Dan Ariely’s paper, which is signing at the beginning versus the end does not decrease dishonesty, which is an analysis of insurance data. Well, this dishonesty paper turned out to have at least part of its data completely fabricated through a random number generator in Excel. As soon as someone plotted their data, it was obvious that someone just made this up.
I have to say he’s one of the three authors on this, and I don’t think it has been determined who fabricated data on that paper. So this is just an example of things that do happen without, actually pointing a finger at anyone. But then why would you even have to fabricate data, when you can be completely selective in choosing data for your analysis? As long as you can get your hands on any data, it’s usually quite hard to show that people have engaged in cherry-picking of their inputs. You would have to have access to source data to prove this.
Matt Teichman:
Are there techniques out there for analyzing
data? Like you could maybe try to measure the level of randomness in
it, to just see sort of mathematically, whether it’s plausible that it
could be not cherry-picked?
Witold Więcek:
Yes. And this is one of the most fun—I
mean, if you’re a boring person—this is one of the most fun things
you can—
Matt Teichman:
—it’s a safe space for boring people to
really nerd out.
Witold Więcek:
There is a whole branch of forensic analysis
of tax records, et cetera, where you will look at distribution of not
just the values reported, but distributions of digits. There is a
really beautiful law in mathematics called Benford’s
law.
Matt Teichman:
Is that the one where everything starts with
one?
Witold Więcek:
No, but it’s one of those, which is about the
distribution of digits and numbers. As far as I know, people still
don’t quite understand why this happens, but it’s quite easy to pick
out things that have been fabricated by people. Usually when it comes
to fabrication of random-seeming data, people are incidentally very
bad, because they will overcompensate. And you can do a really cool
thing with your friends and ask them to make up a sequence of tails
and heads. Usually, you can get really good at detecting which
sequences of tails and heads have been actually generated by flipping
coins and which have been made up by people. The difference is that
people will always try to make it seem more random, whereas if you
start flipping coins, very soon you’ll have four or five or six heads
or tails in a row, and people can’t sit with this “non-random
randomness.”
Anyway, going back to data fabrication, yes, there is cherry-picking, and that is probably as egregious as making up your data. But the much bigger problem, because it’s pervasive, is that people optimize for interesting results at the level not of cherry-picking their data from an existing data set, but of finding data sets that are interesting. Recently, people might have seen this headline, which all the major outlets picked up, about a hundred grand being paid to a very famous economist Alan Krueger to publish a paper on Uber, which was quite favorable in its conclusions to ride hailing apps as an alternative to traditional taxi cabs. Krueger has been criticized widely for taking money to write a paper. I don’t have a strong opinion on the issue of someone taking 100K to write a paper in economics. You can make up your own mind whether this constitutes a conflict of interest, or it actually is productive because this paper wouldn’t have gotten written otherwise.
But what’s interesting is that people are not really picking up on the data side of projects like this, which is increasingly the case in academia, namely: companies providing researchers with datasets and probably already anticipating what sort of conclusions people are going to be drawing from those datasets. So here, even without any conscious research misconduct, or any intention for research misconduct, there is absolutely potential for fudging up some results because the access to information is completely controlled by the private sector.
Matt Teichman:
We’ve looked at a couple different strategies
for publishing a fraudulent study. One is to just fudge the original
input data. Another one is to cherry-pick so that you’ve got real
data, but it’s curated so that the statistical trend in it seems to be
whatever the author of the paper would like it to be. Are there other
ways that you can fudge a statistical study besides these two? For
example, maybe even with perfect data?
Witold Więcek:
This is where the real fun begins, for a
statistician. There are criminals out there who have to use very
crude tools, and there are people who can do their crimes undetected;
now we’re in this category. Sometimes you have to marvel at people’s
ingenuity in misapplication of statistics to prove something. However,
probably as with actual criminals, in 99.9 percent of cases, people
are still using pretty crude tools to make up the results. So how does
it work if we assume that you can have perfect data and you can still
get to whatever answer you want?
The number one concept here comes under the umbrella of what we would call: researcher degrees of freedom. What is the number of ways I can analyze my data, which are slightly different from each other and might lead to different conclusions? If I give you a data and some new miracle drug, or maybe (let’s make this up) there is a new meditation app that has been downloaded by 10,000 users, and we want to show that it has some beneficial outcomes for people.
Matt Teichman:
Makes you live longer.
Witold Więcek:
There you go. Why limit yourselves to “make
you live longer”? Why not it makes you earn more money, or it makes
you sleep longer, or it makes you sleep shorter? (Which can also be
good.) Or it makes you more relaxed, or it makes you more relaxed in
the afternoons, or it makes you more relaxed in the mornings. This is
one aspect of the degrees of freedom, but then measuring things is
really cumbersome. We don’t even have to bother with measuring all of
those things.
What if we only measured one thing, and we have those 10,000 people? We cannot find the increased relaxation levels in the general population. Why don’t we ask about relaxation levels in men who are aged over 40, who are second generation immigrants? You start cutting your statistical sample into finer and finer subgroups until you find something. The way that statistics work is that you’re guaranteed to find something eventually. You can always find a positive claim, as long as you’re not bothered about what kind of positive claim you wanted to make.
Matt Teichman:
That’s interesting.
Witold Więcek:
Sounds really good, right? But this is part
and parcel of how statistics is used in many academic domains. A lot
of experimental psychology claims are of this variety, where some data
has been collected, some hypothesis had been made post hoc, and lo and
behold, they actually happen to be true.
Matt Teichman:
I feel pulled in two directions on this,
because on the one hand, it feels like there is something that’s
probably dishonest about being as open as possible about what
hypothesis you’re trying to test, and only nailing down any kind of
hypothesis after the fact, based on which of these possible hypotheses
could be corroborated by the data you’ve gathered. It feels a little
weird, like you’re just looking for a win. But on the other hand,
isn’t there a legitimate scenario in which you start off exploring one
possibility, and that leads to a dead end? Then it leads you to a
genuinely interesting discovery that’s different from what you started
off with.
Witold Więcek:
That’s the spirit of scientific discovery,
right? It’s just to proceed—
Matt Teichman:
—be open to accidents. So what’s the
difference between being open to accidents and doing this more craven
thing—deliberately being wishy-washy about what you’re checking for,
at the outset?
Witold Więcek:
Statisticians have a way of coping with this,
which is now becoming finally a standard practive in many journals. To
go through the ridiculous example I constructed earlier, it’s called
/multiple hypothesis testing corrections/. A researcher here,
theoretically, can go on, and has any number of hypotheses, but they
will do mathematical or statistical correction, or they will penalize
themselves in their calculations, for the fact that they haven’t
tested one hypothesis. They tested 20 things.
The nice thing is that in statistical inference, we know how likely we are to get spurious results as we increase the number of hypotheses. So we know what kind of correction is needed, as a function of the number of hypotheses we tested. The example that I gave of multiplying hypotheses, always finding some positive claim, and then getting to publish, has very nicely been referred to by Andrew Gelman at Columbia University (who often has pretty fascinating takes on those issues and I would encourage for people to look him up) as the “garden of forking paths,” after that famous Jorge Luis Borges story, which basically expresses a slightly more general principle of a researcher having multiple stages in their analysis. Then they can always just decide to go left or right, left and right, left and right, and if you read the Borges short story, we get to the infinite number of universes by multiplying those choices.
I think I’m probably butchering that story that I haven’t read in years. But it’s important to say that we’re talking about an arms race of sorts here, where a single practice of for example making people correct their—for the number of hypotheses they make only means that they are going to find other outleta for making up scientific facts using statistical analysis, because the number of those decision points is infinite. The modern practice for this is for people to keep themselves honest by—not just in front of their peers but in front of themselves—by preregistering the kind of analysis they’re going to do. But again, this goes back to your question, which I thought was brilliant, that this is creating a tension with the process of scientific discovery.
We have to also allow for this iterative process, for the creative process to come and do data analysis, and this probably goes back to this earlier part of the conversation where we were talking about dividing science between the policemen and the bandits. I guess it’s an open question whether you want to incentivize people not to break the law, as it were, or you want to make them better citizens in the first place. So is there a jail for researchers, or is there a possibility of redemption, or do we just create norms that actually make people commit crime less? It’s funny to frame it like this, but those are valid questions for actually turning out good science.
Matt Teichman:
It seems like the mechanisms we have for that
now are basically: public shaming, and the person gets fired. That’s
jail for researcher who commits fraud.
Witold Więcek:
I think it can be more of a Catholic Church
version of: we just pretend that the problem is not there until it
gets really bad. In the meantime, we’re just going to shift this
person somewhere else.
Matt Teichman:
Or deny someone tenure and then they start
working somewhere else.
Witold Więcek:
Yeah. The “as long as it’s not my problem,
it’s not a problem” kind of attitude. Sometimes, people getting fired
is the best thing that can happen. But it’s still not a great
thing. If this is the only recourse that we have, where we say that
the research misconduct is sort of this binary thing, where just there
are just some bad apples there, and everyone else just getting
along—
Matt Teichman:
—all the gray area cases, even the dark gray
cases we don’t care about. But then only once in a blue moon some
like individual gets scapegoated. That seems like a bad system.
Witold Więcek:
Yeah.
Matt Teichman:
So we’ve uncovered all of these death traps
that are waiting for anyone who wants to draw on heavy-duty
statistical machinery to arrive at a policy conclusion, or new
directions for scientific research, or funding for stuff. There are
all these perils, on the one hand. On the other hand, we need
statistical machinery to help us make these decisions. It’s
unavoidable. So where do we go from here? What recourse do we have to
deal with these various forms of institutional erosion that we’ve just
been discussing?
Witold Więcek:
I think anyone who ever had a statistics class
for their degree will appreciate the fact that statistics are not
taught very well. But when people get to interact with statistical
analysis, they usually are encouraged to think in terms of stats being
something relatively easy. We try to simplify those issues so that an
average—absolutely no offense by picking the first thing that comes
to mind—but an average psychology researcher can publish their
paper. So we teach methods as if they were quite standard, and you
could follow a flowchart to conduct your analysis. It is the same in
medical statistics or empirical economics. The first thing to say is
that we have to treat statistics as something that’s very hard. You
have to just dispense with this conviction that this can be done by
anyone. Of course, I would say this because I’m a statistician, and I
make my money by the fact of helping people who are not statisticians
do their statistical work.
Matt Teichman:
Got to keep yourself in business, right?
Witold Więcek:
Yeah, I mean, that’s why I’m here. [ LAUGHTER
] However, I totally acknowledge in my biosphere. I think having a
dedicated statistician in academia often takes the heat out of a lot
of research misconduct potentialities in those projects. Of course,
where fraud is going to happen, if the senior researcher has an
intention to commit fraud, it’s going to happen anyway.
But the thing in terms of the research groups that are determined to actually get to the bottom of things, the model where the statistician—the analyst—is someone a little bit external, in my experience, it often cuts through a lot of issues with analysis. With people being under pressure to exaggerate claims, people being under pressure to finish their projects, meet the timeline of grant, etc. People having big egos. Externalizing this process can be really good. So I think we don’t commonly think of the roles of a researcher—of someone generating a hypothesis—and the person analyzing data, as separate. But having a bit of this firewall could maybe cut through a lot of those issues, even institutional issues, with how scientific facts are packaged, and this extra degree of confidence that we have to pump into every result that is being presented.
Matt Teichman:
That’s really interesting. I’ve never heard
anyone propose that before. But it immediately suggests a parallel in
computer science. So, the parallel would be there are certain areas in
computer science where you really just want people who have like
devoted their whole careers to work on that. One example would be
cryptographic algorithms. If you’re writing a to-do list app that
somebody can use in their phone, you don’t want the people who are
writing that to come up with new cryptographic algorithms that the app
is going to use. That’s really finicky business. You want experts who
have spent their entire careers getting all the little fine points of
the cryptography algorithm right, because otherwise, your thing is not
going to be secure. The are certain areas where if the work is really
detailed, easy for you to get wrong, and high stakes—high
consequences if you get it wrong—you want to have a division of
labor where specialists do that part, and you just use what they come
up with.
It seems to me that in pure intellectual research areas, there’s a similar situation, where heavy-duty stats number crunching is involved, because it’s so finicky—it’s so error-prone, beginners make huge mistakes, it’s hard for them to avoid making huge mistakes, and the consequences are dire when those huge mistakes happen. This is maybe another situation in which it’s helpful to have that division of labor.
Witold Więcek:
The really funny paradox with statisticians is
that unlike finding a consultant to get your project done, here we’re
talking about people finding a consultant to try to /break/ their
project. The intellectual attitude we’re talking about here is that I
collected a bunch of data, I have a hypothesis, and now I’m asking
someone to think about this in a semi-adversarial way. If there’s a
result, does this result check under every possible permutation of
assumption around those research degrees of freedom?
Matt Teichman:
Witold Więcek, thank you so much for coming
on.
Witold Więcek:
Thanks for having me. This was really nice.