Episode post here. Transcription by Prexie Magallanes.
Matt Teichman:
Hello and welcome to Elucidations, an
unexpected philosophy podcast. I’m Matt Teichman.
Joseph Diller:
I’m Joseph.
Matt Teichman:
With us today is Robin
Hanson, Associate Professor
of Economics at George Mason University, co-host of the Minds Almost
Meeting podcast, and author of The
Age of Em and The Elephant in the
Brain, and he’s here to talk
about prediction markets.
Robin Hanson
Thanks for having me.
Matt Teichman:
The main topic that you’ve worked on, in this
area, is what sometimes gets called prediction markets. I thought
maybe we could open by explaining, in really general terms, what a
prediction market is.
Robin Hanson
Sure. So, you start with a bet like, you know,
will a horse win the race? But let’s pick something more
practical. Will you make a deadline on a project? And you could just
bet with somebody about that: $10 each. And then, if I’m right, I get
$10 from you. If you’re right, you get $10 from me. That’s a
bet. But we can turn that into an asset. We could write a piece of
paper that said, “Pays $10 if the project makes the deadline.” Then,
we could buy and sell that piece of paper, and the price might
fluctuate, and that would be a betting market: a place where people
buy and sell bets.
That’s what we’re basically talking about, when we’re talking about prediction markets: we’re talking about betting markets. And the whole idea is that the price in a betting market is informative about the event. If people are trading this asset that pays $10, if we make the deadline, and the current price is $6, that suggests a 60% chance that we will make the deadline. And you might not know that: maybe as head of the project, you would find that informative, to know that you have a 40% chance of failing.
Matt Teichman:
What exactly is the difference between the bet
working as an office pool, where everybody puts in money, and then
they collect from the general pot if they win, versus this thing where
there’s an asset that people invest in?
Robin Hanson
People often bet, as you say, in pools,
parimutuel pools—that’s, say, how it works at the horse races. In
a parimutuel pool, there’s different pools; there’s the pool where the
horse comes in first, or second, or third, say, or doesn’t do as
well. You put your own money into one of the pools, and then, the rule
is: the winners of the pool get to divide up the other pools in
proportion to how much money they put into the winning pool.
So if $100,000 went into all the pools, and $20,000 went into your pool, that’s a ratio of 5 to 1. If you put in $10 into your pool, then you’ll get $50 total, because you get that fraction of all the other pools. Parimutuel pools are a way to bet, and they do aggregate information, but they don’t do as good a job of a betting market. What happens in a parimutuel pool is: people will be adding to the pool, and they’ll show on the board how much money is in each pool. But you’re well advised to wait until the last possible moment to decide which money to put your pool in, because that’s the moment at which there’s the most information about the current odds, because you might put your money early on, thinking there’s not many people in your pool, and that’s a good investment. Then, later, before the door is closed, lots of people put money into your pool, and then you don’t actually get very good odds. Maybe that wasn’t a good bet for you, at those odds.
When you bet, you want to know what odds you’re going to bet at, to decide whether you think it’s a good bet, and in a parimutuel pool, you only know the odds if you bet at the last possible moment. But in a betting market, when you make a purchase, you know the price of the purchase, at the moment you make the purchase. You can decide: do I want to make this purchase, buy or sale, based on the price you see at that moment? And then, because the prices fluctuate, now information is collected.
The key idea of a betting market is that prices fluctuate up and down, and at any moment, you look at that price and you say, “Is that right? Do I agree with that price?” If you think that price is too low, you have an incentive to buy, and that will push the price up, and if you think the price is too high, you have an incentive to sell, and that’ll push the price down. Thousands or millions of people doing that slowly make the price more informative, as each of them looks at the price and asks, “Does that reflect everything I think I know?”
Matt Teichman:
So in both of these scenarios, the pool
scenario, and the betting market scenario, if each person puts this
amount of money towards guessing that this outcome will happen,
there’s a mathematical formula for determining (once we know whether
or not the thing happened) who gets how much money.
Robin Hanson
You’re right that we could think in general
about a large space of possible systems, and in every system, each
individual would have a number of actions available to them, things
they could say at different times, and they would have different
things they could see about what everybody else has said. There would
be some way in which it was all pulled together into some sort of
consensus that might be the result of the whole process, and then each
of them would face some incentives, in terms of: if I say this at this
time, with this sort of setup for the system, and then this is the
outcome, then this is how much I will gain or lose, in either money,
or reputation, or score, or something like that. So in principle,
there’s this vast possible space of systems, and people have explored
that space, to some degree.
I recommend betting markets as a nice, simple element of that space, that do a good job of aggregating information, and have actually been used a lot over the years. But there is a large space of other possibilities, and if you like, we could explore those.
I can give you some examples. That is, recently, some people have been making tournaments—forecasting tournaments. In forecasting tournaments, individual forecasters forecast particular topics on particular days, and then there’s some formula that takes the forecast, based on a weighted average of who has said what on what days, puts together some aggregate forecast, and then has some scoring system to rate how well each person did, based on the forecast they made, and the other forecasts, and what actually happened. There’s actually a large space of possible formulas you could have for how you produce the consensus, out of everybody’s forecast, and how you score people.
Unfortunately, many people just put together formulas without really thinking through whether they’re incentive-compatible, or whether they do a good job. Sometimes the formulas people make are gameable, where you want to lie about what you think, in order to get more score in the game, and sometimes that messes up the system. But still, forecasting tournaments are much better than most of the other things we do in society, to take groups of people and collect what they know together into a consensus estimate that gives them all good incentives.
Joseph Diller:
In these bets, or forecasts, are there any
conditionals? Like, are bets weighted differently, based on possible
truth of other beliefs?
Robin Hanson
There’s a vast space of possibilities. And so
certainly in principle, you can do all those things. The question is
just: which particular systems have done which particular things? Some
people have done conditional forecasts—scoring conditional
forecasts—but that’s more unusual.
Matt Teichman:
What are some examples of things people have
actually tried to predict, using this method?
Robin Hanson
Betting markets have been around for centuries,
and people have bet on a great many things, for centuries. Of course,
there’s a lot of bets on sports. There are bets on politics, there are
bets on business outcomes, and in some sense, speculative markets are
(more generally) kind of like betting markets. They have a lot of the
same features. So most stock markets, currency markets, commodity
markets—they actually have the same basic feature, that you entice
people to aggregate information into the price by looking at the
current price, and guessing whether it’s too high or too low. When
you’re looking at the price of gold, or something in a commodity
market, it’s less clear what actual fact about the world a high or low
price of gold corresponds to. It’s more indirect. But there definitely
is something about the world you’re learning from the price of gold,
or the price of Google stock, or things like that.
We have a vast experience with speculative markets. Most financial markets are speculative markets, and betting markets are one kind of that, but they have a lot of properties in common, and we’ve had a lot of betting markets over the years. At the moment, there are many betting markets that you can go look up, with sites like PredictIt and Kalshi, where they have bets on lots of celebrity events, and particular economic outcomes: the war in Ukraine, whether the nuke will go off, whether we’ll meet aliens. I mean, there’s just a wide range of forecasts, on many of these sites.
Matt Teichman
The one that I came across, only very
recently—I was in the middle of a faculty meeting, when we were
counting the votes during the 2020 election—I learned about
PredictIt. I was shocked at how much more accurate it seemed to be
than the forecasts they were making on the news. They seemed to call
things like e.g. ‘Georgia is going to go to Biden at this time.’ I
forget the other states that went to him surprisingly. But these
predictions were all made days before they were made by news
outlets. So at least it’s an n of 1; I’m not going to generalize on
the basis of my anecdotal experience, but at least—
Robin Hanson
—well, there is a lot larger “n” to tell you
about. We have a lot of data, on specular markets in general, and
betting markets in particular, about (first of all) their
calibration. For example, when a market says a 70% chance, how many
times out of 10 does that actually happen? Betting markets tend to be
very well-calibrated, since the statistical frequency at which they
are right corresponds to the probability they give. In addition, they
are typically as accurate as (or more accurate than) other things we
compare them with. We often have polls, or committees, or other sorts
of mechanisms to make forecasts, and consistently, when we put up
another mechanism, and we put up a prediction market on the same
topic, at the same time, with comparable resources, then the market
does about as well or substantially better. And that’s the main
reason why you should be interested in them that I can offer. That is,
they consistently work. We do understand roughly why—but maybe
that’s less important than the fact that they do.
Matt Teichman:
Do we have a sense of how much more accurate
they are, approximately?
Robin Hanson
Well, again, it depends on the topic. So if you
took something that everybody knows roughly the same—say we’re going
to roll a dice, and it’s a fair dice, and nobody knows anything about
the dice, other than that it’s a fair dice, then we can all predict a
one-sixth chance of each side of the dice. Then asking you will give
you the same answer as a prediction market. It’ll give us the same
answer as all sorts of sources, because we all know the same. It’s
when some of us know more than others that it becomes interesting. But
problems vary in how difficult they are, right? If we ask about
forecasting, whether the sun will come up tomorrow, we are all very
confident, and it’ll be very hard to judge our relative accuracy
because we’re all going to give the same answer every time, and we’re
always going to be right. There are other problems that are just
really hard, and we’re all bad, and all of our errors are high. So
the interesting question is, on the same topic, comparing
mechanisms, which tends to do better. How accurate they will be
depends on how hard the question is.
Matt Teichman:
So maybe the real principle determining how
accurate these things are is how much the people participating in the
market actually know, like how high their expertise is. Maybe this is
a mechanism for kind of getting everyone to be as much of an expert as
they can be.
Robin Hanson
Some mechanisms require that you do a lot of
curation to set it up. For example, if you have a committee, you have
to decide who’s on the committee. Maybe you decide who has how much
voting power.
Matt Teichman:
Laborious hiring process, blah, blah, blah…
Robin Hanson
Right. So the quality of the committee decision
will depend not just on the committee rules, but on your choice of who
goes in, and on what topics. How well does that match? There are
other mechanisms that are more open, where you don’t have to make as
many choices like that. You can allow people to self-select. And a
betting market can be a mechanism where it’s very open, and you just
let people choose whether to bet. Of course, you can also have a
betting market that’s closed, and you choose that only certain people
are allowed to participate.
Matt Teichman:
Is the idea that if I’m really terrible at
making predictions, I’m going to run out of money and have to stop
betting?
Robin Hanson
In the worst case, that’s how you figure it
out. Hopefully, you get clues earlier, and bail out early. But in the
worst case, that’s how it works.
Matt Teichman:
And then, if I’m really good at them, I keep
making more money, and I get to make more predictions.
Robin Hanson
Right. That’s also a worst case. I mean,
hopefully, you would have some idea what you were good at, and other
people would have some idea, and then you would jump into the things
you’re good at. But in the worst case, you have no idea what you’re
good at, and you just randomly do things, and you find out what you’re
good at by winning.
Joseph Diller:
What if you have key information, that you
can’t really bet as much money on as someone more wealthy, and
therefore you can’t really make as large of an impact with it, as
someone else would, if they had that information?
Robin Hanson
I’m an economics professor, and I study the
economy. In most of the rest of the economy—in an industry like
making cell phones, or watches, or something—the way an industry
works is: the industry combines a number of elements to produce
products and services. So there’s capital, buildings, trucks, et
cetera. There are different kinds of people. People who are good at
marketing, people who are good at engineering, people who are good at
repair, delivery, and they come together in firms to produce the
products and services that people want. They don’t do it individually:
you can help be part of a watch industry without yourself making the
entire watch and doing all the parts. In an industry which is about
producing information, what we want to imagine is firms entering that
industry with teams of people that have an advantage in making that
product, and then successfully gaining customer revenue from being
better than the other firms in the industry. In a betting market, you
don’t have to do it all yourself.
And in fact, in most speculative markets, i.e. stock markets, etc., they are dominated by hedge funds—large organizations that combine capital from investors, and intelligence from individual people, and knowledge from all the contacts they have (and they pay off), to make their many investments. That’s what we want to imagine in betting markets, too.
Joseph Diller:
It seems like this would have a long
time-frame, in comparison to e.g. betting on the 2022 midterms, where
we only have a few months to go.
Robin Hanson
But the 2020 midterms were well-anticipated,
decades ago. We knew when they were coming. Firms often have specialty
products for specialty dates that they know about, long in
advance. They can get together and be prepared to, say, deliver
Christmas products coming on Christmas, and things like that.
Matt Teichman:
Maybe they have some practice from previous
midterm cycles, and we know there are going to continue to be midterm
cycles indefinitely. Of course, as a philosopher, I want to try to
understand the causal mechanisms behind things, and I immediately
start wondering: what is it, exactly, that would make the predictions
of a betting market more accurate than the predictions of—I don’t
know what, pick your favorite alternative—a panel of experts, a
sophisticated computer simulation. Whatever the alternatives are. The
first explanation my brain goes to is: well, you’re really putting
something on the line if you’re going to lose money by making wrong
predictions. That turns up the stakes sufficiently high that people
just end up being more careful in their predictions. Is that why
they’re more accurate?
Robin Hanson
That’s part of it. I would ask you to think in
terms of separating a forum from a competitor in a forum. If you
think about sport teams, the forum is a tree structure of which teams
play who, and what playing field they’re on, and what the rules of the
game are. There’s a process by which the winners become champions and
celebrate it, and then there’s the competitors, the teams and the
coaches, who choose their team strategy, and are trying to win the
competition.
Many of our institutions are of that form, where you’re both wearing shirts, and the shirt industry consists of a forum—that is, places you can go look for to look at shirts, and decide which ones you want. Then, each firm out there that makes shirts has an internal process for deciding which shirts to offer when, and how to make them, and at what costs, etc. So if we ask: how can we all collect information together, in order to produce consensus estimates that are as reliable and informative as can be, then I want to think in terms of a forum in which competitors come together to show that they have a better method for doing that, and then the methods they each use internally. Internal to each of these competitors, say at a hedge fund, they can have computer simulations, they can have smart committees, they can have whatever they want—but they’re competing in a larger forum, which is the speculative market. First, we want to ask, does the larger forum give good incentives to the competitors to do the best job they can, to produce the things we want?
For example, if you thought there was a flaw in the shirt market, whereby the people who had the best shirts wouldn’t win, in terms of getting your attention, because maybe the distributors that you rely on have a bias, in terms of the kind of shirts they want to show you, then that would be a flaw in the forum, that could produce very competent behavior by the competitors, but that was still a problem, because the forum biased the results. Or we can have a great forum, where the competitors are making a bad choice about what strategies to give. But then, if you know what the better strategy is, you could go advise one of these competitors: hey, you should do it differently, because you could win better that way.
Those are the two steps to think about, in terms of how can we all aggregate information together to decide what to believe together. The first claim is that a betting market is a good forum in which different competitors could prove themselves, and be rewarded for contributing to the information we all rely on. And the core idea is that at any one moment, there’s a current price, and there’s a market where you could buy or sell, and if you have information that the market doesn’t, that suggests the price should be higher or lower, then you can immediately and directly profit from that by making a trade, buying or selling, and then waiting until the market knows what you know.
If that happens in an hour, then you get to wait an hour, and then the market will go up to the price that you think it should be, and then you can sell at that point, and walk away with your profit. If it takes 10 years, you may have to wait 10 years to walk away with your profit. But in either case, you’ve got this direct incentive to profit off of your information. So that’s saying this is a good forum. And it’s not just direct information about the topic that you could have. You could have information about everybody else’s biases. As you look at the price, you could say: those people I know, they’re just too gung-ho, and so the price is too high here, and I’m going to sell, to push it down, to reflect that. So you’re inviting everyone not just to learn anything they can about the topic, but to learn anything they can about the other participants’ biases, in order to correct those.
That’s the claim that it’s a good forum, and that’s the claim I say is supported by all this data that we have. But that’s different than “how should each competitor calculate things?” You were talking about: should they have a computer simulation, should they have a mathematical model, should they have a committee of people who talk together, should they have the same employees for 10 years or hire a new person every month? Those are how the competitor organizes, and that’s also true for sporting teams. Each sporting team has choices in basketball, say, about how many team members to have, how fast to rotate people on and off the field, whether or not to go for three-pointers or two-pointers. That’s a different set of strategy questions. But we don’t have to know the answer to that to say that the basketball game itself is a fair forum to find out who’s a good player.
Joseph Diller:
What’s your explanation of cases of failure,
like the 2016 election, where the polling industry did better than
prediction markets? I think
FiveThirtyEight
predicted a 70% chance for Hillary Clinton, and the prediction markets
predicted a 90% chance. I just wonder what your take is.
Robin Hanson
Whenever you’re predicting an event with a
probability—suppose you say there’s a 70% chance something will
happen—you are predicting that 30% of the time, you’ll be
wrong. That is, probabilistic predictions aren’t exact
predictions. The way to evaluate probabilistic predictions is to take
a dataset of many predictions and look overall at the accuracy. You
can look at calibration, and at accuracy. We have had many studies so
far of prediction markets in general, comparing their overall
calibration and accuracy to other sources, like polls, or corrected
polls, or committees, and things like that, and they say very
consistently that the prediction markets either do as well or better.
That doesn’t mean they’ll do better in each case, because they’re
probabilistic predictions. There’s a lot of correlations between
election predictions. Even if you have 50 states, it’s not like
there’s 50 separate data points there, because the surprise could make
them all higher than you expect, or all lower than you expect. So as
far as I can tell, they consistently do better.
Now, we have some other mechanisms that seem to do about as well and nearly as well, and that’s interesting, but I would mainly focus on all the other things in the world where other methods do much worse, and that we have all this potential to apply prediction markets to. We could do the tournament thing: is there another thing that sometimes does as well as a prediction market? Maybe there are, but there’s just all this other stuff we’re doing out there that’s so much worse. And so, what I’m most excited by is just: getting something good like a prediction market out there, where we’re not using anything like that.
Joseph Diller:
Right.
Matt Teichman:
Is there a paper we could point people to,
kind of a meta study of different prediction mechanisms, that present
some of the data on the increased accuracy of prediction markets?
Robin Hanson
I will point you to some of my papers where I’ve
summarized a dozen such studies in the introduction. But it is a
pretty consistent result.
Joseph Diller:
Back to 2016, one criticism is that the
betting markets were bound to reify themselves. Traders treated
market odds as correct possibilities, and didn’t update enough based
on public information. For example, in the last few weeks of the 2016
election, Hillary really took a plummet in sentiment, and
FiveThirtyEight was able to correct for this, but the prediction
markets did not.
Robin Hanson
FiveThirtyEight is a whole complicated system
where they’re making forecasts. It might be that on average,
FiveThirtyEight is as good as the prediction markets it compared to,
but I would want to see a study with a large data set, not just one
particular event, for that. The question is: what other institution
would you point people to? Think about the 2008 financial crash.
That’s a speculative market, a financial market. You could say: the
financial markets didn’t see the crash coming, so there’s a failure
for you. Ha-ha, the markets are wrong, right?
Matt Teichman:
Huh, I hadn’t thought of that. That’s
interesting.
Robin Hanson
Okay. And so I might say, well, at an earlier
date, the markets didn’t know as much as they knew later. That’s why
you saw the price change. So every time the price changes in any of
these markets, that’s evidence that the prior markets didn’t know as
much, because the market later is telling you: look, we now disagree
with what we said before. Our price before was different than it is
now. You could call that a failure, if you like, too, every time the
price changes. Every time the price changes, the market is admitting
that its previous estimate wasn’t so good.
But the question is, do you have another source that would be better (on average) to point people to? I might say, well, if you wanted to anticipate the 2008 crash, who else should you have been listening to, but the markets? Because the way most people found out about the crash was through the markets. There wasn’t another source. Now, there were individuals who were forecasting a crash, and maybe you should have been listening to those, but is there a general way to find those people to make a general institution for forecasting, other than figuring out who was right after the fact? Similarly, about elections, before 2016, I’m not sure we could say that FiveThirtyEight was just always consistently better than the betting markets. If they happened to be better in 2016, that might move the evidence toward them being as good. But hey, it’s just one event. What you really want to look at is the overall stats.
Joseph Diller:
Right. And FiveThirtyEight still got it wrong.
Robin Hanson
But everything is going to be
wrong. Forecasting is hard. The world is complicated. The standard
shouldn’t be, “Were you ever wrong?” The standard should be: who else
was more right, at the same time, with similar resources?
Matt Teichman:
And how can you make yourself more right,
next time you do an important prediction?
Robin Hanson
Which institutions should you rely on? Which
institutions have proved themselves to be as right in the past? But
again, the most interesting thing here is that prediction markets are
clearly much better than the other institutions we have in society for
most other things. We don’t have a FiveThirtyEight for most other
parts of society.
Matt Teichman:
Including the one you opened with: is somebody
going to make a deadline at work? We don’t have a FiveThirtyEight for
that.
Robin Hanson
Exactly. You might have some forecasting wing of
your organization that’s in charge of coming up with predictions for
such things. But our experience is that if you had a betting market—
Matt Teichman:
—I feel like office pools tend to be about
the Oscars and stuff—not about important things.
Robin Hanson
Indeed. That’s a problem.
Matt Teichman:
So part of the proposal is to make them be
about important things.
Robin Hanson
Have more betting markets about things that
matter. The first obstacle is legal; that is, there are often legal
obstacles.
Matt Teichman:
Oh, yeah; I wanted to ask about that. When is
it illegal to do this?
Robin Hanson
The main obstacle to betting markets is gambling
law. And the standard gambling description has three parts. A gamble
is where you put consideration in, you might get consideration out,
and there’s chance in between. That’s a gamble. An ordinary casino,
for example, has that pretty obviously. You walk up with cash, you
roll the dice, and you might walk away with money (or not), depending
on the dice. In order to break that gambling part, you have to break
one of those three parts, or be declared an exception. So almost all
the financial markets you know about—stock markets, commodity
markets, currency markets, option markets, insurance—they were all
once illegal as gambling, because they looked like this. They had
money in, money out, and chance in between.
Over time, we carved out some of these and declared them not to be gambling because they seem to be useful. And that’s a prospect here, too: we could carve out some of these markets and say they are prediction markets, not gambling, because they’re useful. But if you don’t have that available to you, you can also use this three-part definition to avoid gambling laws.
Matt Teichman:
Could we put them in a casino, where it’s
regulated and allowed? Would that be another solution?
Robin Hanson
That’s another way: just to have it be
considered legal gambling. For example, poker players have, at times,
declared themselves not gambling and gotten away with it, by saying
that poker isn’t chance. It’s a “skill,” you see, and therefore not
gambling. So we could try to get away with prediction markets saying:
well, the traders are showing a skill, and therefore, it’s not
gambling. Another way is that you could have the money go to charity,
not back to you. That’s the basis of Long
Bets, which is a website where people bet on
things and then have the money go to charity.
A third way to do this is to not have people put the money or the consideration in. Inside a corporation, like with the deadline, if the company gives each player their stakes, then they bet, and they walk away with their cash. That’s also not gambling, because they didn’t put the money in.
Matt Teichman:
So under our current system, the only way to
make some of these more adventurous prediction markets work is to find
a loophole of this kind.
Robin Hanson
Well, as I said, at the moment, there are some
legal betting markets in the United States, like PredictIt and Kalshi,
but they’re very limited. We might want them to apply to a wider range
of things. For example, Kalshi is applying to allow markets on the
upcoming elections, and PredictIt’s rights were taken away
(apparently) by complaining that they were doing too many political
topics. So you can see there’s an obstacle here to wider legality, but
inside organizations, it’s not an obstacle. I would say the main
obstacle to prediction markets at the moment is not the legality,
because the most interesting applications are these internal
organization applications, and they are mostly legal, with respect to
gambling law.
Matt Teichman:
What are some examples of applications
within an organization? We talked about one: we could maybe have a
market to estimate whether such-and-such project is going to finish in
time. What are similar uses of prediction markets within an
organization?
Robin Hanson
Most salespeople have incentives tied to
sales, but what you need is a benchmark: what’s the expected sales,
if people don’t put in more effort. So prediction markets can give you
sales estimates.
Matt Teichman:
Yeah—any time there’s a metric of success in
a company, you can make predictions about who’s going to be
successful.
Robin Hanson
Right, and you can use that to tie incentives,
as with the sales commission. You can predict events for competitors:
whether they’ll introduce products, or what prices they’ll have for
their products. You can predict the larger economy—recession, boom,
inflation—and you can predict what decisions will be made. But the
thing I’m most interested in, in organizations, is predicting what
decisions you should make.
That’s called a decision market. The idea of a decision market is to say: if we changed our decision, how would the outcomes change? Think about that deadline. We can have a market in a deadline, and it says, okay, there’s a 70% chance we’re going to make the deadline. And now we might ask, well, how does that chance change? If we change who’s in charge, add personnel, change the—
Matt Teichman:
—redefine the scope of the project.
Robin Hanson
Exactly. And then we might get conditional
forecasts that show how the outcome would change by changing some of
the conditions. That is much more directly decision-relevant.
Joseph Diller:
So assuming the success of prediction markets,
what problem do you think prediction markets are trying to solve? How
does it work?
Robin Hanson
I would think of a prediction market as a way
to buy information. That is, if there’s something you want to know
about the future—about what will happen, including
conditionally—then you subsidize a prediction market, and then it
induces participants to come trade and try to profit. That gives you
this information about the answer to your question. And so, if you
don’t want to know anything, this isn’t a very useful product. Or if
there are some things for which knowing them hurts you, then this
product could hurt you. But the key problem is that we need the people
who want the information to pay for it. If there’s just things that
everybody would benefit from knowing, but nobody’s willing to pay for,
then unfortunately this doesn’t happen. That’s just a generic thing in
the economy: nothing happens unless somebody pays for it. So somebody
has to step up and pay for what matters.
Joseph Diller:
We need to change the incentive structure.
Robin Hanson
For the many questions that we all share an
interest in, where we don’t individually want to pay. But we have many
topics, like the project deadline, where there is a party who should
be willing to pay, who does care, and would be the most interested
party. Therefore, they’re the obvious one to pay.
Matt Teichman:
Certainly, if you’re a software company, for
example, setting realistic project deadlines is a notoriously hairy
task, because unexpected difficulties come up. That’s precisely the
kind of difficult prediction that you might want to bring in heavier
muscle to do accurately.
Robin Hanson
Right—now, I think we need to face the fact
that many organizations should want prediction markets, but don’t
seem to.
Matt Teichman:
A lot of it is seat of the pants: we have our
fearless leader decide, or people use their guts.
Robin Hanson
But I think it’s more than just habit and
blindness. I think there’s actually more of a systematic obstacle
here, and that’s something I’ve been trying to think a lot about and
deal with, over the last few decades. I should mention that in terms
of my history, I was here at University of Chicago, down the street at
Burton-Judson
dorms, starting in 1981. I’ve been around here for 41 years, and I
started to do prediction markets and think about that since 1988. That
was 33 years. So I’ve been in this topic for a long time—I’m way
past the initial point of thinking: this sure looks like a cool
idea. Why don’t we tell people about it and see if they might be
interested?
Matt Teichman:
You’ve made some attempts to actually
make it happen.
Robin Hanson
Right, and I’ve seen a lot of reactions. I’ve
seen people try things, get excited, and I’ve seen what people maybe
are reluctant to do, and why they aren’t interested. Those are
important things to engage with—to understand—to try to make the
next steps. I’ve done this talk about: look at this cool possibility,
and look at the potential here for a long time. But once you get past
that, you do need to think about: well, why haven’t people adopted
this? What have they said? Where have been the most successful
applications? What seem to be the real obstacles?
Matt Teichman:
So what would you say is the main reason for
resistance, when you’ve encountered it?
Robin Hanson
If you’re talking about large scale public
things, there’s these gambling limitations. Even when those exist,
there’s the lack of who will pay, but if we go to organizations where
they should want to pay, and where it’s important to them, I’d say we
hit obstacles because prediction markets disrupt the local political
equilibrium. That is, most organizations pretend to be well-honed
machines, that are producing efficient products, and managers mostly
pretend to be spreadsheet managers, who collect information and
calculate the best thing to do. But in fact, managers are mostly
politicians, who are part of coalitions, who are supporting each
other and trying to undermine rival coalitions. Prediction markets can
be disruptive to that. I can explain in more detail how.
Joseph Diller:
It seems to me that you think prediction
markets will promote integrity in forecasts: that they will be more
true. But are you worried about what goes on in gambling, sometimes,
such as fixing? You know, the Chicago Black
Sox is an example,
where the prediction doesn’t confirm reality. Is that a worry of
yours?
Matt Teichman:
What are the potential security holes, if any,
in the system?
Robin Hanson
I call this foul play: the general category of
all the malicious behavior you might be worried about. There are many
different kinds of foul play to walk through, to try to think about
how robust the system is to foul play. Remember always to keep in mind
that the relevant comparison is between this system and a different
system on the same problem. Some problems may just be intractable,
but not any worse for this than others, and for some problems, the
system may do better or worse.
Let’s consider some kinds of foul play. Sabotage, for example, is one kind of foul play. You might go change the world to win a bet. Now, that’s a way in which the system for producing information is producing information, but you don’t want information to be produced that way, because they’re causing the world to be the thing they’re predicting, and then successfully predicting the world to be what they cost. For example, somebody on a project with a deadline might sabotage the project. They might tank it, and make sure it doesn’t make the deadline, in which case they could successfully use their information about having sabotaged the product to make money in the prediction markets, and they are making an accurate forecast. But they’re still undermining the purpose, which was to help make the project work.
Matt Teichman:
Yeah, everybody has got some coworkers in mind
who you’d suspect of doing that. Bob in accounting.
Robin Hanson
So one approach in prediction markets to deal
with, say, a project sabotage is to give everyone a positive stake in
the outcome that you want, which is making the deadline. Then let them
reveal their information by changing their stake above and below that,
but never below zero. Give everybody (initially) $100, if the project
makes the deadline. Then somebody who’s skeptical about making the
deadline, they will sell some of that—you know, 30 of it, move down
to 70 if they make the deadline—and then they’re revealing their
skepticism, through that sell. Other people are optimistic, and buy
more and more, from 100 up to 120, but if they all have to stay above
zero, then they all want the project to succeed. So it’s quite
possible to just make sure people have the right stakes in the
outcomes you want, and then limit the degree to which their betting
stakes can change from that. That avoids the sabotage problem.
That’s not the only kind of foul play you can imagine, of course. Another kind of foul play is gossip, or just lying. That’s a common thing in most speculative markets: some people will just try to lie and say false things, or misleading things, in order to mislead other traders, who would then trade on their misleading claims. That, as you know, is a problem in pretty much all institutions. In most speculative markets, the advice is: well, you’re a fool if you just believe somebody’s claim without substantial supporting evidence, or a track record to make you believe their claims. The fact that they might make money on that is a reason to be wary of all the things they say. That’s an attitude we have in a wide range of institutions.
Joseph Diller:
So is your criticism of institutions like
academia or policy makers, politicians, that they have less
integrity than a prediction market, or—
Robin Hanson
—well, the main claim is the prediction market
forecast will be more accurate, and for each of the other
institutions, we can start to criticize the reasons why it might be
less accurate. Sometimes it’s because they just don’t have any
incentives to do much of anything accurately, or sometimes it’s
because they have poor incentives that make them do the wrong
things. It depends on the other institutions. We could walk through
concrete examples if you’d like. But the claim is that this
institution has a nice combination of incentives and the kind of
information flow—who gets to see what—and the ability to see other
mistakes, and correct them. All these are a nice feature of the
betting markets.
Joseph Diller:
So what are some examples of prediction
markets, applied in the real world?
Robin Hanson
As you know, there are all these betting markets
on elections and they often give you information about the
election. They are unfortunately not very often set up in a way to
give you advice about who to vote for. But for example, there are
often markets in the US presidential elections in who’s likely to be
nominated and who’s likely to win. If you divide the probability of
being nominated into the probability of winning, what you’ll see is
that the ratio there is the conditional probability of winning, if
nominated. That’s, in essence, advice to each party. Who should you
nominate, if you want to win? That’s an example of a decision market,
of giving decision advice. At some points, there have been markets
like that in blockchain and Bitcoin, advising major organizations
about changes in their policies, regarding such coins, and predicting
the value of the coin, conditional on their choice.
Matt Teichman:
What about something more adventurous, like
along the lines of what we discussed? Have there been any companies
that have made hiring decisions on this basis, or any countries that
have legislated laws on this basis? Or anything a little more out
there?
Robin Hanson
I can describe many hypothetical applications,
and what I advise, but in terms of actual applications, it has been
modest. There was someone who made some prediction markets on major
outcomes in the US conditional on whether the Democrat or the
Republican became president—oil prices, stock prices, unemployment,
things like that—but they didn’t get a lot of trading in those
markets. Hopefully someone will try that again. Somebody is at the
moment doing markets like that in the Brazilian national
elections—national Brazilian outcomes, conditional on which party
will win the next elections.
Matt Teichman:
That was another potential security hole I was
wondering about. Suppose we set up a prediction market that’s going
to be the main basis for an important decision, like hiring somebody
at a company. And then, for whatever reason, due to random
circumstance, we can’t get anybody to invest in it. There aren’t
enough people in the market to really render a verdict. Is there
potentially a risk of that happening?
Robin Hanson
If you have a market and nobody comes, then you
can just ignore it, and do whatever you’d do instead. The downside is
low, in that case. The risk would be more if it gave you a misleading
result, and you didn’t know it was a misleading result. Merely having
a failure participation that you can see is less of a problem.
Matt Teichman:
You can have a fallback method for deciding,
if there’s not enough people that end up investing.
Robin Hanson
But you can subsidize these markets, if you want
more participation. There’s a very simple, direct way to take a market
and subsidize it in a way so that the money only goes to the people
who actually end up knowing more about the answer, and doesn’t go to
everyone else. Markets can sometimes be manipulated, in the sense that
some people have an incentive to manipulate it, and that actually is a
substitute for a subsidy. That is, it creates a subsidy when people
are trying to manipulate markets and it doesn’t lower the accuracy. It
actually increases the accuracy when people want to manipulate these
markets. When there’s a decision, you might imagine someone trying to
manipulate the markets, because they want to influence the decision
that the markets will recommend.
Matt Teichman:
So I can’t resist asking about this notion
you’ve dubbed “futarchy” on your blog, and in a
paper, which
contains a very different job description for our legislators. How
does that proposal go?
Robin Hanson
First, let me just describe the basic concept of
futarchy, which is decision markets for governance. I want to
describe it first in the corporate context: that would be the simplest
to apply it to. That’s firing the CEO. The CEO is the most important
position in a firm, and sometimes they need to be fired. The board of
directors is in charge of that, and they’re supposedly too shy about
this.
Matt Teichman:
Maybe they’re friends with the CEO, or
whatever.
Robin Hanson
Exactly. So, here’s how we could do that. In an
ordinary stock market, you trade stock for cash, and then a person
asking whether the current price of e.g. 22 is too high, or too
low—what they need to do is average over all the scenarios they can
think of that the company might be involved in, and, in each scenario,
ask how much is the company worth there, and then average those
together, to produce an overall value for the company.
I want to make two alternative stock markets that both also trade stock for cash, but now each of these markets is going to be called off, if a condition isn’t met. The trades will be as if they never happened, if a condition isn’t met. In one of the markets, we’re going to trade stock for cash, conditional on the CEO staying in office until the end of the quarter, and in the other market, we will make all trades conditional on the CEO not staying in office until the end of the quarter.
Matt Teichman:
And if the condition isn’t met—
Robin Hanson
—the trade never happened.
Matt Teichman:
So all of everybody’s bank balances rewind to
the state they were in before, basically?
Robin Hanson
Right. You really just keep track of their
assets in the different states. And so, these conditional markets will
have a conditional price, and these two conditional prices will be
different than each other, and the price in the unconditional
market. Now, the difference in those prices is advice about whether
to keep the CEO. That is, if the market says that the company is worth
more, if the CEO leaves it to the end of the quarter, then that’s
saying that you should get rid of the CEO. That would be a very simple
way to ask speculators whether or not to keep the CEO, in a way that’s
robust to people trying to manipulate it, and to people having all
sorts of diverse opinions, not knowing really who knows about this,
because that’s what betting markets do well. That’s an example of
futarchy, applied to a particular important decision.
The structure here, more generally, is to have a decision market. We need an outcome that we care about: in this case, the stock price of the company. Then we need some discrete decision options to compare: if we do this, what do we expect the outcome to be, versus if we do that, what do we expect the outcome to be? Now we can apply that more general mechanism to a larger scale governance. We could do many other kinds of corporate governance with this mechanism, because we could keep using the stock price as the outcome. We could ask about firing other people, we could ask about merging or spinning off or buying another firm. We could talk about expanding the firm into new regions.
For each of those kinds of decisions, we could ask this decision market to estimate the value of the company if we make a certain decision, versus if we make an alternative decision. That’s futarchy for a corporation, and it’s relatively easy, because corporations come with this nice numerical outcome measure: the stock price, which represents the value of the firm.
For other kinds of governance, say a city, or even a nation, we need to come up with an alternative numerical measure for the welfare of that unit. For a nation, that would be national welfare. So in order to make futarchy work for a nation, if we need a measure of national welfare—that is a number that we will have in the future about how well off we have been—then we can have a system where anytime someone wants to make a new bill, a proposal for changes in our policies, then we have a decision market where we ask speculators, “Well, what is national welfare if we make this change, versus if we don’t make this change?” Then we could compare those two prices, and that would recommend whether we make this change. And so, futarchy would be a system where we just make that change, whenever the markets recommend it. If the markets say our welfare goes up if we invade Canada, then we invade Canada.
Matt Teichman:
So we keep the part where congressmen consider
what to do, and then we get rid of the part where we do whatever
everybody votes to do, and replace that with whatever the market says
to do? Is that right?
Robin Hanson
We’re going to ask the speculators to pass bills
or not, and not ask the legislature to do that. But we’re going to
ask legislatures: what’s national welfare? What counts for welfare?
Matt Teichman:
How do we measure whether we’re doing better?
Robin Hanson
Right. At the moment, we have things like GDP,
which do encompass a lot of the things we care about, but we know they
also include other things that maybe we care less about, or don’t
include things we do want to include. Then the task of the
legislature is to oversee the extension and measurement of a richer
measure of national welfare. That can include the number of trees, and
international reputation, and leisure time, and all the other things
that we don’t think GDP includes.
Matt Teichman:
It’s almost as if the politicians are becoming
more like philosophers in this scenario!
Robin Hanson
Well, yes. In this scenario—
Matt Teichman:
—obviously, Joseph and I like it.
[ LAUGHTER ]
Robin Hanson
Legislators would be less focused on short-term
management of the economy, and short-term management of policy, and
more focused on what we want in the long run. Then, voters electing
politicians would want to elect politicians who are more focused on
those long-run value judgments than on short-term policy estimations.
Joseph Diller:
So the politicians are in charge of the value,
and the prediction markets are in charge of the policy.
Robin Hanson
Right. The slogan is: “Vote on values, but
bet on beliefs.”
Matt Teichman:
Robin Hanson, thanks so much for joining us.
Robin Hanson
Thanks for having me.
Elucidations isn't set up for blog comments currently, but if you have any thoughts or questions, please feel free to reach out on Twitter!