How good are AGORA forecasts? (continued)

Mark Roulston, Senior Data Scientist •


How good are AGORA forecasts? (continued)

AGORA prediction markets assimilate new information as it becomes available, providing up-to-date forecasts in dynamic situations.

In a previous post we looked at the calibration of forecasts produced by AGORA prediction markets. That is, whether the actual frequencies of events being predicted match the probabilities predicted by AGORA. We found that, so far, the prediction markets run on AGORA have produced forecasts that are consistent with being well calibrated.

“Forecasts that are both sharp and well calibrated are the Holy Grail of forecasting.”

Calibration is an important property for probability forecasts but it isn’t the only one. If I repeatedly predict that the chances of a fair coin coming up heads is 50% my forecasts will be perfectly calibrated but not very informative. If you found someone who repeatedly predicts the chances of heads as either 10% or 90% and their predictions are well calibrated — with heads coming up 10% of the time when they say 10% and 90% of the time when they say 90% — you’d be a lot more impressed (and may invite them to accompany you on a trip to Monte Carlo). This is because their forecasts are sharper. For binary forecasts (will something happen or not) sharpness means forecasts closer to 0 or 100%. When predicting a numerical value, a sharp prediction has a narrower probability distribution, illustrated in Figure 1. Producing forecasts that are sharp is trivial but forecasts that are both sharp and well calibrated are the Holy Grail of forecasting.

Figure 1: An illustration of “sharpness” for a probability forecast of a numerical quantity.

“We can calculate how informative a forecast is using information theory.”

If it is well calibrated, the sharper the forecast the more informative it is. We can formally calculate how informative a forecast is by using ideas from information theory. Consider a situation where there are 8 possible outcomes which initially are equally likely. These outcomes can be represented using 3 bits (000, 001, 010, 011, 100, 101, 110, 111). Learning exactly which outcome has occurred from this initial state of knowledge would hence require 3 bits of information. Now, suppose we have a forecast that has narrowed down the original 8 outcomes to only 4 equally likely outcomes. Representing these outcomes requires 2 bits (00, 01, 10, 11). Possession of the forecast reduced our ignorance from 3 bits to 2 bits, so we can say that the information content of the forecast was 1 bit.

More generally, if the probability assigned to the outcome that actually occurs is p then our ignorance before we knew the outcome was

If the initial probability assigned to the outcome that actually occurs was p(1) and at time t the predicted probability was p(t), then the information gain is the reduction in ignorance given by

Each bit of information gained is equivalent to halving the number of possible outcomes. Expressing the information content of predictions like this is really just a way of interpreting the logarithmic scoring rule proposed by the statistician I.J. Good in 1952. The logarithmic score is similar to the Brier score for evaluating probability forecasts in that they are both proper scores: a forecaster will get the best expected value of a proper score by reporting the probabilities they truly believe.

Figure 2 shows the average information gain during 23 AGORA prediction markets. The markets included were all the public, cash incentivized markets that have been run on the platform to date, including markets for UK temperature and numbers of new cases of COVID-19. The evolution of the information content of the forecasts is shown against the normalized lead time, which is the fraction of time between market opening and settlement remaining until settlement.

Figure 2: The average information gain (in bits) of 23 AGORA forecasts between the time the market opened and final settlement.

Figure 2 illustrates how the information content of AGORA forecasts increases while the prediction markets are open, meaning that they are doing their job of eliciting and aggregating information from the participants. At a normalized lead time of 0.2 AGORA forecasts have effectively reduced uncertainty by a factor of 4 relative to the initial state of knowledge and immediately before settlement this reduction in uncertainty has increased to a factor of 16. We cannot say whether these are “good” reductions in uncertainty because that depends on the nature of the thing being predicted and what alternative forecasts are available. Having one bit of information about the time of sunrise is unimpressive — given its intrinsic predictability — while having one bit of information about next week’s lottery numbers would be phenomenal.

The strength of prediction markets as a forecasting tool is the way they assimilate information. If a participant has a better forecast they have an incentive to trade using that forecast and in doing so inject the information in their forecast into the collective prediction. More people contributing to a forecast reduces the risk of being surprised by “unknown unknowns”.

To speak more about how prediction markets and Hivemind AGORA could help with forecasting in your organisation please get in touch.