We're used to thinking about probabilities like 50% for a coinflip. If you keep flipping a coin, it comes up heads about half the time, just as the probability says. You can see this happening in the graph below, where the actual share of coinflips that come up heads keeps getting closer and closer to the supposed probability.

The probabilities featured here seem different, however. What could a 50% probability mean for something that happens just once? This issue is controversial, but because this is my own forecast, I can provide you with the intended interpretation of my own odds.

When my forecast says something has a certain x% chance of happening, the intended meaning is that after a sufficiently large number of forecasts have been made, close to x% of all events given an x% chance of occurring will have occurred. If you want more precision, I believe that after I have made 100 forecasts with an x% chance, plus or minus 5%, x% ± 10% of them will have truly occurred.

To provide an example, I believe the Democrats have about a 90% chance of winning the Oregon US Senate race. Once I have made 100 predictions yielding a probability between 85% and 95%, I believe somewhere between 80% and 100% of them will have actually occurred. The wide boundaries here are because 100 forecasts isn't a lot, so the actual frequency might not converge on the forecasted frequency by random chance. (Notice how in the coinflip graph, it still deviates a bit from 50% after almost 500 flips.)

Unlike normal probabilities, these forecasted odds are claims about a class of events that the event in question belongs to, not claims about the natural frequency at which a random event like a coinflip occurs. Forecasted odds can be considered wrong when it takes "too long" for the actual frequency to converge on the forecasted frequency. I might claim 100 times that various events have 70% odds, and if by that point just 50% of them actually happened, we could say my forecasts were incorrect or "poorly-calibrated."

This type of "probability" has some interesting properties that distinguish it from normal probabilities. I could forecast that a coin has 90% odds of landing on heads, and even if it comes up tails, the forecast will be "accurate" if I make 9 other forecasts about totally different things like dice rolls and card draws, and get them right 90% of the time, just as claimed. Despite this weird quality, the forecasts provided here have the nice property of mostly reflecting things like polls, which we have good reason to expect to reflect the underlying reality of the situation. Thus, there shouldn't be too many cases where we claim that something more akin to a coinflip has a 90% chance of happening.

I do not have a long record of forecasting, but the forecasts I have made so far have been very good. In 2020, I forecasted that Republicans would win about 48 Senate seats, and they went on to win 50, losing control of the chamber. The average probability I gave to the Republican in each race was about 52%, and they actually won 57% - thus, the empirical frequency was already converging on the claimed one.

In 2024, I forecasted the presidential race between Kamala Harris and Donald Trump. While the overall odds my forecast gave provided a slight edge to Harris, this was essentially the product of random chance, since they were close to 50/50 and I was relying on random simulations. The forecast I provided using my own judgment, however, was almost perfect: I judged that Trump would win, and only failed to predict that Harris would win Nebraska's 2nd district. The post in which I made this forecast can be viewed here. Sadly, my 2020 forecast was an unpublished school project, so you will have to take my word for it. The spreadsheet containing the forecast is here, but of course, a devilish forecaster could fabricate such a thing and make it appear older than it really is.