How the Forecasts Work — Simulation Methodology

A technical explainer for readers who want to know how we arrive at the numbers on the league, tournament, and parity pages — championship odds, playoff-qualification probabilities, and projected final standings.

What this is, and what it isn't. This is an analytical model built for insight and conversation — "how might the season play out, and how sure are we?" It is not a betting tool or a decision-making system. Youth hockey is high-variance and our data is thin (about 34 games per team in a season), so every number here is a genuine probability with real uncertainty around it, not a prediction of what will happen. We've gone out of our way to make the numbers honest about how unsure they are; please read them in that spirit.

1. The question the simulation answers

Given the games played so far, we want the distribution of ways the rest of the season could go:

• Championship probability — how often each team wins its tournament.
• Advancement / qualification probability — how often a team finishes in a spot that matters (top-2 out of a pool, a parity-tournament bid, a playoff berth).
• Projected final standings — the range of places each team could finish.

There is no formula for these. The only honest way to compute them is to play the remaining games thousands of times and count how often each thing happens. That's a Monte Carlo simulation.

2. The two kinds of uncertainty (this is the important part)

A good forecast has to account for two separate sources of randomness, and getting this right is the difference between an honest number and an overconfident one.

1. Game randomness ("aleatoric"). Even if we knew exactly how good two teams were, hockey games are noisy — the better team doesn't always win. A 3.8-goals-per-team league has a lot of bounce.
2. Rating uncertainty ("epistemic"). We don't actually know how good each team is. We estimate it from ~34 games, which is not many. The team that looks best on paper might genuinely be the best — or might just have had a friendly schedule and good luck.

A simulation that only models the first kind looks far more confident than it should. It effectively says "assume our power ratings are exactly correct, now roll the dice" — and the ratings are not exactly correct. We model both.

3. The pipeline, end to end

Step 1 — Rate the teams (Dixon-Coles)

We fit a Dixon-Coles model, the standard approach for goal-scoring sports. It gives every team two numbers:

• an attack rating (how many goals they tend to score), and
• a defense rating (how many they tend to allow),

estimated jointly from every game's final score so that each team's rating is adjusted for the strength of the opponents it actually played. Beating strong teams counts for more than beating weak ones. From any two teams' ratings the model produces the full probability of every possible scoreline — which is what we need to simulate a game.

The ratings are shrunk toward the league average: with only ~34 games of evidence, raw estimates over-react to small samples, so we pull them gently toward "average" by an amount tuned and validated on five years of data. (Full detail: docs/model-research/03-shrinkage.md.)

Step 2 — Capture rating uncertainty (the bootstrap)

To represent the fact that we don't know the ratings, we don't fit them just once. We refit them many times on bootstrap resamples of the season — each resample draws the same number of games at random, with replacement, so each refit sees a slightly different version of the season and produces a slightly different set of ratings. The spread across these refits is our uncertainty about each team's true strength.

(In a future version these refits become draws from a Bayesian posterior, which is the same idea done more formally; the bootstrap is the current implementation. See docs/model-research/12-future-directions.md.)

Step 3 — Play the rest of the season, many times

For each rating sample from Step 2, we run a share of the Monte Carlo trials. In every trial:

• Each remaining game is played by drawing a random scoreline from the Dixon-Coles probabilities for that matchup.
• Already-played games are kept as they really happened — we never re-simulate a completed result.
• Ties that must be broken (a tournament game that can't end level) are resolved by a 50/50 overtime coin flip.
• Standings are computed with the league's real tiebreaker chain, in order: points → regulation wins → head-to-head (for two-way ties only) → goal quotient → regular-season seed.
• Pool winners advance into the crossover bracket (1A vs 2B, 1B vs 2A), and the bracket is played out to a champion.

We tally who finished where. Because Step 3 runs across many different rating samples from Step 2, the final tallies fold together both kinds of uncertainty at once: game-to-game randomness and our uncertainty about the ratings.

Step 4 — Turn tallies into probabilities

A team's championship probability is simply the fraction of all trials in which it won. Advancement and finish-position probabilities are counted the same way.

4. How many simulations, and how precise?

We run 50,000 trials per forecast, split across the rating samples. That's enough that the counting error is about ±0.2 percentage points — far smaller than the uncertainty that actually matters (the rating uncertainty from Step 2). In other words, we deliberately spend the computational budget where it counts: on exploring different plausible team-strength scenarios, not on grinding the coin-flip noise down to a meaningless extra decimal.

5. A worked illustration

To show why Step 2 matters, here is the title race in one division (2017 Platinum), computed two ways — first the old way (one fixed set of ratings), then the honest way (rating uncertainty propagated):

The favorite is still the favorite — but "73%" is the honest number, and the chasers' real chances roughly double. The same mechanism now drives every championship and advancement number on the site. (Reproduce: PYTHONPATH=. uv run python -m scripts.model_lab.sim_uncertainty.)

6. How to read the numbers

• They're probabilities, not predictions. "Team X: 73% to make playoffs" means that if this exact situation played out many times, they'd qualify in about 73 of 100 — and miss in about 27. Upsets are not bugs; they're the 27.
• Round, don't squint. We show rounded figures and ranges on purpose. The difference between 71% and 73% is noise; the difference between 30% and 70% is real.
• Two close teams may be a genuine toss-up. When the ranges overlap, the model is telling you it can't separate them — and that's useful information, not a cop-out.
• It's for fun and discussion. Use it to argue about who's hot, who's underrated, and what needs to happen down the stretch — not to make decisions that matter.

7. Assumptions and limitations (in plain terms)

• Games are treated as independent given the ratings. We don't model momentum, injuries, or who's in net on a given night — we have no reliable data for those, and inventing them would add false precision. (Goaltending is the biggest thing we can't yet see; see docs/model-research/12-future-directions.md.)
• Tournament games are simulated at a neutral site, which is how these tournaments are run.
• Overtime is a 50/50 coin flip. A slightly better team probably wins a bit more than half the time, but we keep this simple and honest rather than inventing an edge we can't measure precisely.
• One season of data. Ratings reflect the current season only; a team that improves mid-season is captured as the season's games accumulate, but we don't yet carry a memory across seasons (a planned improvement).
• The model can't see what isn't in the box score — coaching, line matchups, a kid having a breakout month — beyond how it shows up in goals.

8. Reproducibility and where the code lives

Every simulation stores the random seed it used, so any published number can be regenerated exactly. The implementation is open in the repository:

• Rating model: src/engine/dixon_coles.py
• Simulation engine: src/engine/monte_carlo/ (simulate_tournament, simulate_tournament_ensemble)
• Orchestration + uncertainty bootstrap: src/pipelines/run_engine.py (run_simulation, _bootstrap_param_samples)

Deeper reading

• docs/model-research/13-simulation-forecasting.md — the analysis that motivated the uncertainty-propagation change, with the measured impact.
• docs/model-research/00-INDEX.md — the full model-research program (how the ratings themselves were built and validated).