How do you calculate Brier skill relative to climatology?
Brier skill score expresses the reduction in Brier score relative to a reference forecast. Use the same verification windows, outcomes and weights for both systems.
Start with Brier score practice for probabilistic supercell forecasts.
| Forecast | Brier score | Role |
|---|---|---|
| Model | 0.12 | Verification sample |
| Climatology | 0.16 | Same verification sample |
Worked example
If BS is your forecast’s score and BSᵣ is the reference score, BSS = 1 − BS/BSᵣ. With 0.12 and 0.16 the score is 0.25, a 25% reduction in mean squared probability error relative to the reference. For a climatology reference, estimate the frequency from a suitable training period and apply the reference to the verification windows. The result is undefined when the reference score is zero.
Hint for the exercises
Keep the model score in the numerator and the reference score in the denominator.
Practice questions
- Compute Brier skill from the two scores.
- Find the skill if both scores equal 0.16.
- Compare model A with BS 0.10 and B with BS 0.18 using the same 0.16 reference.
- A reference scores 0.20 and a model reduces that score by 40%. Find the new Brier score.
- Why would reference windows from a quieter season make the comparison unreliable?
- What does BSS = 0.25 tell you about the fraction of events detected?
Worked answers
Answer 1
1−0.12/0.16 = 0.25.
Answer 2
1−1 = 0. It matches the reference.
Answer 3
A gives 0.375. B gives −0.125 and scores worse than the reference.
Answer 4
0.20 × (1−0.40) = 0.12.
Answer 5
The two scores would measure different outcomes and base rates. Evaluate both forecasts against the same eligible cases.
Answer 6
It describes relative squared probability error, not the detection fraction. Detection requires a decision threshold and contingency counts.
- 1−0.12/0.16 = 0.25.
- 1−1 = 0. It matches the reference.
- A gives 0.375. B gives −0.125 and scores worse than the reference.
- 0.20 × (1−0.40) = 0.12.
- The two scores would measure different outcomes and base rates. Evaluate both forecasts against the same eligible cases.
- It describes relative squared probability error, not the detection fraction. Detection requires a decision threshold and contingency counts.