Skip to content

How do you calculate Brier skill relative to climatology?

Brier skill score expresses the reduction in Brier score relative to a reference forecast. Use the same verification windows, outcomes and weights for both systems.

Start with Brier score practice for probabilistic supercell forecasts.

Worked values
ForecastBrier scoreRole
Model0.12Verification sample
Climatology0.16Same verification sample

Worked example

If BS is your forecast’s score and BSᵣ is the reference score, BSS = 1 − BS/BSᵣ. With 0.12 and 0.16 the score is 0.25, a 25% reduction in mean squared probability error relative to the reference. For a climatology reference, estimate the frequency from a suitable training period and apply the reference to the verification windows. The result is undefined when the reference score is zero.

Hint for the exercises

Keep the model score in the numerator and the reference score in the denominator.

Practice questions

  1. Compute Brier skill from the two scores.
  2. Find the skill if both scores equal 0.16.
  3. Compare model A with BS 0.10 and B with BS 0.18 using the same 0.16 reference.
  4. A reference scores 0.20 and a model reduces that score by 40%. Find the new Brier score.
  5. Why would reference windows from a quieter season make the comparison unreliable?
  6. What does BSS = 0.25 tell you about the fraction of events detected?

Worked answers

Answer 1

1−0.12/0.16 = 0.25.

Answer 2

1−1 = 0. It matches the reference.

Answer 3

A gives 0.375. B gives −0.125 and scores worse than the reference.

Answer 4

0.20 × (1−0.40) = 0.12.

Answer 5

The two scores would measure different outcomes and base rates. Evaluate both forecasts against the same eligible cases.

Answer 6

It describes relative squared probability error, not the detection fraction. Detection requires a decision threshold and contingency counts.

Sources