Sampling uncertainty and confidence intervals for the brier score and brier skill score

A. Allen Bradley, Stuart S. Schwartz, Tempei Hashino

研究成果: ジャーナルへの寄稿記事

32 引用 (Scopus)

抄録

For probability forecasts, the Brier score and Brier skill score are commonly used verification measures of forecast accuracy and skill. Using sampling theory, analytical expressions are derived to estimate their sampling uncertainties. The Brier score is an unbiased estimator of the accuracy, and an exact expression defines its sampling variance. The Brier skill score (with climatology as a reference forecast) is a biased estimator, and approximations are needed to estimate its bias and sampling variance. The uncertainty estimators depend only on the moments of the forecasts and observations, so it is easy to routinely compute them at the same time as the Brier score and skill score. The resulting uncertainty estimates can be used to construct error bars or confidence intervals for the verification measures, or perform hypothesis testing. Monte Carlo experiments using synthetic forecasting examples illustrate the performance of the expressions. In general, the estimates provide very reliable information on uncertainty. However, the quality of an estimate depends on both the sample size and the occurrence frequency of the forecast event. The examples also illustrate that with infrequently occurring events, verification sample sizes of a few hundred forecast-observation pairs are needed to establish that a forecast is skillful because of the large uncertainties that exist.

元の言語英語
ページ(範囲)992-1006
ページ数15
ジャーナルWeather and Forecasting
23
発行部数5
DOI
出版物ステータス出版済み - 12 1 2008

Fingerprint

confidence interval
sampling
hypothesis testing
forecast
climatology
experiment

All Science Journal Classification (ASJC) codes

  • Atmospheric Science

これを引用

Sampling uncertainty and confidence intervals for the brier score and brier skill score. / Bradley, A. Allen; Schwartz, Stuart S.; Hashino, Tempei.

:: Weather and Forecasting, 巻 23, 番号 5, 01.12.2008, p. 992-1006.

研究成果: ジャーナルへの寄稿記事

Bradley, A. Allen ; Schwartz, Stuart S. ; Hashino, Tempei. / Sampling uncertainty and confidence intervals for the brier score and brier skill score. :: Weather and Forecasting. 2008 ; 巻 23, 番号 5. pp. 992-1006.
@article{906a406f85e3464abb141d7310c96b43,
title = "Sampling uncertainty and confidence intervals for the brier score and brier skill score",
abstract = "For probability forecasts, the Brier score and Brier skill score are commonly used verification measures of forecast accuracy and skill. Using sampling theory, analytical expressions are derived to estimate their sampling uncertainties. The Brier score is an unbiased estimator of the accuracy, and an exact expression defines its sampling variance. The Brier skill score (with climatology as a reference forecast) is a biased estimator, and approximations are needed to estimate its bias and sampling variance. The uncertainty estimators depend only on the moments of the forecasts and observations, so it is easy to routinely compute them at the same time as the Brier score and skill score. The resulting uncertainty estimates can be used to construct error bars or confidence intervals for the verification measures, or perform hypothesis testing. Monte Carlo experiments using synthetic forecasting examples illustrate the performance of the expressions. In general, the estimates provide very reliable information on uncertainty. However, the quality of an estimate depends on both the sample size and the occurrence frequency of the forecast event. The examples also illustrate that with infrequently occurring events, verification sample sizes of a few hundred forecast-observation pairs are needed to establish that a forecast is skillful because of the large uncertainties that exist.",
author = "Bradley, {A. Allen} and Schwartz, {Stuart S.} and Tempei Hashino",
year = "2008",
month = "12",
day = "1",
doi = "10.1175/2007WAF2007049.1",
language = "English",
volume = "23",
pages = "992--1006",
journal = "Weather and Forecasting",
issn = "0882-8156",
publisher = "American Meteorological Society",
number = "5",

}

TY - JOUR

T1 - Sampling uncertainty and confidence intervals for the brier score and brier skill score

AU - Bradley, A. Allen

AU - Schwartz, Stuart S.

AU - Hashino, Tempei

PY - 2008/12/1

Y1 - 2008/12/1

N2 - For probability forecasts, the Brier score and Brier skill score are commonly used verification measures of forecast accuracy and skill. Using sampling theory, analytical expressions are derived to estimate their sampling uncertainties. The Brier score is an unbiased estimator of the accuracy, and an exact expression defines its sampling variance. The Brier skill score (with climatology as a reference forecast) is a biased estimator, and approximations are needed to estimate its bias and sampling variance. The uncertainty estimators depend only on the moments of the forecasts and observations, so it is easy to routinely compute them at the same time as the Brier score and skill score. The resulting uncertainty estimates can be used to construct error bars or confidence intervals for the verification measures, or perform hypothesis testing. Monte Carlo experiments using synthetic forecasting examples illustrate the performance of the expressions. In general, the estimates provide very reliable information on uncertainty. However, the quality of an estimate depends on both the sample size and the occurrence frequency of the forecast event. The examples also illustrate that with infrequently occurring events, verification sample sizes of a few hundred forecast-observation pairs are needed to establish that a forecast is skillful because of the large uncertainties that exist.

AB - For probability forecasts, the Brier score and Brier skill score are commonly used verification measures of forecast accuracy and skill. Using sampling theory, analytical expressions are derived to estimate their sampling uncertainties. The Brier score is an unbiased estimator of the accuracy, and an exact expression defines its sampling variance. The Brier skill score (with climatology as a reference forecast) is a biased estimator, and approximations are needed to estimate its bias and sampling variance. The uncertainty estimators depend only on the moments of the forecasts and observations, so it is easy to routinely compute them at the same time as the Brier score and skill score. The resulting uncertainty estimates can be used to construct error bars or confidence intervals for the verification measures, or perform hypothesis testing. Monte Carlo experiments using synthetic forecasting examples illustrate the performance of the expressions. In general, the estimates provide very reliable information on uncertainty. However, the quality of an estimate depends on both the sample size and the occurrence frequency of the forecast event. The examples also illustrate that with infrequently occurring events, verification sample sizes of a few hundred forecast-observation pairs are needed to establish that a forecast is skillful because of the large uncertainties that exist.

UR - http://www.scopus.com/inward/record.url?scp=65549136074&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=65549136074&partnerID=8YFLogxK

U2 - 10.1175/2007WAF2007049.1

DO - 10.1175/2007WAF2007049.1

M3 - Article

AN - SCOPUS:65549136074

VL - 23

SP - 992

EP - 1006

JO - Weather and Forecasting

JF - Weather and Forecasting

SN - 0882-8156

IS - 5

ER -