Distributions-oriented verification of probability forecasts for small data samples

A. Allen Bradley, Tempei Hashino, Stuart S. Schwartz

Research output: Contribution to journalArticle

25 Citations (Scopus)

Abstract

The distributions-oriented approach to forecast verification uses an estimate of the joint distribution of forecasts and observations to evaluate forecast quality. However, small verification data samples can produce unreliable estimates of forecast quality due to sampling variability and biases. In this paper, new techniques for verification of probability forecasts of dichotomous events are presented. For forecasts of this type, simplified expressions for forecast quality measures can be derived from the joint distribution. Although traditional approaches assume that forecasts are discrete variables, the simplified expressions apply to either discrete or continuous forecasts. With the derived expressions, most of the forecast quality measures can be estimated analytically using sample moments of forecasts and observations from the verification data sample. Other measures require a statistical modeling approach for estimation. Results from Monte Carlo experiments for two forecasting examples show that the statistical modeling approach can significantly improve estimates of these measures in many situations. The improvement is achieved mostly by reducing the bias of forecast quality estimates and, for very small sample sizes, by slightly reducing the sampling variability. The statistical modeling techniques are most useful when the verification data sample is small (a few hundred forecast-observation pairs or less), and for verification of rare events, where the sampling variability of forecast quality measures is inherently large.

Original languageEnglish
Pages (from-to)903-917
Number of pages15
JournalWeather and Forecasting
Volume18
Issue number5
DOIs
Publication statusPublished - Oct 1 2003

Fingerprint

distribution
forecast
sampling
modeling
experiment

All Science Journal Classification (ASJC) codes

  • Atmospheric Science

Cite this

Distributions-oriented verification of probability forecasts for small data samples. / Bradley, A. Allen; Hashino, Tempei; Schwartz, Stuart S.

In: Weather and Forecasting, Vol. 18, No. 5, 01.10.2003, p. 903-917.

Research output: Contribution to journalArticle

Bradley, A. Allen ; Hashino, Tempei ; Schwartz, Stuart S. / Distributions-oriented verification of probability forecasts for small data samples. In: Weather and Forecasting. 2003 ; Vol. 18, No. 5. pp. 903-917.
@article{5f88ad8ea1984668814a55c8ad9479b9,
title = "Distributions-oriented verification of probability forecasts for small data samples",
abstract = "The distributions-oriented approach to forecast verification uses an estimate of the joint distribution of forecasts and observations to evaluate forecast quality. However, small verification data samples can produce unreliable estimates of forecast quality due to sampling variability and biases. In this paper, new techniques for verification of probability forecasts of dichotomous events are presented. For forecasts of this type, simplified expressions for forecast quality measures can be derived from the joint distribution. Although traditional approaches assume that forecasts are discrete variables, the simplified expressions apply to either discrete or continuous forecasts. With the derived expressions, most of the forecast quality measures can be estimated analytically using sample moments of forecasts and observations from the verification data sample. Other measures require a statistical modeling approach for estimation. Results from Monte Carlo experiments for two forecasting examples show that the statistical modeling approach can significantly improve estimates of these measures in many situations. The improvement is achieved mostly by reducing the bias of forecast quality estimates and, for very small sample sizes, by slightly reducing the sampling variability. The statistical modeling techniques are most useful when the verification data sample is small (a few hundred forecast-observation pairs or less), and for verification of rare events, where the sampling variability of forecast quality measures is inherently large.",
author = "Bradley, {A. Allen} and Tempei Hashino and Schwartz, {Stuart S.}",
year = "2003",
month = "10",
day = "1",
doi = "10.1175/1520-0434(2003)018<0903:DVOPFF>2.0.CO;2",
language = "English",
volume = "18",
pages = "903--917",
journal = "Weather and Forecasting",
issn = "0882-8156",
publisher = "American Meteorological Society",
number = "5",

}

TY - JOUR

T1 - Distributions-oriented verification of probability forecasts for small data samples

AU - Bradley, A. Allen

AU - Hashino, Tempei

AU - Schwartz, Stuart S.

PY - 2003/10/1

Y1 - 2003/10/1

N2 - The distributions-oriented approach to forecast verification uses an estimate of the joint distribution of forecasts and observations to evaluate forecast quality. However, small verification data samples can produce unreliable estimates of forecast quality due to sampling variability and biases. In this paper, new techniques for verification of probability forecasts of dichotomous events are presented. For forecasts of this type, simplified expressions for forecast quality measures can be derived from the joint distribution. Although traditional approaches assume that forecasts are discrete variables, the simplified expressions apply to either discrete or continuous forecasts. With the derived expressions, most of the forecast quality measures can be estimated analytically using sample moments of forecasts and observations from the verification data sample. Other measures require a statistical modeling approach for estimation. Results from Monte Carlo experiments for two forecasting examples show that the statistical modeling approach can significantly improve estimates of these measures in many situations. The improvement is achieved mostly by reducing the bias of forecast quality estimates and, for very small sample sizes, by slightly reducing the sampling variability. The statistical modeling techniques are most useful when the verification data sample is small (a few hundred forecast-observation pairs or less), and for verification of rare events, where the sampling variability of forecast quality measures is inherently large.

AB - The distributions-oriented approach to forecast verification uses an estimate of the joint distribution of forecasts and observations to evaluate forecast quality. However, small verification data samples can produce unreliable estimates of forecast quality due to sampling variability and biases. In this paper, new techniques for verification of probability forecasts of dichotomous events are presented. For forecasts of this type, simplified expressions for forecast quality measures can be derived from the joint distribution. Although traditional approaches assume that forecasts are discrete variables, the simplified expressions apply to either discrete or continuous forecasts. With the derived expressions, most of the forecast quality measures can be estimated analytically using sample moments of forecasts and observations from the verification data sample. Other measures require a statistical modeling approach for estimation. Results from Monte Carlo experiments for two forecasting examples show that the statistical modeling approach can significantly improve estimates of these measures in many situations. The improvement is achieved mostly by reducing the bias of forecast quality estimates and, for very small sample sizes, by slightly reducing the sampling variability. The statistical modeling techniques are most useful when the verification data sample is small (a few hundred forecast-observation pairs or less), and for verification of rare events, where the sampling variability of forecast quality measures is inherently large.

UR - http://www.scopus.com/inward/record.url?scp=0344899039&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0344899039&partnerID=8YFLogxK

U2 - 10.1175/1520-0434(2003)018<0903:DVOPFF>2.0.CO;2

DO - 10.1175/1520-0434(2003)018<0903:DVOPFF>2.0.CO;2

M3 - Article

VL - 18

SP - 903

EP - 917

JO - Weather and Forecasting

JF - Weather and Forecasting

SN - 0882-8156

IS - 5

ER -