pytesmo.metrics.confidence_intervals module

pytesmo.metrics.confidence_intervals.has_analytical_ci(metric_func)[source]

Checks whether an analytical CI implementation is available.

Parameters:

metric_func (callable) – Function that calculates metric value. Must be from pytesmo.metrics.

Returns:

has_ci – True if there is a function with name metric_func.__name__ + "_ci" in pytesmo.metric_cis.

Return type:

bool

pytesmo.metrics.confidence_intervals.tcol_metrics_with_bootstrapped_ci(x, y, z, ref_ind=0, alpha=0.05, method='percentile', nsamples=1000, minimum_data_length=100)[source]

Evaluates triple collocation metrics and bootstraps confidence interval.

This calculates the SNR, error standard deviation, and scaling parameter value using Triple Collocation Analysis and uses bootstrapping to find confidence intervals.

Parameters:

  • x (np.ndarray) – Data to be compared.

  • y (np.ndarray) – Data to be compared.

  • z (np.ndarray) – Data to be compared.

  • ref_ind (int) – Index of reference data set for estimating scaling coefficients. Default: 0 (x)

  • alpha (float, optional) – Confidence level, default is 0.05.

  • method (str, optional) –

    The method to use to calculate confidence intervals. Available methods are:

    • ”percentile” (default): Uses the percentiles of the bootstrapped metric distribution.

    • ”basic”: Uses the percentiles of the differences to the original metric value.

    • ”BCa”: Bias-corrected and accelerated bootstrap.

    For more info and a comparison of the methods, see [Gilleland2010], especially Table 6 on page 38.

  • nsamples (int, optional) – Number of bootstrap samples, default is 1000.

  • minimum_data_length (int, optional) – Minimum amount of data required to do bootstrapping. Default is 100.

Returns:

  • snr_result (tuple) – (snr, lower, upper), where each entry is an array of length 3.

  • err_std_result (tuple) – (err_std, lower, upper), where each entry is an array of length 3.

  • beta_result (tuple) – (beta, lower, upper), where each entry is an array of length 3. Note that beta is always 1 for the reference dataset (see ref_ind), therefore the lower and upper values are set to 1 too.

References

[Gilleland2010]

Gilleland, E. (2010). Confidence Intervals for Forecast Verification (No. NCAR/TN-479+STR). University Corporation for Atmospheric Research. doi:10.5065/D6WD3XJM

pytesmo.metrics.confidence_intervals.with_analytical_ci(metric_func, x, y, alpha=0.05)[source]

Evaluates metric and analytical confidence interval.

This calculates the metric value and an analytical confidence interval. For this to work, the metric function must be from pytesmo.pairwise_metric_cis Use pytesmo.metrics.has_analytical_ci() to check whether it’s available.

Parameters:

  • metric_func (callable or str) – Function that calculates metric value. Must be from pytesmo.metrics, and have an analytical CI function implemented in pytesmo.metric_cis. The metric function must have the following signature: (x : np.ndarray, y : np.ndarray) -> float Alternatively can be the name of the function in pytesmo.metrics.

  • x (np.ndarray) – Data to be compared.

  • y (np.ndarray) – Data to be compared.

  • alpha (float, optional) – Confidence level, default is 0.05.

Returns:

  • m (float) – Metric value

  • lower (float) – Lower bound of confidence interval

  • upper (float) – Upper bound of confidence interval

Raises:

ValueError : – If no analytical CI function is available.

pytesmo.metrics.confidence_intervals.with_bootstrapped_ci(metric_func, x, y, alpha=0.05, method='percentile', nsamples=1000, minimum_data_length=100)[source]

Evaluates metric and bootstraps confidence interval.

This calculates the metric value and uses bootstrapping to find a confidence interval for the metric. This works only for pairwise metrics, use pytesmo.metrics.tcol_metrics_with_bootstrap_ci() for TCA metrics.

Parameters:

  • metric_func (callable) – Function that calculates metric value. The metric function must have the following signature: (x : np.ndarray, y : np.ndarray) -> float

  • x (np.ndarray) – Data to be compared.

  • y (np.ndarray) – Data to be compared.

  • alpha (float, optional) – Confidence level, default is 0.05.

  • method (str, optional) –

    The method to use to calculate confidence intervals. Available methods are:

    • ”percentile” (default): Uses the percentiles of the bootstrapped metric distribution.

    • ”basic”: Uses the percentiles of the differences to the original metric value.

    • ”BCa”: Bias-corrected and accelerated bootstrap.

    For more info and a comparison of the methods, see [Gilleland2010], especially Table 6 on page 38.

  • nsamples (int, optional) – Number of bootstrap samples, default is 1000.

  • minimum_data_length (int, optional) – Minimum amount of data required to do bootstrapping. Default is 100.

Returns:

  • m (float) – Metric value for pairwise metrics

  • lower (float or array of floats) – Lower bound of confidence interval

  • upper (float or array of floats) – Upper bound of confidence interval

References

[Gilleland2010]

Gilleland, E. (2010). Confidence Intervals for Forecast Verification (No. NCAR/TN-479+STR). University Corporation for Atmospheric Research. doi:10.5065/D6WD3XJM