Empirical Bayes Calibration Benchmark

This page details the mathematical basis, methodology, and empirical benchmark results for the calibrate=True flag in random_forest_error(), as adapted from Efron (2014) [1].

Mathematical Basis

The calibration procedure implemented in forestci/calibration.py is based on the g-modeling strategy for Empirical Bayes estimation.

In Efron’s terminology, empirical Bayes can be approached via:

• f-modeling: Parametrically modeling the marginal distribution of the observations in the data space.

• g-modeling: Parametrically modeling the unknown prior distribution \(g(\theta)\) in the parameter space.

forestci implements the g-modeling approach. We assume the true variances \(\theta\) are drawn from an unknown discrete prior \(g(\theta)\) over a grid of possible variances, and we observe noisy estimates \(X \sim \mathcal{N}(\theta, \sigma^2)\).

Following Efron’s exponential family formulation (Eq. 5.1), the prior is modeled as:

\[g(\alpha) \propto \exp(Q \alpha)\]

Implementation specifics vs Efron (2014):

• Basis Matrix (Q): Efron suggests using a natural spline basis. The forestci implementation diverges by using a polynomial basis evaluated on the discrete grid (by default, \(p=2\)), effectively parameterizing the prior as an exponential family with polynomial sufficient statistics.

• Support Constraints: While the raw infinitesimal jackknife (IJ) variance observations used during EB fitting can be negative, the forestci algorithm enforces a strict non-negative mask (\(x \ge 0\)) when evaluating the prior density. This ensures that the latent variance support is strictly non-negative. Calibration does not “remove” the negative observations; rather, it maps them through this non-negative parametric prior to output an expected posterior variance that is inherently non-negative.

• Marginal Likelihood: The marginal density \(f(x)\) is computed via Fast Fourier Transform convolution (scipy.signal.fftconvolve) of the discrete prior and the Gaussian noise kernel, which functions as a discrete approximation of Efron’s continuous formulation (Eq. 5.21).

• Optimization: While Efron uses Fisher scoring, forestci utilizes the BFGS algorithm (via scipy.optimize.minimize) to find the Maximum Likelihood Estimate for \(\alpha\), with a fallback to the Nelder-Mead algorithm to handle numerical precision loss.

• Posterior Calculation: The expected posterior value \(\mathbb{E}[\theta | X]\) calculated in gbayes is derived directly from Efron’s discrete Bayes rule (Eq. 2.12).

Methodology

To evaluate the effectiveness of Empirical Bayes calibration, we compared calibrated and uncalibrated variance estimates across several datasets and low tree counts against a stabilized reference forest.

Reference Stability and Empirical Reference: We use a Random Forest with n_estimators=2000 as an empirical reference, not absolute ground truth. The build does not perform a separate convergence study; finite-forest noise may remain in this reference.

Definitions in the Benchmark:

• Raw IJ Variance: The direct output of the infinitesimal jackknife algorithm, which frequently produces negative estimates due to Monte Carlo noise.

• Clipped Uncalibrated Variance: Raw IJ variance artificially clipped to exactly 0 whenever it drops below 0. This clipping defines the uncalibrated baseline for the variance RMSE comparison and is not equivalent to EB calibration.

• Calibrated EB Variance: The posterior variance estimate generated by the EB algorithm, which is inherently constrained to be non-negative due to the non-negative prior support.

Benchmark Setup:

• Test Scenarios: Independent models with n_estimators set to 50, 100, and 200 trees. Random seed was fixed at 42 across all dataset splits and forest initializations. The test size was set to 0.2.

• Treatments: Variance estimates were computed via random_forest_error() with calibrate=False and calibrate=True.

• Metrics:

  1. Neg Rate: Percentage of raw IJ variance estimates that fell below zero.

  2. Var RMSE: Root Mean Squared Error of the variance estimates compared to the 2000-tree empirical reference variance.

  3. Relative Improvement: Computed as (Var RMSE (Uncal) - Var RMSE (Cal)) / Var RMSE (Uncal) * 100. Positive values mean calibration improves agreement with the 2000-tree empirical reference, while negative values mean calibration worsens it.

Datasets: The benchmark covers both regression and classification across seven datasets, including the repository’s auto_mpg.csv example dataset. 1. auto_mpg.csv: Regression, 392 samples, 7 features. 2. California Housing: Regression, 20,640 samples, 8 features. 3. Diabetes: Regression, 442 samples, 10 features. 4. Breast Cancer: Classification, 569 samples, 30 features. 5. Synthetic Hard: Classification, 2000 samples, 20 features, 10 informative features. 6. Synthetic Reg: Regression, 1000 samples, 10 features, noise=0.1. 7. Wine: Multiclass classification, 178 samples, 13 features, three classes.

Classification rows estimate the variance of predict_proba using an explicit class_index: column 1 for binary tasks and each of the three columns separately for Wine. Each class has its own calibration and reference variance; classes are not pooled or treated as independent. This is variance calibration, not calibration of the predicted probabilities. See the multiclass example.

Benchmark Results

The table is regenerated from examples/generate_calibration_benchmark.py on every documentation build. NumPy calibration sampling is also seeded at 42. For 2,000-estimator confidence widths versus observed prediction errors, see Confidence intervals versus observed prediction error.

Recomputed calibration benchmark

Dataset	Trees	Neg Rate (Uncal)	Var RMSE (Uncal)	Var RMSE (Cal)	Relative Improvement
Auto MPG	50	31.6%	9.998	9.586	4.1%
Auto MPG	100	22.8%	3.790	3.703	2.3%
Auto MPG	200	13.9%	1.132	1.567	-38.4%
California	50	3.3%	3.328	2.335	29.8%
California	100	17.2%	1.046	0.649	38.0%
California	200	31.3%	0.371	0.182	51.1%
Diabetes	50	37.1%	1307.6	322.2	75.4%
Diabetes	100	32.6%	627.3	177.4	71.7%
Diabetes	200	23.6%	401.5	266.9	33.5%
Breast Cancer class 1	50	15.8%	0.056	0.037	33.2%
Breast Cancer class 1	100	17.5%	0.024	0.019	20.2%
Breast Cancer class 1	200	27.2%	0.011	0.007	37.8%
Synth Hard class 1	50	26.0%	0.223	0.091	59.4%
Synth Hard class 1	100	40.0%	0.082	0.023	72.2%
Synth Hard class 1	200	43.5%	0.033	0.007	78.0%
Synthetic Reg	50	26.0%	4291.5	1504.5	64.9%
Synthetic Reg	100	35.5%	1925.7	484.0	74.9%
Synthetic Reg	200	35.0%	766.2	312.6	59.2%
Wine class 0	50	22.2%	0.027	0.016	42.0%
Wine class 1	50	47.2%	0.024	0.005	78.3%
Wine class 2	50	36.1%	0.006	0.005	21.4%
Wine class 0	100	33.3%	0.008	0.005	28.4%
Wine class 1	100	44.4%	0.006	0.004	40.5%
Wine class 2	100	47.2%	0.003	0.002	26.6%
Wine class 0	200	44.4%	0.004	0.003	21.3%
Wine class 1	200	47.2%	0.005	0.004	25.6%
Wine class 2	200	36.1%	0.006	0.003	46.2%

Interpretation and Limitations

Empirical Bayes calibration is a parametric approximation. While it typically improves agreement with the empirical reference, it does not universally improve accuracy in every hypothetical scenario.

Where calibration improves the estimate: Because the EB approach restricts the discrete prior to a non-negative grid, it strictly constrains output estimates to the non-negative support. In many cases—such as the Diabetes dataset or Synthetic Reg—this parametric approximation significantly improves the Var RMSE compared to uncalibrated observations.

Where calibration may have diminishing returns or worsen the estimate: * Bias Introduction: In cases where the raw jackknife variance begins to naturally stabilize or the noise distribution is irregular, the parametric prior can introduce an upward bias or shrinkage effect. Negative relative improvements in the generated table identify cases where calibration worsens agreement with the empirical reference.

Practical Guidance

• Calibration may be useful when computationally constrained to a low number of trees and robust, strictly non-negative variance estimates are required.

• Empirical Bayes calibration is not universally recommended. For large forests (e.g., 1000+ trees), Monte Carlo noise in the infinitesimal jackknife naturally decays. In these environments, calibrate=False minimizes the risk of introducing parametric bias from the Empirical Bayes prior.

References

[1]

2. Efron. “Two Modeling Strategies for Empirical Bayes Estimation”, Statistical Science vol. 29, pp. 285-301, 2014. Available at https://arxiv.org/abs/1409.2677. See also the forest-confidence-interval documentation.