Empirical Bayes Calibration Benchmark

This page details the mathematical basis, methodology, and empirical benchmark results for the calibrate=True flag in random_forest_error(), as adapted from Efron (2014) [1].

Mathematical Basis

The calibration procedure implemented in forestci/calibration.py is based on the g-modeling strategy for Empirical Bayes estimation.

In Efron’s terminology, empirical Bayes can be approached via:

• f-modeling: Parametrically modeling the marginal distribution of the observations in the data space.

• g-modeling: Parametrically modeling the unknown prior distribution g(\theta) in the parameter space.

forestci implements the g-modeling approach. We assume the true variances \theta are drawn from an unknown discrete prior g(\theta) over a grid of possible variances, and we observe noisy estimates X \sim \mathcal{N}(\theta, \sigma^2).

Following Efron’s exponential family formulation (Eq. 5.1), the prior is modeled as:

g(\alpha) \propto \exp(Q \alpha)

Implementation specifics vs Efron (2014):

• Basis Matrix (Q): Efron suggests using a natural spline basis. The forestci implementation diverges by using a polynomial basis evaluated on the discrete grid (by default, p=2), effectively parameterizing the prior as an exponential family with polynomial sufficient statistics.

• Support Constraints: While the raw infinitesimal jackknife (IJ) variance observations used during EB fitting can be negative, the forestci algorithm enforces a strict non-negative mask (x \ge 0) when evaluating the prior density. This ensures that the latent variance support is strictly non-negative. Calibration does not “remove” the negative observations; rather, it maps them through this non-negative parametric prior to output an expected posterior variance that is inherently non-negative.

• Marginal Likelihood: The marginal density f(x) is computed via Fast Fourier Transform convolution (scipy.signal.fftconvolve) of the discrete prior and the Gaussian noise kernel, which functions as a discrete approximation of Efron’s continuous formulation (Eq. 5.21).

• Optimization: While Efron uses Fisher scoring, forestci utilizes the BFGS algorithm (via scipy.optimize.minimize) to find the Maximum Likelihood Estimate for \alpha, with a fallback to the Nelder-Mead algorithm to handle numerical precision loss.

• Posterior Calculation: The expected posterior value \mathbb{E}[\theta | X] calculated in gbayes is derived directly from Efron’s discrete Bayes rule (Eq. 2.12).

Methodology

To evaluate the effectiveness of Empirical Bayes calibration, we compared calibrated and uncalibrated variance and Confidence Interval (CI) estimates across several datasets and low tree counts against a stabilized reference forest.

Reference Stability and Empirical Reference: We established an empirical reference (NOT absolute ground truth) using a Random Forest with n_estimators=2000. We checked the stability of this reference using a 1000-vs-2000 tree comparison. The discrepancy between 1000 and 2000 trees was orders of magnitude smaller than the variance noise observed at 50 trees. This confirms the 2000-tree outputs as a sufficiently stable empirical reference for benchmarking rather than a perfect theoretical ground truth.

Definitions in the Benchmark:

• Raw IJ Variance: The direct output of the infinitesimal jackknife algorithm, which frequently produces negative estimates due to Monte Carlo noise.

• Clipped Uncalibrated Variance: Raw IJ variance artificially clipped to exactly 0 whenever it drops below 0. This clipping is used only to permit a functional CI width comparison for the benchmark and is not equivalent to EB calibration.

• Calibrated EB Variance: The posterior variance estimate generated by the EB algorithm, which is inherently constrained to be non-negative due to the non-negative prior support.

Benchmark Setup:

• Test Scenarios: Independent models with n_estimators set to 50, 100, and 200 trees. Random seed was fixed at 42 across all dataset splits and forest initializations. The test size was set to 0.2.

• Treatments: Variance estimates were computed via random_forest_error() with calibrate=False and calibrate=True.

• Metrics:

  1. Neg Rate: Percentage of raw IJ variance estimates that fell below zero.

  2. Var RMSE: Root Mean Squared Error of the variance estimates compared to the 2000-tree empirical reference variance.

  3. Relative Improvement: Computed as (Var RMSE (Uncal) - Var RMSE (Cal)) / Var RMSE (Uncal) * 100. Positive values mean calibration improves agreement with the 2000-tree empirical reference, while negative values mean calibration worsens it.

Datasets: The benchmark covers both regression and classification across six datasets, including the repository’s auto_mpg.csv example dataset. 1. auto_mpg.csv: Regression, 392 samples, 7 features. 2. California Housing: Regression, 2000 samples, 8 features. 3. Diabetes: Regression, 442 samples, 10 features. 4. Breast Cancer: Classification, 569 samples, 30 features. 5. Synthetic Hard: Classification, 2000 samples, 20 features, 10 informative features. 6. Synthetic Reg: Regression, 1000 samples, 10 features, noise=0.1.

Benchmark Results

Note: In all scenarios below, the calibrated variance estimates strictly eliminated negative values, resulting in a calibrated negative rate of exactly 0.0%. The benchmark methodology is reproduced dynamically via the repository script.

Dataset	Trees	Neg Rate (Uncal)	Var RMSE (Uncal)	Var RMSE (Cal)	Relative Improvement
Auto MPG	50	20.3%	10.3	9.442	8.2%
	100	21.5%	3.756	3.635	3.2%
	200	12.7%	1.077	1.047	2.7%
California	50	35.6%	1.510	1.279	15.3%
	100	11.5%	1.208	1.128	6.6%
	200	61.9%	0.201	0.114	43.3%
Diabetes	50	19.1%	1733.4	939.2	45.8%
	100	41.6%	565.2	230.1	59.3%
	200	23.6%	404.0	267.7	33.7%
Breast Cancer	50	5.3%	0.073	0.033	54.3%
	100	21.9%	0.022	0.013	41.0%
	200	26.3%	0.011	0.007	39.8%
Synth Hard	50	87.5%	0.052	0.032	39.2%
	100	32.8%	0.094	0.047	49.6%
	200	40.0%	0.036	0.016	54.1%
Synthetic Reg	50	18.0%	5180.1	3238.6	37.5%
	100	27.5%	2379.6	1214.4	49.0%
	200	16.0%	1057.9	479.1	54.7%

Interpretation and Limitations

Empirical Bayes calibration is a parametric approximation. While it typically improves agreement with the empirical reference, it does not universally improve accuracy in every hypothetical scenario.

Where calibration improves the estimate: Because the EB approach restricts the discrete prior to a non-negative grid, it strictly constrains output estimates to the non-negative support. In many cases—such as the Diabetes dataset or Synthetic Reg—this parametric approximation significantly improves the Var RMSE compared to uncalibrated observations.

Where calibration may have diminishing returns or worsen the estimate: * Bias Introduction: In cases where the raw jackknife variance begins to naturally stabilize or the noise distribution is irregular, the parametric prior can introduce an upward bias or shrinkage effect. Though our reproduction script showed improvements across all 6 sampled datasets here, extreme noise scenarios with a highly skewed Monte Carlo distribution may experience increased RMSE.

Practical Guidance

• Calibration may be useful when computationally constrained to a low number of trees and robust, strictly non-negative variance estimates are required.

• Empirical Bayes calibration is not universally recommended. For large forests (e.g., 1000+ trees), Monte Carlo noise in the infinitesimal jackknife naturally decays. In these environments, calibrate=False minimizes the risk of introducing parametric bias from the Empirical Bayes prior.

References

[1]

2. Efron. “Two Modeling Strategies for Empirical Bayes Estimation”, Statistical Science vol. 29, pp. 285-301, 2014. Available at https://arxiv.org/abs/1409.2677. See also the forest-confidence-interval documentation.