Dynamic Ensembles

Motivated by the No Free Lunch theorem, which states that no single algorithm is optimal for all problems, ensemble methods combine multiple models to achieve better and more robust predictions than any individual model.

Time series forecasting particularly benefits from ensembles due to:

Varying model performance across different periods
Non-stationary patterns and regime changes
Risk reduction from model selection
State-of-the-art performance in empirical studies

This module implements dynamic ensemble strategies that adapt weights over time to changing patterns or across different series:

Meta-learning: Using a meta-model to predict individual model errors
Regret minimization: Exponentially or polynomially weighted averaging
Windowing: Recent performance-based weighting

class ADE(freq: str, meta_lags: List[int] | None = None, trim_ratio: float = 1, trim_by_uid: bool = True, meta_model=sklearn.multioutput.MultiOutputRegressor)[source]

Bases: BaseADE

Arbitrated Dynamic Ensemble

Dynamic ensemble approach where ensemble members are weighted based on a meta-model that forecasts their error. The ADE approach dynamically combines forecasts from multiple models by learning their error patterns. It’s particularly effective when dealing with heterogeneous time series and when the relative performance of individual models varies over time.

Parameters:

freq (str) – String denoting the sampling frequency of the time series (e.g. “MS”).
trim_ratio (float, optional) – Ratio (0-1) of ensemble members to keep in the ensemble. (1-trim_ratio) of models will not be used during inference based on validation accuracy. Defaults to 1, which means all ensemble members are used.
meta_lags (list of int) – List of lags to be used in the training of the meta-model. Follows the structure of mlforecast. Example: [1,2,3,8].
trim_by_uid (bool, optional) – Whether to trim the ensemble by unique_id (True) or dataset (False). Defaults to True, but this can become computationally demanding for datasets with a large number of time series.
meta_model (object, optional) – Learning algorithm to use in the meta-level to forecast the error of ensemble members. Defaults to a linear LGBM with a default configuration.

References

Cerqueira, V., Torgo, L., Pinto, F., & Soares, C. (2019). Arbitrage of forecasting experts. Machine Learning, 108, 913-944.

Examples

>>> from datasetsforecast.m3 import M3
>>> from neuralforecast import NeuralForecast
>>> from neuralforecast.models import NHITS, NBEATS, MLP
>>>
>>> from metaforecast.ensembles import ADE
>>>
>>> df, *_ = M3.load('.', group='Monthly')
>>>
>>> # ensemble members setup
>>> CONFIG = {'input_size': 12,
>>>           'h': 12,
>>>           'accelerator': 'cpu',
>>>           'max_steps': 10, }
>>>
>>> models = [
>>>     NBEATS(**CONFIG, stack_types=3 * ["identity"]),
>>>     NHITS(**CONFIG),
>>>     MLP(**CONFIG),
>>>     MLP(num_layers=3, **CONFIG),
>>> ]
>>>
>>> nf = NeuralForecast(models=models, freq='M')
>>>
>>> # cv to build meta-data
>>> n_windows = df['unique_id'].value_counts().min()
>>> n_windows = int(n_windows // 2)
>>> fcst_cv = nf.cross_validation(df=df, n_windows=n_windows, step_size=1)
>>> fcst_cv = fcst_cv.reset_index()
>>> fcst_cv = fcst_cv.groupby(['unique_id', 'cutoff']).head(1).drop(columns='cutoff')
>>>
>>> # fitting combination rule
>>> ensemble = ADE(freq='ME', meta_lags=list(range(1,7)), trim_ratio=0.6)
>>> ensemble.fit(fcst_cv)
>>>
>>> # re-fitting models
>>> nf.fit(df=df)
>>>
>>> # forecasting and combining
>>> fcst = nf.predict()
>>> fcst_ensemble = ensemble.predict(fcst.reset_index(), train=df, h=12)

__init__(freq: str, meta_lags: List[int] | None = None, trim_ratio: float = 1, trim_by_uid: bool = True, meta_model=sklearn.multioutput.MultiOutputRegressor)[source]

fit(insample_fcst: pandas.DataFrame, **kwargs)[source]

Parameters:: insample_fcst (pd.DataFrame) – Forecast and actual values dataset formatted like mlforecast cross-validation output. Contains either: - In-sample forecasts (predictions on training data) - Cross-validation results (out-of-sample predictions) Expected columns: - unique_id (or other id_col): Identifier for each series - ds (or other time_col): Timestamp - y (or other target_col): Actual values - model_name: Predictions by a model with name model_name
Returns:: self, with a fitted self.meta_model
Return type:: self

predict(fcst: pandas.DataFrame, train: pandas.DataFrame, h: int, **kwargs)[source]

Combine ensemble member forecasts using the meta-model.

Parameters:

fcst (pd.DataFrame) – Forecasts from individual ensemble members. Expected columns: [‘unique_id’, ‘ds’, ‘model_name1’, ‘model_name2’, …]
train (pd.DataFrame) – Training dataset used to compute recent lags for meta-model input. Expected columns: [‘unique_id’, ‘ds’, ‘y’]
h (int) – Forecast horizon (number of future periods to predict)

Returns:

Combined ensemble forecasts for h periods ahead.

Return type:

pd.Series

update_weights(fcst: pandas.DataFrame, **kwargs)[source]

Update performance statistics of ensemble members based on recent forecasts.

Used to identify and retain top-performing models for ensemble trimming. Updates internal statistics tracking each model’s forecast accuracy.

Parameters:: fcst (pd.DataFrame) – Recent forecasts and actual values for ensemble members. Expected columns: - unique_id: Series identifier - ds: Timestamp - model_name: Predictions with model with name “model_name” - y: Actual values

Notes

Not implemented yet

class MLForecastADE(mlf: mlforecast.MLForecast, sf: statsforecast.StatsForecast | None = None, trim_ratio: float = 1, meta_model=sklearn.multioutput.MultiOutputRegressor)[source]

Bases: ADE

Arbitrated Dynamic Ensemble (ADE) implemented for MLForecast.

Creates a dynamic ensemble where member weights are determined by a meta-model that predicts each model’s forecast error. The meta-model adapts weights based on recent performance and input patterns.

Implementation follows the MLForecast conventions and models. To use ADE with statsforecast or neuralforecast, see ADE.

References

Cerqueira, V., Torgo, L., Pinto, F., & Soares, C. (2019). “Arbitrage of forecasting experts.” Machine Learning, 108, 913-944.

Examples

>>> from datasetsforecast.m3 import M3
>>> from mlforecast import MLForecast
>>> from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV
>>> from sklearn.tree import DecisionTreeRegressor
>>> from sklearn.neighbors import KNeighborsRegressor
>>>
>>> from metaforecast.ensembles import MLForecastADE
>>>
>>> df, *_ = M3.load('.', group='Monthly')
>>>
>>> # ensemble members setup
>>> models_ml = {
>>>     'Ridge': RidgeCV(),
>>>     'Lasso': LassoCV(),
>>>     'Elastic-net': ElasticNetCV(),
>>>     'DT': DecisionTreeRegressor(max_depth=5),
>>>     'DStump': DecisionTreeRegressor(max_depth=1),
>>>     'KNN': KNeighborsRegressor(n_neighbors=30),
>>> }
>>>
>>> mlf = MLForecast(models=models_ml, freq='ME', lags=range(1, 7))
>>>
>>> # make sure fitted=True so we have samples for meta-learning
>>> mlf.fit(df=df, fitted=True)
>>>
>>> # fitting combination rule
>>> ensemble = MLForecastADE(mlf=mlf, trim_ratio=0.5)
>>> ensemble.fit()
>>>
>>> fcst = ensemble.predict(train=df, h=12)

__init__(mlf: mlforecast.MLForecast, sf: statsforecast.StatsForecast | None = None, trim_ratio: float = 1, meta_model=sklearn.multioutput.MultiOutputRegressor)[source]

Initialize the Arbitrated Dynamic Ensemble with MLForecast models.

Parameters:

mlf (MLForecast) – Fitted MLForecast object containing ensemble members. Must be initialized with fitted=True to generate the meta-dataset for error prediction.
sf (StatsForecast, optional) – StatsForecast object containing classical forecasting models to be included in the ensemble alongside MLForecast models.
trim_ratio (float, default=1.0) – Proportion of ensemble members to retain, between 0 and 1. Models are selected based on validation accuracy: - 1.0 keeps all models - 0.5 keeps top 50% of models - Lower values create a more selective ensemble
meta_model (object, optional) – Model used to predict ensemble members’ errors. If None, defaults to LightGBM with linear trees and default parameters.

Notes

The meta_model should support scikit-learn’s fit/predict API.

fit(**kwargs)[source]

Train the meta-model using ensemble members’ in-sample predictions.

Uses cross-validation predictions from the MLForecast object to: 1. Compute forecast errors for each ensemble member 2. Train the meta-model to predict these errors 3. Update model weights based on validation performance 4. If trim_ratio < 1, selects top performing models

predict(train: pandas.DataFrame, h: int, **kwargs)[source]

Generate ensemble forecasts using weighted model combinations.

Parameters:

train (pd.DataFrame) – Training dataset used to compute recent lags for meta-model input. Expected columns: - unique_id: Series identifier - ds: Timestamp - y: Target variable
h (int) – Forecast horizon (number of periods to predict ahead)

Returns:

Combined ensemble predictions.

Return type:

pd.Series

update_weights(fcst: pandas.DataFrame, **kwargs)[source]

Update performance statistics of ensemble members.

See ADE.update_weights() for full documentation.

class MLewa(loss_type: str, gradient: bool, weight_by_uid: bool = False, trim_ratio: float = 1)[source]

Bases: Mixture

Dynamic ensemble using exponentially weighted averaging (EWA).

Implementation inspired by R’s opera package, this class combines forecasts using online learning with exponential weights. Weights are updated based on recent performance to adapt to changing patterns.

See also

MLewa: Variant using exponential weighting
Mixture: Parent class with core functionality

References

Cesa-Bianchi, N., & Lugosi, G. (2006). “Prediction, learning, and games.”

update_weights(fcst: pandas.DataFrame)[source]

Update model performance statistics for dynamic ensemble selection.

Parameters:: fcst (pd.DataFrame) – Dataset containing actual values and model predictions. Expected columns: - unique_id: Series identifier - ds: Timestamp - model_name: Predictions of model with name “model_name” - y: Actual values

class LossOnTrain(trim_ratio: float, weight_by_uid: bool = True)[source]

Bases: Windowing

Weight ensemble members based on training set performance.

An ensemble method that assigns static weights to models based on their training error. Unlike dynamic ensembles, weights are fixed after training and don’t adapt to changing patterns.

Notes

Weights are computed as inverse of training error, giving higher weights to more accurate models. This static weighting assumes relative model performance remains stable over time.

Examples

>>> from datasetsforecast.m3 import M3
>>> from neuralforecast import NeuralForecast
>>> from neuralforecast.models import NHITS, NBEATS, MLP
>>> from metaforecast.ensembles import LossOnTrain
>>>
>>> df, *_ = M3.load('.', group='Monthly')
>>>
>>> # ensemble members setup
>>> CONFIG = {'input_size': 12,
>>>           'h': 12,
>>>           'accelerator': 'cpu',
>>>           'max_steps': 10, }
>>>
>>> models = [
>>>     NBEATS(**CONFIG, stack_types=3 * ["identity"]),
>>>     NHITS(**CONFIG),
>>>     MLP(**CONFIG),
>>>     MLP(num_layers=3, **CONFIG),
>>> ]
>>>
>>> nf = NeuralForecast(models=models, freq='M')
>>>
>>> # cv to build meta-data
>>> n_windows = df['unique_id'].value_counts().min()
>>> n_windows = int(n_windows // 2)
>>> fcst_cv = nf.cross_validation(df=df, n_windows=n_windows, step_size=1)
>>> fcst_cv = fcst_cv.reset_index()
>>> fcst_cv = fcst_cv.groupby(['unique_id', 'cutoff']).head(1).drop(columns='cutoff')
>>>
>>> # fitting combination rule
>>> ensemble = LossOnTrain(trim_ratio=0.8)
>>> ensemble.fit(fcst_cv)
>>>
>>> # re-fitting models
>>> nf.fit(df=df)
>>>
>>> # forecasting and combining
>>> fcst = nf.predict()
>>> fcst_ensemble = ensemble.predict(fcst.reset_index())

__init__(trim_ratio: float, weight_by_uid: bool = True)[source]

Initialize static ensemble with training-based weights.

Parameters:

weight_by_uid (bool, default=True) – Strategy for computing model weights: - True: Separate weights per series - False: Global weights across all series
trim_ratio (float, default=1.0) – Proportion of models to retain in ensemble, between 0 and 1: - 1.0: Keep all models - 0.5: Keep top 50% of models Models are selected based on training performance

Notes

Weight computation involves: 1. Calculate training error for each model 2. Convert errors to weights (inverse relationship) 3. If trim_ratio < 1, select top performing models 4. Normalize weights to sum to 1

update_weights(**kwargs)[source]

Updating the combination weights

Not implemented yet

class BestOnTrain(select_by_uid: bool = True)[source]

Bases: Windowing

Select single best model based on training performance.

A simple baseline ensemble that selects the single best-performing model based on training data accuracy. While not technically an ensemble since it uses only one model, it serves as an important baseline.

Examples

>>> from datasetsforecast.m3 import M3
>>> from neuralforecast import NeuralForecast
>>> from neuralforecast.models import NHITS, NBEATS, MLP
>>> from metaforecast.ensembles import BestOnTrain
>>>
>>> df, *_ = M3.load('.', group='Monthly')
>>>
>>> # ensemble members setup
>>> CONFIG = {'input_size': 12,
>>>           'h': 12,
>>>           'accelerator': 'cpu',
>>>           'max_steps': 10, }
>>>
>>> models = [
>>>     NBEATS(**CONFIG, stack_types=3 * ["identity"]),
>>>     NHITS(**CONFIG),
>>>     MLP(**CONFIG),
>>>     MLP(num_layers=3, **CONFIG),
>>> ]
>>>
>>> nf = NeuralForecast(models=models, freq='M')
>>>
>>> # cv to build meta-data
>>> n_windows = df['unique_id'].value_counts().min()
>>> n_windows = int(n_windows // 2)
>>> fcst_cv = nf.cross_validation(df=df, n_windows=n_windows, step_size=1)
>>> fcst_cv = fcst_cv.reset_index()
>>> fcst_cv = fcst_cv.groupby(['unique_id', 'cutoff']).head(1).drop(columns='cutoff')
>>>
>>> # fitting combination rule
>>> ensemble = BestOnTrain()
>>> ensemble.fit(fcst_cv)
>>>
>>> # re-fitting models
>>> nf.fit(df=df)
>>>
>>> # forecasting and combining
>>> fcst = nf.predict()
>>> fcst_ensemble = ensemble.predict(fcst.reset_index())

__init__(select_by_uid: bool = True)[source]

Initialize best model selector.

Parameters:: select_by_uid (bool, default=True) – Strategy for selecting best performing model: - True: Select best model separately for each series - False: Select single best model across all series

Notes

Per-series selection (select_by_uid=True) allows for more granular model choice but requires sufficient data per series for reliable selection. Global selection may be more robust when individual series are short.

update_weights(**kwargs)[source]

Updating the combination weights

Not implemented yet

class EqAverage(trim_ratio: float = 1, select_by_uid: bool = True)[source]

Bases: Windowing

Combine forecasts using simple average with optional trimming.

A robust ensemble method that equally weights retained models after removing poor performers. Research shows this simple approach often performs competitively with more complex weighting schemes.

References

Jose, V. R. R., & Winkler, R. L. (2008). “Simple robust averages of forecasts: Some empirical results.” International Journal of Forecasting, 24(1), 163-169.

Examples

>>> from datasetsforecast.m3 import M3
>>> from neuralforecast import NeuralForecast
>>> from neuralforecast.models import NHITS, NBEATS, MLP
>>> from metaforecast.ensembles import EqAverage
>>>
>>> df, *_ = M3.load('.', group='Monthly')
>>>
>>> # ensemble members setup
>>> CONFIG = {'input_size': 12,
>>>           'h': 12,
>>>           'accelerator': 'cpu',
>>>           'max_steps': 10, }
>>>
>>> models = [
>>>     NBEATS(**CONFIG, stack_types=3 * ["identity"]),
>>>     NHITS(**CONFIG),
>>>     MLP(**CONFIG),
>>>     MLP(num_layers=3, **CONFIG),
>>> ]
>>>
>>> nf = NeuralForecast(models=models, freq='M')
>>>
>>> # cv to build meta-data
>>> n_windows = df['unique_id'].value_counts().min()
>>> n_windows = int(n_windows // 2)
>>> fcst_cv = nf.cross_validation(df=df, n_windows=n_windows, step_size=1)
>>> fcst_cv = fcst_cv.reset_index()
>>> fcst_cv = fcst_cv.groupby(['unique_id', 'cutoff']).head(1).drop(columns='cutoff')
>>>
>>> # fitting combination rule
>>> ensemble =  EqAverage()
>>> ensemble.fit(fcst_cv)
>>>
>>> # re-fitting models
>>> nf.fit(df=df)
>>>
>>> # forecasting and combining
>>> fcst = nf.predict()
>>> fcst_ensemble = ensemble.predict(fcst.reset_index())

__init__(trim_ratio: float = 1, select_by_uid: bool = True)[source]

Initialize equal-weights ensemble with optional trimming.

Parameters:

select_by_uid (bool, default=True) – Strategy for model selection in trimming: - True: Select best models separately for each series - False: Select best models across all series Per-series selection allows more granular model choice but requires sufficient data per series.
trim_ratio (float, default=1.0) – Proportion of models to retain in ensemble, between 0 and 1: - 1.0: Keep all models (simple average) - 0.5: Keep top 50% of models - Lower values create more selective ensembles

Notes

Models are selected based on validation performance before applying equal weights. As shown in [1], moderate trimming often improves forecast accuracy while maintaining the robustness benefits of equal weighting.

References

[1] Jose, V. R. R., & Winkler, R. L. (2008). “Simple robust averages of forecasts: Some empirical results.” International Journal of Forecasting, 24(1), 163-169.

update_weights(**kwargs)[source]

Updating the combination weights

Not implemented yet

class Windowing(freq: str, select_best: bool = False, trim_ratio: float = 1, weight_by_uid: bool = False, window_size: int | None = None)[source]

Bases: ForecastingEnsemble

Combine forecasts based on recent performance in sliding windows.

A dynamic ensemble that adapts model weights based on accuracy in recent time windows. This approach is particularly suited for non-stationary data where relative model performance changes over time [1], [2].

References

[1] Cerqueira, V., Torgo, L., Oliveira, M., & Pfahringer, B. (2017). “Dynamic and heterogeneous ensembles for time series forecasting.” In IEEE International Conference on Data Science and Advanced Analytics (DSAA) (pp. 242-251).

[2] van Rijn, J. N., Holmes, G., Pfahringer, B., & Vanschoren, J. (2015). “Having a blast: Meta-learning and heterogeneous ensembles for data streams.” In IEEE International Conference on Data Mining (pp. 1003-1008).

Examples

>>> from datasetsforecast.m3 import M3
>>> from neuralforecast import NeuralForecast
>>> from neuralforecast.models import NHITS, NBEATS, MLP
>>> from metaforecast.ensembles import Windowing
>>>
>>> df, *_ = M3.load('.', group='Monthly')
>>>
>>> # ensemble members setup
>>> CONFIG = {'input_size': 12,
>>>           'h': 12,
>>>           'accelerator': 'cpu',
>>>           'max_steps': 10, }
>>>
>>> models = [
>>>     NBEATS(**CONFIG, stack_types=3 * ["identity"]),
>>>     NHITS(**CONFIG),
>>>     MLP(**CONFIG),
>>>     MLP(num_layers=3, **CONFIG),
>>> ]
>>>
>>> nf = NeuralForecast(models=models, freq='M')
>>>
>>> # cv to build meta-data
>>> n_windows = df['unique_id'].value_counts().min()
>>> n_windows = int(n_windows // 2)
>>> fcst_cv = nf.cross_validation(df=df, n_windows=n_windows, step_size=1)
>>> fcst_cv = fcst_cv.reset_index()
>>> fcst_cv = fcst_cv.groupby(['unique_id', 'cutoff']).head(1).drop(columns='cutoff')
>>>
>>> # fitting combination rule
>>> ensemble = Windowing(freq='ME', trim_ratio=.8)
>>> ensemble.fit(fcst_cv)
>>>
>>> # re-fitting models
>>> nf.fit(df=df)
>>>
>>> # forecasting and combining
>>> fcst = nf.predict()
>>> fcst_ensemble = ensemble.predict(fcst.reset_index())

__init__(freq: str, select_best: bool = False, trim_ratio: float = 1, weight_by_uid: bool = False, window_size: int | None = None)[source]

Initialize window-based dynamic ensemble.

Parameters:

freq (str) – Time series sampling frequency (e.g., ‘M’ for monthly, ‘D’ for daily). Used to set default window size if not specified.
select_best (bool, default=False) – Model selection strategy: - True: Use single best model from window - False: Use weighted combination of models Single best can be more aggressive in adapting to changes
trim_ratio (float, default=1.0) – Proportion of models to retain in ensemble, between 0 and 1: - 1.0: Keep all models - 0.5: Keep top 50% of models Models are selected based on window performance
weight_by_uid (bool, default=True) – Whether to compute weights separately for each series: - True: Individual weights per series (may be computationally intensive) - False: Global weights across all series
window_size (int, optional) – Number of recent observations used for performance evaluation. If None, defaults to frequency-based size: - Monthly: 12 observations - Daily: 30 observations etc.

fit(insample_fcst, **kwargs)[source]

Update performance statistics of ensemble members based on recent forecasts.

Used to identify and retain top-performing models for ensemble trimming. Updates internal statistics tracking each model’s forecast accuracy.

Parameters:

insample_fcst (pd.DataFrame) –

Forecast and actual values dataset formatted like mlforecast cross-validation output. Contains either: - In-sample forecasts (predictions on training data) - Cross-validation results (out-of-sample predictions)

Expected columns: - unique_id (or other id_col): Identifier for each series - ds (or other time_col): Timestamp - y (or other target_col): Actual values - model_name: Predictions by a model with name model_name

Returns:

self, with computed self.weights

Return type:

self

predict(fcst: pandas.DataFrame, **kwargs)[source]

Combine ensemble member forecasts based on recent performance.

Parameters:: fcst (pd.DataFrame) – Forecasts from individual ensemble members. Expected columns: [‘unique_id’, ‘ds’, ‘model_name1’, ‘model_name2’, …]
Returns:: Combined ensemble forecasts for h periods ahead.
Return type:: pd.Series

update_weights(**kwargs)[source]

Updating the combination weights

Not implemented yet