The System Maintains Itself — Drift-Triggered Re-Tuning

Drift detection requires a stable measurement. Single-window backtests are too noisy — a model that randomly hit a calm month looks great; a model that hit a rate-decision week looks terrible. Walk-forward across 10+ cutoff dates averages out the noise. This is the only number the drift detector trusts.

# run_weekly_retrain.py — Step 1

def walk_forward_mape(pair, model, n_cutoffs=10):
    cutoffs = sample_cutoffs(start='2023-01-01', end='today', n=n_cutoffs)
    mapes = []
    for cutoff in cutoffs:
        train_df = data[data.date < cutoff]
        test_df  = data[(data.date >= cutoff) & (data.date < cutoff + 30d)]
        model.fit(train_df)
        forecast = model.predict(horizon=30)
        mape = mean(abs(forecast - test_df.actual) / test_df.actual)
        mapes.append(mape)
    return median(mapes)        # median, not mean — robust to single bad cutoff

A single-week MAPE spike isn't drift, it's noise. Drift is persistent. The detector compares this week's walk-forward MAPE to the baseline tuned MAPE; if the ratio exceeds 1.5× for two consecutive weeks, it triggers an Optuna re-tune. Baselines live in tuned_hyperparameters.json; thresholds in config.json.

# source.monitoring.drift_retune

def check_drift(pair, model):
    current  = walk_forward_mape(pair, model)
    baseline = tuned_hyperparameters[(model, pair)]['mape']

    ratio = current / baseline
    history.append(ratio >= 1.5)        # rolling 2-week window

    if len(history) >= 2 and all(history[-2:]):
        # Persistent drift — re-tune
        log_warning(f"DRIFT DETECTED: {model}/{pair} — "
                    f"MAPE {current:.4f} vs baseline {baseline:.4f} "
                    f"({ratio:.1f}x) for 2 consecutive weeks → RE-TUNE TRIGGERED")
        return True
    return False

def retune(pair, model):
    # Optuna 30 trials — narrower than initial 100
    best = optuna.optimize(model, pair, n_trials=30, walk_forward=True)
    tuned_hyperparameters[(model, pair)] = best
    save(tuned_hyperparameters)         # atomic JSON write

Before deploying drift-triggered re-tuning, I simulated 6 years of weekly tunes against the no-tune baseline, per (model, pair). The result was strikingly pair-specific — the same drift detector that helps SGD/SARIMA actively hurts JPY/SARIMA. The production rules below are encoded directly in drift_retune.py.

The weekly retrain script trains all 24 model-pair combinations (3 models × 8 pairs) every Monday — that part is non-negotiable, it's how we measure drift. But re-tuning hyperparameters via Optuna is the expensive step: 30 trials × ~80 seconds each = ~40 minutes of compute per re-tune. Doing this for every pair every week is wasteful.

The most recent successful retrain logged a real drift event. This is what the system was built to do.

# Mondays 9:30 AM via LaunchAgent — com.mockpineapple.weekly-retrain

bash projects/mock_pineapple/run_pipeline.sh --retrain

  STEP 1 · walk-forward MAPE per (model × pair) over 10 cutoffs
        24 fits × ~1-2s each ≈ 30s

  STEP 2 · drift_retune.py
        for each (model, pair):
            check_drift() → returns True/False
            if True and (model, pair) ∉ SKIP_RETUNE: queue_retune()

  STEP 3 · Optuna re-tune queued items (sequential, ~40-60s each)
        Mar 30 example: 3 items queued, 2 actually re-tuned (SARIMA/SGD, LightGBM/SGD)
        save tuned_hyperparameters.json (atomic write)

  STEP 4 · refit production models with new hyperparameters
        write data/trained_models/YYYY-WXX/manifest.json
        sarima.pkl, prophet.pkl, lgbm.pkl per (pair, model)

  STEP 5 · regenerate dashboards
        dashboard/generate_analyst_dashboard.py
        → mape_dashboard.html (model quality, ensemble weights)
        → analyst_report.html  (7-finding narrative)
        → bucketed_mape.html   (per-horizon bucket detail)

  Total wall time: ~25-40 min (depends on how many re-tunes triggered)