Post 4 of the "Building Mochi" series.
Somewhere on my Mac right now, a small statistical model is predicting tomorrow's version of me — my energy, my clarity, whether the week is about to catch up with me. It has been right 14 of the last 15 times it made a call. Because of that track record, it has earned a privilege: Mochi is allowed to use its forecasts when she talks to me.
A second model watches how my mental states flow into each other — focused to overwhelmed to recovering. It runs about 62%. Also allowed to speak, barely.
A third model — the one I had the highest hopes for — tracks my "recovery debt," the slow-draining battery underneath the daily mood. It has been right once in seventeen tries. Six percent. And here is the whole point of this post: I have never heard a word from it. Not because I filter her messages. Because she is built so that a model must earn the right to be heard.
The problem with confident machines
Language models are eager. Ask one to predict your behavior and it will — fluently, plausibly, and with total indifference to whether it's right. The research here is sobering: in formal forecasting tournaments, the best AI ensembles still lose to trained human forecasters, and studies this year showed something more specific — AI is decent at reading your current state and bad at projecting it forward. Inference is not forecasting.
Most AI products handle this by sounding confident anyway. I went the other way.
The gate
Every forecast any of Mochi's models makes is written down at the moment it's made — the prediction, the confidence, the date it settles. When that date arrives, it gets scored against what actually happened. No exceptions, no grading on a curve, and she doesn't get to score her own essay questions; settlement is mechanical.
A model starts life in shadow: it runs, it predicts, it settles — and nobody sees it. Only after at least five settled forecasts at a passing rate does it graduate to surfaceable, meaning its outputs can inform what Mochi says to me. Drift below the bar and it's automatically demoted back to shadow. "She sounds confident" is impossible until "she has been right" is literally true.
The 6% model has been in shadow since the day it was born. The gate did exactly what it was built to do: it took a component I believed in — I designed the thing — and silenced it because reality disagreed with me. I can't tell you how clarifying that is. My opinion of the model didn't matter. Seventeen settled forecasts did.
Why this matters beyond my office
Every personal AI you'll be offered in the next few years is going to predict you — your mood, your churn risk, what you'll buy, when you're persuadable. Almost none of them will tell you their hit rate, and most of them won't know it themselves.
I think calibration-gating should be table stakes for any AI that makes claims about a human being. It costs almost nothing: a ledger, a settlement job, a threshold. What it buys is a system that is structurally incapable of confident nonsense about you — not because the model is smarter, but because the plumbing refuses to transmit unproven confidence.
Next upgrade, already underway: replacing the simple hit-rate with a proper scoring rule (Brier score, for the forecasting nerds), so a lazy model can't pass by predicting the safe answer every day. And a new lane of predictions that settle fast — will I make the gym today, will I approve the morning's draft by ten — because predictions that settle daily let a model prove itself in weeks instead of months.
She reads me well. But reading me was never the hard part. Earning the right to say what comes next — that's the part I made her work for.
Previously: She Dreams at 3 A.M.. Next in the series: A Conscience With One Exit (coming soon).
I build and run Mochi myself. I research and draft these posts with AI assistance, and every claim, number, and story in them is mine and verified by me. Key sources: ForecastBench / the Brier Index (Forecasting Research Institute); DialToM (arXiv:2604.20443) on state inference vs. behavioral forecasting.