Skip to main content

NotesWorking with AI

Agreement is not corroboration

Several sources agreeing is only as informative as their independence.

What I was looking for

Whether disagreement between forecasting models is a useful measure of uncertainty.

What I found

Ensembles, committees and voting agents use agreement as a quality filter. If the voters are correlated, strong consensus can be one opinion counted many times; if they share a fallback, it can be no opinion at all.

In a forecasting ensemble I built, 23 of 37 questions produced the identical triple of answers from the cheap models. The dissent statistic was measuring which constant each model defaults to, not disagreement.

The move

Before I trust a consensus, I ask what the voters share: a model, an input, or a default they fall back to.