Exemplar's curse—Now with 80% more math!

Edit: (Marks 2008) covers essentially the same ground. But there’s no interactive calculator!

Intro

Last time, I outlined the exemplar’s curse in the context of Singapore with a parable and an informal description. Rest easy; I’ve heard your needful clamoring—I’ll now describe the curse more precisely with math.

The exemplar’s curse

Restating the core idea in words: The exemplar’s curse occurs when we select an exemplar from a set of outcomes which resulted from both stochastic and deterministic factors. If many outcomes have similarly compelling deterministic factors, the chosen winner is probably unusually lucky. Math then suggests that the chosen winner will disappoint when the deterministic factors are replicated.

Model

We can model this with the use of random variables. We’ll say that there’s a bundle of deterministic factors which we’ll represent as having a cumulative value (on some presently ill-defined scale of quality) ranging uniformly from $0$ to $D$ where $D$ is finite. Our bundle of stochastic factors range uniformly in value from from $0$ to $S$ where $S$ is finite. Since we observe only visible outcomes rather than underlying causal factors, we see $O = D + S$. The exemplar’s curse is then about the inferrable properties of the causal factors corresponding to the selected maximum $O$ from a set of outcomes $\mathbb{O}$. In other words, if we have a set of observable outcomes $\mathbb{O}$ and select the best outcome $O$ from that set, what can we infer about the underlying structure of $O$—how big are that $O$’s $D$ and $S$?

False exemplars

In the last post, we only went so far as to claim that we should expect replicating causal factors to produce disappointing outcomes due to a sort of reversion to the mean. That is, the stochastic factors $\mathcal{S}_1$ for the maximum outcome $O_1 = \max \mathbb{O}$ are likely better than average ($\frac{S}{2}$). If we generate a new outcome $O_2$ using the same deterministic factors $\mathcal{D}$ that served us well in $O_1$, we should expect our new stochastic factors to be worse $S_2 < S_1$ and so we should expect $O_2 < O_1$.

This leaves a open a compelling retort. One could say, “Even though I’m too optimistic about the eventual outcome, in selecting the exemplar, I’m still selecting the best deterministic factors. That means I’m still making the best choice I can, so no harm done.”

Alas, this is not true. Depending on the parameters, there could be only a vanishingly small chance that the bundle of deterministic factors corresponding to the best outcome (the sum of the deterministic and stochastic factors) is also the best bundle of deterministic factors when looking only at the deterministic factors. In symbols, supposing we have a projection function $p_\mathcal{D} : \mathbb{O} \rightarrow \mathbb{D}$ which finds the $D$ used in outcome $O$, we’re interested in $P(\max \mathbb{D} = p_\mathcal{D}(\max \mathbb{O}))$.

For example, if we choose the max from 1000 outcomes and the value of stochastic factors ranges from 0 to 100 while the value of deterministic factors ranges from 0 to 1, we should be quite surprised if our best outcome actually has the best deterministic factors.

Calculator

We can help build up an intuition around this math using the calculator below. The calculator uses Monte Carlo methods to estimate the probability that the maximum outcome corresponds to the maximum bundle of deterministic factors.

You can change the range of values for stochastic and deterministic factors and change the number of outcomes we’re taking the max from (i.e. Our winner is the winner out of how many contestants?).

This results in:

• The mean value of stochastic factors in best outcomes across Monte Carlo samples. This allows us to estimate the magnitude of expected bias.
• An estimated probability that our observed best outcome actually corresponds to the best bundle of deterministic factors.
• A chart showing all the trials used to produce the estimate. One interesting thing to note about the chart is that the number of points on the diagonal corresponds to the probability of ‘success’ (i.e. the probability that the best outcome also has the best deterministic factors).

Some trends worth noting:

• As the stochastic factors get relatively larger, the probability goes down
• As the deterministic factors get relatively larger, the probability goes up
• As the number of outcomes goes up, the probability goes down and the mean of stochastic factors goes up

This confirms the intuition highlighted by the example a few paragraphs up.

Conclusion

So we see that the exemplar’s curse is even worse than we thought. It’s not just about our outcomes regressing to the mean in spite of making the best possible choice. It will often lead us to pick the wrong exemplar!