Good Player Personnel: Is it Moneyball, Scouting, or Sheer Luck?

Michael Bond of New Scientist Magazine has a fascinating interview with statistician Nassim Nicholas Taleb ("Life is Unpredictable: Get Used to It," 7/5/2006, subscription only), in which Taleb argues that humans are terrible at making predictions and that they tend to place false confidence in statistical models that yield results perhaps no better than mere guessing or sheer serendipity. Along those lines, the idea goes, we strive for explanations and patterns because they give us a greater sense of control, even when those explanations and patterns may be completely illusory. Here are some excerpts from his comments:

Our brains operate on autopilot most of the time . . . Our track record [for predictions] is quite dire. Look at the net, computers, lasers. The internet was designed as a military system, not for chat rooms. The person who first marketed computers didn't think he would sell more than five. The laser was designed by a physicist who had no idea how it might be used. You can't even forecast something that would affect us tomorrow - revolutions, wars, epidemics, political changes, economic variables . . . A forecast is irrelevant unless you have an error rate on it. But if this happened, these people would realise there was no point in forecasting, because their error rate would be so monstrous . . . [Successful forecasting is just luck], and you need a lot of luck to forecast things accurately.

[Question: But don't people make predictions based on history all the time?]

As well as our ability to concoct empirically flawed narratives to explain past events, there are biases in history that we don't seem to be aware of and that make us overestimate the causal links between events: for example, when you see only the winners and not the losers. When you look at the fossil record, you see only the species that left a fossil. You cannot make a generalisation of all species just from fossils - you have to take into account the species that left none. History has a lot of hidden pockets. You can't take it any more seriously than a visit to a museum.

What do Taleb's conclusions suggest about things like Moneyball and the rise of more complex-statistical models for player personnel decisions? Do Moneyball teams succeed for reasons that have nothing to do with empiricial approaches or taking advantage of "market inefficiencies"? Are "market approaches" then receciving too much credit (or blame) in management decisions (e.g., whether it is a good idea to draft or sign a particular player)? Of course, Nassim's conclusions don't validate traditional scouting models either, because they too would invite the same after-the-fact explanations for lucky results.

So statistics or not, is team management really about luck and coming up with an explanation later?

Posted in: