We tend to make decisions based on emotion, intuition, gut feel, and wishful thinking. At the same time, we assume these are the wrong tools for decision-making. In our culture, there is a belief that all decisions and all conclusions must be based purely on logic, reason, science, or statistical evidence. It seems that people feel there is something wrong with conclusions or decisions that are arrived at by any means other than cold, calculating logic. (Never mind, for the moment, people’s demonstrated ability to apply logic.) There is an apparent desire to rid ourselves of emotion, morality, and even personal preference when making choices, even though this seems to be contrary to our nature.
This assumption is so deeply ingrained in our culture that we have formally defined decision-making on any other basis as an error. We call it Base Rate Neglect (regarded as a cognitive bias) or Base Rate Fallacy (regarded as a logical fallacy). But which is the true fallacy: The use of non-logical decision-making methods, or the belief that such methods are to be eschewed categorically?
The phrase “Vulcan choice” is inspired by the character Spock from the old television series, Star Trek. In the Star Trek reality, each humanoid race or species represents a single aspect of human nature taken to an extreme. This is a common literary device in science fiction that allows us to explore particular aspects of the human condition without the distractions of a too-realistic setting.
Thus, the Klingons represent our warrior nature, including aspects of honor, violence, courage, loyalty, and betrayal; the Ferenghi represent our greed and mercantilism, in a way that suggests we are not particularly proud of these traits in ourselves; the Humans represent hubris, extending American Exceptionalism to a planet-wide scale and dictating the structure, principles, and procedures of the Federation, even overriding older, more mature, and more technologically-advanced Federation member societies; the Q Continuum represents those things we cannot control, and for which we ask God’s “grace to accept with serenity.” (Reinhold Niebuhr)
Similarly, the Vulcans represent the conflict between logic and emotion. To free themselves from the problems of emotionalism, which had led to widespread war and suffering in their distant past, the Vulcans developed an extreme form of mental discipline and attempted to eradicate all emotional responses and non-logical considerations from their decision-making. Spock was half Vulcan and half Human. He had an inherent internal conflict in that he tended to suppress his human side in favor of Vulcan logic. He believed logic ought to yield better outcomes than emotion. This proved to be a problematic choice for him, causing him considerable internal turmoil and unhappiness. It was only late in life that the character came to terms with his own nature and found inner peace; and with it, wisdom to complement his accumulated knowledge.
In the episode, “The Doomsday Machine” (episode description), based on Fred Saberhagen’s Berserker concept (berserker stories), an entire ship was consumed by the planet-eater because its crew of 400 Vulcans could not arrive at any useful conclusions based solely on statistical analysis of cold data. The Enterprise, crewed mostly by Humans, prevailed through the use of good old human intuition and creativity.
The Vulcan choice may be problematic for us, as well, as we are not machines and not creatures of pure logic by nature. It is true that we possess some degree of problem-solving ability, but this alone does not define us. Few, if any, breakthrough ideas in business or technology come about by statistical analysis of cold data. While it is true that decisions based only on one or two data points are likely to be poorly reasoned, it is equally true that decisions based solely on statistical analysis are likely to be poorly reasoned.
Except in rare cases of dumb luck, the statistical average does not correspond closely enough with any real-world situation to provide sound guidance for on-the-ground decision-making, absent some other source or sources of information or insight…insight being key. The average family may have 2.4 children, but how many actual families have exactly 2.4 children? After your four-year-old has been in time-out for four minutes, according to the book, and is still screaming, how do the statistical averages help you understand what to do next?
Fortunately, we are not limited to a binary choice between a wild-eyed emotional decision or a cold-hearted Vulcan one. We can learn enough about general patterns to discern the pertinent details in any given situation, and make choices based on a wise selection from among those details. This gives us the best of both worlds: The power of human intuition and judgment bolstered (but not hamstrung) by statistical information. Statistical information can be helpful. Rigorous logic can be helpful. But neither of those alone, nor the two combined, has ever resulted in a human decision of any real significance.
You know what to do after the four-minute time-out for the most human and non-logical of reasons: You know your child. He is not a statistical average, he is a unique individual. When you make that decision, you are not submitting to Base Rate Fallacy. You are using plain old human judgment. That is not an error.
You know the situation in your company. When you try a new technique or method, be it Scrum or Pair Programming or Kanban or whatever, and it doesn’t “work” (whatever that means), you can make a sound judgment call about what to try next. You are using plain old human judgment. That is not an error.
Trust yourself.
I would like to add to this, though I fear I may have just failed to grasp that this is actually at the heart of what you wrote and I merely think it to be a related angle.
Emotions are a form of biological feedback; they are not actually illogical. Rather it is how one responds to them that can be illogical or unreasonable: the person ruled by his appetites or passions even when it is clearly killing him and may even run contrary to larger goals. Sometimes this feedback can be misleading, just like sometimes one can get bad data from a malfunctioning or incorrectly used tool.