Cognitive Biases in Agile Process, Agile Leaders, Agile Practitioners, and Agile User Group Participants

Recently I have been studying cognitive biases. I soon began to think that, "If a person or group of persons value certain biases and they compose some method or process are these biases passed on to the process and are they evident in the process?"

For reference I will use Wikipedia's entry for a list of Cognitive Biases.

Also, I will use Wikipedia's entry for Agile Software Development and the Agile Manifesto.

Of course anything I report will be affected by my own personal biases, as with any report, but this should not cause one to avoid the task but instead to recognize the reality that human bias manifests itself.

Anchoring – the common human tendency to rely too heavily, or "anchor," on one trait or piece of information when making decisions.

Agile Leaders state their opinion that software development had become heavily anchored to processes, tools, comprehensive documentation, contract negotiation, and following a plan.

Agile Leaders recognized the value of these "anchors" but proposed that such biases may actually have adverse affects in software development.

Agile processes reflect the belief of the agile leaders manifesting a shift from these traditional anchors by placing individuals, working software, customer collaboration, and response to change on one side of the balance and the traditional anchors on the other, with the scale tipping in favored weight for the new agile values.

Bandwagon effect – the tendency to do (or believe) things because many other people do (or believe) the same. Related to groupthink and herd behavior.

In my opinion the majority of participants in any software process, traditional or agile, are those on the bandwagon. Bandwagon participation may be justified by an authority bias in that one will say, "Most everyone in the industry is following this process and the process has been defined and recommended by several industry leaders." This opinion coincides with my experience in following company efforts in adopting agile processes.

Bias blind spot – the tendency to see oneself as less biased than other people.

I believe that many agile practitioners and agile user group participants suffer from the blind spot bias.

Confirmation bias – the tendency to search for or interpret information in a way that confirms one's preconceptions.

Many arguments, contentions, and heated debates between different schools of thought concerning software development have their premise based upon building blocks gathered specifically to confirm one's preconceptions. If someone finds a success story from a group that executed a waterfall approach to software development does that make that success transferable? That same goes for an agile shop.

Distinction bias – the tendency to view two options as more dissimilar when evaluating them simultaneously than when evaluating them separately.

During the early years of the agile reformation many tried to contrast the new agile methods with traditional methods by showing them to be very dissimilar. In my opinion the similarities of all software processes are greater than their dissimilarities.

Hyperbolic discounting – the tendency for people to have a stronger preference for more immediate payoffs relative to later payoffs, where the tendency increases the closer to the present both payoffs are.

This bias is particularly interesting in the Agile community. This one bias deserves in-depth treatment on its own. Briefly though I notice agile practitioners weigh immediate "everything" as preferred. For those that recognize such behavior as a bias it can cause conflict during planning meetings and discussions.

Illusion of control – the tendency to overestimate one's degree of influence over other external events.

Traditional software process seemed to suffer greatly from the illusion of control. For me, the recognition of this bias, was one of the main motivators for me to try and figure out a better way to develop software. Big Upfront Design wasn't working on large projects. Therefore I had began to lay ground work for alternative approaches and to identify methods and practices that seemed to contribute to the issues. I remember the CCB, the Change Control Board, where all changes had to go through committee. When I started reading about Extreme Programming I recognized some common concerns and therefore was further intrigued. (Initially I didn't see how XP worked together and started to write a paper against the process, but as I gave real effort in creating real examples of how XP principles didn't apply I soon found that they did.)

Irrational escalation – the phenomenon where people justify increased investment in a decision, based on the cumulative prior investment, despite new evidence suggesting that the decision was probably wrong.

With heavy processes I have experienced this irrational escalation bias many times. "We have hundreds of pages of documentation and we just can't throw out all of that valuable work!"

I see the same with code. We have tens of thousands of lines of code and we can not afford any kind of change to that base.

Agile processes have offered many approaches to the avoidance of this bias. Refactoring, Test First, Continuous Builds, and many other.

Neglect of probability – the tendency to completely disregard probability when making a decision under uncertainty.

Agile processes speak directly to uncertainty and probability. "You Aint Going To Need It" (YAGNI) is an excellent example of this.

Normalcy bias – the refusal to plan for, or react to, a disaster which has never happened before.

Sometimes agile practitioners are accused of the normalcy bias. Through the use of the mantra "Do the simplest thing that works" leads people to believe that there is no planning for disaster. Some people may chose to not plan for disaster but I do not know of any software process that says not to be concerned with the potential of disaster.

Outcome bias – the tendency to judge a decision by its eventual outcome instead of based on the quality of the decision at the time it was made.

I believe that agile processes are outcome bias and that the agile leaders recognized this bias and therefore recommend short delivery cycles.

Semmelweis reflex – the tendency to reject new evidence that contradicts an established paradigm.

I believe that the adoption of any process may be effected by the Semmelweis reflex. Just as groups that practice traditional software processes have had difficulty accepting agile processes, groups that practice agile processes will have difficulty accepting some other process.

Status quo bias – the tendency to like things to stay relatively the same (see also loss aversion, endowment effect, and system justification).

Any individual or group of people may have the status quo bias. Sometimes persons dealing with others that are rejecting some proposed change immediately label the others as suffering with status quo bias. I don't know the name of the bias of labeling others with the most commonly known biases is called, but it definitely exists. This labeling may close communication paths and make it more difficult for change. (I have found it: the fundamental attribution error, correspondence bias or attribution effect.)

Forward Bias - the tendency to create models based on past data which are validated only against that past data.

This bias is tied to estimation. Someday I will have to give thought to "yesterday's weather" as compared to forward bias and the gambler's fallacy.

Primacy effect – the tendency to weigh initial events more than subsequent events.

Traditional processes often suffer from the primacy effect. Agile processes are not immune from the primacy effect either.

In the context of source code, the primacy effect is very real. The abstract idea of code mass may be a result of the primacy effect.

Disregard of regression toward the mean – the tendency to expect extreme performance to continue.

I feel that some advocates of new processes intentionally disregard the regression toward the mean. I have particularly noticed this in various implementations of SCRUM, where the idea is that performance increases should be expected over several months as the practitioner improves in their application of SCRUM.

Stereotyping – expecting a member of a group to have certain characteristics without having actual information about that individual.

I feel that this happens too often in the areas of "swarming" and "cross functional team".

Conclusion

I believe that biases are in full effect in the Leadership of any group and that those biases will be reflected in their proposed methods and processes.

Agile Leadership recognized various biases reflected in traditional processes and addressed some of those biases.

Agile practitioners and Agile user group participants manifest their particular biases as individuals and other biases associated with groups.