The Dunning-Kruger Effect

For a while now, I have felt that the more I know, the more I know that I don't know it all. Confidence in my personal mastery is not growing as quickly as the uncertainty of it, even if I am slowly getting better at things over time.

This is because as I am achieving some level of incremental success in what I am doing, I perceive what further deficiencies I have and where there is more room for improvement. This makes for someone less sure of themselves rather than more confident.

Conversely, I learned that it was pointless to get upset when other people do not realize what is wrong when they are just being themselves. To punish the ignorant for their ignorance would be unfair. It'd be unwise to expect something uncharacteristic of the person. One would be bound for disappointment to hope for attention to detail from the disinterested, initiative from the unmotivated, empathy from the uncaring, mercy from the unforgiving, kind words from the tactless... the list goes on.

One can't throw up ones hands in the air in exasperation and say, "why can't he/she be more thoughtful?" without the answer dawning on you moments later. It's because the person isn't thoughtful.

While watching the ever enlightening TV program "QI" hosted by Stephen Fry, I finally came across a clear explanation why.

At the 21m 12s mark, which you can skip to by clicking here, Stephen asks the question, "How do you know if you are incompetent?".

Stephen goes on to explain, "You don't. There is a thing called the Dunning-Kruger effect. Which is if you are incompetent, you don't know it because the thing that makes you incompetent means you don't realize what the competent thing is..."

"...the fact is, we don't know what we don't know..."

Stephen shows these hilarious mugshots of two burglars, photographed with the rudimentary disguise made from permanent marker:

Since they thought their "disguise" might work, it could explain how the pair was led to the crime and subsequent arrest in the first place.

A clearer explanation of the Dunning-Kruger effect taken from wikipedia:

The Dunning–Kruger effect is a cognitive bias in which unskilled people make poor decisions and reach erroneous conclusions, but their incompetence denies them the metacognitive ability to recognize their mistakes. The unskilled therefore suffer from illusory superiority, rating their ability as above average, much higher than it actually is, while the highly skilled underrate their own abilities, suffering from illusory inferiority.

Actual competence may weaken self-confidence, as competent individuals may falsely assume that others have an equivalent understanding. As Kruger and Dunning conclude, "the miscalibration of the incompetent stems from an error about the self, whereas the miscalibration of the highly competent stems from an error about others". The effect is about paradoxical defects in cognitive ability, both in oneself and as one compares oneself to others.

So it seems that I've managed to intuitively understand half the theory: I do not get frustrated when someone presents themselves to be quite a challenge and moan about who they are. I recognize that their weaknesses can also be their strengths and try to find alternatives to achieve our goals together. I no longer (or try my best not to) just condemn someone as stupid or being difficult and get upset. I try to identify what qualities they have that are causing this or if they actually have different interests in the situation. Then I try to find a way to encourage our cooperation and mutual benefit. It takes creating a situation or adopting a method where we can capitalize on each others strengths and not let our weaknesses undermine the objectives.

I don't always do this well and it doesn't always work but hey it beats putting a fist through a wall.

What I have been failing at, it seems, is to sufficiently recognize that what may seem easy to me, may not be equally easy to others. I don't think I'm very great at all, so it's also rather uncomfortable to receive compliments or praise. When asked by my sales team if a certain creative execution would be feasible, I am more uncertain than certain because I can see so many possibilities of things going wrong. I recognize so many critical variables involved which we may or may not be able to control. The best I can do is give a conditional approval but highlight all the potential pitfalls and work to prevent them or be prepared for them.

This approach usually reduces the chances of being caught unaware of a problem. The result is often success but I experience such discomfort and uncertainty along the way, not being able to feel confident for most of the time.

In the end, when I get congratulated for a job well done, I'm really just relieved that the worst did not happen. I hardly feel any basking glory of achievement (these days). If a project doesn't worry me at all, I move on to devote more time into the projects that do worry me (quite naturally). Somehow I know that I'd only get better at my game by dealing with new and diverse problems, so the attention always lingers on the worrisome.

I hope I don't sound like I am blowing my own horn and inferring that I am "highly competent", but I worry too much and like I said at the start: the more I know, the more I know that I don't know it all. I guess what I'm trying to internalize myself with from all this rambling is: Don't worry so much and be more confident. You'll be fine, Sid. You will.