Thinking

The Brain and Probabilities

The brain is a wonderful machine with an impressive computing power. We can make sense of complex information effortlessly and almost instantly. But it has one big flaw: it does not understand probabilities.

When presented with information, the brain tries to explain it by building a coherent story out of it. To do so quickly, it relies on some shortcuts, which largely ignore probabilities. So the story you get isn’t necessary the most probable, but instead the cheapest it could construct, as long as the story remains plausible.

One of the shortcuts is to trade availability of information for probability (availability heuristic): the information you can recall quickly is deemed more probable than other information. As a result, the probability of sensational events inflates while the probability of mundane events shrinks.

Another of the shortcuts is to only consider what is visible and extrapolate from there (all there is is what you see): only the visible information is considered, without even considering that something could be missing from the picture. Somebody looking nice will be considered a nice person, unless additional negative information about him is given.

The brain tries so hard to build a story that it will see patterns even in random data. It will infer causality very quickly, and with very little. As Kahneman puts it, the brain is “a machine for jumping to conclusion”.

It important to remember the dinstinction between plausible and probablewhen it comes to judgment, because we’re taking decisions all day long.

You hear a project was using a new methodology and was very successful with it so you want to use it as well? Beware the survivorship bias. You don’t know how many projects used the methodology and failed…

You think the biggest risk in your project is a distributed attack from China? Beware the availability heuristic. Your biggest risk might be to not have proper input validation…

You see bug reports for your teams and start detecting a pattern? Beware the law of small number. Your sample size might be too small…

Our brain is hardwired to tell us stories. So it’s very hard to improve on our handling of probabilities. Often, we’re simply not aware that probabilities are at play. And even when we are, it’s really hard to change our instinct in some cases: if you flip 9 times a coin and  got 9 times head, the probability to get tail on the 10th flip is higher, right? Well, actually not. But it’s hard to not feel otherwise.

So, the best remedy is to default to a healthy skepticism and accept that the outcome of many situations in life is simply the result of chance. It might sound like fatalism but it isn’t at all. Your actions will influence the outcome of many situations, but you don’t know how. Don’t buy the first plausible explanation your brain tries to sell you, it’s probably not the right one.

References 

Thinking

Science Book Reviews

Besides IT books, I also read popular science books. Reading is a passive activity and I like to write reviews or summary to better engage with the topic. As usual, it’s only when writting things down that I realized that I didn’t fully understood them.

I will keep track here of my reviews and update this post as needed.

Thinking

Lateral Thinking

Lateral thinking is a term coined by Edward De Bono to characterize the generation of alternative ideas, as opposed to vertical thinking, which generates ideas based on logic and stepwise refinements. Another way to explain lateral thinking in a much common way is “thinking out of the box.”

Often finding the best solution to a problem requires a creative move to go away from the existing solution and start with a new angle. This is were lateral thinking can help.

As a reminder of the power of lateral thinking, let us take an egg and a spoon. You are doing a brunch. How do you provide assistance to help cut the egg?

With vertical thinking you might come up with this solution:


With lateral thinking, maybe with this one:


I was absolutely amazed the first time I saw this device in action. The cut is perfect. Also, I would probably never have come to this solution, no matter how long I stared at my egg.

Each time I discuss a design issue I remember my last brunch and try to take some distance with the situation to go back to the root of the problem to solve and ask: could we do this completely differently?

Sometimes the best way to cut an egg is to not cut it actually.

Thinking

The Zen of Oscar

Writing a good research paper or thesis is hard. It can be very intimidating to figure out the scope of the work needed to be done. Thanks to Oscar, I have been empowered with three adjectives to  think about problems of scope. These adjectives were: breadth, depth and completeness.

Breadth and Depth

At the early stage of my thesis, I prepared a list of action items that I proposed to address for my research. I went to see Oscar for a confirmation that addressing these points would lead to a good thesis. I described him the list of action items and asked “Oscar, if I realize all this, will this make a good thesis?”. He look at me and said “For a good thesis, you must cover a topic with sufficient breadth and depth“. I came back to my office, confused and worried.

What I had failed to understand is that a thesis needs a frame. A frame has a breadth, and depth. The breadth characterizes the perspectives on the idea, and the depth the level of details. Say, your thesis is about dynamic updates. Technical feasibility and user adoption are two perspectives that belong to the breadth dimension. Implementation and formalization are different levels of details that belong to the depth dimension.

Different pieces of work have different ratios of breadth and depth. A position paper might have lots of breadth, but little depth. A paper proposing an optimization of an algorithm has little breadth, but lots of depth. For a thesis, you need a good ratio of both.

Completeness

Later on, I was once invited to prepare an extended version of a conference paper for a journal. I prepared a draft of the changes and went to see Oscar. I asked “Oscar, are my changes enough for an extended journal paper?”. He looked at me and said “For a journal paper, you must do what is necessary for your research to be complete“. I came back to my office, confused and worried.

What I had failed to understand is that maturity of research is not defined by how much has been done in terms of effort, but by the actual amount of speculation left. Research is an incremental process, and a piece of research is complete when all that was needed to support the claim (results or analysis) has been provided. Let’s imagine that you have an implementation of a dynamic update algorithm that you claim is efficient. You must show efficiency in speed and memory use for the research to be complete.

Thinking in terms of breadth, depth and completeness has become a simple technique in my toolkit of planning methods.

Thinking

Mind Blown

There’s lots of things to learn and know. Some are funny trivialities, some are joyful discoveries, some are intriguing theories, some are insightful lessons, … and some are mind-blowing revelations.

Here’s my top 10. Some of them still blow my mind!

Things are only impossible until they’re not. — Jean-Luc Picard

Have fun!

  1. Public-key cryptography
  2. Lamport’s bakery algorithm for mutual exclusion
  3. Meta-circular evaluation and homoiconicity
  4. Storing state with flip-flop
  5. 0.999… = 1
  6. Non-Euclidean Geometry
  7. Imaginary numbers
  8. The necessity of axiom of choice
  9. Fixed-point combinator
  10. Escape velocity
Thinking

Fun with iTune Shuffle and Probabilities

I recently tagged and imported all my mp3 into iTune. I noticed then that there were lots of albums that I had only partially listened to and I decided to use the feature “Party Shuffle” to listen to my library randomly and eventually hear all the songs.

After a couple of weeks, I observed that some songs would reappear in the playlist and were picked twice. Over the weeks the frequency of “re-entry” songs increased with the direct consequence that new music was played less and less. Even though I had already realized that it would not be possible to hear all the songs with approach, I was still surprised by the “re-entry” rate, which I would have intuitively expected to be much lower.

I turned to probability to better understand the situation.

Let’s n be the size of my library. After t songs played randomly, the probability that a given song was played at least once is:

P( song played at least once ) = t / n.

Absolutely not! This probability can be computed with 1 – probability that the song was never played. This gives:

      P( song played at least once ) = 1 – (( n-1 )/ n)  ^ t

More generally, the probability of a song having been played x times is given by the function

P( x ) = (1/n)^x * ( (n-1) / n )^(t-x) * C ( n, x  )

Where C(n,x) is the number of possible permutation. The expanded

P( x ) = (1/n)^x * ( (n-1) / n )^(t-x) *  n! / (n-x) ! x!

Note that the probability that the song was never played (x=0) is still (( n-1 )/ n)  ^ t.

After t songs, the sum P(0) + P(1) + … + P(t) = 1, which proves that the formula is correct.

The average number of songs played in the library after t songs, can be computed with

Avg. played

 = n * P( song played at least once )

= n * ( 1 – ((n-1)/n)^t ) = n – (n-1)^t  / n^(t-1)

The “re-entry” rate, or the probability of hearing a new song can be computed with (n- avg. played) / n which is equivalent to the probability that a given song was never played P(x=0).

The graph bellows shows the probability that a song was never played for a library of 500 songs, after 0, 50, 100, etc. songs. It’s interesting to notice that the probability of new songs fall below 50% after about 300 songs.

iTune_probability