Data Sight and Blindness

How can we avoid being blinded by data overload?

Image by Gerd Altmann from Pixabay

In the 1999 film At First Sight, Val Kilmer plays a man who having been blind for most of his life, gains sight for effectively the first time in his life and struggles to deal with his new found ability and the overflow of information. The movie didn't rate very well but is an apt metaphor for the world of data we find ourselves in 20 years later. Data shines a light on a number of things that we do by instinct, and the initial signs are that much like Val Kilmer's character, we aren't dealing with it too well. In any number of contexts, when faced with data, we individually and collectively still lean towards experience, instinct, or even hearsay. I see this in a vast array of domains and contexts. Ranging from sports fandom, to health and well being, to politics and choices, and of course, in our working lives. The old and gendered joke about why men don't ask for directions is really prompting the question of why we don't use data when it's available!

Val Kilmer & Mira Sorveno, At First Sight

Look around you, the chances are in most spheres of life, you get into discussions and debates which should be easy to settle with the right data. Yet, the arguments are interminable. That one friend who will take a contrary position on some economic policy argument, and argue all evening, or the family member who has acquired a new political colour and wants to convert everybody. Or even the bewildering array of (often contradictory) advice you might get should you disclose a health problem, or suggest that you might want to lose weight or improve your cardiovascular fitness. I'm sure all of these are situations you've found yourself in at some point or the other. Perhaps the work version of this is the one we are actually most familiar with - arguments that should be settled by data but often end up being opinion exchanges.

How much money will the UK actually gain or lose after Brexit? What is the actual impact of electric vehicles or of meat substitutes on the environment? Should you eat bread if you want to lower your cholesterol? What are the top reasons for employee churn? What are the most critical buying influences for your customers? What's the true value of the latest cryptocurrency in the news? For many of these questions, we don't know where to get the data and whether such data is reliable.  But even when we do have the data, we often ignore it.  We forward messages without checking their authenticity, we read 'inside stories' about politics or sports, wanting to believe what we like to hear. Most of us consistently make bad food choices, or lifestyle choices in the face of highly available data. There is a history to this. Data is relatively new, data culture is even newer. We have spent many centuries being hard wired to make important decisions on very limited data.  Decisions that were critical to our survival as individuals or species. So data culture is actually learned behaviour, in a way. Especially when it contradicts our received wisdom or heuristic answers.

I'm listening to the audiobook "A Demon Haunted World: Science as a Candle in the Dark" co-authored by Carl Sagan, and Ann Druyan, his wife, and in the opening lines it becomes clear that the culture of data is in a way the culture of knowledge and science. Every scientific progress we've made and every previously held erroneous belief we've overturned has been done through data and the insistence on data over dogma.  But somewhere along the way, even as science races ahead solving the deepest mysteries of nature from gene folding, to the black hole in the Milky Way, it feels like society has fallen behind, unable to adequately arm itself with data driven critical thinking, and consequently the schism between society and science is growing.

There are 3 distinct problems that create a compounding effect.

Cognitive Biases

Arguably the easiest (or the hardest) problem to fix is the set of our cognitive biases and the challenge of rewiring our brains' decision systems. Of the many dozens of biases, I often find that confirmation bias is the most common. When faced with overload of data sources, all dispensing 'facts' many of which are contradictory, we tend to go with our pre-held beliefs. This could be addressed if there was a reasonably high number of authentic data sources. Which brings me to the second problem...

Data overload and obfuscation

There isn't of course one-authority. There is apparently no such thing as 'without bias'. For any opinion, there is always a counter opinion, and for every fact, a counter fact. In the closing scenes of the movie 'The Big Short', based on Michael Lewis's book about the sub prime mortgage driven financial crash of 2008, Dr Burry, played by Christian Bale says: "People want an authority to tell them how to value things, but they choose this authority not based on facts or results. They choose it because it seems authoritative and familiar. And I am not, nor ever have been, "familiar." But Dr Burry was right in his analysis. He was among the first person who saw foresaw the crash, and he had the data, but nobody believed him. There were too many other familiar faces arguing the opposite. This plays out in every aspect of our lives. We should be able to tell knowledge authority from charlatans and misguided messiahs, given the free access to knowledge and data. But that's where our third problem kicks in.

People want an authority to tell them how to value things

Data sophistication and problem correlation

The world is insanely complex. The kinds of problems we are debating are hugely sophisticated and even experts may be at loggerheads. If you haven't understood RNA, and mRNA, then how can you defend somebody telling you that it's all a conspiracy to spy on you? And that's before we come to Decentralised Finance, distributed Ledgers, or the metaverse, machine learning, and neural networks. Or wars, global supply chain choke points, or the shifting sands of left and right politics? (Or even...was that player really offside?) Many of these are emergent problems, the data about them is still being gathered. And we are a long way away from establishing any kind of correlations. Hence we are overloaded with data but often simultaneously under-nourished with information and facts. Which is when our brains give up and resort to our biases.

But even when the problems aren’t that complicated, the culture of counter-facts means that we have given legitimacy to every opinion, and biases and herd mentality amplify the eddy currents in logical thinking. The past couple of weeks has signalled a sharp drop in technology forecasts and share prices (Amazon, Shopify, Etsy, Ebay) as people go back to pre-pandemic behaviours. But was it that hard to predict, that people wouldn’t continue exactly how they were living their lives when the circumstances changed, and the cost of living soared? Of course the investors would like them to; but for that, see confirmation bias, above. The fact that this correction is a surprise is a bit surprising. (And by the way video technology now tells us exactly when a player is offside, even by a toenail, but the arguments persist).

There are correspondingly three simple rules to beat this. Simple to say, harder to do:

1. We all need to invest time and effort to understand the things that really matter to us, or invest time in finding the right authority

2. Being passive in this environment isn't good enough. We have to actively fight misinformation in our own fields.

3. We need to be vigilant about our own biases and reflect on our decisions and choices.

As I said, easier to say than do. But remember, there’s probably relevant data out there, if you have the energy to find it. Good luck with the journey and may the force be with you.

Reading This Week

Climate change: This scary article says that fossil fuel companies are betting that climate goals will be missed and are making plans accordingly. (The Guardian)

Air Taxis: Are air taxis ready for prime time? This report from TNMT (Transport and Mobility Tech) suggests that automotive, technology, and aviation companies, and start ups are building plans for flying taxis, with over $1bn VC funding in 2020. The company to watch according to the report is Joby Aviation (TNMT) Also read Azeem Azhar’s view here. (Exponential View)

Crypto Death Spiral: You may have seen the carnage this week in crypto prices. This is a deeper look at the story of Stable Coin and Luna. When one cryptocurrency has an algorithmic price, another has an artificially fixed price, and you try and have a fixed exchange rate between them, that is not a sustainable system, and can lead to the death spiral. (Bloomberg)

Quantified Self: The Economist Tech Quarterly has some interesting takes on the quantified self - spanning fitness, wellness, and health. (Economist - Subscription required)

Fintech: Klarna, the BNPL start up has launched an open banking brand with an API strategy and new division - Klarna Kosma. (AltFi)

PS: still trying to bring some regularity to the writing - this year has been crazy so far. See you soon.