In the era of big data, it is easy to imagine that we have all the information we need to make good decisions. But in fact the data we have are never complete, and may be only the tip of the iceberg. Just as much of the universe is composed of dark matter, invisible to us but nonetheless present, the universe of information is full of dark data that we overlook at our peril. In Dark Data, data expert David Hand takes us on a fascinating and enlightening journey into the world of the data we don’t see.
Dark Data explores the many ways in which we can be blind to missing data and how that can lead us to conclusions and actions that are mistaken, dangerous, or even disastrous. Examining a wealth of real-life examples, from the Challenger shuttle explosion to complex financial frauds, Hand gives us a practical taxonomy of the types of dark data that exist and the situations in which they can arise, so that we can learn to recognize and control for them. In doing so, he teaches us not only to be alert to the problems presented by the things we don’t know, but also shows how dark data can be used to our advantage, leading to greater understanding and better decisions.
Today, we all make decisions using data. Dark Data shows us all how to reduce the risk of making bad ones.
David J. Hand is emeritus professor of mathematics and senior research investigator at Imperial College London, a former president of the Royal Statistical Society, and a fellow of the British Academy. His many previous books include The Improbability Principle, Measurement: A Very Short Introduction, Statistics: A Very Short Introduction, and Principles of Data Mining.
"[A] penetrating study of missing (‘dark’) data and its impacts on decisions—skewing stats, enabling fraud, embedding inequity and triggering preventable catastrophes. Advocating ‘data science judo,’ Hand offers expert training, from recognizing when facts are being cherry-picked to designing randomized trials. A book illuminating shadowed corners in science, medicine and policy."—Barbara Kiser, Nature
"A tour de force. . . . Hand is a good and able guide to take us through the many aspects of dark data that are potentially skewing our understanding of real world observations and potential scientific breakthroughs. He writes in an accessible and understandable way too."—Simon Cocking, Irish Tech News
"Well-written and accessible."—Tim Harford, Undercover Economist
"You need to read [Dark Data], and be convinced by David’s reasoning and his examples of cases in which unseen or unreported data play a critical and sometimes even a fatal role. You are likely to walk away with the feeling that the term dark data is indeed a very effective one to arouse both curiosity and suspicion, mixed with happiness that finally a great term was coined by a statistician—and sadness that the statistician is not you."—Xiao-Li Meng, IMS Bulletin
"An exploration of a major problem in data analysis with an attempt of classification, analysing causes, mechanisms, and to some extent also suggest mitigations."—Adhemar Bultheel, European Mathematical Society
"An excellent guide to the many reasons for caution in interpreting data."—Diane Coyle, Enlightened Economist
"When we make decisions in our personal and professional lives, we typically start with some form of data. The very word 'data' derives from the Latin meaning 'something given.' But who gave it? Where is it from? Should I accept it at face value? Opening our eyes to the pitfalls of taking 'something given' for granted, this insightful book should be required reading for everyone in an age when 'fake news' and the explosion of data go hand in hand."—Adrian Smith, director and chief executive of The Alan Turing Institute
"David Hand shines a bright light onto the dark corners of statistics. This is a learned book but a witty, readable, and important one. I learned a lot and so will you."—Tim Harford, author of Fifty Inventions That Shaped the Modern Economy and presenter of the BBC series More or Less
"It is hard to think of anyone having anything at all to do with data-driven decisions who couldn't benefit from reading this book. David Hand effortlessly guides the reader through the many pitfalls of dark data."—Arno Siebes, Universiteit Utrecht
"This unique and much-needed book provides an accessible guide to dark data at a time when general awareness of the phenomenon is declining."—Geert Molenberghs, Universiteit Hasselt and KU Leuven
