Taleb describes the overconfidence of many “experts.” Some people have a motivation to display certainty. For example, auto mechanics always seemed to me to be 100% sure of their diagnosis (“It’s the electrical system”), then when they were wrong, it never would bother them a bit. Setting aside possible fradulence, I think they have a motivation to be certain, because we’re unlikely to follow their advice if they qualify it. In the other direction, academics like me perhaps have a motivation to overstate uncertainty, to avoid the potential loss in reputation from saying something stupid. But in practice, people seem to understate our uncertainty most of the time. Andrew Gelman, 2006

The key thoughts in the book

• Since we can’t control unpredictable events, we should accept uncertainty and seek to maximize our exposure to serendipity, as by putting ourselves in the way of new ideas.
• Since there is such danger in accepting conclusions based on too little information simply because they confirm our beliefs, we should try to remain aware in the present of what we are doing, paying attention to what actually happens and refraining as far as possible from imposing theories on our experience.
• We should recognize our poor record as a species in predicting the future, that we are much better at doing than knowing. Some things are more predictable than others: we are safe enough in expecting tomorrow’s sunrise to plan on breakfast. We can start noticing which situations are most susceptible to black swans, and when we encounter them, remember how little we truly know so our ignorance doesn’t lead us around by the nose.

© 2005 My new crush, Arlene Goldbard

Considering the fact that Taleb blatantly argues that many who consider themselves the rulers of the universe were in fact a group of lucky fools, it is inevitable that many will come away from it with a sense of anger and a refusal to believe it. I am therefore almost surprised that the book has not drawn harsher reviews than it has, for Taleb was certainly not seeking to make friends through the publication of it. I suspect that those who rate the book as poor fall into two general categories: those who were troubled by the thought that a considerable portion of their success may have resulted from luck, and those who are attached to their current views on the workings of the markets and are hostile to any new views on them. These two categories naturally overlap quite often. An important thing to remember is that even if you work very hard, not only are the outcomes of your projects the result, to varying extents, of chance, but chance also played a role in getting you to the position where you can work hard and actaully see it pay off. Considering the complexity of the world we live in, and the infinite forces that push and pull on our lives, this book is critical to anyone who desires an objective veiw of how things come to be…(anonymous reader review on Amazon.com)

Evidence-based management, what is the current definition?
Before you  start measuring things, you need to explain what you mean by evidence-based management. On this site, we use the following definition by Rousseau, Briner and Denyer, 2009: “Evidence-based management is about making decisions through the conscientious, explicit, and judicious use of four sources of information: practitioner expertise and judgment, evidence from the local context, a critical evaluation of the best available research evidence, and the perspectives of those people who might be affected by the decision.”

Levels of evidence
Just reading the definition, the pyramid with the levels of evidence (like the one in use by medicine) is not sufficient. Just putting a symbol D or level III evidence (indicating expert opinion) won’t help us. Most people don’t know what that means.

Rigor and relevance
If you look for systematic reviews in management in ABI/INFORM, you probably won’t find much. So, methodological rigor is very important, but is it still relevant to practice? I currently have the impression that the terms are mutually exclusive. But, that will hopefully change in the future. Even the fact if management is a profession is an active discussion. But there is also good news. At the recent evidence-based management caucus at the Academy of Management in Montreal, 5 people announced ’systematic reviews’. Maybe we can follow a few of these projects on this website.

A suggestion for appraisal

What type(s) of evidence is/are pre-dominantly used in the book?

Practitioner judgement and expertise (practitioner evidence)

Evidence from a local context| (situational evidence)

Critical appraisal of the best available research evidence (C.A.T.) (maybe indicating the level of evidence with a C or a D)

Perspectives from people affected by the decision (group ‘evidence’)

Target audience: beginner, intermediate or expert.

And probably a review of the book on this blog, but that will take some time and effort.

Note: this is probably not enough, but I would like to hear some suggestions for improving the appraisal.

Some of the books/articles with “evidence-based” in their titles hardly seem to be about evidence at all: Maybe authors do this because it’s trendy, and might make people look (which it does, in my case, since I track Google alerts for “evidence-based”). Call me cynical, but I suspect in many cases you could remove the references to “evidence-based” without changing the gist. (Rarely does anyone go to the trouble of defining “evidence”. This makes me want to pound out The Evidence-Based Manifesto …and then my caffeine buzz wears off.)

Now, let’s look at some books that do talk about evidence. First up is Becoming the Evidence-Based Manager: Making the Science of Management Work for You, by Gary P. Latham. He says the book was “written to underscore the scientific aspect of effective management – what is called evidence-based management – in an artful way.” His intent is to “share management techniques that have been proven by valid and reliable research studies to work.”

Latham’s introduction says “The art of management can seldom be taught. The science of management can be taught.” Good point. His focus is on human resources: Hiring top performers, motivating people, etc. He doesn’t delve into evidence-based methods for business strategy, decision-making, etc. This is about organizational psychology. (Here’s Latham on YouTube.)

A gentle introduction. One thing I like about this book is that doesn’t just preach to the converted. Latham addresses an audience that hasn’t focused on “evidence-based” before, So this could be useful to a new manager, or to anyone who hasn’t looked at management scientifically. The presentation isn’t too academic or complicated.

Have some fish: It’s good for you. Not everyone needs to be a philosopher, and not every book needs to ‘meta’ in scope (this one isn’t). Generally speaking, the book gives people fish (telling them about management techniques supported by evidence), rather than teaching them how to fish (i.e., how to gather up their own evidence).

For instance, Chapter 1 is about how to “Use the Right Tools to Hire High-Performing Employees.” The book provides brief case studies: For example, a case where a manager replaced the traditional hiring process with situational interviews, “a method proven to be effective at selecting high performers” (page 143).

Can we look that up? Many references to ‘the evidence’ aren’t supported by a footnote or link (such as statements like “The evidence also shows that …” [page 96]). Latham seems to be addressing an audience that isn’t inclined (or simply lacks the time) to scrutinize the evidence: People who want the evidence to be condensed and presented to them as management advice. Nothing wrong with that. (Though I still would like to see more specific references to the underlying research, and I believe the author could do that without weighting down his discussions.)

The takeaway. Lots of people could use an introduction like this. Hopefully it will give more managers the confidence to ask “Before we move forward, what does the evidence say?” (Eventually leading to more rigorous evaluation of the evidence.)

At the other end of the spectrum is Allen Rubin’s Practitioner’s Guide to Using Research for Evidence-Based Practice. The book “aims to provide human services practitioners what they need to know about various research designs and methods so that when engaging in the EBP process, they can determine which interventions, programs, policies, and assessment tools are supported by the best evidence.”

It’s a process. I especially like how Rubin describes evidence-based practice as a process, not a thing. That mind-set needs to prevail. The book emphasizes that:

- EBP is a process that includes locating and appraising credible evidence as a part of practice decisions.

- EBP is way to designate certain interventions as empirically supported under certain conditions.

Other things I like: The book addresses both quantitative and qualitative studies. And Rubin reminds us that we are seeking answers to EBP questions, and that each one must suit our objective.

This is heavy-duty stuff, but the book is well-written, using language like “recall back in chapter 3 we saw how systematic reviews reside above experiments on the evidentiary hierarchy for answering questions about effectiveness.” And Rubin talks about assessing a research study in relatively straightforward English, and provides practical advice about how to evaluate a study (pages 104-105).

Now we’re getting meta. Evidence about the evidence. Rubin suggests a hierarchy ranking various types of evidence. Level 1 (the best evidence) is systematic review and meta-analysis (page 52): A la Cochrane levels of medical evidence (I, II…).

Chapter 3 talks about “Research Hierarchies: More than one type of hierarchy for more than one type of EBP question.” Chapter 4 is “Criteria for inferring Ineffectiveness.” Chapter 8 covers “Critically appraising systematic reviews and meta-analyses.” And Chapter 9 is “Critically appraising non-experimental quantitative studies.” Good things to know, but not for the novice.

The takeaway. Good for people who think about research design and evidence the way a PhD would – with or without a doctorate. Clearly this kind of effort is appropriate for ‘big’ interventions, or where there’s lots of repeatable stuff to analyze – but it’s too much for making decisions on a smaller scale, or in situations where innovative things are being tried and the available evidence is not very applicable.

Is the goal to be performance-based *and* evidence-based? I’ve often thought about the similarities and differences between so-called ‘performance management’ and ‘evidence-based management’. And I’ve wondered that if these approaches were merged, maybe we’d really have something. (Allow me to digress before we move on to evidence-based book #3.) When narrowly defined, performance management “includes activities to ensure that goals are consistently being met in an effective and efficient manner.” It’s about using metrics, key performance indicators (KPIs), dashboards, etc. to monitor and improve performance. And of course evidence-based management is about making decisions based on the current, best evidence about what works. So I think of performance data as one type of evidence that fits into a wider program to achieve evidence-based management.

I raised the subject of performance management because its concepts are prevalent in our next book, The Intelligent Company: Five Steps to Success with Evidence-Based Management, by Bernard Marr, a consultant on organizational performance. He’s got a lot to say about using data to improve performance, and makes specific recommendations. It’s a serious effort, and Marr cheerfully provides citations and references (in a section at the end of the book). [Note: On amazon there's a downloadable mini-version priced at $9.95 US (40 pages). It's a 2009 article from Financial Management.]

Marr opens by making the connection between evidence-based medicine and business management, explaining the need to identify an important question, gather data, analyze, etc. But a lot of the discussion focuses on “evidence” that can be displayed on management dashboards or presented in reports – high-volume data about specific operational parameters. That’s indeed important, but overlooks useful evidence about business strategy, product management, human resource practices, etc.

The focus here is the data you have – or can go get yourself – and key performance indicators (KPIs) derived from that data: It’s not as much about evidence in the external literature, or systematic reviews. But Marr does a thorough job within his focus area: There’s lots of rigor in his approach, such as testing and proving ideas. And I’m glad that he discusses how best to present information and create reports. He references Stephen Few, a well-known dashboard designer in the performance management/business intelligence field.

Intelligent, you say? The book’s blurb is too hype-y for my taste, breathlessly claiming that “Today’s most successful companies are Intelligent Companies that use the best available data to inform their decision making. This is called Evidence-Based Management and is one of the fastest growing business trends of our times.” (What’s an “Intelligent Company”? And why is the phrase capitalized? This is just consultant-speak.)

The ‘five steps to more intelligent decision making’ are:

1. “More intelligent strategies – by identifying strategic priorities and agreeing your real information needs.
2. More intelligent data – by creating relevant and meaningful performance indicators and qualitative management information liked back to your strategic information needs.
3. More intelligent insights – by using good evidence to test and prove ideas and by analysing the data to gain robust and reliable insights.
4. More intelligent communication – by creating engaging management information packs and dashboards that provide the essential information, packaged in an easy-to-read way.
5. More intelligent decision making – by fostering an evidence-based culture of turning information into actionable knowledge and real decisions.”

Here is Marr’s EbM model. Since he focuses on data, it’s appropriate that he includes enabling technology in the process.

The takeaway. Solid advice. But lots has already been written about data-driven management, performance management, business intelligence, etc. I’m not sure there’s much new here, though the book does a respectable job of presenting it.

Is “evidence-based” meaningless terminology?
Would the point of Marr’s book be substantially different if it instead described “data-driven decision-making”? Would Rubin’s or Latham’s books be the same if they talked about “science-based” instead? And would other works be unaffected if their references to EBM were replaced with something else? Here are my thoughts on “evidence-based ___”:

• We need to talk about scientific methods, not just data, and a good way to do that is to use terminology that addresses evidence and evidence-based actions specifically.
• We need to say what we mean by “evidence”.and “current best” evidence. And that should include evidence from sources both internal and external to an organization.
• We need to discuss the role of technology in supporting “evidence-based ____”. Whether we’re developing, presenting, or distributing evidence, technology is a key variable in the equation: Evidence needs to be more transparent and available on web pages, in documents, and in mobile apps.
• We need to show references when we cite evidence and research. There are ways to do this without making everything look and sound like a textbook.

Where are you on the evidence-based hierarchy? Are you a novice manager, new to the use of science in management? Then books such as Latham’s Becoming the Evidence-Based Manager: Making the Science of Management Work for You might be a good choice. If you’re more interested in quantitative decision-making based on performance data, try Marr’s The Intelligent Company: Five Steps to Success with Evidence-Based Management. If you are up to the task of experimental research design and meta-analysis, then you will enjoy Rubin’s Practitioner’s Guide to Using Research for Evidence-Based Practice.

1. UNDERSTAND WHAT EBMgt MEANS.

Take a look at accounts of what EBMgt is.  See Briner, Denyer, & Rousseau (2009) Evidence-based management: Construct clean-up time? in the Collaborative’s archives. (Practice explaining what EBMgt is to your colleagues. Their questions will help develop your understanding of what it is and isn’t.)

2. MAKE YOUR DECISIONS MORE EXPLICIT.

Analyze a recent decision you have made (alone or with colleagues).

1. What steps did you take at each stage? What information did you use?
2. Did some information influence the decision more than others?
3. Is some evidence missing?
4. How close or how distant are your decisions to the features of EBMgt?

3. ASK “What’s the evidence for this?” WHEN ASSERTIONS ARE MADE WITHOUT CLEAR LINK TO EVIDENCE.

Develop a questioning mindset toward assumptions stated as facts either by you or others.  This approach can inform your conversations and deliberations. It helps shift the mindsets of people you work with, opening up inquiry for more facts and knowledge.

4. DO DIRECTED READING ON SCIENTIFIC EVIDENCE RELATED TO YOUR WORK.

Check out the business section of your local bookstore for books that cite research articles as a basis for their ideas. Avoid books without citations or linked only to pop writings, so-called experts, or news magazines.

Or, go to your local library and search on-line sources of scientific articles. For example, search ABInform or Web of Science for articles that are peer-reviewed (this typically means that the quality of articles has been independently approved by a panel of scientists).  Be sure to check the box that says ‘peer-reviewed’ when you search.

5. MAKE FRIENDS WITH PEOPLE WHO KNOW HOW TO FIND SCIENTIFIC EVIDENCE.

People who know where to find evidence have what is called POINTER KNOWLEDGE—business or social science librarians, college professors, or researchers and evidence-based consultants in your areas of interest. Information searches are easier when you go through people before going to the internet or library.

6. SET PERSONAL LEARNING GOALS TO IMPROVE YOUR DECISION MAKING PROCESSES.

Examples:

a. Regularly integrate organizational facts, potential stakeholder effects, and scientific evidence into my decisions.

b. Make After-Action Reviews of decisions with my colleagues and subordinates to assess strengths and needed improvements.

c. Develop a cognitive tool (e.g., checklist, decision model such as Yates’) to follow when making any important decision.

d. Regularly do directed readings of science-based publications on on-going professional issues (e.g., talent management, market trends, problem-solving processes)

You are on your way to developing your professional acumen and changing your conversation with your colleagues.   Go to Collaborative’s archives for a more details on this post.