Is “evidence-based” meaningless? Three books illustrate the dilemma of using this terminology.

[This is a cross-post from Evidence Soup (not all graphics are included here).] Today we’re looking at three books on “evidence-based ____” , each concerning management in the public or private sector. Together, they demonstrate the wide range of possibilities for evidence-based action. But they also illustrate why it’s difficult for people to point to something and say “Now I see what this EB stuff is all about!”
How does this work, again?

Some of the books/articles with “evidence-based” in their titles hardly seem to be about evidence at all: Maybe authors do this because it’s trendy, and might make people look (which it does, in my case, since I track Google alerts for “evidence-based”). Call me cynical, but I suspect in many cases you could remove the references to “evidence-based” without changing the gist. (Rarely does anyone go to the trouble of defining “evidence”. This makes me want to pound out The Evidence-Based Manifesto …and then my caffeine buzz wears off.)

Now, let’s look at some books that do talk about evidence. First up is Becoming the Evidence-Based Manager: Making the Science of Management Work for You, by Gary P. Latham. He says the book was “written to underscore the scientific aspect of effective management – what is called evidence-based management – in an artful way.” His intent is to “share management techniques that have been proven by valid and reliable research studies to work.”

Latham’s introduction says “The art of management can seldom be taught. The science of management can be taught.” Good point. His focus is on human resources: Hiring top performers, motivating people, etc. He doesn’t delve into evidence-based methods for business strategy, decision-making, etc. This is about organizational psychology. (Here’s Latham on YouTube.)

A gentle introduction. One thing I like about this book is that doesn’t just preach to the converted. Latham addresses an audience that hasn’t focused on “evidence-based” before, So this could be useful to a new manager, or to anyone who hasn’t looked at management scientifically. The presentation isn’t too academic or complicated.

Have some fish: It’s good for you. Not everyone needs to be a philosopher, and not every book needs to ‘meta’ in scope (this one isn’t). Generally speaking, the book gives people fish (telling them about management techniques supported by evidence), rather than teaching them how to fish (i.e., how to gather up their own evidence).

For instance, Chapter 1 is about how to “Use the Right Tools to Hire High-Performing Employees.” The book provides brief case studies: For example, a case where a manager replaced the traditional hiring process with situational interviews, “a method proven to be effective at selecting high performers” (page 143).

Can we look that up? Many references to ‘the evidence’ aren’t supported by a footnote or link (such as statements like “The evidence also shows that …” [page 96]). Latham seems to be addressing an audience that isn’t inclined (or simply lacks the time) to scrutinize the evidence: People who want the evidence to be condensed and presented to them as management advice. Nothing wrong with that. (Though I still would like to see more specific references to the underlying research, and I believe the author could do that without weighting down his discussions.)

The takeaway. Lots of people could use an introduction like this. Hopefully it will give more managers the confidence to ask “Before we move forward, what does the evidence say?” (Eventually leading to more rigorous evaluation of the evidence.)

At the other end of the spectrum is Allen Rubin’s Practitioner’s Guide to Using Research for Evidence-Based Practice. The book “aims to provide human services practitioners what they need to know about various research designs and methods so that when engaging in the EBP process, they can determine which interventions, programs, policies, and assessment tools are supported by the best evidence.”

It’s a process. I especially like how Rubin describes evidence-based practice as a process, not a thing. That mind-set needs to prevail. The book emphasizes that:

- EBP is a process that includes locating and appraising credible evidence as a part of practice decisions.

- EBP is way to designate certain interventions as empirically supported under certain conditions.

Other things I like: The book addresses both quantitative and qualitative studies. And Rubin reminds us that we are seeking answers to EBP questions, and that each one must suit our objective.

This is heavy-duty stuff, but the book is well-written, using language like “recall back in chapter 3 we saw how systematic reviews reside above experiments on the evidentiary hierarchy for answering questions about effectiveness.” And Rubin talks about assessing a research study in relatively straightforward English, and provides practical advice about how to evaluate a study (pages 104-105).

Now we’re getting meta. Evidence about the evidence. Rubin suggests a hierarchy ranking various types of evidence. Level 1 (the best evidence) is systematic review and meta-analysis (page 52): A la Cochrane levels of medical evidence (I, II…).

Chapter 3 talks about “Research Hierarchies: More than one type of hierarchy for more than one type of EBP question.” Chapter 4 is “Criteria for inferring Ineffectiveness.” Chapter 8 covers “Critically appraising systematic reviews and meta-analyses.” And Chapter 9 is “Critically appraising non-experimental quantitative studies.” Good things to know, but not for the novice.

The takeaway. Good for people who think about research design and evidence the way a PhD would – with or without a doctorate. Clearly this kind of effort is appropriate for ‘big’ interventions, or where there’s lots of repeatable stuff to analyze – but it’s too much for making decisions on a smaller scale, or in situations where innovative things are being tried and the available evidence is not very applicable.

Is the goal to be performance-based *and* evidence-based? I’ve often thought about the similarities and differences between so-called ‘performance management’ and ‘evidence-based management’. And I’ve wondered that if these approaches were merged, maybe we’d really have something. (Allow me to digress before we move on to evidence-based book #3.) When narrowly defined, performance management “includes activities to ensure that goals are consistently being met in an effective and efficient manner.” It’s about using metrics, key performance indicators (KPIs), dashboards, etc. to monitor and improve performance. And of course evidence-based management is about making decisions based on the current, best evidence about what works. So I think of performance data as one type of evidence that fits into a wider program to achieve evidence-based management.

I raised the subject of performance management because its concepts are prevalent in our next book, The Intelligent Company: Five Steps to Success with Evidence-Based Management, by Bernard Marr, a consultant on organizational performance. He’s got a lot to say about using data to improve performance, and makes specific recommendations. It’s a serious effort, and Marr cheerfully provides citations and references (in a section at the end of the book). [Note: On amazon there's a downloadable mini-version priced at $9.95 US (40 pages). It's a 2009 article from Financial Management.]

Marr opens by making the connection between evidence-based medicine and business management, explaining the need to identify an important question, gather data, analyze, etc. But a lot of the discussion focuses on “evidence” that can be displayed on management dashboards or presented in reports – high-volume data about specific operational parameters. That’s indeed important, but overlooks useful evidence about business strategy, product management, human resource practices, etc.

The focus here is the data you have – or can go get yourself – and key performance indicators (KPIs) derived from that data: It’s not as much about evidence in the external literature, or systematic reviews. But Marr does a thorough job within his focus area: There’s lots of rigor in his approach, such as testing and proving ideas. And I’m glad that he discusses how best to present information and create reports. He references Stephen Few, a well-known dashboard designer in the performance management/business intelligence field.

Intelligent, you say? The book’s blurb is too hype-y for my taste, breathlessly claiming that “Today’s most successful companies are Intelligent Companies that use the best available data to inform their decision making. This is called Evidence-Based Management and is one of the fastest growing business trends of our times.” (What’s an “Intelligent Company”? And why is the phrase capitalized? This is just consultant-speak.)

The ‘five steps to more intelligent decision making’ are:

1. “More intelligent strategies – by identifying strategic priorities and agreeing your real information needs.
2. More intelligent data – by creating relevant and meaningful performance indicators and qualitative management information liked back to your strategic information needs.
3. More intelligent insights – by using good evidence to test and prove ideas and by analysing the data to gain robust and reliable insights.
4. More intelligent communication – by creating engaging management information packs and dashboards that provide the essential information, packaged in an easy-to-read way.
5. More intelligent decision making – by fostering an evidence-based culture of turning information into actionable knowledge and real decisions.”

Here is Marr’s EbM model. Since he focuses on data, it’s appropriate that he includes enabling technology in the process.

The takeaway. Solid advice. But lots has already been written about data-driven management, performance management, business intelligence, etc. I’m not sure there’s much new here, though the book does a respectable job of presenting it.

Is “evidence-based” meaningless terminology?
Would the point of Marr’s book be substantially different if it instead described “data-driven decision-making”? Would Rubin’s or Latham’s books be the same if they talked about “science-based” instead? And would other works be unaffected if their references to EBM were replaced with something else? Here are my thoughts on “evidence-based ___”:

• We need to talk about scientific methods, not just data, and a good way to do that is to use terminology that addresses evidence and evidence-based actions specifically.
• We need to say what we mean by “evidence”.and “current best” evidence. And that should include evidence from sources both internal and external to an organization.
• We need to discuss the role of technology in supporting “evidence-based ____”. Whether we’re developing, presenting, or distributing evidence, technology is a key variable in the equation: Evidence needs to be more transparent and available on web pages, in documents, and in mobile apps.
• We need to show references when we cite evidence and research. There are ways to do this without making everything look and sound like a textbook.

Where are you on the evidence-based hierarchy? Are you a novice manager, new to the use of science in management? Then books such as Latham’s Becoming the Evidence-Based Manager: Making the Science of Management Work for You might be a good choice. If you’re more interested in quantitative decision-making based on performance data, try Marr’s The Intelligent Company: Five Steps to Success with Evidence-Based Management. If you are up to the task of experimental research design and meta-analysis, then you will enjoy Rubin’s Practitioner’s Guide to Using Research for Evidence-Based Practice.

[carousel keywords="evidence-based management" category="Books" browseNode="2675"]