Studying the Study: Different Kinds of Analysis Yield Contradictory Results For Illinois Wellness Study

August 22, 2018 in Uncategorized

This is Part II of a 2-part post. Check out Part I, Does The Illinois Workplace Wellness Study Say What Everyone Says It Says?.


A lot of questions remain about if and how these programs work. We have observed results for only the first year of our intervention. We are continuing to collect data to evaluate effects in the second and third years.

— Illinois Workplace Wellness Study website

The University of Illinois study rolls on as the researchers demonstrate they are eager to uncover the truth and not just confirm over-simplified pre-existing notions about whether wellness works or doesn’t work. Notice that they called their paper, “What Do Workplace Wellness Programs Do?” rather than using a title that declares the issue put to rest, like, say, “Workplace Wellness Doesn’t Work.”

Magnifying glass study a line chartUltimately, they may very well find that the Illinois program doesn’t yield the desired outcomes (potentially a real kick in the pants for Aetna, one of the researchers’ “collaborators”). Or that it does. If we knew for sure, there’d be no point in the study.

Personally, I’ve never had any reason to believe a wellness program would reduce an employer’s health care costs. But, so far, there isn’t anything in this study I’d cite to support that opinion.

 

Randomized Controlled Trials vs. Observational Studies

One of the most interesting things about the study is its design. It’s a randomized controlled trial (RCT) — a rare sighting in the world of wellness — and the researchers compared their findings to what they would’ve concluded if their data came from a an observational study (the kind that almost all our healthy lifestyle guidance is based on — from “physical activity is good for you” to “don’t inhale too much asbestos”), potentially explaining, as Aaron Carroll argues in his column, why some wellness studies show that wellness does work. Or, as one of the Illinois study’s principal investigators wrote to me in a private correspondence, “Methodology matters.”

Even when methods are about as good as can be, we probably should never trust a single study with high confidence…Take, for example, the randomized controlled trial (RCT). It’s reasonably considered the gold standard of social science methods. When you read the results of a well-conducted RCT, does that mean you can take them and run with it? Not so fast. They may not apply outside the population studied.

– Austin Frakt, co-editor with Aaron Carroll of the Incidental Economist, in Limitations: The Achilles Heel of Single-Study Relevance

No Reason to Expect Improvement

The Illinois Wellness study is and will continue to be important. It has the potential to deliver actionable insights into the value of incentives; the profile of employees who tend to engage in wellness programs; the types of programs that are and aren’t effective; and, ultimately, the behavioral, health, financial, and productivity outcomes we can expect from comparable programs.

For the university, the iThrive program is a good start — more thoroughly thought-out than most new programs. (Thanks to the study’s transparency, a large employer seeking to launch a wellness program could use the study’s published material to develop a program template — though I’d recommend skipping the incentives and the screenings, and adding longer-range plans for a more comprehensive strategy.) But that’s what it is — a start.

In a non-study situation, smart leaders of a “comprehensive” program, seeing that Year 1 activities had no effect on anything, would undertake a quality improvement process and make adjustments accordingly. After all, if there aren’t any behavior changes in Year 1 — and environment, culture, and work design aren’t even on the radar — there’s no reason to expect health, financial, or productivity improvements in the following years.

I admire the researchers’ refraining from sensational conclusions based on their Year 1 data. Now, it’s up to thought leaders with a media platform, and up to us — those responsible for applying research findings to our programs — to exercise the same restraint.

Does The Illinois Workplace Wellness Study Say What Everyone Says It Says?

August 20, 2018 in Employee Wellness Programs, Featured

Welcome to Reality signSeems like every month, the University of Illinois workplace wellness study re-enters the limelight, and earlier this month Aaron Carroll, MD really shoved it center stage in his New York Times piece, Workplace Wellness Programs Don’t Work Well. Why Some Studies Show Otherwise.

This was a randomized controlled study of an employee wellness program. To date, the study results have shown no improvement in health behaviors, health care costs, or productivity. To date.

You can read the full study paper published on the Bureau of Economic Research website. But if you’re not one to wade through a swamp of statistics, check out the study’s very own website for info, updates, and bar charts galore.

Does Feeling Valued Count?

Rather than cherry-picking the facts, allow me to just suggest questions to consider as you learn more about this study:

What does “doesn’t work” mean, anyway? Work to do what?

The study found that the number of program participants who believed their employer was committed to their health and safety increased significantly as an effect of the intervention. Is this important?

In the study paper, how many times do the researchers make the claim that has captured the imagination of Dr. Carroll and many others in the business and health care media, that “wellness doesn’t work”? (You can cheat by using your browser’s “Find” function. Or take a guess. It’s somewhere between -1 and +1.)

Was the study published in a peer-reviewed journal?

“I heavily favor peer-reviewed work.”

— Aaron Carroll, in The Power (and Weakness) of Peer Review, 2011.

How many employers, and how many different kinds of wellness strategies, were included in this study of the University of Illinois wellness program (called iThrive)?

Let’s say you’re running a program for a global manufacturing company or a tech start-up. How comparable is your employee population to the employees at University of Illinois?

A Comprehensive Wellness Program

iThrive is said to be a “comprehensive” wellness program. In my mind, a comprehensive wellness program might include some behavioral programs, cultural strategies, environmental strategies, and, most importantly, organizational strategies that promote healthy work.

Is this a comprehensive program? You be the judge. The core activies and strategies of iThrive:

  • Biometric health screenings
  • Health risk assessments
  • Participation incentives
  • Participation in “one of several activities in the fall and then again in the spring.” Activities included classes on chronic disease management; weight management; tai chi; physical fitness; financial wellness; healthy workplace habits; a tobacco cessation hotline; and an online, self-paced wellness challenge.”

A “Post-Intervention” Time Warp?

  • Screenings were conducted from August 15 to September 16, 2016.
  • Health risk assessment was conducted from September 8 to October 4.
  • Fall wellness activities were held October 10 to December 16.
  • Spring wellness activities were held January 30 to April 25, 2017.
  • “Post-intervention” healthcare utilization was measured for the period August 1, 2016 through July 31, 2017.

Thinking carefully about this timeline, what changes in healthcare utilization patterns would you expect during the first year of the program?

Keep these questions in mind. And I hope you’ll pose a lot more of your own when you read about future findings from this and other studies.


Check out Part II of this post, Studying the Study: Different Kinds of Analysis Yield Contradictory Results For Illinois Wellness Study.