TAMS 5

Critically Reading News Related to Health Care

When reading stories about new methods for curing a disease or a new medication, one has to carefully
and critically read to find out whether what is stated is really true. The questions below will help you to
do so – these are the questions you should ask yourself before you believe the “hype” about the new
wonder drug.

Each question has a little graphic that is a warning, together with the issue that may arise. This is
followed by more details to help you assess whether there is an issue.

1. Does the story adequately discuss the costs of the intervention?

Health care isn’t free. News stories don’t help
patients and consumers if they don’t tackle costs.

When rating a story on this criterion, check whether the story discusses the costs of the intervention,
whether it’s a new device, medication, surgery, exercise fad, therapy or any of the myriad treatments
being studied. It’s not good enough to say, “The cost is much lower than the invasive procedures.”
What is that cost? How much lower? Also, do insurance policies typically cover the intervention?
Might there be other expenses (psychologists, dietitians, required follow-up visits or tests, etc.) that
increase the burden on patients? If they cannot be quantified, they should at least be mentioned.

For early trials or preliminary research, it may be difficult to estimate costs of an experimental
approach early in its development. But a news story can at least cite costs of existing alternatives. If
the new approach is comparable to other approaches then the cost of the alternative methods could
be cited. Bottom line: If it’s not too early to talk about benefits of a new intervention, it’s not too
early to talk about costs.

We also know that some interventions may cost more in the short term, and yet are cost-effective in
the long term because they reduce the need for expensive care down the road. While these longer-
term costs may be difficult or impossible to pin down, good stories make a reasonable attempt to
discuss the implications.

2. Does the story adequately quantify the benefits of the treatment/test/procedure?

New healthcare treatments should work well.
When reading a story, people want to know: How
effective is the intervention? Do the numbers back
it up?

News stories should explain what the researchers measured, in numerical terms. And to use absolute
numbers whenever possible. Many news stories tell us how wonderful a new treatment, test, product
or procedure may be. Few provide helpful numbers to back that up.

Ideally, the story should also explain whether the findings make an actual difference in people’s
lives. If a study says a new drug improved function by 30% in MS patients, what does that mean?
How was that measured? What would an MS patient want to know?

Also, research often isn’t about an actual improvement in health or quality of life, it’s instead
centered around surrogate outcomes, such as changes in blood test scores, or tumor shrinkage. These
endpoints can be useful for researchers (and biotech investors), but surrogate outcomes
do not automatically equal living better or longer. Readers need to know that.

If a story is about preclinical research like a mouse or monkey study, it must point out that
researchers have no guarantee that the intervention will provide the same benefit to people.

And, stories that rely too heavily on patient anecdotes may be unrepresentative of the true benefit.
The same goes for unchallenged exaggerated quotes (“this is a real game-
changer/breakthrough/cure” etc.), which can throw a story out of balance and overshadow the
statistics. When a story is top heavy with personal stories, it makes it hard for readers to sustain their
critical thinking when (if) they get to information that is quantitative. If you hear glowing patient
anecdotes about how well something worked, always ask yourself if that was a representative
example.

3. Does the story adequately explain/quantify the harms of the intervention?

All healthcare interventions carry potential harms.
When reading a news story about a new treatment,
readers should find out what is known about the
harms.

When a story is about a treatment (new drug, surgery, therapy lifestyle change, etc) or screening test,
is should include a discussion of harms and side effects, as well as any measured “adverse events” in
a study.

There are tradeoffs involved in any health care decision. No matter what you choose to do, you stand
to gain something and you stand to lose something. All treatments have potential harms, side effects,
and complications. Also, there is no such thing as a simple blood or screening test. They carry the
risk of a false-positive or false-negative result, which can result in missed diagnosis or
overtreatment, and anxiety.

Ideally, stories discuss both frequency of side effects and severity of side effects, and discuss both
major and minor side effects. What might be minor to one person could be major to another.

Preliminary animal studies or otherwise early research: It’s also important to remember that we
often have no idea what the true impact of a drug is until it’s been used in a large number of people
(and not mice and not cells in a lab), so the story should point out that some harms may be
unknown–especially when the research is far from ready for prime time.

Many stories emphasize or exaggerate potential benefits while minimizing or completely ignoring
potential harms. They may:

• Fail to mention or quantify potential harms.
• Fail to describe the severity of potential harms.
• Fail to account for “minor” side effects that could have a significant impact on a patient’s life.
• Rely too heavily on a patient anecdote about safety or on a researcher’s comment that an

approach “appears to be safe” – without supporting data.

4. Does the story seem to grasp the quality of the evidence?

Ideally, healthcare interventions are subjected to
rigorous testing to prove they work. When reading
about a new intervention, device or diagnostic tool,
people should come away with a sense of how rigorous
the evidence is for the intervention.

The story should include enough details about the study and how it fits into the hierarchy of
evidence. As in: Was it an animal study? An observational study? A small, safety trial? A large,
randomized controlled trial? A meta-analysis or systematic review? A recommendation from a task
force that reviews evidence? And regardless the type of evidence, what was the quality of the
evidence?

Not all studies are equal. Not all evidence is bullet-proof. There is not a certainty or a finality to
everything that appears in the New England Journal of Medicine or any other journal. And just
because it’s a study – or just because it’s published somewhere – the story is not over. That’s not the
way science works.Yet many stories:

• Present anecdotes as evidence of a treatment’s benefits – rather than as a single illustration of its
use.

• Leave out study limitations – even though all studies carry limitations.
• Fail to caution readers about interpretation of uncontrolled data.
• Fail to explain if a primary outcome is a surrogate marker1 or fail to caution readers/viewers

about extrapolating this to health outcomes.
• Fail to point out the limited peer review that may have taken place with findings presented at a

scientific meeting.
• Conflate causation and association – failing to explain limitations of observational studies.
• Get caught up in reporting on the latest study without reporting on larger, better-designed studies

that have been done already.

1 An indirect indicator of a disease state or of its response to therapy. Such markers often include laboratory tests thought to
represent clinical progress accurately. For example, in diabetes mellitus, the glycosylated hemoglobin level is used as a
marker of glycemic control; in AIDS the level of HIV RNA is used as a marker of disease progression.

Questions to ask:

• What are the limitations of the evidence?
• Was the study done in only a few people?
• Was the study done for only a short time? What might happen long-term? Will there be follow-

up?
• Did the study report on an outcome that you really care about – like illness or death? Or did it

only report on test results, markers, or scores?
• Did this information come from a talk presented at a scientific meeting? If so, you should know

this kind of research is often considered preliminary because other experts haven’t had a chance
to thoroughly review it.

• Were the findings from an animal or lab experiment that might not be applicable to human
health?

• Did the information simply present anecdotes as evidence of a treatment’s harms or benefits –
rather than real numbers from the entire study group?

5. Does the story commit disease-mongering?

Sometimes new conditions or diseases actually may
just be opportunities to medicalize (and monetize)
normal states of health. Readers aren’t well-served
by stories that exaggerate or oversell conditions.

There are different forms of “mongering,” including:

• turning risk factors into diseases, with the implication that, then, these must be treated (e.g., low
bone mineral density becomes osteoporosis);

• misrepresentation of the natural history and/or severity of a disease (e.g., early-stage low-grade
prostate cancer);

• medicalization of minor or transient variations in function (e.g. temporary erectile dysfunction);
• medicalization of normal states (aging, baldness, wrinkles, shyness, menopause);
• exaggeration of how common a disorder is (e.g., using rating scales to ‘diagnose’ chronic dry

eye; see “not satisfactory” story examples below).

Identifying disease mongering is a matter of judgment. Sometimes it is obvious. Sometimes there’s a
fine line about whether an article on irritable bowel disorder, erectile dysfunction, restless leg
syndrome or osteoporosis (all of which can be serious for some sufferers) is misrepresenting the
condition to the public.

Note: Most of the conditions we list are not life-threatening. The grand-daddy of disease-mongering
is fixating on LDL cholesterol numbers instead of evaluating whether studies showed a change in
how well or how long people live–and whether treatment results in fewer cases of heart disease or
fewer deaths.

Beware of statistics that may inflate the seriousness of a condition. Beware of articles that
exaggerate the human consequences of a condition – “millions of us are suffering in silence with
toenail fungus.” Suffering? Beware of interviews with “worst-case” patients – holding such patients
up as examples as if their experiences were representative of all with this condition.

The story should discuss the prevalence of the condition under discussion–and if that prevalence
could possibly be inflated.

6. Does the story use independent sources and identify conflicts of interest?

Health care is rife with conflicts of interest. Readers
deserve to know if the sources in the story have any
conflicts of interest. They also deserve to hear the opinion
of objective, independent third-party experts, who are vital
to backstopping exaggerated claims.

Conflicts of interest are common among people making claims about new treatments, tests, products
or procedures. Examples include:

• A trial paid for by the drug manufacturer.
• Researchers employed by or getting fees from a drug company.
• A spokesman for a device manufacturer.
• Doctors who are early adopters and true believers in a new device.
• An inventor or researcher who is seeking patents.

All of these people want their product or their idea to look as good as it can.

News stories should identify the source of the story (news release, journal article, editorial, scientific
meeting presentation, etc.) and explain how likely that source is to be conflicted (e.g., a PR
consultant working for the company said; or Dr. Smith, who received a company grant to perform
the study, said…)

They also must include the input of a trusted independent source with expertise on the subject to
comment on the claims made. (Ideally, more than one such independent source would be used). But
often stories only have one source, and it’s someone with “a dog in the hunt.”

7. Does the story compare the new approach with existing alternatives?

Just because something is “new” in health care
doesn’t mean it’s better. When reading about a
new drug, test, surgery, etc., readers should come
away informed that there are other options on the
table.

Journalists want to cover what’s new. But without a discussion of alternatives, readers can come
away with the impression that the new option is the only good option. Therefore, the new
intervention or finding must be put in context with existing alternatives that have a longer, more
proven track record.

Alternatives to an intervention can mean lifestyle changes, another drug, surgery, no treatment, or
“active surveillance” – declining immediate aggressive therapy while remaining under close medical
care. A good article contains a brief discussion of the other treatments on the table, and how they
compare in effectiveness and cost.

Stories may get rated ‘Not Satisfactory’ if they:

• Discuss a surgical approach without mentioning non-surgical alternative approaches.
• Discuss a new test without mentioning other tests that are available; including the option of not

being screened in the case of a screening test.
• Fail to discuss the advantages/disadvantages of the new idea compared with existing approaches.
• Fail to discuss how the new treatment, test, product or procedure fits into the realm of existing

alternatives.

8. Does the story establish the availability of the treatment/test/product/procedure?

New medical treatments often get lots of attention before
they’re ever made available to the public. After reading a
story about a new intervention, a person should be able
to answer: Is it actually available to me?

Sometimes news stories report on things as if they were available at the corner drug store (or
doctor’s office), when, in fact, they may be years away from even coming on the market. There may
be a lot of marketing hype that suggests that something is well-studied and ready for use–this hype is
good for stock prices, but not patients.

Whenever you hear a prediction about something “expected to be approved by the FDA” in a certain
time period, take it with a grain of salt. It doesn’t get approved until it gets approved, and there may
be many steps left in the approval process. Ignore crystal ball predictions; they usually come from
someone who stands to benefit. It’s also important to know that “approved” medical devices in the
U.S. may not be safe or effective.

Another issue to be aware of is that something might be approved in Europe, but not in the U.S. That
should raise questions about why it hasn’t been submitted for approval in the U.S. yet, or why it
hasn’t gained approval yet. If it’s already approved in the U.S., or doesn’t need to be approved (such
as a new surgical method or a new type of exercise equipment), it’s usually not enough to just say
the intervention is “available.” For example, if it’s a type of heart surgery, do patients have to travel
to specific hospitals that offer that surgery? If it’s a new fitness fad, is it only available in trendy
gyms in Manhattan? If it’s a new drug, will it immediately be stocked on pharmacy shelves?

9. Does the story establish the true novelty of the approach?

Often something that’s lauded as new or novel really
isn’t. Or it may be new, but it’s not better. So, when
reading about a “new” medical intervention, people
deserve to know: What is truly new and different
about this idea?

Drug companies (and device-makers) are very good at promoting their new drugs or products as
“novel” to increase initial sales. Yet many “new” treatments, tests, products or procedures are not
really novel. A “new” drug may be another member of a well-established therapeutic class of drugs.
And even if a drug is the first in a new class of drugs it may offer no more benefit than the drugs that
are widely available.

By focusing on one new idea, it may appear that it’s the only thing being researched in the field.
Putting this in context of past research helps keep hype in check. That backstory can be hard to find
unless the reporter finds a truly independent expert.

We repeat: Newer isn’t always better in health care. And it may not really even be newer. Stories
should establish just how novel the new thing is.

Resources for establishing novelty:

• Clinicaltrials.gov can provide information about other studies that are underway about a specific
treatment or for a particular condition. It can be useful for providing context about something
seemingly innovative.

• With PubMed, you can put in a key word or two and pretty quickly establish whether something
is absolutely unique and if not – how long it’s been around and studied.

10. Does the story appear to rely solely or largely on a news release?

Sometimes news stories rely solely or largely on a
PR news release–and never disclose that the
information came from a news release. Readers
deserve to know if a story includes no original
reporting.

With many news organizations facing tough economic times, financial pressures may lead some
newsrooms to become more open to using news releases as a way to publish content while cutting
costs. News releases can be valid sources of some information. But journalism is charged with the
task of independently vetting claims. Therefore, it is unacceptable to rely on a news release as the
sole source of information–and to not disclose that.