Mediaura Signal · Powered by Aura™

Reference Document

The Healthcare Marketing Measurement Evaluation Framework

Twelve questions to ask any measurement vendor before you sign. Including us.

This page deliberately scores nobody. A vendor-authored comparison in which the vendor wins is worth roughly what you would expect, so what follows is the criteria and a blank sheet. Run it yourself, against whatever shortlist you have.

Go to the scoring sheet → Download the scoring sheet (PDF)

Why This Exists

Every Vendor Passes Their Own Demo

Measurement software is unusually hard to evaluate, because the thing being sold is a number, and a number looks the same whether or not it is correct. The interface reveals nothing. Two products can present identical dashboards while one is estimating causal lift and the other is redistributing credit for conversions that would have happened anyway.

The questions below are the ones that surface the difference. They are ordered so that the earliest ones are the cheapest to ask and the most disqualifying: if a vendor cannot answer the first three, the remaining nine will not save the engagement.

Healthcare gets its own framework for a specific reason. The conversion is an admission recorded in a system no ad platform can see, the path runs through PHI-restricted infrastructure, the major ad platforms will not sign a business associate agreement, and the decision cycle outruns every default attribution window. A measurement product that is entirely adequate for e-commerce can be structurally incapable here, and it will still demo well.

01

The method

What Question Is the Number Actually Answering?

Most measurement disputes are not disagreements about arithmetic. They are two parties answering different questions and assuming they are answering the same one.

1

Is This Number Causal or Correlational?

Attribution assigns credit for conversions that already happened. Causal measurement estimates what would have happened otherwise. Both are legitimate; only one of them tells you what to do with next quarter’s budget. A vendor that uses the words interchangeably has not thought about the difference.

A strong answer
The vendor draws the distinction unprompted, names which of the two their product does, and can explain what their number would look like if the channel were switched off entirely.
A weak answer
"Our attribution is causal." Attribution and causal inference are different classes of method. A product can do both, but not with the same computation.
2

Does It Use Experiments, or Only Observe?

Observational models infer causality from historical variation that someone else chose. Experiments create the variation on purpose. Observational estimates are cheaper and always available; experimental estimates are far more defensible and are the only way to validate the observational ones.

A strong answer
They design geo holdouts or other controlled tests, say who executes them, and can describe how control units get selected and what invalidates an assignment.
A weak answer
Experiments exist on the roadmap, or "our model is so good it does not need them."
3

Does the Specification Exclude Mediators?

A mediator is a variable that sits on the causal path between spend and revenue. Branded search is the classic case: upper-funnel spend drives branded queries, and branded queries drive revenue. Put branded search in the model as a control and it absorbs the credit for the spend that created it. The result looks precise and is systematically wrong.

A strong answer
They know the term, can name which of their inputs they treat as mediators, and can explain the reasoning rather than reciting it.
A weak answer
Every available variable is in the model because more data is better. This is the single most common way a sophisticated-looking measurement product produces a confidently wrong answer.
02

The data

Can It Reach the Place Where the Money Actually Lands?

A measurement system is bounded by the worst part of its data supply. In regulated and multi-location businesses, that is almost never the model.

4

Can It Connect Spend to the Transaction That Actually Matters?

In healthcare the conversion is an admission, not a form fill. In restaurants it is a cover, not a click on a menu link. In services it is a closed contract. If the vendor optimizes toward the proxy because the proxy is what the ad platform can see, you are buying a system that improves the proxy and hopes revenue follows.

A strong answer
They name the systems they read from: EMR, CRM, POS, call tracking, and can describe how a record in that system gets matched back to a campaign.
A weak answer
The deepest conversion available is a form submission or a phone-call event, with the rest left as "we can integrate with that."
5

Can It Handle PHI Safely, and Will It Sign a BAA?

For a covered entity this is a threshold question, not a preference. The major ad platforms will not sign a business associate agreement, which is why so much healthcare analytics starts from a data layer that is either non-compliant or empty. A vendor that treats compliance as a premium tier has told you where it sits in their architecture.

A strong answer
A BAA is standard and executed before data moves. Sensitive form fields the tag recognizes never leave the browser, identifiers are hashed at the point of collection before anything reaches analytics, and the vendor can explain which data crosses which boundary in which direction and how all of it is handled as PHI.
A weak answer
Compliance is an upsell, a checkbox, or a promise that the data is "anonymized" without a description of how.
6

Who Owns the Tracking Layer, and Who Is Watching It?

Tracking infrastructure decays from the moment it is deployed. Pixels stop firing after a release, conversion APIs start reporting to the wrong account, and nobody finds out for a quarter. Every model downstream inherits the damage silently and reports it with full confidence.

A strong answer
Someone is accountable for signal quality as an ongoing operational responsibility, with monitoring that flags collection breaking rather than a dashboard that quietly goes flat, and a clear split between what the vendor monitors and what your team configures on your own sites.
A weak answer
Nobody owns the layer end to end. In that gap, the failure tends to be discovered by whoever is presenting the numbers.
03

The honesty

What Does the System Do When It Does Not Know?

This is the group that separates measurement products from dashboards. Every vendor can produce a number. The question is what happens when the number should not exist.

7

Does It Publish Uncertainty Intervals, and Are They Used?

A point estimate with no interval is an opinion formatted as a fact. Two channels with identical central estimates and wildly different confidence should not receive identical budget decisions.

A strong answer
Intervals appear in the interface where decisions are made, not only in a methodology appendix, and the vendor can say what interval width they consider too wide to act on.
A weak answer
Intervals exist somewhere in the export. If the number in the meeting has no error bar, the error bar does not affect any decision.
8

When Two Methods Disagree, Does It Reconcile Them or Pick One?

Marketing mix models, geo experiments, and attribution routinely disagree, sometimes by a wide margin. That disagreement is information. A product that surfaces one number without telling you the others exist has resolved the conflict by hiding it.

A strong answer
They can describe what happens when the experiment contradicts the model, which one wins under which conditions, and why.
A weak answer
One number, one source, no reconciliation story. Ask what the other methods said.
9

Will It Tell You When It Cannot Answer?

A causal claim requires conditions that are frequently absent: enough variation in spend, a long enough clean pre-period, unconfounded assignment of the test, a valid control to construct. When those conditions fail, the correct output is a refusal, not a smaller number.

A strong answer
The vendor can list their refusal conditions specifically, in public, and can point to a case where the system declined to publish. This is the hardest question on the list to answer well and the most informative when it is answered well.
A weak answer
Every query returns a result. A system that never refuses is not more capable than one that does; it is less honest about the same limits.
04

The proof

Can They Demonstrate It on Your Business Before You Commit?

Every vendor asks you to trust them about the future. You already know what happened in the past, which makes it the only period where you can check their answers.

10

Will It Validate Against a Period Where You Already Know the Answer?

Your history contains natural experiments you did not design: the location that opened, the campaign that paused, the market where the budget got cut. A causal system should be able to recover those events without being told they happened.

A strong answer
They will run on your historical data and be judged on it, with the methodology auditable by your own analyst.
A weak answer
The only evidence available is a demo environment or a case study you cannot verify. A dashboard tour proves the software renders.
11

Does It Model Your Locations Jointly or One at a Time?

If you run many locations, per-location models trained on thin data produce coefficients that are mostly noise, and pooled models trained on the aggregate hide the differences you are trying to find. The middle path is hierarchical: locations borrow strength from the portfolio while keeping their own signal.

A strong answer
They pool across the portfolio and can show how far each location’s estimate moved during pooling and why.
A weak answer
Every location gets its own independent model, or the portfolio gets one blended number. Ask which locations have too little volume to model, because some of them do.
12

Is Measurement Independent of the Platforms Being Measured?

Every ad platform reports its own results, and each one claims credit for the same conversions. When the party that measures performance is also the party being measured, every methodology choice sits next to a conflict of interest. This does not require anyone to act in bad faith. It only requires ordinary judgment calls to break in a consistent direction over time.

A strong answer
Every channel is measured against a single reconciled record, conflicts between sources go to review instead of being quietly resolved, and any media work the vendor also does runs under a separate agreement with the same disclosure rules.
A weak answer
Each platform’s own reporting is the measurement, and added together the platforms claim more conversions than you had. Ask which numbers would have to be wrong for their claims to reconcile.

Start With the Fit Check

Free and honest, with no obligation on your side. If your data can support causal conclusions, you'll get a written fixed-fee quote and timeline. If it can't, you'll know that too.

Start the Fit Check →
Request a demo → Prefer to start with a conversation?