Mediaura Signal · Powered by Aura™

Reference Document

The Healthcare Marketing Measurement Evaluation Framework

Twelve questions to ask any measurement vendor before you sign. Including us.

This page deliberately scores nobody. A vendor-authored comparison in which the vendor wins is worth roughly what you would expect, so what follows is the criteria and a blank sheet. Run it yourself, against whatever shortlist you have.

Go to the scoring sheet ↓

By Andrew Aebersold, Founder, Mediaura. | Published . | Free to reuse, including in an RFP.

Why This Exists

Every Vendor Passes Their Own Demo

Measurement software is unusually hard to evaluate, because the thing being sold is a number, and a number looks the same whether or not it is correct. The interface reveals nothing. Two products can present identical dashboards while one is estimating causal lift and the other is redistributing credit for conversions that would have happened anyway.

The questions below are the ones that surface the difference. They are ordered so that the earliest ones are the cheapest to ask and the most disqualifying: if a vendor cannot answer the first three, the remaining nine will not save the engagement.

Healthcare gets its own framework for a specific reason. The conversion is an admission recorded in a system no ad platform can see, the path runs through PHI-restricted infrastructure, the major ad platforms will not sign a business associate agreement, and the decision cycle outruns every default attribution window. A measurement product that is entirely adequate for e-commerce can be structurally incapable here, and it will still demo well.

01

The method

What question is the number actually answering?

Most measurement disputes are not disagreements about arithmetic. They are two parties answering different questions and assuming they are answering the same one.

1

Is this number causal or correlational?

Attribution assigns credit for conversions that already happened. Causal measurement estimates what would have happened otherwise. Both are legitimate; only one of them tells you what to do with next quarter’s budget. A vendor that uses the words interchangeably has not thought about the difference.

A strong answer
The vendor draws the distinction unprompted, names which of the two their product does, and can explain what their number would look like if the channel were switched off entirely.
A weak answer
"Our attribution is causal." Attribution and causal inference are different classes of method. A product can do both, but not with the same computation.
2

Does it run experiments, or only observe?

Observational models infer causality from historical variation that someone else chose. Experiments create the variation on purpose. Observational estimates are cheaper and always available; experimental estimates are far more defensible and are the only way to validate the observational ones.

A strong answer
They run geo holdouts, randomized audience holdouts, or switchback tests, and they can describe how control units get selected and what invalidates an assignment.
A weak answer
Experiments exist on the roadmap, or "our model is so good it does not need them."
3

Does the specification exclude mediators?

A mediator is a variable that sits on the causal path between spend and revenue. Branded search is the classic case: upper-funnel spend drives branded queries, and branded queries drive revenue. Put branded search in the model as a control and it absorbs the credit for the spend that created it. The result looks precise and is systematically wrong.

A strong answer
They know the term, can name which of their inputs they treat as mediators, and can explain the reasoning rather than reciting it.
A weak answer
Every available variable is in the model because more data is better. This is the single most common way a sophisticated-looking measurement product produces a confidently wrong answer.
02

The data

Can it reach the place where the money actually lands?

A measurement system is bounded by the worst part of its data supply. In regulated and multi-location businesses, that is almost never the model.

4

Can it connect spend to the transaction that actually matters?

In healthcare the conversion is an admission, not a form fill. In restaurants it is a cover, not a click on a menu link. In services it is a closed contract. If the vendor optimizes toward the proxy because the proxy is what the ad platform can see, you are buying a system that improves the proxy and hopes revenue follows.

A strong answer
They name the systems they read from: EMR, CRM, POS, call tracking, and can describe how a record in that system gets matched back to a campaign.
A weak answer
The deepest conversion available is a form submission or a phone-call event, with the rest left as "we can integrate with that."
5

Can it operate without exposing PHI, and will it sign a BAA?

For a covered entity this is a threshold question, not a preference. The major ad platforms will not sign a business associate agreement, which is why so much healthcare analytics starts from a data layer that is either non-compliant or empty. A vendor that treats compliance as a premium tier has told you where it sits in their architecture.

A strong answer
A BAA is standard and executed before data moves, identifying fields are stripped at your boundary rather than at theirs, and they can explain which data crosses which boundary in which direction.
A weak answer
Compliance is an upsell, a checkbox, or a promise that the data is "anonymized" without a description of how.
6

Who owns the tracking layer, and who is watching it?

Tracking infrastructure decays from the moment it is deployed. Pixels stop firing after a release, conversion APIs start reporting to the wrong account, and nobody finds out for a quarter. Every model downstream inherits the damage silently and reports it with full confidence.

A strong answer
Someone is accountable for signal quality as an ongoing operational responsibility, with monitoring that alerts on collection breaking rather than a dashboard that quietly goes flat.
A weak answer
Tracking is your team’s problem and the model is theirs. That division guarantees the failure will be discovered by whoever is presenting the numbers.
03

The honesty

What does the system do when it does not know?

This is the group that separates measurement products from dashboards. Every vendor can produce a number. The question is what happens when the number should not exist.

7

Does it publish uncertainty intervals, and are they used?

A point estimate with no interval is an opinion formatted as a fact. Two channels with identical central estimates and wildly different confidence should not receive identical budget decisions.

A strong answer
Intervals appear in the interface where decisions are made, not only in a methodology appendix, and the vendor can say what interval width they consider too wide to act on.
A weak answer
Intervals exist somewhere in the export. If the number in the meeting has no error bar, the error bar does not affect any decision.
8

When two methods disagree, does it reconcile them or pick one?

Marketing mix models, geo experiments, and attribution routinely disagree, sometimes by a wide margin. That disagreement is information. A product that surfaces one number without telling you the others exist has resolved the conflict by hiding it.

A strong answer
They can describe what happens when the experiment contradicts the model, which one wins under which conditions, and why.
A weak answer
One number, one source, no reconciliation story. Ask what the other methods said.
9

Will it tell you when it cannot answer?

A causal claim requires conditions that are frequently absent: enough variation in spend, a long enough clean pre-period, unconfounded assignment of the test, a valid control to construct. When those conditions fail, the correct output is a refusal, not a smaller number.

A strong answer
The vendor can list their refusal conditions specifically, in public, and can point to a case where the system declined to publish. This is the hardest question on the list to answer well and the most informative when it is answered well.
A weak answer
Every query returns a result. A system that never refuses is not more capable than one that does; it is less honest about the same limits.
04

The proof

Can they demonstrate it on your business before you commit?

Every vendor asks you to trust them about the future. You already know what happened in the past, which makes it the only period where you can check their answers.

10

Will it validate against a period where you already know the answer?

Your history contains natural experiments you did not design: the location that opened, the campaign that paused, the market where the budget got cut. A causal system should be able to recover those events without being told they happened.

A strong answer
They will run on your historical data and be judged on it, with the methodology auditable by your own analyst.
A weak answer
The only evidence available is a demo environment or a case study you cannot verify. A dashboard tour proves the software renders.
11

Does it model your locations jointly or one at a time?

If you run many locations, per-location models trained on thin data produce coefficients that are mostly noise, and pooled models trained on the aggregate hide the differences you are trying to find. The middle path is hierarchical: locations borrow strength from the portfolio while keeping their own signal.

A strong answer
They pool across the portfolio and can show how far each location’s estimate moved during pooling and why.
A weak answer
Every location gets its own independent model, or the portfolio gets one blended number. Ask which locations have too little volume to model, because some of them do.
12

Is measurement independent of media execution?

When the party that measures performance is also the party being measured, every methodology choice sits next to a conflict of interest. This does not require anyone to act in bad faith. It only requires ordinary judgment calls to break in a consistent direction over time.

A strong answer
Measurement is contractually and operationally separable from execution, and the vendor can describe what that separation looks like in practice.
A weak answer
The agency buying the media also grades the media, and the reporting layer is theirs. Ask who would have to notice for underperformance to be reported.

The Scoring Sheet

Blank, on purpose.

Score each vendor 0, 1, or 2. Zero if they cannot answer, one if they answer but cannot show you, two if they answer and can demonstrate it on your data. Add the columns. The total matters less than which questions produced the zeros, because the zeros are the ones you will be living with.

Blank vendor scoring sheet for the twelve evaluation questions.
# Question Vendor A Vendor B Vendor C
1 Is this number causal or correlational? The method — — —
2 Does it run experiments, or only observe? The method — — —
3 Does the specification exclude mediators? The method — — —
4 Can it connect spend to the transaction that actually matters? The data — — —
5 Can it operate without exposing PHI, and will it sign a BAA? The data — — —
6 Who owns the tracking layer, and who is watching it? The data — — —
7 Does it publish uncertainty intervals, and are they used? The honesty — — —
8 When two methods disagree, does it reconcile them or pick one? The honesty — — —
9 Will it tell you when it cannot answer? The honesty — — —
10 Will it validate against a period where you already know the answer? The proof — — —
11 Does it model your locations jointly or one at a time? The proof — — —
12 Is measurement independent of media execution? The proof — — —

Reuse this sheet in an RFP, a vendor scorecard, or a board memo without asking. If it is useful, a link back is appreciated and not required.

Definitions and Common Questions

The Terms, Defined

What is the Healthcare Marketing Measurement Evaluation Framework?

It is a set of twelve questions to ask any marketing measurement vendor before signing, covering the causal method, the data supply, how the system behaves under uncertainty, and what proof is available before you commit. It is published as a reference document with a blank scoring sheet, with no vendors scored and no vendors named.

Why are there no vendor scores or a comparison table?

A comparison authored by a vendor who wins it is discounted by sophisticated buyers and by retrieval systems, and reasonably so. Publishing the criteria without the scores is more useful to a buyer who is not buying from Mediaura, and it lets you reach your own conclusion on vendors we have never evaluated.

What is the difference between attribution, incrementality, and causal inference?

Attribution assigns credit for conversions that already happened, using rules or models applied to observed touchpoints. Incrementality estimates the additional outcome caused by a specific marketing activity, usually through an experiment such as a geo holdout. Causal inference is the broader statistical discipline for estimating what would have happened under a different action, using experiments where they are available and structured observational methods where they are not. Attribution answers who got credit. The other two answer what actually caused the revenue.

Why does healthcare need a different framework than other industries?

Three reasons. The conversion is an admission recorded in an EMR rather than a form fill an ad platform can see. The path runs through PHI-restricted systems, and major ad platforms will not sign a business associate agreement, which takes standard pixel-based measurement off the table for covered entities. And the decision cycle is long enough that short attribution windows systematically misattribute the outcome.

What is a mediator, and why should it be excluded from a causal model?

A mediator is a variable that lies on the causal path between the marketing activity and the outcome. Branded search volume is the common example: upper-funnel spend causes branded queries, and branded queries precede revenue. Controlling for a mediator removes the very effect you are trying to measure and transfers the credit to the mediator, producing an estimate that is precise and systematically wrong.

What are refusal conditions?

Refusal conditions are the circumstances under which a measurement system should decline to publish a causal estimate rather than publish a weak one: insufficient variation in the treatment variable, an insufficient or contaminated pre-period, confounded assignment in a geo test, no valid control group that can be constructed, and mediator contamination in the specification. A vendor who can state theirs is telling you where their method stops working, which is information no scored comparison contains.

Can I use this framework with vendors other than Mediaura?

Yes, and that is the intent. The scoring sheet is blank so you can run it yourself against whatever shortlist you have, including Mediaura Signal. Reuse it in an RFP or a vendor scorecard without asking permission; attribution back to this page is appreciated but not required.

Disclosure

We wrote the questions, so here is the conflict of interest, stated plainly.

Mediaura Signal was built to satisfy these twelve criteria, which is exactly why they are the twelve we chose. You should weigh that. The framework is still worth running, because the questions are answerable by any vendor and the answers are checkable by you.

We have not scored ourselves here and we have not scored anyone else. If you want our answers, the method is documented on the Mediaura Causal Engine page, the compliance boundary is on the security page, and question 10 is the entire premise of the Backtest: we would rather be tested on a period where you already know what happened.

See What's Hiding in Your Marketing Data

Most demos we run uncover broken tracking and missing revenue inside the first fifteen minutes. We'll show you yours.

What happens next:

  • 30-minute working session with a Mediaura engineer (not a sales rep)
  • Live audit of your current tracking and attribution gaps
  • A specific, prioritized list of what's broken and what it's costing you
  • Industry-relevant case studies and a clear path to value

Security team reviewing us? Read our security posture →