Every ad channel takes credit for cases. The harder question is which cases it actually added: the ones that would not have signed without the spend. Attribution can’t answer that. A person who saw your TV spot, then searched your name and clicked a brand ad, gets counted as a brand-search case. A holdout test answers it with a fair comparison. This guide explains how a law firm can run one, using geography, and how to read what comes back. It uses the rules of The Probative Standard, and it builds on our guide to cost per signed case.
Why attribution can’t prove a channel works
Attribution assigns each signed case to a source, such as the last paid click before first contact. That’s useful for running accounts, and every guide on this site uses it. But it measures who got credit, not what caused the case. Some channels, brand search above all, sit at the end of a path that other spending started. They collect credit for cases that would have signed anyway.
The Probative Standard draws the line plainly: any claim that spend “caused”, “created”, “drove” or “added” signed cases needs a test, and without one the word is “attributed” (The Probative Standard). An incremental signed case is one that would not have happened without the spend. It is estimated only in aggregate, from a test, with an interval. No single case is ever labeled incremental.
Lawyers already know this logic. It is the but-for test: but for the spend, would the case have signed? A holdout test is that same question, run on your marketing, which is why Probative calls its on/off tests the But-For Test.
How a geo holdout works
You split the areas you advertise in into two groups. In the treatment group you change the spend: switch the channel off, or step it down by a fixed share. The control group stays on plan. After the test, you compare signed cases in the treatment areas with what they would have been, estimated from the control areas.
Three design choices matter.
1. The unit. Areas must be ones the ad platforms can target and your CRM can assign cases to: counties, ZIP code clusters or media markets. GeoLift, Meta’s open-source tool for geo tests, recommends using the finest level at which campaigns can be targeted (GeoLift). For TV, the unit is usually the media market, because that is how TV is bought.
2. The comparison. Treatment and control areas should be matched on the outcome you care about, signed cases, over a long stable period. GeoLift builds a “synthetic control”: a weighted blend of control areas that tracks the treatment areas closely before the test, using the augmented synthetic control method (GeoLift walkthrough). The gap after the test starts is the estimated effect.
3. Holdout or step. A full holdout switches the channel off in treatment areas. A step test cuts spend by a share, such as 50%, or raises it, such as 30%. The Standard prefers step tests because they answer the question a budget decision actually turns on, what the next dollar buys, and they keep markets open (The Probative Standard). Full holdouts suit channels where off is a real option: brand search, a Local Services Ads category, social prospecting.
The data you need first
A test is only as good as the history it’s matched on. The minimums below follow the Standard and GeoLift’s published practices.
| Requirement | Minimum | Why |
|---|---|---|
| History | 52 weeks of signed cases by area; 26 at the very least | Captures seasonality (GeoLift) |
| Candidate areas | At least 20, each with 25+ pre-test periods | GeoLift’s stated minimum for a usable panel |
| Stable pre-period | 4 to 5 times the test length, no structural break | A new office or intake change breaks the match |
| Data | Daily if possible, every area and date filled | GeoLift strongly prefers daily data and needs no gaps |
| Outcome | Signed cases from leads created in the period, from your CRM | The firm’s records, not platform conversions |
Two firm-specific problems come up often. First, the CRM must record where the client lives, not just where the firm’s office is. Second, the firm needs enough signed cases by area for the comparison to mean anything. That is what the power check is for.
Power and duration
Power is the chance a test detects an effect of a given size if the effect is real. The convention is 80% (PMC). The Standard reads results with two-sided 90% intervals, GeoLift’s default alpha of 0.1 (GeoLift walkthrough), and requires at least 80% power at the test’s minimum detectable effect: the smallest change it can reliably find.
The minimum detectable effect has to be smaller than the smallest effect that would change your decision. If you’d keep a channel even if it added only 15% of the treatment areas’ cases, a test that can only detect 60% is useless to you. GeoLift’s market-selection function simulates designs on your own history and reports power and the minimum detectable effect for each (GeoLift walkthrough).
Duration. The Standard requires four to eight weeks of treatment, never less than four. GeoLift advises covering at least one full purchase cycle (GeoLift). For a law firm, the “purchase cycle” is the time from first contact to signing, which is why the read waits.
The lag window. Count signed cases from leads created during the treatment, and read them only after at least 30 days and after your 80th-percentile lead-to-sign time, whichever is later. Injury firms with slow signings may need longer.
The template
Free template: holdout design, one page
Write these twelve lines down before the test starts
| # | Field | What to write |
|---|---|---|
| 1 | Question | “Does [channel] add signed cases in [practice], and how many per $1,000?” |
| 2 | Expected direction | Fewer cases in treatment areas (holdout or step-down), or more (step-up) |
| 3 | Primary metric | Signed cases from leads created during treatment, from the CRM, by client location |
| 4 | Secondary metrics (2 at most) | Qualified leads; consults held |
| 5 | Design | Full holdout or step test; the share of spend changed |
| 6 | Areas | Unit (county, ZIP cluster, media market); how treatment areas were chosen; share of the market held out |
| 7 | Dates | Pre-period, treatment start and end, read date after the lag window |
| 8 | Power | Minimum detectable effect and power at that effect, from a simulation on your history |
| 9 | Method | Synthetic control or matched markets; tool and version |
| 10 | Integrity checks | Intake staffing, tracking, prices and other channels held steady; checked weekly |
| 11 | Stopping rules | Stop only for an integrity failure or at the firm’s request; an early stop is reported as inconclusive |
| 12 | Decision rule | What happens after a win, a null and an inconclusive result, written now |
Illustrative power table
IllustrativeFewer weekly cases means only large effects can be detected
| Signed cases a week in treatment areas | 4-week test | 6-week test | 8-week test |
|---|---|---|---|
| 5 | about 83% | about 68% | about 59% |
| 10 | about 59% | about 48% | about 42% |
| 25 | about 37% | about 30% | about 26% |
| 50 | about 26% | about 22% | about 19% |
The lesson of Exhibit B is not the exact figures; it is the shape. A firm with five signed cases a week in its test areas can only detect a channel that drives more than half of them. That can still be worth knowing, for example to confirm that a large program is pulling its weight. But it can’t settle a fine-grained budget question. Larger tests, longer tests and bigger spend changes all shrink the detectable effect.
Reading the result: win, null or inconclusive
The Standard sets three outcomes, decided by the interval for incremental signed cases (The Probative Standard).
| Result | Rule | What follows |
|---|---|---|
| Win | The 90% interval excludes zero, in the expected direction | The pre-registered decision runs |
| Null | The interval includes zero and lies within plus or minus the minimum detectable effect | Reported as a result; the “null” decision runs |
| Inconclusive | Anything else, including a failed integrity check or an early stop | No budget change rests on it |
A full readout shows the estimate and its interval, the incremental cost per signed case, the cases attribution gave the channel in the same areas, and the ratio of the two, the incrementality factor. If attribution credited a channel with 20 cases in the test areas and the test estimates it added 8, the factor is 0.4, and the channel’s true cost per signed case is two and a half times what its report says. That example is ILLUSTRATIVE; the point is that the factor can be well below one, and the only way to know is to test.
A result applies to the channel, areas and conditions tested. The Standard expires it after 12 months, or sooner if the channel’s budget moves by more than half, its strategy changes or the platform changes how it works (The Probative Standard).
TV, radio and billboards
Broadcast media gets no clicks, so attribution gives it nothing, which is as wrong as giving it credit for every brand search. Geo tests suit it well, because TV and radio are bought by media market. A typical design steps spend down, or up, in a set of markets while matched markets stay on plan, then compares signed cases from leads created in those markets. Use dedicated phone numbers per medium and a “how did you hear about us?” question at intake alongside the test, but let the test decide. Our personal injury marketing guide covers how broadcast fits an injury firm’s channel mix.
Platform lift studies
Ad platforms offer their own experiments. Google’s Conversion Lift based on geography splits regions into exposed and control groups, can use offline conversion data aggregated to ZIP or city level, and is available through a Google account representative (Google). Studies like this count under the Standard when you can see the design and the data, and the outcome is your own signed cases. When the platform reports a modeled result you can’t inspect, label it “platform-reported”.
Mistakes that spoil a test
- Changing something else at the same time. A new intake hire, a price change or a TV flight in the test period breaks the comparison. GeoLift warns against national media changes during the pre-test or test period (GeoLift).
- Stopping early because it looks good. An early stop is inconclusive by rule.
- Reading too soon. Late signings land after the test ends.
- Using platform conversions as the outcome. The question is signed cases, from your records.
- Testing without power. A test that can’t detect the effect you care about produces a confident-sounding nothing.
What this means for your firm
- Start recording client location in the CRM for every lead and signed case, if you don’t already.
- Pick one question worth money: usually brand search, a Local Services category, or a broadcast flight.
- Run the power check on your own history before committing to a design.
- Write the twelve-line plan down and date it, with the decision rule for each outcome.
- Treat every untested channel’s numbers as attributed, and say so in your reports.
Probative is AI-first SEO and ads for law firms. Tests follow The Probative Standard and are reported in signed cases from your own records; plans that include them are on the pricing page. If another agency runs your marketing, Independent Measurement can design and read a test on its channels. When you’re deciding where test results should move money, see our law firm marketing budget guide.
Marketing information, not legal advice. Any test uses only attorney-approved ads and never reduces intake coverage; your responsible attorney decides what client information leaves the firm.