How to Test Clay Waterfall Accuracy and Credit Cost

Define the exact Clay waterfall under test

This page owns one buyer-controlled workflow validation. It does not repeat the generic CRM enrichment guide, the vendor-specific Apollo data test, the ZoomInfo renewal test, or the prospecting category guide.

State the decision: approve, reject, reorder, narrow, or remediate this exact waterfall for named fields, populations, and CRM actions. Name workspace and plan, workbook/table/version, source, provider accounts and editions, Clay marketplace versus buyer API keys, waterfall type, requested fields, success conditions, CRM destination, observation period, and owners.

Clay’s official waterfall documentation describes ordered providers and successful-provider output. That establishes configurable behavior, not correctness in your population.

Freeze the table, providers, order, and run conditions

Export a configuration manifest before every run: input columns, provider/order, action/version, credentials class, run condition, skip logic, success/fallback rule, returned status, validation step, formulas, lookups, dedupe key, retries, time zone, cache behavior, manual edits, CRM field mapping, overwrite authority, and budget.

Clay documents “Only run if” conditions and existing-data lookups in its credit-conservation guide. A changed condition changes the tested population and spend. Hash raw inputs and outputs; never overwrite the original run.

Freeze the meaning of success per field. “Non-null” is insufficient. A business email may require entity match, type, status, checked time, policy eligibility, and no authoritative conflict. A company size field may require a band, source date, and defined aggregation.

Build a labeled and stratified entity set

Draw accounts and people from the real target population before enrichment. Stratify by country, region, industry, size, parent/subsidiary, common/rare domain, seniority, job family, common/uncommon name, record age, missingness, and known recent changes. Include no-match, duplicate, ambiguous, stale, suppressed, and protected-field cases.

Assign immutable IDs. Create field-level truth from approved authoritative evidence with labeler, observed time, source, confidence, and unknown state. Keep unknown labels out of accuracy denominators rather than treating them as errors or successes. Blind adjudicators to provider identity.

Split development and untouched holdout before tuning. Neither provider output nor test labels may flow into provider-specific run conditions.

Score match, exactness, coverage, staleness, and conflict

• Entity match = correctly resolved intended entities ÷ eligible inputs.
• Field exactness = correct returned values ÷ judged returned values.
• Raw coverage = populated values ÷ eligible inputs.
• Usable coverage = correct, current, policy-eligible, non-conflicting accepted values ÷ eligible inputs.
• Precision = true target-positive returns ÷ all target-positive returns.
• Staleness = stale returned values ÷ returned values with a freshness judgment.
• Conflict rate = matched records with incompatible authoritative values ÷ matched records.

Report counts, unknowns, exclusions, and every measure by field and stratum. The ICO’s guidance on statistical accuracy distinguishes statistical performance from the data-protection accuracy principle and discusses precision/recall tradeoffs; qualified owners must separately assess applicable privacy law and lawful use.

Attribute marginal yield and cost to each provider

Retain one row per attempt: input ID, provider, position, start/end, status, raw value, accepted value, rejection reason, Action, Data Credits, external-provider cost, retry, and final supplier. Provider marginal yield = accepted new values first supplied by that provider ÷ records reaching that provider. Also report share of total accepted values.

Clay’s current usage documentation separates Actions from Data Credits and notes BYO keys shift data cost outside Clay while Actions remain. Verify the live plan, dashboard, and provider console. Credits per accepted record = all consumed Clay Data Credits ÷ accepted usable records; keep Actions and external charges beside it.

A late provider can have low raw yield but high unique value in a critical region. Do not remove it using pooled averages alone.

Test fallback acceptance and false-overwrite cost

Exercise null, error, ambiguous, catch-all, stale, low-confidence, policy-ineligible, and conflict outputs. Confirm each either stops or falls through exactly as the contract says. A provider “success” must not stop the waterfall when the buyer’s acceptance rule fails.

For every conflicting value, calculate false-overwrite cost = remediation labor + workflow/communication cost + recovery cost + attributable business consequence. Keep estimates labeled. Compare fill-empty, propose-review, and overwrite modes separately. Protected CRM fields should have zero unauthorized overwrites.

Record provenance even after final acceptance. Without provider, observed time, status, and rule, later correction and renewal analysis are weak.

Inject retry, error, cache, and dedupe failures

Seed provider timeout, authentication failure, rate limit, malformed output, partial batch, duplicate input, concurrent update, stale cache, empty success, retry after ambiguous response, budget exhaustion, and formula error. Verify retries do not double-charge or create duplicate accepted state beyond the documented design.

Measure attempt completeness, visible-error rate, duplicate-attempt rate, recovery completeness, latency, and credit reconciliation. Re-run selected records with cache intentionally controlled; disclose whether the test measures fresh provider calls or cached results.

Use stable entity and request keys. A duplicated row should not silently become two CRM contacts or two paid enrichments.

Stage CRM writes and prove rollback

Start with CSV, sandbox, or staging objects. Write an authority matrix per field: read-only, fill-empty, propose, overwrite when newer/higher-authority, or never. Preserve old value, new value, source, timestamp, run ID, rule, approver, and rollback value.

Test invalid picklists, recent CRM versus stale enrichment, null clearing, duplicate contacts, reparenting, owner/permission changes, partial write, retry, and suppression. Gate on zero protected writes, zero actionable suppressed records, stable IDs, complete audit evidence, and verified rollback.

The CRM data-quality framework defines ongoing ownership after the pilot.

Run a leakage-safe holdout and provider-order test

Randomly assign untouched holdout records within strata to order A and order B. Use identical inputs, provider versions, acceptance rules, budgets, times, and adjudication. Prevent outputs, cached values, or truth labels from crossing arms. Freeze analysis before opening results.

Compare usable coverage, exactness, critical errors, marginal yield, Actions, Data Credits, external cost, latency, and CRM-risk gates. Because each order can stop at a different provider, raw provider success is not directly comparable without records-reached denominators.

Repeat only on a new holdout after changing rules. Reusing the same labeled records to select and “confirm” an order leaks test information.

Apply hard gates to the worked example

This fictional example is not Clay performance. On 100 eligible records, provider A supplies 42 accepted values. Provider B receives 58 fallbacks and supplies 18 accepted values. The waterfall returns 60 judged values; 54 are correct. It consumes 360 Data Credits.

If hard gates require 95% exactness, zero protected overwrites, complete suppression, reconciled credits, and rollback success, this run fails on exactness even if coverage and cost look attractive. Diagnose by provider, field, and stratum; change on development data; confirm once on a fresh holdout.

Calculate TCO and publish the bounded verdict

Term TCO = Clay platform + Actions tier + Data Credits + BYO provider subscriptions/usage + design and maintenance + labeling/review + CRM integration + monitoring + remediation + security/privacy/procurement + exit. A fictional $24,000 platform/usage cost plus $18,000 providers, $20,000 operations, $10,000 review, $8,000 CRM/security, and $5,000 remediation/exit totals $85,000. Replace inputs with quotes and loaded labor.

Publish configuration hash, providers/order, run rules, sample/strata, truth contract, dates, cache state, metrics and counts, provider contribution, cost, failures, CRM gates, holdout result, exceptions, and retest trigger. The defensible verdict is “this configuration passed these gates on this population,” never “Clay is accurate.”