Fail Conclusively

Design experiments to maximize treatment effect so failures are definitive learnings, not inconclusive results

Sri Batchu
Failure (Compilation)

Fail Conclusively

"Failure is not that you didn't drive revenue, failure is not learning. So it's really important that you learn when you fail. And you can only learn if you've designed the right test and you failed conclusively." - Sri Batchu

What It Is

Most experiments fail—growth experiments typically have around a 30% success rate. The real failure isn't when an experiment doesn't drive revenue; it's when you fail to learn anything from it. Inconclusive failures are the most dangerous outcome because they waste time and create organizational confusion.

When an experiment produces inconclusive results, the same hypothesis gets tested repeatedly. Every new executive or team member who has the same intuition will want to try again. This pattern can continue for years, draining resources without ever producing a definitive answer.

To fail conclusively means designing experiments so that a negative result provides genuine learning—a clear signal that this direction isn't worth pursuing further.

How It Works

The key insight is that experiments succeed in only two ways:

  1. Large sample size (N) - More people in the test produces statistically significant results
  2. Significant treatment effect - A dramatic difference between control and test groups

In B2B contexts, you rarely have the luxury of large N. Consumer companies like Facebook can achieve statistical significance in hours; a B2B company might need two years to reach the same sample size.

To counteract limited sample size, maximize the treatment effect:

  • If you have a hypothesis about what might work, throw all possible tactics and resources at testing it
  • Don't test one small change at a time—combine every lever you believe could move the needle
  • If with maximum effort it still doesn't work, you have conclusive evidence the hypothesis is wrong
  • If it works, you can then run follow-up tests to identify which specific tactics drove results

How to Apply It

  1. Frame the hypothesis broadly - Instead of "this email copy will improve conversion," frame it as "investing heavily in this customer touchpoint will improve conversion"

  2. List all possible tactics - Brainstorm every way you could potentially move the needle on this hypothesis: different triggers, content approaches, personalization, design changes, timing, etc.

  3. Combine everything into one test - Rather than A/B testing individual changes, create a "maximum effort" treatment that includes all your best ideas

  4. Compare to control - Run the comprehensive treatment against your current baseline

  5. Interpret results:

    • If it doesn't work: You've conclusively proven this direction isn't worth pursuing. Document the finding to prevent repeat experiments.
    • If it works: Great! Now run follow-up experiments with subsets of tactics to optimize cost and identify what specifically drives results.

  6. Document the outcome - Create a clear record of what was tested, the comprehensive treatment used, and the conclusion. This prevents future teams from re-running the same experiment.

When to Use It

This framework is especially valuable for:

  • Cross-functional, larger-scale experiments where re-running tests is expensive
  • B2B contexts with limited sample sizes
  • Strategic hypotheses that keep resurfacing across teams or leadership changes
  • Account-based marketing and other high-touch strategies where "we tried it but it didn't work" claims are common

For small, low-cost experiments (like email subject line tests), traditional incremental A/B testing is fine. Use "Fail Conclusively" when the stakes of inconclusive results are high.

Example: Account-Based Marketing

Account-based marketing (ABM) is a common example where teams test half-heartedly, see inconclusive results, then repeat the cycle when new go-to-market leadership arrives.

Instead, apply Fail Conclusively:

  • Select a target account list
  • Combine multiple touchpoints: personalized emails, direct mail, targeted ads, custom content, executive outreach, events
  • Track conversion versus a control group of similar accounts
  • If maximum-effort ABM doesn't outperform the control, you've proven ABM isn't worth the investment
  • If it works, test variations to find the cost-efficient mix

Source

  • Guest: Sri Batchu
  • Episode: "Failure" (Compilation Episode)
  • Key Discussion: (00:33:38) - Designing experiments that fail conclusively
  • YouTube: Watch on YouTube

Related Frameworks

Back to Execution