Every growth guide says the same thing: test your landing page, test your pricing, test your onboarding. But if you run a startup or small business, you’ve probably tried this and watched a test run for weeks without a clear answer.

The problem isn’t your execution. Classic A/B testing was built for companies with massive traffic.

Why A/B testing is hard for startups

01You don’t have enough traffic.

Say your landing page converts at 3% and you want to detect a 10% relative lift (3.0% to 3.3%). At standard confidence and power levels, you need roughly 50,000 visitors per variant. At 5,000 visitors a month, that’s well over a year for one test.

02Tests are slow to set up.

Someone has to form a hypothesis, design the variant, implement it, QA it, and analyze it. In a small team, that someone is also doing five other jobs.

03Winners go stale.

Markets, ad channels, and customer mix change. A result from six months ago may no longer hold, but nobody has time to retest.

04Testing tools weren’t built for you.

Enterprise platforms assume a dedicated growth team. Generative AI copy tools write variants but never measure whether they work.

What good conversion rate optimization looks like for small teams

Effective CRO for startups comes down to three shifts:

• Test bigger swings. Small tweaks like button colors need huge samples to detect. Bold changes to messaging, pricing structure, or onboarding flow produce larger effects you can detect with less traffic.
• Run tests continuously. Instead of one test at a time with long gaps between them, keep experiments always running so learning compounds.
• Automate the busywork. Hypothesis generation, variant deployment, monitoring, and rollout decisions are all candidates for automation.

Where autonomous experimentation fits

Autonomous experimentation applies this approach end to end. Instead of a person manually running each test, a system handles the loop: propose a change, ship it to a slice of traffic, watch for real signal, and promote the winner or roll back a loser.

At NeriLabs, we built the Continuous Conversion Runtime (CCR) around this idea. It optimizes three high-leverage surfaces for startups and SMBs:

• Landing pages: headlines, layouts, and calls to action
• Pricing tiers: structure, packaging, and presentation
• Onboarding flows: the steps between signup and the first moment of value

CCR supports three modes: tests you direct as a founder, behavior-triggered micro-interventions that respond to what a visitor is doing, and autonomous tests launched when the system detects an anomaly, such as a sudden drop in conversion on a page.

How to get started (with or without a tool)

1. Find your biggest leak. Look at your funnel and pick the step with the steepest drop-off.
2. Write one bold hypothesis. “Visitors don’t understand what we do in five seconds” is testable. “Blue might work better than green” is not.
3. Check your sample size before you start. Use a calculator to see whether your traffic can detect the effect you expect. If not, make the change bigger or test a higher-traffic page.
4. Set a stop rule in advance. Decide the duration and success metric before launch to avoid peeking and declaring false winners.
5. Log every result. Even failed tests tell you what your customers don’t care about.

The bottom line

A/B testing still works. What doesn’t work is copying the process of a company with 100x your traffic and a full growth team. Startups win by testing bolder changes, running experiments continuously, and automating everything that doesn’t need human judgment.

The Messaging Gap Every Technical Founder Hits

Every technical founder we talk to hits the same wall. They can explain exactly what their product does. They can't make a stranger want it in one sentence.

Hand the copy to a marketer and the opposite problem shows up. The headline reads well, but it simplifies the mechanism until the product sounds like every other tool in the category.

The message that ships is either too dense to sell or too vague to trust. We're researching a third option: using embeddings and LLMs to write copy that is both.

The Core Idea: Bridge the Gap in Embedding Space

Represent any two sentences as vectors and you can measure how far apart they sit — their cosine similarity. A technical sentence and a buyer-friendly sentence about the same product usually sit far apart.

Take an example. A persuasive hook: "Stop guessing which headline brings buyers." A technical mechanism: "Bayesian adaptive traffic allocation." Different vocabulary, different register, low similarity.

The two connect through an idea in between: learning from outcomes to guide messaging decisions. Generate that connecting sentence and the hook stays honest without reading like a spec sheet.

That's the bet behind this research: an LLM that writes the bridge, not just the headline.

Why the Obvious Approach Falls Short

The naive version of this looks simple: embed the hook, find the technical sentence closest to it, and stitch them together. It doesn't work.

Closeness in embedding space isn't the same as support. A mechanism can share vocabulary with a hook and still fail to justify it — two sentences about "speed" don't mean one explains the other.

The reverse is also true. The most useful mechanism for a hook can sit relatively far from it, because the hook describes an outcome and the mechanism describes how that outcome gets produced.

Prompting an LLM to "write persuasive copy about this feature" has the same failure mode. Without a way to check what the product actually supports, the model drifts toward generic claims no retrieval step ever grounded.

Our Approach: A Four-Stage Pipeline

Rather than retrieve the closest technical sentence to a hook, we retrieve against the customer need sitting between them. The pipeline runs in four stages:

1. Store technical facts with their supported meaning. Each mechanism in the product is tagged with the customer problem it solves, the outcomes it depends on, and the claims it cannot support.
2. Retrieve by customer problem, not jargon. The query asks for a mechanism that solves the hook's underlying problem, not for sentences that merely sound similar to the hook.
3. Rerank for compatibility. Candidates are scored on whether they can honestly support the hook, not on how close they sit in vector space.
4. Generate the bridge, or reject the pairing. When no retrieved mechanism can honestly support the hook, the system says so instead of forcing a connection.

Inside the Pipeline

Retrieval query. Instead of searching for sentences similar to the hook, we search for a verified mechanism that solves the underlying customer problem, returned with its limitations attached, not more marketing copy.

Compatibility reranking. An LLM scores each candidate mechanism against a fixed rubric before anything gets generated:

Factual support is a gate, not just a score. A candidate that sounds compelling but isn't backed by the product facts is rejected outright.

Diversity, used correctly. Maximal marginal relevance (MMR) helps when retrieval returns several near-identical mechanisms. It doesn't establish that a mechanism supports a hook, so we use it to diversify an already-relevant pool, never as a substitute for the compatibility check.

Training the models. Three separate components can be fine-tuned, each for a different failure mode:

The generator is trained on the full task, not isolated sentences, including examples where the correct output is a rejection: "this mechanism doesn't support this hook." Preference training then teaches finer judgment: between two true bridges, which one resolves the hook with a specific, supported detail instead of an unsupported promise.

What This Means for Founders and Growth Engineers

Today, writing a hero line that's both credible and persuasive means sitting down, pulling technical facts, and manually finding the angle that sells without overstating. That's hours spent on every headline, every pricing line, every onboarding prompt.

This research feeds directly into the Write step of our platform, the part of NeriLabs that drafts hypotheses before a test ever ships. The goal is for that step to generate messaging that's already grounded in what the product actually does, so there's less brainstorming before a test can start.

For product-led growth and B2B SaaS teams testing messaging across landing pages, pricing tiers, and onboarding flows, that means more variants worth testing, with less manual drafting between them.

What's Next

This is early research, not a shipped feature. The build order we're following: structured product facts first, then retrieval by customer need, then an LLM-based compatibility reranker, then a prompted generator, with a held-out evaluation set before any fine-tuning.

We'll publish what we learn as the pipeline gets tested against real messaging, including where it says no.

The Promise: Knowing Before Revenue Arrives

Most experiments take weeks to resolve because revenue takes weeks to mature. A visitor who signs up today might not convert to a paying plan for a month.

That delay is expensive. Teams either wait it out, or call a winner off early engagement numbers and hope the pattern holds.

We're researching a third option: a model that turns early behavior into a predicted revenue number, so a team gets a directional read sooner. The question we're actually testing is harder than it sounds. A model can predict revenue well and still be the wrong metric for an experiment.

The Core Idea: A Surrogate Index

The approach is what statisticians call a surrogate index: combine several short-term signals, activation, engagement, early usage, into a single score trained to predict a later outcome. Instead of waiting on revenue, you measure the experiment's effect on that score.

This isn't a new idea. Surrogate endpoints have been studied in clinical trials for decades, where a short-term biomarker stands in for a slow-to-observe outcome like survival. Applying the same logic to growth experiments is reasonable. Applying it safely requires the same scrutiny.

The Causal Catch

A strong revenue predictor is not automatically a valid experiment metric. Those are different claims, and the gap between them is where this kind of model usually breaks.

Here's a concrete failure mode. Historically, visitors who hover over pricing tiers tend to buy, so the model learns to assign predicted value to pricing hovers. Then a new design ships that makes pricing confusing, so more visitors hover out of confusion, not interest. Predicted revenue rises. Actual purchases fall.

The model didn't get worse at prediction on historical data. The intervention changed what the signal means, and the model had no way to know that.

Two assumptions have to hold for a surrogate to be trustworthy as an experiment metric, and both are stronger than ordinary predictive accuracy:

• Pathway capture. The surrogate has to sit on the causal path between the treatment and the downstream outcome, not just correlate with the outcome in observational data.
• Population transfer. The relationship the model learned has to hold in the experimental population under the new treatment, not only in the historical population it was trained on.

This is close to what the surrogate-endpoint literature calls the Prentice criterion: it's not enough for the surrogate to predict the outcome across individuals, the surrogate's treatment effect has to predict the outcome's treatment effect. Most naive implementations only check the first condition.

How We're Approaching It: Three Separated Outputs

Rather than collapse everything into one number, we're researching a design that reports three outputs explicitly, so a team always knows which kind of claim they're looking at:

Keeping these separate matters more than getting any one of them right. A predicted lift that quietly gets treated as a confirmed result is the exact failure mode the pricing-hover example describes.

How We Validate It

A prediction model for this has to be frozen before an experiment starts, with its observation window specified in advance. Changing the model mid-experiment reopens the door to the same failure.

Validation also can't stop at held-out individual visitors. The model needs to be checked against completed randomized experiments, where the true treatment effect is already known. The question isn't "does this model predict individual revenue well," it's "does this model's predicted treatment effect agree with the treatment effect that experiment actually produced."

A model can do well on the first question and fail the second. That gap is specifically what surrogate validation is supposed to catch, and it's the part we're spending the most research time on.

What This Means for Experimentation

Statistics practice already has a name for the discipline this calls for: an overall evaluation criterion, a single pre-committed metric (or small set of them) that a team agrees defines a winning experiment before it ships. Loosely wiring a predicted-revenue model into that decision without validating it against real treatment effects undermines the thing OECs exist to protect.

This research is feeding how we think about the Check step of our platform, the part that decides whether a running experiment is working. If a predicted-revenue signal ships there, it ships labeled for what it is: a conditional estimate, shown alongside the observed behavioral lift, never presented as a confirmed revenue number until revenue has actually matured.

The Business Case, If It Validates

The outcome this is ultimately for is speed: reallocating traffic and budget away from a losing variant before weeks of revenue get spent proving what predicted downstream lift would have flagged sooner.

If pathway capture and population transfer hold up in validation, three outcomes follow directly:

• Faster reallocation. Traffic shifts away from an underperforming variant once the predicted lift and realized behavioral lift disagree, instead of waiting for revenue to fully mature.
• More experiments per quarter. Shorter effective decision time means the same traffic volume supports more tests, so the rate of validated wins compounds instead of being capped by revenue-cycle length.
• Lower cost of a bad variant. Catching a losing design on predicted lift, confirmed against realized treatment effects, limits how much revenue gets exposed to it before it's caught.

None of this is a claim we're ready to make unconditionally. It's the reason the validation work above matters: these outcomes only hold if the surrogate passes pathway capture and population transfer, not just individual-level prediction accuracy.

What's Next

This is active research, not a shipped feature, and not a claim we're making about the product today. The build order: validate surrogate pathway capture first, then test population transfer against completed experiments, then pilot the three-output design with real revenue data.

"Earlier evidence, followed by revenue confirmation" is the claim we think we can eventually defend. "Results in days instead of months" is not something we'll say unconditionally. If it ever appears in our product copy, it will be as a measured pilot result, with the conditions it held under stated next to it.