Teams that ship login, signup, password reset, or 2FA flows eventually hit the same problem, the happy path is easy to automate until a real SMS message is involved. At that point, the test stops being just a browser script and becomes a small distributed system: your app emits an OTP, a provider delivers it, a webhook or inbox watcher captures it, the test extracts the code, and CI decides whether to retry or fail.

That is why a test SMS verification harness is worth treating as infrastructure, not as a throwaway helper. If it is built casually, you get flakes, duplicated codes, timing bugs, and test accounts that are hard to clean up. If it is built deliberately, you get deterministic coverage of the exact flows users rely on, with enough observability to debug failures instead of guessing.

This article walks through a practical architecture for SMS verification testing using Twilio, webhook handling, polling fallbacks, and CI retry logic. It focuses on the parts that fail in real teams, idempotency, cleanup, and the edge cases around asynchronous delivery. It also explains where the maintenance cost starts to outweigh the benefit, and when a maintained platform can be the simpler route.

What a test SMS verification harness needs to prove

A good harness should verify more than “an SMS was sent.” In practice, the important assertions are:

  • The app sends a verification message to the intended number
  • The message contains a usable code or link
  • The code is accepted only once, or within the intended TTL
  • Expired codes are rejected
  • Retry and resend behavior works correctly
  • Incorrect or stale numbers fail cleanly
  • The flow survives webhook delays and transient provider issues

The goal is not to mock away the messaging path, it is to test the parts of the path that break user journeys when message delivery, parsing, or timing behaves unexpectedly.

For background on the categories this sits under, see software testing, test automation, and continuous integration.

Reference architecture

A minimal production-like harness has five pieces:

  1. Application under test that sends verification SMS messages
  2. Twilio number or messaging path that receives the outbound message
  3. Webhook endpoint that captures delivery events or inbound messages
  4. Message store where the test can query the latest OTP by test session
  5. Test runner and CI logic that waits, retries, and cleans up

Twilio’s messaging documentation is the primary reference for the delivery side, webhook configuration, and request/response model, so it is worth reading alongside the implementation work: Twilio Messaging docs.

A simple data flow looks like this:

text Test runner -> app signup/login flow -> SMS provider -> webhook -> message store -> test runner extracts OTP -> app verifies OTP

There are two design choices that shape everything else:

  • Push model, the provider calls your webhook when a message arrives
  • Pull model, the test polls an API or mailbox-like store until the message appears

Most teams end up with a hybrid, webhook first, polling as a fallback when timing or network uncertainty makes delivery events incomplete.

Choose the right test number strategy

The phrase CI test phone numbers covers several patterns, and they are not equivalent.

1. Dedicated real numbers per environment

Each environment gets one or more real SMS-capable numbers. Tests send OTPs to those numbers, then the harness waits for the message to arrive.

Pros:

  • Closest to production behavior
  • Simplifies message routing and debugging
  • Useful when you need actual end-to-end delivery coverage

Cons:

  • Requires lifecycle management for numbers
  • Can be expensive or operationally constrained
  • Numbers can accumulate state if cleanup is weak

2. Shared numbers with session correlation

All tests use the same number, but each request includes a unique session ID or test run ID in metadata, a prefix, or message body.

Pros:

  • Fewer numbers to manage
  • Cheaper and easier to centralize

Cons:

  • More risk of collision under parallel CI
  • Requires stronger filtering logic
  • Harder to reason about when tests overlap

3. Provider-specific test message paths

Some providers expose test flows or sandbox semantics. These are useful for unit-level coverage, but they do not always exercise the full messaging path that production uses.

This is where the distinction matters: if your goal is to validate parsing and business logic, test fixtures may be enough. If your goal is to validate real OTP delivery and recovery, you want a live message path and a harness around it.

Webhook handling, the part most teams underbuild

If you use Twilio webhooks, your harness needs to treat the webhook endpoint as a first-class API. That means:

  • Authenticate requests if possible, or at minimum verify provider signatures
  • Make handlers idempotent
  • Store enough metadata to identify the test run
  • Return quickly, do not block on heavy parsing or downstream requests
  • Persist the raw payload for debugging

A webhook handler that writes the entire payload to a queue or database is usually enough. The test runner can then query the latest matching message by session ID.

A practical Node.js handler might look like this:

import express from 'express';

const app = express(); app.use(express.urlencoded({ extended: false }));

app.post(‘/webhooks/sms’, async (req, res) => { const { From, To, Body, MessageSid } = req.body;

await saveInboundMessage({ messageSid: MessageSid, from: From, to: To, body: Body, receivedAt: new Date().toISOString() });

res.sendStatus(200); });

This is intentionally simple. The important part is not the framework, it is the shape of the data and the idempotency policy.

Idempotency rules that matter

Message providers retry delivery when they do not see a timely 2xx response. Your harness must assume duplicate webhook deliveries are normal.

Common pattern:

  • Use provider message ID as a deduplication key
  • Upsert the record instead of inserting blindly
  • Keep the latest status and timestamp history if you need debugging

If you skip this, your tests can pass on one run and fail on the next because the same message was processed twice, then matched against the wrong session.

How to correlate a message to a test run

A test SMS verification harness lives or dies by correlation. The app under test, the webhook listener, and the test runner all need a shared key.

Good options include:

  • A unique session ID in the verification request metadata
  • A test-only email address or phone alias that maps to a run ID
  • A database row keyed by run ID and environment
  • A message body tag in non-production environments, when acceptable

For example, if your app can accept a verification_context value only in test environments, you can match messages to a run without relying on global state.

typescript

const runId = `ci-${process.env.GITHUB_RUN_ID}-${Date.now()}`;
await page.fill('[data-testid="phone"]', '+15551234567');
await page.evaluate((runId) => {
  window.localStorage.setItem('testRunId', runId);
}, runId);

The specific mechanism is less important than one rule: the message record must map deterministically to one run, even when CI runs in parallel.

If two jobs can claim the same OTP, the harness is not reliable enough for parallel execution.

Polling versus webhook delivery

A webhook is fast when it arrives, but message arrival is still asynchronous. For that reason, SMS webhook polling for tests should exist even if you prefer push delivery.

Use webhooks when

  • You control an always-on test receiver
  • You want low latency
  • You can verify signatures or trust the network path

Use polling when

  • Tests run in ephemeral CI environments
  • The webhook receiver might be unavailable when the message arrives
  • You need a fallback that can recover after a brief outage

A robust implementation often does both, webhook writes to storage, polling reads from storage until a timeout expires.

Example polling loop in Playwright:

typescript

async function waitForOtp(fetchOtp: () => Promise<string | null>, timeoutMs = 30000) {
  const started = Date.now();
  while (Date.now() - started < timeoutMs) {
    const otp = await fetchOtp();
    if (otp) return otp;
    await new Promise(r => setTimeout(r, 1000));
  }
  throw new Error('OTP not received within timeout');
}

The key design decision is the timeout. Too short, and you create flakes. Too long, and CI becomes slow and expensive. Most teams settle by observing typical delivery times in their own stack, then adding enough headroom for transient provider delays and CI jitter.

Build the OTP extractor carefully

Most SMS verification codes are short, but extraction logic is where hidden failures accumulate. Your parser should be strict enough to avoid false positives, but flexible enough to survive harmless message template changes.

Recommendations:

  • Match based on message template or sender when possible
  • Support both 4 digit and 6 digit code formats if your product uses both
  • Normalize whitespace and punctuation
  • Reject stale messages explicitly

A safe parser for a six digit OTP might look like this:

function extractOtp(body: string): string | null {
  const match = body.match(/\b(\d{6})\b/);
  return match ? match[1] : null;
}

That parser is not enough by itself. It should only run after the message has been filtered to the correct sender, recipient, and test session.

CI retry logic, when to retry and when to fail

Retries are useful, but they are easy to abuse. A good harness retries only the uncertainty layer, not the application logic.

Retry candidates:

  • Message not yet visible in the store
  • Webhook delivery delayed briefly
  • CI worker or network hiccup

Do not retry endlessly:

  • Invalid OTP format
  • Wrong recipient number
  • Verification rejected by the app
  • Expired code after timeout

A practical rule is to retry message retrieval, not the final verification assertion. That keeps the signal clear.

GitHub Actions can model this with a small retry wrapper around the extraction step:

name: sms-verification-tests
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm test -- --grep "sms verification"

If you need explicit retries, put them in the test helper so the CI log shows each polling attempt and the elapsed time.

Cleanup, the part that saves you later

The easiest way to make a harness noisy is to ignore cleanup. Messages, verification sessions, and test accounts all leave state behind.

Clean up:

  • OTP rows in the test database after verification or expiry
  • Temporary phone aliases or test account flags
  • Webhook message records older than a retention window
  • Any session locks used to prevent parallel collisions

A common pattern is TTL-based cleanup rather than synchronous deletion on every test. That keeps the test path simple and makes failures easier to inspect.

For example, keep raw inbound webhook payloads for 24 to 72 hours, then delete them automatically. That is usually enough to debug flaky runs without turning the store into a permanent archive.

Failure modes you should design for upfront

These are the failure modes that tend to appear after the first few green runs:

1. Duplicate messages

Provider retries or application retries can create multiple OTP messages for the same session. Your harness should pick the newest matching message and ignore duplicates with the same provider ID.

2. Stale OTPs

If a test polls too slowly, it can accidentally use an expired code. Make the extraction step enforce freshness.

3. Out-of-order delivery

The provider may deliver a resend before the initial message, or a webhook may arrive after the test has already retried. Session-level deduplication is important.

4. Parallel CI collisions

Two jobs using the same test number can race. Solve this with per-job metadata, per-run session IDs, or isolated phone numbers.

5. Environment drift

Twilio numbers, webhook URLs, secrets, and message templates differ between dev, staging, and CI. Put these in configuration and validate them at startup.

6. False positives from fixtures

Mocking only the provider response can hide routing or template bugs. Use real delivery for the flows that matter most.

How to observe and debug failures

A useful harness gives you enough evidence to diagnose failures without adding a debugger to every CI run.

Log these fields:

  • Run ID
  • Recipient number alias or last four digits, where appropriate
  • Provider message SID
  • Message timestamp
  • Sender ID or verified sender
  • OTP extraction result, masked in logs
  • Webhook response status and latency

Store the raw payload separately from the structured test log. That way, you can inspect the exact provider event when something goes wrong.

A dashboard is nice, but even a compact JSON file attached to the CI artifacts is enough to avoid guesswork.

Where a custom harness makes sense

Building your own makes sense when the team needs:

  • Deep integration with product-specific verification rules
  • Control over number allocation and cleanup
  • Data residency or security constraints
  • Message inspection beyond a generic platform workflow
  • Tight coupling to an existing CI and test data strategy

It is also reasonable when the SMS flow is core product logic and deserves first-class test coverage in your own stack.

The tradeoff is maintenance. You are now owning provider integration, test storage, retries, observability, and cleanup logic. That ownership can be justified, but it is still ownership.

When maintained tooling is simpler

If your team mainly wants the SMS-driven user flow covered, and not the plumbing itself, a maintained platform can reduce the moving parts. For example, Endtest, an agentic AI test automation platform,’s Email and SMS Testing is built around real phone numbers and editable, platform-native steps for flows like signup, 2FA, login links, and notifications. For teams that do not want to maintain their own SMS receiver, message store, and retry harness, that can be a simpler alternative.

The decision point is usually not “can we build it?” It is “do we want to own the webhook receiver, storage, retries, cleanup, and ongoing provider changes for the long term?”

A practical implementation checklist

Use this as a build checklist for a first version:

  • Create a unique run ID per CI job
  • Provision a dedicated test number or alias strategy
  • Build a webhook receiver that stores raw payloads
  • Deduplicate messages by provider message ID
  • Add a polling API for the test runner
  • Implement OTP extraction with freshness checks
  • Add retry logic only around message arrival
  • Log provider IDs, timestamps, and correlation keys
  • Expire message records and test sessions automatically
  • Validate configuration at startup

Final take

A test SMS verification harness is one of those systems that looks small until it becomes part of every login, signup, and 2FA regression suite. The engineering work is not hard in isolation, but the failure modes are distributed, webhook retries, message latency, correlation mistakes, cleanup leaks, and CI collisions all show up at once.

If your team has the appetite to own that plumbing, build the harness deliberately and treat it like production-adjacent infrastructure. If your goal is simply to test SMS-driven user flows reliably without maintaining the entire message capture stack, a maintained service can remove a meaningful amount of operational burden.

Either way, the right bar is the same, a verification test should tell you whether a real user could complete the flow, not whether a mocked provider returned a convenient response.