Arena: test calls

Arena calls your assistant over the phone line, the way your callers reach it. A test caller plays a persona, works towards a goal, and can be steered: background noise, a voice in the room, an interruption on a given turn, a silence, key presses, a hang-up. After the call every criterion gets a verdict with its evidence.

Arena is in beta and available on every account.

Test calls are real calls. They run on the same platform as your other calls and are billed the same way, on both legs. An attempt with a criterion that a model judges (must, must_not, audio_must) costs one credit on top for the judge.

How a test call works

1. A suite ties one assistant to one of your phone numbers that routes to it, and to a second number of yours to call from.
2. A test case describes one call: who calls (persona), what they want (goal), what they know (facts and entities), what happens during the call (timeline), what each tool does during the test (tool mode), and what the call is judged on (criteria).
3. A run calls every active case one or more times. Each call is an attempt.
4. The test caller waits until your number answers and checks that the call reached your assistant. If it did not (a person, a voicemail or a phone system answered), it hangs up within five seconds without saying anything, and the attempt does not count.
5. After the call, each criterion gets a verdict: pass, fail or not_applicable.

Requirements

• Arena on for your organization. It is on for every organization. If it has been switched off for yours, every /arena endpoint returns 403 with the error Arena is in beta and not enabled for this organization. Contact support to enable it. When it is on for a whitelabel partner, its child organizations can use it too.
• Two numbers. The number under test must route to the assistant. The test caller calls from another number of yours, over your own SIP trunk, under the same rule as every outbound call. A managed number cannot place calls.
• The caller ID must arrive unchanged. A test call is recognized by the caller number plus an active attempt. If your provider or phone system rewrites the caller ID, the test caller hangs up and the attempt is void with caller_id_mismatch or not_routed_to_assistant.
• Credits. Before a run starts, its cost is estimated from every attempt lasting its full max_duration_s, on both legs, plus the judge where a model decides a criterion. The run is refused if your balance does not cover the estimate. It is an estimate, not an upper bound: a run costs what its calls and its judging cost.

Create a suite

curl -X POST https://api.hmsovereign.com/api/v1/arena/suites \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Core set",
    "assistant_id": "ASSISTANT_ID",
    "number_id": "NUMBER_OF_THE_ASSISTANT",
    "caller_number_id": "NUMBER_TO_CALL_FROM"
  }'

Write a test case

curl -X POST https://api.hmsovereign.com/api/v1/arena/suites/SUITE_ID/cases \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "key": "book-appointment",
    "status": "active",
    "goal": "You want to book an appointment for a leak in your ceiling.",
    "persona_id": "customer-clear",
    "language": "en",
    "audio_profile_id": "street",
    "entities": [{ "key": "postcode", "label": "postcode", "value": "7411 AB" }],
    "timeline": [{ "type": "silence", "at_turn": 2, "duration_s": 6 }],
    "tool_mode": {
      "book_appointment": { "mode": "mock", "response": { "booked": true, "time": "14:00" } }
    },
    "criteria": [
      { "id": "books", "kind": "tool_required", "tool": "book_appointment" },
      { "id": "postcode", "kind": "entity_recall", "entity": "postcode" },
      { "id": "names-company", "kind": "must", "text": "The assistant says the name of the company" },
      { "id": "fast", "kind": "latency_max", "metric": "latency_p95", "max_ms": 1500 }
    ]
  }'

Saving with an existing key creates a new version, unless the definition is the same as the latest version: then you get that version back, so you can send your cases from a repository on every run. A version that was used in a run never changes, so an old result always refers to the case it was run with. A case starts as proposed unless you save it as active; only active cases are run. Use PATCH /arena/cases/{case_id} to activate or retire a version.

GET /arena/catalog lists the personas, audio profiles, timeline events with their fields, channels, criterion kinds and metrics.

The test caller

Entities are said the way you write them in spoken. Without spoken, digits and letters are spelled out one by one.

Timeline

An event happens at a time (at_s, seconds after the call was answered) or on a turn (at_turn, the n-th turn of your assistant), never both.

audio_profile_id sets the background noise for the whole call, at a fixed signal-to-noise ratio. channel processes the caller's voice before it reaches the line: speakerphone, muffled or packet_loss.

Tools during a test

Every tool of the assistant needs a choice in tool_mode. There is no default: a test that books real appointments or places real orders would affect your customers.

A tool without a choice is not run during a test call; the assistant gets an error. A transfer is never placed during a test: the attempt is recorded, with its destination, and the call ends. Set a tool from an MCP server to live under its tool name: it then runs as in a normal call.

Criteria

What platform data or the audio can decide is never handed to a language model. An entity is checked in four steps: whether the caller said it, whether your assistant understood it, whether it ended up in a tool argument, and whether your assistant repeated it.

Run and read the results

curl -X POST https://api.hmsovereign.com/api/v1/arena/suites/SUITE_ID/runs \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "repeats": 3 }'

The response contains the run and its estimate. A suite has one run at a time, and a run has at most 50 attempts. GET /arena/runs/{run_id} returns the attempts with their verdicts, metrics and a recording per track: the test caller, what it heard from your assistant, what your assistant received, and what the timeline played. The recordings are deleted with the calls, under your retention period.

POST /arena/runs/{run_id}/cancel cancels a run. Attempts that are still in the queue are not called and end as void; an attempt that is in progress is finished. The run then ends as cancelled, also when an attempt was still in progress at the moment you cancelled it.

A void attempt does not count and always carries a void_reason:

Routines

A routine runs the active cases of a suite on a schedule, and optionally after a change to your assistant or a platform release. It is off until you enable it, and it always has a monthly budget.

curl -X PUT https://api.hmsovereign.com/api/v1/arena/suites/SUITE_ID/schedule \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "cadence": "daily",
    "window_start": "02:00",
    "window_end": "05:00",
    "timezone": "Europe/Amsterdam",
    "repeats": 3,
    "monthly_budget_eur": 25,
    "on_config_change": true
  }'

GET /arena/suites/{suite_id}/schedule shows the routine with spent_this_month_eur: the actual cost of its finished runs this month, plus the estimate of runs in progress.

Trend and regressions

GET /arena/suites/{suite_id}/trend returns, per case and criterion, the passed and judged attempts of each of the latest runs, with a 95% Wilson interval. Results are never added up across criteria: a total score hides a drop in a single criterion.

A regression is a drop of at least 10 percentage points against the runs of the 14 days before, that is significant (Fisher exact, p < 0.05). A criterion with severity: critical regresses on a single failure. Void attempts never count. current_config_hash against the config_hash of a run tells you whether a result is outdated.

Events

Arena sends events only for a suite that has a routine, also while that routine is off or paused, and only to an https URL: the webhook_url of the routine, or else the webhook of your organization. A suite without a routine sends no events, however you start its runs; its results are in the dashboard and in GET /arena/runs/{run_id}. Events are sent within a few minutes of what they report.

Regressions and pauses are also e-mailed, unless notify_email is off; that does not depend on a webhook URL.

The envelope

The body has the same message envelope as the call webhooks, with arena in place of the call fields:

{
  "message": {
    "id": "4f1e9a52-7c3b-4d0e-9b8a-2c6d5e1f0a93",
    "type": "arena.regression.detected",
    "timestamp": "2026-10-20T02:14:09Z",
    "arena": {
      "suite_id": "0b7c3e2a-5d41-4f8e-a6c9-3e2f1d0b9a87",
      "schedule_id": "9a8b7c6d-5e4f-4a3b-9c2d-1e0f9a8b7c6d",
      "run_id": "6c5d4e3f-2a1b-4c0d-9e8f-7a6b5c4d3e2f",
      "case_key": "book-appointment",
      "criterion_id": "books",
      "critical": false,
      "current": { "passed": 6, "judged": 10 },
      "baseline": { "passed": 29, "judged": 30 },
      "p_value": 0.0011,
      "config_changed": true
    }
  }
}

Signature

Events are signed the same way as the call webhooks. X-Webhook-Timestamp holds the Unix time in seconds, and X-Webhook-Signature holds sha256= followed by the hex HMAC-SHA256 of "{timestamp}.{raw body}". The key is the webhook secret of your organization, also when the routine has its own webhook_url. Without an organization secret, events are sent without X-Webhook-Signature. See Webhook Security for how to verify a signature.

Delivery and retries

A delivery succeeds when your endpoint answers with a 2xx status within 10 seconds. Otherwise the event is sent again, with waits of at least 1 minute, 5 minutes, 30 minutes, 2 hours and 12 hours: at most six attempts in total. Each attempt has the same body and a new X-Webhook-Timestamp and signature. Because an attempt can arrive after your endpoint already handled the event, use message.id to drop duplicates.

Recognize a test call in your backend

On the tested assistant, every webhook of a test call carries call.arena with attempt_id, run_id, case_id and suite_id. Use it to keep test calls out of your own systems. Test calls do not show in your analytics; they do show in your call list and usage. No end-of-call email is sent for a test call.