Prompting per Model

A prompt written for one voice model can be word for word wrong on another. Not wrong in a way you notice: the prompt still reads well, the assistant still answers, and the rule you wrote simply never happens. That is the failure mode to watch for. A rule that does nothing is worse than a missing rule, because it looks like the matter is handled.

Two questions decide what you can put in a prompt at all.

Is there a separate speech step? A pipeline assistant runs speech recognition, a language model and speech synthesis as three models in a row. The language model writes text, and a speech engine reads that text out loud. How you spell something is therefore how it sounds. A speech-to-speech model hears audio and answers in audio, with no text in between, so spelling rules have nothing to act on.

Who decides the turn? On a pipeline and on most speech-to-speech models, voice activity detection sits in front of the model: when the caller starts speaking, the platform stops the audio and the model is told afterwards. The model never gets a say. On a full-duplex model the model itself hears the caller while it speaks and decides whether to stop. Only there does an instruction about interrupting change anything.

The four families at a glance

The prompt generator follows this table for you. If you write by hand or through the API, this page is what it applies.

Pipeline

The language model writes text that a speech engine reads out literally. The spelling is the sound.

Write for the ear.

• Write every word out in full. No abbreviations, no symbols: "at" instead of @, "percent" instead of %.
• No digits in running text. Numbers become words.
• The digits of a phone number, postcode or reference are spoken one by one, separated by spaces or commas.
• Where a numeral collides with another word in your language, write it so it cannot be misread. In Dutch, "één" with accents, or the speech model reads the article.
• Times and dates in spoken form. Do not ask the assistant to convert a clock time it received: a time that is an hour off is far worse than an ugly one. Give examples instead of a conversion rule.

Commas are the only brake that works. Double commas buy nothing. Ellipses, trailing dots and dots between digits do add pause, but they also add audible artefacts, because punctuation a speech model does not recognise is a risk rather than a rest. For real space between two subjects, write a short ordinary sentence with a full stop in between.

Keep answers short. Every sentence still has to pass through recognition delay and synthesis, so a long answer is also a slow one.

Gemini Live

The instruction is fixed for the life of the call. You cannot correct the model halfway through, so everything the assistant needs has to be in the opening instruction or it does not happen. That raises the cost of a long prompt: a sprawling instruction makes this model drift and stall more, not less, and you cannot take it back.

Order matters. Persona first: name, role, character. If you want a particular accent, name the output language alongside it, because accent and language are entangled inside a single audio model. Then the conversational rules in the order you expect them to run, with the things that happen once separated from the loop that repeats. Then the tool instructions, in separate short sentences rather than one long compound clause: "First ask for the postcode. Then call the lookup. Then read the result back." Then the guardrails, with a short example of right and wrong.

Say the language twice, affirmatively.

RESPOND IN DUTCH.
YOU MUST RESPOND UNMISTAKABLY IN DUTCH.

The model infers the spoken language from what it hears, so the instruction is your only lever.

Describe the pace in words, not in punctuation. "Speak calmly, keep answers to one or two sentences" does what a comma cannot do here.

On gemini-3.8-live, give the assistant an explicit rule about ending the call: hang up only after the caller has actually said goodbye, or after your own sign-off has been acknowledged. Never because the line went quiet.

GPT-Live

This is the one model that decides its own turns, so this is the one model where conversational behaviour really is steered by the prompt. Alongside the usual structure it takes three blocks, each under its own heading. Keep the English headings even in a prompt written in another language: it is the form the model was trained on.

Personality
Speak warmly and naturally, at an unhurried pace. Be clear and direct,
not overly cheerful. If the caller is frustrated, acknowledge it briefly
and move to the next helpful step.

Backchannel policy
Stay silent while the caller is speaking. Anything you say while they talk
sounds right through them. One short acknowledgement is allowed only when
the caller pauses inside a longer story and is clearly waiting for a sign
that you are still there. When in doubt, stay silent.

Interruption policy
Stop speaking the moment the caller interrupts you, and listen. Do not
finish your sentence, and do not repeat the whole interrupted message
afterwards: summarise and continue from where it still matters.

Background voices get their own block. The model distinguishes mainly on content rather than on voice, so tell it what to do rather than expecting it to work it out.

Background voices
Your conversation partner is only the person on the call. A television,
people in the room, or noise from the street are not.
A background voice never interrupts you: keep talking, do not react to it,
and never answer a question from someone else.
If the caller is talking to somebody else, a different name, a different
tone, a different subject, do not respond. Wait.
Do not treat a cough, music, or a nearby conversation as a new request.
If you genuinely cannot tell, do not guess. Ask briefly whether they were
talking to you or to someone else.

If the assistant has a backend model, say plainly which requests go there and which stay with the voice model. Delegate when the request needs a lookup or careful reasoning, or when a correction changes work already in progress. Do not delegate when the answer is already in the conversation, or when a one-line clarification is all that is needed. Delegate before answering, and never guess the outcome while waiting. See OpenAI GPT-Live for how the two sets of instructions are configured.

Ending the call is a hand-off too. On GPT-Live, end_call and transfer_call run on the backend model, so the voice model has to delegate them. Add a rule to delegate right after the goodbye, or the assistant says goodbye and stays on the line. The block to use is under Tools.

Grok Realtime

Treat it as a speech-to-speech model with server-side turn-taking: persona first, rules in execution order, tool instructions in separate sentences, guardrails at the end. Pipeline writing rules do not apply, and neither does anything about interrupting or listening sounds.

For an assistant that has to be steered on conversational behaviour — listening sounds, interruptions, other voices in the room — GPT-Live is the model designed for that, and the section above has the blocks to write.

What holds everywhere

Regardless of model, a phone assistant prompt is better for these:

• One question at a time, and the question is the last thing in the turn. A caller starts answering the moment they hear a question, so anything said after it collides with the answer. Put the context before the question, never after.
• Nothing visual. No markdown, no bullet lists, no URLs, no quotation marks, no emoji. Describe where to find something instead of reading out an address.
• Do not ask for an email address unless the task truly needs one. Spelling one out loud is the most fragile minute of a phone call. A phone number is easier to say and to confirm. If you must ask, give the assistant a way out after two failed attempts rather than a third try.
• Name the caller once or twice, not every sentence.
• Say what you are doing before a lookup, and give the caller a moment.

Checklist before you go live

• Does the prompt contain any rule about interrupting or listening sounds? If the assistant is not on GPT-Live, take it out — it is not carried out.
• Does it contain spelling or punctuation rules? If the assistant is on a speech-to-speech model, take them out — there is no speech engine to steer.
• Is the pace described in words, rather than in commas, on a speech-to-speech model?
• On Gemini Live: is the language rule there twice, and is the prompt short enough to read in one go?
• On GPT-Live: is every "sometimes" bounded by a named condition and a maximum?
• On GPT-Live with end_call or transfer_call: does the prompt start with a delegation block that says to delegate right after the goodbye?
• On a pipeline: zero digits, zero symbols, zero abbreviations in what the assistant speaks?

Related

• AI Generation — the prompt generator applies this page for you
• Assistants — where the prompt and the model are configured
• OpenAI GPT-Live — delegation and the two sets of instructions
• xAI Grok
• Call analysis — steering the conversation towards the data you want to extract