Add text to speech to your conversational agent

Beta. This is a starting pattern for pairing a Corti agent with a third-party text to speech provider, not a Corti-hosted feature. Corti does not host, proxy, or bill for text to speech today: your integration connects to the provider you choose directly. Latency, voice quality, and streaming behavior are entirely provider-dependent. Reach out to discuss your use case or share feedback.

This guide assumes a working Corti agent integration is already in place, for example the On-Demand Agent, Wake-Command Agent, or Conversational Agent pattern. It covers only the last step: turning that agent’s text response into audio.

An agent built on the Agentic Framework returns a text response from messageSend. To close the loop into a voice interface, that text needs to become speech the user can hear. This guide walks through pairing your agent with an external text to speech provider: choosing one, sending it the agent’s response, and playing back the result.

Why this is a pairing, not a Corti endpoint

Text to speech models and hosting are widely available and improve quickly, including strong open-source options. Rather than adding a Corti-hosted proxy in front of a fast-moving space, we recommend connecting your agent integration directly to a text to speech provider. That gives you direct control over latency, voice quality, language coverage, and cost, and lets you change providers without waiting on a Corti release.

Choose a text to speech provider

The right provider depends on your priorities. For example, providers with a streaming API (WebSocket or server-sent events) return the first audio chunk well before the full response finishes synthesizing. If low time-to-first-audio matters for your use case, prefer a provider’s streaming endpoint over its plain REST endpoint. A few vendors to consider include the following:

Provider	Good fit when
Cartesia	Low-latency, conversational voice agents; native WebSocket streaming
ElevenLabs	Voice realism and broad multilingual coverage are the priority
OpenAI text to speech	Simple, lower-volume use cases already on the OpenAI platform
Self-hosted / open source (e.g. Chatterbox)	Data residency or on-premises requirements rule out a hosted provider

Verify compliance directly with the provider. The text you send for speech synthesis, including any clinical content in an agent’s response, is processed by that provider under their terms, not Corti’s. Confirm their HIPAA, GDPR, or other data-handling and retention commitments (including any BAA) before sending real patient data through them.

Prompt the agent for speakable text

An agent tuned for a text-based chat interface will often return markdown formatting, bullet lists, or headers, none of which sound right read aloud. Add explicit instructions to the agent’s systemPrompt so its responses are written to be heard, not read:

Speakable-response system prompt

const VOICE_AGENT_PROMPT = `You are a voice assistant. Your responses are converted to speech, so write for listening, not reading.
Use short, complete sentences and plain punctuation.
Never use markdown, bullet points, headers, or emoji.
Spell out symbols instead of using them (say "and" instead of "&", "at" instead of "@").
Keep responses concise: a user hears audio at a fixed pace and cannot skim.`;

Treat this as a starting point, not a final prompt. Test it against real responses from your agent and refine the wording until playback sounds natural for your use case.

Send the agent’s response to your provider

Once messageSend resolves, extract the text from the response and hand it to your chosen provider’s text to speech endpoint. The exact request shape is provider-specific: consult that provider’s docs for authentication, voice selection, and output format. The pattern is the same regardless of provider:

JavaScript

// Note this example shows the v2 A2A message shape directly 
const response = await client.agents.messageSend(agentId, {
  message: {
    role: "ROLE_USER",
    parts: [{ text: userUtterance }],
    messageId: crypto.randomUUID(),
  },
});

// A quick conversational turn returns `message` directly. A long-running task returns `task`, with the reply as the last ROLE_AGENT message in `task.history`.
const replyMessage =
  response.message ??
  [...(response.task?.history ?? [])].reverse().find((m) => m.role === "ROLE_AGENT");
const replyText = replyMessage?.parts.find((part) => "text" in part)?.text;

if (replyText) {
  await speak(replyText);
}

speak is your own function, not part of the Corti API, that calls the chosen provider’s text to speech endpoint with replyText and plays back the resulting audio, for example through the Web Audio API or an <audio> element fed from a MediaSource. Match the playback method to the provider’s output format: compressed formats like mp3 or opus play fine through an <audio> element, but raw PCM output needs the Web Audio API instead.

Guard against empty or partial responses the same way you would for any agent call: if replyText is empty, skip the text to speech request rather than sending blank input to the provider.

Account for end-to-end latency

Time to first audio in this pattern is the sum of three stages: speech to text on the user’s utterance, the agent’s messageSend response time, and the provider’s text to speech synthesis time. Each stage adds up, so:

Prefer providers with a streaming synthesis API to shorten the third stage.
Keep agent responses concise; shorter text reaches the provider’s synthesis step sooner and produces less audio to wait for.
Measure the full pipeline, not just the text to speech call in isolation. A fast provider can still feel slow if it sits behind a slow agent response.

Handle interruptions

Some text to speech providers offer their own barge-in control (for example, a streaming Interrupt or flush/cancel message that stops synthesis server-side). That’s worth using when it’s available, but not every provider offers it, and it only stops the provider’s audio, not your local playback. Detecting the interruption itself doesn’t have to depend on the provider at all: your existing Corti speech to text session can do it, so the same approach works regardless of which provider you paired with above. The core idea: don’t stop listening while the agent’s response plays back. Keep the dictation session running through playback, and treat a new incoming transcript as the user talking over the agent.

Keep the microphone hot during playback

Don’t pause or close the dictation session while speak is running. Track whether playback is active with a simple flag so you can distinguish “the agent is talking” from “the user is talking.”

Turn on echo cancellation

With the speaker and microphone active at the same time, the agent’s own audio can loop back into the mic and be misread as user speech. Set echoCancellation: true on the capture track, the same setting Corti recommends for ambient conversation capture whenever a device’s own speaker output could reach its microphone.

Treat an interim transcript during playback as a barge-in

Listen for the dictation session’s transcript event as usual. If an interim result arrives while isSpeaking is true, the user has started talking over the agent: stop audio playback immediately and cancel the in-flight text to speech request if your provider call is still streaming or generating.

JavaScript

let isSpeaking = false;
let ttsAbortController: AbortController | null = null;

dictation.addEventListener("transcript", (e: CustomEvent) => {
  const { text, isFinal } = e.detail.data;
  if (isSpeaking && text.trim()) {
    interruptAgent();
  }
  if (!isFinal) {
    handleInterim(text);
  } else {
    handleFinal(text);
  }
});

function interruptAgent() {
  isSpeaking = false;
  stopPlayback(); // pause the <audio> element or stop the AudioBufferSourceNode
  ttsAbortController?.abort(); // cancel the provider request if still in flight
  ttsAbortController = null;
}

Feed the new turn through your normal flow

Once the barge-in is handled, the transcript that triggered it continues through the same debounce and turn-finalization logic from the Conversational Agent guide, so it’s sent to the agent as the next turn like any other utterance.

This method stops playback the instant it detects speech, but it doesn’t know how much of the interrupted response the user actually heard. If you need that (for example, to tell the agent what context to assume the user has), check whether your provider reports it: some streaming APIs return spoken-versus-remaining text on interrupt.

Fall back gracefully on provider failures

The text to speech provider is now a dependency your integration doesn’t control. Treat it as one that can fail independently of the agent call: wrap it with a timeout and a fallback so a slow or failed provider request doesn’t stall the conversation or leave the user with nothing.

JavaScript

async function speakOrFallback(replyText: string) {
  try {
    await speak(replyText);
  } catch (err) {
    console.error("Text to speech request failed, falling back to text", err);
    showTextResponse(replyText); // display the reply in the UI instead
  }
}

Set a timeout on the provider request, an AbortController with a budget of a few seconds works for most REST calls, so an unresponsive provider doesn’t stall the turn indefinitely.

What this pattern doesn’t cover

This is a starting point, not a complete voice interface. It does not handle:

Long-running tasks. If the agent needs to do substantial work before responding, this pattern has nothing to say back to the user in the meantime.
SSML or fine-grained prosody control. Handled entirely by your chosen provider, if it supports it.

Next steps

On-Demand Agent

The agent lifecycle and message API, for a single isolated pass.

Wake-Command Agent

Gate the agent behind a spoken wake phrase and hold a multi-turn thread.

Conversational Agent

Always-on voice agent, no wake phrase, every utterance forwarded with speculative prefetch.

Context & Memory

How conversation threads and memory work in the Agentic Framework.

Please contact us for help or questions.

Endpoints

Features

Best Practices

Guides

Resources

Add text to speech to your conversational agent

Why this is a pairing, not a Corti endpoint

Choose a text to speech provider

Prompt the agent for speakable text

Send the agent’s response to your provider

Account for end-to-end latency

Handle interruptions

Fall back gracefully on provider failures

What this pattern doesn’t cover

Next steps

On-Demand Agent

Wake-Command Agent

Conversational Agent

Context & Memory

​Why this is a pairing, not a Corti endpoint

​Choose a text to speech provider

​Prompt the agent for speakable text

​Send the agent’s response to your provider

​Account for end-to-end latency

​Handle interruptions

​Fall back gracefully on provider failures

​What this pattern doesn’t cover

​Next steps

On-Demand Agent

Wake-Command Agent

Conversational Agent

Context & Memory

Why this is a pairing, not a Corti endpoint

Choose a text to speech provider

Prompt the agent for speakable text

Send the agent’s response to your provider

Account for end-to-end latency

Handle interruptions

Fall back gracefully on provider failures

What this pattern doesn’t cover

Next steps