Real-time transcript and Fact generation

​

Overview

The WebSocket Secure (WSS) API enables real-time, bidirectional communication with the Corti system for interaction streaming. Clients can send and receive structured data, including transcripts and facts updates.

This documentation provides a structured guide for integrating the Corti WSS API for real-time interaction streaming.

​

Environment Options

​

Establishing a Connection

Clients must initiate a WebSocket connection using the wss:// scheme and provide a valid interaction ID in the URL.

​

Request

​

Path Parameters

​

Query Parameters

​

Responses

​

101 Switching Protocols

Indicates a successful WebSocket connection. Once connected, the server streams data in the following formats.

​

Data Streams

​

Transcript Stream

{
  "type": "transcript",
  "data": [
    {
    "id": "UUID",
    "transcript": "Patient presents with fever and cough.",
    "final": true,
    "speakerId": -1,
    "participant": { "channel": 0, "role": "doctor" },
    "time": { "start": 1.71, "end": 11.296 }
    }
  ]
}

​

Fact Stream

{
  "type": "facts",
  "data": [
    {
    "id": "UUID",
    "text": "Patient has a history of hypertension.",
    "group": "medical-history",
    "groupId": "UUID",
    "isDiscarded": false,
    "source": "core",
    "createdAt": "2024-02-28T12:34:56Z",
    "updatedAt": "2024-02-28T12:35:56Z"
    }
  ]
}

​

Sending Messages

Clients must send a stream configuration message and wait for a response of type CONFIG_ACCEPTED before transmitting other data.

​

Stream Configuration

​

Example Configuration

{
  "type": "config",
  "configuration": {
    "transcription": {
      "primaryLanguage": "en",
      "isDiarization": false,
      "isMultichannel": false,
      "participants": [
        {
          "channel": 0,
          "role": "multiple"
        }
      ]
    },
    "mode": {
      "type": "facts",
      "outputLocale": "en"
    }
  }
}

Once the server responds with:

{
  "type": "CONFIG_ACCEPTED"
}

Clients can proceed with sending audio or controlling the stream status.

​

Controlling Stream Status

To end the stream, send:

{
  "type": "end"
}

The connection remains open until all transcripts are complete. The server then sends a message of type: "ENDED" and closes the connection.

​

Sending Audio Data

Ensure that your configuration was accepted before starting to send audio and that your initial audio chunk is not too small as it needs to contain the headers to properly decode the audio. We recommend sending audio in chunks of 500ms. In terms of buffering, the limit is 64000 bytes per chunk. Audio data should be sent as raw binary without JSON wrapping.

While we for bandwidth and efficiency reasons recommend utilizing the webm/opus encoding, you can send a variety of common audio formats as the audio you send first passes through a transcoder. Similarly, you do not need to specify any sample rate, depth or other audio settings.

​

Channels, participants and speakers

In a typical on-site setting you will be sending mono-channel audio. If the microphone is a stereo-microphone, you can ensure to set isMultichannel: false and audio will be converted to mono-channel, ensuring no duplicate transcripts are being returned.

In a virtual setting such as telehealth, you would typically have the virtual audio on one channel from webRTC and mix in on a separate channel the microphone of the local client. In this scenario, define isMultichannel: true and assign each channel the relevant participant role, e.g. if the doctor is on the local client and channel 0, then you can set the role for channel 0 to doctor.

Diarization is independent of audio channels and participant roles. If you want transcript segments to be assigned to automatically identified speakers, set isDiarization: true. If false, transcript segments will be returned with speakerId: -1. If set to true, then diarization will try to identify speakers separately on each channel. The first identified speaker on each channel will have transcript segments with speakerId: 0, the second speakerId: 1 and so forth.

​

Error Handling

In case of an invalid or missing interaction ID, the server will return an error before opening the WebSocket.

During a WebSocket session the following messages related to configuration can be returned.

  {"type": "CONFIG_DENIED"} // in case the configuration is not valid
  {"type": "CONFIG_MISSING"}
  {"type": "CONFIG_NOT_PROVIDED"}
  {"type": "CONFIG_ALREADY_RECEIVED"}

In addition, a reason will be supplied, e.g. reason: language unavailable

​

Closing the Connection

To terminate the WebSocket session, send a standard WebSocket close frame, or use:

{
  "type": "end"
}

The connection remains open until all transcripts are complete, at which point the server sends a message of type ENDED and then closes.