Definition of behaviors

Behavioral signals come from how something is said. For every utterance longer than 1 second, the Behavioral API returns 28 signals, grouped in 10 dimensions: 22 behavioral signals and 6 speaker signals (gender and age). It also returns four voice scores (intensity, breathiness, monotony and fluency) and the quality of the recording (signal_quality.snr, .c50, .sdr and .overall).

📊 The 28 signals at a glance

Plus scores between 0 and 1, which have no labels:

Active signals and baselines

Most signals point to something specific, such as emotion: angry, strength: strong or hesitation: yes. The rest are baselines: neutral (in emotion, strength, positivity and engagement), normal, no, non-whispering and non-shouting. Baselines matter too. A calm, neutral call is a finding in itself.

🗣️ The dimensions

Emotion

The basic emotion the speaker expresses through their voice.

• happy: joy, amusement, warmth or excitement.
• angry: irritation, frustration or anger.
• sad: sadness, disappointment or low mood.
• neutral: no clear emotion.

Strength

Arousal: how much energy and activation the voice carries, whatever the emotion. Anger and excitement are usually strong. Sadness and tiredness are usually weak.

• strong: high energy, an activated voice.
• weak: low energy, a flat or tired voice.
• neutral: normal energy.

Positivity

Valence: the sentiment of the speech, estimated from the tone of voice.

• positive: the tone is pleasant or favorable.
• negative: the tone is unpleasant or unfavorable.
• neutral: neither.

Positivity and emotion often agree, but not always. A calm complaint can be emotion: neutral and positivity: negative.

Speaking rate

How fast or slow the speaker talks, compared to speakers in general. The scale is global: the model learned it from a large speech corpus, and it does not adapt to each speaker's own pace.

• fast: faster than most speakers.
• slow: slower than most speakers.
• normal: a typical pace.

Hesitation

Whether the speech shows signs of hesitation, such as pauses, fillers ("uh", "um"), restarts or drawn-out words.

• yes: the speech shows signs of hesitation.
• no: no signs of hesitation.

Engagement

Whether the speaker sounds involved in the conversation.

• engaged: the speaker sounds interested and involved.
• withdrawn: the speaker sounds detached or uninterested.
• neutral: neither.

Whispering

Whether the speaker whispers, i.e. speaks without voicing.

• whispering: the speech is whispered.
• non-whispering: normal voiced speech.

Shouting

Whether the speaker shouts or yells.

• shouting: the speech is shouted.
• non-shouting: speech that is not shouted, including loud or emotional speech.

Gender

The sex of the speaker, estimated from the voice.

• female
• male

Age

The age range of the speaker, estimated from the voice.

• 18 - 22
• 23 - 30
• 31 - 45
• 46 - 65

📏 The scores

The scores below have no labels and no finalLabel. Their prediction holds a single score between 0 and 1.

Intensity

How intense the expressed emotion is. Higher means more intense. It does not say which emotion is intense: for a score of anger, use the angry probability of emotion.

Breathiness

How breathy the voice sounds, i.e. how much audible air there is in the voice. Higher means breathier.

Monotony

How monotone the delivery is: 1 means flat and monotone, 0 means lively and expressive.

Fluency

How fluent the speech sounds to a listener: 1 means smooth and effortless, lower values mean more pauses, fillers, repetitions or restarts. It complements hesitation, which is a yes / no signal.

Signal quality

The quality of the recording rather than of the voice, as four scores where 1 is clean. Each is its own result, named signal_quality.<output>. Low scores tell you when the other results may be less reliable.

• signal_quality.snr: background noise. 1 means no audible noise.
• signal_quality.c50: reverberation. 1 means a dry recording, without echo.
• signal_quality.sdr: clipping. 1 means no clipping.
• signal_quality.overall: one score for the recording. It is close to the worst of the three, so one bad aspect is enough to lower it.

ℹ️ Reading a result

Each dimension comes back as its own result, one per utterance (or per segment, when streaming). The prediction list holds every signal of that dimension with its probability (posterior). The probabilities of one dimension add up to 1. finalLabel is the one answer the API gives for the dimension.

{
  "id": "0",
  "startTime": "0.031",
  "endTime": "6.427",
  "task": "engagement",
  "prediction": [
    { "label": "withdrawn", "posterior": "0.7433" },
    { "label": "neutral",   "posterior": "0.2427" },
    { "label": "engaged",   "posterior": "0.014" }
  ],
  "finalLabel": "withdrawn",
  "level": "utterance"
}

finalLabel is the label with the highest probability. Use finalLabel when you need one answer, and the probabilities when you need a score, for example to track how angry rises over a call.

The signal quality scores come as four results of their own, one after the other, like any other score:

{ "id": "0", "startTime": "0.031", "endTime": "6.427", "task": "signal_quality.snr",     "prediction": [{ "score": "0.8812" }], "level": "utterance" },
{ "id": "0", "startTime": "0.031", "endTime": "6.427", "task": "signal_quality.c50",     "prediction": [{ "score": "0.4105" }], "level": "utterance" },
{ "id": "0", "startTime": "0.031", "endTime": "6.427", "task": "signal_quality.sdr",     "prediction": [{ "score": "0.9530" }], "level": "utterance" },
{ "id": "0", "startTime": "0.031", "endTime": "6.427", "task": "signal_quality.overall", "prediction": [{ "score": "0.4321" }], "level": "utterance" }

All results of one utterance share its id, startTime and endTime. In batch results, gender and age are given per speaker: every utterance of a speaker longer than 1 second gets the same values, averaged over the file. When streaming, gender and age are estimated on each segment.

The probabilities and scores are strings, so convert them to numbers first. Each one is a score from 0 to 1. positivity is valence and strength is arousal, the two dimensions often used in emotion research. Valence and arousal are often given on a scale from -1 to 1. To get that scale, subtract the probabilities of the two opposite labels: positive minus negative for valence, and strong minus weak for arousal. All probabilities and scores are model estimates: use them to compare utterances and follow trends, not as exact measurements.

👥 Other outputs

Besides the outputs above, the API also returns speech attributes with open-ended values: diarization (who speaks), asr (what is said) and language. Speaker IDs such as SPEAKER_00 are generic: the API does not say who is the agent and who is the customer. With embeddings enabled, it also returns speaker embeddings and behavioral embeddings (features). See Retrieve results for the full response schema.

🧑‍💻 Our Technology

Our models estimate each signal from how the voice sounds, not from the words. Each dimension has its own output, which is why every utterance gets a full profile and not a single label.

You can learn more on how to send a file in Submit audio file, or stream live audio with Streaming using the Python SDK.