Continuous valence and arousal

A valence and an arousal score for every utterance

Ask for continuous values to get a score for every utterance (and, in streaming, every segment), from -1 to 1:

ValueWhat it measures
valenceHow positive the speaker sounds: -1 is very negative, 1 is very positive
arousalHow activated the speaker sounds: -1 is very calm, 1 is very activated

Each score is combined from several behavioral signals. Unlike Focus, there is no threshold and no sensitivity: every utterance with behavioral results gets a value.

Continuous values are available for the Behavioral API, in batch and streaming.

🌐 cURL Example (batch)

Submit your audio as usual. When processing is complete, add the continuous query param to the results request:

curl --request GET \
     --url 'https://api.behavioralsignals.com/v5/clients/your-client-id/processes/pid/results?continuous=valence,arousal' \
     --header 'X-Auth-Token: your-api-token' \
     --header 'accept: application/json'

The values are computed when you read the results, so they also work on processes you submitted before, without uploading the audio again. You can combine them with Focus, e.g. ?focus=negative_tone&continuous=valence,arousal.

🐍 Python SDK Example

Batch:

from behavioralsignals import Client

client = Client()

process = client.behavioral.upload_audio(file_path="audio.wav")
result = client.behavioral.wait_for_result(
    pid=process.pid,
    continuous=["valence", "arousal"],
)

for r in result.results:
    if r.task in ("continuous_valence", "continuous_arousal"):
        print(r.st, r.et, r.task, float(r.prediction[0].score))

Streaming:

from behavioralsignals import Client, StreamingOptions
from behavioralsignals.utils import make_audio_stream

client = Client()
audio_stream, sample_rate = make_audio_stream("audio.wav", chunk_size=0.25)

options = StreamingOptions(
    sample_rate=sample_rate,
    encoding="LINEAR_PCM",
    continuous=["valence", "arousal"],
)

for response in client.behavioral.stream_audio(audio_stream=audio_stream, options=options):
    for r in response.results:
        if r.task in ("continuous_valence", "continuous_arousal"):
            print(r.st, r.et, r.task, float(r.prediction[0].score))

In streaming, values come with both segment and utterance results: use the level option to choose which ones you get.

Streaming with gRPC

Add a continuous field to the AudioConfig of the first message (see the proto file):

config = pb.AudioConfig(
    encoding=pb.AudioEncoding.LINEAR_PCM,
    sample_rate_hertz=16000,
    continuous=["valence", "arousal"],
)

Response

Each value is one extra result with task set to continuous_valence or continuous_arousal, and the value in score. It has the same id, startTime, endTime and level as its utterance or segment:

{
  "id": "3",
  "startTime": "12.48",
  "endTime": "16.02",
  "task": "continuous_arousal",
  "prediction": [
    {
      "score": "0.6124"
    }
  ],
  "level": "utterance"
}

Like other score tasks, the score is a string, so convert it to a number first. There is no finalLabel. In batch results, the values come right after the other results of their utterance.

📘

Notes

  • A process that never used continuous values returns the same results as before.
  • Utterances shorter than about 1 second have no behavioral results, so they get no value.
  • An unknown value is rejected: with 400 in batch and INVALID_ARGUMENT in streaming. In batch, an empty continuous is rejected too.
  • In streaming, send continuous in the first message: once it is set, later values are ignored. Continuous values are not available for deepfake detection streams.
  • In batch, the values are not stored: they are added to the response each time you ask for them. A stream's values are stored with its results, and asking for continuous again replaces them.
  • The scores are model estimates: use them to compare utterances and follow trends, not as exact measurements.


Did this page help you?