Get results after completion
You can get results for various levels of processing
📄 Description of the results
The API provides advanced audio analysis capabilities, offering a range of predictions and insights about spoken content. It supports tasks such as speaker diarization, automatic speech recognition (ASR), language identification, and various behavioral and vocal attributes. It assesses gender and age for the detected speakers, emotion, strength (arousal) and positivity of speech, speaking rate, levels of hesitation and engagement, speaking styles such as whispering and shouting, voice scores such as intensity, breathiness, monotony and fluency, and the signal quality of the recording.
Furthermore, our latest update includes deepfake detection. For more information visit the Deepfake Detection API section of the documentation. For this quickstart guide we'll focus on the behavioral API.
Each task returns detailed predictions with confidence scores, enabling comprehensive analysis and understanding of audio data. This rich set of features supports applications in fields like customer service, content analysis, and behavioral research.
The following is the list of attributes that the API calculates:
-
Speaker Diarization:
- Identifies utterances of different speakers within an audio clip.
- Labels each utterance with a unique speaker identifier.
- Marks the start and end time of each speaker turn
-
Automatic Speech Recognition (ASR):
- Transcribes spoken words into text.
- Provides accurate textual representation of the audio content.
-
Language Identification:
- Determines the language used in the audio.
- Returns language labels with confidence scores for multiple languages.
-
Emotion Detection:
- Analyzes the emotional tone of the speech.
- Classifies emotions such as "angry," "neutral," "happy," and "sad."
-
Strength Detection:
- Analyzes the strength (arousal) of the speech.
- The levels are: "weak", "neutral" and "strong"
-
Gender and Age Prediction:
- Assesses the speaker's gender and age range.
- Gives one value per speaker for the whole file, with a probability for each label.
-
Positivity Analysis:
- Evaluates the positivity (valence) of each utterance.
- Classifies speech as "positive," "neutral," or "negative."
-
Speaking Rate:
- Measures the rate of speech.
- Labels the speech as "slow," "normal," or "fast."
-
Hesitation Detection:
- Identifies moments of hesitation in speech.
- Classifies each utterance with "yes" or "no" for hesitation.
-
Engagement Assessment:
- Analyzes the level of engagement of the speaker.
- Classifies engagement as "engaged," "neutral," or "withdrawn."
-
Speaking Style:
- Detects whispered and shouted speech ("whispering" / "non-whispering", "shouting" / "non-shouting").
- Scores breathiness, monotony and fluency from 0 to 1.
-
Intensity:
- Scores how intense the expressed emotion is, from 0 to 1.
-
Signal Quality:
- Scores the quality of the recording from 0 to 1: background noise, reverberation, clipping and an overall score.
Notes
- Due to model limitations, behavioral results (
emotion,strength, etc.) are available in every utterance longer than 1 sec. For shorter utterances, onlydiarization,asrandlanguageare available.
🖥️ Retrieving the results - without writing code
If you submitted the audio through the Deepfakes Dashboard, the results appear there once processing completes: per-utterance behavioral insights and a deepfake verdict.
🌐 Retrieving the results - using the API
Once the status of your job changes to “2: completed” you can send a GET results request, using the unique pid, that returns a media type/JSON containing the results.
curl --location 'https://api.behavioralsignals.com/v5/clients/<your-client-id>/processes/<pid>/results' \
--header 'X-Auth-Token: <your-api-token>'Example response:
{
"pid": <your-pid>,
"cid": <your-cid>,
"code": 2,
"message": "Processing Complete",
"modelVersion": "7.7.0",
"results": [
{
"id": "0",
"startTime": "0.069",
"endTime": "4.67",
"task": "asr",
"prediction": [
{
"label": " Life will give you whatever experience is most helpful for the evolution of your consciousness."
}
],
"finalLabel": " Life will give you whatever experience is most helpful for the evolution of your consciousness.",
"level": "utterance"
},
{
"id": "0",
"startTime": "0.069",
"endTime": "4.67",
"task": "diarization",
"prediction": [
{
"label": "SPEAKER_00"
}
],
"finalLabel": "SPEAKER_00",
"level": "utterance"
},
{
"id": "0",
"startTime": "0.069",
"endTime": "4.67",
"task": "language",
"prediction": [
{
"label": "en",
"posterior": "0.9970703125"
},
{
"label": "la",
"posterior": "0.0005102157592773438"
},
{
"label": "zh",
"posterior": "0.0003731250762939453"
}
],
"finalLabel": "en",
"level": "utterance"
},
{
"id": "0",
"startTime": "0.069",
"endTime": "4.67",
"task": "gender",
"prediction": [
{
"label": "female",
"posterior": "0.9555"
},
{
"label": "male",
"posterior": "0.0445"
}
],
"finalLabel": "female",
"level": "utterance"
},
{
"id": "0",
"startTime": "0.069",
"endTime": "4.67",
"task": "age",
"prediction": [
{
"label": "46 - 65",
"posterior": "0.5991"
},
{
"label": "18 - 22",
"posterior": "0.3442"
},
{
"label": "31 - 45",
"posterior": "0.0567"
},
{
"label": "23 - 30",
"posterior": "0.0"
}
],
"finalLabel": "46 - 65",
"level": "utterance"
},
{
"id": "0",
"startTime": "0.069",
"endTime": "4.67",
"task": "emotion",
"prediction": [
{
"label": "angry",
"posterior": "0.3789"
},
{
"label": "neutral",
"posterior": "0.3691"
},
{
"label": "happy",
"posterior": "0.2325"
},
{
"label": "sad",
"posterior": "0.0196"
}
],
"finalLabel": "angry",
"level": "utterance"
},
{
"id": "0",
"startTime": "0.069",
"endTime": "4.67",
"task": "positivity",
"prediction": [
{
"label": "neutral",
"posterior": "0.5427"
},
{
"label": "negative",
"posterior": "0.3294"
},
{
"label": "positive",
"posterior": "0.128"
}
],
"finalLabel": "neutral",
"level": "utterance"
},
{
"id": "0",
"startTime": "0.069",
"endTime": "4.67",
"task": "strength",
"prediction": [
{
"label": "neutral",
"posterior": "0.5839"
},
{
"label": "strong",
"posterior": "0.3992"
},
{
"label": "weak",
"posterior": "0.0169"
}
],
"finalLabel": "neutral",
"level": "utterance"
},
{
"id": "0",
"startTime": "0.069",
"endTime": "4.67",
"task": "speaking_rate",
"prediction": [
{
"label": "fast",
"posterior": "0.5784"
},
{
"label": "normal",
"posterior": "0.3864"
},
{
"label": "slow",
"posterior": "0.0351"
}
],
"finalLabel": "fast",
"level": "utterance"
},
{
"id": "0",
"startTime": "0.069",
"endTime": "4.67",
"task": "hesitation",
"prediction": [
{
"label": "no",
"posterior": "0.9661"
},
{
"label": "yes",
"posterior": "0.0339"
}
],
"finalLabel": "no",
"level": "utterance"
},
{
"id": "0",
"startTime": "0.069",
"endTime": "4.67",
"task": "engagement",
"prediction": [
{
"label": "engaged",
"posterior": "0.8753"
},
{
"label": "neutral",
"posterior": "0.1239"
},
{
"label": "withdrawn",
"posterior": "0.0008"
}
],
"finalLabel": "engaged",
"level": "utterance"
},
{
"id": "0",
"startTime": "0.069",
"endTime": "4.67",
"task": "intensity",
"prediction": [
{
"score": "0.287"
}
],
"level": "utterance"
}
]
}
🐍 Retrieving the results - using the SDK
With the Python SDK you can get the results of an individual process as such:
from behavioralsignals import Client
from behavioralsignals.utils import print_results
client = Client()
result = client.behavioral.get_result(pid=12345) # your process ID
print_results(result.results)result is an object of type ResultResponse (defined here), with the same schema as the JSON above.
Updated 1 day ago