Is it possible to get subtitles or a timed script with batch synthesis text to speech avatar?

d m 5

Using batch text-to-speach or batch avatar API, is it possible to get subtitles on the generated video? Or even better, getting a script of the text with time stamps.

I was hoping to do some front end shenanigans by creating texts highlights, as the text gets read, but I would probably need that time-stamped text for that.

Can i do this with this Azure's API? Or would I need to do something like using speech to text on my generated audio to get that "time-stamped script"?

1 answer

Sina Salam 14,161

Hello d m,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

For more clarity about your questions on batch synthesis text to speech avatar in Azure AI Speech:

Is it possible to get subtitles or a timed script with batch synthesis text to speech avatar?

Yes, you can get a timed script using Azure's Batch Synthesis API for text-to-speech avatars - https://learn.microsoft.com/en-us/azure/ai-services/speech-service/text-to-speech-avatar/batch-synthesis-avatar-properties)

is it possible to get subtitles on the generated video Or even better, getting a script of the text with time stamps.

API does not directly generate subtitles, you can use the timing details from the synthesis summary to create subtitles - https://learn.microsoft.com/en-us/azure/ai-services/speech-service/text-to-speech-avatar/batch-synthesis-avatar

Can i do this with this Azure's API? Or would I need to do something like using speech to text on my generated audio to get that "time-stamped script"?

You can achieve this with Azure's API by leveraging the timing details provided in the synthesis summary. Check the two links above.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

d m 5 Reputation points

2024-09-16T13:53:47.8866667+00:00
Hello, my original question was very vague, I'm sorry for that.

Here is what I meant:

From what my goal is and from what i gather are the API capabilities, my current goal is to get the [nnnn].word.json file when using batch synthesis for avatar text-to-speech.

I've tried to enable word bounday to get the time-stamped "script" when generating an avatar text-to-speech video.

Everything is running fine but the only outputs im getting are the video and the standard summary.json file

Here is my script:

# Function to submit a batch synthesis job def

When i say I'm not getting the [nnnn].word.json file I'm refering this:
""" If sentence boundary data was requested ("sentenceBoundaryEnabled": true), then a corresponding [nnnn].sentence.json file is included in the results. Likewise, if word boundary data was requested ("wordBoundaryEnabled": true), then a corresponding [nnnn].word.json file is included in the results.

Here's an example word data file with both audio offset and duration in milliseconds:

[ { "Text": "The", "AudioOffset": 50, "Duration": 137 }, { "Text": "rainbow", "AudioOffset": 200, "Duration": 350 }, { "Text": "has", "AudioOffset": 562, "Duration": 175 }, { "Text": "seven", "AudioOffset": 750, "Duration": 300 }, { "Text": "colors", "AudioOffset": 1062, "Duration": 625 }, { "Text": ".", "AudioOffset": 1700, "Duration": 100 } ]

"""
which I found here: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/batch-synthesis#batch-synthesis-results

As you can see in my code, I included the "wordBoundaryEnabled"= True in the payload properties, yet I'm not getting the "[nnnn].word.json" file, just the video and the summary.json file, which is just this:

{ "jobID": "9efb93a4-0f1d-4a02-8374-74cf640b9f16", "status": "Succeeded", "results": [ { "contents": [ "A recente onda de calor em Portugal levou a um aumento significativo de alertas de incêndio em várias regiões do país. O Instituto Português do Mar e da Atmosfera (IPMA) emitiu avisos vermelhos para os distritos de Castelo Branco, Santarém e Portalegre, devido às temperaturas que ultrapassaram os 40 graus Celsius. " ], "status": "Succeeded", "videoFileName": "4bbb121d760b4701bbf01aa7b4fdbea7/ceaa743a-db15-4f29-ad36-cb2627fc3f58/0001.webm", "TalkingAvatarCharacter": "max", "TalkingAvatarStyle": "formal" } ] }

d m 5

I've tried to enable word boundary to get the time-stamped "script" when generating an avatar text-to-speech video.

Everything is running fine but the only outputs im getting are the video and the standard summary.json file

# Function to submit a batch synthesis job
def submit_synthesis(text, voice, character, style, output_container_url, video_crop):
    synthesis_id = str(uuid.uuid4())  # Generate a unique synthesis ID
    url = f'https://{SERVICE_REGION}.api.cognitive.microsoft.com/avatar/batchsyntheses/{synthesis_id}?api-version=2024-08-01'
    
    headers = {
        'Ocp-Apim-Subscription-Key': SUBSCRIPTION_KEY,
        'Content-Type': 'application/json'
    }

    # Payload with avatarConfig for video generation
    payload = {
        "inputKind": "PlainText",
        "synthesisConfig": {
            "voice": voice  # Voice chosen by the user
        },
        "inputs": [
            {
                "content": text
            }
        ],
        "avatarConfig": {
            "talkingAvatarCharacter": character,  # Character chosen by user
            "talkingAvatarStyle": style,  # Style chosen by user
            "videoFormat": "webm",  # Specify video format
            "videoCodec": "vp9",  # Codec for video
            "subtitleType": "soft_embedded",  # Subtitles option
            "backgroundColor": "transparent",  # Background for avatar
            "videoCrop": video_crop  # Set the appropriate video crop based on the user's selection
        },
        "properties": {
            "destinationContainerUrl": output_container_url,  # Blob storage URL with SAS token
            "timeToLiveInHours": 744,  # How long to keep the job
            "wordBoundaryEnabled": True

        }
    }

    response = requests.put(url, headers=headers, json=payload)
    
    if response.status_code == 201:
        logger.info('Batch avatar synthesis job submitted successfully')
        job_id = response.json()["id"]
        return job_id
    else:
        logger.error(f'Failed to submit batch avatar synthesis job: {response.text}')
        return None

# Function to check the synthesis job status
def get_synthesis_status(job_id):
    url = f'https://{SERVICE_REGION}.api.cognitive.microsoft.com/avatar/batchsyntheses/{job_id}?api-version=2024-08-01'
    headers = {
        'Ocp-Apim-Subscription-Key': SUBSCRIPTION_KEY
    }

    response = requests.get(url, headers=headers)
    if response.status_code == 200:
        # Check and print the full response for debugging purposes
        job_status = response.json()
        logger.info(f'Job status response: {job_status}')
        return job_status
    else:
        logger.error(f'Failed to get batch synthesis job status: {response.text}')
        return None

# Streamlit interface
def main():
    st.title("Azure Talking Avatar Video Generator")
    
    # Language and voice selection
    lang = st.selectbox('Choose the language', list(lang_voices.keys()))
    voice = lang_voices[lang][1]
    
    # User input for text
    text_input = st.text_area(f'Type the text in {lang}:')

    # Character selection
    selected_character = st.selectbox('Choose the avatar character:', list(character_styles.keys()))
    
    # Dynamically update styles based on character selection
    available_styles = character_styles[selected_character]
    selected_style = st.selectbox('Choose the avatar style:', available_styles)
    
    output_container_url = st.text_input("Enter Output Container URL (with SAS Token)")

    # Add an option to select avatar view (Full Body, Top Half, Head)
    avatar_view = st.selectbox('Choose avatar view', ["Full Body", "Top Half", "Head"], index=0)

    # Adjust video cropping based on avatar view selection
    if avatar_view == "Full Body":
        video_crop = {
            "topLeft": {"x": 0, "y": 0},
            "bottomRight": {"x": 1920, "y": 1080}
        }
    elif avatar_view == "Top Half":
        video_crop = {
            "topLeft": {"x": 0, "y": 0},
            "bottomRight": {"x": 1920, "y": 540}
        }
    else:  # Head
        video_crop = {
            "topLeft": {"x": 700, "y": 0},
            "bottomRight": {"x": 1220, "y": 500}
        }

    submit_button = st.button("Submit Job")
    
    if submit_button:
        if text_input.strip() and output_container_url.strip():
            # Check if selected style is valid for the character
            if selected_style in character_styles[selected_character]:
                with st.spinner("Processing..."):
                    job_id = submit_synthesis(text_input, voice, selected_character, selected_style, output_container_url, video_crop)
                    if job_id:
                        st.success(f'Job submitted! Job ID: {job_id}')
                        st.info("Checking job status. Please wait...")
                        while True:
                            job_status = get_synthesis_status(job_id)
                            
                            # Check for a valid response and process it
                            if job_status and job_status["status"] == "Succeeded":
                                st.success("Avatar video generated successfully!")
                                
                                # Log the full response to ensure correct output
                                logger.info(f'Full job status response: {job_status}')
                                
                                # Extract video URL from the response
                                destination_container_url = job_status['properties'].get('destinationContainerUrl', '')
                                relative_video_path = job_status['outputs'].get('result', '')

                                # Construct the full video URL
                                if destination_container_url and relative_video_path:
                                    # Ensure no double slashes in the final URL
                                    if destination_container_url.endswith('/'):
                                        destination_container_url = destination_container_url[:-1]
                                    
                                    video_url = f"{destination_container_url}/{relative_video_path}"
                                    
                                    # If the video URL is valid, display the video
                                    if video_url.startswith("http"):
                                        st.markdown(f"Download the avatar video [here]({video_url})")
                                        st.video(video_url)
                                    else:
                                        st.error("Video URL seems invalid or incomplete.")
                                else:
                                    st.error("Video URL seems invalid or incomplete.")
                                break
                            
                            elif job_status and job_status["status"] == "Failed":
                                st.error("Failed to generate the avatar video.")
                                break
                            else:
                                time.sleep(5)  # Wait for a few seconds before checking again
            else:
                # Show a warning if the style is not compatible with the selected character
                st.warning(f'The character "{selected_character}" cannot use the chosen style. '
                        f'Please choose from the available styles: {", ".join(available_styles)}')
        else:
            st.error("Please provide the text and output container URL.")

When i say I'm not getting the "timed script" (aka the [nnnn].word.json file), I'm refering this:
""" If sentence boundary data was requested ("sentenceBoundaryEnabled": true), then a corresponding [nnnn].sentence.json file is included in the results. Likewise, if word boundary data was requested ("wordBoundaryEnabled": true), then a corresponding [nnnn].word.json file is included in the results.

Here's an example word data file with both audio offset and duration in milliseconds:

[   {     "Text": "The",     "AudioOffset": 50,     "Duration": 137   },   {     "Text": "rainbow",     "AudioOffset": 200,     "Duration": 350   },   {     "Text": "has",     "AudioOffset": 562,     "Duration": 175   },   {     "Text": "seven",     "AudioOffset": 750,     "Duration": 300   },   {     "Text": "colors",     "AudioOffset": 1062,     "Duration": 625   },   {     "Text": ".",     "AudioOffset": 1700,     "Duration": 100   } ]

""" which I found here: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/batch-synthesis#batch-synthesis-results

As you can see in my code, I included the "wordBoundaryEnabled"= True in the payload properties, yet I'm not getting the "[nnnn].word.json" file, just the video and the summary.json file, which is just this: "

{
  "jobID": "9efb93a4-0f1d-4a02-8374-74cf640b9f16",
  "status": "Succeeded",
  "results": [
    {
      "contents": [
        "A recente onda de calor em Portugal levou a um aumento significativo de alertas de incêndio em várias regiões do país. O Instituto Português do Mar e da Atmosfera (IPMA) emitiu avisos vermelhos para os distritos de Castelo Branco, Santarém e Portalegre, devido às temperaturas que ultrapassaram os 40 graus Celsius. "
      ],
      "status": "Succeeded",
      "videoFileName": "4bbb121d760b4701bbf01aa7b4fdbea7/ceaa743a-db15-4f29-ad36-cb2627fc3f58/0001.webm",
      "TalkingAvatarCharacter": "max",
      "TalkingAvatarStyle": "formal"
    }
  ]
}

Pritam Suwal Shrestha 0 Reputation points

2024-12-13T07:14:52.48+00:00
I tried using

subtitleType: 'external_file'

But no external file is generated. Other options: soft_embedded and hard_embedded work fine. But when external_file is used, it does not produce any external file.

avatarConfig.subtitleTypeType of subtitle for the avatar video file could be external_file, soft_embedded, hard_embedded, or none. This property is optional, and the default value is soft_embedded.
My config json:

avatarConfig: { customized: false, talkingAvatarCharacter: character, talkingAvatarStyle: characterStyle, videoFormat: 'webm', videoCodec: 'vp9', subtitleType: 'external_file', videoCrop: { topLeft: { x: 550, y: 50 }, bottomRight: { x: 1400, y: 1000 } } }

Why is it not working? What i want is external subtitle file seperately from video so i can use it anyway i like.

Share via

Is it possible to get subtitles or a timed script with batch synthesis text to speech avatar?

1 answer

Your answer