How to avoid Jailbreak errors using O1 model

Question

Hello,

We have been running into constant Jailbreak errors when using O1 models on Azure. The same prompts on OpenAI works without any errors.

Our subscription has been approved to disable content filter, however in this link, it states configurable content filters are not available for o1* models. Is there a way to get around this right now ?

Configurable content filters are not available for

A sample response

openai.BadRequestError: Error code: 400 - {'error': {'message': "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: 

Answer 1

Hello Bhala Moorthy,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you would like to know how you can avoid Jailbreak errors using O1 model.

To avoid Jailbreak Errors with O1 Models on Azure you will need to understand that Azure’s implementation of OpenAI models, such as O1, comes with stricter, non-configurable content management policies compared to OpenAI's direct platform. This limitation ensures compliance with Azure's safety standards, meaning that even approved subscriptions to disable content filters cannot override this restriction for O1 models. https://github.com/Azure/azure-sdk-for-java/issues/42094 - this means that even if your subscription is approved to disable content filters, the O1 models might still be subject to Azure's default content management policies.

Secondly, to minimize errors, refine prompts to avoid ambiguous or sensitive language. This can involve rephrasing questions to remove potentially flaggable terms. For example, instead of asking, “Describe how to bypass security measures,” you could say, “Provide recommendations for improving cybersecurity practices.” Testing prompts incrementally can also reveal patterns that trigger filters, allowing systematic adjustments.

Also, leverage Azure's diagnostic capabilities to capture specific reasons why prompts fail. Logging error messages like BadRequestError and identifying flagged patterns can guide future refinements. Although Azure doesn't provide direct tools for content filtering diagnostics, analyzing service logs in tools like Azure Monitor can be invaluable.

You might need to opt for an alternative model, if O1 models fail to meet requirements due to content filtering, explore other Azure-supported models like GPT-4 or GPT-4 Turbo. These models might offer greater flexibility and align better with your use case.

Lastly, implement robust error-handling mechanisms to dynamically adjust prompts when errors occur. Use automated retry logic to refine and reattempt filtered prompts. The following Python code demonstrates this approach:

import openai
def call_openai_with_retry(prompt, max_retries=3):
    for attempt in range(max_retries):
        try:
            response = openai.Completion.create(
                engine="text-davinci-002",
                prompt=prompt,
                max_tokens=100
            )
            return response
        except openai.error.InvalidRequestError as e:
            if "filtered" in str(e):
                prompt = modify_prompt(prompt)  # A function to simplify prompts
                continue
            raise e
    raise Exception("Failed after multiple retries.")
def modify_prompt(prompt):
    # Example logic to simplify or adjust the prompt
    return f"Can you explain: {prompt}"

To read more - Azure OpenAI Service Documentation - https://learn.microsoft.com/en-us/azure/ai-services/openai and https://github.com/Azure/azure-sdk-for-java/issues/42094

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.