@Chloe Hall, Thanks for posting in Q&A. From your description, I know the conditional access policy is not working.
To clarify our issue, please check the following information:
1, Please check if we set on instead of Report only in the conditional access.
2, What cloud app we choose, did we choose the one we want like SharePoint, OneDrive, and Exchange?
3, It seems the screen shot is not uploaded successfully. Please get screen shot of the conditional access policy settings to know more better. As a note hide any sensitive information when we provide screen shots.
4, Check the sign in log to see if any more we can find.
https://learn.microsoft.com/en-us/entra/identity/monitoring-health/howto-troubleshoot-sign-in-errors
https://learn.microsoft.com/en-us/entra/identity/conditional-access/troubleshoot-conditional-access
Please check the above information and if there's any update, feel free to let us know.
If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.