Hello Ayush Malik,
Thank you for reaching out to Microsoft!
To restrict the permission set to some sites, please find the below work around which might help resolve the issue.
You can restrict the permissions to specific sites using Sites.Selected permission instead of AllSites.Fullcontrol.
Please find the below useful reference guides for the same:
Link: Using Sites.Selected permission
Link: Controlling app access on specific SharePoint site collection
Hope this helps.
If the answer is helpful, please click Accept Answer and kindly upvote it. If you have any further questions about this answer, please click Comment.