As you have pointed out, the Group has to be a Security Group or a Distribution list for it to appear in the toke group claim.
Only groups ie (security groups, directory roles or distribution lists) will be in the group claim not application ids.
In your scenario,
Best solution:
- Have App B (Target API) Look for custom roles in the token
- Create custom roles on App B registration
- Assign the roles to a Security Group X (on App B's Service Principal)
- Add App A to the Security Group X. As long as App A is a member of Security Group X, the roles assigned to that Group on App B will be in the token requested with scope
api://{app registration id B}/.default
Note that by just having App A in Security Group X, the token will have group id for Security Group X because App A is a member(this does not depend on App B's relation to this Group).