Security groups do not have an associated mailbox, thus there are no OOO/auto-reply settings to configure. Moreover, security groups are not valid Exchange recipients and cannot receive mail.
A mail-enabled security group can receive messages, but you still cannot configure OOO on it. The usual workaround is to configure it on one of the members of the group instead. If this is the approach you end up using, make sure to also set the SendOofMessageToOriginatorEnabled flag on the group:
Set-DistributionGroup groupname -SendOofMessageToOriginatorEnabled $true