KVA doesn't have permissions to delete a Key Vault. I could be wrong, but a KVA is just giving you admin rights to the vault contents, not rights to the Azure KV resources, which are generally managed at the subscription level.
Refer to the following table in Azure. Only Key Vault Purge Operator(s) can do this. Key Vault Data Access Admin may also work but that seems more skewed to managing access to the data.