Protecting Sensitive Data with Azure Key Vault
By Stephen Lewontin, Solution Architect
In this week's post I'm going to look at a recently introduced Azure component—Key Vault—that deals with a basic security issue for many deployments: how to safely protect and manage sensitive data within the Azure environment. There is already excellent published getting started and step-by-step documentation for Key Vault, so I'll start with just a quick overview: I want to devote most of this post to a deeper dive on some of the underlying security issues and show you some useful patterns for addressing these in a real application context.
The samples I'll show you come from a recent engagement with a major financial institution where the ability to address the customer's rigorous security requirements was key to winning the engagement. The customer had asked us to implement Azure-based analytics for their newly launched Twitter presence. This required authenticated access to the Twitter REST APIs we used to get input data such as client tweets, replies, followers, and mentions. For this we chose to use the Twitter application-only authentication mechanism (based on OAuth2 Client Credential Grant Flow), which provides read-only access to a subset of feed data.
We made the choice of app-only authentication since the majority of the analytics that the customer was interested in were accessible with this mechanism. The customer was potentially interested in some feed data not accessible via this mechanism—for example, direct messages—but these were only available with a level of access that would also have permitted an application to post to the client's feed. Obviously, we had no intention of writing code that would post, but the customer was understandably concerned about generating such powerful credentials, so we agreed to stick to a lower level of access. This is a perfect example of adhering to the principal of least privilege, which states that a component must only have access to the information and resources that are necessary for its legitimate purpose. Since our application must not be able to post to the customer's Twitter feed, it seemed prudent to avoid giving it the credentials to do so .
Keeping Secrets Secret
To authenticate we needed to construct a credential from a pair of parameters called the consumer key and consumer secret. While the access allowed by Twitter application-only authentication is limited, we agreed with the customer that the security parameters should be well-protected in our implementation. We chose to store the consumer credential parameters as Azure Key Vault secrets.
Key Vault supports two classes of storage: keys and secrets. Key storage is intended to provide secure storage and access for encryption keys. It provides methods that allow an application to encrypt and decrypt data using these keys without requiring that the application have access to the actual private key data. Secret storage provides safe storage for other kinds of sensitive application data, and was a good fit for the Twitter authentication parameters we were using.
Coding Key Vault Access
The basic pattern for using Key Vault is:
- Register your application with Azure Active Directory (AAD)
- Create an Azure Key Vault instance
- Store critical secrets in the key vault
- Code access to those secrets in your application
- Configure the application to safely pick up Key Vault authentication parameters
Let's start with a quick overview of these steps. If you haven't already done so, I'd recommend walking through the step-by-step basics I mentioned above to get the full context before continuing here.
Step 1 is shown as performed in the Azure classic portal, first by adding a new application in the Azure Active Directory Applications tab:
Following this you create a client ID and key in the configuration tab for the newly created application:
Note the client ID and key: these are the authentication parameters your application will use later to access the Key Vault.
Steps 2-3 are performed using Microsoft Power Shell. I'll cover only the key steps here since the existing step-by-step documentation has a very complete sample. You can also refer to the Azure Key Vault cmdlets documentation for more details on the Power Shell interaction. The first steps, shown below, create a key vault instance and grant your application permissions to it.
PS C:\> New-AzureResourceGroup -Name KeyVaultBlogSampleRG -Location 'East US 2'
ResourceGroupName : KeyVaultBlogSampleRG
Location : eastus2
ProvisioningState : Succeeded
. . .
PS C:\> New-AzureKeyVault -VaultName KeyVaultBlogSampleKV -ResourceGroupName KeyVaultBlogSampleRG -Location 'East US 2'
Vault Name : KeyVaultBlogSampleKV
Resource Group Name : KeyVaultBlogSampleRG
. . .
Vault URI : https://KeyVaultBlogSampleKV.vault.azure.net
Tenant ID : 72f988bf-86f1-41af-91ab-2d7cd011db47
. . .
PS C:\> Set-AzureKeyVaultAccessPolicy -VaultName KeyVaultBlogSampleKV -ServicePrincipalName 'd935bb32-5f81-4cd0-9ccd-a07665adee39' -PermissionsToSecrets get -PermissionsToKeys wrapkey, unwrapkey, encrypt, decrypt
From a security perspective, there are a couple of key things to note on these steps. The Key Vault instance is implicitly linked to the default Active Directory instance for the subscription where the user is logged in. In effect, the active directory instance acts as a security boundary for the key vault instance. We'll see below how this affects access to the secrets you have stored in the Key Vault. The ServicePrincipalNameparameter to the last Power Shell cmdlet is the ID of your application from the first step, where you registered your application with AD.
A second thing to point out is that the code explicitly assigns the Key Vault instance to a resource group. Resource groups provide a common management boundary for a set of related resources and can be managed directly in the Azure preview portal. For our Twitter solution, we actually had several scheduled applications running to deal with various components of the Twitter feed and other data processing, and these were all writing to Azure Blob and other storage. We assigned all of these Azure resources to the same resource group and created the Key Vault instance in that group. Among other benefits, having all of these resources accessible as a group enhances the customer's ability to manage security issues across the whole set.
The second step, shown below, is to add some secrets to the key vault. In this case, I'm adding the Twitter consumer secret. Once the secret is added, Power Shell shows an Id for the added secret in the form of a URI. Keep a note of this since it is the datum that the application will need later to retrieve the actual secret value from the key store.
PS C:\> $csvalue = ConvertTo-SecureString 'ac592......f37' -AsPlainText -Force
PS C:\> Set-AzureKeyVaultSecret -VaultName KeyVaultBlogSampleKV -Name TwitterConsumerSecret
-SecretValue $csvalue
Vault Name : keyvaultblogsamplekv
Name : TwitterConsumerSecret
Version : 876d12c73f2d4788abac9980c33aa8d6
Id : https://keyvaultblogsamplekv.vault.azure.net:443/secrets/TwitterConsumerSec
ret/876d12c73f2d4788abac9980c33aa8d6
Enabled : True
In this step you'll notice the –AsPlainText flag. This doesn't affect how Key Vault protects your secret in storage, but it does mean that your secret is not automatically encrypted in memory either by Power Shell or your application code. We'll discuss the whole issue of encrypting secrets a bit further on.
Step 4 shows a snippet of .NET C# code that accesses the key vault and retrieves a secret:
// If not present, run NuGet Install-Package Microsoft.Azure.KeyVault
Using Microsoft.Azure.KeyVault
//Get the secret identified by id from the KeyVault
public static string getVaultSecret(string id)
{
try
{
var kvClient =
new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(KeyVault.GetToken));
return kvClient.GetSecretAsync(id).Result.Value;
}
catch (Exception e)
{
return null;
}
}
//Implement GetToken callback invoked by the KeyVaultClient
private async static Task<string> GetToken(string authority, string resource, string scope)
{
var authContext = new AuthenticationContext(authority);
//Alternatively, use certificate authentication
ClientCredential clientCred = new ClientCredential(
ConfigurationManager.AppSettings["ClientID"],
ConfigurationManager.AppSettings["ClientKey"]);
AuthenticationResult result = await authContext.AcquireTokenAsync(resource, clientCred);
if (result == null)
throw new InvalidOperationException("No Key Vault token");
return result.AccessToken;
}
The Twitter application logic calls getVaultSecret with the Key Vault Id URI of one of the Twitter authentication parameters. For example, to get the Twitter client secret that we added to the key vault in the previous steps, your code would pass the Id URI shown in the power shell window after adding the secret. The KeyVaultClient implementation in turn invokes the callback defined by the GetToken method to authenticate the application to the Key Vault. This code is pretty similar to what you will find in other published Key Vault samples. What I want to focus on here is the use of the Configuration Manager to get the Key Vault credentials for the application.
First, let's step back and think about what it means to protect the sensitive Twitter authentication parameters. Any access to these parameters must be authenticated, and Key Vault provides just this protection: only authenticated applications can access the secrets stored in the Key Vault. But, of course, this would seem to involve a conundrum: to authenticate to Key Vault, an application must ultimately have access to some other secrets: the application's Key Vault Client ID and Key, which were created in step 1! Just as we don't want to expose the Twitter parameters by embedding them in code, it seems logical that we don't want to hard code the Key Vault parameters either.
A Safe Configuration Pattern
Ultimately, Azure addresses this problem by partitioning the problem space: we assume that the Azure infrastructure protects the runtime environment of our application via adequate user authentication. For example, when a user with admin privileges installs and configures an Azure component (in this case, our Twitter application), the Azure infrastructure has sufficiently authenticated the user and properly protects the runtime environment from unauthorized access. Most important, this means that the runtime configuration and environment of our application cannot be accessed or modified by unauthorized users. We can make use of this to safely configure the Key Vault authentication used by our application.
One way to do this takes advantage of a neat feature of Azure App Service apps. Such apps are always associated with a Web Application host, which can be used to manage and provide them with runtime configuration. The host Web Application settings can be configured directly in the Azure portal, and the configuration parameters set there are inherited by the hosted App Service app at run time. For our Twitter application we chose to deploy our code as an Azure Web Job . Web Jobs can be scheduled to run at regular intervals and are ideal for the kind of repeated short term workloads required to process the Twitter API calls.
Here's what the configuration of the Web App that hosts our Web Job looks like, using the Azure Web App Settings in the Azure Portal:
Here I've set the ClientID and ClientKey config parameters to the values generated when I registered the Twitter App with Azure AD back in Step 1. (I've also set the Key Vault Secret Id URIs of the two Twitter API parameters, which will be used by the code that calls getVaultSecret as discussed above.) All these parameters will be inherited by any Web Jobs hosted by this Web App, so they can be pulled from the Configuration Manager as shown in C# the code sample.
To summarize the pattern:
- Store sensitive data as Key Vault secrets
- Deploy your application code as one of the Azure App Service application types (in this case, we chose a Web Job)
- Set configuration parameters in the Host Web application that can be used to authenticate to Key Vault and to access the stored secrets.
You then write your application based on a few simple principals:
- No secrets or other confidential data are ever explicitly contained in application code and configuration files
- Code only accesses secrets at the point where they are required at runtime
- Code never stores, writes to console, or otherwise shares secrets.
One thing you may have noticed in the C# code is that there is no explicit reference to the Active Directory that holds the authentication credentials. By default the KeyVaultClient assumes that these are stored in the primary AD for the subscription in which the application is running. I haven't seen any official documentation about how to change this, though presumably one could do this by modifying the implementation of the authentication callback to use a different authority identifier.
Before we leave this topic, it has probably occurred to you that this whole pattern begs the question, why not just store the client's Twitter parameters directly in the Web App runtime configuration and avoid all of the indirection? The logic here is that the Twitter authentication parameters are valuable customer resources that may be used in multiple contexts, and the Key Vault provides a secure way to manage these resources on behalf of any Azure component that is authorized to make use of them. The Key Vault credentials are different: these are application-specific resources so it makes sense to provision them in the authenticated context provided by the Azure portal for each managed application.
One of the key security principals that is implicitly being applied here is to compartmentalize management of privileged data to security domains for which this is appropriate. An instance of Key Vault is used to manage the Twitter keys as a shared resource in the customer's environment, with access granted by whomever manages the Twitter account on an as-needed basis to specific applications and users. Applications are then responsible for managing only their application-specific Key Vault access tokens. In fact, we were able to write, test, and deploy our Twitter analytics application without ever actually seeing the Twitter keys stored by the customer in the Key Vault, and we were careful during development to make sure that these values where never saved or logged by our code, even in debug mode, so that there was no chance that they would be inadvertently leaked.
How much security is enough?
We adopted a fairly conservative approach to keeping the Twitter authentication parameters secure, but we could have done more in two areas:
Certificate Authentication
Our application could have authenticated to Key Vault using an X.509 certificate. This uses a somewhat more secure public key-based method to authenticate to the Key Vault, where the application does not need to supply a secret key to the Key Vault API. Of course the application still needs to safely store the public keys, which can be done using Web Application settings similarly to how we stored the KeyVault credentials. There's no security magic here: we're still relying on the Azure Web Application runtime environment to protect the stored certificate.
One caveat: the published examples use test certificates generated by the developer and, initially, stored on the developer's machine. They don't address the larger certificate management issues which you will run into when doing a customer deployment, such as how to manage certificate storage, expiration, and renewal. Security conscious customers typically have key management policies and infrastructure in place, and you may need to integrate your application's key management with this
Encryption
The Twitter authentication parameters are stored unencrypted. To return to the Power Shell interaction we used to store the Twitter consumer secret:
PS C:\> $csvalue = ConvertTo-SecureString 'ac592......f37' -AsPlainText -Force
As I mentioned previously, the –AsPlainText flag means that the data passed to the Key Vault contain the plain text supplied by string parameter, which in our case is unencrypted text. This doesn't mean that this value is exposed by the Key Vault. According to the Key Vault documentation, Key Vault "accepts the data and stores it securely", but the documentation also suggests that "for highly sensitive data, clients should consider additional layers of protection for data that is stored in Azure Key Vault; for example by pre-encrypting data using a separate protection key." The implication is that the Key Vault keeps your value safe internally, but can obviously make no guarantees about what happens to the data once it is retrieved by an application.
For the moderately sensitive Twitter keys we were storing, this seems sufficient, but for more sensitive data—for example, Twitter keys that give post access to the client's Twitter feed—additional encryption probably makes sense. Key Vault provides a way to securely encrypt such secrets, though the process requires some extra steps using encryption keys in Key Vault's key storage. I'll discuss this in a subsequent post.
One other area of theoretical concern is how our application handles these secure data internally. By following the rules we stated above—such as never storing or writing the values to the console—we've eliminated many potential vulnerabilities. But there is a class of attack that relies on gaining access to the memory used by the executing application and reading sensitive values temporarily stored there. Since our code does convert the secrets into plain text strings internally, this might seem like a concern. In practice, such attacks are much less likely to be possible for applications running in the context of Azure PaaS. If you are concerned about this, however, Key Vault does support handing keys and secrets as .NET secure strings, which are encrypted in memory. This may also be a topic for a future posting.
Summary
Azure Key Vault provides an essential PaaS component to safeguard sensitive data. You can use Key Vault most effectively by thinking about how to protect the whole chain of sensitive data—in the sample case, both the client's Twitter authentication parameters and our application's Key Vault authentication parameters—and by following basic security principles such as least privilege and information compartmentalization. I've shown you some basic deployment patterns that accomplish this and suggested some further steps that may be relevant with more sensitive data. Even if your application doesn't follow these patterns exactly—for example, your application runs as a custom ADF activity rather than an Azure App Service—you should be able to apply similar principals. I'll discuss further examples in future posts.