Valet Key Pattern

Use a token or key that provides clients with restricted direct access to a specific resource or service in order to offload data transfer operations from the application code. This pattern is particularly useful in applications that use cloud-hosted storage systems or queues, and can minimize cost and maximize scalability and performance.

Context and Problem

Client programs and web browsers often need to read and write files or data streams to and from an application’s storage. Typically, the application will handle the movement of the data—either by fetching it from storage and streaming it to the client, or by reading the uploaded stream from the client and storing it in the data store. However, this approach absorbs valuable resources such as compute, memory, and bandwidth.

Data stores have the capability to handle upload and download of data directly, without requiring the application to perform any processing to move this data, but this typically requires the client to have access to the security credentials for the store. While this can be a useful technique to minimize data transfer costs and the requirement to scale out the application, and to maximize performance, it means that the application is no longer able to manage the security of the data. Once the client has a connection to the data store for direct access, the application cannot act as the gatekeeper. It is no longer in control of the process and cannot prevent subsequent uploads or downloads from the data store.

This is not a realistic approach in modern distributed systems that may need to serve untrusted clients. Instead, applications must be able to securely control access to data in a granular way, but still reduce the load on the server by setting up this connection and then allowing the client to communicate directly with the data store to perform the required read or write operations.

Solution

To resolve the problem of controlling access to a data store where the store itself cannot manage authentication and authorization of clients, one typical solution is to restrict access to the data store’s public connection and provide the client with a key or token that the data store itself can validate.

This key or token is usually referred to as a valet key. It provides time-limited access to specific resources and allows only predefined operations such as reading and writing to storage or queues, or uploading and downloading in a web browser. Applications can create and issue valet keys to client devices and web browsers quickly and easily, allowing clients to perform the required operations without requiring the application to directly handle the data transfer. This removes the processing overhead, and the consequent impact on performance and scalability, from the application and the server.

The client uses this token to access a specific resource in the data store for only a specific period, and with specific restrictions on access permissions, as shown in Figure 1. After the specified period, the key becomes invalid and will not allow subsequent access to the resource.

Figure 1 - Overview of the pattern

It is also possible to configure a key that has other dependencies, such as the scope of the location of the data. For example, depending on the data store capabilities, the key may specify a complete table in a data store, or only specific rows in a table. In cloud storage systems the key may specify a container, or just a specific item within a container.

The key can also be invalidated by the application. This is a useful approach if the client notifies the server that the data transfer operation is complete. The server can then invalidate that key to prevent its use for any subsequent access to the data store.

Using this pattern can simplify managing access to resources because there is no requirement to create and authenticate a user, grant permissions, and then remove the user again. It also makes it easy to constrain the location, the permission, and the validity period—all by simply generating a suitable key at runtime. The important factors are to limit the validity period, and especially the location of the resource, as tightly as possible so that the recipient can use it for only the intended purpose.

Issues and Considerations

Consider the following points when deciding how to implement this pattern:

• Manage the validity status and period of the key. The key is a bearer instrument that, if leaked or compromised, effectively unlocks the target item and makes it available for malicious use during the validity period. A key can usually be revoked or disabled, depending on how it was issued. Server-side policies can be changed or, in the ultimate case, the server key it was signed with can be invalidated. Specify a short validity period to minimize the risk of allowing subsequent unwarranted operations to take place against the data store. However, if the validity period is too short, the client may not be able to complete the operation before the key expires. Allow authorized users to renew the key before the validity period expires if multiple accesses to the protected resource are required.
• Control the level of access the key will provide. Typically, the key should allow the user to perform only the actions necessary to complete the operation, such as read-only access if the client should not be able to upload data to the data store. For file uploads, it is common to specify a key that provides write-only permission, as well as the location and the validity period. It is vital to accurately specify the resource or the set of resources to which the key applies.
• Consider how to control users’ behavior. Implementing this pattern means some loss of control over the resources to which users are granted access. The level of control that can be exerted is limited by the capabilities of the policies and permissions available for the service or the target data store. For example, it is usually not possible to create a key that limits the size of the data to be written to storage, or the number of times the key can be used to access a file. This can result in huge unexpected costs for data transfer, even when used by the intended client, and might be caused by an error in the code that causes repeated upload or download. To limit the number of times a file can be uploaded or downloaded it may be necessary, where possible, to force the client to notify the application when one operation has completed. For example, some data stores raise events the application code can use to monitor operations and control user behavior. However, it may be hard to enforce quotas for individual users in a multi-tenant scenario where the same key is used by all the users from one tenant.
• Validate, and optionally sanitize, all uploaded data. A malicious user that gains access to the key could upload data aimed at further compromising the system. Alternatively, authorized users might upload data that is invalid and, when processed, could result in an error or system failure. To protect against this, ensure that all uploaded data is validated and checked for malicious content before use.
• Audit all operations. Many key-based mechanisms can log operations such as uploads, downloads, and failures. These logs can usually be incorporated into an audit process, and also used for billing if the user is charged based on file size or data volume. Use the logs to detect authentication failures that might be caused by issues with the key provider, or inadvertent removal of a stored access policy.
• Deliver the key securely. It may be embedded in a URL that the user activates in a web page, or it may be used in a server redirection operation so that the download occurs automatically. Always use HTTPS to deliver the key over a secure channel.
• Protect sensitive data in transit. Sensitive data delivered through the application will usually take place using SSL or TLS, and this should be enforced for clients accessing the data store directly.

Other issues to be aware of when implementing this pattern are:

• If the client does not, or cannot notify the server of completion of the operation, and the only limit is the expiry period of the key, the application will not be able to perform auditing operations such as counting the number of uploads or downloads, or preventing multiple uploads or downloads.
• The flexibility of key policies that can be generated may be limited. For example, some mechanisms may allow only the use of a timed expiry period. Others may not be able to specify a sufficient granularity of read/write permissions.
• If the start time for the key or token validity period is specified, ensure that it is a little earlier than the current server time to allow for client clocks that might be slightly out of synchronization. The default if not specified is usually the current server time.
• The URL containing the key will be recorded in server log files. While the key will typically have expired before the log files are used for analysis, ensure that you limit access to them. If log data is transmitted to a monitoring system or stored in another location, consider implementing a delay to prevent leakage of keys until after their validity period has expired.
• If the client code runs in a web browser, the browser may need to support cross-origin resource sharing (CORS) to enable code that executes within the web browser to access data in a different domain from the originating domain that served the page. Some older browsers and some data stores do not support CORS, and code that runs in these browsers may not be able to use a valet key to provide access to data in a different domain, such as a cloud storage account.

When to Use this Pattern

This pattern is ideally suited for the following situations:

• To minimize resource loading and maximize performance and scalability. Using a valet key does not require the resource to be locked, no remote server call is required, there is no limit on the number of valet keys that can be issued, and it avoids a single point of failure that would arise from performing the data transfer through the application code. Creating a valet key is typically a simple cryptographic operation of signing a string with a key.
• To minimize operational cost. Enabling direct access to stores and queues is resource and cost efficient, can result in fewer network round trips, and may allow for a reduction in the number of compute resources required.
• When clients regularly upload or download data, particularly where there is a large volume or when each operation involves large files.
• When the application has limited compute resources available, either due to hosting limitations or cost considerations. In this scenario, the pattern is even more advantageous if there are many concurrent data uploads or downloads because it relieves the application from handling the data transfer.
• When the data is stored in a remote data store or a different datacenter. If the application was required to act as a gatekeeper, there may be a charge for the additional bandwidth of transferring the data between datacenters, or across public or private networks between the client and the application, and then between the application and the data store.

This pattern might not be suitable in the following situations:

• If the application must perform some task on the data before it is stored or before it is sent to the client. For example, the application may need to perform validation, log access success, or execute a transformation on the data. However, some data stores and clients are able to negotiate and carry out simple transformations such as compression and decompression (for example, a web browser can usually handle GZip formats).
• If the design and implementation of an existing application makes it difficult and costly to implement. Using this pattern typically requires a different architectural approach for delivering and receiving data.
• If it is necessary to maintain audit trails or control the number of times a data transfer operation is executed, and the valet key mechanism in use does not support notifications that the server can use to manage these operations.
• If it is necessary to limit the size of the data, especially during upload operations. The only solution to this is for the application to check the data size after the operation is complete, or check the size of uploads after a specified period or on a scheduled basis.

Example

Microsoft Azure supports Shared Access Signatures (SAS) on Azure storage for granular access control to data in blobs, tables, and queues, and for Service Bus queues and topics. An SAS token can be configured to provide specific access rights such as read, write, update, and delete to a specific table; a key range within a table; a queue; a blob; or a blob container. The validity can be a specified time period or with no time limit.

Azure SAS also supports server-stored access policies that can be associated with a specific resource such as a table or blob. This feature provides additional control and flexibility compared to application-generated SAS tokens, and should be used whenever possible. Settings defined in a server-stored policy can be changed and are reflected in the token without requiring a new token to be issued, but settings defined in the token itself cannot be changed without issuing a new token. This approach also makes it possible to revoke a valid SAS token before it has expired.

Note

For more information see Introducing Table SAS (Shared Access Signature), Queue SAS and update to Blob SAS in the Azure Storage Team blog and Shared Access Signatures, Part 1: Understanding the SAS Model on MSDN.

The following code demonstrates how to create a SAS that is valid for five minutes. The GetSharedAccessReferenceForUpload method returns a SAS that can be used to upload a file to Azure Blob Storage.

public class ValuesController : ApiController{  private readonly CloudStorageAccount account;  private readonly string blobContainer;
  ...
  /// <summary>  /// Return a limited access key that allows the caller to upload a file   /// to this specific destination for a defined period of time.  /// </summary>  private StorageEntitySas GetSharedAccessReferenceForUpload(string blobName)  {    var blobClient = this.account.CreateCloudBlobClient();    var container = blobClient.GetContainerReference(this.blobContainer);    var blob = container.GetBlockBlobReference(blobName);    var policy = new SharedAccessBlobPolicy    {      Permissions = SharedAccessBlobPermissions.Write,      // Specify a start time five minutes earlier to allow for client clock skew.      SharedAccessStartTime = DateTime.UtcNow.AddMinutes(-5),      // Specify a validity period of five minutes starting from now.       SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(5)    };    // Create the signature.     var sas = blob.GetSharedAccessSignature(policy);    return new StorageEntitySas    {      BlobUri = blob.Uri,      Credentials = sas,      Name = blobName    };  }  public struct StorageEntitySas  {    public string Credentials;    public Uri BlobUri;    public string Name;  }
}

Note

The complete sample containing this code is available in the ValetKey solution available for download with this guidance. The ValetKey.Web project in this solution contains a web application that includes the ValuesController class shown above. A sample client application that uses this web application to retrieve a SAS key and upload a file to blob storage is available in the ValetKey.Client project.

Related Patterns and Guidance

The following patterns and guidance may also be relevant when implementing this pattern:

• Gatekeeper Pattern. This pattern can be used in conjunction with the Valet Key pattern to protect applications and services by using a dedicated host instance that acts as a broker between clients and the application or service. The gatekeeper validates and sanitizes requests, and passes requests and data between the client and the application. This pattern can provide an additional layer of security, and reduce the attack surface of the system.
• Static Content Hosting Pattern. This pattern describes how to deploy static resources to a cloud-based storage service that can deliver these resources directly to the client in order to reduce the requirement for expensive compute instances. Where the resources are not intended to be publicly available, the Valet Key pattern can be used to secure them.

More Information

• The article Introducing Table SAS (Shared Access Signature), Queue SAS and update to Blob SAS on the Azure Storage Team blog.
• Shared Access Signatures, Part 1: Understanding the SAS Model on MSDN.
• Shared Access Signature Authentication with Service Bus on MSDN.

You can download the "Cloud Design Patterns – Sample Code" from the Microsoft Download Center at https://aka.ms/cloud-design-patterns-sample.

Next Topic | Previous Topic | Home | Community