Working with memory mapped files in .NET 4
I have been exploring different new features that come with the .NET 4, beyond the most popular ones like dynamic types and covariance; I was interested in performance enhancements. For this reason I am going to publish a couple of blog entries were I explore these different features.
Memory mapped files may sounds alien to the managed code developer but it has been around for years, what is more, it is so intrinsic in the OS that practically any communication model that wants to share data uses it behind the scenes.
So what is it? A memory mapped file allows you to reserve a region of address space and commit physical storage to a region (hmmm, sounds like virtual memory, isn’t it?) but the main difference is that the physical storage comes from a file that is already on the disk instead of the memory manager. I will say that it has two main purposes:
· It is ideal to access a data file on disk without performing file I/O operations and from buffering the file’s content. This works great when you deal with large data files.
· You can use memory mapped files to allow multiple processes running on the same machine to share data with each other.
The memory mapped file is the most efficient way for multiple processes on a single machine to communicate with each other. What is more, if we check other IPC methods we can see the following architecture:
Impressive, isn’t it? Now you have the power of this technology available on the System.IO namespace (instead of using the Pinvoke approach).
Now let’s quickly explore how it works. We have two types of memory mapped files models, one model is using a custom file, this can be any file that the application accesses it and the other one using the page file that we are going to share it with the memory manager (this is the model that most of the technologies above use).
Let’s explore the custom file model. The first thing that we need to do is to create a FileStream to the file that we are going to use, this can be an existing file or a new file (keep in mind that you should open this file as shared, otherwise no other process will be able to access it!). With the stream in place, we can now create the memory mapped file. Let’s see an example:
using System.IO.MemoryMappedFile;
MemoryMappedFile MemoryMapped = MemoryMappedFile.CreateFromFile(
new FileStream(@"C:\temp\Map.mp", FileMode.Create), // Any stream will do it
"MyMemMapFile", // Name
1024 * 1024, // Size in bytes
MemoryMappedFileAccess.ReadWrite); // Access type
I have use one of the simplest constructor, we define the stream to use and we provide a name. The object needs to know the size of it in bytes and the type of access that we need. This will create the memory mapped file but to start using it we will need a map view. We can create one using the following syntax:
MemoryMappedViewAccessor FileMapView = MemoryMapped.CreateViewAccessor();
This map covers the file from the first byte until the end. If we need now to write or read information from it we just call the map view methods with the correct offset.
int Number = 1234;
FileMapView.Write(0, Number);
FileMapView.Write<Container>(4, ref MyContainer);
We can see that we can write built in types or custom types with the generic version. The good thing about the memory mapped files is persistence, as soon as you close it the contents will be dumped on the disk, this is a great scenario for sharing cached information between applications.
Now if we want to read from it, the other process needs also to create a memory mapped file, we can use the other static initialize that opens an existing one or creates one if it does not exist.
MemoryMappedFile MemoryMapped = MemoryMappedFile.CreateOrOpen(
new FileStream(@"C:\temp\Map.mp", FileMode.Create), // Any stream will do it
"MyMemMapFile", // Name
1024 * 1024, // Size in bytes
MemoryMappedFileAccess.ReadWrite); // Access type
Create the map view and read it:
using (MemoryMappedViewAccessor FileMap = MemoryMapped.CreateViewAccessor())
{
Container NewContainer = new Container();
FileMap.Read<Container>(4, out NewContainer);
}
That’s it, really simple isn’t it? Now, there is a small drawback with this approach and is related to the size of the memory mapped file. If you don’t know in advance you will need to create a large file
“just in case”, this can be a very large file with a lot of wasted space, it would be nice to have the ability to reserve the space instead of committing all of it, isn’t it?
In order to solve this issue you can use the page file, this has the advantage of allowing you to commit data on the fly but introduces another issue: you don’t own the file and the map will last until the last handle is destroyed. But think about it, for certain scenarios this is absolutely valid.
Now if we revisit the sample we will need to change some initialization parameters, for this particular one I will use the full constructor so I can introduce some other features that are also applicable to the custom files one.
MemoryMappedFileSecurity CustomSecurity = new MemoryMappedFileSecurity();
MemoryMappedFile PagedMemoryMapped = MemoryMappedFile.CreateNew(
@"Salvador", // Name
1024 * 1024, // Size
MemoryMappedFileAccess.ReadWrite, // Access type
MemoryMappedFileOptions.DelayAllocatePages, // Pseudo reserve/commit
CustomSecurity, // You can customize the security
HandleInheritability.Inheritable); // Inherit to child process
The memory mapped file security allows you to customize who or which process can have access to the resource, this can be quite important when you want to protect sensitive information and you don’t want other processes changing the file map. You can explore that object and see all the different settings that you can change. The reference is here. If we explore the construction of the memory mapped file we can see that there is no stream, we just name the resource. This will create an association between a section of the file based on the size and the map name, this is how both processes will access the file. Note as well that I am setting the property “DelayAllocatePages”, this implements a pseudo reserve/commit model where we will use the space once is needed. Finally, the other interesting parameter is the handle inheritance model, this will allow to share this resource with a child process if is required.
The access to the file uses the same syntax as the previous example, remember that if you close the memory mapped file this will be non accessible, this issue catches many developer.
Finally, another interesting area is the creation of multiple map views, these can work on the same memory mapped file accessing different areas of the files. This will allow you to properly protect the content and allowing you to synchronize the access.
You can do this defining different offsets and lengths when you create your map view.
MemoryMappedViewAccessor WriteMap = MemoryMapped.CreateViewAccessor(0, 1024,
MemoryMappedFileAccess.ReadWrite);
MemoryMappedViewAccessor ReadMap = MemoryMapped.CreateViewAccessor(1025, 1024,
MemoryMappedFileAccess.Read);
Now you can enjoy the power of this high performance data sharing model on your applications when you compile them with .NET 4.0! Note that this blog is based on beta 1 information, I will check the post once is released.
Comments
Anonymous
June 09, 2009
The comment has been removedAnonymous
June 20, 2009
.Net Framework 4.0 introduces memory mapped files. Memory mapped files are useful when you need to doAnonymous
June 24, 2009
The comment has been removedAnonymous
June 24, 2009
Hi Cameron, VS2008 does not know how to handle .NET 4 unfortunately. Therefore the IDE can not use the new compiler. You can write the code but you will need to manually call the VS2010 compiler to link it to .NET 4. You can still use memory mapped files in VS2008 using PInvoke, but I imagine that is not the solution that you are looking after. Regards, SalvadorAnonymous
August 20, 2009
Salva, Could you please elaborate more on how to "manually call the VS2010 compiler to link it to .NET 4." I am thinking that I would like to take the approach where I write a DLL in VS2010 .NET 4.0 to handle the reading-writing/caching of the data in the memory mapped file and reference this DLL in my VS2008 .NET 3.5 project. The reason is that my application is heavily dependent on MS Robotics Studio which is not 4.0 ready, but I need to process very large image data sets. Thanks, ScottAnonymous
August 24, 2009
Hi E. Scott Anderson The short answer is you can't. You can do it the other way around as .NET 4 allows you to run side-by-side libraries (i.e running a .NET4 app and referencing a .NET 2/3/3.5 DLL). The problem is that the process running under CLR 2 (.net 2/3/3.5) does not know how to interpret a DLL compiled under CLR 4. You can still access the memory mapped files using the PInvoke model, it works very well. You can find more info here: http://msdn.microsoft.com/en-us/library/ms810613.aspx RegardsAnonymous
March 29, 2010
Do you know if memory mapped files will work for a file on a distributed file system?Anonymous
March 29, 2010
Hi John, No, it does not work with UNC or remote drives as is for same machine process intercom. RegardsAnonymous
July 09, 2010
What is memory mapped file? Is this new technology? Never heard before.Anonymous
July 31, 2010
is there a code link for this example?Anonymous
October 16, 2010
Hi, if MemoryMappedFile is created from <b>dbData.mdb</b> (1.5GB) file. MemoryMappedFile MemoryMapped = MemoryMappedFile.CreateFromFile(FilePath); MemoryMappedViewAccessor FileMap = MemoryMapped.CreateViewAccessor()) how do i put this in this connection string. Microsoft.Jet.Oledb.4.0; Data Source=??Anonymous
October 21, 2010
@Jetha No, it's not really a new technique (though I haven't come across it before either). The MSDN article Salva linked to are more than 17 years old, so it has been around since the very first version of Windows NT! :-)Anonymous
January 19, 2011
So ,in memory maped file, can user updates in the file also? If so, does it controls the versions?Anonymous
May 11, 2012
What is the maximum size of the mapped file. Is it bounded or unbounded?Anonymous
July 30, 2012
Does anyone know for sure if when you pull from MMF, does the local App process create it's own memory storage for the contents in the shared memory? or does it reference the memory directly? I ask this question because I am in thinking about using a 4gb file in MMF, and if it pulls down a local memory for each app(process) that uses the resource, I will be looking at a lot of RAM. Any ideas?Anonymous
January 09, 2013
- I believe it is only sticking into memory chunks/pages of a file at a time. How does it release it back ? How does it know when we are done?
- Should we still make a copy of the original file and put it in a temp location then read it into mmf from there or is it ok to just read it from the original now? (how does that play if someone overwrites the original as we are reading from it?
- How do we ensure the mmf is always using the latest since we might constantly be updating a particular file?
- What is the server hit of using this? I imagine its better than not using it but want to know more.
- Anonymous
January 12, 2013
I tested 2 apps - one sending WM_COPYDATA (Windows Messaging) and other with out of proc COM server consumed by VB script. Both from All in One Code Framework. I second case Process Explorer shows handle of "ALPC port" with name "RPC ControlOLE44EE3FE7D3E04B7EA505A092387B". Seems it is en.wikipedia.org/.../Local_Procedure_Call. It differs from Named Shared Memory and "The memory mapped file is the most efficient way for multiple processes on a single machine to communicate with each other. " not always true. And COM/OLE and may be RPC uses not only shared memory but ALPC either.