Mapped Files give very fast access to huge amounts of data
Sometimes you write a program and it requires lots of extra storage. Perhaps it needs to deal with lots of information. If the requirements are large there are several alternatives to consider:
1. use main process memory (HeapAlloc, VirtualAlloc, or any API that eventually calls these, such as C++ new, C# new, etc).
a. 32 bit processes are limited to 2^32 = 4 Gigabytes. That’s about the size of a full length movie on a DVD, and less than the amount of memory on many smartphones.
b. Customer’s may run your program with large inputs, causing Out of Memory crashes.
2. Use an external process
a. Need to deal with inter-process communications
3. Use a database
a. Could be internal, local, network, or cloud based
b. Use a web service of some sort to fetch the data.
4. Use Memory Mapped Files
In general, the further the data is, the more expensive (in terms of time) to process it. Perhaps the information is a temporary aggregation of local data (like calculating intermediate results). You may just want a simple way to read and write huge amounts of data from your program.
Most modern operating systems, such as Windows, provide ways to manage memory that allow multiple programs to run at seemingly the same time. In reality, this is mostly an illusion. When I start PowerPoint, Word, Visual Studio, Visual FoxPro, Notepad, Excel, OutLook etc. on my machine, they seem to be running at the same time. I can switch between them at will. However, on this Surface Pro 4, there are 4 cores, each of which can separately fetch and process instructions from memory simultaneously. Each running application gets a little bit of execution time on the processor core when needed. The OS is very efficient at swapping applications in and out of execution, allowing the various threads of the applications to be mapped into execution.
This swapping of execution context happens very often and it must be extremely efficient. The “Execution Context” needs to be swapped very quickly (perhaps thousands of times per second) It includes things like the register contents (which includes the stack pointer, instruction pointer, along with the general purpose registers).
But that’s not all that needs to be swapped out: the memory that OutLook was using needs to be swapped out as well, and that of Excel needs to be put in. This memory swapping also has to be done very fast. It can’t be an operation like “copy all this memory here to that location there” because that’s a huge amount of swapping.
Thankfully, modern processors have memory management units that map huge amounts of memory by just changing a few special values in things called Global or Local Descriptor Tables.
Now it’s pretty clear that Excel, Word, PowerPoint, OutLook share a lot of the same code. The Office Ribbon is common code. There is a lot of other code that is common to Office, as well as to all other Windows applications. For example, when a program calls an API function to create a file or allocate some virtual memory, the routines in the Windows API that support these functions are also mapped into the process. These APIs typically live in DLLs, like Kernel32.dll, User32.dll and some of these DLLs are mapped into every 32 bit process. Not only are they mapped into every process, they are mapped into the same addresses in each process. This allows them to share the same memory segments. (The base address at which, say “user32.dll” is loaded is constant, and used to be the same for every machine running the same version of Windows. Virus authors took advantage of the fact that “WriteFile” was always at address 0xef01004. Since Windows Vista, the addresses at which standard Windows DLLs get loaded are randomized (see Address Space Layout Randomization)
This efficient memory swapping mechanism can be used to a program’s advantage: a small fixed region of memory can be used as a “sliding” view window that can be moved across a huge file.
Below is some sample code and some tests that demonstrate the use and performance of using Memory Mapped Files.
When you run the code below, you might be quite surprised by the performance of file mapping. It is very fast, is an integral part of the operating system and has been tuned for years.
Code notes:
1. the main code is in C++ and the test code is in C#. This is a handy way to write tests in C#, even if the target code is native.
2. Check out the use of the interop callback delegate MemMapDelegate between C++ and C#
3. Observe the use of MapViewOfFileEx and the preferred address
4. The difference between growing a PageFile backed MappedFile and a Disk File based one.
5. Run the code with Process Explorer open showing the Virtual Size, working set, and Page Faults of the test running process.
6. Run the code with Explorer open to your Desktop and watch the file get created and grow as the code runs (for the file on disk based scenario)
7. The performance numbers are surprising! Accessing 50 gigabytes of data from a 4 gigabyte process at very high speed is a breeze.
Start Visual Studio. (for testing the instructions below, I used the free community edition of Visual Studio)
File->New->Project->C++ Windows->Win32 Win32Project. “NativeMemMapDll”
In the Win32 Application Wizard, choose Application Type: “Dll”, then Finish
Paste in the contents of “NativeMemMapDll.cpp” from below
Project->Add->New Item->C++ Header file. Name it “MyMemMap.h”
Paste in the contents of <MyMemMap.h> from below.
Now we need a way to call the code:
Right click on Solution node in Solution Explorer->Add->New Project->Visual C#->Test->Unit Test Project “UnitTestProject1”
We need a build event for the C++ project: when finished building, copy the output to the test code:
Right Click on NativeMemMapDll solution->Properties->Configuration Properties->Build Events->Post-Build Event.
Choose “All Configurations “ at the top to apply the same Build Event to Debug and Release buidls.
Edit the Command Line Post Build Event to say:
echo post Build Event
xcopy /fdy "$(TargetPath)" "$(SolutionDir)UnitTestProject1"
Build the project once so the Build event runs.
Then choose the UnitTestProject1 project and click on the button at the top of Solution Explorer to “Show All Files”. This will make the native Dll “NativeMemMapdDll.Dll” show in the solution explorer.
Right click on it, then choose “Include in Project”, then right click on it again->Properties.
Change the “Copy to Output Directory” property from “Do not copy” to “Copy if Newer”. This will deploy the dll to the test execution folder
The unit test runner process can’t handle large memory (Increase the memory available to your tests)
Do this once from a VS command prompt.
Editbin "C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 14.0\Common7\IDE\COMMONEXTENSIONS\MICROSOFT\TESTWINDOW\te.processhost.managed.exe" /LargeAddressAware
<NativeMemMapDll.cpp>
// NativeMemMapDll.cpp : Defines the exported functions for the DLL application.
//
#include "stdafx.h"
#include <functional>
#include <atlbase.h>
#include <map>
#include "MyMemMap.h"
typedef int(__cdecl *MyCallBack)(int mode, int index, BSTR *pbstr);
using namespace std;
const int nMaxSize = 2000;
WCHAR g_string[nMaxSize + 1] = { L'\0' };
// we need a very fast data generator so the time measured isn't in allocating/copying data
// have a single static large string of "a"s. Return a string of length == requested length mod 2000
WCHAR *makeString(int i, int *pSize = nullptr)
{
static int g_lastTimeNull = -1;
auto ndx = i % nMaxSize; // desired length
if (g_string[0] == NULL) // initialize
{
for (int i = 0; i < nMaxSize; i++)
{
g_string[i] = L'a';
}
g_string[nMaxSize] = L'\0';
}
if (g_lastTimeNull != -1) // restore string to original
{
g_string[g_lastTimeNull] = L'a';
}
g_lastTimeNull = ndx; // remember where we put a NULL
g_string[ndx] = NULL; // put a NULL at the desired length
if (pSize != nullptr)
{
*pSize = ndx;
}
return g_string;
}
LONGLONG DoMemGrowNoMemMap(WCHAR *desc, int mapMode, int numItems, MyCallBack callback)
{
map<int, CComBSTR> dictData;
LONGLONG nBytesAlloc = 0;
for (int i = 0; i < numItems; i++)
{
int len;
CComBSTR bstrfoo = makeString(i, &len);
nBytesAlloc += 2 * (len + 1);
dictData.emplace(i, bstrfoo);
}
// now verify the stored data
int i = 0;
for each (auto ent in dictData)
{
ASSERTF((!wcscmp(ent.second, makeString(i)), L"Mismatch chars"));
i++;
}
callback(3, 0, &CComBSTR(desc)); // send results back
return nBytesAlloc;
}
template< typename MemMapType>
LONGLONG DoMemMapGrowWithMemMap(WCHAR *desc, MappingMode mapMode, UINT initialFileSize, UINT initialViewSize, int numItems, MyCallBack callback)
{
MemMapType memMap("Test", mapMode, initialFileSize, initialViewSize);
map<int, MemMapType::MapFileLocator> dictData;
for (int i = 0; i < numItems; i++)
{
int nLen;
CComBSTR foo = makeString(i, &nLen);
auto nSize = 2 * (nLen + 1); // include nullterm
auto mapFileLocator = memMap.AllocateAndPut(nSize, foo);
dictData.emplace(i, mapFileLocator);
}
// now verify the stored data
// Note: cheating: the stored data is not a CComBSTR, but a copy of the data. This avoids an additional CComBSTR copy and d'tor
int i = 0;
for each (auto ent in dictData)
{
auto addr = (WCHAR *)memMap.Dereference(ent.second);
ASSERTF((!wcscmp(addr, makeString(i)), L"Mismatch chars"));
i++;
}
auto str = string(CW2A(desc)) + (mapMode == 0 ? "WithPageFile " : "WithFileOnDisk ") + memMap._stats.GetStats();
callback(3, 0, &CComBSTR(str.c_str()));
return memMap._stats.nTotalAllocs;
}
template< typename MemMapType>
LONGLONG DoMemWithDeletion(WCHAR *desc, MappingMode mapMode, UINT initialFileSize, UINT initialViewSize, int numItems, MyCallBack callback)
{
MemMapType memMap("Test", mapMode, initialFileSize, initialViewSize);
map<int, MemMapType::MapFileLocator> dictData;
for (int i = 0; i < numItems; i++)
{
int nLen;
CComBSTR bstrfoo = makeString(i, &nLen);
auto nSize = 2 * (nLen + 1); // include nullterm
auto mapFileLocator = memMap.AllocateAndPut(nSize, bstrfoo);
dictData.emplace(i, mapFileLocator);
}
for each (auto ent in dictData)
{
memMap.Deallocate(&ent.second);
}
ASSERTF((memMap._stats.nTotalDeletions == memMap._stats.nTotalAllocs, L"total deletions must be same as total allocs"));
auto str = string(CW2A(desc)) + (mapMode == 0 ? "WithDeletionPageFile " : "WithDeletionFileOnDisk ") + memMap._stats.GetStats();
callback(3, 0, &CComBSTR(str.c_str()));
return memMap._stats.nTotalAllocs;
}
template< typename MemMapType>
LONGLONG DoMemRand(WCHAR *desc, MappingMode mapMode, UINT initialFileSize, UINT initialViewSize, int numItems, MyCallBack callback)
{
MemMapType memMap("Test", mapMode, initialFileSize, initialViewSize);
srand(1); // seed
map<int, MemMapType::MapFileLocator> dictData;
using mpair = pair<int, MemMapType::MapFileLocator>;
vector<mpair> vecAllocs;
for (int i = 0; i < numItems; i++)
{
CComBSTR bstrfoo;;
callback(0, 1, &bstrfoo);
auto verb = rand() * 100.0 / RAND_MAX; //RAND_MAX == 32767
if (verb < 10)
{
for (int j = 0; j < 100; j++)
{
CComBSTR bstr2;
callback(0, j, &bstr2);
auto nSize = 2 * (SysStringLen(bstr2) + 1); // include nullterm
auto mapFileLocator = memMap.AllocateAndPut(nSize, bstr2);
dictData.emplace(i, mapFileLocator);
vecAllocs.push_back(mpair(i, mapFileLocator));
}
}
else
if (verb < 20) //delete
{
auto ndx = (UINT)(vecAllocs.size() *((double)rand() / RAND_MAX));
if (ndx < vecAllocs.size())
{
auto entToRemove = vecAllocs[ndx];
auto mhToRemove = entToRemove.second;
vecAllocs.erase(vecAllocs.begin() + ndx);
memMap.Deallocate(&mhToRemove);
}
}
else if (verb < 40) // dereference some value
{
auto ndx = (UINT)(vecAllocs.size() *((double)rand() / RAND_MAX));
if (ndx < vecAllocs.size())
{
auto ent = vecAllocs[ndx];
CComBSTR bstr((OLECHAR *)memMap.Dereference(ent.second));
callback(1, ent.first, &bstr);
}
}
else
{ // add
auto nSize = 2 * (SysStringLen(bstrfoo) + 1); // include nullterm
auto mapFileLocator = memMap.AllocateAndPut(nSize, bstrfoo);
dictData.emplace(i, mapFileLocator);
vecAllocs.push_back(mpair(i, mapFileLocator));
}
}
for each (auto ent in dictData)
{
memMap.Deallocate(&ent.second);
}
auto str = string(CW2A(desc)) + " " + (mapMode == 0 ? "RandWithPageFile " : "RandWithFileOnDisk ") + memMap._stats.GetStats();
callback(3, 0, &CComBSTR(str.c_str()));
return memMap._stats.nTotalAllocs;
}
extern "C" LONGLONG _declspec(dllexport) __stdcall DoMemMap(WCHAR *desc, int verb, int mapMode, int is64, UINT initialFileSize, UINT initialViewSize, int numItems, MyCallBack callback)
{
LONGLONG res = 0;
try
{
switch (verb)
{
case 0:
res = DoMemGrowNoMemMap(desc, 0, numItems, callback);
// MessageBoxA(0, "", "c", 0);
break;
case 1:
if (is64)
{
res = DoMemMapGrowWithMemMap<MemMap<UINT64>>(desc, (MappingMode)mapMode, initialFileSize, initialViewSize, numItems, callback);
}
else
{
res = DoMemMapGrowWithMemMap<MemMap<UINT>>(desc, (MappingMode)mapMode, initialFileSize, initialViewSize, numItems, callback);
}
break;
case 2: // with deletion
if (is64)
{
res = DoMemWithDeletion<MemMap<UINT64>>(desc, (MappingMode)mapMode, initialFileSize, initialViewSize, numItems, callback);
}
else
{
res = DoMemWithDeletion<MemMap<UINT>>(desc, (MappingMode)mapMode, initialFileSize, initialViewSize, numItems, callback);
}
break;
case 3: // random
if (is64)
{
res = DoMemRand<MemMap<UINT64>>(desc, (MappingMode)mapMode, initialFileSize, initialViewSize, numItems, callback);
}
else
{
res = DoMemRand<MemMap<UINT>>(desc, (MappingMode)mapMode, initialFileSize, initialViewSize, numItems, callback);
}
}
}
catch (const MemMapException&)
{
auto x = 2;
// callback(3, 0, &CComBSTR(ex.message)); // send results back
}
return res;
}
</NativeMemMapDll.cpp>
<MyMemMap.h>
#pragma once
#include <Windows.h>
#include "shlobj.h"
#include "shellapi.h"
#include <unordered_map>
#include <atlbase.h>
/*
Memory mapped file implementation by Calvin Hsia
see https://microsoft-my.sharepoint.com/personal/calvinh_microsoft_com/Documents/Shared%20with%20Everyone/Presentations/MapFileDict.wmv
*/
struct MemMapException : std::exception
{
MemMapException(char *msg) : exception(msg)
{
}
};
#if 1 //_DEBUG
void __cdecl vsassertf(bool fTest, WCHAR *szFmt, ...)
{
if (!fTest)
{
static BOOL fDisableThisAssert = FALSE;
if (!fDisableThisAssert)
{
va_list arglist;
WCHAR szMsg[4096];
va_start(arglist, szFmt);
_vsnwprintf_s(szMsg, _countof(szMsg), szFmt, arglist);
szMsg[_countof(szMsg) - 1] = 0;
va_end(arglist);
MessageBoxW(0, szMsg, L"MemMapFailure", 0);
throw new MemMapException(CW2A(szMsg));
// _ASSERT_EXPR(0, szMsg);
}
}
}
#define ASSERTF(args) vsassertf args
#else
#define ASSERTF(args)
#endif
using namespace std;
const UINT AllocationGranularity = 0x10000; // 64k
enum MappingMode
{
// use system paging file to store data
// advantage: don't have to worry about disk cleanup
// disadvantage: to grow, need to create a new one and copy old data into it
MappingModePageFile,
// use disk file to store data
// to grow, just recreate a file mapping with the same file and larger size. much faster to grow
// disadvantage: need to clean up when done.
MappingModeFileOnDisk
};
template <typename T> // T= UINT for 32 bit, UINT64 for 64 bit
class MemMap // for mapping MemSpect internal storage to PageFile or file on disk
{
public:
typedef struct
{
// offset into the file mapping (either UINT for up to 4 gig, or UINT64 for larger
T ulOffset;
// size of allocation in bytes
UINT ulSize;
} MapFileLocator;
MemMap(
string mapName, // unique per use in a single process.
MappingMode mapmode = MappingModePageFile,
ULONGLONG initialFileSize = AllocationGranularity,
UINT initialViewSize = 2 * AllocationGranularity
);
~MemMap();
LPVOID AllocateSpace(UINT requestSize, _Out_ MapFileLocator *pLocator);
// allocate space and put something in it
MapFileLocator AllocateAndPut(UINT requestSize, _In_ void *theData)
{
MapFileLocator locator;
auto addr = AllocateSpace(requestSize, &locator);
memcpy(addr, theData, requestSize);
return locator;
}
// Get a reference to the allocated memory
void *Dereference(MapFileLocator locator);
// free the memory (add to freelist)
void Deallocate(MapFileLocator *pLocator);
struct stats
{
T ulOffsetFreeMem; // the next available memory
UINT nFileMaps; // # of different file mappings created (# resizes)
UINT nCompactions;
UINT nAllocs; // # of alloc requests
UINT nDeletions; // # of deletion requests
UINT nAllocsUsingFreeList;
ULONGLONG nTotalAllocs; // total # of bytes allocated (never decreases)
ULONGLONG nTotalDeletions; // total # of bytes deleted (never decreases)
ULONGLONG ulMemSize; // current size of mapping file (not view size)
UINT nMapViews; // # of times a view was mapped into mem
UINT nFreeListSize;
UINT uiViewSize;
ULONGLONG initialFileSize;
string GetStats()
{
char buff[1000];
sprintf_s(buff, "#Allocs =%d #del = %d #recycle=%d TotAlloc= %lld TotDel = %lld nFileMaps=%d\r\n nMapViews = %d nCompcts = %d, nFreeListsize = %d, ulMemSize = %lld, ViewSize = %d, InitSize = %lld",
nAllocs,
nDeletions,
nAllocsUsingFreeList,
nTotalAllocs,
nTotalDeletions,
nFileMaps,
nMapViews,
nCompactions,
nFreeListSize,
ulMemSize,
uiViewSize,
initialFileSize
);
return string(buff);
}
};
stats _stats;
private:
string _mapName;
MappingMode _MappingMode;
using vecMapFileLocator = vector<MapFileLocator>;
using freeList = unordered_map<UINT, vecMapFileLocator *>;
freeList _FreeLists;
wstring outFileName;
struct MappingData
{
HANDLE _hFileMapping; // handle to the file mapping.
HANDLE _hFileHandle; // for Paging File, INVALID_HANDLE_VALUE, else FileHandle of disk file
LPVOID _mapBase; // base address of mapping
T _ulBaseOffset; // offset into file for mapping
UINT _currentMappedViewSize;
ULONGLONG _dwFileSize; // total size of underlying mapped object
MappingData() :_hFileMapping(0),
_mapBase(0),
_ulBaseOffset(0),
_currentMappedViewSize(0),
_dwFileSize(0)
{
_hFileHandle = INVALID_HANDLE_VALUE;// INVALID_HANDLE_VALUE = System paging file
}
};
void Resize(ULONGLONG newSize);
void UnmapView(LPVOID dwAddr);
MappingData _currentMapping;
const UINT nRoundVal = 4;// << 2; // power of 2 >=4
UINT Roundit(UINT dwSize)
{
auto res = dwSize;
auto remainder = res %nRoundVal;
if (remainder != 0)
{
res += nRoundVal - remainder;
}
return res;
}
MappingData CreateAMapping(ULONGLONG newSize)
{
static int mapIndex = 0; // to create a unique name per process
MappingData mapData;
WCHAR szBuf[100];
_stats.nFileMaps++;
_snwprintf_s(szBuf, _countof(szBuf), _TRUNCATE, L"%S%d_%d", _mapName.c_str(), GetCurrentProcessId(), mapIndex);
if (_MappingMode == MappingModeFileOnDisk)
{
if (_currentMapping._hFileHandle == INVALID_HANDLE_VALUE) // if we already have the file on disk. we're resizing
{
wchar_t desktopFolderName[MAX_PATH];
SHGetFolderPath(
NULL, //hWnd,
CSIDL_DESKTOP,
NULL, //token
0, //dwFlags
desktopFolderName
); // C:\Users\calvinh\Desktop
outFileName = desktopFolderName;
outFileName.append(L"\\");
outFileName += szBuf;
outFileName += L".txt";
mapData._hFileHandle = CreateFile(outFileName.c_str(),
GENERIC_WRITE | GENERIC_READ,
0,
0,
CREATE_ALWAYS,
FILE_ATTRIBUTE_TEMPORARY,
NULL);
mapIndex++; // we've use this filename already
}
else
{
mapData._hFileHandle = _currentMapping._hFileHandle; // use same handle;
}
}
else
{
mapIndex++; // ensure pagefile name is different each time
}
mapData._hFileMapping = CreateFileMappingW(
mapData._hFileHandle,
nullptr, //GetSecurityAttributes(), // security
PAGE_READWRITE,
(UINT)((newSize & (MAXULONGLONG - MAXUINT)) >> 32), // sizeHigh 0xFFFFFFFF00000000
(UINT)(newSize & MAXUINT), // sizeLow 0xFFFFFFFF
szBuf//name;
);
if (mapData._hFileMapping == 0)
{
auto err = GetLastError();
ASSERTF((false, L"Can't create file mapping %x", err));
}
mapData._dwFileSize = newSize;
return mapData;
}
LPVOID MapView(MapFileLocator locator, MappingData *mappingData = nullptr)
{
T ulOffset = locator.ulOffset;
UINT numBytesToMap = locator.ulSize;
LPVOID mappedAddress = nullptr;
auto newBaseOffset = (ulOffset / AllocationGranularity) * AllocationGranularity;
auto leftover = ulOffset - newBaseOffset;
LPVOID preferredAddress = nullptr;
if (mappingData == nullptr)
{
mappingData = &_currentMapping;
}
auto desiredViewSize = _stats.uiViewSize > _currentMapping._dwFileSize ? (UINT)_currentMapping._dwFileSize : _stats.uiViewSize;
if (mappingData->_mapBase != nullptr)
{
// if what we want to map is already mapped (from the current mapping data)
bool fFits = ulOffset >= mappingData->_ulBaseOffset &&
ulOffset + numBytesToMap < mappingData->_ulBaseOffset + mappingData->_currentMappedViewSize;
if (fFits && leftover + numBytesToMap < mappingData->_currentMappedViewSize)
{
mappedAddress = (LPVOID)((UINT)(mappingData->_mapBase) + ((UINT)(newBaseOffset - mappingData->_ulBaseOffset + leftover)));
}
else
{
preferredAddress = mappingData->_mapBase;
UnmapView(mappingData->_mapBase);
ASSERTF((numBytesToMap <= AllocationGranularity, L"#bytes to map should be AllocationGran"));
if (leftover + numBytesToMap > desiredViewSize)
{
desiredViewSize += AllocationGranularity;
}
if (newBaseOffset + desiredViewSize >= mappingData->_dwFileSize)
{
desiredViewSize = (UINT)(mappingData->_dwFileSize - newBaseOffset);
}
}
}
if (mappedAddress == nullptr)
{
_stats.nMapViews++;
// try at preferred address
mappedAddress = MapViewOfFileEx(
mappingData->_hFileMapping,
FILE_MAP_READ | FILE_MAP_WRITE,
(UINT)((newBaseOffset & (MAXULONGLONG - MAXUINT)) >> 32), // sizeHigh
(UINT)(newBaseOffset & MAXUINT), // sizeLow
desiredViewSize,
preferredAddress
);
if (mappedAddress == nullptr)
{
if (preferredAddress != nullptr)
{
// try again at any address
mappedAddress = MapViewOfFileEx(
mappingData->_hFileMapping,
FILE_MAP_READ | FILE_MAP_WRITE,
(UINT)((newBaseOffset & (MAXULONGLONG - MAXUINT)) >> 32), // sizeHigh
(UINT)(newBaseOffset & MAXUINT), // sizeLow
desiredViewSize,
nullptr
);
}
if (mappedAddress == nullptr)
{
auto err = GetLastError();
ASSERTF((0, L"Can't MapView %d %d %d", ulOffset, numBytesToMap, err));
}
}
mappingData->_mapBase = mappedAddress;
mappingData->_currentMappedViewSize = desiredViewSize;
mappingData->_ulBaseOffset = newBaseOffset;
mappedAddress = (LPVOID)((UINT)mappedAddress + (int)leftover);
}
return mappedAddress;
}
void CompactIt()
{
}
};
template <typename T>
MemMap<T>::MemMap(
string mapName,
MappingMode mapmode /*= MappingModePageFile*/,
ULONGLONG initialFileSize /*= AllocationGranularity*/,
UINT initialViewSize /*= 2 * AllocationGranularity*/
)
{
_mapName = mapName;
_MappingMode = mapmode;
_stats = { 0 };
_stats.initialFileSize = initialFileSize;
_stats.uiViewSize = initialViewSize;
}
template <typename T>
MemMap<T>::~MemMap()
{
if (_currentMapping._mapBase != 0)
{
UnmapView(_currentMapping._mapBase);
}
if (_currentMapping._hFileMapping != INVALID_HANDLE_VALUE)
{
CloseHandle(_currentMapping._hFileMapping);
}
if (_currentMapping._hFileHandle != INVALID_HANDLE_VALUE)
{ // delete the file
CloseHandle(_currentMapping._hFileHandle);
remove(CW2A(outFileName.c_str()));
}
}
template <typename T>
LPVOID MemMap<T>::AllocateSpace(UINT requestSize, _Out_ MapFileLocator *pLocator)
{
LPVOID addr = nullptr;
auto nSize = Roundit(requestSize);
_stats.nAllocs++;
MapFileLocator locator = { 0 };
if (nSize > 0)
{
if (nSize >= _stats.uiViewSize)
{
ASSERTF((false, L"requested size too big %x", nSize));
*pLocator = locator;
return addr;
}
if (_currentMapping._dwFileSize == 0) // initialization
{
Resize(_stats.initialFileSize);
}
auto res = _FreeLists.find(nSize);
int vecSize;
if (res != _FreeLists.end() &&
(vecSize = res->second->size()) > 0) // found a freelist of desired size && not empty list
{
locator = res->second->at(vecSize - 1);
res->second->pop_back();
_stats.nFreeListSize -= nSize;
_stats.nAllocsUsingFreeList++;
}
else
{
if (_stats.ulOffsetFreeMem + nSize >= _stats.ulMemSize)
{
CompactIt();
if (_stats.ulOffsetFreeMem + nSize >= _stats.ulMemSize)
{
// gotta grow
auto newSize = _stats.ulMemSize * 2;
Resize(newSize);
}
}
locator.ulOffset = _stats.ulOffsetFreeMem;
locator.ulSize = (UINT)nSize;
_stats.ulOffsetFreeMem += nSize;
}
_stats.nTotalAllocs += nSize;
addr = MapView(locator, &_currentMapping);
ASSERTF((addr != nullptr, L"Mapview failed for alloc req size %d", requestSize));
}
*pLocator = locator;
return addr;
}
template <typename T>
void MemMap<T>::UnmapView(LPVOID dwAddr)
{
auto res = UnmapViewOfFile(dwAddr);
ASSERTF((res == TRUE, L"unmapView failed %x", res));
}
template <typename T>
void *MemMap<T>::Dereference(MapFileLocator locator)
{
void *pResult = MapView(locator);
return pResult;
}
template <typename T>
void MemMap<T>::Deallocate(MapFileLocator *pLocator)
{
auto res = _FreeLists.find(pLocator->ulSize);
_stats.nDeletions++;
vecMapFileLocator *pvecMapFileLocator;
if (res == _FreeLists.end())
{
pvecMapFileLocator = new vector<MapFileLocator>();
_FreeLists[pLocator->ulSize] = pvecMapFileLocator;
}
else
{
pvecMapFileLocator = res->second;
}
pvecMapFileLocator->push_back(*pLocator);
_stats.nTotalDeletions += pLocator->ulSize;
_stats.nFreeListSize += pLocator->ulSize;
}
template <typename T>
void MemMap<T>::Resize(ULONGLONG newSize)
{
if (_currentMapping._dwFileSize == 0)
{
_stats.ulOffsetFreeMem = 0;
}
if (newSize > _stats.ulMemSize)
{
if (_currentMapping._hFileMapping != 0) // prior data?
{
if (_currentMapping._hFileHandle != INVALID_HANDLE_VALUE) // if file on disk, let's extend it
{
// first unmap
UnmapView(_currentMapping._mapBase);
// close the file mapping
CloseHandle(_currentMapping._hFileMapping);
_currentMapping = CreateAMapping(newSize);
}
else
{
MappingData newMappingData = CreateAMapping(newSize);;
UINT nBlocks = (UINT)(_stats.ulMemSize / AllocationGranularity);
for (UINT i = 0; i < nBlocks; i++)
{
MapFileLocator locator = { i * AllocationGranularity, AllocationGranularity };
auto oldAddr = MapView(locator, &_currentMapping);
auto newAddr = MapView(locator, &newMappingData);
memcpy(newAddr, oldAddr, AllocationGranularity);
}
UnmapView(_currentMapping._mapBase);
if (_currentMapping._hFileHandle != INVALID_HANDLE_VALUE) // if file on disk
{
CloseHandle(_currentMapping._hFileHandle);
}
CloseHandle(_currentMapping._hFileMapping);
_currentMapping = newMappingData;
}
}
else
{
_currentMapping = CreateAMapping(newSize);
}
_stats.ulMemSize = newSize;
}
}
</MyMemMap.h>
<UnitTest1.cs>
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.Runtime.InteropServices;
namespace UnitTestProject1
{
[TestClass]
public class UnitTest1
{
public const int AllocationGranularity = 65536;
[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
public delegate int MemMapDelegate(int mode, int itemIndex, [MarshalAs(UnmanagedType.BStr)] ref string bstr);
enum MapVerb
{
MapVerbBaseline, // just allocate memory and free in a STL map
MapVerbGrow, // use the MemMap class to store data in STL map
MapVerbWithDeletions, // add in deleting data
MapVerbRandom // add, delete, deref randomly accessed data
}
enum MappingMode
{
MappingModePageFile,
MappingModeFileOnDisk
}
[DllImport("NativeMemMapdll.dll")]
public static extern Int64 DoMemMap(
[MarshalAs(UnmanagedType.LPWStr)] string desc,
int verb,
int mapMode, // 0 pagefile, 1 fileondisk
int Is64,
uint initialFileSize, // init size of mapping file. defaults to 64k, will grow on demand and incur costs of growing
uint initialViewSize, // size of view. Defaults to 2 * 64k
int numItems,
MemMapDelegate callback);
public TestContext TestContext { get; set; }
string MakeString(int i)
{
return new string('a', i % 2000);
}
#if DEBUG
const int NumIterations = 1;
const int NumberItems = 100000;
#else
const int NumIterations = 4;
const int NumberItems = 1000000;
#endif
void DoTest(MapVerb nVerb, MappingMode mapMode, int is64, uint initialsize = 65536, uint initialViewSize = 2 * 65536, int numItems = NumberItems, int nIter = NumIterations)
{
// bug te.processhost.managed.exe is not large address aware, causing out of memory failures for large tests.
// link /dump /headers "C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 14.0\Common7\IDE\COMMONEXTENSIONS\MICROSOFT\TESTWINDOW\te.processhost.managed.exe"
// Editbin "C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 14.0\Common7\IDE\COMMONEXTENSIONS\MICROSOFT\TESTWINDOW\te.processhost.managed.exe" /LargeAddressAware
for (int iter = 0; iter < nIter; iter++)
{
var totBytesAlloc = DoMemMap(TestContext.TestName, (int)nVerb, (int)mapMode, is64, initialsize, initialViewSize, numItems,
(int mode, int i, ref string str) =>
{
if (mode == 3)
{
TestContext.WriteLine($"{DateTime.Now.ToString()}\r\n {str}");
}
else
{
if (mode == 0)
{
str = MakeString(i);
}
else
{
if (nVerb != MapVerb.MapVerbRandom)
{
Assert.IsTrue(str == MakeString(i), $"value incorrect {i}");
}
}
}
return 0;
}
);
TestContext.WriteLine($"Total bytes = {totBytesAlloc:n0}");
}
}
[TestMethod]
public void PerfGrowNoMemMap()
{
DoTest(MapVerb.MapVerbBaseline, MappingMode.MappingModePageFile, is64: 0);
}
[TestMethod]
public void PerfMemMapGrowWithPageFile()
{
DoTest(MapVerb.MapVerbGrow, MappingMode.MappingModePageFile, is64: 0);
// DoTest(MapVerb.MapVerbGrow, MappingMode.MappingModePageFile, is64: 0, initialsize: (uint)(Math.Pow(2, 20)), initialViewSize: (uint)(Math.Pow(2, 20)));
}
[TestMethod]
public void PerfMemMapGrowWithDiskFile()
{
DoTest(MapVerb.MapVerbGrow, MappingMode.MappingModeFileOnDisk, is64: 0);
}
[TestMethod]
public void PerfMemMapGrowWithDiskFile64()
{
// 40 Gig Allocate in 1 minute
DoTest(MapVerb.MapVerbGrow, MappingMode.MappingModeFileOnDisk, is64: 1);
}
[TestMethod]
public void MemMapWithDeletionPageFile()
{
DoTest(MapVerb.MapVerbWithDeletions, MappingMode.MappingModePageFile, is64: 0);
}
[TestMethod]
public void MemMapWithDeletionDiskFile()
{
DoTest(MapVerb.MapVerbWithDeletions, MappingMode.MappingModeFileOnDisk, is64: 0);
}
[TestMethod]
public void MemMapWithDiskFile64()
{
DoTest(MapVerb.MapVerbGrow, MappingMode.MappingModeFileOnDisk, is64: 1, numItems: 5000000, nIter: 1);
}
[TestMethod]
// [Ignore]
public void MemMapRandWithDiskFile64()
{
DoTest(MapVerb.MapVerbRandom, MappingMode.MappingModeFileOnDisk, is64: 1, numItems: 500000, nIter: 1);
}
[TestMethod]
public void MemRand()
{
DoTest(MapVerb.MapVerbRandom, MappingMode.MappingModePageFile, is64: 0, numItems: 100000);
}
[TestMethod]
public void MemRandDiskFile()
{
DoTest(MapVerb.MapVerbRandom, MappingMode.MappingModeFileOnDisk, is64: 0, numItems: 100000);
}
}
}
</UnitTest1.cs>