Making parts of Windows CE Device Driver Code Non-Pageable
Posted by Wes Barcalow
Following on to Sue’s previous posts describing the paging pool and memory management, I wanted to talk about how drivers can be made pageable for additional virtual memory savings.
Windows CE has features to allow for more data and code to be used on a device than the available RAM. It does this by ‘paging’ resources into RAM from fixed or read-only storage (ROM/Flash), and discarding pages if the overall amount of RAM available in the system becomes too low. In systems where code cannot execute directly from ROM, this paging is the only available way to use storage to offset RAM usage. This is the case for NAND Flash, which is more perfomant and of lower cost than NOR Flash (which does allow XIP or eXecute In Place).
Some code and data in the system is read (‘paged’) into RAM and ‘locked’ there – it is marked as non-pageable after it is loaded. This code and data must be available, namely when the storage it was retrieved from is no longer available. For example, to achieve the best power saving on entry to a low-power or deep-idle mode, it is preferable to turn off the power to a NAND Flash chip.
Applications are typically pageable, since the operating system completely stops the threads of applications before entering a low power mode. At this point, since the application's code will not be executed and it's data cannot be accessed, such code and data can be ‘paged out’ and is not needed. For device drivers things are slightly different. Most drivers written for Windows CE / Windows Mobile are by default loaded non-pageable by device manager. This means that no matter how big the driver is, it takes up all the RAM it wants to once it is loaded – none of it can be paged out. In the case of user mode drivers, udevice.exe loads the driver instead of device manager, but it too uses the same criteria for choosing between pageable and non-pageable modes.
With an increase of functionality or flexibility in a driver comes an increase in size. A camera driver that supports many formats or many features may be very large. However, if the camera is not used for a long time, then the RAM resources taken up by it is not being put to efficient use. It makes sense to make this type of driver pageable by default instead.
To make a driver pageable, these steps have to be taken.
1) Tell device manager you want the driver to be pageable.
2) Tell the kernel that pageable mode is allowed.
3) Identify and flag code that is needed to be non-pageable.
The last step is slightly more complicated than the first two steps. Even though you may have a large driver, you may still need portions of it to be non-pageable. The most important parts of a Windows CE / Windows Mobile driver that cannot be paged out are functions that execute when the file system is not in operation. If you do not have such functions in your driver then you do not have to worry about making them non-pageable.
Marking a driver as pageable needs to happen in two steps; the first is with a registry setting for that driver. It may or may not already have a “Flags” registry entry. To enable the driver to be paged ensure there is a registry value named “Flags” of type DWORD entry and that in its value the DEVFLAGS_LOADLIBRARY bit is set (0x02). If there are other flag bits set, simply logical ‘or’ this with what is already there.
Here is an example of what a GPIO driver registry setting might look like in platform.reg:
[HKEY_LOCAL_MACHINE\Drivers\BuiltIn\GPIO]
"Dll"="gpio.dll"
"Flags"=dword:10002 ;Trusted caller only & pageable
...
The second step for marking a driver pageable is ensuring the ‘M’ flag of the binary image builder file (BIB file) is not set. The purpose of the ‘M’ flag is to inform the kernel not to demand page the driver, thus forcing the driver to be completely loaded into RAM.
Here is an example of what a GPIO bib file entry might look like that allows the driver to be loaded in a pageable mode by the kernel:
msm7x00_gpio.dll $(_FLATRELEASEDIR)\gpio.dll NK SH
Notice the flags at the end of the statement, there is no ‘M’ flag. A user wishing to force the driver into a non-pageable mode would use “SHM” instead of “SH”. Or alternatively, a user wishing to force the driver into a non-pageable mode would clear the DEVFLAGS_LOADLIBRARY bit in the registry. Either approach is valid.
It is also worth pointing out that a trusted user can potentially change the registry after run time, thus changing a driver from non-pageable to pageable and back again. The bib file flag, however, is built into the image and cannot be overridden. Both are viewed as equally secure as only a trusted caller can change the registry, though the bib file flag provides a predictable pageable status when loading the driver.
The final, more complicated step from above is to identify and isolate code that can’t be pageable. As mentioned above, this is code that runs in single threaded mode where the file system cannot page in or out code and data. The most well-known examples of this are:
- XXX_PowerUp
- XXX_PowerDown
- Interrupt Service Threads and Interrupt Service Routines (ISTs and/or ISRs) that may execute while the file system is inactive.
- Read-Only constants that are accessed by these functions.
- Any supporting code called by these functions.
- All code associated with the file system path, as it is responsible for bringing in new pages.
Once the code is identified, it should be wrapped in compiler #pragma statements to inform the linker about the properties of the code. Below is an example of making xxx_PowerUp and xxx_PowerDown non-pageable.
#pragma comment(linker, "/section:.no_page,ER!P")
#pragma code_seg(push, ".no_page")
XXX_PowerDown()
{
//Perform single-threaded power off logic
}
XXX_PowerUp()
{
//Perform single-threaded power on logic
}
UtilityFuncOne()
{
// Non-Paged utility function that can be called by
// both page and non-paged code
}
#pragma code_seg(pop)
UtilityFuncTwo()
{
// Paged utility function that can only be called by other
// paged code.
}
This sample code shows the XXX_PowerDown and XXX_PowerUp code being marked as pageable. This will allow the processor to access this code in RAM while the file system is not in operation (during suspend and resume operations). UtilityFuncOne is also in the non-paged section of code, thus making it safe to call from within XXX_PowerUp/Down. However the UtilityFuncTwo code is outside of the non-paged area, and therefore pageable and at risk of not being available if the processor were to try to access it while performing suspend / resume operations.
To test for drivers marked as pageable that are critical to suspend, resume, and shutdown code paths the registry key PageOutAllModules can be used to instruct the kernel to page out all code. This can be used to find drivers that use pageable code when calling XXX_PowerUp and XXX_PowerDown API’s while the file system is inactive. By generating page faults, problematic drivers can be identified more easily. Below is what the registry key looks like:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Power]
"PageOutAllModules"=dword:1
Set this registry key and force the system to suspend, resume, or shutdown. The OS will then page out all code marked as pageable and proceed with suspend / resume / shutdown operation. If a critical driver is improperly marked as pageable then this process will generate a page fault and device will die. This technique will help ensure that all drivers a properly marked as pageable/non-pageable when preparing to release the device to market.
By making your driver pageable you can decrease the load on the system for resources while a component or feature is not being used. It is important to take care as outlined above to make sure some important parts of your driver can still function even though in general the bulk of it is ‘paged’.
Comments
- Anonymous
February 28, 2008
PingBack from http://www.biosensorab.org/2008/02/28/making-parts-of-windows-ce-device-driver-code-non-pageable/ - Anonymous
March 01, 2008
Hey I came to know that exFAT support is handled by the CE team, not Windows team. Do you plan to make exFAT available for XP? What's the point of keeping it exclusive to Vista if it's free? It'll only hamper adoption. - Anonymous
March 03, 2008
Thanks for the informative post. One clarification, data is not paged out in the general case because there is no backing store to page it to. The post refers to code and data being paged out, but that is only true for the special case of readonly data.Regards,Dean