Troubleshooting a paged pool leak in Windows 

Operating Systems

Windows server 2008,

Windows Server 2008 R2,

Windows server 2003,

Windows server 2003 R2

Problem

Here we will fix a real paged pool leak on your system so that we can put to use the techniques to track down the leak.

Solution

When a program needs to store some temporary information during execution, it can dynamically request a chunk of memory from the system. However, the system has a fixed amount of total memory available. If one application uses up all of the system’s free memory, then other applications will not be able to obtain the memory that they require. The implications of a “memory starved” application can range from a graceful shutdown to an unexpected crash. Most large scale applications regularly request memory, so running out of system memory tends to have a domino effect. Even if the applications do not terminate, the system will slow down to a crawl—or even hang—in low memory conditions. Clearly, none of these results are desirable, so the system never wants to run out—or run low—of memory.

responsibility of each application to “free” dynamically requested memory when they are finished using it. Freeing the memory returns it to the system, where it can be re–allocated to another application when needed. When an application dynamically allocates memory, and does not free that memory when it is finished using it, that program has a memory leak. The memory is not being used by the application anymore, but it cannot be used by the system or any other program either.

Memory leaks add up over time, and if they are not cleaned up, the system eventually runs out of memory. Most everyone has seen the “Your computer is running low of virtual memory” message box on Windows when memory gets too high. It is typically accompanied by horribly slow response time, and often the user can’t even close the wasteful application because of this sluggishness. The only response at that point is to reboot the computer.

Memory leaks are bad on any system. For the automotive platform, however, they become much more of a significant problem. This is due to the following:

Applications stay in memory when not in use. Typically, the operating system can clean up after a program which leaks memory after it terminates. However, since we do not allow the users to easily shut down automotive applications, Windows CE cannot clean up after any ill–behaved applications.

System run–time is expected to be unlimited. If all else fails, a system reboot will clean up all memory leaks. However, although we do support system reboots, our goal is that the user will never need to do so. Since you do not need to “reboot” your car stereo every few weeks because it stops working, or because the performance becomes poor, it is unacceptable to need to do so on the automotive platform.

Systems typically have lower total available system memory. Desktop applications can get away with being sloppy with memory when they run on systems with 64 megabytes of RAM and more, and with large hard drives to store swapped out virtual memory pages. But the automotive platform may have only 16 megabytes total memory, and that includes process and storage memory together! In order words, that memory must store what would be in RAM on a desktop PC, as well as what would be on the hard drive.

Paged and nonpaged pools serve as the memory resources that the operating system and device drivers use to store their data structures. The pool manager operates in kernel mode, using regions of the system’s virtual address space (described in the Pushing the Limits post on virtual memory) for the memory it sub-allocates. The kernel’s pool manager operates similarly to the C-runtime and Windows heap managers that execute within user-mode processes.  Because the minimum virtual memory allocation size is a multiple of the system page size (4KB on x86 and x64), these subsidiary memory managers carve up larger allocations into smaller ones so that memory isn’t wasted.

Coming to our experiment, we are used number of utilities for testing the memory leakage issues.

Resources used:

• NotMyFault 

• Process Explorer 

• Poolmon 

• Strings 

• TaskManger

It is the responsibility of each application to “free” dynamically requested memory when they are finished using it. Freeing the memory returns it to the system, where it can be re–allocated to another application when needed. When an application dynamically allocates memory, and does not free that memory when it is finished using it, that program has a memory leak. The memory is not being used by the application anymore, but it cannot be used by the system or any other program either.

Memory leaks add up over time, and if they are not cleaned up, the system eventually runs out of memory. Most everyone has seen the “Your computer is running low of virtual memory” message box on Windows when memory gets too high. It is typically accompanied by horribly slow response time, and often the user can’t even close the wasteful application because of this sluggishness. The only response at that point is to reboot the computer.

Memory leaks are bad on any system. For the automotive platform, however, they become much more of a significant problem. This is due to the following:

Applications stay in memory when not in use. Typically, the operating system can clean up after a program which leaks memory after it terminates. However, since we do not allow the users to easily shut down automotive applications, Windows CE cannot clean up after any ill–behaved applications.

System run–time is expected to be unlimited. If all else fails, a system reboot will clean up all memory leaks. However, although we do support system reboots, our goal is that the user will never need to do so. Since you do not need to “reboot” your car stereo every few weeks because it stops working, or because the performance becomes poor, it is unacceptable to need to do so on the automotive platform.

Systems typically have lower total available system memory. Desktop applications can get away with being sloppy with memory when they run on systems with 64 megabytes of RAM and more, and with large hard drives to store swapped out virtual memory pages. But the automotive platform may have only 16 megabytes total memory, and that includes process and storage memory together! In order words, that memory must store what would be in RAM on a desktop PC, as well as what would be on the hard drive.

Paged and nonpaged pools serve as the memory resources that the operating system and device drivers use to store their data structures. The pool manager operates in kernel mode, using regions of the system’s virtual address space (described in the Pushing the Limits post on virtual memory) for the memory it sub-allocates. The kernel’s pool manager operates similarly to the C-runtime and Windows heap managers that execute within user-mode processes.  Because the minimum virtual memory allocation size is a multiple of the system page size (4KB on x86 and x64), these subsidiary memory managers carve up larger allocations into smaller ones so that memory isn’t wasted.

Coming to our experiment, we are used number of utilities for testing the memory leakage issues.

Resources used:

Except Task manager all the utilities are downloaded from Microsoft Sysinternals.

To download simply hold Ctrl key and select the item.

The leak will be generated by the NotMyFault tool from Sysinternals.

When we run NotMyFault.exe, it loads the device driver Myfault.sys and presents the following dialog box

Click the Leak Paged button. This causes NotMyFault to begin sending requests to the Myfault device driver to allocate paged pool. (Do not click the Do Bug button or you will experience a system crash; this button is used todemonstrate various types of crashes.)

NotMyFault will continue sending requests until you click the Stop Paged button. Note that paged pool is not normally released even when you close a program that has caused it to occur (by interacting with a buggy device driver); the pool is permanently leaked until you reboot the system. However, to make testing easier, the Myfault device driver detects that the process was closed and frees its allocations.

While the pool is leaking, first open Task Manager and click on the Performance tab. You should notice Kernel Memory (MB): Paged climbing.

You can also check this with Process Explorer’s System Information display. (Click on View and then System Information.)

To determine the pool tag that is leaking, run Poolmon and press the B key to sort by the number of bytes.

 Press P twice so that Poolmon is showing only paged pool. You should notice the pool tag “Leak” climbing to the top of the list. (Poolmon shows changes to pool allocations by highlighting the lines that change.)

 Now press the Stop Paged button so that you don’t exhaust paged pool on your system.

 Run Strings (from Sysinternals) to look for driver binaries that contain the pool tag “Leak”:

Strings %SystemRoot%\system32\drivers\*.sys | findstr Leak

This should display a match on the file Myfault.sys, thus confirming it as the driver using the “Leak” pool tag.

Once we had identified the device driver, we could identify the manufacturer and work on it for the resolution..

It’s worth bearing in mind that the same technique can also be used to troubleshoot paged pool problems as well.