High Pages out in Windows 2003 64 bit

Question

Recently we've installed ten application servers to help our AS/400. We've moved all the users to the logon group and all uses only this servers.
As I said the central instance is AS/500 v5r3, the aplication server are Windows 2003 64 with sp1.The kernel is also 64 bit ( 640 unicode ). The sap release is 4.7 enterprise. The configuration of the server is:
- 2 Opteron processor with two kernel each.
- 16 Gb of RAM.
- 20 Gb of paging file ( following the SAP recommendations ).
The sap parameters have also been changed. Using perfmon and the st06 transactions i can see that the page out is too high. Is any reason for that ? Any recommendation ?

Answer 1

no, not yet. But we will do, when we have a fix for it. Until then the situation is as it has developed in the past 12 Years:
The page out story is as old as SAP R/3 on Windows and related to the mechanism called extended memory within R/3 and Windows Memory Management. It is not a special issue to the 64 Bit Windows platforms.
In order to support bigger memory configurations on 32 Bit Windows and Linux extended memory (ABAP Memory) is implemented in a way that only the active user context of an ABAP user is part of a work process virtual address space. Before a work process processes a user request the memory pages of the user context (abap memory of the modus etc) are mapped into the work process virtual address space. At the end of the processing in the work process the context is rolled out - the related memory pages are detached from the work process virtual address space.
In Windows Memory Management this action (unmapping) is handeld in a similar way as the Working Set Trimmer works (which gets active when system comes in a short of memory state): unmapped unmodified pages where added to a standby list, modified pages are added to the modified page list. In all Windows versions the pagewriter will start cleaning up the modified page list (writing modified pages back to their backing store -pagefile- the page changes the status to unmodified and is added to the standby list) if the modified pages list contains more than a certain number of pages. The different Windows versions have different algorythms how many pages are written back to pagefile, but the threshold for starting the action is identically from Windows 2000 Professional to Windows 2003 Server Datacenter Edition for IA64: 800 Pages - or 3.2 MB (a very,very small number)
In the past Microsoft explains that the reason for this behaviour is related to prepare memory shortage situations: as long as the system has sufficient resources it will preventively page out modified memory pages in a moderately manner. In case of a memory shortage they can drop the pages off the standby list and can immediately use the physical memory for new memory requests, without first writing them to pagefile. When Windows 2003 was introduced we obtained a significantly higher pagout rate compared to Windows 2000. Microsoft explains this with the necessity of some changes in memory management caused of kernelmode locks we had in some configurations. As long as sufficient memory is provided these pagesouts should not be a bootleneck for the overall performance of the system, claimed Microsoft.
Comparing the situation today and 7 years (4.6C days) ago you can see, thattwo factors have changed dramatically:
1. processor performance has increased rapidly.
2. the average size of a user context increased in a similar way and with them the number of modified pages in a transaction.
but one thing did not change that fast: disk spead. capacity has increased a lot but the number of spindles, surfaces and RPMs are stil the same. So a random accesses to a 4k page needs nearly the same time as 3 years ago.
we have shown Microsoft where the problem sticks in their operating system by modifying the Windows Management variable which defines the modified page list maximum in the Windows kernel using the Windows Kernel Debugger.
It immediately brings down pageouts to 0 the overall pageout rate is reasonable.
It's like Doping: it helps a lot, but is completly unsupported and you can't oversee the side effects.
But you know: changing this part of Windows code needs extended testing - there is a lot of other software running on them - not only SAP and fixing this issue must not harm the performance of any other software (like Database, Exchange, Internet Information Server......).
Maybe it helps to open a message at Microsoft, in order to show them they have customers having some problems with the operating system performance.
regards
Peter
regards
Peter