Short term memory allocator and Cache memor is out of memory

Question

Hi,
I have three NW 6.5 sp8 servers in production. One of these, the one which holds Pervasive sql 9.7 began to show the following errors:
Cache memory allocator out of available memory.
Short term memory allocator is out of memory.
360396 attempts to get more memory failed.
request size in bytes 1048576 from Module SERVER.NLM
I show here segstats.txt:
*** Memory Pool Configuration for : DBASE_SERVER
Time and date : 02:42:36 AM 12/02/2012
Server version : NetWare 6.5 Support Pack 8
Server uptime : 11d 04h 35m 28s
SEG.NLM uptime : 0d 00h 01m 17s
SEG.NLM version : v2.00.17
Original Memory : 4,292,812,800 bytes (4.00 GB)
ESM Memory : 805,302,272 bytes (768.0 MB)
0xFFFFFFFF --------------------------------------------------------------
| Kernel Reserved Space |
| |
| Size : 180,355,071 bytes (172.0 MB) |
| |
0xF5400000 --------------------------------------------------------------
| User Address Space (L!=P) |
| |
| User Pool Size : 671,088,640 bytes (640.0 MB) |
| High Water Mark : 559,710,208 bytes (533.8 MB) |
| PM Pages In Use : 1,855,488 bytes (1.8 MB) |
| |
0xCD400000 --------------------------------------------------------------
| Virtual Memory Address Space (L!=P) |
| |
| VM Address Space : 2,369,781,760 bytes (2.21 GB) |
| Available : 801,435,648 bytes (764.3 MB) |
| Total VM Pages : 800,870,400 bytes (763.8 MB) |
| Free Clean VM : 785,563,648 bytes (749.2 MB) |
| Free Cache VM : 15,306,752 bytes (14.6 MB) |
| Total LP Pages : 0 bytes (0 KB) |
| Free Clean LP : 0 bytes (0 KB) |
| Free Cache LP : 0 bytes (0 KB) |
| Free Dirty : 0 bytes (0 KB) |
| NLM Memory In Use : 1,767,256,064 bytes (1.65 GB) |
| NLM/VM Memory : 1,751,785,472 bytes (1.63 GB) |
| Largest Segment : 2,097,152 bytes (2.0 MB) |
| Lowest Kernel Page: 0 bytes (0 KB) |
| : [0x00000000] |
| High Water Mark : 2,243,096,576 bytes (2.09 GB) |
| Alloc Failures : 370,804 |
| |
0x40000000 --------------------------------------------------------------
| File System Address Space (L==P or L!=P) |
| |
| FS Address Space : 1,067,290,624 bytes (1017.8 MB) |
| Available : 108,978,176 bytes (103.9 MB) |
| Largest Segment : 3,362,816 bytes (3.2 MB) |
| |
| NSS Memory (85%) : 613,683,200 bytes (585.3 MB) |
| NSS (avail cache) : 610,455,552 bytes (582.2 MB) |
| |
0x00627000 --------------------------------------------------------------
| DOS / SERVER.NLM |
| |
| Size : 6,451,200 bytes (6.2 MB) |
| |
0x00000000 --------------------------------------------------------------
Total NLMs loaded on the server: 307
Top 20 Memory Consuming NLMs
NLM Name Version Date Total NLM Memory
================================================== =============
1. NWMKDE.NLM 9.70.07 Nov 14, 2008 813,035,623 bytes (775.4 MB)
2. SERVER.NLM 5.70.08 Oct 3, 2008 467,216,096 bytes (445.6 MB)
3. NSS.NLM 3.27.02 Nov 11, 2009 203,168,848 bytes (193.8 MB)
4. NCPL.NLM 3.02 May 6, 2009 41,854,837 bytes (39.9 MB)
5. NWSQLMGR.NLM 9.70.07 Nov 14, 2008 39,309,132 bytes (37.5 MB)
6. DS.NLM 20217.07 Jan 30, 2009 24,851,303 bytes (23.7 MB)
7. APACHE2.NLM 2.00.63 Apr 25, 2008 19,863,493 bytes (18.9 MB)
8. CIOS.NLM 1.60 Feb 12, 2008 10,569,567 bytes (10.1 MB)
9. OWCIMOMD.NLM 3.02 Nov 27, 2007 9,318,616 bytes (8.9 MB)
10. APRLIB.NLM 0.09.17 Apr 25, 2008 8,959,760 bytes (8.5 MB)
11. APACHE2.NLM 2.00.63 Apr 25, 2008 7,702,469 bytes (7.3 MB)
12. FATFS.NLM 1.24 Aug 27, 2007 5,859,413 bytes (5.6 MB)
13. NWPA.NLM 3.21.02 Oct 29, 2008 4,990,686 bytes (4.8 MB)
14. PKI.NLM 3.32 Aug 25, 2008 4,069,957 bytes (3.9 MB)
15. WS2_32.NLM 6.24.01 Feb 14, 2008 3,623,596 bytes (3.5 MB)
16. NWMPM100.NLM 9.70.07 Nov 14, 2008 3,597,747 bytes (3.4 MB)
17. NWODBCEI.NLM 9.70.07 Nov 14, 2008 3,459,159 bytes (3.3 MB)
18. PORTAL.NLM 4.03 Sep 22, 2008 3,404,576 bytes (3.2 MB)
19. JVM.NLM 1.43 Oct 16, 2008 2,701,919 bytes (2.6 MB)
20. NLDAP.NLM 20218.11 Jan 30, 2009 2,579,131 bytes (2.5 MB)
Top 20 NLM - Memory Trends
NLM Name Original Memory Current Change
================================================== =========
1. NWMKDE.NLM 842,068,071 bytes 813,035,623 bytes -27.7 MB
2. SERVER.NLM 463,894,240 bytes 467,216,096 bytes 3.2 MB
3. NSS.NLM 203,168,848 bytes 203,168,848 bytes 0 KB
4. NCPL.NLM 41,850,741 bytes 41,854,837 bytes 4 KB
5. NWSQLMGR.NLM 39,092,044 bytes 39,309,132 bytes 212 KB
6. DS.NLM 24,896,359 bytes 24,851,303 bytes -44 KB
7. APACHE2.NLM 19,855,301 bytes 19,863,493 bytes 8 KB
8. CIOS.NLM 10,569,567 bytes 10,569,567 bytes 0 KB
9. OWCIMOMD.NLM 9,277,656 bytes 9,318,616 bytes 40 KB
10. APRLIB.NLM 8,959,760 bytes 8,959,760 bytes 0 KB
11. APACHE2.NLM 7,702,469 bytes 7,702,469 bytes 0 KB
12. FATFS.NLM 5,859,413 bytes 5,859,413 bytes 0 KB
13. NWPA.NLM 4,957,918 bytes 4,990,686 bytes 32 KB
14. PKI.NLM 4,135,493 bytes 4,069,957 bytes -64 KB
15. WS2_32.NLM 3,619,500 bytes 3,623,596 bytes 4 KB
16. NWMPM100.NLM 3,597,747 bytes 3,597,747 bytes 0 KB
17. NWODBCEI.NLM 3,459,159 bytes 3,459,159 bytes 0 KB
18. PORTAL.NLM 3,400,480 bytes 3,404,576 bytes 4 KB
19. JVM.NLM 2,701,919 bytes 2,701,919 bytes 0 KB
20. NLDAP.NLM 2,505,403 bytes 2,579,131 bytes 72 KB
Logical Memory Summary Information
================================================== ===============================
File System Cache Information
FS Cache Free : 4,591,616 bytes (4.4 MB)
FS Cache Fragmented : 104,386,560 bytes (99.6 MB)
FS Cache Largest Segment : 3,362,816 bytes (3.2 MB)
Logical System Cache Information
LS Cache Free : 0 bytes (0 KB)
LS Cache Fragmented : 722,448,384 bytes (689.0 MB)
LS OS Reserved Data : 333,455,360 bytes (318.0 MB)
LS Cache Largest Segment : 2,097,152 bytes (2.0 MB)
LS Cache Largest Position : 2DE00000
Summary Statistics
Total Address Space : 4,294,967,296 bytes (4.00 GB)
Total Free : 4,591,616 bytes (4.4 MB)
Total Fragmented : 826,834,944 bytes (788.5 MB)
Highest Physical Address : CFE53000
User Space : 671,088,640 bytes (640.0 MB)
User Space (High Water Mark) : 559,710,208 bytes (533.8 MB)
NLM Memory (High Water Mark) : 2,243,096,576 bytes (2.09 GB)
Kernel Address Space In Use : 2,572,759,040 bytes (2.40 GB)
Available Kernel Address Space : 43,929,600 bytes (41.9 MB)
Memory Summary Screen (.ms)
================================================== ===============================
KNOWN MEMORY Bytes Pages Bytes Pages
Server: 3487425552 851422 Video: 8192 2
Dos: 86000 20 Other: 131072 32
FS CACHE KERNEL NLM MEMORY
Original: 3483172864 850384 Code: 46854144 11439
Current: 108978176 26606 Data: 27242496 6651
Dirty: 0 0 Sh Code: 49152 12
Largest seg: 3362816 821 Sh Data: 20480 5
Non-Movable: 81920 20 Help: 172032 42
Other: 4235538432 4292855635 Message: 1236992 302
Avail NSS: 610439168 149033 Alloc L!=P: 1661366272 405607
Movable: 8192 2 Alloc L==P: 14843904 3624
Total: 1751785472 427682
VM SYSTEM
Free clean VM: 785563648 191788
Free clean LP: 0 0
Free cache VM: 15306752 3737
Free cache LP: 0 0
Free dirty: 0 0
In use: 1855488 453
Total: 801435648 195663
Memory Configuration (set parameters)
================================================== ==============================
Auto Tune Server Memory = ON
File Cache Maximum Size = 1073741825
File Service Memory Optimization = 1
Logical Space Compression = 1
Garbage Collection Interval (ON) = 299.9 seconds
VM Garbage Collector Period (ON) = 300.0 seconds
server -u<number> = 671088640
NSS Configuration File:
C:\NWSERVER\NSSSTART.CFG
File does not exist,
or is zero byte in size.
DS Configuration File:
SYS:\_NETWARE\_NDSDB.INI
File does not exist,
or is zero byte in size.
TSAFS Memory Information/Configuration
================================================== ==============================
Cache Memory Threshold : 1%
Read Buffer Size : 65536 bytes
Max Data Sets for Read Ahead : 2
Read Threads Per Job : 4
NSS Memory Information/Configuration
================================================== ==============================
Current NSS Memory Settings
Cache Balance Percentage : 85%
Cache Memory Allocated : 585.3 MB
Available Cache from NSS : 582.2 MB
Current NSS Caching Percentages
Buffer cache hit percentage : 63%
Name Tree cache hit percentage : 94%
File cache hit percentage : 99%
NSS Flush Status: Not Flushed
Server High/Low Water Mark Values
================================================== ==============================
NLM Memory High Water Mark = 2,243,096,576 bytes
File System High Water Mark = 443,108 bytes
User Space Information:
User Space High Water Mark = 559,710,208 bytes
Committed Pages High Water Mark = 87 pages
Mapped VM Pages High Water Mark = 3,875 pages
Reserved Pages High Water Mark = 400,103 pages
Swapped Pages High Water Mark = 3,785 pages
Available Low Water Mark = 294,670,336
ESM Memory High Water Mark = 173 pages
It seems that server.nlm is growing without limits. When tat occurs, I have the mentioned errors.
Though NWMKDE seems to have grown. It remains steady around the showed values.
I'm not brave enough to apply the memcalc's recommended fixes because the following line:
set file cache maximum size=822083584
returns an error saying the minimun value should be 1073741824.
Can someone help me because I'm completely blind here.
Thanks in advance.
Gabriel

Answer 1

I take it this is primarily a database server, in which case it's OK that Btrieve is using so much memory? You wouldn't want this to be a general file server too. Is the memory error causing any actual problem?
Server is asking for only 1mb, and due to fragmentation there is little free memory (actually 2mb left, which is a little odd, but neither here nor there).
Also, let's see your bti.cfg, which is the Btrieve config file. I'll paste in below an ArcServe TID on Btrieve using excessive memory:
Symptoms
Btrieve was upgraded to version 8.5 during the installation of ARCserve r11.1. The cachesize in the BTI.cfg microkernel section is at 20 MB (20480). (Pervasive would like this setting placed to 20% of the server memory or database size which ever is less.) The server will keep adding 20 additional Megs of memory to the total amount of memory the server is using for database transactions after each backup job. This can be verified by performing the following at the server console:
LOAD MONITOR
Scroll down to System Resources under Available Options and hit enter.
Scroll down to Alloc Memory (Bytes) and hit enter.
Locate NWMKDE.nlm in the Resource Tags list.
Sort by memory bytes and you will slowly see nwmkde.nlm move to the top of the usage list. Unless the server is rebooted the small memory allocations stays at the increased amount.
Explanation
Starting with Btrieve version 8.5 and higher, Pervasive has been working to make the Btrieve database more dynamic. They have created a two-tier memory allocation approach. The first level is controlled by the cache size setting in the BTI.cfg. If this becomes inadequate, the second level will be accessed. The default setting for the second level is 60% of the server's total memory.
The following line in the BTI.cfg will control the second level of memory caching:
MaxCacheUsage=60; default is 60% of memory.
An example would be a server with 100 MB of memory and the following settings in sys:\system\bti.cfg:
[microkernel]
cachesize=20480
MaxCacheUsage=60
This will cause the nwmkde.nlm to use 20 MB (20480) of memory initially and grow up to 60 percent of the total server memory or 60 MB.
Now you also have to throw Max worker threads into the mix. A setting of Max worker threads = 3 in the BTI.cfg > Btrieve Communications Manager section will also use server memory. It will use 1 MB per thread. In this example, 3 Megs of additional memory will be used. That will bring the total amount of memory used by nwmkde.nlm to 20 MB (20480) + 3 MB = 23 MB when the server is first booted. After running some backups, this number could go up to as high as 60 MB (60% of server memory) if the server dynamically requires it.
Resolution
The MaxCacheUsage=60 setting must be set down from this 60% number. Pervasive recommends setting this from 0 to 20. The server needs to be rebooted for this change to take effect.