Mpss support on solaris 10

Question

I would like to enable large heap pages for an application that may be experiencing high TLB miss rates, but I am having difficulty getting the tools to manipulate the page size to work correctly.
On SPARC, I can get the stack to use large pages, but not the heap. On x86, I cannot get either the stack or the heap to use large pages.
Here is a transcript of some experiments to adjust the page size for 'cat', first run on SPARC, and then repeated on an x86 machine.
user@sparc ~> cat /etc/release
Solaris 10 1/06 s10s_u1wos_19a SPARC
Copyright 2005 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 07 December 2005
user@sparc ~> cat
^Z
Suspended
user@sparc ~> pmap -s `pgrep cat` | egrep "stack|heap"
0002A000 16K 8K rwx-- [ heap ]
FFBFE000 8K 8K rw--- [ stack ]
user@sparc ~> fg
cat
^C
user@sparc ~> pagesize -a
8192
65536
524288
4194304
user@sparc ~> ( setenv LD_PRELOAD mpss.so.1 ; setenv MPSSHEAP 4M ; setenv MPSSSTACK 4M ; cat )
^Z
Suspended
user@sparc ~> pldd `pgrep cat`
23276: cat
/usr/lib/mpss.so.1
/lib/libc.so.1
/lib/libgen.so.1
/platform/sun4u/lib/libc_psr.so.1
user@sparc ~> pmap -s `pgrep cat` | egrep "stack|heap"
0002A000 16K 8K rwx-- [ heap ]
0002E000 3912K - rwx-- [ heap ]
FF800000 4096K 4M rw--- [ stack ]
user@sparc ~> fg
( setenv LD_PRELOAD mpss.so.1; setenv MPSSHEAP 4M; setenv MPSSSTACK 4M; cat )
^C
user@sparc ~> ppgsz -o heap=4M,stack=4M cat
^Z
Suspended
user@sparc ~> pmap -s `pgrep cat` | egrep "stack|heap"
0002A000 16K 8K rwx-- [ heap ]
0002E000 3912K - rwx-- [ heap ]
FF800000 4096K 4M rw--- [ stack ]
user@sparc ~> fg
ppgsz -o heap=4M,stack=4M cat
^C
user@sparc ~>
Here, using either mpss.so.1 or ppgsz, I can adjust the stack page size, but not the heap.
The same set of tests run on x86:
user@x86 ~> cat /etc/release
Solaris 10 6/06 s10x_u2wos_09a X86
Copyright 2006 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 09 June 2006
user@x86 ~> cat
^Z
Suspended
user@x86 ~> pmap -s `pgrep cat` | egrep "stack|heap"
08047000 4K 4K rw--- [ stack ]
0806A000 12K 4K rw--- [ heap ]
user@x86 ~> fg
cat
^C
user@x86 ~> pagesize -a
4096
2097152
user@x86 ~> ( setenv LD_PRELOAD mpss.so.1 ; setenv MPSSHEAP 2M ; setenv MPSSSTACK 2M ; cat )
^Z
Suspended
user@x86 ~> pldd `pgrep cat`
19155: cat
/usr/lib/mpss.so.1
/lib/libc.so.1
/lib/libgen.so.1
user@x86 ~> pmap -s `pgrep cat` | egrep "stack|heap"
08047000 4K 4K rw--- [ stack ]
0806A000 12K 4K rw--- [ heap ]
0806D000 1612K - rw--- [ heap ]
user@x86 ~> fg
( setenv LD_PRELOAD mpss.so.1; setenv MPSSHEAP 2M; setenv MPSSSTACK 2M; cat )
^C
user@x86 ~> ppgsz -o heap=2M,stack=2M cat
^Z
Suspended
user@x86 ~> pmap -s `pgrep cat` | egrep "stack|heap"
08047000 4K 4K rw--- [ stack ]
0806A000 12K 4K rw--- [ heap ]
0806D000 1612K - rw--- [ heap ]
user@x86 ~> fg
ppgsz -o heap=2M,stack=2M cat
^C
user@x86 ~>
In this case, I cannot adjust either the heap or the stack size with either method.
The sparc machine is an Ultra IIe. The x86 machine is an AMD Opteron(tm) Processor 146.
Any insight would be appreciated.
Thanks.

Answer 1

your problem is aligment, the first heap/stack pages will be assigned before the mpss environment variables are read, it will not be aligned suitably for big pages so it will be small pages. If you then do a 8mb malloc you will see that the heap will grep by > 8m, it will grow in small pages to a 4mb alignment and then grow in 4mb pages, the same with the stack.
#include <sys/types.h>
#include <stdlib.h>
#include <unistd.h>
int
main ( int argc,char *ragv[]) {
char * fpp;
int i;
fpp = malloc(8192000);
for ( i = 0; i < 8192000;i = i+4096) {
fpp[i] = 23;
printf("fpp @ 0x%p to 0x%lx\n",
fpp, (uint32_t)fpp+8192000);
pause();
timu-home ksh: test1 &
fpp @ 0x8060a08 to 0x8830a08
[2] 3610
timu-home ksh: pmap -xs 3610 | grep -i heap
08061000 3708 3708 3708 - 4K rwx-- [ heap ]
08400000 8192 8192 8192 - 4M rwx-- [ heap ]
timu-home ksh:
I suspect the same is true of the stack, if you keep recursing to a 4m aligned
boundary it wil use 4m pages, else you can force it by moving your main to a
thread and pass you own 4m aligned buffer to thr_create() to use as a stack.
tim