10.2.0.4 Agent startup fails "/libjvm.so, because No such file or directory

Question

Hi Team,
10.2.0.4 Agent startup fails with below errors.Customer is having 30 agents in failover on the same system .Issue is happening for his newly installed agents and existing agents also.
When he stop few agents(like 26th agent and 27th agent... ) and he able to start the agent which is failed to start with below error.Issue looks like lack of some resources to start the agent.
oracle@sorapx2:/u00/app/oracle/ [AGENTFSP] emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
sh: 28400 Killed
/usr/lib/hpux32/dld.so: Cannot map text for library</pkg/FSP/u00/app/oracle/product/agent10g/jdk/jre/lib/IA64N/hotspot/libjvm.so>: mmap(0x0, 0xd51870, 0x5, 0x41, 3, 0x0) returns Not enough space.
oracle@sorapx2:/pkg/FSP/u00/app/oracle/product/agent10g/ [AGENTFSP] emctl start agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
sh: 20806 Killed
Agent is already running
oracle@sorapx2:/pkg/FSP/u00/app/oracle/product/agent10g/ [AGENTFSP] emctl upload agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
sh: 22988 Killed
/usr/lib/hpux32/dld.so: Cannot map text for library</pkg/FSP/u00/app/oracle/product/agent10g/jdk/jre/lib/IA64N/hotspot/libjvm.so>: mmap(0x0, 0xd51870, 0x5, 0x41, 3, 0x0) returns Not enough space.
+ No issues with space:
oracle@sorapx2:/pkg/FSP/u00/app/oracle/product/agent10g/ [AGENTFSP] bdf .
Filesystem kbytes used avail %used Mounted on
/dev/vgfsp1/lvol1 5144576 1973904 3146008 39% /pkg/FSP/u00
Agent also gives below error:
oracle@sorapx2:/u00/app/oracle/emdiag/log/ [AGENTGCP] emctl status agent
Error: failed to load /pkg/GCP/u00/app/oracle/product/agent10g/jdk/jre/lib/IA64N/server/libjvm.so, because No such file or directory
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
sh: 25923 Killed
/usr/lib/hpux32/dld.so: Cannot map text for library</pkg/GCP/u00/app/oracle/product/agent10g/lib32/libclntsh.so.10.1>: mmap(0x0, 0x21cded0, 0x5, 0x41, 3, 0x0) returns Not enough space.
Here emctl agent start command is trying to to load a 32 bit library in stead of the 64 bit.which is causing the issue:
There are a 32 Bit Library and a 64Bit Library available for the agent
oracle@vorakx2:/pkg/HYK/u00/app/oracle/product/agent10g/lib/ [HYK]
file libagtsh.so.1.0
libagtsh.so.1.0: ELF-64 shared object file - IA64
oracle@vorakx2:/pkg/HYK/u00/app/oracle/product/agent10g/lib32/ [HYK]
file libagtsh.so.1.0
libagtsh.so.1.0: ELF-32 shared object file - IA64
set below things properly still same problem.:
export ORACLE_HOME=<path of the agent10g installation>
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
Please suggest me further
+what emctl status agent is loading 32 bit library
+how to resolve the "Not enough space."
Environment Details:
10.2.0.4 OMS
10.2.0.4 Agent
Operating System HP-UX B.11.31 U (64)
SR 3-4636880901
Thanks
Narendra

Answer 1

Hi Narendra,
I just checked the link and it was ok. So maybe try again?
Copy/past Alain's post:
Hi,
-Buf cache is too high: SAM->kernel-> change dbc_pct_max from 50% to 10%.
-Whith GPM and command "extract" you can find which process consumme too much memory.
-Install lastest Quality Pack.
-With 32 bits processes, you can have problem with memory quadrant restriction. <= you have 64 bit so this is not of use to you
-Check kernel parameter related to max memory used by each process: maxdsiz
-Verify with swapinfo -tam and GPM -> memory.
See doc http://docs.hp.com/en/1218/mem_mgt.html
http://docs.hp.com/en/5580/Misconfigured_Resources.pdf
Alain.
Eric