Autoconfig fail EBS R12 Apps-Node with EBS 11gR2 RAC

Question

Platform: HPUX IA 64-11.31
DB: 11.2.0.3
Nodes: 2
We were following metalink note 823587.1 and have successfully converted single-instance database of EBS R12 to 20node RAC. When we are trying to do steps of Section-3.8 of the note the autoconfig is running in error. See the below error message from the "adconfig.log" file:
cat /etc/hosts
=========
127.0.0.1 localhost.localdomain localhost
#Public IP
172.16.101.23 ts1db1.bukhatir.ae ts1db1
172.16.101.24 ts1db2.bukhatir.ae ts1db2
#Vip
172.16.101.44 ts1_vip1.bukhatir.ae ts1_vip1
172.16.101.45 ts1_vip2.bukhatir.ae ts1_vip2
#inerconnect
10.0.0.1 ts1_prv1.bukhatir.ae ts1_prv1
10.0.0.2 ts1_prv2.bukhatir.ae ts1_prv2
172.16.101.20 ts1apps1.bukhatir.ae ts1apps1
172.16.101.21 ts1apps2.bukhatir.ae ts1apps2
=========
Generate Tns Names
Logfile: /locapps1/apps/apps/TEST_ts1apps1/admin/log/04101202/NetServiceHandler.log
Classpath : /ts1apps/apps/apps_st/comn/java/lib/appsborg2.zip:/ts1apps/apps/apps_st/comn/java/classes
Updating s_tnsmode to 'generateTNS'
UpdateContext exited with status: 0
AC-50480: Internal error occurred: java.lang.Exception: Error while generating listener.ora.
Error generating tnsnames.ora from the database, temporary tnsnames.ora will be generated using templates
Instantiating Tools tnsnames.ora
Tools tnsnames.ora instantiated
Web tnsnames.ora instantiated
adgentns.pl exiting with status 2
ERRORCODE = 2 ERRORCODE_END
xecuting script in InstantiateFile:
/locapps1/apps/apps/TEST_ts1apps1/admin/install/adgendbc.sh
script returned:
adgendbc.sh started at Tue Apr 10 12:03:56 UAE 2012
SQL*Plus: Release 10.1.0.5.0 - Production on Tue Apr 10 12:03:57 2012
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Enter value for 1: Enter value for 2: Enter value for 3: Connected.
[ APPS_DATABASE_ID ]
Application Id : 0
Profile Value : TEST
Level Name: SITE
INFO : Updated/created profile option value.
PL/SQL procedure successfully completed.
Commit complete.
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
==============================
* * * * DBC PARAMETERS * * * *
==============================
fnd_jdbc_buffer_min=1
fnd_jdbc_buffer_max=5
fnd_jdbc_buffer_decay_interval=300
fnd_jdbc_buffer_decay_size=5
fnd_jdbc_usable_check=false
fnd_jdbc_context_check=true
fnd_jdbc_plsql_reset=false
====================================
* * * * NO CUSTOM PARAMETERS * * * *
====================================
Unique constraint error (00001) is OK if key already exists
Creating the DBC file...
java.sql.SQLException: The Network Adapter could not establish the connection
Database connection to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=ts1_vip1.bukhatir.ae)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=ts1_vip2.bukhatir.ae)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=TEST))) failed
ADD call failed with exit code 1
Updating Server Security Authentication
java.sql.SQLException: Invalid number format for port number
Database connection to jdbc:oracle:thin:@host_name:port_number:database failed
Updating Server Security Authentication failed with exit code 1
Restoring DBC file from backed up location /locapps1/apps/apps/TEST_ts1apps1/appltmp/TXK/TEST_Tue_Apr_10_12_03_2012.dbc
adgendbc.sh ended at Tue Apr 10 12:04:01 UAE 2012
adgendbc.sh exiting with status 1
ERRORCODE = 1 ERRORCODE_END
See the network configuration files from the environment:
===========
Dbhome
===========
/orahome/oradb/app/product/11.2.0.3/network/admin/TEST1_ts1db1/listener.ora
LISTENER_TEST =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = ts1_vip1.bukhatir.ae)(PORT = 1521)(IP = FIRST)))
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = ts1db1)(PORT = 1521)(IP = FIRST)))
SID_LIST_LISTENER_TEST =
(SID_LIST =
(SID_DESC = (ORACLE_HOME = /orahome/oradb/app/product/11.2.0.3)(SID_NAME = TEST1))
STARTUP_WAIT_TIME_LISTENER_TEST = 0
CONNECT_TIMEOUT_LISTENER_TEST = 10
TRACE_LEVEL_LISTENER_TEST = OFF
LOG_DIRECTORY_LISTENER_TEST = /orahome/oradb/app/product/11.2.0.3/network/admin
LOG_FILE_LISTENER_TEST = TEST1
TRACE_DIRECTORY_LISTENER_TEST = /orahome/oradb/app/product/11.2.0.3/network/admin
TRACE_FILE_LISTENER_TEST = TEST1
ADMIN_RESTRICTIONS_LISTENER_TEST = ON
SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER_TEST = OFF
IFILE=/orahome/oradb/app/product/11.2.0.3/network/admin/TEST1_ts1db1/listener_ifile.ora
/orahome/oradb/app/product/11.2.0.3/network/admin/TEST1_ts1db1/tnsnames.ora
TEST=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1_vip1.bukhatir.ae)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=TEST)
(INSTANCE_NAME=TEST1)
TEST1=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1_vip1.bukhatir.ae)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=TEST)
(INSTANCE_NAME=TEST1)
TEST1_FO=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1_vip1.bukhatir.ae)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=TEST)
(INSTANCE_NAME=TEST1)
TEST_FO=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1_vip1.bukhatir.ae)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=TEST)
(INSTANCE_NAME=TEST1)
TEST1_LOCAL=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1_vip1.bukhatir.ae)(PORT=1521))
TEST_BALANCE=
(DESCRIPTION=
(ADDRESS_LIST=
(LOAD_BALANCE=YES)
(FAILOVER=YES)
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1_vip1.bukhatir.ae)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=TEST)
TEST_REMOTE=
(DESCRIPTION=
(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1_vip1.bukhatir.ae)(PORT=1521))
#TEST_REMOTE=
# (DESCRIPTION=
# (ADDRESS_LIST=
# (ADDRESS=(PROTOCOL=tcp)(HOST=tsscan.bukhatir.ae)(PORT=1521))
TEST1_local=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1_vip1.bukhatir.ae)(PORT=1521))
extproc_connection_data =
(DESCRIPTION=
(ADDRESS_LIST =
(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROCTEST1))
(CONNECT_DATA=
(SID=PLSExtProc)
(PRESENTATION = RO)
IFILE=/orahome/oradb/app/product/11.2.0.3/network/admin/TEST1_ts1db1/TEST1_ts1db1_ifile.ora
===========
Gridhome
===========
/gridhome/oragrid/11.2.0/grid/network/admin/listener.ora
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) # line added by Agent
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent
/gridhome/oragrid/11.2.0/grid/network/admin/endpoints_listener.ora
LISTENER_TS1DB1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ts1_vip1)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=172.16.101.23)(PORT=1521)(IP=FIRST)))) # line
added by Agent
===========
Listener EBShome
===========
cd $TNS_ADMIN
/locapps1/apps/apps/TEST_ts1apps1/ora/10.1.2/network/admin/listener.ora
APPS_TEST =
(ADDRESS_LIST =
(ADDRESS= (PROTOCOL= TCP)(Host= ts1apps1)(Port= 1629))
SID_LIST_APPS_TEST =
(SID_LIST =
( SID_DESC = ( SID_NAME = FNDSM )
( ORACLE_HOME = /ts1apps/apps/tech_st/10.1.2 )
( PROGRAM = /ts1apps/apps/apps_st/appl/fnd/12.0.0/bin/FNDSM )
( envs='MYAPPSORA=/ts1apps/apps/apps_st/appl/APPSTEST_ts1apps1.env,PATH=/usr/bin:/usr/ccs/bin:/bin,FNDSM_SCRIPT=/locapps1/apps/apps/TEST_ts1apps1/admin/scripts/gsmst
art.sh' )
( SID_DESC = ( SID_NAME = FNDFS )
( ORACLE_HOME = /ts1apps/apps/tech_st/10.1.2 )
( PROGRAM = /ts1apps/apps/apps_st/appl/fnd/12.0.0/bin/FNDFS )
( envs='EPC_DISABLED=TRUE,NLS_LANG=American_America.AL32UTF8,LD_LIBRARY_PATH=/ts1apps/apps/tech_st/10.1.2/lib32:/ts1apps/apps/tech_st/10.1.2/lib:/ts1apps/apps/tech_s
t/10.1.2/jdk/jre/lib/IA64N:/ts1apps/apps/tech_st/10.1.2/jdk/jre/lib/IA64N/server:/ts1apps/apps/apps_st/appl/sht/12.0.0/lib,SHLIB_PATH=/ts1apps/apps/tech_st/10.1.2/lib32:/ts1apps/apps
/tech_st/10.1.2/lib:/ts1apps/apps/tech_st/10.1.2/jdk/jre/lib/IA64N:/ts1apps/apps/tech_st/10.1.2/jdk/jre/lib/IA64N/server:/ts1apps/apps/apps_st/appl/sht/12.0.0/lib,LIBPATH=/ts1apps/ap
ps/tech_st/10.1.2/lib32:/ts1apps/apps/tech_st/10.1.2/lib:/ts1apps/apps/tech_st/10.1.2/jdk/jre/lib/IA64N:/ts1apps/apps/tech_st/10.1.2/jdk/jre/lib/IA64N/server:/ts1apps/apps/apps_st/ap
pl/sht/12.0.0/lib,APPLFSTT=TEST_BALANCE;TEST;TEST_FO,APPLFSWD=/locapps1/apps/apps/TEST_ts1apps1/appl/admin;/locapps1/apps/apps/TEST_ts1apps1/appltmp;/ts1apps/apps/apps_st/comn/webapp
s/oacore/html/oam/nonUix/launchMode/restricted' )
STARTUP_WAIT_TIME_APPS_TEST = 0
CONNECT_TIMEOUT_APPS_TEST = 10
TRACE_LEVEL_APPS_TEST = OFF
LOG_DIRECTORY_APPS_TEST = /locapps1/apps/apps/TEST_ts1apps1/logs/ora/10.1.2/network
LOG_FILE_APPS_TEST = APPS_TEST
TRACE_DIRECTORY_APPS_TEST = /locapps1/apps/apps/TEST_ts1apps1/logs/ora/10.1.2/network
TRACE_FILE_APPS_TEST = APPS_TEST
ADMIN_RESTRICTIONS_APPS_TEST = ON
IFILE = /locapps1/apps/apps/TEST_ts1apps1/ora/10.1.2/network/admin/TEST_ts1apps1_listener_ifile.ora
SUBSCRIBE_FOR_NODE_DOWN_EVENT_APPS_TEST = OFF
/locapps1/apps/apps/TEST_ts1apps1/ora/10.1.2/network/admin/tnsnames.ora
TEST = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1db1)(PORT=1521))
(CONNECT_DATA=(SID=TEST1))
# Net8 definitions for FNDFS and FNDSM on the HTTP server node - ts1apps1
FNDFS_ts1apps1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
FNDFS_ts1apps1.bukhatir.ae = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
# For when the profile FS_SVC_PREFIX is set these entries will be used
FNDFS_TEST1_ts1apps1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
FNDFS_TEST1_ts1apps1.bukhatir.ae = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
FNDSM_ts1apps1_TEST1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDSM))
FNDSM_ts1apps1.bukhatir.ae_TEST1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDSM))
# Net8 definitions for FNDFS and FNDSM on the forms server node - ts1apps1
FNDFS_ts1apps1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
FNDFS_ts1apps1.bukhatir.ae = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
# For when the profile FS_SVC_PREFIX is set these entries will be used
FNDFS_TEST1_ts1apps1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
FNDFS_TEST1_ts1apps1.bukhatir.ae = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
FNDSM_ts1apps1_TEST1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDSM))
FNDSM_ts1apps1.bukhatir.ae_TEST1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDSM))
# Net8 definitions for FNDFS and FNDSM on the administration server node - ts1apps1
FNDFS_ts1apps1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
FNDFS_ts1apps1.bukhatir.ae = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
# For when the profile FS_SVC_PREFIX is set these entries will be used
FNDFS_TEST1_ts1apps1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
FNDFS_TEST1_ts1apps1.bukhatir.ae = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
FNDSM_ts1apps1_TEST1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDSM))
FNDSM_ts1apps1.bukhatir.ae_TEST1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDSM))
# Net8 definitions for FNDFS and FNDSM on the concurrent processing server node - ts1apps1
FNDFS_ts1apps1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
FNDFS_ts1apps1.bukhatir.ae = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
# For when the profile FS_SVC_PREFIX is set these entries will be used
FNDFS_TEST1_ts1apps1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
FNDFS_TEST1_ts1apps1.bukhatir.ae = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDFS))
FNDSM_ts1apps1_TEST1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDSM))
FNDSM_ts1apps1.bukhatir.ae_TEST1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ts1apps1)(PORT=1629))
(CONNECT_DATA=(SID=FNDSM))
IFILE=/locapps1/apps/apps/TEST_ts1apps1/ora/10.1.2/network/admin/TEST_ts1apps1_ifile.ora

Answer 1

Yes, following message is also reported in the adconfig.log:
Unique constraint error (00001) is OK if key already exists
Creating the DBC file...
java.sql.SQLException: The Network Adapter could not establish the connection
Database connection to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=ts1_vip2.bukhatir.ae)(PORT=1523))(ADDRESS=(PROTOCOL=tcp)(HOST=
ts1_vip1.bukhatir.ae)(PORT=1523)))(CONNECT_DATA=(SERVICE_NAME=TEST))) failed
ADD call failed with exit code 1
Updating Server Security Authentication
java.sql.SQLException: Invalid number format for port number
Database connection to jdbc:oracle:thin:@host_name:port_number:database failed
Updating Server Security Authentication failed with exit code 1
Restoring DBC file from backed up location /locapps1/apps/apps/TEST_ts1apps1/appltmp/TXK/TEST_Mon_Apr_16_13_13_2012.dbc
adgendbc.sh ended at Mon Apr 16 13:13:46 UAE 2012
adgendbc.sh exiting with status 1
ERRORCODE = 1 ERRORCODE_END
.end std out.
.end err out.
Did you see Autoconfig has failed on Apps tier on adgendbc.cmd with error: ADD call failed with exit code 1, UPDATE call failed with exit code 1 [ID 359739.1]?But we have the file version:
$Header: AdminAppServer.java 120.11.12010000.6 2010/04/13 15:24:03 fskinner ship $