Node eviction while registring a new DSN

Question

Hi,
I encounter a node eviction when trying to register a DSN in cache grid configuration. I tried about 6 times with the same result. Could you please take a look?
My software run on Solaris 10.
My system consists of 2 DSN: tt_0.0.0.1 and tt_0.1.0.1
I have 2 servers with TT and CRS installed, node 06 hosts active tt_0.0.0.1 and standby tt_0.1.0.1 / node 07 hosts standby tt_0.0.0.1 and standby tt_0.1.0.1
Oracle Server is hosted on 07
I managed to register tt_0.0.0.1:
server06$ ttisql tt_0.0.0.1
Command> call ttcachestart ;
Command> call ttcacheuidpwdset('pin', 'pin'); # pin is a datastore user (uid)
Command> call ttGridCreate('ttgridPin');
Command> call ttGridNameSet('ttgridPin');
Command> call ttGridAttach(1, '0.0.0.1_active', 'dfso4e06', 53381, '0.0.0.1_stdby', 'dfso4e07', 53382);
Command> call ttrepstart ;
Command> call ttgridnodestatus;
< TTGRIDPIN, 1, 1, T, DFSO4E06, TTGRIDPIN_TT_0.0.0.11_1A, 10.197.65.67, 53381, T, DFSO4E07, TTGRIDPIN_TT_0.0.0.12_1B, 10.197.65.185, 53382 >
Command> exit
server07$ ttIsql tt_0.1.0.1
Command> call ttcachestart ;
Command> call ttcacheuidpwdset('pin', 'pin');
Command> call ttGridNameSet('ttgridPin');
Command> call ttGridAttach(1, '0.1.0.1_active', 'dfso4e07', 53383, '0.1.0.1_stdby', 'dfso4e06', 53384);
Command> call ttrepstart ;
Everything works fine, below is my cachegrid configuration:
ttadmin@dfso4e06:/exec/products/CRS/v11.1.0.7/log/dfso4e06$ ttIsql tt_0.0.0.1
Command> call ttGridNameSet('ttgridPin');
Command> call ttgridnodestatus;
< TTGRIDPIN, 1, 1, T, DFSO4E06, TTGRIDPIN_TT_0.0.0.11_1A, 10.197.65.67, 53381, T, DFSO4E07, TTGRIDPIN_TT_0.0.0.12_1B, 10.197.65.185, 53382 >
< TTGRIDPIN, 2, 1, T, dfso4e07, TTGRIDPIN_0.1.0.1_active_2A, 10.197.65.185, 53383, F, dfso4e06, TTGRIDPIN_0.1.0.1_stdby_2B, 10.197.65.67, 53384 >
Note: the F status (detached) of the standby server which should go to T (attached) as soon as i successfully run "ttCWAdmin -create"
My CRS / TT integration looks like this:
ttadmin@dfso4e06:/exec/products/CRS/v11.1.0.7/log/dfso4e06$ ttCWAdmin -status
Master Datastore 1:
Host:dfso4e06
Status:AVAILABLE
State:ACTIVE
Master Datastore 2:
Host:dfso4e07
Status:AVAILABLE
State:STANDBY
I used ttCWAdmin framework to register that DSN. Now I want to do the same with the other one: tt_0.1.0.1.
I run:
ttadmin@dfso4e07:/exec/applis/BRM/timesten/TimesTen/tt1121_HA/info$ ttCWAdmin -create -dsn tt_0.1.0.1
Successful connection with Oracle clusterware stack
replication DDL = create active standby pair "TT_0.1.0.1" on "dfso4e07","TT_0.1.0.1" on "dfso4e06" RETURN TWOSAFE
Enter internal UID with ADMIN privileges: pin
Enter password for the same UID:
IMDB Cache is Enabled
Enter Oracle password for the same UID:
Enter any phrase for password encryption:
Data store exists and uid/pwd are verified on host dfso4e07
Create active standby pair replication scheme on host dfso4e07 and make it the active master? (Y/N)
As soon as i will hit 'Y' + Enter the countdown to eviction will start from the other node (the 06), see:
ttadmin@dfso4e06:/exec/products/CRS/v11.1.0.7/log/dfso4e06$ tail -f alertdfso4e06.log
2011-03-30 02:42:37.166
[cssd(3323)]CRS-1612:node dfso4e07 (2) at 50% heartbeat fatal, eviction in 14.783 seconds
2011-03-30 02:42:38.175
[cssd(3323)]CRS-1612:node dfso4e07 (2) at 50% heartbeat fatal, eviction in 13.773 seconds
2011-03-30 02:42:45.244
[cssd(3323)]CRS-1611:node dfso4e07 (2) at 75% heartbeat fatal, eviction in 6.703 seconds
2011-03-30 02:42:49.283
[cssd(3323)]CRS-1610:node dfso4e07 (2) at 90% heartbeat fatal, eviction in 2.663 seconds
2011-03-30 02:42:50.292
[cssd(3323)]CRS-1610:node dfso4e07 (2) at 90% heartbeat fatal, eviction in 1.653 seconds
2011-03-30 02:42:51.302
[cssd(3323)]CRS-1610:node dfso4e07 (2) at 90% heartbeat fatal, eviction in 0.643 seconds
2011-03-30 02:42:51.954
[cssd(3323)]CRS-1607:CSSD evicting node dfso4e07. Details in /exec/products/CRS/v11.1.0.7/log/dfso4e06/cssd/ocssd.log.
the last log won't say much.
Below, my sys.odbc.ini:
[ODBC Data Sources]
tt_0.0.0.1=TimesTen 11.2.1 Driver
tt_0.1.0.1=TimesTen 11.2.1 Driver
# New data source definitions can be added below.
[tt_0.0.0.1]
DataStore=/exec/applis/BRM/brm7.4/tt/db_files/tt_0.0.0.1
UID=pin
PWD=pin
OracleNetServiceName=PINDB
DatabaseCharacterSet=UTF8
ConnectionCharacterSet=UTF8
PLSQL=1
oraclepwd=pin
OracleId=PINDB
Driver=$TIMESTEN_HOME/lib/libtten.so
#Shared-memory size in megabytes allocated for the datastore.
PermSize= 1024
#Shared-memory size in megabytes allocated for temporary data partition, generally half the size of PermSize.
#TempSize=32
PassThrough=0
#Use large log buffer, log file sizes
LogFileSize=512
LogBufMB=512
#Async repl flushes to disk before sending batches so this makes it faster on Linux
LogFlushMethod=2
#Limit Ckpt rate to 10 mb/s
CkptFrequency=200
CkptLogVolume=0
CkptRate=10
Connections=200
#Oracle recommends setting LockWait to 30 seconds.
LockWait=30
DurableCommits=0
CacheGridEnable=1
[tt_0.1.0.1]
DataStore=/exec/applis/BRM/brm7.4/tt/db_files/tt_0.1.0.1
UID=pin
PWD=pin
OracleNetServiceName=PINDB
DatabaseCharacterSet=UTF8
ConnectionCharacterSet=UTF8
PLSQL=1
oraclepwd=pin
OracleId=PINDB
Driver=$TIMESTEN_HOME/lib/libtten.so
#Shared-memory size in megabytes allocated for the datastore.
PermSize= 1024
#Shared-memory size in megabytes allocated for temporary data partition, generally half the size of PermSize.
## JB ##TempSize=32
PassThrough=0
#Use large log buffer, log file sizes
LogFileSize=512
LogBufMB=512
#Async repl flushes to disk before sending batches so this makes it faster on Linux
LogFlushMethod=2
#Limit Ckpt rate to 10 mb/s
CkptFrequency=200
CkptLogVolume=0
CkptRate=10
Connections=200
#Oracle recommends setting LockWait to 30 seconds.
LockWait=30
DurableCommits=0
CacheGridEnable=1
Note: pin is the cache admin i created - i also created and populated cache groups within the tt_0.0.0.1
below my cluster.oracle.ini
[repdb1_1121]
MasterHosts = dfso4e06,dfso4e07
ScriptInstallDir = /exec/applis/BRM/timesten/TimesTen/tt1121_HA/info/crs_scripts
[cachedb1_1121]
MasterHosts = dfso4e06,dfso4e07
ScriptInstallDir = /exec/applis/BRM/timesten/TimesTen/tt1121_HA/info/crs_scripts
CacheConnect = Y
[tt_0.0.0.1]
MasterHosts = dfso4e06,dfso4e07
ScriptInstallDir = /exec/applis/BRM/timesten/TimesTen/tt1121_HA/info/crs_scripts
CacheConnect = Y
ReturnServiceAttribute = RETURN TWOSAFE
GridPort = 53381,53382
[tt_0.1.0.1]
MasterHosts = dfso4e07,dfso4e06
ScriptInstallDir = /exec/applis/BRM/timesten/TimesTen/tt1121_HA/info/crs_scripts
CacheConnect = Y
ReturnServiceAttribute = RETURN TWOSAFE
GridPort = 53383,53384

Answer 1

Since I changed some Timeout parameters, I have a new error coming up:
ttadmin@dfso4e07:/users/applix/vpb/ttadmin$ ttCWAdmin -create -dsn tt_0.1.0.1
Successful connection with Oracle clusterware stack
replication DDL = create active standby pair "TT_0.1.0.1" on "dfso4e07","TT_0.1.0.1" on "dfso4e06" RETURN TWOSAFE
Enter internal UID with ADMIN privileges: pin
Enter password for the same UID:
IMDB Cache is Enabled
Enter Oracle password for the same UID:
Enter any phrase for password encryption:
Data store exists and uid/pwd are verified on host dfso4e07
Create active standby pair replication scheme on host dfso4e07 and make it the active master? (Y/N)Y
Waiting for confirmation from hosts...
Active Standby pair replication scheme created on host dfso4e07, which will be the active master at start
====================================================================
Warning: If data store(s) already exist(s) on host(s) dfso4e06, they may be destroyed when the active master is duplicated.
If the data store(s) need(s) to be preserved, please back up the data store(s) manually, before executing
ttCWAdmin -start Command
====================================================================
Registering TimesTen Cluster with Oracle Clusterware
====================================================================
Number of unregistered dsn resources = 2
Registering dsn resources:... Registration complete.
====================================================================
====================================================================
Number of unregistered service resources = 2
Registering service resources:... (ttCWAdmin:) crsctl.c(14915): TT48004: clscrs_register_resource failed with status = 200.
Warning: some resource could not be registered
====================================================================
(ttCWAdmin:) crsctl.c(1789): TT48014: Failed to register the cluster for DSN TT_0.1.0.1 with Oracle Clusterware.
I perform the required check:
oracle@dfso4e07:/users/applix/vpb/oracle
crsctl check crsCluster Synchronization Services appears healthy
Cluster Ready Services appears healthy
Event Manager appears healthy
I found this interesting though in ttcwerrors.log:
2011-03-31 01:35:37.26 Err : : 10032: (ttCRSmaster:TT_0.0.0.1) ttctl.c(366): 08001:[TimesTen][TimesTen 11.2.1.6.6 ODBC Driver][TimesTen]TT0830: Cannot create data store
file. OS-detected error: Permission denied -- file "db.c", lineno 11505, procedure "sbDbConnect"
2011-03-31 01:35:37.26 Err : : 10032: (ttCRSmaster:TT_0.0.0.1) ttCRSmaster.c(1978): Native error 830 for the dsn TT_0.0.0.1
2011-03-31 01:35:39.74 Err : : 10032: (ttCRSmaster:TT_0.0.0.1) ttctl.c(366): 08001:[TimesTen][TimesTen 11.2.1.6.6 ODBC Driver][TimesTen]TT0830: Cannot create data store
file. OS-detected error: Permission denied -- file "db.c", lineno 11505, procedure "sbDbConnect"
2011-03-31 01:35:42.25 Err : : 10032: (ttCRSmaster:TT_0.0.0.1) ttctl.c(366): 08001:[TimesTen][TimesTen 11.2.1.6.6 ODBC Driver][TimesTen]TT0830: Cannot create data store
file. OS-detected error: Permission denied -- file "db.c", lineno 11505, procedure "sbDbConnect"
I tried to run truss, but couldnt find out where it was denied...
Any help?