HA Oracle and ASM
Hi, I wonder if anyone has experiences in running the above configuration in SUN Cluster. As far as I know, ASM is not yet supported by the SC Oracle HA Agent and for this reason ASM is just certified for RAC on SC. Anyway, there is no good reason why ASM should not be used for HA solutions as well (instead of RAW devices or a filesystem based configuration), even though the ASM instance must be started by a GDS ressource before the database comes up. Are there any plans to integrate ASM into the SC Oracle HA Agent ?
Thanks, Marc
Tim,
thanks for this and sorry for my late response. Anyway, I disagree with most of the issues:
1) Yes, the hostname is stored in the OCR, but without CRS, the OCR is not shared, but per default locally stored beneath $ORACLE_HOME, for instance: /orahome/oracle/product/10.2.0.2/db/cdata/localhost/local.ocr
So there is no problem with OCR as long as $ORACLE_HOME is locally stored on each cluster node. Furthermore there is a library provided by SUN for older OAS environments to keep the hostname consistent to OAS in SUN Cluster regardless to the real hostname of the cluster node (libloghost_64.so). This could additionally solve the problem in case of a shared $ORACLE_HOME, but in fact there is no need for a shared $ORACLE_HOME and local installations avoid this problem at all.
2) I fully agree, you will have a local ASM instance on each node using it's own OCR (see above).
3) I also agree, the only point to take care about is to integrate startup, shutdown and monitoring of the ASM instance(s) needded to serve the database into SC. This can be done easily by using GDS.
4) Yes the greate advantage of ASM is that DG configuration information is stored as part as header information on the disks. But there is no need zero out the configuration at all to import the DGs to another server. Just shutdown the original ASM instance first (as without RAC the ASM instances shoud be prevented to import the DGs concurrently), switch over the DCS and start the other ASM instance on the failover node. By starting the instance ASM imports all DGs stored on the disks which match the asm_diskstring, in my case asm_diskstring='/dev/did/rdsk/*', because there is just one database.
Moreover this way of failing over ASM is supported by Oracle (for instance for an OAS infrastructure database, see http://download-east.oracle.com/docs/cd/B25006_01/install.1012/install/ha_cfc.htm#BABEADDA). As mentioned in my previous posting I tested this constellation running a local ASM instance (+ASM) on each cluster node and one failover database:
Resource Name Node Name State Status Message
Resource: HW1P-lh-res fix Online Online - LogicalHostname online.
Resource: HW1P-lh-res foxi Offline Offline - LogicalHostname offline.
Resource: HW1P-dg-res fix Online Online
Resource: HW1P-dg-res foxi Offline Offline
Resource: HW1P-asm-res fix Online Online - Service is online.
Resource: HW1P-asm-res foxi Offline Offline
Resource: HW1P-db-res fix Online Online
Resource: HW1P-db-res foxi Offline OfflineI have to admit, that things will get more difficult in case of several failover databases in the same cluster. But there is generally no problem to have several ASM instance on each node, each of them serving another database. So after all I can't see any technical reason for not supporting ASM for failover databases in SUN Cluster. In fact this could be also a solution to get rid of problems in environments using stoarge-based data replication. As DG information is stored on the disks and each disk matching the asm_diskstring will be scanned, ASM-DGs can be imported, even if the DIDs are differrent due to a storage failover. Thus there is no need to combine/fake DIDs like in SVM to use storage-based replication.
Similar Messages
-
Solaris 10 and Hitachi LUN mapping with Oracle 10g RAC and ASM?
Hi all,
I am working on an Oracle 10g RAC and ASM installation with Sun E6900 servers attached to a Hitachi SAN for shared storage with Sun Solaris 10 as the server OS. We are using Oracle 10g Release 2 (10.2.0.3) RAC clusterware
for the clustering software and raw devices for shared storage and Veritas VxFs 4.1 filesystem.
My question is this:
How do I map the raw devices and LUNs on the Hitachi SAN to Solaris 10 OS and Oracle 10g RAC ASM?
I am aware that with an Oracle 10g RAC and ASM instance, one needs to configure the ASM instance initialization parameter file to set the asm_diskstring setting to recognize the LUNs that are presented to the host.
I know that Sun Solaris 10 uses /dev/rdsk/CwTxDySz naming convention at the OS level for disks. However, how would I map this to Oracle 10g ASM settings?
I cannot find this critical piece of information ANYWHERE!!!!
Thanks for your help!You don't seem to state categorically that you are using Solaris Cluster, so I'll assume it since this is mainly a forum about Solaris Cluster (and IMHO, Solaris Cluster with Clusterware is better than Clusterware on its own).
Clusterware has to see the same device names from all cluster nodes. This is why Solaris Cluster (SC) is a positive benefit over Clusterware because SC provides an automatically managed, consistent name space. Clusterware on its own forces you to manage either the symbolic links (or worse mknods) to create a consistent namespace!
So, given the SC consistent namespace you simple add the raw devices into the ASM configuration, i.e. /dev/did/rdsk/dXsY. If you are using Solaris Volume Manager, you would use /dev/md/<setname>/rdsk/dXXX and if you were using CVM/VxVM you would use /dev/vx/rdsk/<dg_name>/<dev_name>.
Of course, if you genuinely are using Clusterware on its own, then you have somewhat of a management issue! ... time to think about installing SC?
Tim
--- -
Hi,
I have noticed following 'strange' behaviour of Oracle Restart and ASM.
starting position:
-bash-3.2 $ crsctl status resource -t
NAME TARGET STATE SERVER STATE_DETAILS
Local Resources
ora.DATA.dg
ONLINE ONLINE oracle-restart
ora.LISTENERASM.lsnr
ONLINE ONLINE oracle-restart
ora.asm
ONLINE ONLINE oracle-restart Started
Cluster Resources
ora.cssd
1 ONLINE ONLINE oracle-restart
ora.diskmon
1 ONLINE ONLINE oracle-restartstep 1:
-bash-3.2 $ srvctl stop asm
-bash-3.2 $ srvctl stop diskgroup -g data
-bash-3.2 $ srvctl disable diskgroup -g datastep 2:
via sqlplus start ASM instance
SQL> startup
ASM instance started
Total System Global Area 283930624 bytes
Fixed Size 2212656 bytes
Variable Size 256552144 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled
SQL> select * from v$asm_diskgroup;
GROUP_NUMBER NAME SECTOR_SIZE BLOCK_SIZE
ALLOCATION_UNIT_SIZE STATE TYPE TOTAL_MB FREE_MB HOT_USED_MB
COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
COMPATIBILITY
DATABASE_COMPATIBILITY V
1 DATA 512 4096
1048576 MOUNTED EXTERN 10236 10177 0
59 0 10177 0
GROUP_NUMBER NAME SECTOR_SIZE BLOCK_SIZE
ALLOCATION_UNIT_SIZE STATE TYPE TOTAL_MB FREE_MB HOT_USED_MB
COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
COMPATIBILITY
DATABASE_COMPATIBILITY V
11.2.0.0.0
10.1.0.0.0 N
-bash-3.2 $ crsctl status resource -t
NAME TARGET STATE SERVER STATE_DETAILS
Local Resources
ora.DATA.dg
OFFLINE OFFLINE oracle-restart <== funny !!!
ora.LISTENERASM.lsnr
ONLINE ONLINE oracle-restart
ora.asm
ONLINE ONLINE oracle-restart Started
Cluster Resources
ora.cssd
1 ONLINE ONLINE oracle-restart
ora.diskmon
1 ONLINE ONLINE oracle-restartIs this behaviour a 'feature' or bug?
Anyone had similar experience?
thanks,
goranHi,
asm resource is depending on diskgroup resource ... if diskgroup res. is not available, crsctl status shows offline, I would expect asm should be also shown as 'offline' (and brought offline) as they are dependent.
What is the point of managing resources via srvctl when it doesn't take care of dependencies? For me it's wrong.ora.asm : is ASM Instance
ora.*.dg : is Diskgroup
ora.*.dg is dependent of ora.asm, not to the contrary.
I can have more than one diskgroup and want only one diskgroup disabled, so I need the ASM Instance (ora.asm) online.
Important:
If you shut down the database with SQL*Plus, Oracle Restart does not interpret this as a database failure and does not attempt to restart the database.
Similarly, if you shut down the Oracle ASM instance with SQL*Plus or ASMCMD, Oracle Restart does not attempt to restart it.
An important difference between starting a component with SRVCTL and starting it with SQL*Plus (or another utility) is the following:
When you start a component with SRVCTL, any components on which this component depends are automatically started first, and in the proper order.
When you start a component with SQL*Plus (or another utility), other components in the dependency chain are not automatically started; you must ensure that any components on which this component depends are started.
Oracle Restart also manages the weak dependency between database instances and the Oracle Net listener (the listener): When a database instance is started, Oracle Restart attempts to start the listener. If the listener startup fails, then the database is still started. If the listener later fails, Oracle Restart does not shut down and restart any database instances.
It makes no sense Oracle Restart to shut down all environment (databases) because the listener down.
Regards,
Levi Pereira -
How do I map Hitachi SAN LUNs to Solaris 10 and Oracle 10g ASM?
Hi all,
I am working on an Oracle 10g RAC and ASM installation with Sun E6900 servers attached to a Hitachi SAN for shared storage with Sun Solaris 10 as the server OS. We are using Oracle 10g Release 2 (10.2.0.3) RAC clusterware
for the clustering software and raw devices for shared storage and Veritas VxFs 4.1 filesystem.
My question is this:
How do I map the raw devices and LUNs on the Hitachi SAN to Solaris 10 OS and Oracle 10g RAC ASM?
I am aware that with an Oracle 10g RAC and ASM instance, one needs to configure the ASM instance initialization parameter file to set the asm_diskstring setting to recognize the LUNs that are presented to the host.
I know that Sun Solaris 10 uses /dev/rdsk/CwTxDySz naming convention at the OS level for disks. However, how would I map this to Oracle 10g ASM settings?
I cannot find this critical piece of information ANYWHERE!!!!
Thanks for your help!Yes that is correct however due to use of Solaris 10 MPxIO multipathing software that we are using with the Hitachi SAN it does present an extra layer of complexity and issues with ASM configuration. Which means that ASM may get confused when it attempts to find the new LUNs from the Hitachi SAN at the Solaris OS level. Oracle Metalink note 396015.1 states this issue.
So my question is this: how to configure the ASM instance initialization parameter asm_diskstring to recognize the new Hitachi LUNs presented to the Solaris 10 host?
Lets say that I have the following new LUNs:
/dev/rdsk/c7t1d1s6
/dev/rdsk/c7t1d2s6
/dev/rdsk/c7t1d3s6
/dev/rdsk/c7t1d4s6
Would I set the ASM initialization parameter for asm_diskstring to /dev/rdsk/c7t1d*s6
as correct setting so that the ASM instance recognizes my new Hitachi LUNs? Solaris needs to map these LUNs using pseudo devices in the Solaris OS for ASM to recognize the new disks.
How would I set this up in Solaris 10 with Sun multipathing (MPxIO) and Oracle 10g RAC ASM?
I want to get this right to avoid the dreaded ORA-15072 errors when creating a diskgroup with external redundancy for the Oracle 10g RAC ASM installation process. -
Oracle 10g ASM and RAC configuration
Hi all,
I want to ask to everybody something about Oracle 10g RAC and ASm configuration. We plan to migrate to Oracle 10g from 9i, and we will begin configuring oracle but we have to decide which configuration are the best.
Our materials are bellow:
Hardware: RP 3440 (HP)
OS : HPUX 11i Ver 1
Storage: EVA 4000 (eva disk group)
The problem is:
Our supplier recommand us to use HP serviguard + HP serviceguard extension for RAC+ RAc and Raw device as configuration.
But we want to use Oracekl Clusterware + RAC + ASM
My question is if anybody know what is the best configuration, we want to use ASm.
Can HP serviguard use ASM.
Some documentations or link explain oracle RAC and ASM configuration will be appreciate.
Thanks for your help.
Regards.
raitsarevoHi,
I want to use RAC for clustering. My shared disk is a NetApp Filer FS250 that can only be mounted throught NFS. RAW devices have to be excluded, alsi OCFS i think works similar to raw devices and need to see phisical disk to create partition. ASM instead can work on an NFS mount?
Ste
Visit http://www.stefanocislaghi.it/ -
Oracle database 10g RAC and ASM configuration
Hi all,
I want to ask to everybody something about Oracle 10g RAC and ASm configuration. We plan to migrate to Oracle 10g from 9i, and we will begin configuring oracle but we have to decide which configuration are the best.
Our materials are bellow:
Hardware: RP 3440 (HP)
OS : HPUX 11i Ver 1
Storage: EVA 4000 (eva disk group)
The problem is:
Our supplier recommand us to use HP serviguard + HP serviceguard extension for RAC+ RAc and Raw device as configuration.
But we want to use Oracekl Clusterware + RAC + ASM
My question is if anybody know what is the best configuration, we want to use ASm.
Can we use HP serviguard and ASM.
Some documentations or link explain oracle RAC and ASM configuration will be appreciate.
Thanks for your help.
Regards.
raitsarevoHello,
there's no extra RAC software package, but the option is only offered, if one of the supported cluster layers for the respective OS has been installed before.
10.1.0.3 looks like a complete redesign, but anyway it is a patch, you have to install 10.1.0.2 first. -
Oracle database 10g RAC and ASm installation
Hi all,
I want to ask to everybody something about Oracle 10g RAC and ASm configuration. We plan to migrate to Oracle 10g from 9i, and we will begin configuring oracle but we have to decide which configuration are the best.
Our materials are bellow:
Hardware: RP 3440 (HP)
OS : HPUX 11i Ver 1
Storage: EVA 4000 (eva disk group)
The problem is:
Our supplier recommand us to use HP serviguard + HP serviceguard extension for RAC+ RAc and Raw device as configuration.
But we want to use Oracekl Clusterware + RAC + ASM
My question is if anybody know what is the best configuration, we want to use ASm.
Can we use HP serviguard and ASM.
Some documentations or link explain oracle RAC and ASM configuration will be appreciate.
Thanks for your help.
Regards.
raitsarevoHi Experts ,
Using this command
$ emca
STARTED EMCA at Mon Mar 09 16:13:13 GMT+07:00 2009
Enter the following information about the database to be configured
Listener port number:
Following error comes:
Exception in thread "main" java.lang.NoClassDefFoundError: oracle/sysman/emSDK/conf/InventoryLoader
at oracle.sysman.emcp.EMConfig.getFreePorts(EMConfig.java:4225)
at oracle.sysman.emcp.EMConfig.checkParameters(EMConfig.java:994)
at oracle.sysman.emcp.EMConfig.perform(EMConfig.java:265)
at oracle.sysman.emcp.EMConfigAssistant.invokeEMCA(EMConfigAssistant.java:692)
at oracle.sysman.emcp.EMConfigAssistant.performSetup(EMConfigAssistant.java:641)
at oracle.sysman.emcp.EMConfigAssistant.statusMain(EMConfigAssistant.java:340)
at oracle.sysman.emcp.EMConfigAssistant.main(EMConfigAssistant.java:180)
Moreover this command does not work on Aix 5.3
emca -config dbcontrol db -repos recreate
Regards,
Edited by: LazyDBA10g on Mar 16, 2009 6:20 AM -
Oracle Grid Infrastructure and ASM Tech Supprot
Considering purchase the Tech Support from Oracle for Grid Infrastructure and ASM. Is there a web site/link I can go to?
Is the tech support for ASM covered under the Oracle Database tech support package or I need a separate support package for it?
-
Oracle RAC and ASM for SAP in SLES
Hello,
We are installing our SAP ERP with Oracle RAC 11.2.0.2 on SLES 11 x86_64. According note 527843 (Oracle RAC support in the SAP environment), this is supported for SAP. But as a requirement ASM Cluster File System must be setup.
But when Oracle Grid is installed, ACFS installation returns the following error:
ACFS-9459: ADVM/ACFS is not supported on this OS version: 'sles-release-11.1-1.152'
So Oracle GRID has been installed without ACFS, so we will not be able to use Oracle Clusterware for providing high availability of SAP.
As this is supposed to be a feasible combination (Oracle RAC 11.2.0.2 with ASM on SLES 11 SP1, Netweaver 7.0), I'm wondering if it is a bug, a lack of documentation or something I'm missing.
Kind regards,
FermíRead this below Oracle Note:
ACFS not supported on certain platforms [ID 1075058.1]
oracle@node1:~/app/oracle/product/grid/log/node1> tail -1 alertesiha.log
[client(14083)]CRS-10001:ADVM/ACFS is not supported on SUSE
Then these can be ignored, since ACFS/ADVM may not be supported for your platform. As of this writing, Sun, AIX, SuSE 10 are all supported in 11.2.0.2 only. HP-UX, SuSe 11 and RH6.0, as well as Oracle UEK ((2.6.32-100*) are not yet supported in 11.2.0.2. Keep in mind that this does not prevent your Grid Infrastructure stack (Clusterware and ASM) from starting. This is just an informational message for these platforms
Extract from SAP Note:
RAC 11.2.0.2 (x86 & x86_64 only):
Oracle Clusterware 11.2.0.2 + ASM/ACFS 11.2.0.2 (currently only for SLES10, RHEL5, OL 5.x (without UEK))
Thanks
Srikanth M -
11g CRS and ASM, 10.2.0.4 database, how to set up RMAN
Hi all, I am fairly green when it comes to RAC so please bear with me. I have a Linux 2 node RAC environment. The servers are lux148 and lux149. The 2 node cluster is running 11g CRS and ASM. The database is 10.2.0.4. It had to be set up this way in order for IBM DataStage to work. The database name is fictrp0. The instances are fictrp01 (lux148) and fictrp02 (lux149). I am trying to register the database with an RMAN catalog. I am under the impression that I need to register the database (fictrp0) not the instances (fictrp01 and fictrp02) with the RMAN catalog. Is this correct? So I log into lux148 and set my environment so the ORACLE_SID=FICTRP0. From the command line I issue the following:
fictrp0:/u01/app/oracle> rman target / catalog rman102/[email protected]
The command returns the following:
Recovery Manager: Release 10.2.0.4.0 - Production on Wed Oct 29 14:05:50 2008
Copyright (c) 1982, 2007, Oracle. All rights reserved.
connected to target database (not started)
connected to recovery catalog database
Is this normal "connected to target database (not started)"? I was expecting to see a DBID=FICTRP0 for the target. If this is typical, how will RMAN know the ID of the database? Should I be trying to register an instance perhaps such as FICTRP01? If so do I need to register both instances (FICTRP01 and FICTRP02)?
Bottom line I am very confused on how RAC and RMAN work together. Any help would be greatly appreciated.You no need to register the instance,You need to register only the database and database only will have DBID
-
Oracle 11g ASM - looking for some answers
Hello,
I reviewed a bunch of stuff about Oracle 11g ASM from various sources, but still haven't figured out some answers. Perhaps someone could help me to find the missing links please.
Q1) ASMLib seems to require partitions. ASM recognizes partitions as well as disk devices. So why not using partitions in general? At least I could use ASMLib later without having to reconfigure the drives and loosing all data. Correct?
Q2) The ASMLib driver is not installed by default, and unlike other ASM components is not available from the public yum repository for OEL 5.5. After some searching I found it available for download, but why is it not more obvious. Should it not be used anymore? http://www.oracle.com/technetwork/topics/linux/downloads/rhel5-084877.html
Q3) ASM will not see drives or partitions unless the device ownership is set to Oracle. When Enterprise Linux starts, it resets all block devices to the owner of root. The solution is to create a udev rules file: e.g.:
/etc/udev/rules.d/99-oracle.rules:
KERNEL=="sda", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdb", OWNER="oracle", GROUP="dba", MODE="0660"
Before using ASMLlib, I need to configure the driver, which will ask me for the default user and group who owns the partitions (/etc/init.d/oracleasm configure)
I conclude that using ASMLib will therefore not require a udev rules file, correct?
Q4) The Oracle documentation outlines that ASMLIB is an application programming interface (API) developed by Oracle to simplify the operating system–to-database interface and to exploit the capabilities of vendors' storage arrays on Linux-based systems. What does this mean, examples? Isn't this what ASM does in general, why ASMLIB?
Q5) I installed ASM and would like to re-configure it. Apparently ASM alone does not provide an interface other than sqlplus. In order use ASM with Enterprise Manager, I need to install a database or EM 11g (?) Grid control. Correct?
Thanks,
MarkusHi,
So you got answers for 3 4 and 5
Q1) ASMLib seems to require partitions. ASM recognizes partitions as well as disk devices. So why not using partitions in general? At least I could use ASMLib later >without having to reconfigure the drives and loosing all data. Correct?Using ASMLib, you can use your understandable disk partitions names rather than creating raw devices and adding entry in /etc/raw file and changing rights on every startup, so it makes life easier. Read post of "wcoekaer" from following thread
ASMLib and Linux block devices
Q2) The ASMLib driver is not installed by default, and unlike other ASM components is not available from the public yum repository for OEL 5.5. After some >searching I found it available for download, but why is it not more obvious. Should it not be used anymore? http://www.oracle.com/technetwork/topics/linux/downloads/rhel5-084877.html
I am not sure about OEL 5, but i have seen it in OEL 4 and i can install this package during OEL 4 installation. But i believe you should be able to find this package in one of the Disks of OEL 5
Salman -
After bring the Cluster and ASM is up, I'm not able to start the rdbms ins.
I am not able to bring the database up.
Cluster and ASM is up and running.
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/oracle/app/oracle/product/11.1.0/db_1/dbs/initTMISC11G.ora'That is wre the confusion is: I coud nt find it under O_H/dbs,
However the log file indicates that it is using spfile which is located at :
spfile = +DATA_TIER/dintw10g/parameterfile/spfile_dintw10g.ora
Now it checks only the /dbs directory only i guess.
Could you please how can i bring it up now.
Just for the information: ASM instance is up. -
How to delete client from oracle 10g ASM
Hi,
I would like to know how to delete client from oracle 10g ASM, while configuring Dataguard in same windows os i have by mistake created the standby in ASM.
However later i have corrected the datafile destination. My Primary database is in ASM and the standby in disk.
Could anyone please tell me how do i delete the client from ASM. On view from asmcmd it shows db_unknown and from v$asm_client
GROUP_NUMBER INSTANCE_NAME
DB_NAME STATUS
SOFTWARE_VERSION
COMPATIBLE_VERSION
1 stby
stby CONNECTED
10.2.0.1.0
10.2.0.1.0
Thanks and regards,
shaanHi Deepak,
Pls see the result from the view, i wanted to delete stby
SQL> select * from v$asm_client;
GROUP_NUMBER INSTANCE_NAME
DB_NAME STATUS
SOFTWARE_VERSION
COMPATIBLE_VERSION
1 prim
prim CONNECTED
10.2.0.1.0
10.2.0.1.0
2 prim
prim CONNECTED
10.2.0.1.0
10.2.0.1.0
1 stby
stby CONNECTED
10.2.0.1.0
10.2.0.1.0
regards,
shaan -
HI
Can someone give the difference between RAID 10 and ASM
From my understanding both is doing stripping of data.
If we are implementing RAID 10 ,then do we need to implement ASM (if yes what is the advantage)
Thanksuser10394804 wrote:
Can someone give the difference between RAID 10 and ASMWhat is the difference between a car and the road? ASM=vehicle. RAID=road.
In other words, ASM is a Storage (or Volume) Manager System that runs on different RAID implementations. The RAID implementation can be at the actual storage layer (hardware RAID).. in which case RAID is external to ASM. Or ASM can itself be used to implement RAID to run on (software RAID).
From my understanding both is doing stripping of data.ASM automatically stripes data across disks (e.g. RAID10 disks) in a diskgroup. RAID10 is a combination of striping and mirroring. So yes, both use striping - but ASM != RAID10. These are 2 very different layers in the storage tier.
If we are implementing RAID 10 ,then do we need to implement ASM (if yes what is the advantage)Yes. ASM manages the storage layer for you. It can perform dynamic load balancing. With ASM 11r2, there are even more features that are introduced that makes ASM a very important part in tying the storage layer effectively and efficiently with the o/s layer and the database layer.
Oracle specifically recommends using ASM for RAC. -
Hi All,
i am getting an error message in asm_alert log file saying "NOTE: ASMB process exiting due to lack of ASM file activity".
This leads to frequent crashing of NODE 1. Please check below detail error and suggest solution.
Thu Mar 24 07:05:11 2011
LMD0 (ospid: 32493) has not called a wait for 94 secs.
GES: System Load is HIGH.
GES: Current load is 55.87 and high load threshold is 20.00
Thu Mar 24 07:06:32 2011
LMD0 (ospid: 32493) has not called a wait for 174 secs.
GES: System Load is HIGH.
GES: Current load is 71.23 and high load threshold is 20.00
Thu Mar 24 07:06:36 2011
+Trace dumping is performing id=[cdmp_20110324070635]+
Thu Mar 24 07:07:49 2011
+Trace dumping is performing id=[cdmp_20110324070635]+
Thu Mar 24 07:08:16 2011
Waiting for clusterware split-brain resolution
Thu Mar 24 07:18:17 2011
Errors in file /u01/app/oracle/diag/asm/asm/+ASM1/trace/+ASM1_lmon_32484.trc (incident=60073):+
ORA-29740: evicted by member 1, group incarnation 120
Incident details in: /u01/app/oracle/diag/asm/asm/+ASM1/incident/incdir_60073/+ASM1_lmon_32484_i60073.trc+
Thu Mar 24 07:18:19 2011
+Trace dumping is performing id=[cdmp_20110324071819]+
Errors in file /u01/app/oracle/diag/asm/asm/+ASM1/trace/+ASM1_lmon_32484.trc:+
ORA-29740: evicted by member 1, group incarnation 120
LMON (ospid: 32484): terminating the instance due to error 29740
System state dump is made for local instance
System State dumped to trace file /u01/app/oracle/diag/asm/asm/+ASM1/trace/+ASM1_diag_32459.trc+
+Trace dumping is performing id=[cdmp_20110324071820]+
Instance terminated by LMON, pid = 32484
Thu Mar 24 07:18:31 2011
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 172.20.223.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 172.20.222.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/asm_1/dbs/arch
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 11.1.0.7.0.
Using parameter settings in server-side pfile /u01/app/oracle/product/11.1.0/asm_1/dbs/initASM1.ora+
System parameters with non-default values:
large_pool_size = 12M
instance_type = "asm"
cluster_database = TRUE
instance_number = 1
asm_diskstring = "ORCL:*"
asm_diskgroups = "REDO01"
asm_diskgroups = "REDO02"
asm_diskgroups = "DATA"
asm_diskgroups = "RECOVERY"
diagnostic_dest = "/u01/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance
+172.20.223.25+
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
Thu Mar 24 07:18:36 2011
PMON started with pid=2, OS id=23120
Thu Mar 24 07:18:36 2011
VKTM started with pid=3, OS id=23123 at elevated priority
VKTM running at (20)ms precision
Thu Mar 24 07:18:36 2011
DIAG started with pid=4, OS id=23127
Thu Mar 24 07:18:37 2011
PING started with pid=5, OS id=23129
Thu Mar 24 07:18:37 2011
PSP0 started with pid=6, OS id=23131
Thu Mar 24 07:18:37 2011
DIA0 started with pid=7, OS id=23133
Thu Mar 24 07:18:37 2011
LMON started with pid=8, OS id=23135
Thu Mar 24 07:18:37 2011
LMD0 started with pid=9, OS id=23137
Thu Mar 24 07:18:37 2011
LMS0 started with pid=10, OS id=23148 at elevated priority
Thu Mar 24 07:18:37 2011
MMAN started with pid=11, OS id=23152
Thu Mar 24 07:18:38 2011
DBW0 started with pid=12, OS id=23170
Thu Mar 24 07:18:38 2011
LGWR started with pid=13, OS id=23176
Thu Mar 24 07:18:38 2011
CKPT started with pid=14, OS id=23218
Thu Mar 24 07:18:38 2011
SMON started with pid=15, OS id=23224
Thu Mar 24 07:18:38 2011
RBAL started with pid=16, OS id=23237
Thu Mar 24 07:18:38 2011
GMON started with pid=17, OS id=23239
lmon registered with NM - instance id 1 (internal mem no 0)
Reconfiguration started (old inc 0, new inc 124)
ASM instance
List of nodes:
+0 1 2+
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* allocate domain 1, invalid = TRUE
* allocate domain 2, invalid = TRUE
* allocate domain 3, invalid = TRUE
* allocate domain 4, invalid = TRUE
* domain 0 valid = 1 according to instance 1
* domain 1 valid = 1 according to instance 1
* domain 2 valid = 1 according to instance 1
* domain 3 valid = 1 according to instance 1
* domain 4 valid = 1 according to instance 1
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
LMS 0: 0 GCS shadows traversed, 0 replayed
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
Thu Mar 24 07:18:40 2011
LCK0 started with pid=18, OS id=23277
ORACLE_BASE from environment = /u01/app/oracle
Thu Mar 24 07:18:41 2011
SQL> ALTER DISKGROUP ALL MOUNT
NOTE: cache registered group DATA number=1 incarn=0xf7063e39
NOTE: cache began mount (not first) of group DATA number=1 incarn=0xf7063e39
NOTE: cache registered group RECOVERY number=2 incarn=0xf7063e3a
NOTE: cache began mount (not first) of group RECOVERY number=2 incarn=0xf7063e3a
NOTE: cache registered group REDO01 number=3 incarn=0xf7163e3b
NOTE: cache began mount (not first) of group REDO01 number=3 incarn=0xf7163e3b
NOTE: cache registered group REDO02 number=4 incarn=0xf7163e3c
NOTE: cache began mount (not first) of group REDO02 number=4 incarn=0xf7163e3c
NOTE:Loaded lib: /opt/oracle/extapi/32/asm/orcl/1/libasm.so
NOTE: Assigning number (1,0) to disk (ORCL:ASM_DATA1)
NOTE: Assigning number (1,1) to disk (ORCL:ASM_DATA2)
NOTE: Assigning number (2,0) to disk (ORCL:ASM_RECO1)
NOTE: Assigning number (3,0) to disk (ORCL:ASM_LOG1)
NOTE: Assigning number (4,0) to disk (ORCL:ASM_LOG2)
kfdp_query(): 5
kfdp_queryBg(): 5
NOTE: cache opening disk 0 of grp 1: DATA1 label:ASM_DATA1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache opening disk 1 of grp 1: DATA2 label:ASM_DATA2
NOTE: cache mounting (not first) group 1/0xF7063E39 (DATA)
kjbdomatt send to node 1
kjbdomatt send to node 2
NOTE: attached to recovery domain 1
NOTE: LGWR attempting to mount thread 1 for diskgroup 1
NOTE: LGWR mounted thread 1 for disk group 1
NOTE: opening chunk 1 at fcn 0.10794571 ABA
NOTE: seq=81 blk=1313
NOTE: cache mounting group 1/0xF7063E39 (DATA) succeeded
NOTE: cache ending mount (success) of group DATA number=1 incarn=0xf7063e39
kfdp_query(): 6
kfdp_queryBg(): 6
NOTE: cache opening disk 0 of grp 2: RECO1 label:ASM_RECO1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 2/0xF7063E3A (RECOVERY)
kjbdomatt send to node 1
kjbdomatt send to node 2
NOTE: attached to recovery domain 2
NOTE: LGWR attempting to mount thread 1 for diskgroup 2
NOTE: LGWR mounted thread 1 for disk group 2
NOTE: opening chunk 1 at fcn 0.10436377 ABA
NOTE: seq=48 blk=4298
NOTE: cache mounting group 2/0xF7063E3A (RECOVERY) succeeded
NOTE: cache ending mount (success) of group RECOVERY number=2 incarn=0xf7063e3a
kfdp_query(): 7
kfdp_queryBg(): 7
NOTE: cache opening disk 0 of grp 3: LOG1 label:ASM_LOG1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 3/0xF7163E3B (REDO01)
kjbdomatt send to node 1
kjbdomatt send to node 2
NOTE: attached to recovery domain 3
NOTE: LGWR attempting to mount thread 1 for diskgroup 3
NOTE: LGWR mounted thread 1 for disk group 3
NOTE: opening chunk 1 at fcn 0.229332 ABA
NOTE: seq=30 blk=10690
NOTE: cache mounting group 3/0xF7163E3B (REDO01) succeeded
NOTE: cache ending mount (success) of group REDO01 number=3 incarn=0xf7163e3b
kfdp_query(): 8
kfdp_queryBg(): 8
NOTE: cache opening disk 0 of grp 4: LOG2 label:ASM_LOG2
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 4/0xF7163E3C (REDO02)
kjbdomatt send to node 1
kjbdomatt send to node 2
NOTE: attached to recovery domain 4
NOTE: LGWR attempting to mount thread 1 for diskgroup 4
NOTE: LGWR mounted thread 1 for disk group 4
NOTE: opening chunk 1 at fcn 0.225880 ABA
NOTE: seq=30 blk=10556
NOTE: cache mounting group 4/0xF7163E3C (REDO02) succeeded
NOTE: cache ending mount (success) of group REDO02 number=4 incarn=0xf7163e3c
kfdp_query(): 9
kfdp_queryBg(): 9
NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1
SUCCESS: diskgroup DATA was mounted
kfdp_query(): 10
kfdp_queryBg(): 10
NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2
SUCCESS: diskgroup RECOVERY was mounted
kfdp_query(): 11
kfdp_queryBg(): 11
NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3
SUCCESS: diskgroup REDO01 was mounted
kfdp_query(): 12
kfdp_queryBg(): 12
NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 4
SUCCESS: diskgroup REDO02 was mounted
SUCCESS: ALTER DISKGROUP ALL MOUNT
Thu Mar 24 08:26:28 2011
Starting background process ASMB
Thu Mar 24 08:26:28 2011
ASMB started with pid=20, OS id=9597
NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds
Thu Mar 24 08:27:39 2011
Starting background process ASMB
Thu Mar 24 08:27:39 2011
ASMB started with pid=25, OS id=10735
NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds
+[oracle@qa1crmrac1 trace]$ tail -1500 alert_ASM1.log
Thu Mar 24 07:05:11 2011
LMD0 (ospid: 32493) has not called a wait for 94 secs.
GES: System Load is HIGH.
GES: Current load is 55.87 and high load threshold is 20.00
Thu Mar 24 07:06:32 2011
LMD0 (ospid: 32493) has not called a wait for 174 secs.
GES: System Load is HIGH.
GES: Current load is 71.23 and high load threshold is 20.00
Thu Mar 24 07:06:36 2011
+Trace dumping is performing id=[cdmp_20110324070635]+
Thu Mar 24 07:07:49 2011
+Trace dumping is performing id=[cdmp_20110324070635]+
Thu Mar 24 07:08:16 2011
Waiting for clusterware split-brain resolution
Thu Mar 24 07:18:17 2011
Errors in file /u01/app/oracle/diag/asm/asm/+ASM1/trace/+ASM1_lmon_32484.trc (incident=60073):+
ORA-29740: evicted by member 1, group incarnation 120
Incident details in: /u01/app/oracle/diag/asm/asm/+ASM1/incident/incdir_60073/+ASM1_lmon_32484_i60073.trc+
Thu Mar 24 07:18:19 2011
+Trace dumping is performing id=[cdmp_20110324071819]+
Errors in file /u01/app/oracle/diag/asm/asm/+ASM1/trace/+ASM1_lmon_32484.trc:+
ORA-29740: evicted by member 1, group incarnation 120
LMON (ospid: 32484): terminating the instance due to error 29740
System state dump is made for local instance
System State dumped to trace file /u01/app/oracle/diag/asm/asm/+ASM1/trace/+ASM1_diag_32459.trc+
+Trace dumping is performing id=[cdmp_20110324071820]+
Instance terminated by LMON, pid = 32484
Thu Mar 24 07:18:31 2011
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 172.20.223.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 172.20.222.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/asm_1/dbs/arch
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 11.1.0.7.0.
Using parameter settings in server-side pfile /u01/app/oracle/product/11.1.0/asm_1/dbs/initASM1.ora+
System parameters with non-default values:
large_pool_size = 12M
instance_type = "asm"
cluster_database = TRUE
instance_number = 1
asm_diskstring = "ORCL:*"
asm_diskgroups = "REDO01"
asm_diskgroups = "REDO02"
asm_diskgroups = "DATA"
asm_diskgroups = "RECOVERY"
diagnostic_dest = "/u01/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance
+172.20.223.25+
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
Thu Mar 24 07:18:36 2011
PMON started with pid=2, OS id=23120
Thu Mar 24 07:18:36 2011
VKTM started with pid=3, OS id=23123 at elevated priority
VKTM running at (20)ms precision
Thu Mar 24 07:18:36 2011
DIAG started with pid=4, OS id=23127
Thu Mar 24 07:18:37 2011
PING started with pid=5, OS id=23129
Thu Mar 24 07:18:37 2011
PSP0 started with pid=6, OS id=23131
Thu Mar 24 07:18:37 2011
DIA0 started with pid=7, OS id=23133
Thu Mar 24 07:18:37 2011
LMON started with pid=8, OS id=23135
Thu Mar 24 07:18:37 2011
LMD0 started with pid=9, OS id=23137
Thu Mar 24 07:18:37 2011
LMS0 started with pid=10, OS id=23148 at elevated priority
Thu Mar 24 07:18:37 2011
MMAN started with pid=11, OS id=23152
Thu Mar 24 07:18:38 2011
DBW0 started with pid=12, OS id=23170
Thu Mar 24 07:18:38 2011
LGWR started with pid=13, OS id=23176
Thu Mar 24 07:18:38 2011
CKPT started with pid=14, OS id=23218
Thu Mar 24 07:18:38 2011
SMON started with pid=15, OS id=23224
Thu Mar 24 07:18:38 2011
RBAL started with pid=16, OS id=23237
Thu Mar 24 07:18:38 2011
GMON started with pid=17, OS id=23239
lmon registered with NM - instance id 1 (internal mem no 0)
Reconfiguration started (old inc 0, new inc 124)
ASM instance
List of nodes:
+0 1 2+
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* allocate domain 1, invalid = TRUE
* allocate domain 2, invalid = TRUE
* allocate domain 3, invalid = TRUE
* allocate domain 4, invalid = TRUE
* domain 0 valid = 1 according to instance 1
* domain 1 valid = 1 according to instance 1
* domain 2 valid = 1 according to instance 1
* domain 3 valid = 1 according to instance 1
* domain 4 valid = 1 according to instance 1
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
LMS 0: 0 GCS shadows traversed, 0 replayed
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
Thu Mar 24 07:18:40 2011
LCK0 started with pid=18, OS id=23277
ORACLE_BASE from environment = /u01/app/oracle
Thu Mar 24 07:18:41 2011
SQL> ALTER DISKGROUP ALL MOUNT
NOTE: cache registered group DATA number=1 incarn=0xf7063e39
NOTE: cache began mount (not first) of group DATA number=1 incarn=0xf7063e39
NOTE: cache registered group RECOVERY number=2 incarn=0xf7063e3a
NOTE: cache began mount (not first) of group RECOVERY number=2 incarn=0xf7063e3a
NOTE: cache registered group REDO01 number=3 incarn=0xf7163e3b
NOTE: cache began mount (not first) of group REDO01 number=3 incarn=0xf7163e3b
NOTE: cache registered group REDO02 number=4 incarn=0xf7163e3c
NOTE: cache began mount (not first) of group REDO02 number=4 incarn=0xf7163e3c
NOTE:Loaded lib: /opt/oracle/extapi/32/asm/orcl/1/libasm.so
NOTE: Assigning number (1,0) to disk (ORCL:ASM_DATA1)
NOTE: Assigning number (1,1) to disk (ORCL:ASM_DATA2)
NOTE: Assigning number (2,0) to disk (ORCL:ASM_RECO1)
NOTE: Assigning number (3,0) to disk (ORCL:ASM_LOG1)
NOTE: Assigning number (4,0) to disk (ORCL:ASM_LOG2)
kfdp_query(): 5
kfdp_queryBg(): 5
NOTE: cache opening disk 0 of grp 1: DATA1 label:ASM_DATA1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache opening disk 1 of grp 1: DATA2 label:ASM_DATA2
NOTE: cache mounting (not first) group 1/0xF7063E39 (DATA)
kjbdomatt send to node 1
kjbdomatt send to node 2
NOTE: attached to recovery domain 1
NOTE: LGWR attempting to mount thread 1 for diskgroup 1
NOTE: LGWR mounted thread 1 for disk group 1
NOTE: opening chunk 1 at fcn 0.10794571 ABA
NOTE: seq=81 blk=1313
NOTE: cache mounting group 1/0xF7063E39 (DATA) succeeded
NOTE: cache ending mount (success) of group DATA number=1 incarn=0xf7063e39
kfdp_query(): 6
kfdp_queryBg(): 6
NOTE: cache opening disk 0 of grp 2: RECO1 label:ASM_RECO1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 2/0xF7063E3A (RECOVERY)
kjbdomatt send to node 1
kjbdomatt send to node 2
NOTE: attached to recovery domain 2
NOTE: LGWR attempting to mount thread 1 for diskgroup 2
NOTE: LGWR mounted thread 1 for disk group 2
NOTE: opening chunk 1 at fcn 0.10436377 ABA
NOTE: seq=48 blk=4298
NOTE: cache mounting group 2/0xF7063E3A (RECOVERY) succeeded
NOTE: cache ending mount (success) of group RECOVERY number=2 incarn=0xf7063e3a
kfdp_query(): 7
kfdp_queryBg(): 7
NOTE: cache opening disk 0 of grp 3: LOG1 label:ASM_LOG1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 3/0xF7163E3B (REDO01)
kjbdomatt send to node 1
kjbdomatt send to node 2
NOTE: attached to recovery domain 3
NOTE: LGWR attempting to mount thread 1 for diskgroup 3
NOTE: LGWR mounted thread 1 for disk group 3
NOTE: opening chunk 1 at fcn 0.229332 ABA
NOTE: seq=30 blk=10690
NOTE: cache mounting group 3/0xF7163E3B (REDO01) succeeded
NOTE: cache ending mount (success) of group REDO01 number=3 incarn=0xf7163e3b
kfdp_query(): 8
kfdp_queryBg(): 8
NOTE: cache opening disk 0 of grp 4: LOG2 label:ASM_LOG2
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 4/0xF7163E3C (REDO02)
kjbdomatt send to node 1
kjbdomatt send to node 2
NOTE: attached to recovery domain 4
NOTE: LGWR attempting to mount thread 1 for diskgroup 4
NOTE: LGWR mounted thread 1 for disk group 4
NOTE: opening chunk 1 at fcn 0.225880 ABA
NOTE: seq=30 blk=10556
NOTE: cache mounting group 4/0xF7163E3C (REDO02) succeeded
NOTE: cache ending mount (success) of group REDO02 number=4 incarn=0xf7163e3c
kfdp_query(): 9
kfdp_queryBg(): 9
NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1
SUCCESS: diskgroup DATA was mounted
kfdp_query(): 10
kfdp_queryBg(): 10
NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2
SUCCESS: diskgroup RECOVERY was mounted
kfdp_query(): 11
kfdp_queryBg(): 11
NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3
SUCCESS: diskgroup REDO01 was mounted
kfdp_query(): 12
kfdp_queryBg(): 12
NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 4
SUCCESS: diskgroup REDO02 was mounted
SUCCESS: ALTER DISKGROUP ALL MOUNT
Thu Mar 24 08:26:28 2011
Starting background process ASMB
Thu Mar 24 08:26:28 2011
ASMB started with pid=20, OS id=9597
NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds
Thu Mar 24 08:27:39 2011
Starting background process ASMB
Thu Mar 24 08:27:39 2011
ASMB started with pid=25, OS id=10735
NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds
Do i need to set the compatible parameter?
Regards,
VishIt looks to me like your server is absolutely buried, and ASM may just be an innocent bystander. What is going on in the database when this happens? Also, run sar samples at 30 second intervals up to when this happens to see what is happening. It's overhead, but you need to find what is causing the problem with the server(s).
Are you swapping?
Maybe you are looking for
-
Reg-Creating Data source in BI
Hi All, I need to crate a data source in BI By using the following tables TBTCP, TBTCO, TRDIRT, TSTC..First two tables having common relationship, where as there is no relationship between TRDIRT and TSTC tables....For this i am planning to do by cre
-
Blue Screen with installing new Windows on already pre-installed one
Hello everyone, I am sorry if that topic has already been dealt with, and if I am asking about the same thing. I found a few posts relating to my problem, but it did not fully help, so... I bought a M70 in Germany, it has Win XP HE already installed,
-
Streams & Sequential Access Files ....
I have two sequential files that were created from 'FileInputStream' ... TRANSACTIONS.DAT and MASTERFILE.DAT I have wrapped a 'FileInputSteam' with a 'ObjectInputStream' for both, which will be used for comparisons. I know how I could compare the two
-
BBM status update is extra sensitive....
I have been facing problem with BBM status update and display name screens. They are extar sensitive and change immediately before you actually decide to press enter. Any solutions?
-
Workflow -Inconsistent behavior of wf_engine.setitemattrdate
I have customized HR self service workflow for termination. I am using wf_engine.setitemattrdate to set the value for all the custom attributes, sometimes the attributes do not come through in the notification, though they are derived; they work as e