ACE module status is unknown "FSM_FT_STATE_STANDBY_COLD",
suddenly ACE module status is unknown “FSM_FT_STATE_STANDBY_COLD”, after reloading the FT worked fine , i need to know the root cause for this ?
do a 'shut' / 'no shut' of the ft vlan interface.
If that does not bring the interface up, check on the cat6k if FT vlan assigned to SVCLC group associated with ACE.
You should also plan an upgrade to A2(2.4) since a similar problem was fixed.
Gilles.
Similar Messages
-
ACE Module: Recover a real server probe-failed status
How does the ACE module recover a real server that has entered a probe-failed status state? We are doing some testing, purposely dropping a servers interface. ACE recognizes the server as being down and show it in a probe-failed state. When we bring the system's interface back up, will ACE see this and automatically bring the state back into Operational status, or does someone have to do something on the ACE module?
ACE continues to probe servers that are down or probe_failed. As soon as a server starts responding again its state will switch to alive again.
Nothing to be done.
Gilles. -
Hi,
I am trying to configure FT on ACE modules, with the following commands
ft interface vlan 20
ip address 172.16.20.1 255.255.255.252
peer ip address 172.16.20.2 255.255.255.252
no shutdown
ft peer 1
heartbeat interval 300
heartbeat count 10
ft-interface vlan 20
ft group 1
peer 1
priority 150
associate-context Admin
inservice
The moment I enter the command 'ft interface vlan 20', it gives a prompt that 'interface vlan20 is not associated with ft', how do I resolve this ? Do I need to enable something ?Hi have the following config which seems to be working fine for me... check your vlan20 interface is up
ft interface vlan 212
ip address 172.31.1.221 255.255.255.252
peer ip address 172.31.1.222 255.255.255.252
no shutdown
ft peer 1
heartbeat interval 300
heartbeat count 20
ft-interface vlan 212
ft group 2
peer 1
priority 50
peer priority 150
associate-context Admin
inservice
HQ-ACE1/Admin# sh int
vlan212 is up, administratively up
Hardware type is VLAN
MAC address is 00:23:5e:25:72:f1
Mode : routed
IP address is 172.31.1.221 netmask is 255.255.255.252
FT status is standby
Description:not set
MTU: 1500 bytes
Last cleared: never
Last Changed: Tue Sep 6 12:46:06 2011
No of transitions: 1
Alias IP address not set
Peer IP address is 172.31.1.222 Peer IP netmask is 255.255.255.252
Assigned from the Supervisor, up on Supervisor
8654909 unicast packets input, 735611030 bytes
1151150 multicast, 161 broadcast
0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops
13020418 unicast packets output, 1672055521 bytes
0 multicast, 163 broadcast
0 output errors, 0 ignored -
Cisco ACE module missing licence file - no connectivity
Hi,
We have 2 ACE modules that were delivered without any licenses.
There is no IP connectivity whatsoever to these modules and I'm guessing this is due to the fact there are no licenses installed.
Have tried asking Cisco to no avail - and am not sure if there is an actual problem with them or not.
The VLANs are assigned correctly and I can see inbound ICMP echo from the 6509 that its hosted in, but no outbound packets ever leave the ACE. I've applied a mgmt policy to enable ping/telnet/ssh etc.
switch/Admin# sh vlans
Vlans configured on SUP for this module
vlan4 vlan30-31 vlan160 vlan180-195 vlan360 vlan380-395 vlan560 vlan580-
595 vlan760 vlan780-795
switch/Admin# sh ip int bri
Interface IP-Address Status Protocol
vlan4 10.119.127.196 up up
vlan30 10.119.127.241 up up
vlan31 10.119.127.245 up up
interface vlan 4
description ACE Mgmt interface for Admin Context
ip address 10.119.127.196 255.255.255.224
service-policy input REMOTE_MGMT
no shutdown
vlan4 is up
Hardware type is VLAN
MAC address is 00:1f:ca:7b:6f:33
Mode : routed
IP address is 10.119.127.196 netmask is 255.255.255.224
FT status is non-redundant
Description:ACE Mgmt interface for Admin Context
MTU: 1500 bytes
Last cleared: never
Alias IP address not set
Peer IP address not set
Assigned from the Supervisor, up on Supervisor
Config download failures : 1
2980 unicast packets input, 16363862 bytes
240857 multicast, 3026 broadcast
0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops
0 unicast packets output, 187712 bytes
0 multicast, 2933 broadcast
0 output errors, 0 ignored
switch/Admin# sh arp
Context Admin
================================================================================
IP ADDRESS MAC-ADDRESS Interface Type Encap NextArp(s) Status
================================================================================
10.119.127.193 00.00.00.00.00.00 vlan4 GATEWAY - * 3 req dn
10.119.127.196 00.1f.ca.7b.6f.33 vlan4 INTERFACE LOCAL _ up
10.119.127.245 00.1f.ca.7b.6f.33 vlan31 INTERFACE LOCAL _ up
10.119.127.241 00.1f.ca.7b.6f.33 vlan30 INTERFACE LOCAL _ up
================================================================================
Total arp entries 4
The ARP table for the adjacent switch SVI has a valid MAC upon reboot, but soon after resets to 00.00.00.00.00.00
Problem is that once Cisco eventually send me the license file I have no way of TFTP'ing it to the ACE module.
Any suggestions/advice?Thanks for the info - so I should at least be able to connect to a license-less ACE at least, but these modules seem to have a problem.
If the modules are reloaded (from the ACE) or reset (from the Supervisor) they initially have the ARP entry (however still cannot communicate to the attached Supervisor via SVI) which eventually resets.
Info as requested:
switch/Admin# sh resource usage
Allocation
Resource Current Peak Min Max Denied
Context: Admin
conc-connections 9 9 0 0 0
mgmt-connections 0 0 0 0 0
proxy-connections 0 0 0 0 0
xlates 0 0 0 0 0
bandwidth 0 76 0 125000000 296849008
throughput 0 76 0 0 296849008
mgmt-traffic rate 0 0 0 125000000 0
connection rate 0 2 0 0 15
ssl-connections rate 0 0 0 0 0
mac-miss rate 0 0 0 0 0
inspect-conn rate 0 0 0 0 0
acl-memory 0 6336 0 0 11
sticky 0 0 0 0 0
regexp 0 0 0 0 0
syslog buffer 0 0 0 0 0
syslog rate 0 0 0 0 24
Context: APPLICATION
conc-connections 0 0 2000000 0 0
mgmt-connections 0 0 25000 0 0
proxy-connections 0 0 262144 0 0
xlates 0 0 262144 0 0
bandwidth 0 0 125000000 125000000 0
throughput 0 0 125000000 0 0
mgmt-traffic rate 0 0 0 125000000 0
connection rate 0 0 250000 0 0
ssl-connections rate 0 0 250 0 0
mac-miss rate 0 0 500 0 0
inspect-conn rate 0 0 1500 0 0
acl-memory 0 0 19650480 0 0
sticky 0 0 419430 0 0
regexp 0 0 262144 0 0
syslog buffer 0 0 1048576 0 0
syslog rate 0 0 25000 0 0
Context: BACK_END
conc-connections 0 0 2000000 0 0
mgmt-connections 0 0 25000 0 0
proxy-connections 0 0 262144 0 0
xlates 0 0 262144 0 0
bandwidth 0 0 125000000 125000000 0
throughput 0 0 125000000 0 0
mgmt-traffic rate 0 0 0 125000000 0
connection rate 0 0 250000 0 0
ssl-connections rate 0 0 250 0 0
mac-miss rate 0 0 500 0 0
inspect-conn rate 0 0 1500 0 0
acl-memory 0 0 19650480 0 0
sticky 0 0 419430 0 0
regexp 0 0 262144 0 0
syslog buffer 0 0 1048576 0 0
syslog rate 0 0 25000 0 0
Context: FRONT_END
conc-connections 0 0 2000000 0 0
mgmt-connections 0 0 25000 0 0
proxy-connections 0 0 262144 0 0
xlates 0 0 262144 0 0
bandwidth 0 0 125000000 125000000 0
throughput 0 0 125000000 0 0
mgmt-traffic rate 0 0 0 125000000 0
connection rate 0 0 250000 0 0
ssl-connections rate 0 0 250 0 0
mac-miss rate 0 0 500 0 0
inspect-conn rate 0 0 1500 0 0
acl-memory 0 0 19650480 0 0
sticky 0 0 419430 0 0
regexp 0 0 262144 0 0
syslog buffer 0 0 1048576 0 0
syslog rate 0 0 25000 0 0
Context: TEST_DEV
conc-connections 0 0 2000000 0 0
mgmt-connections 0 0 25000 0 0
proxy-connections 0 0 262144 0 0
xlates 0 0 262144 0 0
bandwidth 0 0 125000000 125000000 0
throughput 0 0 125000000 0 0
mgmt-traffic rate 0 0 0 125000000 0
connection rate 0 0 250000 0 0
ssl-connections rate 0 0 250 0 0
mac-miss rate 0 0 500 0 0
inspect-conn rate 0 0 1500 0 0
acl-memory 0 0 19650480 0 0
sticky 0 0 419430 0 0
regexp 0 0 262144 0 0
syslog buffer 0 0 1048576 0 0
syslog rate 0 0 25000 0 0
switch/Admin# sh cde health
CDE BRCM INTERFACE
======================
Packets received 3357
Packets transmitted 12
Broadcom interface CRC error count 0
BRCM VOQ status [empty] [not full]
BRCM pull status [pulling]
CDE HYPERION INTERFACE
======================
Packets received 7668407
Packets transmitted 967915
Short packets drop count 0
Fifo Full drop count 0
Protocol error drop count 0
FCS error drop count 0
CRC error drop count 0
Num times flow control triggered on hyp interface 0
Num self generated multicast packets filtered 967915
HYP IXP0 VOQ status [empty] [not full]
HYP IXP1 VOQ status [empty] [not full]
HYP SLOW VOQ status [empty] [not full]
HYP tx pull status [pulling]
CDE IXP0 INTERFACE
======================
Packets received 964680
Packets transmitted 6581196
Num bad pkts recvd on fast spi channel0 0
Num bad pkts recvd on slow spi channel8 0
Num bad pkts recvd on fast spi channel2 0
Num bad pkts recvd on slow spi channel4 0
IXP0 Fast VOQ status [empty] [not full]
IXP0 BRCM VOQ status [empty] [not full]
IXP0 pull status [pulling]
IXP0 spi src status [healthy]
IXP0 spi snk status [healthy]
CDE1 SWITCH1 INTERFACE
======================
Packets received (hyp, ixp0) 3241
Packets received (bcm) 6
Packets received (daughter card 0) 0
Packets received (daughter card 1) 0
Packets Errors received (hyp, ixp0) 0
Packets Errors received (bcm) 0
Packets Errors received (daughter card 0) 0
Packets Errors received (daughter card 1) 0
Packets transmitted (ixp1) 122653
Packets transmitted (nitrox) 0
Packets Errors transmitted (ixp1) 0
Packets Errors transmitted (nitrox) 0
CDE2 SWITCH2 INTERFACE
======================
Packets received (ixp1) 122653
Packets received (nitrox) 0
Packets Errors received (ixp1) 0
Packets Errors received (nitrox) 0
Packets transmitted (hyp, ixp0) 3241
Packets transmitted (broadcom) 6
Packets transmitted (daughter card 0) 0
Packets transmitted (daughter card 1) 0
Packets Errors transmitted (ixp1) 0
Packets Errors transmitted (nitrox) 0
Packets Errors transmitted (daughter card 0) 0
Packets Errors transmitted (daughter card 1) 0
CDE IXP1 INTERFACE
======================
Packets received 3247
Packets transmitted 122653
Num bad pkts recvd on fast spi channel0 0
Num bad pkts recvd on slow spi channel8 0
Num bad pkts recvd on fast spi channel2 0
Num bad pkts recvd on slow spi channel4 0
IXP1 Fast VOQ status [empty] [not full]
IXP1 BRCM VOQ status [empty] [not full]
IXP1 pull status [pulling]
IXP1 spi src status [healthy]
IXP1 spi snk status [healthy]
CDE NITROX INTERFACE
======================
Packets received 0
Packets transmitted 0
Num bad pkts recvd on fast spi channel0 0
Num bad pkts recvd on slow spi channel8 0
Num bad pkts recvd on fast spi channel2 0
Num bad pkts recvd on slow spi channel4 0
NTX Fast VOQ status [empty] [not full]
NTX BRCM VOQ status [empty] [not full]
NTX pull status [pulling]
NTX spi src status [healthy]
NTX spi snk status [healthy]
== Backplane ==
ITASCA_SYS_CNTL1 0x300 data 0x61f0000
ITASCA_SYS_CNTL2 0x304 data 0x80630000 -
Reuse of context in ACE module
Hi all, just have a question about som reuse of resources in a ACE module context. I don't want to make a new context, and can reuse most of the existing configuration in one of my context. The config is not complex and difficult, but I'm not sure if I can do this.
The primary goal is to loadbalance 2 webservers with a new vip, new serverfarm, stickygroup, policy-map and different nat-pool.
Since I haven't decided the ip addresses to be used, they are just xx in the config below.
The changes I want to implement are in bold. Will this work for me?
probe http WEBGUI_D2
description Probe for http mot webgui
interval 10
passdetect interval 10
passdetect count 1
request method get url /D2/auth/login.aspx
expect status 200 302
header User-Agent header-value "IDENTITY"
rserver host cwi003
description content server logon
ip address 10.163.22.27
inservice
rserver host cwi004
description content server logon
ip address 10.163.22.28
inservice
rserver host cwi503
description content server logon 2
ip address 10.163.22.23
inservice
rserver host cwi504
description content server logon 2
ip address 10.163.22.24
inservice
serverfarm host SF_LOGON_D2
probe WEBGUI_D2
rserver cwi003 80
inservice
rserver cwi004 80
inservice
serverfarm host SF_LOGON2_D2
probe WEBGUI_D2
rserver cwi503 80
inservice
rserver cwi504 80
inservice
sticky ip-netmask 255.255.255.255 address source STICKYGROUP1
timeout 20
replicate sticky
serverfarm SF_LOGON_D2
serverfarm SF_LOGON2_D2
class-map match-all VS_LOGON_D2
3 match virtual-address 10.163.22.13 any
class-map match-all VS_LOGON2_D2
3 match virtual-address 10.163.22.xx any
policy-map type loadbalance first-match PM_ONE_ARM_LB
class class-default
sticky-serverfarm STICKYGROUP1
policy-map multi-match PM_ONE_ARM_MULTI_MATCH
class VS_LOGON_D2
loadbalance vip inservice
loadbalance policy PM_ONE_ARM_LB
nat dynamic 5 vlan 1240
class VS_LOGON2_D2
loadbalance vip inservice
loadbalance policy PM_ONE_ARM_LB
nat dynamic 6 vlan 1240
interface vlan 1240
description Client_server
ip address 10.163.22.11 255.255.255.0
peer ip address 10.163.22.12 255.255.255.0
access-group input INBOUND
nat-pool 5 10.163.22.14 10.163.22.17 netmask 255.255.255.192 pat
nat-pool 6 10.163.22.xx 10.163.22.xx netmask 255.255.255.192 pat
service-policy input PM_ONE_ARM_MULTI_MATCH
no shutdown
ip route 0.0.0.0 0.0.0.0 10.163.22.1
BR
GeirThanks for your reply.
Hope I understand you correct. This sould be the config I need to paste into the existing context.
rserver host cwi503
description content server logon 2
ip address 10.163.22.23
inservice
rserver host cwi504
description content server logon 2
ip address 10.163.22.24
inservice
serverfarm host SF_LOGON2_D2
probe WEBGUI_D2
rserver cwi503 80
inservice
rserver cwi504 80
inservice
sticky ip-netmask 255.255.255.255 address source STICKYGROUP2
timeout 20
replicate sticky
serverfarm SF_LOGON2_D2
class-map match-all VS_LOGON2_D2
3 match virtual-address 10.163.22.xx any
policy-map type loadbalance first-match PM_ONE_ARM_LB2
class class-default
sticky-serverfarm STICKYGROUP2
policy-map multi-match PM_ONE_ARM_MULTI_MATCH
class VS_LOGON2_D2
loadbalance vip inservice
loadbalance policy PM_ONE_ARM_LB2
nat dynamic 6 vlan 1240
interface vlan 1240
nat-pool 6 10.163.22.xx 10.163.22.xx netmask 255.255.255.192 pat
Br
Geir -
Simple SLB with the ACE Module
Hello,
i have some problems with a ACE module i am currently tesing.
I have a simple Serverfarm with two Servers.
But there seems to be some Problems with the Loadbalancing i not understand:
1) I use Round Robin, but the ACE seems to put me serval times to the same server. I notice this, because i have different content on both servers, also different URLs.
2) withz the show serverfarm statement the total connects do not increment.
switch/slb-c1# show serverfarm webfarm
serverfarm : webfarm, type: HOST
total rservers : 2
----------connections-----------
real weight state current total
---+---------------------+------+------------+----------+--------------------
rserver: web1
10.0.33.201:0 8 OPERATIONAL 0 0
rserver: web2
10.0.33.200:0 8 OPERATIONAL 0 0
switch/slb-c1# show service-policy L4_LB_VIP
Status : ACTIVE
Interface: vlan 300
service-policy: L4_LB_VIP
class: L4_VIP_CLASS
loadbalance:
L7 loadbalance policy: L7_SLB_POLICY
VIP Route Metric : 77
VIP Route Advertise : DISABLED
VIP ICMP Reply : ENABLED
VIP State: INSERVICE
curr conns : 0 , hit count : 15
dropped conns : 0
client pkt count : 10198 , client byte count: 420991
server pkt count : 23367 , server byte count: 34915173
I have attatched the Config.
Any Idea what is going on?what version do you have ?
I would recommend to run the very recent A1.4.
This is something that really should work.
Gilles. -
Hi,
I configured a new serverfarm with leastconns predictor for two servers on our ACE module Version A2(2.3). Probes (show probes XX detail) to the servers are successful and both servers are operational (show serverfarm APPLI detail) but connections are directed only to one server.
When I deactived the server which is receiving the connections (no inservice), the ACE start to direct connection to the second server.
There are several serverfarm, configured the same way, that are Loadbalancing traffic as correctly.
Here is a sample of my config
serverfarm host TEST_443
predictor leastconns
probe TEST_443_PROBE01
rserver TEST_RS01 443
inservice
rserver TEST_RS02 443
inservice
sticky http-cookie TEST_HTTPS TEST_443_STKY
cookie insert
timeout 720
replicate sticky
serverfarm TEST_443
probe http TEST_443_PROBE01
port 443
interval 20
passdetect interval 60
passdetect count 5
request method get url /test
expect status 302 302
connection term forced
policy-map type loadbalance first-match TEST_L7PLB_HTTPS
class class-default
sticky-serverfarm TEST_443_STKY_SF
insert-http X-Forwarded-Proto header-value "https"
insert-http X-Forwarded-For header-value "%is"
policy-map multi-match SLB-HTTP-POLICY
class TEST_L4VIP_HTTPS
loadbalance vip inservice
loadbalance policy TEST_L7PLB_HTTPS
loadbalance vip icmp-reply active
loadbalance vip advertise active
nat dynamic 1 vlan 202
appl-parameter http advanced-options PERSIST
ssl-proxy server TEST_SSL_PROXY_SERVER
PS : ACE uptime is 291days, could that impact ACE behavior ?
Thanks for any troubleshooting hintsLooking at this on my phone but it looks like you L7 policy is referencing a sticky server farm that does not exist.
ie TEST_443_STKY_SF is incorrect name for sticky
If that's not it. Then check that the first server actually has a number of conns on it when a new connection is established. Sometimes when both servers have 0 conns - new incoming conns will always go to the first server
Regards
Stephen
===============================
Free network configuration management software at www.rconfig.com
Sent from Cisco Technical Support iPhone App -
Rac resource status is unknown only vip and asm is running fine
i have just install oracle 10g clusterware and oracle software on RHEL 5
only asm and vip is working fine ..rest of the resource not working status is unknown
[oracle@rac1 bin]$ ./crs_stat -t
Name Type Target State Host
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE UNKNOWN rac1
ora.rac1.gsd application ONLINE UNKNOWN rac1
ora.rac1.ons application ONLINE UNKNOWN rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE UNKNOWN rac2
ora.rac2.gsd application ONLINE UNKNOWN rac2
ora.rac2.ons application ONLINE UNKNOWN rac2
ora.rac2.vip application ONLINE ONLINE rac2
i tried to start is manually but its return error like this
[oracle@rac1 bin]$ ./crs_stop ora.rac1.LISTENER_RAC1.lsnr -f
Attempting to stop `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
Stop of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded.
[oracle@rac1 bin]$ ./crs_start ora.rac1.LISTENER_RAC1.lsnr -f
Attempting to start `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
`ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
CRS-0215: Could not start resource 'ora.rac1.LISTENER_RAC1.lsnr'.
[oracle@rac1 bin]$ ./crs_start ora.rac1.LISTENER_RAC1.lsnr -f
CRS-1028: Dependency analysis failed because of:
'Resource in UNKNOWN state: ora.rac1.LISTENER_RAC1.lsnr'
CRS-0223: Resource 'ora.rac1.LISTENER_RAC1.lsnr' has placement error.
[oracle@rac1 bin]$ ./crs_start ora.rac2.LISTENER_RAC2.lsnr -f
CRS-1028: Dependency analysis failed because of:
'Resource in UNKNOWN state: ora.rac2.LISTENER_RAC2.lsnr'
CRS-0223: Resource 'ora.rac2.LISTENER_RAC2.lsnr' has placement error.
i reboot the system 3 times but problem is same plz help me to solve this problem.../opt/app/crs/log/rac1/alertrac1.log output..
[cssd(5441)]CRS-1605:CSSD voting file is online: /dev/raw/raw2. Details in /opt/app/crs/log/rac1/cssd/ocssd.log.
2012-12-19 05:17:02.561
[cssd(5441)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .
2012-12-19 05:17:03.998
[crsd(4718)]CRS-1012:The OCR service started on node rac1.
2012-12-19 05:17:04.028
[evmd(5327)]CRS-1401:EVMD started on node rac1.
2012-12-19 05:17:12.456
[crsd(4718)]CRS-1201:CRSD started on node rac1.
2012-12-19 05:17:23.668
[cssd(5441)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2012-12-19 07:23:46.211
[cssd(5216)]CRS-1605:CSSD voting file is online: /dev/raw/raw2. Details in /opt/app/crs/log/rac1/cssd/ocssd.log.
2012-12-19 07:23:49.399
[cssd(5216)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .
2012-12-19 07:23:50.458
[crsd(4709)]CRS-1012:The OCR service started on node rac1.
2012-12-19 07:23:50.490
[evmd(5098)]CRS-1401:EVMD started on node rac1.
2012-12-19 07:23:55.776
[crsd(4709)]CRS-1201:CRSD started on node rac1.
2012-12-19 07:25:00.583
[cssd(5216)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2012-12-20 00:09:11.199
[cssd(5286)]CRS-1605:CSSD voting file is online: /dev/raw/raw2. Details in /opt/app/crs/log/rac1/cssd/ocssd.log.
2012-12-20 00:09:14.907
[cssd(5286)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .
2012-12-20 00:09:16.446
[evmd(5128)]CRS-1401:EVMD started on node rac1.
2012-12-20 00:09:16.459
[crsd(4756)]CRS-1012:The OCR service started on node rac1.
2012-12-20 00:10:02.406
[crsd(4756)]CRS-1201:CRSD started on node rac1.
2012-12-20 00:10:39.220
[cssd(5286)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
/opt/app/crs/log/rac1/crsd/crsd.log output:-
2012-12-20 00:09:15.606: [ CRSD][7390912]0Daemon Version: 10.2.0.1.0 Active Version: 10.2.0.1.0
2012-12-20 00:09:15.606: [ CRSD][7390912]0Active Version and Software Version are same
2012-12-20 00:09:15.606: [ CRSMAIN][7390912]0Initializing OCR
2012-12-20 00:09:15.801: [ OCRRAW][7390912]proprioo: for disk 0 (/dev/raw/raw1), id match (1), my id set (1669906634,1028247821) total id sets (1), 1st set (1669906634,1028247821), 2nd set (0,0) my votes (2), total votes (2)
2012-12-20 00:09:16.264: [ OCRMAS][3065346960]th_master:12: I AM THE NEW OCR MASTER at incar 1. Node Number = 1
2012-12-20 00:09:16.310: [ OCRRAW][3065346960]proprioo: for disk 0 (/dev/raw/raw1), id match (1), my id set (1669906634,1028247821) total id sets (1), 1st set (1669906634,1028247821), 2nd set (0,0) my votes (2), total votes (2)
2012-12-20 00:09:16.524: [ CRSD][7390912]0ENV Logging level for Module: allcomp 0
2012-12-20 00:09:16.528: [ CRSD][7390912]0ENV Logging level for Module: default 0
2012-12-20 00:09:16.534: [ CRSD][7390912]0ENV Logging level for Module: COMMCRS 0
2012-12-20 00:09:16.536: [ CRSD][7390912]0ENV Logging level for Module: COMMNS 0
2012-12-20 00:09:16.549: [ CRSD][7390912]0ENV Logging level for Module: CRSUI 0
2012-12-20 00:09:16.556: [ CRSD][7390912]0ENV Logging level for Module: CRSCOMM 0
2012-12-20 00:09:16.559: [ CRSD][7390912]0ENV Logging level for Module: CRSRTI 0
2012-12-20 00:09:16.562: [ CRSD][7390912]0ENV Logging level for Module: CRSMAIN 0
2012-12-20 00:09:16.564: [ CRSD][7390912]0ENV Logging level for Module: CRSPLACE 0
2012-12-20 00:09:16.567: [ CRSD][7390912]0ENV Logging level for Module: CRSAPP 0
2012-12-20 00:09:16.570: [ CRSD][7390912]0ENV Logging level for Module: CRSRES 0
2012-12-20 00:09:16.573: [ CRSD][7390912]0ENV Logging level for Module: CRSOCR 0
2012-12-20 00:09:16.576: [ CRSD][7390912]0ENV Logging level for Module: CRSTIMER 0
2012-12-20 00:09:16.582: [ CRSD][7390912]0ENV Logging level for Module: CRSEVT 0
2012-12-20 00:09:16.586: [ CRSD][7390912]0ENV Logging level for Module: CRSD 0
2012-12-20 00:09:16.590: [ CRSD][7390912]0ENV Logging level for Module: CLUCLS 0
2012-12-20 00:09:16.593: [ CRSD][7390912]0ENV Logging level for Module: OCRRAW 0
2012-12-20 00:09:16.596: [ CRSD][7390912]0ENV Logging level for Module: OCROSD 0
2012-12-20 00:09:16.600: [ CRSD][7390912]0ENV Logging level for Module: CSSCLNT 0
2012-12-20 00:09:16.603: [ CRSD][7390912]0ENV Logging level for Module: OCRAPI 0
2012-12-20 00:09:16.606: [ CRSD][7390912]0ENV Logging level for Module: OCRUTL 0
2012-12-20 00:09:16.609: [ CRSD][7390912]0ENV Logging level for Module: OCRMSG 0
2012-12-20 00:09:16.613: [ CRSD][7390912]0ENV Logging level for Module: OCRCLI 0
2012-12-20 00:09:16.651: [ CRSD][7390912]0ENV Logging level for Module: OCRCAC 0
2012-12-20 00:09:16.671: [ CRSD][7390912]0ENV Logging level for Module: OCRSRV 0
2012-12-20 00:09:16.678: [ CRSD][7390912]0ENV Logging level for Module: OCRMAS 0
2012-12-20 00:09:16.678: [ CRSMAIN][7390912]0Filename is /opt/app/crs/crs/init/rac1.pid
2012-12-20 00:09:16.956: [ CRSMAIN][7390912]0Using Authorizer location: /opt/app/crs/crs/auth/
2012-12-20 00:09:17.080: [ CRSMAIN][7390912]0Initializing RTI
2012-12-20 00:09:17.085: [CRSTIMER][2845059984]0Timer Thread Starting.
[ clsdmt][2866039696]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=rac1DBG_CRSD))
2012-12-20 00:09:17.236: [ CRSRES][7390912]0Parameter SECURITY = 1, running in USER Mode
2012-12-20 00:09:17.236: [ CRSMAIN][7390912]0Initializing EVMMgr
2012-12-20 00:09:17.475: [ COMMCRS][2834570128]clsc_connect: (0xa4e7cc8) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth))
2012-12-20 00:09:18.437: [ COMMCRS][2834570128]clsc_connect: (0xa45b6b8) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth))
2012-12-20 00:09:18.888: [ COMMCRS][2834570128]clsc_connect: (0xa45af68) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth))
2012-12-20 00:09:19.575: [ COMMCRS][2834570128]clsc_connect: (0xa456f50) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth))
2012-12-20 00:09:20.029: [ COMMCRS][2834570128]clsc_connect: (0xa45b330) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth))
2012-12-20 00:09:47.675: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [172339896] retval lht [-27] Signal CV.
2012-12-20 00:09:55.947: [ CRSMAIN][7390912]0CRSD locked during state recovery, please wait.
2012-12-20 00:10:01.118: [ CRSMAIN][7390912]0CRSD recovered, unlocked.
2012-12-20 00:10:01.127: [ CRSMAIN][7390912]0QS socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=ora_crsqs))
2012-12-20 00:10:02.329: [ CRSMAIN][7390912]0CRSD UI socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=CRSD_UI_SOCKET))
2012-12-20 00:10:02.400: [ CRSMAIN][7390912]0E2E socket on: (ADDRESS=(PROTOCOL=tcp)(HOST=rac1-priv)(PORT=49896))
2012-12-20 00:10:02.401: [ CRSMAIN][7390912]0Starting Threads
2012-12-20 00:10:02.406: [ CRSMAIN][7390912]0CRS Daemon Started.
2012-12-20 00:10:09.239: [ CRSRES][2740161424]0startRunnable: setting CLI values
2012-12-20 00:10:09.612: [ CRSRES][2729671568]0startRunnable: setting CLI values
2012-12-20 00:10:10.089: [ CRSRES][2740161424]0Attempting to start `ora.rac1.vip` on member `rac1`
2012-12-20 00:10:10.147: [ CRSRES][2729671568]0Attempting to start `ora.rac2.vip` on member `rac1`
2012-12-20 00:10:25.883: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:25.895: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:25.907: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:25.929: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.009: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.036: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.077: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.099: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.112: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.124: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.138: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.156: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.181: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.197: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.213: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.225: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.239: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.253: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.265: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.275: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.288: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.299: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.312: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.324: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.335: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.351: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.366: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.379: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.389: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.400: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.414: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.426: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.438: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.449: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.460: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.473: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.486: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.500: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.513: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.523: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.537: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.551: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.563: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.574: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.587: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.607: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.620: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.636: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.650: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.662: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.680: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:26.694: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [176455408] retval lht [-27] Signal CV.
2012-12-20 00:10:40.640: [ CRSRES][2740161424]0Start of `ora.rac1.vip` on member `rac1` succeeded.
2012-12-20 00:10:42.653: [ CRSRES][2729671568]0Start of `ora.rac2.vip` on member `rac1` succeeded.
2012-12-20 00:10:44.290: [ CRSRES][2740161424]0startRunnable: setting CLI values
2012-12-20 00:10:44.600: [ CRSRES][2729671568]0StopResource: setting CLI values
2012-12-20 00:10:44.601: [ CRSRES][2740161424]0Attempting to start `ora.rac1.ASM1.asm` on member `rac1`
2012-12-20 00:10:44.669: [ CRSRES][2729671568]0Attempting to stop `ora.rac2.vip` on member `rac1`
2012-12-20 00:10:46.051: [ CRSRES][2729671568]0Stop of `ora.rac2.vip` on member `rac1` succeeded.
2012-12-20 00:10:46.154: [ COMMCRS][2696936336]clsc_send_msg: (0xa84ecb0) NS err (12571, 12560), transport (530, 111, 0)
2012-12-20 00:10:46.154: [ CRSCOMM][2729671568]0CLSC connection failed, ret = 9
2012-12-20 00:10:46.154: [ CRSEVT][2729671568]0invokepeer ret 200
2012-12-20 00:10:46.302: [ CRSRES][2729671568]0Remote start never sent to rac2: X_E2E_NotSent : Failed to connect to node: rac2
(File: caa_CmdRTI.cpp, line: 492
2012-12-20 00:10:46.303: [ CRSRES][2729671568][ALERT]0Remote start for `ora.rac2.vip` failed on member `rac2`
2012-12-20 00:10:46.446: [ CRSEVT][2740161424]0CAAMonitorHandler :: 0:Action Script /opt/app/oracle/product/db_1/bin/racgwrap(start) timed out for ora.rac1.ASM1.asm! (timeout=600)
2012-12-20 00:10:46.446: [ CRSAPP][2740161424]0StartResource error for ora.rac1.ASM1.asm error code = -2
2012-12-20 00:10:46.558: [ CRSRES][2729671568]0startRunnable: setting CLI values
2012-12-20 00:10:46.625: [ CRSEVT][2740161424]0CAAMonitorHandler :: 0:Action Script /opt/app/oracle/product/db_1/bin/racgwrap(stop) timed out for ora.rac1.ASM1.asm! (timeout=600)
2012-12-20 00:10:46.626: [ CRSAPP][2740161424]0StopResource error for ora.rac1.ASM1.asm error code = -2
2012-12-20 00:10:46.665: [ CRSRES][2729671568]0Attempting to start `ora.rac2.vip` on member `rac1`
2012-12-20 00:10:46.750: [ CRSRES][2740161424]0X_OP_StopResourceFailed : Stop Resource failed
(File: rti.cpp, line: 1698
2012-12-20 00:10:46.750: [ CRSRES][2740161424][ALERT]0`ora.rac1.ASM1.asm` on member `rac1` has experienced an unrecoverable failure.
2012-12-20 00:10:46.750: [ CRSRES][2740161424]0Human intervention required to resume its availability.
2012-12-20 00:10:46.938: [ CRSRES][2740161424]0startRunnable: setting CLI values
2012-12-20 00:10:46.978: [ CRSRES][2740161424]0Attempting to start `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
2012-12-20 00:10:47.541: [ CRSEVT][2740161424]0CAAMonitorHandler :: 0:Action Script /opt/app/oracle/product/db_1/bin/racgwrap(start) timed out for ora.rac1.LISTENER_RAC1.lsnr! (timeout=600)
2012-12-20 00:10:47.541: [ CRSAPP][2740161424]0StartResource error for ora.rac1.LISTENER_RAC1.lsnr error code = -2
2012-12-20 00:10:47.807: [ CRSEVT][2740161424]0CAAMonitorHandler :: 0:Action Script /opt/app/oracle/product/db_1/bin/racgwrap(stop) timed out for ora.rac1.LISTENER_RAC1.lsnr! (timeout=600)
2012-12-20 00:10:47.807: [ CRSAPP][2740161424]0StopResource error for ora.rac1.LISTENER_RAC1.lsnr error code = -2
2012-12-20 00:10:48.181: [ CRSRES][2740161424]0X_OP_StopResourceFailed : Stop Resource failed
(File: rti.cpp, line: 1698
2012-12-20 00:10:48.181: [ CRSRES][2740161424][ALERT]0`ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` has experienced an unrecoverable failure.
2012-12-20 00:10:48.181: [ CRSRES][2740161424]0Human intervention required to resume its availability.
2012-12-20 00:10:50.692: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [177094664] retval lht [-27] Signal CV.
2012-12-20 00:11:00.139: [ CRSRES][2729671568]0Start of `ora.rac2.vip` on member `rac1` succeeded.
2012-12-20 00:11:00.257: [ CRSRES][2729671568]0StopResource: setting CLI values
2012-12-20 00:11:00.269: [ CRSRES][2729671568]0Attempting to stop `ora.rac2.vip` on member `rac1`
2012-12-20 00:11:00.882: [ CRSRES][2729671568]0Stop of `ora.rac2.vip` on member `rac1` succeeded.
2012-12-20 00:11:00.938: [ CRSRES][2729671568]0Attempting to start `ora.rac2.vip` on member `rac2`
2012-12-20 00:11:23.937: [ CRSRES][2729671568]0Start of `ora.rac2.vip` on member `rac2` failed.
2012-12-20 00:11:24.051: [ CRSRES][2729671568]0startRunnable: setting CLI values
2012-12-20 00:11:24.067: [ CRSRES][2729671568]0Attempting to start `ora.rac2.vip` on member `rac1`
2012-12-20 00:11:36.257: [ OCRSRV][3044367248]th_select_handler: Failed to retrieve procctx from ht. constr = [177090520] retval lht [-27] Signal CV.
2012-12-20 00:11:43.061: [ CRSAPP][2729671568]0StartResource error for ora.rac2.vip error code = 1
2012-12-20 00:11:46.191: [ CRSAPP][2740161424]0CheckResource error for ora.rac1.vip error code = 1
2012-12-20 00:11:46.196: [ CRSRES][2740161424]0In stateChanged, ora.rac1.vip target is ONLINE
2012-12-20 00:11:46.197: [ CRSRES][2740161424]0ora.rac1.vip on rac1 went OFFLINE unexpectedly
2012-12-20 00:11:46.197: [ CRSRES][2740161424]0StopResource: setting CLI values
2012-12-20 00:11:46.211: [ CRSRES][2740161424]0Attempting to stop `ora.rac1.vip` on member `rac1`
2012-12-20 00:11:46.422: [ CRSRES][2729671568]0Start of `ora.rac2.vip` on member `rac1` failed.
2012-12-20 00:11:46.815: [ CRSRES][2686446480]0startRunnable: setting CLI values
2012-12-20 00:11:46.848: [ CRSRES][2729671568]0startRunnable: setting CLI values
2012-12-20 00:11:47.139: [ CRSRES][2686446480]0Attempting to start `ora.rac1.ons` on member `rac1`
2012-12-20 00:11:47.163: [ CRSRES][2729671568]0Attempting to start `ora.rac1.gsd` on member `rac1`
2012-12-20 00:11:47.266: [ CRSRES][2675956624]0Attempting to start `ora.rac2.gsd` on member `rac2`
2012-12-20 00:11:47.571: [ CRSRES][2665466768]0Attempting to start `ora.rac2.ons` on member `rac2`
2012-12-20 00:11:49.679: [ CRSEVT][2686446480]0CAAMonitorHandler :: 0:Action Script /opt/app/crs/bin/racgwrap(start) timed out for ora.rac1.ons! (timeout=600)
2012-12-20 00:11:49.680: [ CRSAPP][2686446480]0StartResource error for ora.rac1.ons error code = -2
2012-12-20 00:11:49.710: [ CRSEVT][2729671568]0CAAMonitorHandler :: 0:Action Script /opt/app/crs/bin/racgwrap(start) timed out for ora.rac1.gsd! (timeout=600)
2012-12-20 00:11:49.710: [ CRSAPP][2729671568]0StartResource error for ora.rac1.gsd error code = -2
2012-12-20 00:11:49.794: [ CRSEVT][2686446480]0CAAMonitorHandler :: 0:Action Script /opt/app/crs/bin/racgwrap(stop) timed out for ora.rac1.ons! (timeout=600)
2012-12-20 00:11:49.794: [ CRSAPP][2686446480]0StopResource error for ora.rac1.ons error code = -2
2012-12-20 00:11:49.813: [ CRSRES][2686446480]0X_OP_StopResourceFailed : Stop Resource failed
(File: rti.cpp, line: 1698
2012-12-20 00:11:49.813: [ CRSRES][2686446480][ALERT]0`ora.rac1.ons` on member `rac1` has experienced an unrecoverable failure.
2012-12-20 00:11:49.813: [ CRSRES][2686446480]0Human intervention required to resume its availability.
2012-12-20 00:11:49.839: [ CRSEVT][2729671568]0CAAMonitorHandler :: 0:Action Script /opt/app/crs/bin/racgwrap(stop) timed out for ora.rac1.gsd! (timeout=600)
2012-12-20 00:11:49.839: [ CRSAPP][2729671568]0StopResource error for ora.rac1.gsd error code = -2
2012-12-20 00:11:49.865: [ CRSRES][2729671568]0X_OP_StopResourceFailed : Stop Resource failed
(File: rti.cpp, line: 1698
2012-12-20 00:11:49.865: [ CRSRES][2729671568][ALERT]0`ora.rac1.gsd` on member `rac1` has experienced an unrecoverable failure.
2012-12-20 00:11:49.865: [ CRSRES][2729671568]0Human intervention required to resume its availability.
2012-12-20 00:11:50.076: [ CRSRES][2740161424]0Stop of `ora.rac1.vip` on member `rac1` succeeded.
2012-12-20 00:11:50.079: [ CRSRES][2740161424]0ora.rac1.vip RESTART_COUNT=0 RESTART_ATTEMPTS=0
2012-12-20 00:11:50.103: [ CRSRES][2740161424]0ora.rac1.vip failed on rac1 relocating.
2012-12-20 00:11:50.520: [ CRSRES][2740161424]0Attempting to start `ora.rac1.vip` on member `rac2`
2012-12-20 00:11:50.773: [ CRSRES][2675956624][ALERT]0`ora.rac2.gsd` on member `rac2` has experienced an unrecoverable failure.
2012-12-20 00:11:50.774: [ CRSRES][2675956624]0Human intervention required to resume its availability.
2012-12-20 00:11:50.810: [ CRSRES][2665466768][ALERT]0`ora.rac2.ons` on member `rac2` has experienced an unrecoverable failure.
2012-12-20 00:11:50.810: [ CRSRES][2665466768]0Human intervention required to resume its availability.
2012-12-20 00:12:13.625: [ CRSRES][2740161424]0Start of `ora.rac1.vip` on member `rac2` failed.
2012-12-20 00:12:13.994: [ CRSRES][2686446480]0startRunnable: setting CLI values
2012-12-20 00:12:26.925: [ COMMCRS][2824080272]clsc_receive: (0xa975368) Lock release 1 failed, rc 2
2012-12-20 00:12:26.925: [ COMMCRS][2824080272]clsc_receive: (0xa975368) error 2
2012-12-20 00:12:33.062: [ CRSAPP][2686446480]0StartResource error for ora.rac2.vip error code = 1
[root@rac1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
[root@rac1 bin]#
current output of crs_stat
[root@rac1 bin]# ./crs_stat -t
Name Type Target State Host
ora....SM1.asm application ONLINE UNKNOWN rac1
ora....C1.lsnr application ONLINE UNKNOWN rac1
ora.rac1.gsd application ONLINE UNKNOWN rac1
ora.rac1.ons application ONLINE UNKNOWN rac1
ora.rac1.vip application ONLINE OFFLINE
ora....SM2.asm application ONLINE OFFLINE
ora....C2.lsnr application ONLINE OFFLINE
ora.rac2.gsd application ONLINE UNKNOWN rac2
ora.rac2.ons application ONLINE UNKNOWN rac2
ora.rac2.vip application ONLINE OFFLINE -
Team,
Below is a snapshot of the HTTP Probe that I currently have confgured:
probe http http-probe
interval 10
passdetect interval 3
request method head
expect status 200 200
My question is, what if one of my reservers has a bad URL but the subsystem in IIS is responding with a 200 status? How do I protect myself from this situation and have the ACE module take this rserver out of rotation?
Thank you,
John...For example instead of root url you can probe a specific url pointing to the app like
probe http http-probe
interval 10
passdetect interval 3
request method head
request method get url /testpage.html
expect status 200 200
Where tespage.html is the app specific page.
There could also be situations where requirement would be to keep track of the backend server along with the front end/Web server and mark the Web server down if a backend server (like application /database server is down).
This can be achieved by if APP can be tweaked by developers such that it make calls to backend servers (like DB server) and populate a page with some value from the database. In http probe you can look for that value and if that value doesnt exist then you can mark the server down.
for e.g in following example if ACE will mark the server down if it gets 200 ok but doesnt get "DBISUP" in response
probe http http-probe
interval 10
passdetect interval 3
request method head
request method get url /checkdbpage.html
expect status 200 200
expect regex DBISUP
HTH
Syed Iftekhar Ahmed -
ACE module hung and required hard reset !!Plz help
ACE module had bit flip and it was hunged after that.I was not able to run any command(i.e For ex if i run show ft status nothing was displayed).I was not able to run any command on the standby ACE as well is this could be both the ACE module ACTIVE?
Manuaaly reboot from the ACE did not work. I had to forced hardare reset from cat 6500.
Is this a bug or strange behaviour?
I am running ACE A2(2.3) version on the module.
Thanks
ALEXUsually in the case of the bit flip the ace will reset itself, which clears the problem. In order to understand what is happining to your ACE, you would have to open a TAC case, and provide show tech information, as well as any files that were generated in the "core:" directory. You can view these using the command "dir core:"
It seems odd that the standby ACE also wouldn't respond to any command input. Did you have to reset it as well? If you had to reset it as well, then it may have encountered the same conditions that caused the hang on the primary.
Was there any syslog messages generated on the 6500 switch during the time? -
Unable to import PKCS12 certificate file to ACE module
Hi,
I'm currently in the process of replacing my CSS-appliances with the ACE module. So far everything's been smooth, but when I'm trying to import a certificate file to the respective context using the "crypto import"-command, ACE can't recognize the filetype, it's just marked as UNKNOWN. On the CSS I had to specify PKCS12 as the fileformat, but this is apparently not an options on the ACE. Does anyone know the equivalent commmand on how to import a PKCS12-file to the ACE?
Thankx
/Ulrich
PS! I haven't created a cert chaingroup, as I was told this would not be necessary.Hi Ulrich,
Short answer is you cannot import PKCS12 format. You'll need to extract the component parts into PEM format outside of the ACE and then use crypto import.
You will also need a chaingroup unless this is a self-signed certificate. Again any intermediate and root certificates will need to be in PEM format.
HTH
Cathy -
Voice Mail Device Status is Unknown
My Voice Mail Port: CiscoUM1-VI1 has a status of Unknown, with the IP registered as 10.0.1.2 (Unity)
I'm at a loss as to why its at this.If it's just 1 port and you've already tried resetting it countless times, chances are you'll need to reboot Unity to get it registered again.
HTH
java
If this helps, please rate
www.cisco.com/go/pdihelpdesk -
A problem with ACL in the class-map on the ACE module
Hi all,
I configured the following on the ACE module:
object-group network test
host 192.168.1.21
host 192.168.1.22
host 192.168.1.23
object-group service port
tcp eq www
tcp eq 8080
access-list T line 8 extended permit object-group port object-group test any
I tried to configure a class-map for matching this ACL:
ACE-4710-2/Lab-OPT-11(config)# class-map match-any TEST_C
ACE-4710-2/Lab-OPT-11(config-cmap)# match access-list T
Error: Cannot associate acl having object-group ACEs in class-map.
So couldn't I configure the class-map by using ACL with object-groups involved? Is it the bug or the normal behaviour? Because the customer uses object-groups in ACLs and he has to configure ACL without object-groups for the traffic classification. It is horrible.
Thank you
RomanHi Roman,
I'm afraid it's the expected behavior. You cannot use an ACL with object-groups inside a class-map.
Regards
Daniel -
How to Virtual IP configuration in ACE module?
Hi,
I am in the process of configuring load balancing on ACE module but struggling to configure virtual IP address for ACE module.
I'm working on ACE30 module and using software version A5 (1.2). ACE module is in slot of Catalyst 6504 switch.
Can anybody please post the steps/commands to perform this activity? An early response would be appreciated.
Regards,
Rachit.Hi Rachit,
Here is a basic configuration example:
access-list Allow_Access line 10 extended permit ip any any
rserver host test
ip address 10.198.16.98
inservice
rserver host test2
ip address 10.198.16.93
inservice
serverfarm host test
rserver test 80
inservice
rserver test2 80
inservice
sticky http-cookie test group2
cookie insert
serverfarm test
class-map match-all VIP
2 match virtual-address 10.198.16.122 tcp eq www
policy-map type loadbalance first-match test
class class-default
sticky-serverfarm group1
policy-map multi-match clients
class VIP
loadbalance vip inservice
loadbalance policy test
loadbalance vip icmp-reply active
nat dynamic 1 vlan 112
interface vlan 112
ip address 10.198.16.91 255.255.255.192
access-group input Allow_Access
nat-pool 1 10.198.16.122 10.198.16.122 netmask 255.255.255.192 pat
service-policy input NSS_MGMT
service-policy input clients
no shutdown
ip route 0.0.0.0 0.0.0.0 10.198.16.65
Here is the configuration guide:
http://tools.cisco.com/squish/101AD
Cesar R -
Has anyone upgraded the Ironport ESA to 8.5.6-074 and had the issues of Raid status showing unknown?
Has anyone upgraded the Ironport ESA to 8.5.6-074 and had the issues of Raid status showing unknown? After we upgraded our appliances we are having issues with our ESA appliances showing the RAID status as unknown. When we reported the issue to CISCO we were updated there were no issues reported at all. Could anyone please confirm if you have experienced the same issue.
You should see OPTIMAL - meaning the drives in the C170 are in good health/status:
myc680.local> version
Current Version
===============
UDI: C680 V FCH1611V0B2
Name: C680
Product: Cisco IronPort C680 Messaging Gateway(tm) Appliance
Model: C680
Version: 8.5.6-074
Build Date: 2014-07-21
Install Date: 2014-07-29 11:16:34
Serial #: xxx-yyy1611Vzzz
BIOS: C240M3.1.4.5.2.STBU
RAID: 3.220.75-2196, 5.38.00_4.12.05.00_0x05180000
RAID Status: Optimal
RAID Type: 10
BMC: 1.05
There are times post-reboot, that you'll see and get notification of RAID sub-optimal --- meaning that the appliance is running through on a health-check of the appliance's RAID. You should be getting a notification once RAID status has returned to OPTIMAL, or as per the older OS revisions, READY:
myc170.local> version
Current Version
===============
UDI: C170 V01 FCH1428V06A
Name: C170
Description: Cisco IronPort C170
Product: Cisco IronPort C170 Messaging Gateway(tm) Appliance
Model: C170
Version: 7.6.3-019
Build Date: 2013-06-09
Install Date: 2014-09-12 13:52:24
Serial #: xxxxxxD87B39-yyyyyy8V06A
BIOS: 9B1C115A
RAID: 02
RAID Status: READY
RAID Type: 1
BMC: 2.01
Maybe you are looking for
-
Solaris nis client can't connect to Windows 2008 NIS Server across subnets
I have been using Microsoft Windows Server for NIS for years to centralize some accounts accessing Solaris and Linux OS. Windows 2003 R2 Indentity Management for UNIX version of Server for NIS was last working version with Solairs. After upgrading Do
-
I have just reloaded and no difference
-
BIS Sync period with Google calendar
Hello to everybody, I need to sync by BIS service my Google calendar. How many old and future events can the BIS service sync? I have old events from 1999 and future events until 2012. Is it possible to sync this large period by BIS service? Thanks B
-
Automatically creation of column by creating it in another list
Hi, I have two list (list1 and list2) and what I want is if I create any column in list1 the same will create in list2 with same description and property. Kindly help me in this. thanks, Vidhu
-
I am unable to install ANY Adobe product on my Macintosh
I am unable to install ANY Adobe product on my Macintosh iMac running 10.9 Tried to update Adobe Reader and was unable to. Removed Reader and tried to reinstall, no luck. Now my copy of Adobe Acrobat Pro will not run or reinstall. I have spend over a