Design ? about SNMP operation in ACE module ... Traps sent to different Mgmt Stations

Good Day everyone,
I searched the site, and I could not find the answer I was looking for, so If anyone happens to know or point me to a link I would greatly appreciate it.
Topic:
Can ACE module sent different Traps (oid) to different management station? Split decision processing to send specific traffic to specific stations, based on the alert it has detected.
Scenario:
Our network equipments have a demarc point on what devices are managed via SNMP (Traps, syslog, EMS, etc...); Routers, Switches, ACE modules, and so forth.
However, we are not responsible for the App Servers assigned to various broadcast domains.
Customer would like to receive Notification from the ACE module when a Real Server is taken out of rotation , when specific probes have failed.
My team manages the ACE module, so any alerts from the ACE will be sent to the management station configured in our network.
Unfortunately I do not have a Test Lab to test my theory, so any help would be greatly appreciated before I submit my Production configs.
Design Requirements:
Customer would like the following traps generated and sent to their management station:
1) Real Server host name
2) TCP port
3) Real Server IP address
4) If capable, percentage threshold for each real server, based on the prediction configured for each Server Farm
5) Can a NetIQ agent be download on the ACE module to communicate with the NetIQ management station?
As always thank you for any help you can provide, and if you happen to be around Huntsville Alabama/USA.. you got a cold beer waiting for you!!!!
Cheers,
-raman

Gilles,
Thank you for your prompt answer.
When you have time please look over the following question and let me know if it is possible to implement, if the Proxy server is not an option?
Can a Custom TCL script be executed to sent an notification via SMPT if a health probe fails?
The SMTP message will contain the server info (IP address, Host name, TCP port).
The script procedure will execute certain actions based on the returned result.
Thanks,
raman
P.S
Sorry about not being up to speed on TCL. I am reading up on the TCL capability, and trying to provide some options to my customer.

Similar Messages

Design question: ACE module connected to 2 different L3 engine while in bridge mode

fellow engineers,
i have been working on a design model , where the ACE mldule will provide SLB for both virtual and real servers. we have been deploying several UCS systems and the customer would like to use the ACE as our Enterprise SLB layer
configured in bridcge mode.
the msfc within the 6509 provide the L3 routing. however we may extends multiple vlans (v160-v163) via nexus switch layer (7k,5k,2k) to a FW appliance which now is the svi interface for the extended vlans. these vlans will be configured on a dedicated context.
the extension is based on the bridge mode operation as follow:
need help with the following:
1) if i have 4 bvi's configured, do i need to have default route configured?
2) my total count for vlans are: v160-v163 for server vlans, and v101 is the management vlan. the svi for this vlan is on the msfc card. the server GW are pointing to each dedicated svi's on the FW+L3 apliance.
3) if my default route on the context is pointing to the v160 svi on the FW+L3 engine, will that prevent the return traffic for other vlans ( v161-v163) from the ace toward the client?
4) is default route neccessary if you hae the ace in bridge mode.
it was brought to my attention that if you have multiple vlans configured in bridge mode pointing to another L3 engine, then each vlan would have to be configured on seperate context since you can only have one default route per context.
i appreciate any feedback on this inquiry. if you need additional information please le me know.
thanks and best regards,
raman azizian

Hi Raman,
You can have up to eight default routes in one context. What the ACE is doing with the entries is to create a ARP-entry with the name GATEWAY. If you need more then eight entries, just declare gateway as rservers. In that case the ARP-entry is stored as RSERVER instead of GATEWAY. The trick is to tell ACE to learn the MAC-address for the IP-address and store it int the ARP-table. The ACE never learn for itself a MAC-address. Don't forget mac-sticky enable on vlan's facing gateway.
I'm running one context in bridge mode and have 18 bvi's with FW and Router 6509 as gateways.
Exampel:
Interface to ROUTER 6509
interface vlan 300
bridge-group 300
no normalization
mac-sticky enable
access-group input BPDU
access-group input alla
access-group output alla
service-policy input lb-int-vlan300
no shutdown
rserver host 300GATEWAY
ip address 164.135.121.47
inservice
A#1/prod1# sho arp | i 164.135.121.47
164.135.121.47 00.08.e3.ff.fc.14 vlan300   RSERVER    4775   239 sec      up
A#1/prod1#
Interface to FIREWALL
interface vlan 802
bridge-group 802
no normalization
mac-sticky enable
access-group input BPDU
access-group input alla
access-group output alla
service-policy input lb-int-vlan802
no shutdown
rserver host 802GATEWAY
ip address 192.168.137.1
inservice
192.168.137.1   00.23.33.6a.bf.80 vlan802   RSERVER    4785   5 sec        up
Regards
Mats

VMs change OID and fails SNMP probe on ACE module

I am setting up least loaded load-balancing on a serverfarm. It seems to work fine when all servers have the same OIDs, but the VMs change OIDs for the CPU utilization every time they vMotion or reboot, and that causes the SNMP probe to fail.
Is there any known solution to fix that problem?
Thank you,
Mark

Hi Mark,
If the OID's change itself on servers then this is expected. I am aware of anything that we can do on ACE to update the OID's automatically.
Regards,
Kanwal

ACE modules not syncing up

Hi,
I was adding logging and snmp to my ACE modules this weekend. I first made the changes to the primary ACE module and did a wr mem; I then went to my secondary module and noticed that the modules did not sync.
After some troubleshooting; I decided to reboot the secondary module, when the module came back, it was in sync.
As anyone run into this issue before? What is the command that will show me who is my primary module and the state of the modules?
I am running ACE code: A2.1.2
Regards,
John...

Thank you for your reply; I think that this was my problem:
14:1007 => Feb 01 07:57:27: ha_process_message:1818 Running sync info: mode 0, s
tatus 0, reason Detected license mismatch with peer, disabling running-config au
to sync
14:1008 => Feb 01 07:57:27: ha_process_message:1822 Startup sync info: mode 0, s
tatus 0, reason Detected license mismatch with peer, disabling running-config au
to sync
I first upgraded the license on my primary and made my changes, then tried to sync. The only problem I see here is that when I did the wr mem the module starting to sync and said that the sync process was complete.
John...

Configuring ACE Module for Redundancy

Hi Sir,
I'm configuring fault tolerance between two ACE modules installed on two different Catalyst 6513 switches. I have one Admin context and 3 user contexts.
Do I need to configure 4 "ft group", i.e. one context per group? E.g. config:
ft group 1
peer 1
priority 110
peer priority 105
associate-context Admin
inservice
ft group 2
peer 1
priority 110
peer priority 105
associate-context ace-context1
inservice
ft group 3
peer 1
priority 105
peer priority 110
associate-context ace-context2
inservice
ft group 4
peer 1
priority 105
peer priority 110
associate-context ace-context3
inservice
Can you also explain the purpose of configuring an alias IP address on the client-facing VLAN interface? I understand we need an alias IP address on the server-facing VLAN interface to provide a virtual gateway address to the servers. But what's the use of an alias IP on the client-side?
Thank you.
B.Rgds,
Lim TS

Hi Gilles,
I have configured FT for all user contexts as well as for the admin context. It works. My FT config is identical to the one I posted in this thread. Of course, one has to define the "ft interface vlan" and "ft peer" before configuring FT groups.
I noticed a few things:
(1) After the initial FT config, subsequent FT groups just need to be configured on the active Admin context and it will be replicated to the standby ACE, with the priority correctly reversed.
(2) You will get the message "NOTE: Configuration mode has been disabled on all sessions" when you log in to a standby context.
(3) The hostname of the active Admin context is not synced to the standby ACE. Do you know why?
One issue I encountered in one of the user contexts is as follows:
ace1/ace-context-1# sh run int
Generating configuration....
interface vlan 950
description *** Client-Facing VLAN ***
ip address 10.1.35.5 255.255.255.0
alias 10.1.35.4 255.255.255.0
peer ip address 10.1.35.6 255.255.255.0
access-group input ACL_VL950_IN
service-policy input REMOTE_MGMT
service-policy input MY_LB
no shutdown
interface vlan 951
description *** Connection to Real Servers ***
ip address 10.1.36.2 255.255.255.0
alias 10.1.36.1 255.255.255.0
peer ip address 10.1.36.3 255.255.255.0
access-group input ACL_VL951_IN
service-policy input NAT_REAL
no shutdown
This is the active context. It can ping to 10.1.35.4 (alias) and 10.1.35.6 (peer) over VLAN 950 (client-side). It can ping alias 10.1.36.1 over VLAN 951 (server-side) but can't ping to peer 10.1.36.3. The ACL_VL951_IN permits ip any any. Do you know why?
Secondly, I can remotely ping to alias 10.1.35.4 but can't telnet to it (I'm expecting it to telnet to the active context). I have to telnet to 10.1.35.5. Is this normal behavior?
Please advise.
Thank you.
B.Rgds,
Lim TS

ACE Module SNMP limits

I am monitoring an ACE module using snmp. The values returned from certain OIDs are graphed using Cacti. I found the 64 bit counters on interfaces for the ACE wrap at 10,000,000,000 instead of 2^64. Now that I have configured cacti to expect the wrap at 10 billion, I am concerned about the 32 bit counters. I am querying this snmp oid to get L7 connection counter
cslbxStatsL7PolicyConns
1.3.6.1.4.1.9.9.254.1.1.1.1.8
Should I expect this counter to wrap at 2^32 or a lower value?

The maximum value for a 32bit OID should be 4294967296, I do have a value in my lab that is above 1 billion for that counter, so I wouldn't think there is an issue immediately. One common issue - when you clear stats manually, the counter will reset to 0. As well, I found an internal bug that that suggested some pocket case within the code could have cleared stats incorrectly, but it has never been seen since. There is a guess that someone logged into the test bed and cleared it without permission, but it was not able to be verified. Hence the bug was created to investigate the code, turned up nothing, and was junked accordingly.
What you might want to do is keep a sharp eye on the counter. When it looks like it rolls, login to the context you are polling and take a look at the accounting log. If you find that someone cleared the logging, that answers the question. If not - log a TAC case and we can replicate your exact configuration/code version in our lab to see if there what the deviation is that causes it to clear. A bug would be logged and fixed.
Regards,
Chris Higgins

Monitoring the Cisco ACE module with SNMP

We use 2 redundant Cisco ACE loadbalancer in our datacenter
The models are ACE20-MOD-K9 with software A2(2.0)
Does anybod know how to monitor the environment (cpu, memory) of such a module with snmp?
We were not able to find an applicable MIB for that module.
The CISCO-PROCESS-MIB.oid (ftp://ftp.cisco.com/pub/mibs/oid/CISCO-PROCESS-MIB.oid) seems not to reflect the correct oid's.
What are the correct oid's for cpu and memory?
Where can I find a detailed documentation for snmp-monitoring the cisco ace module?
thanks

Hi Patrik,
to monitor the ACE I use these two MIB's:
ftp://ftp.cisco.com/pub/mibs/v2/CISCO-SLB-MIB.my
ftp://ftp.cisco.com/pub/mibs/v2/CISCO-ENHANCED-SLB-MIB.my
Example for CPU:
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Normale Tabelle";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-qformat:yes;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin:0cm;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:"Times New Roman";
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
cpmCPUTotalEntry 1.3.6.1.4.1.9.9.109.1.1.1.1
The resource usage and other interesting things you will find with a MIB browser.
Achim

Want to know about ACE module in 6509 : load-balancing concept

Hi,
I am quite new in this field , where i need to configure and understand the concept of load-balancing through ACE.
In my existing network set-up , i have some application servers as well as some other servers where i am looking for load-balancing.
I have gone through some of the site and cisco site as well and i came across ACE module which can be installed in 6509 switch.
I have 6509 switch as well but before going for installing the ACE module I am keen to understand below things:
1) what is difference between CSM or any other product load-balancer and ACE module :
Gone through site as well , but not getting proper answer or comparison.
1) I have some of the server configured with clustering and getting one virtual IP, In this case , will ACE work ?
2) If suppose i go for configuring different IP address with all server IP :
How do i achieve it ?
3) what is Virtual IP concept in ACE because i do not have and other ACE module then why do i need virtual IP ?
4) will the load-balancing happens based on destination based or session based ?
Please share the knowledge. It would be great help for me to go ahead with ACE and configure it and understand all the application ?

Hello,
1) what is difference between CSM or any other product load-balancer and ACE module :
There are several differences but to say simply, you get higher performance and more features with ACE module/appliance comparing others.
One big difference is that with ACE seriese, you can configure multiple contexts on one box (virtual load-balancers on one box) that makes us possible to provide a virtual load-balancer to a customer. In that way, the customer can access and makes changes on only the virtual box. You can split management domain for each customers. Also using contexts, you can assign certain resources available on the hardware for each contexts according to their service contract.
ACE serise has specific hardware chip for supporting SSL termination but some others do not.
For instance, you need a CSM-S, or a CSM and a SSL module to terminate SSL.
The other thing I should mention is that our most recent product is ACE serise that means it has longer product roadmap.
Let me try clarifying your other questions.
3) what is Virtual IP concept in ACE because i do not have and other ACE module then why do i need virtual IP ?
4) will the load-balancing happens based on destination based or session based ?
I think I'd better to put 3) and 4) first.
Virtual ip address (VIP) is the address to which client accesses.
VIP is tied with a serverfarm or serverfarms, in a serverfarm one or multiple rservers can be configured.
"serverfarm" is a group of "rservers".
"rserver" means real-server that has an ip address and processes transactions.
When a client accesses to the VIP, ACE picks up a rserver according to algorithm.
If you configure a VIP that is tied with a serverfarm where only one rsever is configured, client accesses to the virtual ip address are
all forwarded to the rserver.
If you configure a VIP that is tied with a serverfarm where multiple rsevers are configured, client accesses to the virtual ip address are
balanced among those rservers.
If you configure multiple VIPs, client accesses to those VIPs are forwareded to corresponding rservers according to configuration.
1) I have some of the server configured with clustering and getting one virtual IP, In this case , will ACE work ?
ACE load-balances connections to configured rservers.
If the clustered servers are sharing one virtual ip address and you configure the virtual ip address as a rserver, all connections are
sent to the virtual ip address. That is not "load-balancing" on ACE... You need multiple rservers to which ACE load-balances connections.
2) If suppose i go for configuring different IP address with all server IP :
How do i achieve it ?
You can configure those ip addresses as rserver ip address.
Multiple rservers are tied into a group, "serverfarm".
I'm not certain about your culstered servers but I guess you can configure each ip addresses in the culster as rservers.
Then put those rservers in a serverfarm.Client accesses to a virtual ip address configured on ACE for the serverfarm.
This way connections are load-balanced among those rservers depending on load-balancing algorithm you choose.
Above is just an overveiw. ACE gives you granular control not mentioned above.
I can provide more specific information if you tell me details of what you are trying to archive with ACE.
Regards,
Kimihito.

ACE Module Routed design

Hi all,
I have a requirement to install 2 ACE Modules into two 6509 chassis'
We want to run the ACE modules in a live/live scenario so we can utilise the two ACE modules
So we want to split the VIPS so we have some live on one ACE and others on the other.
Also the ACE modules will be setup in routed mode. We have a number of subnets we want to use on the client side - 3 to be exact, and there will be another 3 different subnets on the server side
A few points which are confusing me
For each subnet would i have to configure a SVI? And if so you can only have 1 SVI per contect so that would mean creating a context and a SVI for each subnet?
Are there any example configs which could help me out?
Any help would be appreciated
Thanks
James

See the config example here:
http://www.cisco.com/en/US/products/hw/modules/ps2706/products_configuration_example09186a00809c3048.shtml
Normally you only need one client-side subnet per context, but multiple ones work too.
You'd create an SVI on MSFC for the client-side subnets only, otherwise server traffic would bypass the ACE.
Also keep in mind when you do active/active, it's done on the context level.
That means you need to create at least two contexts in addition to the Admin context. (although you can technically run things in /Admin)
Go through the example above, and the config guides below and you'll be all set:
http://www.cisco.com/en/US/products/ps6906/tsd_products_support_model_home.html

ACE modules reloaded

HI Experts,
We had some issue with Datacentre ACE modules. Both primary and DR ACE modules got restarted in 16 hours difference.
Unfortunately Syslog was not configured on the ACE and local logging got cleared after restart.
The current IOS version is A2(3.2). The modules uptime was around 300 Days.
Here is the log from 6509 switch during the restart
Primary DC 6509-1 .
Jul 10 18:52:05.383 WAT: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks
.Jul 10 18:56:47.291 WAT: %SNMP-5-MODULETRAP: Module 9 [Down] Trap
Jul 10 18:56:47.127 WAT: %OIR-SP-3-PWRCYCLE: Card in module 9, is being power-cycled off (Reset - Module Reloaded During Download)
Jul 10 18:56:47.271 WAT: %C6KPWR-SP-4-DISABLED: power to module in slot 9 set off (Reset - Module Reloaded During Download)
Jul 10 18:57:00.951 WAT: %OIR-SP-3-PWRCYCLE: Card in module 9, is being power-cycled off (Module not responding to Keep Alive polling)
Jul 10 18:57:00.951 WAT: %C6KPWR-SP-4-DISABLED: power to module in slot 9 set off (Module not responding to Keep Alive polling)
Jul 10 19:01:57.172 WAT: %DIAG-SP-6-RUN_MINIMUM: Module 9: Running Minimal Diagnostics...
.Jul 10 19:01:59.256 WAT: %SNMP-5-MODULETRAP: Module 9 [Up] Trap
Jul 10 19:01:58.700 WAT: %DIAG-SP-6-DIAG_OK: Module 9: Passed Online Diagnostics
Jul 10 19:01:59.256 WAT: %OIR-SP-6-INSCARD: Card inserted in slot 9, interfaces are now online
.Jul 10 19:02:04.548 WAT: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks
DR DC 6509-1 .
Jul 11 09:42:05.759: %LINK-5-CHANGED: Interface TenGigabitEthernet9/1, changed state to administratively down .
Jul 11 09:42:05.763: %SNMP-5-MODULETRAP: Module 9 [Down] Trap
.Jul 11 09:42:05.763: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet9/1, changed state to down
Jul 11 09:42:05.599: %OIR-SP-3-PWRCYCLE: Card in module 9, is being power-cycled off (Reset - Module Reloaded During Download)
Jul 11 09:42:05.747: %C6KPWR-SP-4-DISABLED: power to module in slot 9 set off (Reset - Module Reloaded During Download)
Jul 11 09:42:05.767: %LINK-SP-5-CHANGED: Interface TenGigabitEthernet9/1, changed state to administratively down
Jul 11 09:42:05.771: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface TenGigabitEthernet9/1, changed state to down .
Jul 11 09:42:14.535: %SVCLC-5-SVCLCNTP: Could not update clock on the module 9, rc is -1
Jul 11 09:42:19.395: %OIR-SP-3-PWRCYCLE: Card in module 9, is being power-cycled off (Module not responding to Keep Alive polling)
Jul 11 09:42:19.395: %C6KPWR-SP-4-DISABLED: power to module in slot 9 set off (Module not responding to Keep Alive polling)
Jul 11 09:47:15.819: %DIAG-SP-6-RUN_MINIMUM: Module 9: Running Minimal Diagnostics... .
Jul 11 09:47:19.871: %MLS_RATE-4-DISABLING: The global switching mode is now 'truncated'. Disabling the Layer2 Rate Limiters. .
Jul 11 09:47:19.903: %SNMP-5-MODULETRAP: Module 9 [Up] Trap Jul 11 09:47:19.633: %DIAG-SP-6-DIAG_OK: Module 9: Passed Online Diagnostics Jul 11 09:47:19.905: %OIR-SP-6-INSCARD: Card inserted in slot 9, interfaces are now online .
Jul 11 09:47:21.079: %LINK-5-CHANGED: Interface TenGigabitEthernet9/1, changed state to administratively down
Jul 11 09:47:20.912: %LINK-SP-3-UPDOWN: Interface TenGigabitEthernet9/1, changed state to down
Jul 11 09:47:21.080: %LINK-SP-5-CHANGED: Interface TenGigabitEthernet9/1, changed state to administratively down
.Jul 11 09:47:25.039: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks
.Jul 11 09:47:25.047: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet9/1, changed state to up
Jul 11 09:47:24.520: %LINK-SP-3-UPDOWN: Interface TenGigabitEthernet9/1, changed state to down
Jul 11 09:47:25.056: %LINK-SP-3-UPDOWN: Interface TenGigabitEthernet9/1, changed state to up
Jul 11 09:47:25.060: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface TenGigabitEthernet9/1, changed state to up
Please let me did anyone face this issue before or is it any known BUG?

HI All, Thanx for the help. Got the resaon from show version output.
last boot reason: NP 1 Failed : SRAM Parity Error Chan 3
Also got the TAC comment on SRAM party error
The SRAM parity error presented in the core file is not due to a software issue.
The issue is the result of a "bit-flip" within the SRAM itself which can occur as a
result of environmental conditions. This "bit-flip" is rectified by a simple reboot of
the system, which would occur with the generation of the core file. Cisco internal
testing and customer experience has shown that these types of issues can occur
with very low frequency, but do not required an RMA of the device.
If there are multiple instances of this issue on the same module, a proactive RMA/EFA
of the device would be in order.
ACE is susceptible to this because of the way it uses SRAM to store control information
and packet data as opposed to scratch-pad storage. Almost any 1-bit flip will be detected as a
parity error. Cisco has recognized the issue and is taking action to ensure this will not be
an issue on the next generation of the ACE module. The next generation module design
and timeline is currently under review.
Thnx again for the help
Aslam

Do i have a dead ACE module?

I rebooted one of my HA ACE modules and it hasn't come back up. The logs on the 6500 show the following..
Mar 23 08:54:25: %DIAG-SP-6-RUN_COMPLETE: Module 4: Running Complete Diagnostics...
Mar 23 08:54:28: %SVCLC-5-SVCLCVTPMODE: VTP mode is set to non-transparent
Mar 23 08:54:28: %SNMP-5-MODULETRAP: Module 4 [Up] Trap
Mar 23 08:54:27: %DIAG-SP-6-DIAG_OK: Module 4: Passed Online Diagnostics
Mar 23 08:54:28: %OIR-SP-6-INSCARD: Card inserted in slot 4, interfaces are now online
Mar 23 08:54:43: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 08:55:18: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks
Mar 23 08:57:30: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 09:07:23: %SNMP-5-MODULETRAP: Module 4 [Down] Trap
Mar 23 09:07:23: SP: The PC in slot 4 is shutting down. Please wait ...
Mar 23 09:07:56: SP: PC shutdown completed for module 4
Mar 23 09:08:06: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set off (Fabric channel errors)
Mar 23 09:15:48: %DIAG-SP-6-RUN_COMPLETE: Module 4: Running Complete Diagnostics...
Mar 23 09:15:50: %DIAG-SP-6-DIAG_OK: Module 4: Passed Online Diagnostics
Mar 23 09:15:51: %SVCLC-5-SVCLCVTPMODE: VTP mode is set to non-transparent
Mar 23 09:15:51: %SNMP-5-MODULETRAP: Module 4 [Up] Trap
Mar 23 09:15:51: %OIR-SP-6-INSCARD: Card inserted in slot 4, interfaces are now online
Mar 23 09:16:06: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 09:16:41: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks
Mar 23 09:17:45: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 09:28:00: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 09:28:46: %SNMP-5-MODULETRAP: Module 4 [Down] Trap
Mar 23 09:28:46: SP: The PC in slot 4 is shutting down. Please wait ...
Mar 23 09:29:19: SP: PC shutdown completed for module 4
Mar 23 09:29:29: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set off (Fabric channel errors)
Mar 23 09:37:11: %DIAG-SP-6-RUN_COMPLETE: Module 4: Running Complete Diagnostics...
Mar 23 09:37:13: %SVCLC-5-SVCLCVTPMODE: VTP mode is set to non-transparent
Mar 23 09:37:13: %SNMP-5-MODULETRAP: Module 4 [Up] Trap
Mar 23 09:37:12: %DIAG-SP-6-DIAG_OK: Module 4: Passed Online Diagnostics
Mar 23 09:37:13: %OIR-SP-6-INSCARD: Card inserted in slot 4, interfaces are now online
Mar 23 09:37:28: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 09:38:03: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks
Mar 23 09:38:15: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
The output of the ACE console is the following....
System Bootstrap, Version 12.2[123],
Copyright (c) 1994-2009 by cisco Systems, Inc.
Slot 4 : Running DEFAULT rommon image ...
.ACE platform with 1048576 Kbytes of main memory
.Loading disk0:c6ace-t1k9-mz.A2_3_4.bin. Please wait ....
Uncompressing Linux...
Starting the kernel...
INIT: version 2.78 booting
Mounting Second Ramdisk ....
Second Ramdisk successfully mounted
Configuring network interfaces.
CF dump: Register callback functions
dosfsck 2.11, 12 Mar 2005, FAT32, LFN
/dev/cf: 11 files, 26575/63414 clusters
FAT FS is ok
Compact Flash size 1014624(in 1k blocks) ...
Core file size 204800
Available free size in cf is 589424 (in 1k blocks) ...
set_coredump 2.11, 12 Mar 2005, FAT32, LFN
first_cluster = 0x5102 num_cluster = 0x40 (64)
inserting procfs
inserting isan_kthread
inserting wiremod
inserting klib
inserting resdrv
inserting tlv
inserting sse
inserting kpss
inserting sdwrap
creating sdwrap device
inserting klm_tl
creating tl device
inserting klm_scp
inserting klm_mts
creating mts0 device
creating mtscfg0 device
inserting utaker
creating utaker0 device
creating utaker1 device
inserting sysmgr-hb
creating sysmgr-hb device
inserting modlock
creating modlock device
inserting bufmgr
inserting pkt_fifo
inserting encdec
creating encdec device
inserting pseudo
inserting drammap mod
creating drammap device
inserting ixp_dnld
creating ixp_dnld device
inserting sysdrv
creating sysdrv device
New registry installed.
INIT: Entering runlevel: 3
inserting i2c module
inserting ssa driver
inserting cde driver
inserting bf_dnld driver
inserting pfm_drv driver
inserting regaccess driver
inserting bf_nvram driver
Firmware compiled 21-Jan-11 13:14 by integ Build [25600]
ACE Daughter boards DB1 not present DB2 not present.
downloading fpga to cde 1
Read 3262454 bytes from ./cde1_core.bit
FPGA Date: 2007/12/18 Time: 14:22: 0
CDE 1 download successful
downloading fpga to cde 2
Read 2377744 bytes from ./cde2_core.bit
FPGA Date: 2007/ 8/15 Time: 20:59:47
CDE 2 download successful
FPGA Programming Done
CDE 1 revision ID 0403
CDE 2 revision ID 0402
enabling cde 0 interrupts
finished CDE setup
Configuring NP 1 Memory
Configuring NP 2 Memory
Waiting for NP 1 SRAM memory to clear...success
Downloading NP 1 Image
Waiting for NP 2 SRAM memory to clear...success
Downloading NP 2 Image
..... 0x4eef60 (5173088) bytes downloaded
..... 0x4eef60 (5173088) bytes downloaded
Loading Nitrox driver.
PCI device 177d:0002
Writing register at address 3838 with e00
size = 8108
Ctx memory range(0x0000000-0x10000000)
Cleared 262144 1024-byte blocks in 5 requests.
N2SetupMicrocode: failed; error code 3
Writing register at address 3898 with 1
N2LoadMicrocode: failed; error code 3
N2LoadMicrocode: failed; error code 3

Hello Akhtar,
Can you upload the command: #show version?
Can you upload the dir core: , hopefully the ACE might have generated some core dumps which might help us to determine the failure?
Here you have a link about getting the core dumps:
http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_%28ACE%29_Troubleshooting_Guide_--_Overview_of_ACE_Troubleshooting#Copying_Core_Dumps
Have you experienced this issue before? Did you experience this issue during a high peak of traffic?
Did you apply any change in the configuration?
#show tech-support and core dumps would help to determine if this was a hardware failure or a software defect
Jorge

ACE-module Restarted

Hi
One of my ACE module got restarted The following are the error messsages in the 6500 switches
Oct 22 13:38:40.411: %OIR-SP-3-PWRCYCLE: Card in module 9, is being power-cycled off (Module not responding to Keep Alive polling)
Oct 22 13:38:40.439: %C6KPWR-SP-4-DISABLED: power to module in slot 9 set off (Module not responding to Keep Alive polling)
The IOs version of the ACE is :- disk0:c6ace-t1k9-mz.3.0.0_A1_3b.bin
Switch Os version is : s72033-advipservicesk9_wan-mz.122-18.SXF7.bin
Could anybody tell me Is there any BUG in the IOS ?Or What could be the possible reason ?
Thanks in Advance
Dinesh

i have similar problem. the catalyst restart the ace, and ace doesn't work.
Also i have two catalyst conected in trunk, and i have two ace, one in each catalyst. No redundancie. both have the same problem.
ACE ios is: boot system image:c6ace-t1k9-mz.A2_1.bin
catalyst ios is: s72033-ipservicesk9_wan-mz.122-33.SXH2a.bin
log from catalyst:
17w2d: %CONST_DIAG-SP-6-HM_TEST_SP_INFO: TestAsicSync[1]: last_busy_percent[6%], Tx_Rate[3292], Rx_Rate[232]
17w2d: %CONST_DIAG-SP-2-HM_MOD_RESET: Resetting Module 1 for software recovery, Reason: Failed TestAsicSync
17w2d: %OIR-SP-3-PWRCYCLE: Card in module 1, is being power-cycled off (Diagnostic Failure)
17w2d: %HA_EM-6-LOG: Mandatory.go_asicsync.tcl: GOLD EEM TCL policy for TestAsicSync
17w2d: %SNMP-5-MODULETRAP: Module 1 [Down] Trap
17w2d: %C6KPWR-SP-4-DISABLED: power to module in slot 1 set off (Diagnostic Failure)
17w2d: %SVCLC-5-SVCLCVTPMODE: VTP mode is set to non-transparent
17w2d: %SNMP-5-MODULETRAP: Module 1 [Up] Trap
17w2d: %DIAG-SP-6-RUN_MINIMUM: Module 1: Running Minimal Diagnostics...
17w2d: %DIAG-SP-6-DIAG_OK: Module 1: Passed Online Diagnostics
17w2d: %OIR-SP-6-INSCARD: Card inserted in slot 1, interfaces are now online
17w2d: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks
17w2d: %FABRIC-SP-6-TIMEOUT_ERR: Fabric in slot 5 detected excessive flow-control on channel 0 (Module 1, fabric connection 0)
17w2d: %CONST_DIAG-SP-6-HM_TEST_SP_INFO: TestAsicSync[1]: last_busy_percent[6%], Tx_Rate[6293], Rx_Rate[298]
ANY advise?

Reuse of context in ACE module

Hi all, just have a question about som reuse of resources in a ACE module context. I don't want to make a new context, and can reuse most of the existing configuration in one of my context. The config is not complex and difficult, but I'm not sure if I can do this.
The primary goal is to loadbalance 2 webservers with a new vip, new serverfarm, stickygroup, policy-map and different nat-pool.
Since I haven't decided the ip addresses to be used, they are just xx in the config below.
The changes I want to implement are in bold. Will this work for me?
probe http WEBGUI_D2
description Probe for http mot webgui
interval 10
passdetect interval 10
passdetect count 1
request method get url /D2/auth/login.aspx
expect status 200 302
header User-Agent header-value "IDENTITY"
rserver host cwi003
description content server logon
ip address 10.163.22.27
inservice
rserver host cwi004
description content server logon
ip address 10.163.22.28
inservice
rserver host cwi503
description content server logon 2
ip address 10.163.22.23
inservice
rserver host cwi504
description content server logon 2
ip address 10.163.22.24
inservice
serverfarm host SF_LOGON_D2
probe WEBGUI_D2
rserver cwi003 80
   inservice
rserver cwi004 80
   inservice
serverfarm host SF_LOGON2_D2
probe WEBGUI_D2
rserver cwi503 80
   inservice
rserver cwi504 80
   inservice
sticky ip-netmask 255.255.255.255 address source STICKYGROUP1
timeout 20
replicate sticky
serverfarm SF_LOGON_D2
serverfarm SF_LOGON2_D2
class-map match-all VS_LOGON_D2
3 match virtual-address 10.163.22.13 any
class-map match-all VS_LOGON2_D2
3 match virtual-address 10.163.22.xx any
policy-map type loadbalance first-match PM_ONE_ARM_LB
class class-default
   sticky-serverfarm STICKYGROUP1
policy-map multi-match PM_ONE_ARM_MULTI_MATCH
class VS_LOGON_D2
   loadbalance vip inservice
   loadbalance policy PM_ONE_ARM_LB
   nat dynamic 5 vlan 1240
class VS_LOGON2_D2
   loadbalance vip inservice
   loadbalance policy PM_ONE_ARM_LB
   nat dynamic 6 vlan 1240
interface vlan 1240
description Client_server
ip address 10.163.22.11 255.255.255.0
peer ip address 10.163.22.12 255.255.255.0
access-group input INBOUND
nat-pool 5 10.163.22.14 10.163.22.17 netmask 255.255.255.192 pat
nat-pool 6 10.163.22.xx 10.163.22.xx netmask 255.255.255.192 pat
service-policy input PM_ONE_ARM_MULTI_MATCH
no shutdown
ip route 0.0.0.0 0.0.0.0 10.163.22.1
BR
Geir

Thanks for your reply.
Hope I understand you correct. This sould be the config I need to paste into the existing context.
rserver host cwi503
description content server logon 2
ip address 10.163.22.23
inservice
rserver host cwi504
description content server logon 2
ip address 10.163.22.24
inservice
serverfarm host SF_LOGON2_D2
probe WEBGUI_D2
rserver cwi503 80
    inservice
rserver cwi504 80
    inservice
sticky ip-netmask 255.255.255.255 address source STICKYGROUP2
   timeout 20
   replicate sticky
   serverfarm SF_LOGON2_D2
class-map match-all VS_LOGON2_D2
   3 match virtual-address 10.163.22.xx any
policy-map type loadbalance first-match PM_ONE_ARM_LB2
class class-default
    sticky-serverfarm STICKYGROUP2
policy-map multi-match PM_ONE_ARM_MULTI_MATCH
class VS_LOGON2_D2
    loadbalance vip inservice
    loadbalance policy PM_ONE_ARM_LB2
    nat dynamic 6 vlan 1240
interface vlan 1240
nat-pool 6 10.163.22.xx 10.163.22.xx netmask 255.255.255.192 pat
Br
Geir

Simple SLB with the ACE Module

Hello,
i have some problems with a ACE module i am currently tesing.
I have a simple Serverfarm with two Servers.
But there seems to be some Problems with the Loadbalancing i not understand:
1) I use Round Robin, but the ACE seems to put me serval times to the same server. I notice this, because i have different content on both servers, also different URLs.
2) withz the show serverfarm statement the total connects do not increment.
switch/slb-c1# show serverfarm webfarm
serverfarm : webfarm, type: HOST
total rservers : 2
----------connections-----------
real weight state current total
---+---------------------+------+------------+----------+--------------------
rserver: web1
10.0.33.201:0 8 OPERATIONAL 0 0
rserver: web2
10.0.33.200:0 8 OPERATIONAL 0 0
switch/slb-c1# show service-policy L4_LB_VIP
Status : ACTIVE
Interface: vlan 300
service-policy: L4_LB_VIP
class: L4_VIP_CLASS
loadbalance:
L7 loadbalance policy: L7_SLB_POLICY
VIP Route Metric : 77
VIP Route Advertise : DISABLED
VIP ICMP Reply : ENABLED
VIP State: INSERVICE
curr conns : 0 , hit count : 15
dropped conns : 0
client pkt count : 10198 , client byte count: 420991
server pkt count : 23367 , server byte count: 34915173
I have attatched the Config.
Any Idea what is going on?

what version do you have ?
I would recommend to run the very recent A1.4.
This is something that really should work.
Gilles.

Bizarre ACE module behavior

Hi,
I configured a new serverfarm with leastconns predictor for two servers on our ACE module Version A2(2.3). Probes (show probes XX detail) to the servers are successful and both servers are operational (show serverfarm APPLI detail) but connections are directed only to one server.
When I deactived the server which is receiving the connections (no inservice), the ACE start to direct connection to the second server.
There are several serverfarm, configured the same way, that are Loadbalancing traffic as correctly.
Here is a sample of my config
serverfarm host TEST_443
predictor leastconns
probe TEST_443_PROBE01
rserver TEST_RS01 443
    inservice
rserver TEST_RS02 443
    inservice
sticky http-cookie TEST_HTTPS TEST_443_STKY
cookie insert
timeout 720
replicate sticky
serverfarm TEST_443
probe http TEST_443_PROBE01
port 443
interval 20
passdetect interval 60
passdetect count 5
request method get url /test
expect status 302 302
connection term forced
policy-map type loadbalance first-match TEST_L7PLB_HTTPS
class class-default
    sticky-serverfarm TEST_443_STKY_SF
    insert-http X-Forwarded-Proto header-value "https"
    insert-http X-Forwarded-For header-value "%is"
policy-map multi-match SLB-HTTP-POLICY
class TEST_L4VIP_HTTPS
    loadbalance vip inservice
    loadbalance policy TEST_L7PLB_HTTPS
    loadbalance vip icmp-reply active
    loadbalance vip advertise active
    nat dynamic 1 vlan 202
    appl-parameter http advanced-options PERSIST
    ssl-proxy server TEST_SSL_PROXY_SERVER
PS : ACE uptime is 291days, could that impact ACE behavior ?
Thanks for any troubleshooting hints

Looking at this on my phone but it looks like you L7 policy is referencing a sticky server farm that does not exist.
ie TEST_443_STKY_SF is incorrect name for sticky
If that's not it. Then check that the first server actually has a number of conns on it when a new connection is established. Sometimes when both servers have 0 conns - new incoming conns will always go to the first server
Regards
Stephen
===============================
Free network configuration management software at www.rconfig.com
Sent from Cisco Technical Support iPhone App

Design ? about SNMP operation in ACE module ... Traps sent to different Mgmt Stations

Similar Messages

Maybe you are looking for