High CPU utilization on CSS 11501 version sg0750303

Question

Hi everyone,
I have the problem about High CPU utilization on CSS 11501 version sg0750303.
Our customer has used one pair of CSS 11501 (active-standby).
As a matter of convenience, called "Old CSS" after here in this post.
However traffic via Old CSS had been increasing so customer decided to add one more
pair (active-standby) of CSS to separate traffic.
Yesterday we installed new two CSS 11501 version sg0750303 (active-standby).
As a matter of convenience, called "New CSS" after here in this post.
Today, active CSS 11501 and standby CSS 11501 which were installed yesterday (New CSSs)
indicates High CPU utilization.
Active CSS 11501:
Peak CPU utilization: about 85%
Average CPU Utilization: about 60%
Standby CSS 11501:
Peak CPU utilization: about 40%
Average CPU Utilization: unknown
I do not understand why CPU utilization of both New CSSs become high.
The traffic pass through New CSS less than Old CSS, because the traffic is separated into
Old CSS and New CSS.
And CSS's configuration parameters (service, content, access-list) also less than Old CSS,
because real servers are also separated into Old CSS and New CSS.
Old CSS indicated average of CPU utilization about 20% before installing New CSSs yesterday,
in spite of all traffic pass through Old CSS only.
I wrote "New CSS remains High CPU utilization", however end users do not feel the
performance issue (e.g., performance delay, communication failure and so on) and
the traffic pass through New CSS normally.
So I have the question "CSS 11501 sg0750303 remains High CPU utilization on normal situation ?"
And customer uses MTRG to poll SNMP for Old CSSs and New CSSs.
So I have the question "CSS 11501 sg0750303 become High CPU utilization in case of receiving
SNMP polling ?".
Or if this situation is abnormal we need to start investigation.
Would you please let me know how do we investigate this situation.
I found the DDTS CSCek57080 "Performance issue using arrowpoint-cookie with ASR".
Release note of this DDTS says that
A customer was using a CSS pair configuration where arrowpoint-cookie
is being used along with a redundant-index on many content rules. When
the flow rate increased to a few hundred flows/sec, the peer message
queue of the CSS receiving ASR related message began to fill up.
When the peer message queue became over subscribed, the CPU increased
and the CSS became unstable.
New CSSs have configured redunrant-index on two content rules, and end users do not feel the
performance issue (e.g., performance delay, communication failure and so on) and
the traffic pass through New CSS normally.
So I think this DDTS does not related to this case.
Your information would be greatly appreciated.
Best regards,

Answer 1

Gilles,
Thank you very much for your cooperation.
I got the capture you instructed us.
The following are additional information from our customer.
At time user traffic path through the active CSS, active CSS indicates;
CPU utilization always range of 30% - 40%
Peak CPU utilization about 60% - 80%
At time there is no user traffic pass through active CSS, active CSS indicates;
CPU utilization always range of 0% - 5%
Attached files are named "Active CSS.log" and "Standby CSS.log".
"Active CSS.log" is captured on active CSS and "Standby CSS.log" is captured on
standby CSS.
I found the following process is using resource by looking the output of
"shell 1 1 spyReport" command.
On active CSS,
tFlowMgrPktR 8ba24070 50 26% ( 1469) 20% ( 26)
On standby CSS,
fmPeerMsgTas 8a511510 50 16% ( 176) 10% ( 7)
Your comment would be greatly appreciated.
Best regards,