VPN HW-1-PACKET ERROR

Question

Hi all,
my customer is seen lots of errors shown below
%VPN_HW-1-PACKET_ERROR: slot: 2 Packet Encryption/Decryption error, Output Authentication error:srcadr=y.y.y.y,dstadr=x.x.x.x,size=1392,handle=0x5BA7
%VPN_HW-1-PACKET_ERROR: slot: 2 Packet Encryption/Decryption error, Output Authentication error:srcadr=y.y.y.y,dstadr==x.x.x.x,size=1392,handle=0x5BA7
%VPN_HW-1-PACKET_ERROR: slot: 2 Packet Encryption/Decryption error, Output Authentication error:srcadr=y.y.y.y,dstadr==x.x.x.x,size=1392,handle=0x5BA7
%VPN_HW-1-PACKET_ERROR: slot: 2 Packet Encryption/Decryption error, Output Authentication error:srcadr=y.y.y.y,dstadr=x.x.x.x,size=1392,handle=0x5AB0
%VPN_HW-1-PACKET_ERROR: slot: 2 Packet Encryption/Decryption error, ESP Pad Length:srcadr=y.y.y.y,dstadr=x.x.x.x,size=96,handle=0x5AB0
I have looked in to the show commands and could not see anything which points me to an issue
===============================================
router#show crypto eli
Hardware Encryption Layer :   ACTIVE
Number of crypto engines = 1 .
CryptoEngine-0 (slot-2) details.
Capability-IPSec : IPPCP, 3DES, AES, RSA
IKE-Session   :   229 active,  5120 max, 0 failed
DH-Key        :    12 active,  5120 max, 0 failed
IPSec-Session :  1034 active, 10230 max, 0 failed
router#Show pas vam interf
VPN Acceleration Module Version II+ in slot : 2
        Statistics for Hardware VPN Module since the last clear
         of counters 241663 seconds ago
      368408630 packets in                   368408630 packets out
   203515582983 bytes in                  202789234970 bytes out
           1524 paks/sec in                       1524 paks/sec out
           6737 Kbits/sec in                      6713 Kbits/sec out
              0 pkts compressed                      0 pkts not compressed
              0 bytes before compress                0 bytes after compress
          1.0:1 compression ratio                1.0:1 overall
        3981449 commands out                   3981449 commands acknowledged
        Last 5 minutes:
         935955 packets in                      935955 packets out
           3119 paks/sec in                       3119 paks/sec out
       13198000 bits/sec in                   13123845 bits/sec out
Errors:
   ppq full errors         :        0   ppq rx errors           :        5
   cmdq full errors        :        0   cmdq rx errors          :        0
   ppq down errors         :        0   cmdq down errors        :        0
   no buffer               :        0   replay errors           :      471
   dest overflow           :        0   authentication errors   :      252
   Other error             :        0   Raw Input Underrun      :        5
   IPSEC Unsupported Option:        0   IPV4 Header Length      :        0
   ESP Pad Length          :        5   IPSEC Decompression     :        0
   AH ESP seq mismatch     :        0   AH Header Length        :        0
   AH ICV Incorrect        :        0   IPCOMP CPI Mismatch     :        0
   IPSEC ESP Modulo        :        0   Unexpected IPV6 Extensio:        0
   Unexpected Protocol     :        0   Dest Buf overflow       :        0
   IPSEC Pkt is fragment   :        0   IPSEC Pkt src count     :        0
   Invalid IP Version      :        0   Unwrappable             :        0
   PPTP Duplicate packet   :        0   PPTP Exceed max missed p:        0
   RNG self test fail      :        0   DF Bit set              :        0
   Hash Miscompare         :        0   Unwrappable object      :        0
   Missing attribute       :        0   Invalid attrribute value:        0
   Bad Attribute           :        0   Verification Fail       :        0
   Decrypt Failure         :        0   Invalid Packet          :        0
   Invalid Key             :        0   Input Overrun           :        0
   Input Underrun          :        0   Output buffer overrun   :        0
   Bad handle value        :        0   Invalid parameter       :        0
   Bad function code       :        0   Out of handles          :        0
   Access denied           :        0   Out of memory           :        0
   NR overflow             :        0   pkts dropped            :      257
Warnings:
   sessions_expired        :        0   packets_fragmented      :        0
   general                 :        0   compress_bypassed       :        0
HSP details:
   hsp_operations          : 67829762   hsp_sessions            :     1258
Slot 2:
        VAM2+ Encryption/Compression engine, Port adapter
        Port adapter is analyzed
        Port adapter insertion time 6w3d ago
        EEPROM contents at hardware discovery:
        Hardware Revision        : 1.0
        PCB Serial Number        : AAAAAAAAAAAAA
        Part Number              : 73-9571-06
        Board Revision           : A0
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
        Deviation Number         : 0
        Product (FRU) Number     : SA-VAM2+
        Version Identifier       : V02
        Top Assy. Part Number    : 68-2288-06
        CLEI Code                : CNDQASMGAA
        EEPROM format version 4
        EEPROM contents (hex):
          0x00: 04 FF 40 04 B0 41 01 00 C1 8B 4A 41 46 31 33 33
          0x10: 32 42 50 43 44 82 49 25 63 06 42 41 30 03 00 81
          0x20: 00 00 00 00 04 00 88 00 00 00 00 CB 94 53 41 2D
          0x30: 56 41 4D 32 2B 20 20 20 20 20 20 20 20 20 20 20
          0x40: 20 89 56 30 32 20 87 44 08 F0 06 C6 8A 43 4E 44
          0x50: 51 41 53 4D 47 41 41 FF FF FF FF FF FF FF FF FF
          0x60: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
          0x70: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
------------------ show pas isa interface ------------------
VPN Acceleration Module Version II+ in slot : 2
        Statistics for Hardware VPN Module since the last clear
         of counters 3204653 seconds ago
     6653099487 packets in                  6653099487 packets out
  3413929884758 bytes in                 3416877945028 bytes out
           2076 paks/sec in                       2076 paks/sec out
           8522 Kbits/sec in                      8529 Kbits/sec out
              0 pkts compressed                      0 pkts not compressed
              0 bytes before compress                0 bytes after compress
          1.0:1 compression ratio                1.0:1 overall
       51878047 commands out                  51878047 commands acknowledged
        Last 5 minutes:
         996655 packets in                      996655 packets out
           3322 paks/sec in                       3322 paks/sec out
       13649702 bits/sec in                   13570320 bits/sec out
Errors:
   ppq full errors         :        0   ppq rx errors           :       11
   cmdq full errors        :        0   cmdq rx errors          :        0
   ppq down errors         :        0   cmdq down errors        :        0
   no buffer               :        0   replay errors           :     5548
   dest overflow           :        0   authentication errors   :      872
   Other error             :        0   Raw Input Underrun      :       11
   IPSEC Unsupported Option:        0   IPV4 Header Length      :        0
   ESP Pad Length          :       11   IPSEC Decompression     :        0
   AH ESP seq mismatch     :        0   AH Header Length        :        0
   AH ICV Incorrect        :        0   IPCOMP CPI Mismatch     :        0
   IPSEC ESP Modulo        :        0   Unexpected IPV6 Extensio:        0
   Unexpected Protocol     :        0   Dest Buf overflow       :        0
   IPSEC Pkt is fragment   :        0   IPSEC Pkt src count     :        0
   Invalid IP Version      :        0   Unwrappable             :        0
   PPTP Duplicate packet   :        0   PPTP Exceed max missed p:        0
   RNG self test fail      :        0   DF Bit set              :        0
   Hash Miscompare         :        0   Unwrappable object      :        0
   Missing attribute       :        0   Invalid attrribute value:        0
   Bad Attribute           :        0   Verification Fail       :        0
   Decrypt Failure         :        0   Invalid Packet          :        0
   Invalid Key             :        0   Input Overrun           :        0
   Input Underrun          :        0   Output buffer overrun   :        0
   Bad handle value        :        0   Invalid parameter       :        0
   Bad function code       :        0   Out of handles          :        0
   Access denied           :        0   Out of memory           :        0
   NR overflow             :        0   pkts dropped            :      883
Warnings:
   sessions_expired        :        0   packets_fragmented      :        0
   general                 :        0   compress_bypassed       :        0
HSP details:
   hsp_operations          : 63582808   hsp_sessions            :     1270
------------------ show pas isa controller ------------------
Controller Information of Slot 2:
Encryption Mode = IPSec
Addresses of Rings and instance structure:
Low Priority Queue:
    OMQ=0xE3A4CC0, OMQ Shadow = 0x64B69FD8, {139, 139, 0, 256}
    PKQ=0xE3A8D00, PKQ Shadow = 0x64B6EC0C, {105, 105, 0, 256}
    ERQ=0xE3ACD40, ERQ Shadow = 0x64B73840, {0, 0, 0, 256}
Heartbeat info: <Addr, Value> = <0xE3B7EC0, 25CD87A>
Running default HSP (addr=0x63973630, size=809364)
hsp_version: 3.4(3) (PRODUCTION)
History Log location: 0x6464F9A0, end: 0x6465B520
High Priority Rings:
   TX:  0x0E3B0D80 {h=100, t=100, queued=0}
  SRC:  0x0E3B3640 TX Shadow: 0x64B78474, {h=196, t=196}
   RX:  0x0E3B1DC0 {100, 100, 256}
   RX Pool: 0x00000000 RX Pool(hsp): 0x0E3B2E00 RX Pool Shadow: 0x64B868A8, {255, 254, 511}
Instance Structure address: 0x64B63548, ce_id=0
Misc registers:
HIFN_REG=0x3D000000
IndexReg = 0xFFFF688A
PPQ registers:
 CMD  ring: ctl=0x18, head=0x64, tail=0x64 [0xE3B13C0, 0xE3B13C0]
 SRC  ring: ctl=0x10, head=0xC4, tail=0xC4 [0xE3B3C60, 0xE3B3C60]
 RSLT ring: ctl=0x5C, head=0x64, tail=0x64 [0xE3B2400]
 FREE ring: ctl=0x2000058, head=0x100, tail=0xFF [0xE3B3200]
PCI registers:
 pci_stat=0x60000, pci_intr_err_addr=0x0
hifn_driver_vary: 0x00000100
VAM2+ Active FLASH Image is: (V1) Default Image
Image Info for (V1) FLASH Default Image
Image Token == 0x2FADED2C: Image is Valid
Version        [0x03090002,0x03900201]
ChkSum    from [0xBFC00000,0xBFC2F618) is 0x00000000 with Comp==0x6C08F152
Image ChkSum is Valid
Image Info for (V2) FLASH Downloaded Image
Image Token == 0xFFFFFFFF: Image is NOT Valid
==========================================================
per Cisco error lookup, I need to escalate this to Cisco TAC, before I do that, does anyone has any idea what's causing it. My initial thoughts were HW issue
thanks in advance
Lance

Answer 1

any one? any idea?