Cisco C3650 causes "crit - arp req detected an IP conflict" alerts on Juniper Netscreen Firewall.

Question

Hi All,
I am posting this issue on the Cisco community site, I've also posted it in the J-net discussion forums.
I have three Netscreen-25 firewalls on my LAN. Two are configured as a NSRP pair/cluster and the thrid is a standalone firewall. All firewalls were running ScreenOS 5.4.0r27.0 (Firewall+VPN) and they work/worked perfectly, though a few weeks ago I noticed that all the netscreen firewalls were logging critical errors:
One FW shows this - logged every 30 seconds
crit - arp req detected an IP conflict (IP 10.2.26.242, MAC 88f0310dba31) on interface ethernet1
Other FW shows this - logged every 30 seconds
arp req detected an IP conflict (IP 10.30.235.242, MAC 88f0310dba31) on interface ethernet2
Both show the same MAC.
Now I don't appear to have any problems with network services, but the these log entries are causing concern.
I have a 100% switched cisco network. I was able to track the MAC address down to a new Cisco C3650 48 port switch which i recently installed. As soon as I disconnect the switch, the critical alerts stop. As soon as I plug the C3650 switch back into the network the alerts start coming in. I have not configured this new C3650 in any special way, I have configured it in the same as all my other Cisco switches. If I plug a Cisco 3560, or 2960 (basically any other cisco switch i got) I do not get the alerts on the Netscreen FW's.
I have upgraded the software on my cisco switch to the latest version (IOS XE 03.03.04SE) and have upgraded one of my Netscreen firewalls to ScreenOS 5.4.0r28a.0 (Firewall+VPN) - the latest version. But still the critical "arp req detected an IP conflict" alerts are coming in every 30 seconds.
It's got to be something to do with the new Cisco 3650 - though I don't know what it could be. On the networking side of things everything seems to be working OK.
Please can anybody advise as what the problem might be?
Thanks in advance.

Answer 1

Hi All,
I have updated my post on the juniper forum, so will update this thread too with the same information...
Firstly thanks for your replies. I have RSTP enabled on all my switches. These new Cisco C3650 series switches are connected to the exsiting switches (in a fibre ring) using a SFP modules /fibre patch leads.
In the current setup I cannot see how there could be a layer 2 loop because the 3650 is connected via a single physical link, whether that be using a SFP module/fibre patch lead or a single gigabit ethernet port directly connected using a cat5e patch lead into another gigabit ethernet port. So in both cases only 1 link/path exists.
On the netscreen-25 the critical error reports the MAC address of the connected/trunk link port on the Cisco 3650:
"arp req detected an IP conflict (IP 10.2.26.242, MAC 88f0310df431) on interface ethernet1"
On the cisco this is the:
xxxxxxx-hh1-cat15#sh interfaces gigabitEthernet 1/1/1
GigabitEthernet1/1/1 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 88f0.310d.f431 (bia 88f0.310d.f431)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 16000 bits/sec, 25 packets/sec
  5 minute output rate 6000 bits/sec, 9 packets/sec
     350837 packets input, 29807313 bytes, 0 no buffer
     Received 234182 broadcasts (156724 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 156724 multicast, 0 pause input
     0 input packets with dribble condition detected
     119555 packets output, 9923683 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     12154 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
xxxxxx-hh1-cat15#
And second Cisco 3650 also triggers a similar alert:
on the Netscreen-25
"arp req detected an IP conflict (IP 10.2.26.242, MAC 88f0310dba31) on interface ethernet1"
On the Cisco 3650:
xxxxx-hh1-cat14#sh interfaces gigabitEthernet 1/1/1
GigabitEthernet1/1/1 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 88f0.310d.ba31 (bia 88f0.310d.ba31)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 1596000 bits/sec, 156 packets/sec
  5 minute output rate 83000 bits/sec, 77 packets/sec
     5236243 packets input, 4667733334 bytes, 0 no buffer
     Received 1400163 broadcasts (930724 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 930724 multicast, 0 pause input
     0 input packets with dribble condition detected
     2353505 packets output, 204910425 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     75948 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
xxxxxx-hh1-cat14#
As per above if I change the uplink port on the Cisco 3650, all that happens is the MAC address reported on the Netscreen changes to show the MAC of the new physically connected port.
If I connect the switches redundantly, the STP recalculates and as expected some ports go into the BLK states. But in the end the Netscreen will still report the MAC addresses of the active/FWD'ing trunk link ports. As I have two Cisco 3650's I get alerts for two MAC addresses.
I must stress that if I replace any of the new Cisco 3650 with the older Cisco 3560, 3560v2, 2960 series switches (connected in exactly the same way) I do NOT get any alerts. I only get alerts when i plug in the Cisco C3650.
So something definitely to do with new switches, but I can't see what it can be?
If I can provide anymore info that you need please let me know..
Regards