Hanging out doing an all night maintenance, get to say hi!! live to you guys on the other side of the planet.. wet rainy night here. ..
snore....snore...snore
I would have responded, I was dealing with wannacry and was up... but I was doing a screen share... boss doesn't like it when I do non-work stuff on a screen share...
I was working on the 13th. Started 6509 upgrade form 12.2 to 15.1
Started at 10:30 and finished by 1:30. 8) 8) 8)
Quote from: LynK on May 16, 2017, 12:48:14 PM
I was working on the 13th. Started 6509 upgrade form 12.2 to 15.1
Started at 10:30 and finished by 1:30. 8) 8) 8)
I had to roll back, 2 hours upgrade, 1/2 hour of troubleshooting, 2 hours rollback.
(5k has like 20 FEX connected)
For some reason the antique StoneGate we run wasn't able to populate it's ARP table when we went from NX-OS 5.1 to NX-OS 5.2 - odd.
5K switch would see everything in it's MAC address table (l2 switch) and the upstream 6500 Catalyst (l3 switch) would see everything in ARP too,
but the Stonegate not. Through troubleshooting we found that configuring a static entry for the SVI on the StoneGate wouldn't hold through reboots.
Also found that when one of the 5K's goes down, the active/active StoneGate looses connections and everything breaks.
Something to due with our dual homed architecture and VLANs suspending when one 5K switch goes down.
Plan is to rehome the StoneGate outside connections to the 6500 Catalyst and try the upgrade again in a few weeks.
I also learned from Cisco that the upgrades go quicker when you shut the dual homed FEX connections on the secondary 5K. this way they upgrade and finish and don't forever reload (though doesn't fix the VLAN suspension issue). Without saving the config, upgrade the secondary, and all the links come back up proper.
Quote from: ristau5741 on May 17, 2017, 06:48:08 AM
For some reason the antique StoneGate we run wasn't able to populate it's ARP table when we went from NX-OS 5.1 to NX-OS 5.2 - odd.
okay... interesting
Quote
active/active StoneGate looses connections and everything breaks.
I think I found your issue. active/active is an ARP NIGHTMARE. Can you change this to A/P?
Quote
I also learned from Cisco that the upgrades go quicker when you shut the dual homed FEX connections on the secondary 5K.
Mental note taken. Are you using VPC+? Any considerations to changing this, to make it easier on yourself?
Quote from: LynK on May 17, 2017, 08:38:35 AM
Quote from: ristau5741 on May 17, 2017, 06:48:08 AM
For some reason the antique StoneGate we run wasn't able to populate it's ARP table when we went from NX-OS 5.1 to NX-OS 5.2 - odd.
okay... interesting
Quote
active/active StoneGate looses connections and everything breaks.
I think I found your issue. active/active is an ARP NIGHTMARE. Can you change this to A/P?
Quote
I also learned from Cisco that the upgrades go quicker when you shut the dual homed FEX connections on the secondary 5K.
Mental note taken. Are you using VPC+? Any considerations to changing this, to make it easier on yourself?
will have to check that A/P think, but when we roll back to 5.1 all is good. don't know why A/P should work better in 5.2
DC is closing, NOBODY is interested in spending time or effort to make changes to the End of Service architecture.
more than to run a few cables as work around to passify the security folks,
Wet and rainy. Manchester UK always wet and raining :( :( :(
We get a 25c weather warning here once every 10-15 years followed by the news telling everyone the elderly are going to die in the sweltering heat.
So be grateful you have good weather :smug:
25c causes your elderly to perish in sweltering heat?
We issue cold weather advisories for 25c or below here in Texas... :lol:
So true.
Maybe the rain is best for the UK. From the look of it, you lot seem to have trouble figuring out snow:
(http://i.imgur.com/nYOGvB3.gif)
Best to leave that business to the Scandinavians, I suppose.
Hey again, making another all night attempt as a redo for that failed maintenance on May 13th.
freakin' weird,
on the 6500 where the VLAN SVI is
I see the mac address of the firewall in the CAM,
I see the network in the CEF table
I see the IP address in the ARP table as INCOMPLETE.
I can't ping the firewall A/A VIP.
firewall doesn't see MAC address of the SVI gateway and cant ping it either.
as we roll back again, all works normally/.
firewall is connected directly to the 5K's, we moved it from the FEX to the 5K.
Quote from: ristau5741 on June 03, 2017, 12:21:29 AM
freakin' weird,
on the 6500 where the VLAN SVI is
I see the mac address of the firewall in the CAM,
I see the network in the CEF table
I see the IP address in the ARP table as INCOMPLETE.
I can't ping the firewall A/A VIP.
firewall doesn't see MAC address of the SVI gateway and cant ping it either.
as we roll back again, all works normally/.
firewall is connected directly to the 5K's, we moved it from the FEX to the 5K.
So I look up incomplete IP addresses... https://supportforums.cisco.com/document/11216/packets-are-not-being-forwarded-due-incomplete-entries-arp-table
But that's from, like, 8 years ago... and then I read another article that complains about how Cisco implements the RFC for ARP... and that's from 7 years ago... Has this been a Cisco issue for a very long time?
typically ya can't do layer 3 without layer 2
so was it working OK on the FEX, then you basically lost ARP when you moved it to the chassis port?
Did you try clearing ARPs? Was vPC involved?
smells like some kinda bug (a Nexus bug? You don't say!!!!! ROFL).
Quote from: wintermute000 on June 05, 2017, 07:50:21 AM
so was it working OK on the FEX, then you basically lost ARP when you moved it to the chassis port?
Did you try clearing ARPs? Was vPC involved?
smells like some kinda bug (a Nexus bug? You don't say!!!!! ROFL).
no we lost ARP when we upgraded the Nexus 5.1 to 5.2 only for the outside interface of the StoneGate FW-5000
tried clearing arp. the MAC never populated back to the ARP table on the 5.2 code on either side. no vPC involved.
Yay, Cisco was able to reproduced the issue in the lab.
Just wish I could stick around here until I found out the cause.
I'll have to stay in touch with the team and find out.
no you don't, mr security big shot! ;)
Quote from: ristau5741 on June 16, 2017, 06:00:09 AM
Yay, Cisco was able to reproduced the issue in the lab.
Always a cause to celebrate!
Before the lab repro:
:ckfacepalm:
After the lab repro:
:bole: