Whats the tried and tested way of configuring OSPF between a pair of ASA's and layer 3 core switches?
/30 or /29 p2p between both asas (assuming ha pair) and each switch and between switches. Metric to force inferior path to and from the secondary.
I'm troubleshooting an issue with an ASA cluster, OSPF peering with two L3 switches in separate locations on one VLAN. So basically, a broadcast VLAN/network.
The ASA is getting the same routes from both neighbours. In the DB, I see the LSAs have the same metric but for some reason the ASA only installs one route in the routing table. Unfortunately it's the one with the least optimal path. Shouldn't the ASA be doing equal-cost loadbalancing by default or is this different on the ASA platform?
Is your ASA cluster active/active or active/passive.
I am supposing active/passive. They are 5515X's.
If they had two different P2P subnets it would be easy to increase the cost on one and be done with it, but here they are in the same segment so raising the cost on the ASA interface itself would have the same effect on both the routes.
That's exactly why I recommended my approach. Keep us posted with yours as ecmp should be ospf default as you've stated
I know, I don't even see a reason why they would need OSPF in this case as the ASA is basically a stub. HSRP on that transit network would do just fine.
But it's a branch site that "manages their own equipment". Their original complaint was bad WAN performance so I'm gathering as much as I can.
Sorry to hijack your thread, Dieselboy, but it's an example of how not to configure it :)
Check the asa equivalent of show ip cef or the juniper show route forwarding table
Quote from: wintermute000 on February 03, 2015, 06:00:47 AM
/30 or /29 p2p between both asas (assuming ha pair) and each switch and between switches. Metric to force inferior path to and from the secondary.
Wintermute, I don't think your statement would work at all. For an ASA active/standby setup you need the interfaces on the inside to be able to communicate with each other for monitoring. This will consume 2 IP's. Then if you have 2 core switches this will consume another 2 IP's so a /30 would definitely not work. In any case, I could use a /8 or a /24 - the size of the broadcast domain makes no difference in all honesty.
Quote from: SimonV on February 03, 2015, 08:51:35 AM
I'm troubleshooting an issue with an ASA cluster, OSPF peering with two L3 switches in separate locations on one VLAN. So basically, a broadcast VLAN/network.
The ASA is getting the same routes from both neighbours. In the DB, I see the LSAs have the same metric but for some reason the ASA only installs one route in the routing table. Unfortunately it's the one with the least optimal path. Shouldn't the ASA be doing equal-cost loadbalancing by default or is this different on the ASA platform?
Simon, I have your exact setup but a different issue. I have 2 x ASA5515X's which I've configured OSPF to two separate layer 3 switches. The switches are Nexus and to get around the vPC issue (which I don't think affects the 3000 series nexus anyway), I have a separate port channel between the two Nexus running the VLAN that the ASA's are connected on. So to explain more simply, each ASA inside interface is connected to each switch on VLAN 5. VLAN 5 SVI on the switches should have 2 peers, one to the other switch and one to the primary ASA. I did think that there may be an OSPF peer to the secondary ASA but this is not seen. To allow the ASA's to communicate to each other for the monitoring, VLAN 5 is connected between the switches on a trunk which is not part of the vPC setup. It's 2 cables for cable resilience. So in this setup we can ignore the fact that the switches are Nexus. The two switches are layer 3 devices connected with a link.
What I am seeing is everything working perfect except on one switch, the primary ASA is showing as EXSTART. I ran some debugs and the logs showed that the issue is due to an MTU mismatch. I know for a fact that this is not the case. I checked all layer 3 interfaces on all devices and the MTU is unchanged at 1500. As a test, I put in the ospf mtu ignore and this made no difference at all, which kind of backs up this not being an MTU issue. I then removed the MTU ignore command.
If I shut down the VL5 interface on the primary switch, OSPF will come up to the second switch (where it's currently EXSTART) if I clear the process, so this would say to me the config is fine.
I was thinking over night over this and TBH I don't really need OSPF on the firewalls but it's handy if it's running on them since we're growing quite quickly and there's only me here. But if I cannot isolate this problem for the sake of stability I'll remove OSPF and just use static routes inbound to the HSRP IP. The other benefit of OSPF was the VPN injecting the static routes.
Here's the config:
ASA
interface GigabitEthernet0/2
description INSIDE INTERFACE
mac-address 00a0.c9c0.0501 standby 00a0.c9c0.0502
nameif INSIDE
security-level 100
ip address 192.168.5.4 255.255.255.0 standby 192.168.5.5
ospf hello-interval 1
ospf message-digest-key 1 md5 *****
ospf authentication message-digest
router ospf 1
router-id 192.168.5.4
network 192.168.5.0 255.255.255.0 area 0
log-adj-changes
redistribute static subnets route-map REDIST
SW1
interface Vlan5
description INTERNAL OSPF NEXUS AND ASA PEER LINKS
no shutdown
no ip redirects
ip address 192.168.5.2/24
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 7 ******
ip ospf hello-interval 1
no ip ospf passive-interface
ip router ospf 1 area 0.0.0.0
hsrp 249
preempt delay minimum 60 reload 60
timers 1 4
ip 192.168.5.1
router ospf 1
log-adjacency-changes
passive-interface default
SW2
interface Vlan5
description INTERNAL OSPF NEXUS AND ASA PEER LINKS
no shutdown
no ip redirects
ip address 192.168.5.3/24
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 7 *****
ip ospf cost 10
ip ospf hello-interval 1
no ip ospf passive-interface
ip router ospf 1 area 0.0.0.0
hsrp 249
preempt delay minimum 60 reload 60
priority 90
timers 1 4
ip 192.168.5.1
router ospf 1
log-adjacency-changes
passive-interface default
I raised a TAC under the 3048 Nexus
TAC engineer I've dealt with before and he's pretty good. He requested quite lot of information from me, including a diagram to illustrate - took me about 30 mins or so to collect all the logs and show outputs he wanted. Came back to me a few hours later and stated couldn't see anything wrong.
Then we had a webex with an ASA engineer as well and we all agreed there should be no problem with the config. So they're going to lab it up and reproduce. We also collected captures and more debugs.
I think that may be the nexus switch is doing something with the OSPF packet as it traverses the port-channel, although it shouldn't be. It only decrements the TTL when going across the vPC link and there is definitely no vPC involved.
I'll let you know what TAC says.
my mistake, too many routers and not enough fw work over the last year, forgot the HA clustering its one virtual unit with a floating ip.
When you say nexus vPC link are you referring to the vPC peer link? and you're excluding VLAN5 from the peer link trunk?
Static routes FTW on a HA link. You don't want to have the devices go into a failover fugue state because a route update goes awry.
Quote from: Dieselboy on February 03, 2015, 08:31:42 PM
In any case, I could use a /8 or a /24 - the size of the broadcast domain makes no difference in all honesty.
It can matter if you work for a big organization and you are running out of RFC1918 IP address; I would fire someone who waste a /8.
Quote from: AnthonyC on February 04, 2015, 10:59:01 AM
Quote from: Dieselboy on February 03, 2015, 08:31:42 PM
In any case, I could use a /8 or a /24 - the size of the broadcast domain makes no difference in all honesty.
It can matter if you work for a big organization and you are running out of RFC1918 IP address; I would fire someone who waste a /8.
Indeed. Big companies treat IP addresses like the last few drops of fuel in a Mad Max movie.
Quote from: wintermute000 on February 04, 2015, 02:16:01 AM
my mistake, too many routers and not enough fw work over the last year, forgot the HA clustering its one virtual unit with a floating ip.
When you say nexus vPC link are you referring to the vPC peer link? and you're excluding VLAN5 from the peer link trunk?
:)
Yes exactly that. the vPC peer link decrements the TTL so when OSPF hello's are sent by the ASA, the second switch wont form an OSPF relationship with the ASA if the hello comes across the vPC link. So to get around this, I did the following (I knew this initially so I planned it this way):
1. create VLAN 5
2. connect 2 additional links between the switches and form port-channel 6
3. Po6 has switchport trunk allowed vlan 5
4. connect the ASA primary and secondary inside interfaces into switchport access VLAN 5 on the switch
The above should have worked fine. however both switches would not form OSPF.
Now, the new problem: there's no ospf at all, there's no Po6 at all. VLAN 5 is now on Po1 which is the vPC peer link. As there's no ospf this is fine.
HSRP is in use on VLAN 5 so the ASA can route back to the network.
Static default route from the core pointing to the ASA primary address.
When I ping the standby ASA I never get a reply. I did a ping from the second switch and it's fine. If I source from example VLAN 10 I do not get a reply.
I ran some captures and the ASA receives the ping. It's reply is targeted for the HSRP address which is switch 1. Switch 1 somehow doesn't forward it to the second switch. I put a workaround in place which is to enable ip redirects on vlan 5.
Waiting for TAC's response. May be there's a bug in the nexus code which is causing these probs.
Quote from: deanwebb on February 04, 2015, 08:35:24 AM
Static routes FTW on a HA link. You don't want to have the devices go into a failover fugue state because a route update goes awry.
I thought exactly this when I did my testing and lost all access to the internet because OSPF screwed up. I gave TAC the day to come to me with why this was failing and they didn't know and still don't. So I had no second thought about stripping out OSPF. I don't like the risk. TBH it's more configuration to have OSPF running there. I Removed the OSPF config and it's replaced with 3 static routes and 2 prefix lists in a route map to be redistributed.
Quote from: AnthonyC on February 04, 2015, 10:59:01 AM
It can matter if you work for a big organization and you are running out of RFC1918 IP address; I would fire someone who waste a /8.
Ahh but then you're talking about policy, best practice and specific requirements based on an audit or documentation. I'm talking technically about a technical function. I would laugh if I ever saw a /8 actually used in a design in that manner.
Did you put in the vPC peer gateway command? (as otherwise it would cause problem with HSRP)
@Dieselboy,
Why do you think the N3K is immune to the issue of peering over a vPC? I ask because AFAIK only the N6K doesn't have that issue (though I can't find the CiscoLive preso that mentions that).
My experience with Nexus is that although some stuff works for the N5K, it may not work for the N7K, and what works on one version or line card doesn't work on the other, so as a principle I always refer to the official config guides for the software version and for the specific Nexus I'm working on for verification on what works and what doesn't, and be very attentive to detail.
What's odd is the config guides for the N3K don't say anything about routing peering in combination with a vPC...but the N5K and N7K guides explicitly state what works and what doesn't (e.g. running a secondary L2 trunk between the switches for SVI L3 peering on the peer devices is supported on the N5K, but not on the N7K), so unless you can find documentation that states that your setup is supported on the N3K (or your TAC buddy says differently), I wouldn't trust this to work even if you prune VLAN 5.
In any case, the "recommended" way of doing routing in your case is L2 vPC from Nexus to your ASAs, static routes on the Nexus pointing to the ASA, then redistributing those into your IGP. Look here starting on page 104:
http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Aug2014/CVD-DataCenterDesignGuide-AUG14.pdf
The specific config on that doc applies to the N5K, but unless you find documentation that excludes the traditional issues from the N3K, don't assume anything :-).
You forego dynamic routing from the ASA directly, but if you just follow that guideline you'll save yourself the headaches I went through in trying to figure out where the OSPF packet went, why adjacencies weren't forming, and learning all the discrepancies among the Nexus family and software versions.
Quote from: AnthonyC on February 05, 2015, 09:52:56 PM
Did you put in the vPC peer gateway command? (as otherwise it would cause problem with HSRP)
Yes peer-gateway is enabled but it would not cause a problem with HSRP without it enabled. Not in this scenario.
Quote from: killabee on February 06, 2015, 12:58:11 AM
@Dieselboy,
Why do you think the N3K is immune to the issue of peering over a vPC? I ask because AFAIK only the N6K doesn't have that issue (though I can't find the CiscoLive preso that mentions that).
My experience with Nexus is that although some stuff works for the N5K, it may not work for the N7K, and what works on one version or line card doesn't work on the other, so as a principle I always refer to the official config guides for the software version and for the specific Nexus I'm working on for verification on what works and what doesn't, and be very attentive to detail.
What's odd is the config guides for the N3K don't say anything about routing peering in combination with a vPC...but the N5K and N7K guides explicitly state what works and what doesn't (e.g. running a secondary L2 trunk between the switches for SVI L3 peering on the peer devices is supported on the N5K, but not on the N7K), so unless you can find documentation that states that your setup is supported on the N3K (or your TAC buddy says differently), I wouldn't trust this to work even if you prune VLAN 5.
In any case, the "recommended" way of doing routing in your case is L2 vPC from Nexus to your ASAs, static routes on the Nexus pointing to the ASA, then redistributing those into your IGP. Look here starting on page 104:
http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Aug2014/CVD-DataCenterDesignGuide-AUG14.pdf
The specific config on that doc applies to the N5K, but unless you find documentation that excludes the traditional issues from the N3K, don't assume anything :-).
You forego dynamic routing from the ASA directly, but if you just follow that guideline you'll save yourself the headaches I went through in trying to figure out where the OSPF packet went, why adjacencies weren't forming, and learning all the discrepancies among the Nexus family and software versions.
I definitely don't think n3k is immune to vPC issues, actually quite the opposite. I think you have misunderstood my posts or not read them fully. " I wouldn't trust this to work even if you prune VLAN 5." - it definitely will work if you prune VLAN 5 since pruning vlan 5 from the vPC trunk is removing it from the vPC configuration.
I've still yet to read that design guide! :) it's on my list.
I had TAC look at the secondary issue regarding not being able to ping the 2nd ASA. The guy believed he found a forwarding table issue (like a CEF problem, but the Nexus equivalent). As a test, we failed over HSRP by adjusting the priorities and everything began working fine. We then failed it back and everything remained working fine and has been ever since. On my list of things to do is to bring the switches up to a later code as they've been up for 500+ days. I've not yet had a chance to look through the release notes to justify it though, ie for security fixes. However Cisco advised the above is not normal and an upgrade is likely in order. I also think the above is what caused our OSPF not to work.
In any case, static routing is working fine and is less hassle. :)
Hmmm, now I'm confused because I sure did read your post where you made this comment:
"I have 2 x ASA5515X's which I've configured OSPF to two separate layer 3 switches. The switches are Nexus and to get around the vPC issue (which I don't think affects the 3000 series nexus anyway)..."
So what vPC issue are you referring to that you don't think affects the N3K?
Nexus is bleeding edge bug fun time. 500+ days, upgrayed that shiz!!!!!!!!
Not surprised its looking like a bug as I've found cisco design guides specifically outlining your topology and the solution of running the vlan across a non-vPC.
A colleague of mine is dealing with 2 separate N7k issues where the sup randomly reboots. LOLOLOL (and you know 7k = its going to be a pretty darned important switch at a pretty darned important site/DC)
https://tools.cisco.com/bugsearch/bug/CSCui72592 (https://tools.cisco.com/bugsearch/bug/CSCui72592)
Quote from: killabee on February 08, 2015, 10:02:28 PM
Hmmm, now I'm confused because I sure did read your post where you made this comment:
"I have 2 x ASA5515X's which I've configured OSPF to two separate layer 3 switches. The switches are Nexus and to get around the vPC issue (which I don't think affects the 3000 series nexus anyway)..."
So what vPC issue are you referring to that you don't think affects the N3K?
This part: "(which I don't think affects the 3000 series nexus anyway).." was referring to the differences between n5k and n7k where they treat vPC slightly differently, that's all. It's still documented to not peer OSPF across a vPC though.
Quote from: wintermute000 on February 09, 2015, 12:25:31 AM
Nexus is bleeding edge bug fun time. 500+ days, upgrayed that shiz!!!!!!!!
I don't really want to. It's all working fine (now), it's completely internal with no customer access secured by a firewall. My biggest concern is upgrading it and introducing bugs / issues. Even if I do read the release notes Cisco's "internal bug / private bug" system can make the release notes somewhat un-useful. Really annoys me :)