I have been banging my head on this one and cant figure out why the tunnel will not light up. Not even ISAKMP SA...
So I have an existing tunnel in place and need to shift it to a new destination IP and new destination subnets. When I stand up the new tunnel I get nothing. Packet-Tracer shows the old tunnel hits the correct NAT but the new tunnel does not and hits the global default NAT.
These are basic static NAT entries with source and destination hosts. All I did was add the new destination IPs to the ACLs referenced in the NAT statement.
What am I missing here???
static (inside,outside) 10.19.176.25 access-list inside_nat_static_20
access-list inside_nat_static_20 extended permit ip host GROUPA host xxx.xxx.xxx.119
access-list inside_nat_static_20 extended permit ip host GROUPA host xxx.xxx.xxx.110
access-list inside_nat_static_20 extended permit ip host GROUPA host xxx.xxx.xxx.142 <---this line is what I added
Is that in the same subnet range as .119 and .110?
I'm not sure what causes the tunnels to hit the global like this. I'm no ASA expert. But what you need to do is change the NAT order, easiest with ASDM, so that the new one comes up first before the old tunnel. Also this order can affect your crypto maps as well. I kill myself on this constantly.
Edited to contain proper engrish
On my side they are all setup as host statements with /32
Static routes also point to each host as a /32.
Shifted the ACE entry to the top and still no dice.
Stupid question, do I need to clear xlate or anything to re-apply the ACL/NAT changes?
Wouldn't hurt to try that.
Now that it is higher in the list than the old tunnel does packet tracer show the same results?
Nope no change. I also did a clear xlate.
The more I work on these ASAs the more I hate them.
How about the crypto maps? Is there an entry with a lower number that will match prior to the new tunnel?
Quote from: routerdork on September 23, 2015, 01:35:28 PM
How about the crypto maps? Is there an entry with a lower number that will match prior to the new tunnel?
There is the original tunnel but the interesting traffic ACL does not match the new destination IPs I added.
Do you have proxy-ids defined for each host, on each end of the tunnel?
Quote from: deanwebb on September 23, 2015, 01:58:40 PM
Do you have proxy-ids defined for each host, on each end of the tunnel?
Looking at the config I dont think so. No sure what the other end looks like though.
Can you paste your packet tracer results? Or the pertinent information to the tunnels?
Phase: 1
Type: FLOW-LOOKUP
Subtype:
Result: ALLOW
Config:
Additional Information:
Found no matching flow, creating a new flow
Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 0.0.0.0 0.0.0.0 outside
Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_access_in_1 in interface inside
access-list inside_access_in_1 extended permit ip any any
Additional Information:
Forward Flow based lookup yields rule:
in id=0xac456388, priority=12, domain=permit, deny=false
hits=644099416, user_data=0xac456348, cs_id=0x0, flags=0x0, protocol=0
src ip=0.0.0.0, mask=0.0.0.0, port=0
dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 4
Type: CONN-SETTINGS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xac681480, priority=7, domain=conn-set, deny=false
hits=398409650, user_data=0xb207e4d8, cs_id=0x0, flags=0x0, protocol=6
src ip=0.0.0.0, mask=0.0.0.0, port=0
dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xadeae910, priority=0, domain=permit-ip-option, deny=true
hits=738004768, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip=0.0.0.0, mask=0.0.0.0, port=0
dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 6
Type: IDS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xad7fa058, priority=50, domain=ids, deny=false
hits=644475458, user_data=0xad7f91a0, cs_id=0x0, flags=0x0, protocol=0
src ip=0.0.0.0, mask=0.0.0.0, port=0
dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 7
Type: FOVER
Subtype: standby-update
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xadeabc80, priority=20, domain=lu, deny=false
hits=398705818, user_data=0x0, cs_id=0x0, flags=0x0, protocol=6
src ip=0.0.0.0, mask=0.0.0.0, port=0
dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 8
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside) 1 0.0.0.0 0.0.0.0
match ip inside any outside any
dynamic translation to pool 1 (<OUTSIDE IP> - <OUTSIDE IP>)
translate_hits = 371278761, untranslate_hits = 13163439
Additional Information:
Dynamic translate SERVER-IP/80 to <OUTSIDE IP>/88 using netmask 255.255.255.255
Forward Flow based lookup yields rule:
in id=0xace6b3a0, priority=1, domain=nat, deny=false
hits=375332119, user_data=0xace6b300, cs_id=0x0, flags=0x0, protocol=0
src ip=0.0.0.0, mask=0.0.0.0, port=0
dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 9
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
nat (inside) 1 0.0.0.0 0.0.0.0
match ip inside any outside any
dynamic translation to pool 1 (<OUTSIDE IP> - <OUTSIDE IP>)
translate_hits = 371278774, untranslate_hits = 13163439
Additional Information:
Forward Flow based lookup yields rule:
in id=0xace6b630, priority=1, domain=host, deny=false
hits=158172805, user_data=0xace6b300, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip=0.0.0.0, mask=0.0.0.0, port=0
dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 10
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0xacb91b60, priority=0, domain=permit-ip-option, deny=true
hits=569718276, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip=0.0.0.0, mask=0.0.0.0, port=0
dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 11
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 853760457, packet dispatched to next module
Module information for forward flow ...
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_divert_fragment
snp_ids
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat
Module information for reverse flow ...
snp_fp_inspect_ip_options
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_divert_fragment
snp_ids
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat
Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow
On working tunnels, Phase 8 matches the correct static NAT and then Phase 10 is VPN.
Give me a bit and Ill get config on here too.
crypto map outside_map 37 match address outside_37_cryptomap
crypto map outside_map 37 set peer <Peer IP>
crypto map outside_map 37 set transform-set ESP-3DES-SHA
crypto map outside_map 37 set security-association lifetime seconds 28800
crypto map outside_map 37 set security-association lifetime kilobytes 4608000
crypto map outside_map 65535 ipsec-isakmp dynamic outside_dyn_map
crypto map outside_map interface outside
tunnel-group <Peer IP> type ipsec-l2l
tunnel-group <Peer IP> ipsec-attributes
pre-shared-key *
access-list outside_37_cryptomap extended permit ip object-group INTERNAL-SERVERS object-group REMOTE-SERVERS
access-list outside_37_cryptomap extended permit ip object-group INTERNAL-SERVERS-1 object-group REMOTE-SERVERS
NAT config is listed above. IPS and names are changed to protect the guilty.
Looking at your config above, which is pretty much identical to how we do our tunnels, the only difference is how we do our NAT. What is your NAT trying to accomplish? Is just using one NAT statement to allow access for each tunnel? Also what code are you running?
nat (INSIDE,OUTSIDE) source static insert_source_object_group insert_source_object_group destination static insert_destination_object_group insert_destination_object_group route-lookup
So the equivalent to yours would be...
nat (inside,outside) source static INTERNAL-SERVERS INTERNAL-SERVERS destination static REMOTE-SERVERS REMOTE-SERVERS route-lookup
nat (inside,outside) source static INTERNAL-SERVERS-1 INTERNAL-SERVERS-1 destination static REMOTE-SERVERS REMOTE-SERVERS route-lookup
Code is old. 8.0...
We have a reserved range for NATed traffic for Vendor tunnels. So its just basically there so we dont overlap with their space. These servers just do a 1-to-1 for all vendors.
Quote from: that1guy15 on September 23, 2015, 05:46:08 PM
Code is old. 8.0...
We have a reserved range for NATed traffic for Vendor tunnels. So its just basically there so we dont overlap with their space. These servers just do a 1-to-1 for all vendors.
Please upgrade so as to avoid major security fail. Thank you.
My ASA-fu is weak now but I am curious about this and can't see anything obvious so please keep us in the loop!
I'd just TAC it TBH - throw the show run and packet tracer at them.
Yeah pretty much said the same thing when I took over these firewalls. I got denied multiple times because they dont see just reason for a possible outage. The cut to 8.3 looks like a nightmare for these guys as they have not been administered well....
I now have been approved to replace them Q1 next year so they would rather wait till then to take an outage and are fine with them as-is...
Gotta love it!!
Quote from: wintermute000 on September 23, 2015, 08:20:55 PM
My ASA-fu is weak now but I am curious about this and can't see anything obvious so please keep us in the loop!
I'd just TAC it TBH - throw the show run and packet tracer at them.
Yup. TAC call will be first thing in the morning. Figured Id give you guys first crack.
Thanks for the help so far!!
Quote from: that1guy15 on September 23, 2015, 08:27:09 PM
Quote from: wintermute000 on September 23, 2015, 08:20:55 PM
My ASA-fu is weak now but I am curious about this and can't see anything obvious so please keep us in the loop!
I'd just TAC it TBH - throw the show run and packet tracer at them.
Yup. TAC call will be first thing in the morning. Figured Id give you guys first crack.
Thanks for the help so far!!
If TAC says to upgrade the code...
:haha1:
ah yeah dude. I HATE calling TAC on these guys. Its their easy out on everything and they dig in and call it a day.
"What the PSU is on fire? Yeah you are running old code. Upgrade to 9.x and then we can help disable the fire."
Back to basics
Checking what you are trying to accomplish. What's the exact parameter to obtain the packet tracer output?
From my reading of it:
Traffic FROM inside interface TO outside interface
FROM GROUPA TO xxx.xxx.xxx.142
NAT source IP i.e. GROUPA to 10.19.176.25
and then your crypto map ACL 37 is supposed to match interesting traffic which is (due to order of operations) 10.19.176.25 to xxx.xxx.xxx.142?
And this works for the first 2 lines of access-list inside_nat_static_20, just not the new one?
static (inside,outside) 10.19.176.25 access-list inside_nat_static_20
access-list inside_nat_static_20 extended permit ip host GROUPA host xxx.xxx.xxx.119
access-list inside_nat_static_20 extended permit ip host GROUPA host xxx.xxx.xxx.110
access-list inside_nat_static_20 extended permit ip host GROUPA host xxx.xxx.xxx.142 <---this line is what I added
I'd be curious to see a packet-tracer from the one that works, then compare it to the one that doesn't.
@wintermute000
That is correct. To clarify the NAT statement is called by the original tunnel for the original two entries and is still functioning. The new tunnel calls the same NAT statement but only for the new entry.
So the reason I asked about code...back in the 8.1/8.2 days I ran into a bug where a change on a tunnel caused it to no longer put traffic across the tunnel until the ASA was reloaded :barf:
Quote from: routerdork on September 24, 2015, 08:49:31 AM
So the reason I asked about code...back in the 8.1/8.2 days I ran into a bug where a change on a tunnel caused it to no longer put traffic across the tunnel until the ASA was reloaded :barf:
Might want to give that a try, even though they're outage-averse.
TAC picked it up pretty quick. Its mostly me being stupid with how ASAs do tunnels.
Two things:
1) NAT Exemption was not in place for the new traffic flow so the default NAT was being used. I was under the impression the default NAT was checked last after everything. But I guess it goes first. OK...
2) It was mentioned earlier in the thread. Interesting traffic being matched via ACL overlapped with the original tunnel so it would not stand up the new tunnel. I assumed since it matched different ACEs of the ACL it would differentiate between the two. I assumed wrong.
So working with the other end I have two options to move forward. Update the original tunnel with a secondary tunnel destination and have them kill the old tunnel on their end, or remove the old tunnel completely on my side and stand up the new tunnel.
It looks like that1guy needs to spend some time and get alot more familiar with the ASA line. :doh: :wall:
Thanks for posting the solution!!
At least it wasn't the code :pub:
#2 is good to know.
As for #1 I'm not understanding something - I thought about NAT exemption but in my mind, if you match the NAT exemption, you're... exempt. Hence no-nat, not even the default NAT.
If your NAT order was correct (as mentioned earlier) your static NAT should be hit before your default? (or am I too used to post 8.3 where static is before dynamic?)
This is 8.0 code... Yeah check the packet tracer and you can see it never hits the right NAT. Once they where added to the NAT-exemption Phase 8 showed up as a NAT-EXEMPT and Phase 9 hit the proper NAT and Phase 10 was VPN.
Dude this cut went rough. Really shows I am not very strong with ASA VPNs. First I was able to take an outage and pull down the old tunnel and setup the new one clean. But after dropping in the code no dice. Do a debug and see the SPIs dont match both sides. OK... Working with the other side I was using specific host when they where setup with a /28 for their side. So Phase 1 would complete but Phase 2 would terminate because there were no matching SPIs...
Cleaned that up and still no go. Other dude figured out I was setup for 3DES and he was AES. Corrected and we are MM_STATE and traffic is flowing.
Id love to see how to catch that out of the debugs. Im sure its pretty clear when you get the logs. Debug with IPSEC really are your best friend.
Im gonna go drink now...
I love IPSEC debugs. Some of the best error messages are in those. My favorite is "paranoid keepalives."
Quote from: deanwebb on September 24, 2015, 06:31:26 PM
I love IPSEC debugs. Some of the best error messages are in those. My favorite is "paranoid keepalives."
I love all debugs. Some of my best CCIE study time was in the middle of debugs. I really wish I could run them in production more.
Also love me some packet captures.
Quote from: that1guy15 on September 24, 2015, 07:58:07 PM
Also love me some packet captures.
Amen. If there's one thing that I learned long before Cisco realized the importance of it and added it to the CCIE track... it's that I am extremely thankful for, though hated it at the time, working in tech support staring at packet captures all day. Learning Wireshark and how to analyze protocol communication and compare it with RFCs/other docs was probably one of the single most important skills I've ever learned.
It's amazing how something so simple can be so profound. An example was where a customer was claiming their web proxy was slowing down the network, which was partially true, but not the proxy's fault. What was the root issue? Slow/inconsistent DNS responses. I took a packet capture and created a simple I/O graph creating a filter showing DNS requests and DNS responses... you could EASILY see over time the missed queries. Why? Narrowed it down to the fact they were using an old DNS record by hostname instead of IP addresses (Which we at Websense always recommended IP addresses for a reason), and several IP addresses in that record were no longer functional.
Thank youuuuuuuuu Wireshark.
I'm still confused as to the NAT exemption. If you add something to NAT exemption, you are saying DO NOT NAT it. And it would show up in packet tracer.
How did adding the traffic you want to be NAT - to the 'DO NOT NAT" list - fix the issue?
And worse, how does your working traffic hit the NAT exemption, then hit another NAT rule after that?!??! confused....
nat (inside) 0 access-list inside_nat0_outbound
ACL is used to match traffic that should be excluded. I agree this seems very backwards. But this NAT is always checked first.
yes, so if you add it to this line, your traffic is NOT NAT. Its the same pre and post 8.3, except 8.3 is defined as explicit i.e. source same dest same.
I thought you DID want to source NAT through your VPN?