Some hard won knowledge to flag with y'all security guys
a lot of recently updated linux/BSD based open source FW (including commercial derivatives such as sonicwall etc.) run a new version of the 'nix IPSEC daemon called strongswan, not the old raccoon.
This does NOT play nice with cisco's implementation of IKEv1 i.e. all the old school IPSEC tunnels we've all been running, if you have multiple SAs (i.e. multiple lines in your crypto interesting traffic ACL). The behaviour is random and intermittent e.g. some SAs will get 'stuck' intermittently, rebooting will generally fix it but only briefly, etc.
The only fix (aside from downgrading the FW back to a version using the old raccoon daemon) is to migrate the Cisco end to IKEv2.
I ran into this specific issue on PFsense, and it took away a weekend and a lot of stress at a long term client facing a production meltdown that I thought was my fault... before he told me 'you know, I did upgrade these FWs a few days ago....'. LOL. Found this issue documented, and it is confirmed to also affect sonicwall, symantec to name two other vendors
https://doc.pfsense.org/index.php/Upgrade_Guide#Problems_with_Rekeying_with_Multiple_Phase_2_Entries (https://doc.pfsense.org/index.php/Upgrade_Guide#Problems_with_Rekeying_with_Multiple_Phase_2_Entries)
IKEv2 is STRONGLY recommended by this security professional. IKEv1 has been compromised, so this is one more reason to switch to v2.
If you do government IA stuff you have to do IKEv2
I'm aware of the failings of IKEv1 - on the ground, there are a million and one of these deployments still around + it still ends up as the default 'copy and paste' deployment in plenty of places.
In my soon to be former workplace (hahahahahahahahahah :dance: ), I cound no less than 20 managed services environments, all large enterprise, and only the financial firms/banks are using IKEv2 across the board.
SMB market is even worse, they're all running off copy and paste scraps from blogs originating 5-10 years ago - hey the config still works so nobody thinks twice, esp the typical small business all-in-one techies