We have been working on some solutions with syslog, and we tried kiwi (to the recommendation of some of those here) and it is atrocious. Down right abysmal. We are currently demoing logzilla, and I HIGHLY recommend everyone give it a try. It is amazing. We are really enjoying it, and by the way they have 25% off promotion until the end of December.
We haven't had any issues with SPLUNK. Not sure what your requirements are though.
no probs here with syslogd, with a bit of creative scripting to bzip daily logs and clean up the log directories.
search for things via grep
Graylog, ELK stack, or splunk if you're made of money.
splunk. It probably is the largest scaling solution out there (AWS.... we have sold it and developed packages for banks etc. - its really more of a database than a syslog solution)
My biggest stink is I have yet to see a syslog provider give you the ability to have separate panels with device filters, so I can have one for my MPLS routers, MPLS switches, HQ Access, HQ DC, Firewall, etc.
it would be so nice.
You can do it ghetto style using facilities as categories
Gravedig because I'm in charge of finding a syslog for the network team at my work. Need a solution for a global company with about 500 locations, 4 of which are massive data centers. Need to look at HP Arcserve and Splunk, but will also get to look at 2-3 others. We need something with a really easy to use front end so our level 1/level 2 people can use it within 15-30 minutes of being told that they're going to do some work with syslog.
Splunk is awesome but for large deployments it's typically used as a db to a custom front end or monitoring solution
We have an entire ps department who do nothing but program splunk
Quote from: wintermute000 on January 21, 2016, 05:26:59 PM
Splunk is awesome but for large deployments it's typically used as a db to a custom front end or monitoring solution
We have an entire ps department who do nothing but program splunk
Why do the front ends need to be customized? Is that missing in splunk, or is that just pretty standard in any syslog product to have to customize front ends?
the splunk front end is pretty generic - its actually regarded more as a general purpose unstructured DB (no schema) that is wicked fast than a pure syslog product, though syslog is one of its common applications
It does work fine IMO but you have to basically curate filter/query templates and know how to tweak them. Not tried automating reports or alerts etc. but I'd imagine thats a big part of custom integrations. I've only really driven the free version (limited to 10k a day or something like that)
There's a whole ecosystem of products now though, I'd recommend talking to a splunk consultancy to get the max value. BUt I can tell you for a fact that my company does a lot of splunk business, all for very large organisations like yours, to develop custom syslog/reporting systems and integrate into monitoring, ticketing etc. For example we have two guys who sit in a large telco who've been exclusively developing custom splunk for the last 5 years and constantly integrating new systems/parts of the network.
http://www.splunk.com/en_us/products/splunk-enterprise.html (http://www.splunk.com/en_us/products/splunk-enterprise.html)
If you just want a simple syslog appliance that sends emails then perhaps its overkill.
I do need to do this here... There's a log viewer in our Cacti.
What's free, vs what's not free? The cacti one I think uses syslogd, and the front end is cacti which you can filter based on per device and date / time.. Seems okay, but I've not properly spent the time to set it up yet. And it's free.
We used cacti at my old work, including for syslog, heavily customised bit worked fine
Basic splunk is free for up to 10k logs a day
Your syslog should be linked with your device monitoring system. What are you using to monitor your devices? There should be some sort of syslog capability there, along with the alerting functions. It needs to be integrated, for example a device sends out a syslog event for a failed fan, no use for the alert to go into the syslog database without an event alert being sent. Maybe I'm starting to wade into the weeds of SNMP, where your device sends traps to the monitoring system and an alert is generated. syslogs are pretty useless if you don't know (can't find) and can't react.
I have yet to get to use it but we use LogRhythm here. I'm supposed to go to the training in Boulder, CO at some point this year.
http://www.sans.org/reading-room/whitepapers/logging/ins-outs-system-logging-syslog-1168
https://tools.ietf.org/html/rfc5424
^ Gonna read those today before diving into marketing slicks. Always best to be armed with knowledge.
We/I use syslog-ng and ELK. Its much like splunk just opensource and doesnt have all the pre-made plugins..
Sent from my iPhone using Tapatalk
Reading over the product descriptions for Arcsight, Splunk, Secnology... now I'm wondering if what we really want is a network monitoring solution, whether or not it accepts syslog as an input. These products all seem like they're set up as big data solutions and what we want most is something that will fire off an alert when a switch fan is about to fail or when a router interface goes down.
I've generally seen all-in-one type solutions in smaller networks (10k endpoints etc.), but it sounds like your network is huge, and you're going to want a proper distributed DB backend anyway - make sure your integrated monitoring solution scales to the appropriate size and availability.
Don't discount the issue of performance/indexing/searching either. with a huge DB thats taking tens of thousands of syslogs a day (or hundreds of thousands a day if someone turns on a debug? LOL). It may be a wiser move long term to have a separate back end and front end.
Log insight :rock: :rock:
Syslog is something I work with a lot at my current job. A good document on this is actually from Cisco;
Building Scalable Syslog Management Solutions
http://www.cisco.com/c/en/us/products/collateral/services/high-availability/white_paper_c11-557812.html
We use syslog for troubleshooting, cyber incident correlation, alerting and triggering tasks, etc. Based on what you posted above you are probably going to want a distributed model where you put a collector at each location. Then you can aggregate that back to a main central system from each collector. The main reason to do this is because syslog is by default UDP. Some systems will allow you to change this, but not everything.
Even the systems that do support syslog over TCP have some issues. I have seen two fairly major applications fail TCP syslog. One basically treated it as UDP just using TCP. So it would send the log message, but not buffer the message until it was acknowledged. So what would happen is a packet would get dropped, and the receiver would constantly request retransmission which the sending system couldn't do (it already flushed the message). Eventually the connection would reset. The second one would buffer the messages, but never had a timeout. So if the TCP session would not establish the messages would buffer till the system ran out of memory. Both of these were reported to the vendors (and I hope fixed), but we stopped trusting TCP syslog implementations.
We drop a collector at major sites, and have everything at that site pointed there using normal syslog. Small sites that don't rate their own collector get pointed to the most reliable major site. Then the systems guys do magic, and pull the files back to the central server which then gets forwarded to different systems. Our SIEM gets a copy, a generic syslog-ng box gets a copy so us geeks can grep stuff, Splunk gets a copy so management types can see pretty graphs, etc. The downside is the central collector is not real time. I think the lag is about five minutes because of the "systems magic" that collects everything. This isn't a big headache as if I am trying to do something in real time I am looking at local logs.
Some other issues you will run into.
1. There is a very common system in use almost everywhere that does not support syslog nativity. Windows. This means installing an agent on all your Windows systems. We all love installing more agents right?
2. Cisco rate limiting syslog. Normally not an issue, but Cisco will log at the rate of the slowest interface it is set to log to. This means if you are logging console syslog will rate limit to 9.6kbps. If you are logging ACLs you can easily exceed this. Easy fix, just do "no logging console". There is a log message that shows up when syslog is dropped because of this. Grep for rate-limit in your Cisco logs.
3. Make sure time is set correctly on everything. It really sucks to try correlating messages when one system is in PST, one EST, and one is not synced to time at all, and is 12 minutes off UTC.
-Otanx
Great pointers and points, thanks. This syslog will only be for the network devices. Server guys can go pound sand if they want to use it for anything. :)
If you only do network devices you are missing out. Lets look at a common issue. Server guy calls and says your firewall is blocking him. Surprisingly he knows the source and destination IP and ports. You look at your logs that only have network devices. No logs, and you tell him it isn't the firewall because there are no denies being logged. However, if you had system logs as well, you might see a log message that says;
"Aug 4 13:23:00 centos kernel: IPTables-Dropped: IN=em1 OUT= MAC=a2:be:d2:ab:11:af:e2:f2:00:00 SRC=192.168.2.115 DST=192.168.1.23 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=9434 DF PROTO=TCP SPT=58428 DPT=443 WINDOW=8192 RES=0x00 SYN URGP=0"
Then you get to point out to the systems guy that it was his own system blocking the connection, and that he will need to fix that. Then you get to ask why you as a network guy are better at troubleshooting system issues than he is. I find that shaming systems guys is the only way to train them to troubleshoot first before blaming the network.
Being serious the more things you can get logs from the better. Otherwise you will be missing important information that may help solve a problem.
-Otanx
I agree 100% about the more devices logged, the better.
In my firm, however, we're so big that the server guys are a tower, network is a tower, client is a tower, etc. Towers everywhere. Backups and antimalware are towers, that's how many towers we got. Getting cross-tower cooperation is culturally... difficult... I know because I'm doing NAC and having to build all those bridges and keep the ones that are built up from burning down.
So, the solution is network device-only. We need the monitoring set up soon, as there's another solution (Orion) about to expire.
Quote from: Otanx on January 23, 2016, 11:21:05 AM
Server guy calls and says your firewall is blocking him. Surprisingly he knows the source and destination IP and ports.
What is this fantastic parallel universe of which you speak? :glitch:
Attached a screenshot off my lab splunk - as you can see its very much a db interface
We will be rolling out Graylog, small one-VM setup to begin with to index our switches and some FWs. Have it running at home and already alerting me for a handful of events.
You can have it listen on different ports for different classes of devices, then create streams, counters and statistics based on every aspect of the log messages. Everything you can configure is just an Elasticsearch syntax which you can copy and paste to other parts of the system. Takes some linux-fu to get it up and running but once the basics are in it's very straightforward.
And it's opensource :professorcat:
Fun twist... looks like I won't be doing a syslog vendor comparison, after all.
Someone didn't just consider Splunk. He executed a PO and bought Splunk. Lots of it. Because, Splunk!
Looks like I get to read a lot of Splunk documentation in the next few days.
:challenge-accepted:
Quote from: jinxer on January 22, 2016, 02:12:00 PM
We/I use syslog-ng and ELK. Its much like splunk just opensource and doesnt have all the pre-made plugins..
Sent from my iPhone using Tapatalk
Neat, we are using similar toolchains to build our own but we use fluentd instead of syslog-ng.
Quote from: deanwebb on January 28, 2016, 06:36:42 PM
Fun twist... looks like I won't be doing a syslog vendor comparison, after all.
Someone didn't just consider Splunk. He executed a PO and bought Splunk. Lots of it. Because, Splunk!
Looks like I get to read a lot of Splunk documentation in the next few days.
:challenge-accepted:
splunk is very memory and CPU intensive, without it, runs like a snail.
Just in case......
https://splunkbase.splunk.com/app/1918/ (https://splunkbase.splunk.com/app/1918/)
:awesome:
Nice to know.
In other news, I just downloaded 25 Splunk PDF manuals. Time to read them Spanish novels!
Quote from: deanwebb on January 28, 2016, 06:36:42 PM
splunk is very memory and CPU intensive, without it, runs like a snail.
hungry like
Intel x86 64-bit chip architecture
2 CPUs, 6 cores per CPU (12 cores total), at least 2 Ghz per core
12 GB RAM