Hi all – I'm looking at logging issues for the first time, been reading up on syslog. Is there a log format tailored for firewalls and IDS/IPS? Syslog seems like an old format designed for any device, servers, etc, not specific to networks or firewalls.
Are there any binary formats? Log bloat looks amazingly bad with these text and JSON logs. It would be neat to have an efficient binary format that renders as text as needed.
For example, I don't want to store an IP address as the plain text `168.212.226.204`. That can just be 4 bytes – four 8-bit unsigned integers – not 15 bytes (12 for the numerals, three for the periods). The periods can be inferred by the type definition.
The status codes, severity, ports, etc. can be one byte each, or even just a few bits. Firewall rule number could be two bytes tops. Yes, the text formats have lots of repeats like IPs and protocols, and standard compressors like gzip or Zstd will efficiently compress those repeated strings, but I don't see any reason to default to the bloated form, the binary form will have the same repeats (just shorter), and the compression is a non-trivial compute load. The larger the input data, the more resources the compression codec will need. Anyway, text data just annoys me when it's not necessary.
What else is there besides syslog? I know about rsyslog, syslog-ng, and NXlog, but I think those are tools, not formats. I think they all emit syslog, or consume syslog, and they just have richer config options or something. I know a little about Graylog, but I bailed when they mentioned JSON. It's like everyone thinks disk is free.
Thanks.
No comments:
Post a Comment