adamflott.com

Adamantium - II - Logging Ecoysystem

Date: 2020-02-04

1 Collection/Shippers

Standalone (e.g. not a programming library) log collectors/shippers.

1.1 Filebeat

1.1.1 Statements

From https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html

1.1.2 Observations

1.2 fluentd

1.2.1 Statements

1.2.2 Observations

From https://logz.io/blog/fluentd-vs-fluent-bit/

1.3 fluentbit

1.4 Logagent

1.5 Logstash

1.6 rsyslog

1.7 Vector

https://vector.dev/

1.7.1 Statements

1.7.2 Observations


2 Programming Language Collection/Shippers

Specific libraries for various programming libraries that can do interesting things with collecting or shipping logs.

2.0.1 Gollum

Gollum is an n:m multiplexer that gathers messages from different sources and broadcasts them to a set of destinations.

Gollum originally started as a tool to MUL-tiplex LOG-files (read it backwards to get the name). It quickly evolved to a one-way router for all kinds of messages, not limited to just logs. Gollum is written in Go to make it scalable and easy to extend without the need to use a scripting language.

3 Generalized Collection

3.1 Apache Kafka


4 Cloud Services

4.0.1 sematext

4.0.2 LogDNA

4.0.2.1 Statements

From https://logdna.com/technology/

4.0.2.2 Misc

4.0.3 Stackdriver

4.0.4 Loggly

4.0.5 Loki

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

4.0.6 LogDevice

LogDevice is a scalable and fault tolerant distributed log system. While a file-system stores and serves data organized as files, a log system stores and delivers data organized as logs. The log can be viewed as a record-oriented, append-only, and trimmable file.

LogDevice is designed from the ground up to serve many types of logs with high reliability and efficiency at scale. It's also highly tunable, allowing each use case to be optimized for the right set of trade-offs in the durability-efficiency and consistency-availability space. Here are some examples of workloads supported by LogDevice:

4.0.7 Splunk

4.0.8 NewRelic

4.0.9 Graylog

4.0.10 Flume

4.0.11 Papertrail/Timber.io

4.0.12 Scalyr

4.0.13 Sumologic

4.0.14 sentry.io

4.0.15 rollbar

4.0.16 CloudWatch

4.0.17 DataDog

4.0.18 Coralogix

4.0.19 Logentries

4.0.20 Humio

4.0.20.1 Statements

4.0.21 Seq

4.0.21.1 Statements

4.0.22 Logz.io

4.0.23 Honeycomb


4.1 Specific

4.1.1 ElasticSearch

4.1.1.1 https://news.ycombinator.com/item?id=21952254

My main advice is avoid ELK. I have no clue how Elastic managed to convince the world that Elasticsearch should be the default log database when it is terrible for logs.

4.1.1.2 https://news.ycombinator.com/item?id=21951804

We used to use an ELK cluster but it was always breaking - I'm sure this stuff can be reliable but we just wanted an easy way to search ~300GB of logs (10GB/day)

5 TODO

https://www.tibco.com/resources/datasheet/tibco-loglogic-log-management-intelligence