- 1 Logging Services
- 1.1 Collection/Shippers
- 1.2 Cloud
- 1.2.1 sematext
- 1.2.2 LogDNA
- 1.2.3 Stackdriver
- 1.2.4 Loggly
- 1.2.5 Loki
- 1.2.6 LogDevice
- 1.2.7 Splunk
- 1.2.8 NewRelic
- 1.2.9 Graylog
- 1.2.10 Flume
- 1.2.11 Papertrail/Timber.io
- 1.2.12 Scalyr
- 1.2.13 Sumologic
- 1.2.14 sentry.io
- 1.2.15 rollbar
- 1.2.16 CloudWatch
- 1.2.17 DataDog
- 1.2.18 Coralogix
- 1.2.19 Logentries
- 1.2.20 Humio
- 1.2.21 Seq
- 2 Comments & Opinions
1 Logging Services
- Fluentd decouples data sources from backend systems by providing a unified logging layer in between.
- Fluentd's 500+ plugins connect it to many data sources and outputs while keeping its core simple.
- 5,000+ data-driven companies rely on Fluentd. Its largest user currently collects logs from 50,000+ servers.
Gollum is an n:m multiplexer that gathers messages from different sources and broadcasts them to a set of destinations.
Gollum originally started as a tool to MUL-tiplex LOG-files (read it backwards to get the name). It quickly evolved to a one-way router for all kinds of messages, not limited to just logs. Gollum is written in Go to make it scalable and easy to extend without the need to use a scripting language.
1.1.7 Apache Kafka
- From Kubernetes, syslog, or code libraries to REST APIs, LogDNA supports more than 30+ integrations to ingest log data.
- Auto-parse the most popular formats, or request custom parsing from any data source.
- Search, monitor, and analyze logs across multiple deployments, and see results in a single pane.
- Save your search query as a view so you can access it later, much like a shortcut.
- Enterprise-level authentication and custom controls for all team members.
- LogDNA is HIPAA, SOC2, PCI, Privacy Shield and GDPR compliant.
- participated in Y Combinator in 2015 https://newscenter.io/2017/11/former-y-combinator-partners-lead-7-million-series-logdna/
Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.
LogDevice is a scalable and fault tolerant distributed log system. While a file-system stores and serves data organized as files, a log system stores and delivers data organized as logs. The log can be viewed as a record-oriented, append-only, and trimmable file.
LogDevice is designed from the ground up to serve many types of logs with high reliability and efficiency at scale. It's also highly tunable, allowing each use case to be optimized for the right set of trade-offs in the durability-efficiency and consistency-availability space. Here are some examples of workloads supported by LogDevice:
- Write-ahead logging for durability
- Transaction logging in a distributed database
- Event logging
- Stream processing
- ML training pipelines
- Replicated state machines
- Journals of deferred work items
- Humio ingests log data as quickly as it comes, without indexing, and regardless of bursts. Efficiently stored data means you can ingest terabytes of data per day and search it all in a matter of seconds.
- Humio is index-free, and it works with any structured or unstructured data format. Because you don’t need to define fields up front, you can ask any question with live or archived data and experience fast response times.
- The Humio engine was built from scratch to ensure that ingest and search scales to terabytes per day. Humio has virtually no latency even at huge volumes. And with constant focus to optimize the use of infrastructure and storage, Humio requires very few resources.
- Application logs are the most useful data available for detecting and solving a wide range of production issues and outages. Seq makes it easier to pinpoint the events and patterns in application behavior that show your system is working correctly — or why it isn't.
- Seq is built for modern structured logging with message templates. Rather than waste time and effort trying to extract data from plain-text logs with fragile log parsing, the properties associated with each log event are captured and sent to Seq in a clean JSON format. Message templates are supported natively by ASP.NET Core, Serilog, NLog, and many other libraries, so your application can use the best available diagnostic logging for your platform.
- Seq accepts logs via HTTP, GELF, custom inputs, and the seqcli command-line client, with plug-ins or integrations available for .NET Core, Java, Node.js, Python, Ruby, Go, Docker, message queues, and many other technologies.
2 Comments & Opinions
My main advice is avoid ELK. I have no clue how Elastic managed to convince the world that Elasticsearch should be the default log database when it is terrible for logs.
We used to use an ELK cluster but it was always breaking - I'm sure this stuff can be reliable but we just wanted an easy way to search ~300GB of logs (10GB/day)