Firebird Flight Wiki: Logging & Log Analysis
Firebird Flight

Logging & Log Analysis

Logs provide critical insight into how the system is functioning and can point to issues such as application crashes, security breaches, or resource bottlenecks. By understanding the logging infrastructure and exploring key log files, system administrators can quickly diagnose problems and enhance security.

Contents
  1. Introduction
  2. rsyslog & systemd-journald
  3. Key Log Files
  4. Log Rotation & Retention
  5. Log Analysis Tools
  6. Conclusion

1. Introduction

Linux provides multiple mechanisms for logging, including the rsyslog daemon and systemd-journald. Depending on the distribution and version, logs might be stored in traditional text files under /var/log or within a journal managed by systemd. Understanding where logs are kept and how to parse them is essential for troubleshooting performance, security, and application-level issues.

2. rsyslog & systemd-journald

In many modern systems, both rsyslog and systemd-journald work together. By default, journald captures logs in a binary journal, and rsyslog may forward them into plain-text files under /var/log.

rsyslog

rsyslog is a Syslog daemon that can process, filter, and route system logs. Its config resides in /etc/rsyslog.conf and /etc/rsyslog.d/. Common log streams include:

  • /var/log/syslog (Debian/Ubuntu) or /var/log/messages (Red Hat/CentOS).
  • /var/log/auth.log: Authentication-related messages.
  • /var/log/kern.log: Kernel log messages.

You can forward logs to remote servers or filter by severity (debug, info, warning, etc.).

systemd-journald

systemd-journald collects logs from the kernel, early boot, and systemd units. It stores them in binary form under /run/log/journal (volatile) or /var/log/journal (persistent), viewable with journalctl.

  • journalctl -b shows logs since last boot.
  • journalctl -u [service] shows logs for a specific systemd service.
  • journalctl --vacuum-size=[size] can limit the journal size for retention.

3. Key Log Files

Even with journald in the background, many distros still maintain plain-text logs in /var/log for convenience. Here are a few important examples:

Log File Purpose / Content
/var/log/syslog or /var/log/messages Catches a wide range of system messages (e.g., general daemons, non-critical info).
/var/log/auth.log Authentication logs (login, sudo, SSH) – invaluable for security auditing.
/var/log/kern.log Records kernel messages, which can point to hardware or driver issues.
/var/log/dmesg Boot and kernel ring buffer messages, similar to dmesg output.
/var/log/apache2/ or /var/log/nginx/ Web server logs (access and error), crucial for diagnosing HTTP-related issues.
Table 1: Common Plain-Text Log Files

Additionally, other services (e.g., MySQL, Postfix) typically maintain logs in /var/log, sometimes under subdirectories named after the service.

4. Log Rotation & Retention

Without periodic rotation, log files can grow indefinitely and consume all disk space. Linux systems often rely on logrotate to rotate and compress logs on a schedule specified in /etc/logrotate.conf and /etc/logrotate.d/ entries.

Basic logrotate Usage

Each service or system log file typically has a corresponding config that defines:

  • How frequently to rotate (daily, weekly, monthly).
  • How many old copies to keep (e.g., rotate 4 logs, then delete the oldest).
  • Whether to compress rotated logs (compress directive).
  • Post-rotate actions (e.g., service reload).

Example snippet from /etc/logrotate.d/syslog might look like: 

/var/log/syslog {
  daily
  rotate 7
  compress
  missingok
  postrotate
    /usr/bin/systemctl reload rsyslog >/dev/null 2>&1 || true
  endscript
}

This means /var/log/syslog will rotate daily, keep 7 copies, compress older logs, and reload rsyslog after rotation to start a fresh file.

5. Log Analysis Tools

Searching or monitoring logs manually with grep and tail is common, but bigger environments benefit from advanced analytics or indexing solutions. A few examples:

grep, awk, sed

The classic command-line trifecta for filtering and transforming text, invaluable for quick, custom analysis:

  • grep "error" /var/log/syslog finds lines containing "error".
  • awk '{print $5}' /var/log/auth.log extracts the 5th space-delimited field from each line.
  • sed -n '10,20p' /var/log/syslog prints lines 10 through 20 only.

logwatch or goaccess

logwatch parses system logs and emails a summary of notable events daily, while goaccess provides a terminal-based or web-based dashboard for analyzing web server logs (e.g., Nginx, Apache).

Centralized Logging / SIEM

Larger deployments may forward logs to Elasticsearch, Splunk, or Greylog for indexing and search, or use a Security Information & Event Management (SIEM) platform to correlate events across multiple systems.

Elastic Stack is a respected and well-known tool. Your familiarity with it would be greatly helpful to solve some of the challenges soon to be implemented!

6. Conclusion

Whether you’re reading logs directly from /var/log, using journalctl, or analyzing data in a full-fledged SIEM, effective log management is essential for diagnosing issues and maintaining system health. By learning how logs are rotated, where they’re stored, and which tools can parse them, you’ll be well-equipped to handle both routine troubleshooting and detailed forensics when something goes wrong.

For more advanced logging strategies, consider implementing log shipping to a centralized server or exploring solutions like the ELK stack (Elasticsearch, Logstash, Kibana) for deeper search and visualization capabilities.