Log Analysis with journald & syslog

Logs are the memory of a Linux system. If a server was attacked, the evidence is almost always first visible in its logs: failed logins, service restarts, cron jobs running where none existed, unexpected commands in the shell history. Knowing where logs live and how to query them is the single highest-leverage skill in Linux forensics.

The modern logging landscape

Two systems coexist on almost every modern distribution:

• systemd-journald — captures everything from services that log to stdout/stderr, stores it in a binary, indexed journal under /var/log/journal/, and by default keeps it in memory only if the directory doesn't exist.
• rsyslog / syslog-ng — the classic text log daemon, writing /var/log/messages, /var/log/syslog, /var/log/auth.log, and more. On many systems it also forwards journal entries into text files.

The classic /var/log

Rotated files carry numeric suffixes (auth.log.1, .2.gz) — always read them too. Attackers frequently clear the current logs but miss the rotated archives.

Querying the systemd journal

# all boot-time entries since the last boot
journalctl -b

# logs for a specific unit
journalctl -u sshd

# since a specific time
journalctl --since "2026-08-10 00:00" --until "2026-08-10 06:00"

# kernel messages only
journalctl -k

# export format - useful for analysis tooling
journalctl -o export -b -1 > prev_boot.export

# a service's stdout/stderr (e.g. a web app)
journalctl -u nginx.service --no-pager

From a dead image, you can point journalctl at the copied journal directory:

cp -a /mnt/image/var/log/journal /evidence/journal
journalctl --directory=/evidence/journal -u sshd --no-pager

Authentication & compromise clues

Some of the highest-value patterns to grep for:

# brute force attempts
grep "Failed password" /evidence/auth.log* | awk '{print $1,$2,$3,$11}' | sort | uniq -c | sort -rn | head

# successful root logins / logins as unusual users
grep -E "Accepted (password|publickey)" /evidence/auth.log* | grep -v "from 10\.\|from 192\.168\." 

# sudo usage
grep "sudo" /evidence/auth.log* | grep -iE "command|COMMAND" | tail -50

# session openings / new users added
grep -E "session opened|useradd" /evidence/auth.log*

# new cron entries (a classic persistence method)
grep -iE "cron.*INSTALL|NEW" /evidence/syslog* /evidence/cron*

Extracting logs from an image

Don't boot the image. Mount read-only, or pull files with The Sleuth Kit:

# find the /var/log directory inode
fls -rd /evidence/image.dd -f ext4 -p /var/log | head -40

# dump auth.log with icat (replace NNNN with the inode)
icat -f ext4 /evidence/image.dd NNNN > /evidence/auth.log

# or use loop mount read-only (kernel must support ext4)
mount -o ro,loop,noexec,nodev /evidence/image.dd /mnt/img
cp -a /mnt/img/var/log /evidence/var_log
umount /mnt/img

A log analysis checklist

1. Inventory what log sources exist: journal directory, /var/log files, rotations, wtmp/btmp.
2. Establish the time baseline: system clock, timezone, time of suspected incident.
3. Look at authentication first: brute force, new users, password changes, root logins.
4. Look for persistence: cron, systemd units, rc.local, authorized_keys changes.
5. Look for data exfiltration or lateral movement: scp/sftp sessions, unusual outbound connections.
6. Check for log tampering: gaps in time, empty auth.log where content is expected, journal cleared.

← Previous: Recovering Deleted Files on ext4  ·  Next: Memory Forensics with Volatility →