Logs: journalctl, rsyslog and a persistent journal

Two log systems working together

On RHEL 9, systemd-journald receives log messages from the kernel, from services' output, and from programs using syslog. It stores them in a structured, indexed journal. rsyslog then reads from the journal and writes classic text files in /var/log. Both are useful: journalctl has powerful filters, and text files are easy to grep and to ship to a central log server.

Reading the journal

$ journalctl -u sshd                   # one service
$ journalctl -u httpd --since "1 hour ago"
$ journalctl -p err -b                 # errors and worse, this boot
$ journalctl -f                        # follow, like tail -f
$ journalctl -n 20                     # last 20 lines
$ journalctl --since "2026-10-19 09:00" --until "2026-10-19 09:30"
$ journalctl _PID=1301                 # one process
$ journalctl -b -1                     # the previous boot (needs a persistent journal)
$ journalctl -o verbose -n 1           # every field of one entry

Priorities, from most to least serious: 0 emerg, 1 alert, 2 crit, 3 err, 4 warning, 5 notice, 6 info, 7 debug. -p err shows level 3 AND everything more serious (0 to 3). journalctl only shows everything to root and to members of the wheel or systemd-journal groups; others see only their own messages.

Making the journal persistent

By default the journal setting is Storage=auto: journald writes to /var/log/journal if that directory exists, otherwise only to /run/log/journal, which is in memory and lost at every reboot. Without a persistent journal you cannot read logs from before the last reboot, which is exactly when you need them after a crash. Check with ls /var/log/journal.

# mkdir -p /var/log/journal
# systemctl restart systemd-journald
# journalctl --list-boots
IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
 -1 4b1c9e2d3f...                    Sun 2026-10-18 09:01:02 IST Sun 2026-10-18 23:59:10 IST
  0 9a7f6e5d4c...                    Mon 2026-10-19 09:00:58 IST Mon 2026-10-19 11:05:31 IST

Persistent journals are limited in size (by default up to 10% of the file system, with a cap). journalctl --disk-usage shows the size, and journalctl --vacuum-time=2weeks removes older entries. Classic text files are rotated by logrotate, configured in /etc/logrotate.conf and /etc/logrotate.d/.

Writing your own log lines

$ logger -p user.notice "Backup finished: 12 GB"
$ journalctl -n 1
Oct 19 11:10:02 web01 asha[7720]: Backup finished: 12 GB

Common mistakes

• Looking for yesterday's crash with journalctl -b -1 and getting "Specifying boot ID or boot offset has no effect, no persistent journal was found." Make the journal persistent before you need it.
• Reading only /var/log/messages. Login problems are in /var/log/secure; SELinux denials are in /var/log/audit/audit.log.
• Forgetting that time filters use the server's time zone. Check with timedatectl when comparing with users' reports.