What’s new in RHEL7: Persistent journal log

Systemd journal logs are by default not persistent. They are stored in memory (kernel ring buffer) are /run/log/journal. This means, journal logs are available only from the current boot.

# journalctl --list-boots
0 f27172ee431b4012af52b7623468e2fc Fri 2016-10-21 06:16:53 CDT—Fri 2016-10-21 06:17:22 CDT

Logs from previous boot are not available. However, since journal logs are forward to rsyslog are default, you can still see journal logs from previous boot from /var/log/messages. You will not be able to view logs from previous boots using the journalctl command. Continue reading


What’s new in RHEL7: Viewing the journal

In a Linux distro using systemd such as RHEL7, you can view the logs by rumning journalctl.

# journalctl
-- Logs begin at Sun 2016-09-25 00:00:38 CDT, end at Wed 2016-10-12 05:37:48 CDT. --
Sep 25 00:00:38 server.example.com systemd-journal[232]: Runtime journal is using 8.0M (max allowed 802.0M, trying to leave 1.1G free
Sep 25 00:00:38 server.example.com systemd-journal[232]: Runtime journal is using 8.0M (max allowed 802.0M, trying to leave 1.1G free
Sep 25 00:00:38 server.example.com kernel: Initializing cgroup subsys cpuset
Sep 25 00:00:38 server.example.com kernel: Initializing cgroup subsys cpu

The output will be very similar to what you see when you run “cat /var/log/messages”. You can scroll up and down and search by typing / because journalctl uses less pager. You will notice that errors are highlighted in red. Continue reading

What’s new in RHEL7: The Journal

Applications and daemons in Linux will either send logs to syslog or write directly to log files (Apache does that). For years, various implementations of syslog (rsyslog, syslog-ng, sysklogd) have been the sole logging service. In RHEL7, with the rise of Systemd, there’s a new logging service which is a part of Systemd called Journal and Journald being its daemon. Rsyslog and Journal co-exist in RHEL7 and they can write to reach other.

The Journal is:

If you are wondering what the journal is, here’s an explanation in a few words to get you up to speed: the journal is a component of systemd, that captures Syslog messages, Kernel log messages, initial RAM disk and early boot messages as well as messages written to STDOUT/STDERR of all services, indexes them and makes this available to the user. It can be used in parallel, or in place of a traditional syslog daemon, such as rsyslog or syslog-ng. For more information, see the initial announcement.

Syslog stores messages from applications as is in text files. If the application is compromised and the hacker manipulates the message, syslog will store them as it is. Journal appends metadata (_PID, _UID) about the sending application to the incoming messages and stores in binary format. Therefore you can always trust what’s in the Journal. Also Journal log is structured (stored as key, value) and indexed which means searching and filtering is fast. You view syslog messages using traditional Unix utilities like cat, less. To view Journal logs, you need an utility called journalctl.

When you run “systemctl status service-name” you will see a couple lines of logs about the service. This is comes from the journal.

To learn more about Journal:

Stale Virtual Machine Swapfile location in ESXi

While attempting to configure Virtual Machine Swapfile location in ESXi, I get an error
The object has already been deleted or has not been completely created“.

In vCenter, after navigating to Configuration-> Virtual Machine Swapfile location, the “Edit” is greyed out disallowing me to configure the swapfile location.

The problem is, it was pointing to a datastore that is no longer mounted.
Ssh to the host:

~ # grep -i HostLocalSwapDir  /etc/vmware/esx.conf
/adv/Mem/HostLocalSwapDir = "/vmfs/volumes/0391a5b2-65784bc7"
/adv/Mem/HostLocalSwapDirEnabled = "1"
~ # ls -l /vmfs/volumes/0391a5b2-65784bc7
ls: /vmfs/volumes/0391a5b2-65784bc7: No such file or directory

The host has been re-purposed from a different cluster and its datastores had been unmounted, including the swap datastore. To fix this

1) vi /etc/vmware/esx.conf and I replaced the stale datastore with the new swap datastore. This did not resolve the issue

2) Next I restarted hostd “/etc/init.d/hostd restart” after which I could configure the swapfile location from the vCenter GUI.