UOMF: Logging With Emacs Org Mode

• Updates
  • 2021-11-19: Note on the frequency of error messages in my current setup.

This is an article from a series of blog postings. Please do read my "Using Org Mode Features" (UOMF) series page for explanations on articles of this series.

In this article, I'm going to explain why and how I moved certain notifications and error logs to Org mode.

During normal usage of a GNU/Linux system errors and logs are generated. They are located in /var/log (or its corresponding systemd locations). If you are using cron (or its corresponding systemd mechanism) to automate things, any output of these scheduled tasks is sent to the local user's email inbox.

The /var/log files are good for analyzing a situation. They are not really designed for notifying the user when something is wrong and needs attention.

Notification emails that go to the local user's email inbox were fine until more than a decade ago. In former times, people were really using their local email system. Emails from the (email) provider were transferred to the local mail server (MTA) and read by an email program (MUA). In this situation, additional emails from the local system were easily seen and handled.

Meanwhile, local email inboxes are ignored most of the time. But still, cron and other services are populating those local inboxes with notifications that end up being ignored as well.

Some IT specialists might set up a personal central logging server, collecting log events from all kind of services and hosts. With a proper filter and prioritization mechanism, this can be a fine solution. However, it requires knowledge and effort.

For the reasons above, I set up a monthly recurring todo that reminded me to take a look at my local inboxes which is a polling approach. I skim through all the emails and look, if there is something that needs my personal attention. Typically, I did this via mutt -f /var/mail/$USER as well as for the local root. And this for all my hosts.

While working on my recurring task "go through the local inboxes and check if there is some issue with my cron jobs" I noticed that many errors get unnoticed in-between those manual checks. And most of the time, the manual checks do not uncover any issue at all. This task is tedious and boring. When something is tedious and boring, there is always a better way of dealing with it.

So I wanted to look for a different approach, following the interrupt concept: whenever there is something happening, I want to get notified right away.

For notifying a user on her/his desktop, there are multiple methods. Some of the most frequently used ones are:

• Pop-up messages like OS-built-in solutions, growl (macOS, Windows) or libnotify (a standardized solution, GNOME/GNU/Linux).
• Some kind of bridge to the mobile phone of the user.
• Inject email to the (external) email setup of the user.

Being an intense user of Emacs Org mode, I was using my agenda to notify me when I accidentally deleted a substantial number of lines. Whenever that happened, I created a heading in an errors.org file which is part of my agenda. This heading had a title, a link to the git commit where the loss was registered and a time-stamp that recurred every day. This way, I got an agenda line on each day that reminds me of that issue as long as I delete this heading in the error.org file.

While this worked perfectly well for over five years, I did not understood the potential until recently.

My agenda shows everything I want. When I write "everything", I mean it. So why not using this "log to the agenda"-idea on a more general basis?

Recently, I summarized many handy Python functions that deal with parsing or generating Org mode elements into a library. I added a new function to this library: generate_heading()

Then I wrote a decent Python wrapper script for this function as stand-alone tool: appendorgheading. Now I may append arbitrary headings to existing Org mode files. For example, this provides an additional possibility to capture anything into Org mode.

For this use-case, I add headings with notifications to my errors.org which is part of my agenda files list.

In case something needs my attention, I see it on my agenda.

Please do note that I applied this method only to selected tasks in my crontab. Those selected tasks generate probably one to three error messages on my agenda per week. Just to give you an idea on my personal situation and the amount of error messages processed by that method.

If you want to know how I'm using this right now, here are some examples.

Here are three cron jobs with stripped paths and inserted line breaks for legibility:

20 4 * * * <PATH>/vk-cronjob-sherri-memacs-update-collected-git.sh \
     > <PATH>/vk-cronjob-sherri-memacs-update-collected-git.log 2>&1 || \
     /usr/local/bin/appendorgheading \
        --title "vk-cronjob-sherri-memacs-update-collected-git.sh: return value not zero" \
        --filecontent <PATH>/vk-cronjob-sherri-memacs-update-collected-git.log \
        --section "file:<PATH>/vk-cronjob-sherri-memacs-update-collected-git.sh"

0 6 * * * <PATH>/vk-cronjob-sherri-appendorgheading-if-file-too-old.sh \
     <PATH>/agenda-export.ics 1 "check with id:2019-11-19-HOWTO-check-and-fix-Org-agenda-radicale"

40 4 * * * /usr/bin/emacs --batch --load /home/vk/.emacs.d/init.el \
     --eval '(progn (my-export-agenda))' \
     > <PATH>/vk-cronjob-sherri-emacs-batch-my-export-agenda.log 2>&1 || \
     /usr/local/bin/appendorgheading \
         --title "emacs --batch my-export-agenda: return value not zero" \
         --filecontent <PATH>/vk-cronjob-sherri-emacs-batch-my-export-agenda.log	  

As you can see, they follow the same principle:

 <a program> <re-directing output to a file> || <call appendorgheading>	  

The trick is that following the POSIX concept of exit status, the appendorgheading command is only executed when the program does report an error. If the program finishes without any error, appendorgheading is not involved at all.

As you can see, the command line parameters of my appendorgheading are minimal. This can be accomplished by storing my default settings in $HOME/.appendorgheading which looks like:

[DEFAULT]
output = /home/vk/org/errors.org
level = 2
daily = True	  

In the second example cron job, you might have noticed a slightly different situation. Here, I'm using another wrapper-script for the general case of checking the age of files. I do have several auto-generated files where I want to get notified when something happens to the process and the resulting files are not re-generated for a certain period of time. This is handled via vk-cronjob-sherri-appendorgheading-if-file-too-old.sh:

#!/bin/sh
set -o errexit

FILENAME="${1}"
MAXDAYS="${2}"
HELPTEXT="${3}"

OLD=$(stat -c %Y "${FILENAME}")
NOW=$(date +%s)
DIFFDAYS=$(( ($NOW - $OLD) / (60*60*24) ))
DIFFHRS=$(( ($NOW - $OLD) / (60*60) ))

[ $DIFFDAYS -gt $MAXDAYS ] && /usr/local/bin/appendorgheading --title "\"${FILENAME}\" is older than $DIFFDAYS days" \
        --quiet \
        --section "File is $DIFFHRS hours old: $(ls -la ${FILENAME})\n\n${HELPTEXT}"
exit 0	  

It takes three command line arguments:

1. The name of the file to check for age.
2. The number of maximum days which is OK for the check.
3. Some kind of description text that is used for the content of the error heading.

Earlier I mentioned that I get notified when I remove more than a given threshold of lines in one of my Org mode files. I now updated it with the appendorgheading method accordingly.

I've set up this concept only a few days ago. Since then, I already noticed several issues that went undetected for too long. To me, it's a splendid improvement in my overall concept.