CLOSED: [2020-01-01 Wed 22:42] SCHEDULED: <2020-01-01 Wed> :PROPERTIES: :CREATED: [2019-12-31 Tue 01:29] :ID: 2019-12-31-appendorgheading :END: :LOGBOOK: - State "DONE" from "DONE" [2021-11-19 Fri 08:58] - State "DONE" from "STARTED" [2020-01-01 Wed 22:42] :END: - Updates - 2021-11-19: Note on the frequency of error messages in my current setup. This is an article from a series of blog postings. Please do read [[id:2019-09-25-using-orgmode][my "Using Org Mode Features" (UOMF) series page]] for explanations on articles of this series. In this article, I'm going to explain why and how I moved certain notifications and error logs to Org mode. *** How Logging It Is Usually Done During normal usage of a GNU/Linux system errors and logs are generated. They are located in =/var/log= (or its corresponding [[https://en.wikipedia.org/wiki/Systemd][systemd]] locations). If you are using [[https://en.wikipedia.org/wiki/Cron][cron]] (or its corresponding systemd mechanism) to automate things, any output of these scheduled tasks is sent to the local user's email inbox. The =/var/log= files are good for analyzing a situation. They are not really designed for notifying the user when something is wrong and needs attention. Notification emails that go to the local user's email inbox were fine until more than a decade ago. In former times, people were really using their local email system. Emails from the (email) provider were transferred to the local mail server ([[https://en.wikipedia.org/wiki/Message_transfer_agent][MTA]]) and read by an email program ([[https://en.wikipedia.org/wiki/Mail_user_agent][MUA]]). In this situation, additional emails from the local system were easily seen and handled. Meanwhile, local email inboxes are ignored most of the time. But still, cron and other services are populating those local inboxes with notifications that end up being ignored as well. Some IT specialists might set up a personal central logging server, collecting log events from all kind of services and hosts. With a proper filter and prioritization mechanism, this can be a fine solution. However, it requires knowledge and effort. *** Why I Wanted Something Different For the reasons above, I set up a monthly recurring todo that reminded me to take a look at my local inboxes which is a [[https://en.wikipedia.org/wiki/Polling_(computer_science)][polling]] approach. I skim through all the emails and look, if there is something that needs my personal attention. Typically, I did this via =mutt -f /var/mail/$USER= as well as for the local root. And this for all my hosts. While working on my recurring task "go through the local inboxes and check if there is some issue with my cron jobs" I noticed that many errors get unnoticed in-between those manual checks. And most of the time, the manual checks do not uncover any issue at all. This task is tedious and boring. *When something is tedious and boring, there is always a better way of dealing with it.* So I wanted to look for a different approach, following the [[https://en.wikipedia.org/wiki/Interrupt][interrupt]] concept: whenever there is something happening, I want to get notified right away. *** Usual Desktop Notification Mechanisms For notifying a user on her/his desktop, there are multiple methods. Some of the most frequently used ones are: - Pop-up messages like OS-built-in solutions, [[http://growl.info/][growl]] (macOS, Windows) or [[https://github.com/GNOME/libnotify][libnotify]] (a [[https://developer.gnome.org/notification-spec/][standardized solution]], GNOME/GNU/Linux). - Some kind of bridge to the mobile phone of the user. - Inject email to the (external) email setup of the user. *** The Org Way of Doing Things Being an intense user of Emacs [[id:2018-01-26-orgmode][Org mode]], I was [[id:2014-08-20-org-losses-determining-post-commit][using my agenda to notify me when I accidentally deleted a substantial number of lines]]. Whenever that happened, I created a heading in an =errors.org= file which is part of my agenda. This heading had a title, a link to the git commit where the loss was registered and a time-stamp that recurred every day. This way, I got an agenda line on each day that reminds me of that issue as long as I delete this heading in the =error.org= file. While this worked perfectly well for over five years, I did not understood the potential until recently. My agenda shows everything I want. When I write "everything", [[https://github.com/novoid/Memacs/][I mean it]]. So why not using this "log to the agenda"-idea on a more general basis? Recently, I summarized many handy Python functions that deal with parsing or generating Org mode elements into [[https://github.com/novoid/orgformat][a library]]. I added a new function to this library: =generate_heading()= Then I wrote a decent Python wrapper script for this function as stand-alone tool: [[https://github.com/novoid/appendorgheading][appendorgheading]]. Now I may append arbitrary headings to existing Org mode files. For example, this provides an additional possibility to capture anything into Org mode. For this use-case, I add headings with notifications to my =errors.org= which is part of my agenda files list. #+CAPTION: My errors_misc.org file showing the auto-generated entries for three isues. #+ATTR_HTML: :align center :width 630 :linked-image-width original [[tsfile:2020-01-01T11.12.26 Emacs org mode - errors_misc with expanded details 984x590 -- screenshots publicvoit.png][2020-01-01T11.12.26 Emacs org mode - errors_misc with expanded details 984x590 -- screenshots publicvoit.png]] In case something needs my attention, I see it on my agenda. #+CAPTION: My agenda showing three issues that require attention. #+ATTR_HTML: :align center :width 630 :linked-image-width original [[tsfile:2020-01-01T11.09.14 Emacs org mode agenda with three errors - blurred 914x381 -- screenshots publicvoit.png][2020-01-01T11.09.14 Emacs org mode agenda with three errors - blurred 914x381 -- screenshots publicvoit.png]] Please do note that I applied this method only to selected tasks in my crontab. Those selected tasks generate probably one to three error messages on my agenda per week. Just to give you an idea on my personal situation and the amount of error messages processed by that method. *** Details on My Setup If you want to know how I'm using this right now, here are some examples. Here are three =cron= jobs with stripped paths and inserted line breaks for legibility: #+BEGIN_EXAMPLE 20 4 * * * /vk-cronjob-sherri-memacs-update-collected-git.sh \ > /vk-cronjob-sherri-memacs-update-collected-git.log 2>&1 || \ /usr/local/bin/appendorgheading \ --title "vk-cronjob-sherri-memacs-update-collected-git.sh: return value not zero" \ --filecontent /vk-cronjob-sherri-memacs-update-collected-git.log \ --section "file:/vk-cronjob-sherri-memacs-update-collected-git.sh" 0 6 * * * /vk-cronjob-sherri-appendorgheading-if-file-too-old.sh \ /agenda-export.ics 1 "check with id:2019-11-19-HOWTO-check-and-fix-Org-agenda-radicale" 40 4 * * * /usr/bin/emacs --batch --load /home/vk/.emacs.d/init.el \ --eval '(progn (my-export-agenda))' \ > /vk-cronjob-sherri-emacs-batch-my-export-agenda.log 2>&1 || \ /usr/local/bin/appendorgheading \ --title "emacs --batch my-export-agenda: return value not zero" \ --filecontent /vk-cronjob-sherri-emacs-batch-my-export-agenda.log #+END_EXAMPLE As you can see, they follow the same principle: : || The trick is that following the POSIX concept of [[http://tldp.org/LDP/abs/html/exit-status.html][exit status]], the =appendorgheading= command is only executed when the program does report an error. If the program finishes without any error, =appendorgheading= is not involved at all. As you can see, the command line parameters of my =appendorgheading= are minimal. This can be accomplished by storing my default settings in =$HOME/.appendorgheading= which looks like: #+BEGIN_EXAMPLE [DEFAULT] output = /home/vk/org/errors.org level = 2 daily = True #+END_EXAMPLE In the second example cron job, you might have noticed a slightly different situation. Here, I'm using another wrapper-script for the general case of checking the age of files. I do have several auto-generated files where I want to get notified when something happens to the process and the resulting files are not re-generated for a certain period of time. This is handled via =vk-cronjob-sherri-appendorgheading-if-file-too-old.sh=: #+BEGIN_SRC sh #!/bin/sh set -o errexit FILENAME="${1}" MAXDAYS="${2}" HELPTEXT="${3}" OLD=$(stat -c %Y "${FILENAME}") NOW=$(date +%s) DIFFDAYS=$(( ($NOW - $OLD) / (60*60*24) )) DIFFHRS=$(( ($NOW - $OLD) / (60*60) )) [ $DIFFDAYS -gt $MAXDAYS ] && /usr/local/bin/appendorgheading --title "\"${FILENAME}\" is older than $DIFFDAYS days" \ --quiet \ --section "File is $DIFFHRS hours old: $(ls -la ${FILENAME})\n\n${HELPTEXT}" exit 0 #+END_SRC It takes three command line arguments: 1. The name of the file to check for age. 2. The number of maximum days which is OK for the check. 3. Some kind of description text that is used for the content of the error heading. Earlier I mentioned that I get notified when I remove more than a given threshold of lines in one of my Org mode files. [[id:2014-08-20-org-losses-determining-post-commit][I now updated it with the =appendorgheading= method]] accordingly. I've set up this concept only a few days ago. Since then, I already noticed several issues that went undetected for too long. To me, it's a splendid improvement in my overall concept.