π

UOMF: My Current Org Mode Files and Heading Structure

Show Sidebar

This is an article from a series of blog postings. Please do read my "Using Org Mode Features" (UOMF) series page for explanations on articles of this series.

Similar to my article explaining my file hierarchy, this long article describes my current Org mode files, how they are named, what main headings they contain and some statistics. If you're not an Emacs Org mode fetishist, this is probably not going to be of high interest to you.

I did not sit down and decided on the structure and names of my Org mode files and heading structure properly. The things I'm going to show you are the current results of a "natural development". It wasn't always the way it is now and it most probably will change in the future as well. Some things are "good enough" to stay like they are despite the fact that I can imagine a better concept. Some things stay as they are because they are difficult to change because of migration cost in form of manual effort.

Therefore, there is no hidden message "this is how it should be done". Far from that. Please do not think that I will convince anybody to do it exactly as it happens to be at my side here.

If you expect a clean setup with consistent patterns, you will be disappointed. I even described stuff that is somewhat embarrassing, considering my interest in PIM and self-organization.

This article is for people who are really curious on how my Org mode files are organized. It's a reference I'll use myself. Maybe I'll do a similar article in a couple of years from now and highlight differences. Maybe this is a one-hit wonder due to the considerable amount of time I had to spend describing all those things. Let's see.

Here is an overview of the headings of this article:

Files in General

Most of my Org mode files are locate within ~/org/. These are my main files (ignoring their archive companions) sorted according to their number of lines, largest on top. The three main sections are a rough separation of large, mid-size and smaller files.

Lines Size Filename Business
126235 5,1M ~/org/notes.org
77699 2,3M ~/org/contacts.org
66021 3,0M ~/org/hardware.org
53570 2,4M ~/org/public_voit.org
45801 2,0M ~/org/misc.org
39329 1,7M ~/org/company2_general.org x
32816 1,6M ~/org/company1_general.org x
26178 1,1M ~/org/company2_role.org x
17485 603K ~/org/company1_product.org x
17198 627K ~/org/projects.org
15290 580K ~/org/references.org
10563 542K ~/org/company3.org x
10024 432K ~/org/finances.org
6594 261K ~/org/lecturer1.org x
4739 202K ~/org/house.org
4249 222K ~/org/issues.org
4054 143K ~/org/archive.org
2731 92K ~/org/yankpad.org
1659 47K ~/org/foodandbeverages.org
969 32K ~/org/yankpad-company2.org x
859 39K ~/org/inbox.org
607 34K ~/org/private.org
28 1,1K ~/org/tags-company3.org x
25 534 ~/org/tags-company3.org x
18 1,5K ~/org/tags-personal.org x
8 176 ~/org/errors.org
5 238 ~/org/errors_public_voit.org
2 112 ~/org/tmp.org

Some file names were renamed here for privacy purpose ("company", "lecturer") or to make it more self-explanatory for you ("house", "finances").

Statistics of the .*\.org files: 

    25221 headings in   568602 total lines
        9586 task headings
        15635 non-task headings
        169 tagged with @KarlVoit
        2019 tagged with "@.+" but not "@KarlVoit"

    1879 open tasks:
        TODO:      1481
        STARTED:   294
        WAITING:   104

    7707 finished tasks:
        CANCELLED: 1004
        DONE:      6703

The archive files are these:

Lines Size Filename Business
118224 4.0M ~/org/misc.org_archive
63251 2.0M ~/org/archive.org_archive
55283 1.9M ~/org/company1_product.org_archive x
53059 1.5M ~/org/inbox.org_archive
41332 1.5M ~/org/projects.org_archive
33573 1.2M ~/org/company1_general.org_archive x
9320 1023K ~/org/issues.org_archive
24958 867K ~/org/house.org_archive
11796 439K ~/org/lecturer.org_archive x
9504 362K ~/org/hardware.org_archive
7828 265K ~/org/company2.org_archive x
8483 225K ~/org/contacts.org_archive
3302 138K ~/org/company2_role.org_archive x
2340 65K ~/org/notes.org_archive
115 3.8K ~/org/public_voit.org_archive
98 2.8K ~/org/finances.org_archive

Statistics of the .*\.org_archive files: 

    17564 headings in   442468 total lines
        12672 task headings
        4892 non-task headings
        6 tagged with @KarlVoit
        2682 tagged with "@.+" but not "@KarlVoit"

    199 open tasks:
        TODO:      151
        STARTED:   23
        WAITING:   25

    12473 finished tasks:
        CANCELLED: 1260
        DONE:      11213

Additionally to that, I've got some Memacs archive files in a sub-directory:

Lines Size Filename Business
525650 113M ~/org/memacs/files.org_archive
403645 26M ~/org/memacs/www.org_archive
300135 7.9M ~/org/memacs/phonelog.org_archive
91818 6.7M ~/org/memacs/tweets.org_archive
69284 2.9M ~/org/memacs/SMS.org_archive
57750 1.9M ~/org/memacs/phonecalls.org_archive
53656 2.8M ~/org/memacs/archive.org_archive
40487 1.8M ~/org/memacs/git.org_archive
35513 969K ~/org/memacs/datebk6.org_archive
33103 8.0M ~/org/memacs/mbox.org_archive
23155 1.4M ~/org/memacs/delicious.org_archive
7630 1.4M ~/org/memacs/news.org_archive
5920 538K ~/org/memacs/bank.org_archive

Their statistics: 

    745766 headings in   1649953 total lines
        1861 task headings
        743905 non-task headings
        0 tagged with @KarlVoit
        0 tagged with "@.+" but not "@KarlVoit"

    219 open tasks:
        TODO:      110
        STARTED:   107
        WAITING:   2

    1642 finished tasks:
        CANCELLED: 1
        DONE:      1641

A few smaller files are located in different locations:

The following sections describe the content of the files.

Non-Org Comments

I'm using Syncthing to synchronize most of my Org files between my devices. On my home computer that runs 24/7, I do have gitwatch configured to commit any change to git version control system.

This way, whenever I save a file on my home computer the changes get committed. Any other device that is online syncs back its Org mode file changes to my home computer which then auto-commits those changes as well.

This way, I do have a full history of all of my Org mode changes back to May 16 2011. As of 2020-04-02, the size of the .git folder is 150 megabytes (almost 70,000 commits). At least this is the size after my weekly git gc invocations. Before this weekly garbage collecting task, the size is usually between 260 and 400 megabytes. Extremely impressive efficiency of git in my eyes.

As a side-mark, this also enables me to detect unusual numbers of deleted content which results in a warning on my agenda. You might be interested in this workflow in case you worry about accidentally deleting collapsed parts of your Org data.

Main Files

So far, I tried to use fewer but larger files. This enables me to mix different types of headings such as notes, todos and blog entries. The common theme is that I try to keep all things that are related to each other at one place independent of their "type".

There are some exceptions to that rule. For example, contact information is managed within contacts.org, even when a person is (only) related to one specific project, event, issue and so forth which do have their spot in different files.

All of my main files are part of my agenda except yankpad, archive, private and tmp.

Some General Comments and the File ~/org/notes.org

This is my personal knowledge base. It's a large Org-based data-base if you will. Other people might use any Org-based Zettelkasten method instead. I might as well try this in the future when I can wrap my head around moving from one big file to many smaller ones.

As I have explained in the section "The Problem With Strict Hierarchies" in this article, the real world can not be mapped into a strict hierarchy. The same holds true for the strict hierarchy of Org mode files and their heading structures. As a workaround, I try to keep it as straight-forward as I can. Additionally, I do maintain a mental map in my brain as a workaround. In general, I do have one single spot for each kind of thing. And if this is not that easy to determine because there are multiple spots that may fit, I am using technical workarounds. Usually, those consist of :id: property links between different sub-hierarchies.

So whenever you wonder, I most probably would be confused as well and decided in my mental model to use one preferred way of handling the situation. Sometimes, I even re-discover large parts of my Org mode hierarchies I have forgotten and - in the worst case - started a different "spot" for this kind of information somewhere else. That's the way it is with this kind of technology and the human brain. Maybe a technology like Remembrance Agents could help here. Some guy who set up a modern version of it did not find out that this is an efficient tool.

These issues apply to many of my Org mode files. Due to the nature of the content of notes.org, these issues appear more often in that file than in my other files.

The main structure (with a few more detailed branches) of my notes.org file with highlighted headings where I spend most of the time:

File ~/org/contacts.org

There is a blog article in my (long-term) blogging pipeline about managing contacts. Until it's done, read this article where I describe how I'm able to assign one single email address to each contact.

The main structure (with a few more detailed branches) of this file with highlighted headings where I spend most of the time:

File ~/org/hardware.org

I can not understand how anybody is able to deal with things without an overview of the past issues/things related to one specific thing. I may look up how I fixed things in the past. It has high value to me.

Please notice the difference to the general notes.org: in hardware.org, everything is about specific instances of something. Here, I'm collecting, e.g., Linux issues for a certain computer in contrast to "how to do X in GNU/Linux". Sometimes, it is hard to differ between them for me. Whenever I'm in doubt, I may create an id: link somewhere else to connect two dots.

File ~/org/public_voit.org

Many blog articles are written and archived here. As you can see in other hierarchy examples, I tend to write blog articles wherever I like. This was one of the main reasons I wrote my own blog software. This way, blog articles related to my bicycle are within hardware.org accordingly. However, there are many articles that do not fit anywhere in my Org mode files. And those articles are located here.

File ~/org/misc.org

Yes, this file is a very large insider-joke to anybody familiar with Personal Information Management. Even I do have some big pile named "miscellaneous".

This file started as something that holds everything that has no place somewhere else. Meanwhile, it may turned out to be more specific. I never cared to find a better name.

Business-Related Files

My business--related files are:

These are the common patterns that developed over time:

When I had to create a separate Org mode file (e.g., for a different role), I had to put common tags in separate files. They were included in all business-related Org files accordingly:

Current approach is to limit myself for one file for the current company/job which has some benefits. For example, I may have one common glossary using radio targets that got linked in all data related to one company.

File ~/org/tags-personal.org

Here, I maintain my set of tags. Not all of my Org files include this file. Basically, it's for notes.org (mainly bookmarks) and the tags for my blog.

I started using those tags before I even understood how tagging works. Therefore, I'm still violating my own rules on how to use tags by having too many pre-defined tags. I guess most of the older ones are not in use by me any more. I'll have to schedule some time to replace the more specific tags with some more general ones, reducing the total number of different tags.

File ~/org/projects.org

This is a fork from misc.org because of a growing amount of content here.

I already described how I define an Org mode project. My projects.org holds all kinds of projects that don't fit somewhere else. Sometimes, they would fit somewhere else but I decided to put them here anyway. I'm not always consistent myself. ;-)

A few examples of the probably hundred unfinished projects of my pipeline:

File ~/org/references.org

I developed this literature/reference management system during writing my PhD thesis, approximately in 2011. It consists of entries like: 

 **** DONE Voit2012b - TagTrees: Improving Personal Information Management using Associative Navigation :PIM:
 CLOSED: [2012-11-18 Sun 14:43]
 :PROPERTIES:
 :CREATED: <2012-09-17 Mon 17:48>
 :ID: Voit2012b
 :END:

 [[bib:Voit2012b][Voit2012b.bib]]
 [[pdf:Voit2012b][Voit2012b.pdf]]

 ***** Abstract

 #+BEGIN_QUOTE
 This dissertation gives an overview of research related to Personal
 Information Management (pim). After discussing some challenges that
 [...]
 #+END_QUOTE

The structure is quite simple. In the following headings, there are headings for each article, paper or book like the one example above:

After finishing my PhD and leaving academia, I did not add much content to the file. A few years ago, I started to add simplified headings for each book I read: 

 **** <2017-11-17 Fri>-<2017-12-03 Sun> Dan Brown: Origin
 :PROPERTIES:
 :CREATED:  [2017-11-18 Sat 15:21]
 :END:

 - read on my mobile
 - awesome thriller!

File ~/org/finances.org

The original (German) file name is finanzen_behoerden_versicherungen.org which may translate to finances_authorities_insurances.org. For the sake of simplicity, I used an abbreviated name in this article.

File ~/org/lecturer1.org

I'm in the lucky position to be able to conduct a lecture on Personal Information Management at the University of Applied Sciences St. Pölten in my spare time. This file holds everything related to this.

It was not until a few weeks ago that this file also contained the script for the students. It is now a separate Org mode document (not described here) within its own git repository. The way it is supposed to be.

File ~/org/house.org

This one started when I organized the building of our house. I was using it more often back then. Large parts are just for reference I guess. Some things got stuck though. Maybe I'm moving large parts to my archive.org file some day.

File ~/org/issues.org

Someday, I though it would be nice to have one spot for all the open software-related issues and bugs. Either they are just personal, e.g., when Emacs is misbehaving in my opinion, or they are official bugs that I reported to open source projects.

In reality, this file contains issues that are not fitting somewhere else. So I introduced the tag issues. All of the projects in this file got this tag. The issues and bugs located somewhere else got this tag as well. This way, I could derive a general bug/issue view. I never do that but I got this option.

The rest of the issues and bugs is mostly located within the corresponding software headings in notes.org or they are sub-headings for specific devices in hardware.org. Whatever suits me.

Another inconsistency of mine. I might get rid of this in future. For now, it does not hurt too bad.

File ~/org/archive.org

Well, this is an empty file. Way larger is the corresponding archive.org_archive which I used to archive all kinds of stuff. Many things from my pre-Org mode era: converted from my great PIM-time with PalmOS, ZIM, old email and Usenet data and so forth.

Furthermore, it contains whole Org mode files that lost its purpose. For example, it contains the content of my phd.org after I finished my PhD thesis. It contains tagstore.org from one of the projects I ran during my PhD. After I had to give up some plans for academia, I moved my postdoc.org to this archive.

I may have used a better name such as cementary.org or good_old_times.org or such. ;-)

File ~/org/yankpad.org

I moved from plain yasnippet to yankpad because most of my text snippets are Org mode snippets anyway. Therefore, I prefer to manage them within an Org mode file.

I do have way more than fifty snippets. Many of them are quite short, some of them are really large. As I mentioned in this article discussing task dependencies, one of my most complex "snippet" is the one I use to organize a lecture. It contains all the tasks I have to do to handle one class in one term. There are probably fifty tasks from setting up the lecture material for the online representation up to sending in the marks. All those tasks are linked via dependencies such that new TODO keywords are added to one to many headings when one task gets done.

Here are some of my short ones:

File ~/org/foodandbeverages.org

Not that you think that I'm a chef, cooking all the time. Quite the opposite. However, there are some recipes for food and cocktails I wanted to write down. Here is the spot for this. Almost never visited.

File ~/org/inbox.org

When I do capture stuff on my Android smartphone mostly from RSS/Atom, Twitter, Mastodon, Signal or mobile browsing, it lands here.

Those things then get either:

I tried to get rid of the last category. So far, I failed. Currently, I do have 248 items that I wanted to look at some point in time. One of the embarrassing parts of my own PIM.

The structure is simple: one level of headings, sorted by time added, most of them consisting of an URL only.

File ~/org/private.org

Everybody has some delicate content that needs to be handled with extra care. Maybe it's your bucket list, your passwords, your voting history or some other stuff you want to make sure that nobody else puts an eye onto.

I'm using :crypt: tags within the other files as well. Somehow, stuff landed in this file. It is not synchronized to all other devices and contains a flat list of headings that each do have the :crypt: tag.

File ~/org/errors.org

This is one of my newest Org mode agenda files.

After publishing a Python library to generate Org mode content, I wrote appendorgheading which exposes this functionality for easy usage from within shell scripts. While that being very general, I came up with a specific purpose for my daily life: I used this to switch some log- or notification workflows to Org mode.

This is an empty file, except for a brief header comment and the main heading entry. If one of my cron-jobs fail, this gets populated with a notification (a time-stamp that is recurring daily), description on the problem, optional error message output and a link to Org mode headings with directions on how to fix this topic if I already collected information on frequent issues.

I do love this workflow.

Writing those lines, I recognized that this file holds a new heading that informs me that the agenda export failed since more than one day. I'll handle this right now. Usually this issue is solved by fixing a broken :id: link or similar.

File ~/org/errors_public_voit.org

Similar to the errors.org file, this file gets populated with errors that occur when my blog is re-generated. Since I wrote my own blog software, I'm limiting myself to the Org mode syntax supported by it. This is why it is very rare that I get errors.

In future, this may be merged with errors.org.

File ~/org/tmp.org

This is an Org mode file which holds headings that are just for testing stuff. I may remove all of them any time. I consider this as my scratch-buffer but for Org mode.

Memacs Files

Memacs is a platform I wrote for getting non-Org mode stuff into Org mode. Each data source is associated with one Memacs module. The resulting data typically is written into Org mode archive files in order not to clutter my agenda.

Here are my personal active Memacs files that get re-created by cron-jobs:

File ~/org/memacs/files.org_archive

Using the filenametimestamps module, a daily snapshot of all of my files that start with an ISO date-stamp is generated. It looks like that: 

 * Memacs for file name time stamp          :Memacs:filedatestamps:
 ** <2019-06-03 Mon 22:15> [[file:/home/user/institutions/telco/2019-06-03 phone bill.pdf][2019-06-03 phone bill.pdf]]
 ** <2007-03-03 Sat 00:27> [[file:[...]/2007/2007-03-02 CLT07/2007-03-03T00.27_img_0475 at the Pub.jpg][2007-03-03T00.27_img_0475 at the Pub.jpg]]

Those lines provide a lookup-table from file base-names to their whole and absolute path. This enables me to link to files independent of the directory they are stored or include them into my blog using a similar mechanism that is independent of the path. All of this is described on this page.

I consider this one of my killer features when using Org mode.

File ~/org/memacs/www.org_archive

This is a meanwhile old artifact from the time where I was able to use a Firefox add-on that stored all web content on my file system. In other words, this stored everything I browsed the web for into HTML files, images and other files onto my disk.

Following the Quantified self approach and the ideas behind Memacs, I got a full history of my web activity. Not only in form of links but in form of actual content. I was able to compare how certain web pages changed over time and a web page going offline was not an issue to me. The answer to the question that you've got in your mind right now is: two to four Gigabytes per month. A very reasonable figure, considering the cost of storage these days. If you run the numbers on nowadays disks, this is a bit over one single Euro per year of storage cost (neglecting backup space).

For practical reasons, I decided to ran the filenametimestamp module separately for this sub-hierarchy to get a separate index.

With the major changes in the add-on API interface in 2017, I've lost this add-on and did not find a replacement yet.

Therefore, this is a static archive of my 2012 to 2017 complete browser history.

File ~/org/memacs/phonelog.org_archive

This file gets generated by the simplephonelogs module. My Android phone got smart with Tasker. Using this framework, I can now easily log events on my Android device into a simple text file. This file then gets converted and interpreted by Memacs.

With this workflow, my agenda gets populated with lots of interesting context information:

I also created a simple Android UI to quickly capture arbitrary log text to this workflow.

File ~/org/memacs/tweets.org_archive

I periodically download my Twitter data-set using twython and process it with the Twitter module. Still on my todo list: a similar workflow for Mastodon.

I just realized that I did not execute this process for quite some time due to high manual effort.

File ~/org/memacs/SMS.org_archive

Before I switched to Signal, I was using SMS Backup and Restore and this module to generate the corresponding Org mode file. Signal backups are encrypted which is great but also jeopardized my workflow here.

With signal-back I found a manual solution to decrypt the backup data. Until I do not implement a Memacs module for this result, this file is a static archive of my mobile messages.

File ~/org/memacs/phonecalls.org_archive

With the phonecalls module, I converted the Call Logs Backup & Restore archive to Org mode files.

I just realized that I did not execute this process for quite some time.

File ~/org/memacs/git.org_archive

The Git module provides me all of my Git commits on my agenda.

File ~/org/memacs/datebk6.org_archive

This static archive holds many years of data from DateBk6 from PalmOS. This was an amazing piece of great PIM software I used way longer than PalmOS was a large thing. I more or less switched directly from PalmOS DateBk6 to Org mode in 2011.

Back then, I wrote a Python script that converted my PIM data into Org mode format for archiving purposes.

File ~/org/memacs/mbox.org_archive

Mbox is a storage format for emails. On a monthly basis, I convert the emails that are older than fourteen months from my usual maildir to mbox format in order to archive them. Using this non-standard Memacs module, I convert it to this Org mode file.

The module is non-standard because it's a quick hack without using the usual framework functions of Memacs. Somebody should move it to the Memacs framework some day. ;-)

File ~/org/memacs/delicious.org_archive

Delicious once was the perfect solution for web bookmarks. Unfortunately, the service entered a long and painful demise. So I had to switch to my current Org mode based bookmark method. The data from the period before is archived here via the RSS module.

File ~/org/memacs/news.org_archive

Fortunately, there is still a more or less active Usenet community at our local University of technology which I'm part of. Since my news client uses the same mbox standard format as mentioned above to store its postings, I am using the same Memacs module to derive this file.

File ~/org/memacs/bank.org_archive

Another non-standard Memacs module is the one for my bank statements. I regularly have to download CSV files from my net-banking site. This module then converts all of my banking events to Org mode.

Concluding Remarks

If you have read through the whole article, I think you must be either insane or really dedicated to Org topics. (No offense.)

I'm very interested in your lessons learned (if any). So please leave a comment below so that I might blog about your favorite workflows in a separate article.

Related articles that link to this one:

Comment via email (persistent) or via Disqus (ephemeral) comments below: