π

A Draft Workflow for Advanced Project Management Using Org Mode and Org Edna

Show Sidebar

Updates:

I'm currently dreaming of a workflow that allows complex project management within Org mode. Please do read my article how I define projects within Org mode. I won't repeat those things in this article here.

So far, I was using task dependencies via org-depend, writing manual :TRIGGER: and :BLOCKER: properties. This works perfectly fine. However, when you want to use more advanced features, you need to look for something different. For example, I sometimes want to schedule a task on the day I finalize a related one. Or I want to schedule a follow-up task three working-days after marking a specific task as done. Org-depend only allows to set todo keywords and has no functionality to control scheduled dates or other advanced stuff.

With having defined task dependencies explicitly, a full-blown project management workflow is more than just a nice to have. For example, I generate all tasks related to organizing and conducting a lecture via a very long, complex and static yankpad template. This way, I only have to maintain the snippet template and get all tasks for a semester with all of their dependencies using a short keyboard command.

When you can not apply a static template to your project, you have to define the dependencies of the tasks manually. Using an advanced (and more complex) syntax, this is a tedious and error-prone task to do. Therefore, I want to automate and semi-automate as much as possible so that project management does not have to deal with manually write cumbersome property syntax.

The Basic Concept (My Requirements)

These are the basic concepts I want to follow when managing a project using task dependencies:

  1. Manual effort for defining dependencies should be minimized.
  2. Next tasks that might be worked on within a project are very easily spotted. Only actionable tasks are visible on your agenda.
    • Compare to the Getting Things Done method and how it is emphasizing the importance of being able to spot the next task to work on.
    • A task is only scheduled and only has a TODO keyword when it is ready to be considered in your workflow.
      • See exceptions due to current technical limitations.
    • This requires task dependencies being explicitly defined using Org Edna:
      • :BLOCKER: properties are redundant and optional: I like to use them to visualize dependent tasks.
      • :Effort: properties are optional but handy for deriving (Gantt) charts and clocking.
      • Multiple entries for :BLOCKER: and :TRIGGER: properties are merged to one single line. In real world, you will likely use M-x org-edna-edit to interactively work with those properties.
    • For me, deadlines are for hard deadlines only.
      • Therefore, they are seldomly used in my world.
  3. Task dependencies should be visible within both tasks, the prerequisite as well as the follow-up task.
  4. Changes or delays should be applied with minimum effort.
  5. There should be the possibility to generate an on-demand project status overview.

The Draft Workflow (Method)

My desired workflow is not finalized or completed yet. See the "FIXXME" parts where I'd love to get volunteers interested in contributing. Whatever code I will come up by myself, most probably will be implemented in Python and not elisp.

  1. MANUALLY: Create a new Org mode heading for the project.
  2. SUPPORTED: Invoke C-c C-x b (org-tree-to-indirect-buffer()) on project heading.
    • This is for focus only: ignore the rest of the potentially large Org buffer while working within a specific sub-hierarchy.
    • Think of switching to the version from my configuration with improvements by alphapapa.
  3. SUPPORTED: Mark project heading as such.
    • Via my-mark-as-project() from my configuration (and this blog article):
      1. adding a :project: tag
      2. property COOKIE_DATA set to "todo recursive"
      3. leading progress indicator
    • FIXXME:
      • add appropriate column view mode property
      • ask for a descriptive project tag and add it
      • ask for short project acceptance criteria and add "<criteria> → close project" heading that closes the generated project heading ID as well
  4. MANUALLY: Adding a descriptive project tag such as GProj for "Garage Project".
    • I like this to add some context in derived views such as my agenda.
  5. MANUALLY: Brain-storm new project as simple list.
    • Move around list items in a flat list.
  6. SUPPORTED: Convert list items to headings.
    • Via marking list and invoking C-c *org-toggle-heading().
  7. MANUALLY/OPTIONAL: Add task effort estimates if you need them for other purpose.
    • Switch to column view via C-c C-x C-c.
    • Add effort estimates via e (on a value).
    • Quit column view via q (on a value).
  8. SEMI-MANUALLY: Add unique :ID: properties to the headings.
    • I prefer :ID: over :Custom_ID: as described in this article.
    • I generate IDs via my-id-get-or-generate() from my configuration because I prefer human-readable IDs instead of auto-generated UUIDs that consists of arbitrary digits and characters.
      • FIXXME: automate missing ID generating with the future function for adding trigger dependencies (see below) as it can be done for org-super-links using that snippet.
  9. FIXXME: add trigger dependencies via a cool function as described on this (rejected) feature request.
    • Adding :TRIGGER: and :BLOCKER: properties in an interactive way + tab-completion of IDs.
    • Notice that org-edna is able to understand IDs with the id: prefix starting with version 1.1.2. This results in navigable links/
  10. MANUALLY: Mark non-depending initial tasks with an active TODO keyword.
    • I personally prefer NEXT for tasks I might start doing now and WAITING for blocked tasks by external parties.
  11. Start working on the project tasks.
  12. FIXXME: derive a read-only status chart from the current state of the headings any time.
  13. FIXXME: writing a hydra with all kinds of cool tools for supporting project management. Some ideas are:
    • Insert a new task between two tasks that do have a dependency defined: A→B then gets modified to A→NEW→B with modifying corresponding :TRIGGER: and :BLOCKER: properties.
    • Remove a task with all references within :TRIGGER: and :BLOCKER: properties.

Currently, four steps are at least partly supported by functions. Five steps are manual tasks and most likely stay that way. At least three steps can be dramatically improved by developing functions for them.

Here is a very primitive elisp macro I was using for defining a default :TRIGGER: property with a id:<my-current-id> from the kill-ring (I got the ID via my-id-get-or-generate() from my configuration):

(setq last-kbd-macro
   [?\C-c ?\C-p ?\C-c ?\C-x ?p ?T ?R ?I ?G ?G ?E ?R return ?\C-y ?\C-a right right ?s ?\C-d ?\( ?\C-e ?\) ?  ?t ?o ?d ?o ?\( backspace ?! ?\( ?N ?E ?X ?T ?\) ?  ?s ?c ?h ?e ?u ?d backspace backspace ?d ?u ?l ?e ?d ?\( backspace ?! ?\( ?\" ?. ?\" ?\) return])	  

While this would reflect my personal use-case for project management very well, there are even more possible future extensions. You could assign one or more persons to a task which supports capacity planning tasks. Or you can integrate with external project management tools. I'm sure that different project managers have different requirements and ideas here.

An Example Project: The Garage Project

For everybody unfamiliar with org-edna and task dependencies, I created an example project for refurbishing a garage. You can skip this section if you don't need details on the syntax I'm planning to use.

Please note that I'm not done with switching from org-depend to org-edna yet. Therefore, there might be some things an experienced org-edna user does differently. In those cases, please send me a comment!

 *** NEXT [0/2] Example project: Garage Project                  :GProj:project:
 :PROPERTIES:
 :COLUMNS:  %40ITEM %6Effort(Effort){:} %60BLOCKER
 :COOKIE_DATA: todo recursive
 :END:

 **** NEXT Find local stores for paint
 :PROPERTIES:
 :Effort:   3h
 :TRIGGER:  if ids("GProj-Choose-color-for-paint") !done? then ids("GProj-Buy-paint") todo!(NEXT) scheduled!(".") endif
 :ID:       GProj-Find-local-stores-for-paint
 :END:

 **** NEXT Choose color for paint
 :PROPERTIES:
 :Effort:   3h
 :ID:       GProj-Choose-color-for-paint
 :TRIGGER:  if ids("GProj-Find-local-stores-for-paint") !done? then ids("GProj-Buy-paint") todo!(NEXT) scheduled!(".") endif
 :END:

 **** Buy paint
 :PROPERTIES:
 :Effort:   3h
 :ID:       GProj-Buy-paint
 :BLOCKER:  ids("GProj-Find-local-stores-for-paint" "GProj-Choose-color-for-paint")
 :TRIGGER:  ids("GProj-Move-car-to-nearby-parking-lot") todo!(NEXT) scheduled!(".") ids("GProj-Remove-bikes") todo!(NEXT) scheduled!(".") ids("GProj-Remove-stuff-from-shelves") todo!(NEXT) scheduled!(".")
 :END:

 "Buy paint" is only scheduled when both prior tasks are marked as DONE thanks
 to the "if ids(...) !done? then ids(...) ... endif" trick above.

 Until org-edna is going to support this, there is a viable workaround at:
 https://github.com/mskorzhinskiy/org-linked-tasks

 **** Move car to nearby parking lot
 :PROPERTIES:
 :Effort:   1h
 :ID:       GProj-Move-car-to-nearby-parking-lot
 :BLOCKER:  ids("GProj-Buy-paint")
 :TRIGGER:  ids("GProj-Garage-is-empty") todo!(DONE) scheduled!(".")
 :END:

 Please notice that as of 2020-08-14 and org edna version 1.0.2,
 headings without an active keywords can be marked as DONE despite the
 fact that they're blocked. I've reported this bug already and hope for
 a fix.

 **** Remove bikes
 :PROPERTIES:
 :Effort:   1h
 :ID:       GProj-Remove-bikes
 :BLOCKER:  ids("GProj-Buy-paint")
 :TRIGGER:  ids("GProj-Garage-is-empty") todo!(DONE)
 :END:

 **** Remove stuff from shelves
 :PROPERTIES:
 :Effort:   2d
 :ID:       GProj-Remove-stuff-from-shelves
 :BLOCKER:  ids("GProj-Buy-paint")
 :TRIGGER:  ids("GProj-Remove-shelves") todo!(NEXT) scheduled!("++1d")
 :END:

 **** Remove shelves
 :PROPERTIES:
 :Effort:   1d
 :ID:       GProj-Remove-shelves
 :BLOCKER:  ids("GProj-Remove-stuff-from-shelves")
 :TRIGGER:  ids("GProj-Garage-is-empty") todo!(DONE)
 :END:

 **** Garage is empty
 :PROPERTIES:
 :ID:       GProj-Garage-is-empty
 :BLOCKER:  ids("GProj-Move-car-to-nearby-parking-lot" "GProj-Remove-bikes" "GProj-Remove-shelves")
 :TRIGGER:  ids("GProj-Paint-walls-and-floor") todo!(NEXT) scheduled!("++1d")
 :END:

 **** Paint walls and floor
 :PROPERTIES:
 :Effort:   1d
 :ID:       GProj-Paint-walls-and-floor
 :BLOCKER:  ids("GProj-Garage-is-empty")
 :TRIGGER:  ids("GProj-Re-install-shelves") todo!(NEXT) scheduled!("++2d")
 :END:

 **** Re-install shelves
 :PROPERTIES:
 :Effort:   8h
 :ID:       GProj-Re-install-shelves
 :BLOCKER:  ids("GProj-Paint-walls-and-floor")
 :TRIGGER:  ids("GProj-Bring-back-bikes-into-garage") todo!(NEXT) scheduled!(".")  ids("GProj-Bring-back-car-into-garage") todo!(NEXT) scheduled!(".")
 :END:

 **** Bring back bikes into garage
 :PROPERTIES:
 :Effort:   1h
 :ID:       GProj-Bring-back-bikes-into-garage
 :BLOCKER:  ids("GProj-Re-install-shelves")
 :TRIGGER:  ids("GProj-Celebrate-and-close-project") todo!(NEXT) scheduled!(".")
 :END:

 **** Bring back car into garage
 :PROPERTIES:
 :Effort:   1h
 :ID:       GProj-Bring-back-car-into-garage
 :BLOCKER:  ids("GProj-Re-install-shelves")
 :TRIGGER:  ids("GProj-Celebrate-and-close-project") todo!(NEXT) scheduled!(".")
 :END:

 **** Celebrate and close project
 :PROPERTIES:
 :BLOCKER: consider(all) rest-of-siblings-wrap
 :ID:       GProj-Celebrate-and-close-project
 :END:	  

For the sake of simplicity, I modified the generated IDs so that the leading ISO date-stamp is replaced by GProj- instead.

As you can see, I'm not using absolute dates for task planning as classic project management often does. My personal projects don't have deadlines and they develop according to my spare time, spanning over weeks, months and often even years. Therefore, personal projects often require a stricter management than my business projects.

Note that "Find local stores for paint" and "Choose color for paint" do feature rather complicated :TRIGGER: expressions. This is the technical solution that Jouke has found here to achieve, that "Buy paint" gets only "active" (scheduled + todo keyword) when both blocker tasks are marked as done and not just one of them. This can get even more complicated when "Buy paint" would be depending on more than two tasks.

As a consequence, I am currently not using efforts at all. The only reason I added effort estimates here is to play around with methods to visualize project status.

Visualizations

At first, I tried to derive a classic Gantt chart because it is the most obvious visualization when it comes to project management.

The new elgantt project won't support task estimations more fine-grained than days. PlantUML Gantt does have the same restriction although the forum lists some discussions to introduce hours and even smaller periods.

Since my personal projects are dealing with smaller tasks, planning on a day-basis is a no-go to me. However, I personally also do not need visualizations that map the tasks to a calendar or resources. I just want to get a brief overview as a nice-to-have thing, that's all. This is the reason why I most probably will not work with effort estimates in the long run for personal projects.

With the two Gantt tools failing to support my use-cases, I manually generated a simple chart using PlantUML Use-Case diagrams. Automating the chart generating for an arbitrary Org sub-hierarchy reflecting a project is only a matter of time and should be straight-forward. Elisp-savvy volunteers could provide a general solution here.

The PlantUML source has an initial/general section which defines colors, and the project itself. The following section with the first half of the nodes I ordered by "PlantUML link type" so that you're able to understand the different PlantUML syntax elements. The last section holds the remaining nodes and is arranged by Org heading which most likely reflects the parsing order and the results by generated code:

' skin settings for changing certain nodes according to their state
skinparam usecase {
    BackgroundColor<< NEXT >> LightGray
    BackgroundColor<< FINAL NEXT >> LightGray

    BackgroundColor<< STARTED >> PaleGreen
    BackgroundColor<< FINAL STARTED >> PaleGreen

    BackgroundColor<< DONE >> LimeGreen
    BackgroundColor<< FINAL DONE >> LimeGreen

    BackgroundColor<< FINAL >> LightSkyBlue
}

' this is default anyway
top to bottom direction

' from the project heading
' (**TITLE**) as (ID)
(**Garage Project**) as (GProj)

' ------------------------------------------------------
'     the items of the first nodes in logical order
' ------------------------------------------------------

' a list of all heading titles with their IDs as alias
'(TITLE\n//ESTIMATE//) as (ID)
'(TITLE) as (ID)
'(TITLE) as (ID) << TODOKEYWORD >>     ' TODOKEYWORD = FINAL when no trigger is defined
(Find local stores for paint\n//3h//) as (GProj-Find-local-stores-for-paint) << NEXT >>
(Choose color for paint\n//3h//) as (GProj-Choose-color-for-paint) << NEXT >>
(Buy paint\n//3h//) as (GProj-Buy-paint)

' all headings that got no blocker tasks:
' (PROJECTID) -down-> (HEADINGID)
(GProj) -down-> (GProj-Find-local-stores-for-paint)
(GProj) -down-> (GProj-Choose-color-for-paint)

' trigger properties
' (HEADINGID) -down-> (HEADINGID) : TODOKEYWORD
(GProj-Find-local-stores-for-paint) -down-> (GProj-Buy-paint) : NEXT
(GProj-Choose-color-for-paint) -down-> (GProj-Buy-paint) : NEXT
(GProj-Buy-paint) -down-> (GProj-Move-car-to-nearby-parking-lot) : NEXT
(GProj-Buy-paint) -down-> (GProj-Remove-bikes) : NEXT
(GProj-Buy-paint) -down-> (GProj-Remove-stuff-from-shelves) : NEXT

' blocker properties
' (HEADINGID) .up.>> (HEADINGID)
(GProj-Buy-paint) .up.> (GProj-Find-local-stores-for-paint)
(GProj-Buy-paint) .up.> (GProj-Choose-color-for-paint)

' ------------------------------------------------------
'     the rest of the nodes in the order of parsing
' ------------------------------------------------------

(Move car to nearby parking lot\n//1h//) as (GProj-Move-car-to-nearby-parking-lot)
(GProj-Move-car-to-nearby-parking-lot) .up.> (GProj-Buy-paint)
(GProj-Move-car-to-nearby-parking-lot) -down-> (GProj-Garage-is-empty) : DONE

(Remove bikes\n//1h//) as (GProj-Remove-bikes)
(GProj-Remove-bikes) .up.> (GProj-Buy-paint)
(GProj-Remove-bikes) -down-> (GProj-Garage-is-empty) : DONE

(Remove stuff from shelves\n//2d//) as (GProj-Remove-stuff-from-shelves)
(GProj-Remove-stuff-from-shelves) .up.> (GProj-Buy-paint)
(GProj-Remove-stuff-from-shelves) -down-> (GProj-Remove-shelves) : NEXT ++1d

(Remove shelves\n//1d//) as (GProj-Remove-shelves)
(GProj-Remove-shelves) .up.> (GProj-Remove-stuff-from-shelves)
(GProj-Remove-shelves) -down-> (GProj-Garage-is-empty) : DONE

(Garage is empty) as (GProj-Garage-is-empty)
(GProj-Garage-is-empty) .up.> (GProj-Move-car-to-nearby-parking-lot)
(GProj-Garage-is-empty) .up.> (GProj-Remove-bikes)
(GProj-Garage-is-empty) .up.> (GProj-Remove-shelves)
(GProj-Garage-is-empty) -down-> (GProj-Paint-walls-and-floor) : NEXT ++1d

(Paint walls and floor\n//1d//) as (GProj-Paint-walls-and-floor)
(GProj-Paint-walls-and-floor) .up.> (GProj-Garage-is-empty)
(GProj-Paint-walls-and-floor) -down-> (GProj-Re-install-shelves) : NEXT ++2d

(Re-install shelves\n//8h//) as (GProj-Re-install-shelves)
(GProj-Re-install-shelves) .up.> (GProj-Paint-walls-and-floor)
(GProj-Re-install-shelves) -down-> (GProj-Bring-back-bikes-into-garage) : NEXT
(GProj-Re-install-shelves) -down-> (GProj-Bring-back-car-into-garage) : NEXT

(Bring back bikes into garage\n//1h//) as (GProj-Bring-back-bikes-into-garage)
(GProj-Bring-back-bikes-into-garage) .up.> (GProj-Re-install-shelves)
(GProj-Bring-back-bikes-into-garage) -down-> (GProj-Celebrate-and-close-project) : NEXT

(Bring back car into garage\n//1h//) as (GProj-Bring-back-car-into-garage)
(GProj-Bring-back-car-into-garage) .up.> (GProj-Re-install-shelves)
(GProj-Bring-back-car-into-garage) -down-> (GProj-Celebrate-and-close-project) : NEXT

(Celebrate and close project) as (GProj-Celebrate-and-close-project) << FINAL >>	  

The first chart reflects the situation right after planning the project. You can see that the project has an initial node at the top and gray nodes below which are ready to go. The yellow ones (default color) are not scheduled and do not have an todo keyword yet. They're not actionable at the moment. :TRIGGER: dependencies are visualized and blocking back-links as well.

Garage project with the initial status before starting to work. (click for a larger version)

While this is nice to look at, a more value is derived from the visualization when the project is in an ongoing state:

Garage project status after starting to work. (click for a larger version)

If there is a dangling task that relates to an error when defining the dependencies, the visualization can help to spot issues. In the following example, I removed the :TRIGGER: property of the "Remove bikes" task:

Garage project with a dangling in-between node. (click for a larger version)

Next Steps

While project management is not one of my main use-cases when working with Org and my personal projects, it would improve my ability to follow long-running projects with many tasks more efficiently.

Now that I've written down all my thoughts and ideas, it's easier for me to communicate the bigger picture when joining discussions or creating feature requests.

Furthermore, there is a chance that you do have similar project management requirements and with your ability to code elisp, you might want to help automating some of the manual steps of that workflow.

Comment from yantar92

Yantar92 wrote me an email with some very good points. I'm quoting some of his arguments and my answer here.

First of all, he stated that project management consists of more than just defining tasks. He is absolutely right on this. This is true. I omitted it because in contrast to the title, my article's focus was not project management in general but Org mode task dependencies in the context of project management.

For a full-blown project management workflow, there are much more things missing such as resource tracking, cost tracking, and so forth.

Why don't just use capture template? This will automatically narrow the buffer, apply the :project: tag, add properties, progress indicator, prompt you for the descriptive project tag (via :%^G: in your capture template)

Also true.

The reason is that a large percentage of projects are not started as a project. Therefore, there are some tasks that "explode" into a project.

If the majority of my projects would be projects right from the start, I would use a yankpad snippet instead which automates some things I mentioned in my workflow.

I feel that you are over-complicating the dependency management. Let's consider your garage project example. Most of dependencies can be trivially defined by introducing hierarchy into your task list.
Note that I use different todo-keyword convention: - all the tasks must have a keyword - the tasks that can be done next are only the tasks with NEXT keyword /and/ with all the parent tasks also marked NEXT - Marking NEXT task DONE switches the following sibling TODO task to NEXT

I once followed this pattern as well. Somehow, it ended up with a different approach at my side. I don't use TODO any more, just NEXT. The TODO keyword was replaced by no keyword (and optional some TRIGGER property waiting to fire to set it) or the SOMETIME keyword.

I'd rather not say that one of those two approaches is inferior to the other though. Just different conventions for the same, I guess.

 * NEXT Garage project

 ** NEXT Buy paint
 *** NEXT Chose pain colour
 *** NEXT Find local stores for the paint
 *** TODO Buy the chosen paint

 ** TODO Empty the garage
 *** NEXT Remove bikes
 *** NEXT Move car to nearby parking
 *** NEXT Remove shelves
 **** NEXT Remove staff from shelves
 **** TODO Remove the shelves

 ** TODO Paint walls and floor

 ** TODO Re-install shelves

 ** TODO Bring things back into the garage
 *** NEXT Bring back the bikes
 *** NEXT Bring back the car	  
The above project structure will let you manipulate the dependencies simply by moving the headlines around and changing their todo-keywords with built-in org-mode functionality.

Absolutely true for this example.

I'd love to have a very low-profile feature compared to the :ordered: t property but with an additional "moving" keyword. org-depend offers something similar with chain-siblings(KEYWORD). To my surprise, org-edna does not offer the same feature in a simple way.

Of course, it does not allow complex task dependencies (when some task is triggered by completing multiple sub-tasks in different hierarchies). However, they are quite rare in practice (according to my experience).

In mine as well.

Automating dependency management in these rare cases may be still useful, but if things go away from initial plan in such complex projects, any universal automation workflow often turns out to be not suitable. I find it more practical using weekly review to resolve complex dependency (and other) problems, as advised by David Allen.

Perfectly well approach.

While my draft workflow reflects a certain complexity to demonstrate advanced dependency features of org-edna, most projects are straight forward as you have pointed out already.

However, in my case there are not so rare cases where I want to define a dependency between tasks that are not within the same project as well. For example when I want to take a look at a specific features of a tool which is not available on my slow-moving Debian GNU/Linux stable, I might define a dependency from the system upgrade task to the software test task. Those two headings are not even in the same Org mode file.

My example project demonstrates dependency management. I might have caused a certain expectation by choosing the project management story. Therefore, I hope I could explain my underlying thoughts with these lines a bit more.

Thank you for your great comments!

Promising Approach: org-linker-edna

The org-linker-edna project is a very promising approach to deliver many missing pieces to my workflow. It is able to create simple dependency links with :TRIGGER: and :BLOCKER: properties between headings.

Links

Andy read this article and discussed some aspects with me via Mastodon. He came up with his own workflow he's describing on his blog article. I very much recommend you to read through his approach as he's following absolutely valid simplifications for projects that don't need all of the complexity I mentioned above.

Comment via email or via Disqus comments below: