π

Pieter Hintjens: Confessions of a Necromancer

Show Sidebar

If you have worked with ZeroMQ, OpenAMQ, AMQP or Xitami, you have worked with code by Pieter Hintjens, a Belgian software developer.

Out of a tragic personal situation, my attention was brought to a blog article of Pieter: A Protocol for Dying. This is quite an article to chew on. Pieter was diagnosed with a rare type of cancer with not much time left. He was very open with his disease and wrote quite a few articles about his situation. In 2016, he underwent voluntary euthanasia. If you are interested in learning from this situation, I recommend browsing through his blog.

Here, I want to draw your attention to a different article by him: Confessions of a Necromancer. This is some sort of IT-related autobiography which begins in the old times of Microcomputer. Pieter wrote down his journey through many IT projects, companies and customers.

It's a long article. Printed to a PDF file I read on my e-ink reader, it had 43 pages without the comments.

He wrote his findings and lessons learned where we all can learn from his experience. In the following section you see some of the text I highlighted. Read the article yourself if you want to get the whole picture.

Some Quotes from the Text

And it's all about me, the smartest guy in the room, always with an answer, almost impossible to beat in an argument.

This is a professional road trip, for my fellow programmers.

There has always been a thread of insanity in our business. We're hooked on doing the impossible. Clients lie to themselves that they've hired a team that know what they're doing. The team lie to their clients that they're in control. The sales people lie. The marketing department lies.

Yet for most of my career, my special talent was to make projects work, no matter how impossible the technical challenges. I am a really good technical architect, able to understand systems at the lowest and the highest levels. I can write all the code myself, or I can explain to people exactly what to make, and it all fits together like laser-cut blocks.

Technology is a slave to personality. It doesn't matter how good the design, when there are unresolved problems in the organization.

Gradually I shifted from technical architect to social architect. From caring about technical designs to caring about people and the psychology that drives them. Because in the end, this is what seems to make the difference between a working project and a failure.

Every machine had its own concept of file structures, organization models, compilation procedures, and so on. Imagine learning ten different varieties of Linux, each from a different planet.

You could develop and test on a PC, at home, and the same code would run unchanged on an IBM mainframe.

The more your clients pay you, the more they appreciate you and listen to what you say.

Since no-one had told me it was impossible, I took the documentation and a test system, and began to play.

I learned an important lesson about consultants: they are professional liars.

If you have the trust of your client, and s/he has real power, you have done half the work already.

This was a perfect software project. Working with good matériel; decent hardware and a non-insane operating system. Working with a smart team that know their stuff, and a client who likes and trusts you. With total freedom to build things the right way. Where the hardware and software bends to your will. Where you can take the risks, analyze them, and design them away.

Lesson learned: identify the riskiest parts of your architecture and bring them under your full control.

The lesson here is that making systems more complex, to try to make them "reliable" will usually make them less reliable. What we were experiencing was "split brain" syndrome, the bane of any clustering design. Allowing the PCs to randomly decide they were in charge was a Bad Idea. Letting them decide to switch back to normal mode made it much worse.

Additional lesson: fail-over automatically and recover manually.

Don't expect random work from random vendors to magically work together.

It is shocking how many projects are based on betting against the tide. By this I mean forcing an application to run on some antiquated, expensive, inflexible, and painful platform, when new, cheap, flexible, and enjoyable alternatives exist.

Here are some of the reasons:

Threads must never block, and all I/O must be asynchronous.

You don't need actual system threads.

We've learned by 2016 that sharing so-called "mutable" data between threads is a Really Bad Idea.

We developed in C, which is less than ideal for the grunt work of business applications.

Manpower's competitors tended to grow by acquisition and then leave every business unit to do its own thing. That was far more organic, and cheaper, than trying to rationalize all their IT systems.

The foundation for a good project is: a competent client who knows the business and has power of decision; a full-stack team that can deal with the work, at all technical levels; and a technical platform that is both dependable and tractable.

Let's take the prototype we have, I said, and improve it little by little. You make requests, we'll make changes, you test the results. How often can you make new releases, they asked us. Every day, if you want it, I replied. We agreed on two or three times per week.

The lesson here is, when you are a technology company, use your own technology in your projects. Don't accept client requirements that sabotage your own vision and future. It isn't worth the short term gains.

Much later I came to frame this more brutally as, "never build closed source, period."

Occasional users cannot learn complex UIs.

The lesson here is a devious one, and that is that you often don't know what the real problem is until you get very close to it. We were lucky our approach fit Manpower's way of working.

MTS would panic and shutdown threads when too many users worked at once. There was no way to fix this. It left the database with dangling locks that ended up killing the entire system. We had to limit the number of users, and move to larger machines.

Either you love Africa or you hate it. There is no easy middle ground. Even stepping foot in Africa as a European is a political act, conscious or not. It took me many years to understand it as a continental prison filled with innocents, as I've written in my book Culture & Empire.

In Nigeria, there is no trust. All business runs on the assumption that theft will happen if it can.

We learned quickly that a meeting for a certain day meant, "we'll arrive at some point during the day." Traffic jams could last several hours, even though from the brewery to Victoria Island, and the financial district, was just 15 minutes' drive.

One thing you quickly learn in Africa: patience.

I learned that email is surprisingly robust, even though it has no delivery guarantees.

Lesson here is: you can do a lot with little, if you are creative.

I did what I often did in those days, when starting a new project. I sat down and wrote a design spec.

Lesson here is, don't take "impossible" as an answer. Everyone lies, not always deliberately. We just have our assumptions and ignorance and we believe we're telling the truth.

There were some good lessons here:

It is ironic that a "successful" project can be a failure, while catastrophically bad projects can push you through to better things.

Lesson learned: don't make stuff and then try to sell it unless you are growing an existing client base.

Lesson learned: breaking into markets you don't know is probaby impossible.

Lesson: mobile phone operators are crooks who steal billions, fifty cents at a time.

Yet without shutting down our projects and going through the pain of firing friends, iMatix would have gone bankrupt.

Lesson: be aware of your expenditure and manage your losses. You can survive a long time with less income if you are in tight control of what you spend.

Second lesson: it is no favor to pay people to do idle work. When you hire someone, tell yourself, and them, one day this will be over. Today I far prefer working with self-employed partners because that doesn't need to be stated, it's explicit.

"Pieter, I want to make you a deal. If you remove your name as author of the spec, and let me tell people I wrote it, I'll get you the rights to OpenAMQ so you can start a business on it."

Lesson: sometimes success is your greatest problem, and sometimes bad events can have great outcomes.

Lesson: don't try to fix existing organizations. Start new ones. It's sad yet there we are.

If you can't envision a complex protocol as layers, you shouldn't be in the business of protocol design.

If we'd managed to build a thriving community around OpenAMQ, it would have survived. So the lesson here is simple: community before code.

In toxic projects like AMQP, when do you give up? I've tended to try to make things work until the bitter end. I'd stop only when there was no money left to invest, or literally at the edge of burnout. I've justified and rationalized this in different ways. Even now, I'll argue that bad projects (like bad relationships) are the only way to really learn.

The best answer I've found is the one I explain in that book. Diagnose the situation, observe carefully, intervene to turn it around, terminate when you are healed.

Good software is used by people to solve real problems. Good software saves people money, or makes them a profit. It can be buggy, incomplete, undocumented, slow. Yet it can also be good. You can always make good software better yet it's only worth doing when it's already good.

I think the core lessons are: be patient, don't give up, and always be learning. You can turn even the most crappy situation into valuable lessons. Teach them to others. Be happy with what you have yet always strive to improve things. Don't let people flatter you into playing their games. When things get weird, keep a log. Love and respect good people. Learn to keep the assholes at a distance. Don't get hung-up on the past. Be nice to people, even those trying to hurt you. Speak up when things are bad, and tell the truth. Trust your emotions yet check where they come from. Don't be afraid of taking risks, and learn to identify and manage risks. Solve one problem at a time. Be generous. Teach others whenever you can. Remember Sturgeon's Law.

Comment via email (persistent) or via Disqus (ephemeral) comments below: