About

The short version: I make data move, reliably, and I'd rather it were boring.

Background

I'm a backend and data engineer living in Amsterdam. I studied applied mathematics briefly and computer science more seriously, and have spent about twelve years working on the unglamorous middle of systems: the part between where data gets created and where somebody looks at a number and makes a decision. I like that part. It has fewer meetings, and when it goes wrong you can usually prove it.

I've been on both sides of the "backend" and "data" line long enough to think the line is mostly organisational. A pipeline that people rely on at 07:00 is a production service whether or not anyone calls it that, and a service that emits events is a data producer whether or not the team that runs it has ever heard of schemas.

What I work with

postgres

More than anything else. Most of my day is SQL, EXPLAIN output, and reading other people's migrations.

python

Glue, pipelines, and anything that has to be read by people who don't write code for a living.

go

Services that need to be small, start fast and stay up. I'm not religious about it; it's just what the last few years have looked like.

kafka

When there's a real reason for it, and not before. The reason is usually "more than one consumer" or "we need to replay", and I write both down before agreeing.

airflow

Orchestration, with opinions. See the sensors note.

dbt

The transformation layer in the warehouse. It made SQL reviewable, which is most of what I wanted.

I've spent time with ClickHouse and BigQuery and liked both for what they're for. I've spent time with Spark and would, on balance, rather not. I still think most "streaming" architectures I've been shown would be happier as a Postgres table and a cron job, and I've usually been right about that, though not always.

Where I've worked

In roughly this order:

• A logistics company with more trucks than engineers, where I learned that a forty-minute batch job everyone relies on is a production service, and that nobody will tell you this until it fails on a Sunday.
• A payments start-up that didn't make it, but had the best code review culture I've seen before or since. I still write commit messages the way they taught me.
• The data platform team at a large retailer, where the scale made everything interesting and the org chart made everything slow. Most of the Postgres partitioning experience comes from here.
• Currently, as a contractor, a scale-up in the energy sector, on a team of five that is responsible for more Kafka topics than it would like.

What I'm curious about

• Incremental computation done properly: differential dataflow and its cousins, and why so little of it has made it into everyday tooling.
• How to make data pipelines testable without a test suite that takes an hour and a staging warehouse that costs as much as production.
• Whether the "modern data stack" was a good idea, or just a good decade for vendors. I go back and forth.
• Cycling infrastructure, as a user rather than an engineer. Living here spoils you.
• Bread. Badly, so far.

This site

Hand-written HTML and one CSS file. No analytics, no cookies, no fonts or scripts loaded from anywhere else; the pages weigh what they look like they weigh. Notes are written when I've finally understood something, which happens rarely and irregularly. The date on a note is when I published it; I edit old ones without ceremony when they turn out to be wrong.

Contact

I don't put an email address on here, and I don't link out to accounts; both attract more mail than they're worth. If you know me, you know how to reach me. If you don't and a note is wrong, it will stay wrong until I notice, which I do, eventually.