I was working for KFC, in a leadership role, when during the stand-up a problem came up. I had previously worked extensively on the database system there, so I knew that it could be solved easily with a couple of SQL statements. After the standup, the people who were involved with the issue stayed behind as I shared my screen, loaded up DataGrip (this was before I fell in love with neovim) and started rattling out the SQL queries.

The only part I really remember was that I started collecting the little bits of data together in WITH statements before I joined them together.

What I remember well though is the awe with which it was received — which surprised me. I didn’t think it was particularly complicated — I mean, I put it together on the fly — how complicated could it be?

The engineers on the call were so impressed, they bandied around terms like “sql god,” and I just brushed it off.

The DBA role tends to be pretty lonely because you’re often the only person doing it in a team — I was. It has a lot of depth and breadth, and plenty of mazes to get lost into before we even get into the operational aspects. Without a witness, it’s easy to forget that some of these things are pretty hard.

I appreciated having these witnesses that day.

I started learning SQL when I was ten years old. I was sitting in on the DBase III+ classes that my uncle taught at home. I remember struggling to understand JOIN statements.

I didn’t get that much into SQL back then — there were too many toys to play with. I was feeding an obsession to learn absolutely everything I could about this magical machine.

The next time I got a real chance to play with databases was MS SQL Server in 1999. I was thrilled to be playing with this “big boy” database system. Shortly after, I was the sole technical building an online course for Scottish Enterprise.

What I remember the most? One late evening, while I was building something, I accidentally DROPped the Pages table. It contained all the course content. I had no backups.

It was the first real terrifying moment of my career. It felt like the blood was draining out of my body as I worked out alternative plans. Worst case, I’d need to reconstruct it again from the source data — unpleasant and time consuming, but do-able.

Before doing that though, I figured I’d give the hosting company a call. They were just up the road from me.

I remember calling them, and asking if by chance they had backups. Her name was Alison and luckily for me, they had backups, and she restored it for me — while on the call. I felt like I was back at school, which I’d only left a couple of years earlier — and Alison was a nice teacher. She reminded me about the importance of backups and also that we should pay for it. Getting that data back was a huge sense of relief.

After that, I used transactions around any data modification statements and refused to work with databases that did not support proper transactions, if I could help it. I also made sure to have backups. Any write statement became much scarier. I learnt to do a SELECT before an update with the same clauses. I did not want that feeling again — of all my blood draining out of my body.

A few years later, I was building megabus.com and I picked PostgreSQL for the database. It had transactions (of course) but it also handled extremely high load much better than the main competitor at the time — MySQL.

I loved working with PostgreSQL — and got into all kinds of mischief like building a replication system for it. The most difficult SQL I wrote was when we developed an updated version of the schema and we were working out how to migrate the data. Chris and I worked on that and it was hard. The first version of it that we wrote was estimated to take two weeks to complete — we needed it to complete in a few hours at the most.

We convoluted and reconvoluted the data to trim the times down. We were working our server so hard that one of the statements that Chris tested caused the kernel on the server to oops and crash. I was very impressed — though sadly we were not able to figure out why it crashed or to reproduce it.

Postgres did not have materialised views back then, so a lot of the time, we had to use temporary tables to help speed things up. Sometimes we needed temporary tables just to make it easier to reason about what we were doing, and break apart some lengthy and complicated statements. I got intimately familiar with EXPLAIN and EXPLAIN ANALYZE.

I dreamt about the results from the EXPLAIN statements.

It was exhausting work. In the end, we got it down to a few hours — short enough that we could do the upgrade overnight.

Several years later, I was responsible for the databases at KFC, and a bug had mangled some loyalty data. The stamps were allocated by an application which had a bug and had allocated wrong stamps.

The shape of the data was such that the calculations were easy to do imperatively in code, and hard to do from the database. On the other hand, the amount of data to process was so large that it would noticeably slow down the database, and potentially take a long time to complete.

So, the question was whether I could do some triage on the data first. Could I then help the engineers identify the quickest solution for the majority of cases.

I was running the queries on a follower, so the data was live, but what I could do was limited — no direct mangling of data. I learned a whole bunch of new tricks that PostgreSQL now had like materialised views, window functions, WITH statements, and more. I loved it.

With some elbow grease, I was able to do the analysis required to assess the damage and to accelerate deployment of fixes to the data.

I learned a lot about the database structure and how it all hung together over those couple of days.

When I think about that day, on the call after the stand-up, I only think about how easily I was putting those queries together. I might have written those queries on the fly, but I was building them for years before that.

The blood rushing out of my body, the convolution and reconvolution, and getting to learn how someone else’s data structures hang together — it all came together that morning on the call — and made it look easy.

A few years later, at Kpler, when I optimised a naive but complex query which took over 2 seconds to under 500ms, I remembered that video call. Just because someone knows how to navigate something, doesn’t mean it’s easy.

What does expertise actually mean? The engineers on the call were impressed — but I dismissed it, partly on the basis that they just didn’t know SQL as well as I did.

Working in tech, it’s hard to have a witness — someone who understands enough to see how difficult something is to solve — but not so much that they too take their expertise for granted.