I then started my own company and ran that for nearly two decades, everything largely self-taught. After that, because of the peculiar shape of my CV, and “falling between a lot of stools” as a friend pointed out, interviews were… tricky!

I remember one such interview a few years ago. He was good at diffusing any anxiety and it was a pretty relaxed interview. He wanted to know if I knew architectural design patterns - a term I hadn’t heard before. He gave me examples, circuit breaker, pub / sub and a third one I can’t remember anymore. I recognised pub / sub and the other one I think - but was not aware of circuit breaker. He explained it to me - something about reducing the risk of a downstream service failing.

It made sense - but I couldn’t remember any instance where I’d done that and I said as much.

No big deal! We moved on.

As you may have noticed from my recent flurry of posts, I’ve been going through a bit of an excavation of all things kraya, and discovering little nuggets that I’d forgotten all about.

Claude code went through all of my emails, all the code repositories and put together interesting things I had done. The purpose was mainly to pick out things that could be good blog posts, stories, or just reminders.

Claude had found two circuit breakers.

The first one was built by the seat of my pants in 2004. There was an active marketing campaign on megabus and the system was struggling. I’d plumbed the depths of the tech stack - the web servers, the database server to get every last ounce of performance out of the system.

What I really needed was to slow down the deluge of people coming into the site - just a little bit. I needed to prevent the snowball effect, and I tried a simple way to achieve it. I wrote the following around my 23rd birthday.

(The eagle eyed among you might notice a bug in the above code. It wasn’t resolved for years. Coding live on a production system tends to create bugs. See if you can find it before reading on.)

(Regular readers may also remember a part of this story from I Know People Like You)

The system already tracked searches - so I just needed to check it.

This bit of code checks to see how many of the searches that started in the last two minutes were incomplete. Except that’s not what it did - I forgot to actually use $timestamp so it only checked the current minute’s worth.

I considered failing and letting the user retry manually - but I didn’t want to force the user to take action unless absolutely necessary. This one waited up to 15 seconds and then let the search happen anyway.

If the user was still waiting after 15 seconds, might as well try and do a search and see what happens. In hindsight, it might have been better to fail at that point. If there were too many searches after 15 seconds, the database server was likely already snowballing.

This bit of code survived through to the end of the PHP codebase apart from minor tweaks. The Java version had layers of circuit breakers. Session limits, rate limiters and threadpool configuration for scaling up. None of it though, was called a circuit breaker - not by me or by the team.

So, have I built a circuit breaker? Thinking back, what threw me off was the wording. I had the feeling that the circuit breaker would “protect” another service.

In my mind, the database wasn’t another service - it was a part of the same service.

By the time it was 2015, I did not.

I had poured blood, sweat and tears into a ticketing system that kraya built — first for megabus, then for Polskibus. It had broken me, and we were now just limping along.

By 2015, we were spending 101–286 hours each month on a support contract that paid for 60. I raised this with the client and suggested a minimum of a 100% increase. They refused. Instead, they minimised their requests to just about 80 hours each month. Without the development work, I was now making a loss each month providing the resources for this support contract.

I could cancel the contract, but that would involve letting them have the source code, which was otherwise kraya’s property. There was a six month notice period — six months of providing all the support they need to run and to migrate the system to their team.

At some point, I went from taking on the challenge to holding the fort.

I could see that any effort to make things easier would backfire. I built them a feature for free — they complain about one bug in it — and they need it fixed urgently. I loosen the rules and deploy an additional server just before the weekend, letting them know about the risks. Something goes wrong and they are surprised.

The hardest thing for me to change was the belief that the client’s wins were my wins. If anything, the client’s wins were my loss. It meant more unpaid work in the support contract. It meant more complex systems to support and maintain — while every penny was being questioned.

I started to say no. No, we will not deploy new servers on a Thursday because the weekend is too close. No, we will not reduce the QA time on this bit of functionality you want. No, we will not restore reports which could display erroneous data.

It all came to a head in one specific instance where the CEO insisted on speaking to me because they needed something deployed urgently. They needed us to work through the evening or the weekend. I said no. They offered to pay double. I said no. They were not happy.

A few months later, they cancelled the contract.

I remember the meeting. It was amicable and friendly. They asked if I could keep the staff on till the end of the year - they wanted backup in case anything went wrong during the migration.

I wouldn’t want to let them go before Christmas anyway.

I had a brief conversation with my brother to decide what to do next. I already knew that there was nothing left to keep running. We kept it running for three more months until the end of December.

I remember going into the main office, gathering everyone around and delivering the news. Polskibus had cancelled. We are shutting kraya down. We’ll keep going until the end of December.

I remember the Christmas party. It had often ended up being drinks, and being out all night. It didn’t this year - we went home after the meal. The atmosphere was one of sadness and relief. We’d all been through the fires and made it out the other end. It was over.

It was the end of what I’d built over 15 years.

I felt nothing.

It was time for kraya to take a leap. We should take the megabus.com ticketing system to the next level and build a distributed ticketing system that could potentially scale infinitely.

I’d picked JBoss because it was backed by Red Hat - it had all these features and capabilities - a lot of which we needed. The other option we considered was Glassfish, but it just didn’t have the features we needed. There were other options but that involved prohibitive licensing fees.

We costed it out at £650k and a year. It was unrealistic but I could not imagine Stagecoach paying more, or giving us more time. They wanted it for £500k and in six months. I should’ve pushed back, but we’d built a booking system over the weekend, only a few months back - this should be possible, right?

It wasn’t.

We got a year in the end — we didn’t ask for it, and we didn’t know until we were most of the way down the path. When I heard about the delay it was a mix of relief and regret. We’d burned through most of the budget on getting in contractors who were leaving imminently. We could have got fewer people on board, but as permanent staff.

We launched to Canada first, and that wasn’t too bad. Then it was the US, and the nightmare started. The UK launched last on my birthday in 2010.

In the intervening years, we had a budget shortfall of £150k and because we’d rushed to build the product, we’d cut corners, and all the contractors had left. We even had to let go of some of the permanent staff. I asked Stagecoach if they’d fund us the extra £150k as we’d asked at the start. They said no. They renegotiated the contract. They demanded more oversight - and increased our reporting obligations.

Load testing was already a part of our process and we tested the new system under load. We identified issues and fixed them. On paper, it looked good.

It wasn’t.

The problem wasn’t load per se. It was the stability of the system. It struggled to stay up for extended periods of time. The more nodes there were, the worse it was.

It was exactly the kind of problem that was hard to replicate and hard to test. The main option we had was to think through what all it could be and to take stabs in the dark. I put together a hit list and worked through it methodically. Convincing Stagecoach to spend the money was sometimes harder than solving the problem.

Ultimately, the one thing that pushed us over the line in terms of stability was staggered nightly automated restart of each node.

I thought I was done working with software that needed regular restarts to stay functional and at first resisted this. We are running enterprise level software, and that too on Linux. If I was comfortable with such shenanigans, I might have stayed with Windows. But we were running out of options. It was a hail Mary - it worked.

I had been working with Linux and related software for years by that point. I, in fact had servers that had not been restarted for literal years at that point, with services that were running just as long. A lot of these services didn’t even need a restart on config change - just a reload.

None of these services even had a paid tier. That should have been the clue.

I cannot imagine having to restart PostgreSQL nightly or even Apache. I still do not understand how something slated for the enterprise market can have leaks that would warrant a regular restart to keep it working.

Years later, when I looked up issues around JBoss, I realised that it was notorious for a whole slew of problems with JGroups and clustering. I remember scouring the internet for details on any kind of issues and coming up empty.

I’d fallen into a marketing trap. JBoss was no Apache — it was the commercial product of Red Hat. I thought all enterprise open source software would be of the same calibre. It wasn’t.

From what I understand, Stagecoach spent millions and maybe two years building the whole ticketing system inhouse. kraya limped along for a few more years before being shuttered.

The people involved, though, fortunately seem to have gotten through it largely unscathed. It gives me a great deal of joy to see so many of the juniors I’d hired now CTO’s, VP’s, Directors.

I believed that every victory was ours but when something went wrong, it belonged to me. After all, I made every choice. I chose to pursue the Java EE ticketing system. I chose JBoss. I chose to keep going.

I was in a narrowing path, with fewer and fewer options.

I knew I was the only one holding that line - I didn’t know that there was another option.

Bit by bit, I lost all sense of what I was paying to fix these mistakes.

It was the same every time. I would drag myself up to answer the phone - my body, my mind screamed at me, but I had gotten good at overriding every instinct through sheer willpower. I could hear the apologetic tone on the other side, and I could recognise some of the voices after a while. I mustered up all of my strength to be and sound as awake as possible. I needed to be professional even if I was still in my underwear.

I had a glass of water on my bedside table. I’d pick that up and head to my office in the spare room. The computer was always on and always ready to go — like me I guess. I’d log on to the servers, and check the logs. If I can identify which one fell out of the group, I can restart just that one. If I was too late or if the issue had escalated, I’d have to restart the whole cluster — shut them all down, give them a few seconds, then bring each one up, while keeping an eye on them. I could do it half asleep after a while.

Falling back asleep wasn’t a breeze either - I was tired - exhausted - but I was now also wired. Waking up in the morning was harder - the alarm would go off and my body would be limp. I still remember the sheer power of will to drag myself into the shower, then carry on with the rest of the day.

Of 266 incidents over about two years, I answered 156.

I remember one particular night, though I do not remember how many times I’d woken up beforehand. I was already tired.

megabus.com had gone offline. I got an alert. “But someone else is on call tonight,” I told them. “We already tried them twice,” came the reply. I had to deal with this. I had to deal with this.

I remember sitting at my desk at home working on fixing it. At some point, something was different, though I don’t remember what. While I was working on fixing it, I remember being overcome with an overwhelming impulse to get up from the chair and walk away — I almost imagined myself walking away. I resisted and shut down that impulse. I fixed megabus as I had always done. In fixing megabus though, something broke inside me, somewhere deep, in the very core of my being. I was never the same again.

I analysed the system top to bottom, inside and out. I even waded through JVM internals.

It got incrementally better, more stable. I think I rewrote every component that wasn’t the core ticketing system. In the end, what pushed it over the line were two unexpected changes. Automated nightly restarts of each node in the cluster and a rate limiter.

On the 10th December 2012, the system, now serving Polskibus, had a sale event. We had a bank of screens on a wall with all the key stats for the system. It looked cool, and we felt a bit like we were on a TV show. At peak, nearly 15,000 concurrent sessions — six or seven times the average. Over 30,000 bookings in a single day, three times more than the normal amounts across all systems.

We watched it closely, all day. Nothing broke. Nothing screamed. Everyone smiled, but there was no celebration.

We were also working on the fringe website. We integrated the website with the brand spanking new ticketing system - which cost nearly £900k.

We were also hosting websites for Boots, Kellogg’s Food Service and dozens of other clients.

All of this was held together by three or four developers, two systems administrators and me.

On the 13 June (incidentally, I got married on the same date years later), as I was just getting ready for a wild night on the town, a call comes through - which John answers.

I still remember them laughing and then doing a double take “oh, you’re serious? let me get Shri”

It was the fringe. We’d already known that they were having trouble with their ticketing system. I’d even pitched in, made suggestions - looked at their code to try and help, but none of that was enough. I expected an update.

They wanted to know if we could put together an interim booking system for them over the weekend. I wasn’t sure. I told them I’d speak to my team and get back to them.

I wasn’t involved with the work on the fringe up until this point. I knew very little about it. I was focused on megabus.com. The US version of the site had a big marketing campaign happening in a few days and that was what I was meant to be focused on.

By the time I put the phone down, Chris, who had been the lead on the fringe already had a answer. “We can do it!”

“But how - it’s got to take more than a weekend - right?”

The fringe website was already built well and had a clean layer interfacing with the new ticketing system. In fact, that was the bulk of the work that year.

So.. Chris told me - all we would have to do is to implement the functionality within that thin layer, fattening it up.

He believed we could do it. I believed him.

I hopped in a cab, headed over to the fringe to talk it through. I didn’t promise them we’d be able to get something ready by Monday, but I promised we’d do our best.

We were in the office on the weekend, writing code. I remember working on the basket, sending diffs over email and generally having a good time.

I even had some megabus US fun to keep me entertained in the form of issues with loading sheets - I was already in the office, so it was one step easier to fix.

One of the bits of functionality which took a surprising amount of time was the seat allocation. None of us had to worry about that before - it was just capacity management on megabus. For the fringe though, we had to allocate actual seats with seat numbers and everything.

With the fringe we had multiple tables which all joined together (thanks hibernate) to encode a tremendous amount of detail about the seating plans - including their physical location on a map.

It was too much detail for us, so we had to simplify it all down to get it working in the timeframe. We kept most of the rest of the structures intact to keep the data migration easier once the ticketing system was fixed.

By Monday, we had each managed at the most 6 hours of sleep each of the previous three nights. I still have vivid memories of a suit of armour that we put together using packing material while we were waiting for bits of data or details of logic.

I broke the MySQL replication at 01:24, fixed by 01:32

I remember the delirium setting in. Email sent to the client with “fun fun fun fun fun fun fun” as the subject

There were random emails to my brother “I’m still here!”

I also remember making makeshift beds with bubblewrap to get a wee nap here and there. We were all so exhausted - pumped up on coffee and nicotine.

Finally at 03:35 on the Tue email to client: “DONE DONE DONE DONE DONE NODE NODE NODE NODE”

Then at 05:33, requesting a PostgreSQL server rebuild for megabus US for their marketing campaign.

At 10am on Tuesday, the fringe is finally able to sell tickets. The website promptly fell over from the load, but we nurse it back and it sells 65k+ tickets in the first week.

It would be at least two more weeks before the ticketing system is fixed and brought back in.

For the work we did for them that year, and the previous one, we effectively only charged about 30% - because that’s all they could afford. This year, we asked if they could put our name on the website.

Over the next two weeks(while megabus US was on their marketing campaign), we fought many battles. There were 750 duplicate bookings. Numerous customer complaints (thanks to our name being on the website) - almost all of them blaming us for the failure of the ticketing system. People did not understand that we put in the interim one, not the one that failed.

Press releases went out from the fringe - only two credited us. Both misspelt the company name. Both called us a web design company — which, we were not, had never been, and had no interest in becoming.

In truth, I wanted to be a hero - I think we all did. What we really wanted was an acknowledgement of what we had done - which was nowhere to be found. We got paid though - at least for a part of our effort.

For many years after that, I would tell people with pride - “did you know - I saved the fringe, back in 2008,” which was inevitably met with something like “oh, did they have a problem that year?”

It started off small, and little by little, I carried more and more. I became we, and we carried more and more. Before we realised, we were carrying a great deal. Ultimately, though, if something went seriously wrong, it would be on my shoulders.

The chart does not capture the scaling of the organisation, or other departments like tech support or hosting, which had dozens of clients.

The section at the top is the number of active committers for that quarter. You can see my on my todd at the start and the rise and the fall of the dev team.

The very peak of it was in 2008. We built a booking system over the weekend for the Edinburgh festival fringe because their brand new £800k+ system could only handle one person at a time.

There were a handful of us building it while I was simultaneously prepping and managing the megabus US systems for a sales campaign. At the same time we were operating megabus across the UK, USA, and Canada, Oxford Tube, Coach USA, the Fringe website itself and numerous other smaller hosting clients like Boots, Kelloggs Food Service, and so on.

We were a total of ~4 developers and two systems administrators holding all of these together.

I was recently reminded of a story a friend of mine told me. Before he was my friend, he worked with me, and one of the things we did together was one the big megabus deployments when we migrated to a Java EE ticketing system.

One part of the migration was the data. I had developed a tool to migrate the data and on the evening - everything was prepared, we were off peak, and we had taken the site offline. He ran the script, which went on for a wee while and it failed. It was not meant to do that.

The way he tells the story, he told me about the failure. I come over, look at the errors, say “hmmm, that’s interesting,” and head off to have a cigarette. A few minutes later I go over to my desk type away furiously, then asked him to run it again.

It worked, and completed.

I remember that night. I don’t remember what the problem was or how I fixed it, but I do remember that moment when I went over to see how it had failed. In the short walk from my desk to his, I reiterated in my mind, all the possible backup plans - with the worst case scenario being to call off the migration on that day. We would do it another day. It would cost money, but it would be do-able. I was ok with that.

I was curious as to where my limits were, so I kept pushing, until I would meet with a wall.

There were no rails and there were no railings - just a cliff edge, unmarked… I didn’t know that - I expected a brick wall.

The worst of it would be only a few years later, in 2011. We built a new Java EE ticketing system for a fraction of what it should have cost in about 30% of the time it needed.

I personally responded to over 250 out of hours emergency tickets over an 18 month period. That was hard!

I had run off a cliff edge, and like the roadrunner in the cartoons, it took a while before I realised there was no ground beneath me.

A few years after I ran off the cliff, the company shut down. A few years later, I would start my active recovery journey through therapy. A few years later still, when I felt ready for a leadership role, I was asked to lead a problematic team - a role they struggled to fill for a while.

The team worked hard and delivered but struggled with the perception of poor delivery. Trust was thin, stress was high and morale was low. The situation was so bad that the week before I was supposed to start, the scrum master who was supposed to be my guide through it all quit.

I was warned by multiple people that this job was loaded with problems. I took on the job anyway, without a real guide, straight into multiple serious issues.

I loved it and managed to turn the whole thing around in my first week. Delivered key items, laid the foundations of trust and improved morale. It took a bit longer to bed everything down. Within weeks, I was asked if I would take on leading the entire digital team.

It was here, many years later, I got a sense of how unusual it was for such a tiny team to do so much.

It was here, many years later, I got a sense of how it is to have guardrails, to have support, peers to lean on.

It was here, for the first time I realised that the job didn’t have to be a lonely one.

I was 22 years old.

When I built the original prototype for megabus.com, I built it using PHP + PostgreSQL. I put together a document detailing my reasoning for these choices. I quoted 33 days for it, built it over six weeks and charged £13,200.

The support contract was £350/month - for one day a month. On the first day, megabus.com sold 200 orders.

When megabus had its first expansion, I was up overnight bringing new servers online and scaling it live. I loved it - my code was finally being tested.

Over a week, I’d probably burned through many days of effort. I remember the project manager specifically asking me to invoice for the extra work I put into it. I even said that I would - except I didn’t.

I’ve had a long time to think about this - why did I not send that invoice? I even had approval.

The answer, as with most things of this nature is complicated. I loved the work and I didn’t want it to end. I didn’t want a potential conflict trying to figure out what a reasonable amount was to charge. I felt that I should have done a better job in the first place - I felt responsible that I had not told them that scaling of this nature would not have worked without prep work.

I had not scaled anything before.

I was 22 years old.

I was super grateful that someone believed in me. I naively assumed that they saw all the extra effort I was putting in and that they would reward me for it - that they would have my back.

I remember adding a bunch of different bits of functionality because I wanted it there. I didn’t want to go through the process of quoting for it, and it getting potentially rejected, not to mention the waiting for decisions. One key bit of functionality I remember is adding in a percentage load column for the loading sheets. I built it, showed it - they loved it! It went live. I did not charge for it.

At this point, the vast majority of my time was spent on megabus - very little of it actually paid for.

At a glance, based on the emails sent, I probably spent a minimum of 10 days each month supporting megabus when I was charging for one day.

In Jan 2004 - I proposed doubling the contract to two days for £4,800/year. It probably kicked in in Feb 2004. By March 2004, the site exceeded that revenue each day.

In the following months, I probably spent, on average a minimum of at least double the time I was paid for. I should have charged for it.

I grew the team, and the support contract based on the minimum I needed to maintain the product - not based on the amount of time I was spending.

For my 28th birthday, my girlfriend at the time organised a cake which was a image representing kraya - which was basically megabus. I felt bad that she thought that kraya was the most important thing in my life - she was right - but it still felt bad. kraya had other clients at the time, but my time wasn’t monopolised by other clients, or indeed by kraya - my heart still belonged to megabus.

And it would all have been fine too, except for a grave miscalculation I made.

In 2010, after trying to rebuild the ticketing system for £500k, and making some mistakes with people I trusted, kraya ended up in £150k in the hole. We needed some money urgently.

I was desperate and naively, I reached out to stagecoach for help. I thought they were my friend - that they would have my back.

They understandably lost a great deal of trust in my ability to manage and lead my company. I trusted the wrong person - but that was still my mistake. They were right.

I thought that I’d built up enough goodwill that they would help me through this. I’d felt I would have way more than that “in the bank” in terms of goodwill. I learned that professional relationships do not work that way that dark afternoon, standing outside my office on the phone, in the rain.

They didn’t make my life easier. Instead, I’d ended up rattling the cage - they were now panicked - realising their over-reliance on an organisation that could disappear at any point.

Instead of support, I had further actions, renegotiating the contract and what felt like punitive, and definitely invasive reporting obligations.

I was hurt and angry. I had poured my heart, my soul - hey, my very life into this product that I loved.

Suffice it to say - I got no help - no loan, no offer of investment - though they did suggest buying us outright - which I rejected.

I signed a contract under circumstances I would not wish on anyone.

The best I got from them was a challenge - if we were really spending more time than we were charging for - prove it. I did! We documented every minute we were spending - I wasted my time on spreadsheets, pointless meetings and work to try and rebuild the broken trust.

We went from £300k in the hole to £200k profit within a year. We charged for a whole year in support around 20% of what the system made in a day.

I was 28 years old.

Around the same time, I was also dealing with the operational aftermath of trying to build a java EE ticketing system over six months for £500k. I thought it would take a year and cost £1m. In hindsight, it needed two years and probably three million pounds.

Over 18 months, I personally answered over 150 out of hours emergency calls. We had a rota and others on call too - but I took the vast majority of these calls. I felt bad putting others through what I knew was gruelling.

All of this led me down a narrower and narrower path to a serious breakdown - though I didn’t know enough to name it until many years later. All I knew - all I felt was that something broke in me.

We managed to resolve all of the issues, but the deployment of that version kept getting pushed.

Stagecoach cancelled the contract in 2012. They had started building a ticketing system in-house two years prior - the cost of my grave mistake. I wasn’t able to make the meeting - I was in India, and at the same time as the meeting, I was meeting for the first time the one who is now my wife.

I was 28 years old. I spent the next 15 years putting myself back together.

How much did it cost them to build it inhouse? If I had charged for my time from the start, would we all have been better off?

I still feel something deep inside me every time I see a megabus - a sense of pride mixed in with a deep sense of sadness - not for what I lost - but for what could have been.

I am 44 years old, and I am starting again.