By the time it was 2015, I did not.

I had poured blood, sweat and tears into a ticketing system that kraya built — first for megabus, then for Polskibus. It had broken me, and we were now just limping along.

By 2015, we were spending 101–286 hours each month on a support contract that paid for 60. I raised this with the client and suggested a minimum of a 100% increase. They refused. Instead, they minimised their requests to just about 80 hours each month. Without the development work, I was now making a loss each month providing the resources for this support contract.

I could cancel the contract, but that would involve letting them have the source code, which was otherwise kraya’s property. There was a six month notice period — six months of providing all the support they need to run and to migrate the system to their team.

At some point, I went from taking on the challenge to holding the fort.

I could see that any effort to make things easier would backfire. I built them a feature for free — they complain about one bug in it — and they need it fixed urgently. I loosen the rules and deploy an additional server just before the weekend, letting them know about the risks. Something goes wrong and they are surprised.

The hardest thing for me to change was the belief that the client’s wins were my wins. If anything, the client’s wins were my loss. It meant more unpaid work in the support contract. It meant more complex systems to support and maintain — while every penny was being questioned.

I started to say no. No, we will not deploy new servers on a Thursday because the weekend is too close. No, we will not reduce the QA time on this bit of functionality you want. No, we will not restore reports which could display erroneous data.

It all came to a head in one specific instance where the CEO insisted on speaking to me because they needed something deployed urgently. They needed us to work through the evening or the weekend. I said no. They offered to pay double. I said no. They were not happy.

A few months later, they cancelled the contract.

I remember the meeting. It was amicable and friendly. They asked if I could keep the staff on till the end of the year - they wanted backup in case anything went wrong during the migration.

I wouldn’t want to let them go before Christmas anyway.

I had a brief conversation with my brother to decide what to do next. I already knew that there was nothing left to keep running. We kept it running for three more months until the end of December.

I remember going into the main office, gathering everyone around and delivering the news. Polskibus had cancelled. We are shutting kraya down. We’ll keep going until the end of December.

I remember the Christmas party. It had often ended up being drinks, and being out all night. It didn’t this year - we went home after the meal. The atmosphere was one of sadness and relief. We’d all been through the fires and made it out the other end. It was over.

It was the end of what I’d built over 15 years.

I felt nothing.

It was time for kraya to take a leap. We should take the megabus.com ticketing system to the next level and build a distributed ticketing system that could potentially scale infinitely.

I’d picked JBoss because it was backed by Red Hat - it had all these features and capabilities - a lot of which we needed. The other option we considered was Glassfish, but it just didn’t have the features we needed. There were other options but that involved prohibitive licensing fees.

We costed it out at £650k and a year. It was unrealistic but I could not imagine Stagecoach paying more, or giving us more time. They wanted it for £500k and in six months. I should’ve pushed back, but we’d built a booking system over the weekend, only a few months back - this should be possible, right?

It wasn’t.

We got a year in the end — we didn’t ask for it, and we didn’t know until we were most of the way down the path. When I heard about the delay it was a mix of relief and regret. We’d burned through most of the budget on getting in contractors who were leaving imminently. We could have got fewer people on board, but as permanent staff.

We launched to Canada first, and that wasn’t too bad. Then it was the US, and the nightmare started. The UK launched last on my birthday in 2010.

In the intervening years, we had a budget shortfall of £150k and because we’d rushed to build the product, we’d cut corners, and all the contractors had left. We even had to let go of some of the permanent staff. I asked Stagecoach if they’d fund us the extra £150k as we’d asked at the start. They said no. They renegotiated the contract. They demanded more oversight - and increased our reporting obligations.

Load testing was already a part of our process and we tested the new system under load. We identified issues and fixed them. On paper, it looked good.

It wasn’t.

The problem wasn’t load per se. It was the stability of the system. It struggled to stay up for extended periods of time. The more nodes there were, the worse it was.

It was exactly the kind of problem that was hard to replicate and hard to test. The main option we had was to think through what all it could be and to take stabs in the dark. I put together a hit list and worked through it methodically. Convincing Stagecoach to spend the money was sometimes harder than solving the problem.

Ultimately, the one thing that pushed us over the line in terms of stability was staggered nightly automated restart of each node.

I thought I was done working with software that needed regular restarts to stay functional and at first resisted this. We are running enterprise level software, and that too on Linux. If I was comfortable with such shenanigans, I might have stayed with Windows. But we were running out of options. It was a hail Mary - it worked.

I had been working with Linux and related software for years by that point. I, in fact had servers that had not been restarted for literal years at that point, with services that were running just as long. A lot of these services didn’t even need a restart on config change - just a reload.

None of these services even had a paid tier. That should have been the clue.

I cannot imagine having to restart PostgreSQL nightly or even Apache. I still do not understand how something slated for the enterprise market can have leaks that would warrant a regular restart to keep it working.

Years later, when I looked up issues around JBoss, I realised that it was notorious for a whole slew of problems with JGroups and clustering. I remember scouring the internet for details on any kind of issues and coming up empty.

I’d fallen into a marketing trap. JBoss was no Apache — it was the commercial product of Red Hat. I thought all enterprise open source software would be of the same calibre. It wasn’t.

From what I understand, Stagecoach spent millions and maybe two years building the whole ticketing system inhouse. kraya limped along for a few more years before being shuttered.

The people involved, though, fortunately seem to have gotten through it largely unscathed. It gives me a great deal of joy to see so many of the juniors I’d hired now CTO’s, VP’s, Directors.

I believed that every victory was ours but when something went wrong, it belonged to me. After all, I made every choice. I chose to pursue the Java EE ticketing system. I chose JBoss. I chose to keep going.

I was in a narrowing path, with fewer and fewer options.

I knew I was the only one holding that line - I didn’t know that there was another option.

Bit by bit, I lost all sense of what I was paying to fix these mistakes.

It was the same every time. I would drag myself up to answer the phone - my body, my mind screamed at me, but I had gotten good at overriding every instinct through sheer willpower. I could hear the apologetic tone on the other side, and I could recognise some of the voices after a while. I mustered up all of my strength to be and sound as awake as possible. I needed to be professional even if I was still in my underwear.

I had a glass of water on my bedside table. I’d pick that up and head to my office in the spare room. The computer was always on and always ready to go — like me I guess. I’d log on to the servers, and check the logs. If I can identify which one fell out of the group, I can restart just that one. If I was too late or if the issue had escalated, I’d have to restart the whole cluster — shut them all down, give them a few seconds, then bring each one up, while keeping an eye on them. I could do it half asleep after a while.

Falling back asleep wasn’t a breeze either - I was tired - exhausted - but I was now also wired. Waking up in the morning was harder - the alarm would go off and my body would be limp. I still remember the sheer power of will to drag myself into the shower, then carry on with the rest of the day.

Of 266 incidents over about two years, I answered 156.

I remember one particular night, though I do not remember how many times I’d woken up beforehand. I was already tired.

megabus.com had gone offline. I got an alert. “But someone else is on call tonight,” I told them. “We already tried them twice,” came the reply. I had to deal with this. I had to deal with this.

I remember sitting at my desk at home working on fixing it. At some point, something was different, though I don’t remember what. While I was working on fixing it, I remember being overcome with an overwhelming impulse to get up from the chair and walk away — I almost imagined myself walking away. I resisted and shut down that impulse. I fixed megabus as I had always done. In fixing megabus though, something broke inside me, somewhere deep, in the very core of my being. I was never the same again.

I analysed the system top to bottom, inside and out. I even waded through JVM internals.

It got incrementally better, more stable. I think I rewrote every component that wasn’t the core ticketing system. In the end, what pushed it over the line were two unexpected changes. Automated nightly restarts of each node in the cluster and a rate limiter.

On the 10th December 2012, the system, now serving Polskibus, had a sale event. We had a bank of screens on a wall with all the key stats for the system. It looked cool, and we felt a bit like we were on a TV show. At peak, nearly 15,000 concurrent sessions — six or seven times the average. Over 30,000 bookings in a single day, three times more than the normal amounts across all systems.

We watched it closely, all day. Nothing broke. Nothing screamed. Everyone smiled, but there was no celebration.

Truth be told - it also very nearly killed me! I had a breakdown which left me an emotional and mental wreck, as if I’d been hit by a bus (10 points if you tell me which one!)

It has taken me the better part of 15 years to recover from that, and in the meantime I tried working for others and I enjoyed it. In the end, though, I think once you have run your own company for a while, it is incredibly difficult to do anything else. It was inevitable that if I ever got well enough, that I would want to start something of my own again.

It was a long and hard journey, but I am finally better, and I am ready to start again. I am taking all the hard earned lessons into this venture, and I want to do it even more differently this time.

I promised myself that this time around, I would:

• Surround myself with people who will challenge me and expand my perspective.
• Accept the fact that I do not have all the answers
• Do things the way that we are meant to:
  • Bring in expertise we do not have
  • Have someone in the team right from the start focused on marketing
  • Technically, BDD, TDD, Agile, Lean, ““The only way to go fast is to go well””
• Focus on people first
• Focus on the journey, not the destination

One thing I realised as part of this journey is that doing things differently means that all the tools, technologies, and processes out there don’t quite fit.

With #muster, we are starting with one of the easier parts, and streamlining the developer experience. We are ending up building a lot of other things in the process, but they all take longer.

With choosing to do things in a very different way, we don’t know exactly how it’ll all pan out, but it will an interesting journey