Test Wars Episode III: Return of the Dev-Confidence
There's one topic that consistently proves challenging to grasp and implement: CI/CD – specifically, how to transition to a world where every Pull Request (PR) is a deployment to production.
I remember a conversation with a large corporation. They asked, "How many releases do you do a year?" I smiled and replied, "I don't know, more than a thousand?" They were stunned. With over 200,000 employees, they were still managing monthly releases for their main applications.
So, what's the secret sauce to achieving this seemingly mythical state of continuous deployment?
It takes time: Usually, this transformation takes between 6 months to a year for small teams [less than 50 people].
It takes a village: Your team culture needs to evolve, and often, your talent strategy too.
It takes discipline: The way you work is probably not conducive to CI/CD.
It takes software: People can only move so fast; the right tools are essential.
And the most exciting part? This entire process is accelerating, thanks to what some call vibe-coding, or as you know it, AI!
So, How Did We Do It?
It's actually very straightforward when you break it down:
(0) PRs need to be small. This is fundamental. Small changes reduce entropy in your system. If your changes are minor, the probability of introducing bugs diminishes significantly. Every PR gets max. 2 peer reviews and then automatically merges to master. That's it. Move on.
(1) Every development is feature flag protected. For any mid-sized or major change, you must be able to roll back instantly. Ideally, you have two mechanisms:
- An 'immediate rollback' capability (at my latest venture, we can do this in under a minute).
- Feature flags to meticulously control who sees what, and when.
(2) Shift left – and let AI help. Now, more than ever, it's crucial to invest time upfront: thinking through what you want to code, and critically, how you are going to test it. Defining your testing strategy becomes paramount. AI is proving to be a powerful ally in this "shift left" approach, aiding in everything from test case generation to analysis.
A few missteps to avoid
Over-focusing on unit and integration tests vs automated E2E
Time and again, we've seen this: a service implements a new API or modifies an existing one. Unit and integration tests pass with flying colors. But then, users (human or otherwise) interact with the functionality in a slightly different way, and things break. That's the killer.
The Fix: Invest heavily in a robust suite of End-to-End (E2E) tests that cover actual user workflows. This ensures functionality remains consistent from the user's perspective and reduces rework across the entire organization.
The Result: Even as usage volumes continue to skyrocket, we've reduced the portion of the team dedicated to "Keep The Lights On" (KTLO) activities from 20% to less than 5%. This is largely thanks to this strategic shift in testing.
Underestimating putting ideas into words
So many times, a feature that seems simple and straightforward on the surface turns into a technical quagmire. We realize too late that we chose the wrong data type for our database, the wrong foreign key, or that we were inadvertently trying to solve an NP-hard problem. Now, the whole thing needs a rethink. At best, it's a database migration; at worst, we're saddled with a legacy decision that will slow down the team for years.
The Fix: The simple act of sitting down, thinking through the functionality, writing it down, and reviewing it with the team – a process that shouldn't take more than a few hours – saves an incredible amount of time in the long run.
The Result: It's funny, many engineers who worked with me through this transformation initially resisted this "process." Now, they tell me how they've successfully implemented it in their new teams. Once a Tech Lead sees it in action for six months or more, everyone embraces it.
Bonus Point: When you adopt this upfront thinking, you'll find your engineering team becomes more proactive. They develop insights and feedback for Product, UX, and QA because they've already thought through the entire challenge. Your team moves beyond 'quick fixes' to focus on the value they deliver.
Achieving true CI/CD is a journey, not a destination. It requires patience, discipline, the right culture, and leveraging smart tools, including the ever-evolving capabilities of AI.
Thank you for reading! I'm always happy to explore any of these concepts in more depth.