You Can't Automate a Process That Doesn't Exist

• Role: Lead Platform / DevOps Engineer (Consulting)
• Timeline: 3 Months (after 2 years of previous internal failures)
• Stack Context: AWS, ECS, Atlantis, Terraform, Databricks, Spark, Airflow

The Background & The “Two-Week Coffee Tour”

A mid-sized company operating at the intersection of SEO optimization, marketing spend, and e-commerce. They had six distinct engineering teams—Frontend, Crawling, Analytics, and others.

The business case for a single, homogeneous pipeline was obvious to everyone. Homogeneity means eliminating duplicated effort, slashing infrastructure maintenance overhead, and making developer onboarding across different teams practically seamless. On paper, it’s a no-brainer.

Yet, they had three failed attempts to build this unified pipeline, stretching over two painful years.

When I stepped in, the mandate from leadership was presented as a purely technical challenge: “Go build the unified pipeline.”

Instead of opening an IDE or drawing boxes on a whiteboard, I stopped. I spent the first two weeks doing nothing but talking to team leads, product managers, and individual engineers.

To the CTO, this looked like a total lack of progress. I could practically hear him tapping his fingers on the desk, wondering why his highly-paid consultant was just drinking coffee and chatting with people instead of writing YAML files.

But I needed to map the invisible chaos.

The Root Cause: Automating a Ghost

The two-week diagnostic phase revealed exactly why the previous three attempts had crashed and burned. It wasn’t a tool problem. It was a process problem.

The teams were operating in completely different universes:

• Tech Stacks: A wild mix of Python, Java, Kotlin, and Node.js.
• Git Workflows: Some used feature branches, others used production/staging branches. Some ran builds on Pull Request creation; others built only after a merge.
• Infrastructure as Code (IaC): Half the teams kept their Terraform code in the same repository as their application workloads. The other half maintained dedicated infrastructure repos.

In short, there was no standard engineering process.

A CI/CD pipeline is not magic. It is merely an automated conveyor belt. You cannot build an automated conveyor belt for a manufacturing process that changes every time a different worker steps onto the floor.

Before designing a tool, we had to design and negotiate the process itself.

The Architecture of Compromise: Golden Paths vs. Autonomy

To get six teams to agree on a single pipeline without causing an engineering rebellion, we had to separate the non-negotiable security/stability guardrails from the day-to-day tooling preferences. We established a “Golden Path” approach built on three pillars:

1. The Non-Negotiables (and the cost of behavioral change)

We established a strict, universal gate. No code could reach staging or production without:

• Pre-merge unit tests.
• Dependency vulnerability scanning.
• Static Application Security Testing (SAST).
• Test coverage validation.

But let’s be honest: implementing these gates was not a technical configuration; it was a forced behavioral shift. It meant forcing several teams to radically change their daily habits.

If you force engineers to change how they work without getting their genuine buy-in, they will view your platform as bureaucratic friction. At best, they will be frustrated; at worst, they will actively bypass the system, creating “Shadow IT” and secret deployment backdoors.

We had to spend significant time and political capital explaining why these rules existed, proving their value, and winning the teams over. The engineers had to believe in the destination before we forced them onto the road.

2. The Standard Interface (The Humble Makefile)

Different technologies inherently require different build, test, and security scanning mechanics. You cannot treat a JVM-based monolith the exact same way you treat a lightweight Python web scraper. Attempting to force-feed a single, rigid pipeline script to all of them is a recipe for instant developer friction. We needed explicit points of customization and extensibility—places where teams could inject their own toolchains without risking global pipeline stability.

Dockerfiles solve isolation, but they don’t solve orchestration abstraction at the execution layer. So, instead of over-engineering a complex, custom YAML configuration parser, we decided to use something older than almost everyone working at the company: a humble Makefile with a standardized interface.

Every repository, whether running Kotlin, Python, or Node.js, had to support a few basic commands: make test, make scan, and make build. The pipeline didn’t need to understand how Python dependencies were packaged or how Java resolved its classpaths; it just ran the make target. As for the output? We piped all test results into good old, universally-supported JUnit-compatible XML reports. Why reinvent the wheel when the industry solved this problem decades ago? The technology inside the container remained the team’s business. The interface to the platform became ours.

3. The IaC Hybrid Model

If you want to start an infinite argument in a room full of DevOps engineers, just ask them how to structure Infrastructure as Code. I know at least five competing “industry best practices” for this, and most of them are highly opinionated and completely incompatible.

As a battle-scarred platform engineer, I might have personally leaned towards Terragrunt or a more rigid repository structure. But pragmatism always beats dogma. Since most of the teams were already using Atlantis for pull-request-driven Terraform orchestration, we decided to stick with it.

Atlantis allowed us to preserve repo structure flexibility. We killed the structural debate by implementing a hybrid approach. Global infrastructure (networking, ECS clusters, shared databases) lived in a central, platform-controlled repository. Service-specific workloads (ECS task definitions, micro-services) lived right next to the application code. This allowed engineers to keep using the directory structures natural to their specific frameworks, without sacrificing global state safety.

It also helped that the tech stack was built on AWS ECS. In this context, ECS was an absolute blessing. It meant we could keep everything nicely encapsulated in native Terraform, completely avoiding the nightmare of mixing IaC with Helm charts, Kustomize templates, or Kubernetes YAML soup.

The Strategic Sacrifice: Cutting the Data Team

One of the most critical decisions of this project was knowing when not to standardize.

From Day 0, it was clear that the Data Team’s workflow was completely incompatible with standard software engineering practices. They were running Airflow, Spark, and Databricks. Trying to force their heavy, stateful data pipelines into a lightweight application deployment pipeline would have delayed the entire project by six months and alienated a key team.

So, I made a conscious call: we excluded them from the initial rollout.

We built the core software engineering pipeline first, proved it worked, and got the other five teams running smoothly. Only then, with credibility on our side, did we return to the Data Team.

The sequel? We eventually built a tailored, secondary pipeline for them, complete with a custom internal testing framework designed specifically for Databricks and Spark. But diving into the details of that here would be textbook scope bloat—and we have enough of that in our industry already…

The Result

Six weeks after starting, we began the technical implementation. By the end of month three, the unified pipeline was fully deployed, tested, and actively adopted by all target teams.

The CTO stopped worrying about my coffee consumption.

Executive Summary & Verdict

If you are currently planning a platform engineering or developer velocity initiative, keep these two realities in mind:

1. The Technical Principle

A CI/CD pipeline is just a tool that automates a process. It cannot automate a process that doesn’t exist. If your team’s workflows are chaotic, automation will only generate chaos at scale, faster.

2. The Cultural Principle

Importing “best practices” from past gigs feels like a great shortcut. Often it is — except instead of a route to efficiency, it’s just a faster lane to the exact same cliff. Only this time, driven with supreme confidence.

By day, I help companies scale their systems and engineering organizations without scaling the chaos. By night, under the pen name Adam Korga, I dissect the tech industry’s underlying absurdities, organizational politics, and failure patterns.