Reliability on Tachera W Sasi, Backend & Infra Engineer (Go, Self-Hosted, Observability)

The One-Word Bug That Broke Our Worker for 6 Days

Tue, 28 Apr 2026 00:00:00 +0000

—2" tags: [“devops”, “docker”, “golang”]

For six days, our system was lying to us

Not crashing. Not failing loudly. Just quietly pretending everything was fine while doing absolutely nothing.

And the cause?

One wrong word in a shell script.

Everything Looked Fine

We run a pretty standard setup:

A Go API (core)
A background worker (minion)
Redis for queues
Docker Compose holding it all together

We recently moved bulk SMS processing into the worker using Asynq. Clean separation, better scalability, the usual win.

We deployed.

The API responded perfectly:

200 OK
"bulk SMS queued successfully

Except… no SMS was ever sent.

The Worker Was Dead Silent

First instinct: check the worker logs.

Nothing.

No errors. No panics. No trace of the task ever being picked up.

Just one quiet warning after retries were exhausted.

Meanwhile, everything else worked:

login alerts ✅
email notifications ✅
scheduled jobs ✅

Only bulk SMS failed.

At this point, we did what engineers always do when the signal is weak:

We guessed.

We Investigated the Wrong Things

We went deep into the stack:

Suspected Redis corruption
Questioned queue priorities
Looked for serialization bugs
Added panic recovery middleware
Logged full payloads
Added startup diagnostics

We deployed again.

Still nothing.

And this is where most people waste days. We almost did.

Because we were debugging behavior… not reality.

The Subtle Clue

There was one tiny inconsistency.

Our worker used to log:

Worker started

We had changed it to:

Worker started (server + scheduler running)

After deploy, the logs still showed the old message.

That should have been the end of the investigation right there.

Our code wasn’t running.

The Real Problem

The issue wasn’t in Go.
It wasn’t in Redis.
It wasn’t in queues.

It was in this script:

docker compose build --no-cache api 
docker compose up -d --remove-orphans

And this config:

services:
 core:
 minion:
 db:
 redis:

Look closely.

We were building api.

There is no api.

Docker didn’t complain. It didn’t warn. It didn’t fail.

It just did nothing.

And then docker compose up happily started containers using old cached images.

So for six days:

The API sometimes rebuilt
The worker never rebuilt
The system kept deploying successfully
Our fixes never actually shipped

We were debugging code that wasn’t even running.

The Fix Was Embarrassing

- docker compose build --no-cache api
+ docker compose build --no-cache core minion

That’s it.

Two words instead of one.

We deployed again.

Immediately:

New startup logs appeared
Bulk SMS tasks were picked up
Jobs executed in milliseconds

System fixed.

What This Actually Teaches

1. If your fix doesn’t change behavior, assume it never deployed

Stop overthinking.

Before touching Redis, queues, or concurrency… ask:

“Is my new binary even running?”

If you skip this, you deserve the hours you’re about to lose.

2. Silent failures are worse than crashes

Crashes force action.

Silent no-ops pass CI, pass deploys, and waste days.

docker compose build should not succeed when the service doesn’t exist.

But it does.

So you have to defend against that yourself.

3. Your deployment pipeline is part of your codebase

People treat scripts like they’re disposable.

They’re not.

That one-line deploy.sh caused more damage than any bug in the Go code.

Review it. Test it. Break it intentionally.

4. Logging is not optional for workers

If your worker can fail quietly, it will.

You need:

startup fingerprints (commit hash, build time)
task-level logs
explicit success/failure visibility

Otherwise, you’re blind.

5. Rename things properly or don’t rename them at all

api became core.

The script didn’t.

That mismatch cost six days.

Refactors don’t end when the code compiles.

The Real Lesson

This wasn’t a debugging failure.

It was a discipline failure.

We trusted the pipeline without verifying it.
We chased complex explanations before ruling out simple ones.
We debugged symptoms instead of checking fundamentals.

And that’s how a one-word mistake beats experienced engineers.

Originally published on Medium.

Why Most Engineers Shouldn't Build Distributed Systems (And Why I Did Anyway)

Thu, 05 Mar 2026 00:00:00 +0000

—2"

Distributed systems are one of the most fascinating areas in software engineering.

They’re also one of the fastest ways to accidentally create massive complexity.

A lot of engineers jump straight into microservices, container orchestration, and distributed architectures because it sounds scalable and modern.

But in reality, distributed systems introduce an entirely new category of problems.

When your system runs on a single machine, many things are simple.

Function calls are instant.
Memory access is predictable.
Failures are easier to understand.

For example, a local function call might look like this:



func processOrder(orderID string) error {
 order := getOrder(orderID)
 chargeCustomer(order)
 updateInventory(order)
 return nil
}

Everything happens inside one process.
No network. No timeouts. No partial failures.

But once your system becomes distributed, the same operation might involve multiple services communicating over the network.



func processOrder(orderID string) error {
 order, err := orderService.Get(orderID)
 if err != nil {
 return err
 }


 err = paymentService.Charge(order.CustomerID, order.Total)
 if err != nil {
 return err
 }


 err = inventoryService.Reserve(order.Items)
 if err != nil {
 return err
 }


 return nil
}

Now things become much more complicated.

What happens if the payment service succeeds but the inventory service fails?

What if the network times out but the request actually succeeded?

What if the service crashes halfway through the workflow?

These are common distributed system problems.

Failures Become Normal

In distributed systems, failures are not exceptions.

They are expected behavior.

A simple HTTP call can fail in many ways:



resp, err := http.Get("http://node-service/tasks")


if err != nil {
 // network failure 
 retry()
}

But what kind of failure is it?

The server might be down
The network might be slow
DNS might fail
The request might succeed but the response gets lost

Handling these cases correctly requires retries, timeouts, and idempotent operations.

Scheduling Work Across Machines

One of the challenges I encountered while building a cloud orchestration system was scheduling workloads across multiple nodes.

A very simplified scheduler might look like this:



type Node struct {
 ID string
 Capacity int
 Used int
}


func schedule(nodes []Node, job Job) *Node {
 for i := range nodes {
 if nodes[i].Capacity-nodes[i].Used >= job.Resources {
 return &nodes[i]
 }
 }
 return nil
}

This looks simple.

But real systems quickly introduce more challenges:

nodes going offline
jobs failing midway
resource fragmentation
network delays
distributed state

A production system requires heartbeat monitoring, leader election, failure recovery, and task rescheduling.

Suddenly the architecture grows significantly.

Debugging Becomes Harder

Debugging distributed systems is also more complex.

In a monolith, you can follow logs sequentially.

In a distributed system, a single request might pass through several services:



API Gateway -> Auth Service -> Order Service -> Payment Service -> Inventory Service

To debug issues, you often need distributed tracing tools to reconstruct the path of a request.

Without observability, diagnosing issues becomes extremely difficult.

Do Most Companies Need This?

In many cases, the answer is no.

A well designed monolith can scale surprisingly far.

Many successful products run on monolithic architectures much longer than engineers expect.

Distributed systems should only be introduced when they solve a clear scaling or reliability problem.

But There Is Still Value in Learning Them

Even though most systems don’t need distributed architectures immediately, understanding them provides enormous value.

Once you understand how systems fail across networks and machines, you begin designing software differently.

You think more about:

fault tolerance
system boundaries
reliability
observability

Those lessons improve every backend system you build.

Distributed systems are powerful tools.

But like all powerful tools, they should be used carefully.

Originally published on Medium.