Debugging production without redeploying

The loop everyone knows

A customer reports something your logs do not explain. The line you need was never printed. So you add it, open a pull request, wait for CI, deploy, wait for the condition to happen again, and read the result. Often it shows that you logged the wrong variable, and the loop starts over.

Each turn costs hours, and the cost is not only time. While you wait, the incident stays open, and the pressure to try something faster grows: attach a debugger to a live process, add a print that changes timing, flip a value to see what happens. That instinct is how an investigation turns into a second incident.

Two problems hiding in one loop

The slow loop has two separate causes, and they need separate fixes.

The first is instrumentation that cannot answer questions after the fact. A log line that says "payment failed" without the order, the attempt number or the upstream response forces a redeploy for every follow-up question. A trace that stops at the first queue or service boundary hides exactly the hop where the problem usually is.

The second is no safe way to look at live state. When the only tools are "redeploy with more logging" or "poke the running system", teams alternate between being slow and being dangerous.

Make the payload answer the next question

Write logs and errors for the person who will read them during an incident, not for the person writing the code. A useful diagnostic payload carries the identifiers that let you follow one request across systems, the inputs that decided the branch taken, and the outcome of the call that failed. It leaves out what you would never search for.

Carry trace context across every boundary you control: HTTP calls, queues, background jobs. A trace that survives the hop into a worker turns "somewhere after checkout" into a specific job and a specific line.

Probes that cannot change what they inspect

When you do need to look at running code, use probes that are read-only by construction: they read state and report it, and they cannot write, retry or mutate. The rule is simple to state and easy to forget under pressure, which is why it is worth writing down as a rule rather than relying on judgement during an incident.

This matters even more when an AI assistant is helping with the investigation. An agent that is allowed to "try a fix" against a live system will eventually try one. An agent constrained to read-only probes can explore freely, because the worst it can do is waste a query.

Read logs like a budget

The last step is often skipped: reading. Dumping thousands of undifferentiated log lines into a chat with an assistant burns its context on noise and buries the one line that matters. Filter first, by request identifier, time window and severity, and give the reader, human or model, the smallest slice that can still answer the question.

Putting it into practice

The Production Telemetry and Live Debugging Rules turn these habits into four Cursor rules: diagnostic payloads and live probes always on, trace context and log reading scoped to your observability code. The pack adds a playbook for the order to look in, from the outside inwards, and zero-redeploy probe examples for Node.js, Python and Go.

For the related problem of a model changing behaviour under a feature with no deploy at all, see your LLM feature got worse and nothing in your diff changed.