In this chapter
Every hard problem feels the same at first.
An intermittent failure nobody can reproduce. An integration that refuses to come up. A performance target you can’t hit. A system doing something the design says is impossible. The details differ; the feeling doesn’t. It’s the flat, slightly sick moment of I don’t know how to get through this.
The first instinct is always the same too: try things. Change something, rebuild, power-cycle, hope. Resist it. Random action feels like progress, but it destroys the evidence, burns the schedule, and — worst of all — sometimes makes the symptom go away without telling you why, which means it will be back, at a worse time, in front of a customer.
Take control of the situation.
A problem you can’t solve yet is manageable. A situation you don’t control is not. Before any technical work, convert the crisis into a bounded, written, owned problem. Control does not come from having the answer — it comes from writing things down.
Stabilize first. If something is actively failing — a fielded system, a demo tomorrow, a customer watching — the first move is containment, not diagnosis. Get to a known state: fall back to the last working configuration, reduce scope to the part that works, or safe the system entirely. You cannot think clearly while the fire is spreading, and nobody else can either.
Write the problem statement. One or two sentences, in observable terms: what happens, under what conditions, and how that differs from expected behavior. “The link drops” is not a problem statement. “The receiver loses lock for 2–5 seconds, roughly every 10–20 minutes, only above 40 °C ambient” is. If you can’t write the statement yet, that’s fine — it just means your first task is gathering the observations that let you write it, and now you know that.
Inventory what you actually have. Four lists, on paper:
- Known — facts you have directly observed or measured. Nothing else goes on this list.
- Assumed — things everyone believes but nobody has verified. This list is where the bug usually lives.
- Unknown — what you would need to know to explain the behavior.
- Changed — everything different since it last worked: code, hardware, environment, configuration, people.
You control a situation the moment you can say: here is what we know, here is what we assume, here is what we’re testing next, and here is when we’ll know more. You do not need the answer to be in control. You need the next step.
Analyze until it’s testable.
You can’t test “the system is broken.” You can test “the failure is upstream of the demodulator.” Analysis is the act of cutting a vague problem into pieces small enough that each one has a yes-or-no experiment attached. Three habits do most of the work.
Decompose along the boundaries you already have. Draw the system as a chain of blocks with defined interfaces — the same boundaries you designed and integrated along. The failure lives in a block or in an interface between blocks. The structure of the system tells you where you’re allowed to cut; this is half the payoff of having designed for testability in the first place.
Binary-search the failure space. Pick the test that eliminates half of the remaining possibilities, not the test that’s easiest to run. Signal clean at the midpoint of the chain? You just excluded half the system. Fault follows the unit when you swap hardware? It’s the unit, not the environment. Three good cuts turn eight suspects into one.
Rank your hypotheses like an engineer, not a fan.
- List every explanation that fits all the evidence — not just your favorite. A hypothesis that ignores one observation is wrong, however elegant.
- Rank by probability times cost-to-test. A likely cause with a five-minute test goes first. An unlikely cause needing a week of setup goes last, even if it’s more interesting.
- Write the disqualifying observation for each. Before you run a test, state what result would prove the hypothesis wrong. If no result could disqualify it, it isn’t a hypothesis — it’s a hunch.
And from the first minute: keep a written log. What you tried, what you observed, timestamped. Intermittent problems get solved by correlation across days of notes, not by memory. The log is also the raw material for every status report you’ll owe later in this chapter.
Test solutions: check the boxes.
The full system has too many variables to reason about at once. A test solution — a harness, a stub, a bench setup, a stripped-down replica — isolates one question and answers it definitively. Each answer is a box checked, and checked boxes don’t get re-argued in meetings.
- One hypothesis per test. Change one variable; hold everything else fixed.
- Smallest setup that produces a definitive answer. A 50-line harness beats a full build.
- Reproduce the failure before attempting the fix. If you can’t make it fail on demand, you can’t prove you fixed it.
- Cheapest disqualifying test first. Spend an hour eliminating a suspect before spending a week confirming one.
- Record the result either way. A hypothesis eliminated is progress, not failure.
- Keep the harness. It is tomorrow’s regression test.
The checkbox discipline. Maintain the hypothesis list as a literal checklist. Every entry carries a status: untested, eliminated (with the evidence), or confirmed (with the evidence). This does three things at once: it stops you from re-testing what’s already settled, it makes progress visible to anyone who asks, and it forces honesty — an entry cannot move to “eliminated” on a feeling.
A receiver was losing lock for 2–5 seconds every 10–20 minutes — but only above 40 °C ambient. The team’s first week went to rebuilding firmware and reseating boards: random action, no evidence, no progress. The reset was a ranked hypothesis list that fit all the observations, including the temperature dependence: a reference oscillator drifting with temperature, a supply sagging under thermal load, a firmware timing bug fed by a temperature-dependent sensor path, and a connector expanding intermittently.
Then the test solutions, cheapest disqualifier first. The oscillator alone went into a thermal chamber with a frequency counter logging — a thirty-minute setup. The board ran from a bench supply with current logging. The firmware ran against a simulated sensor feed at fixed temperature. Each rig answered exactly one question.
The lesson. Box one confirmed it: the oscillator drifted past the tracking loop’s pull-in range at 43 °C. The other three boxes were eliminated with logged evidence nobody could re-litigate. Two days of directed tests closed what a week of thrashing on the full system couldn’t touch — and the thermal-chamber rig became a permanent acceptance screen for every oscillator after that.
Tie it back into the real system.
Proving something on the bench is not the same as fixing the system. The bench proved the mechanism; the system has the interactions. The last mile — from validated test result to deployed fix — is where careless engineers reintroduce the chaos they just spent a week eliminating.
Port the fix under real constraints. Your harness didn’t have the real timing, the real power budget, the real concurrency, or the real environment. Re-apply the fix inside the actual system and re-run the reproduction case there. If the failure was intermittent, decide up front how long a clean run must be before you call it fixed — statistically, not hopefully. A failure that struck every twenty minutes needs hours of clean soak, not one good demo.
Then close the loop completely:
- Reproduce the original failure in the real system one final time — then apply the fix and show it gone under identical conditions.
- Run the regression suite. Verify the fix didn’t break what worked.
- Promote the test harness into the permanent test set, so this failure class stays caught forever.
- Update the documents that were wrong. The assumption you disproved is written down somewhere, and the next engineer will trust it.
- Ask where else the same mechanism applies. A root cause usually has siblings.
Speed is a byproduct.
Fast solutions do not come from skipping the method. They come from the method: a tight problem statement prevents wandering, ranked hypotheses prevent expensive detours, small harnesses turn around in minutes instead of full-build hours, and the log prevents re-doing work you already did. Every shortcut above buys minutes now and costs days later.
Two disciplines keep the speed honest. Timebox every test. If an experiment can’t produce its answer in the time you allotted, that is data too: either the test is wrong-sized or the cause is buried deeper than you ranked it. Take workarounds deliberately. When the schedule forces a patch before the root cause is fixed, that can be the right engineering call — but write down what’s temporary, why, and when it comes out. A workaround you chose is engineering. A workaround you drifted into is debt.
Communicate clearly.
While you’re heads-down, everyone above and around you is deciding whether to trust you, help you, or panic. Clear communication is what buys you the room to do the work. Silence buys you a daily status meeting.
Every update is the same four lines, whatever the audience:
- What we know. Observed facts only, with numbers. Not theories.
- What we’ve ruled out. The checked boxes, with the evidence. This is what makes progress visible before the answer is found.
- What we’re testing next. The current hypothesis and the experiment against it.
- When we’ll know more. A specific time for the next update — and then deliver it at that time, even if the update is “the test is still running.”
And a few rules that keep trust once you have it. Separate fact from hypothesis in every sentence — “the oscillator drifts 8 ppm” is a different kind of statement from “we suspect the loop can’t track it,” and your credibility depends on your listeners always knowing which kind they’re hearing. No surprises: bad news travels from you, first, with a plan attached. Write for the reader — the program manager needs impact and dates; the engineer next to you needs waveforms and register values. Never say “it should work now”; say what was proven, how, and what soak time remains. And when you’re stuck, say so and ask — stating a problem cleanly to another person solves a remarkable fraction of them before he or she says a word.
Stabilize and write the problem down. Split it until each piece has a test. Run the cheapest disqualifying tests first and check the boxes in writing. Prove the fix in the real system, not just on the bench. And at every step, tell people what you know, what you’ve ruled out, what’s next, and when you’ll report again. That is what control looks like before you have the answer.