Nobody hires an engineer for the problems with known answers. The job is the ones where you do not know what is wrong or how to proceed. This chapter is the method for exactly that moment: take control of the situation, cut it into testable pieces, check the boxes one at a time, and land the answer back in the real system, fast, and without anyone above you panicking.
Four UDP packets left the sender within five milliseconds of each other; a packet capture on the loopback interface said so, with timestamps. The receiver, polling far faster than that, reported them arriving twenty milliseconds apart. Twenty milliseconds, in a system built to move samples to a DAC on a hard clock, is a lifetime. Nothing in the code did anything slow. Nothing in the kernel had changed. The numbers simply disagreed with each other, and every engineer in the room had a theory, which is the same as having none.
An intermittent failure nobody can reproduce. An integration that refuses to come up. A performance target you cannot hit. A system doing something the design says is impossible. The details differ; the feeling is the same. It is the flat, slightly sick moment of I do not know how to get through this.
The first instinct is always the same too: try things. Change something, rebuild, power-cycle, hope. Resist it. Random action feels like progress, but it destroys the evidence, burns the schedule, and, worst of all, sometimes makes the symptom go away without telling you why, which means it will be back, at a worse time, in front of a customer.
The difference between panic and professionalism is having a method you trust for the moment knowledge runs out. The method has five moves: take control of the situation, analyze it into testable hypotheses, build small test solutions that check boxes one at a time, tie what you proved back into the real system, and communicate clearly the entire way. It works on a failing RF link, a crashing process, a stalled vendor, and most things in between.
Every hard problem starts with the same sick feeling, and the method is what you reach for instead of the reboot button.
A problem you cannot solve yet is manageable. A situation you do not control is the dangerous thing. Before any technical work, convert the crisis into a bounded, written, owned problem. Control comes from writing things down, long before it comes from having the answer.
Stabilize first. If something is actively failing, a fielded system, a demo tomorrow, a customer watching, the first move is containment rather than diagnosis. Get to a known state: fall back to the last working configuration, reduce scope to the part that works, or safe the system entirely. You cannot think clearly while the fire is spreading, and nobody else can either.#
Write the problem statement. One or two sentences, in observable terms: what happens, under what conditions, and how that differs from expected behavior. “The link drops” is a complaint. “The receiver loses lock for 2 to 5 seconds, roughly every 10 to 20 minutes, only above 40 C ambient” is a problem statement. If you cannot write the statement yet, that is fine; it means your first task is gathering the observations that let you write it, and now you know that.#
Inventory what you actually have. Four lists, on paper:#
The four lists. The known list is short and strict. The assumed list is where the bug usually lives.
Known: facts you have directly observed or measured. Nothing else goes on this list.
Assumed: things everyone believes but nobody has verified. This list is where the bug usually lives.
Unknown: what you would need to know to explain the behavior.
Changed: everything different since it last worked: code, hardware, environment, configuration, people.
You control a situation the moment you can say: here is what we know, here is what we assume, here is what we are testing next, and here is when we will know more. You need the next step, long before you need the answer.
Stabilize, write the problem statement in observable terms, and fill in the four lists; control arrives with the next step, long before the answer.
You cannot test “the system is broken.” You can test “the failure is upstream of the demodulator.” Analysis is the act of cutting a vague problem into pieces small enough that each one has a yes-or-no experiment attached. Three habits do most of the work.
Decompose along the boundaries you already have. Draw the system as a chain of blocks with defined interfaces, the same boundaries you designed and integrated along. The failure lives in a block or in an interface between blocks. The structure of the system tells you where you are allowed to cut; this is half the payoff of having designed for testability in the first place.#
Binary-search the failure space. Pick the test that eliminates half of the remaining possibilities rather than the test that is easiest to run. Signal clean at the midpoint of the chain? You just excluded half the system. Fault follows the unit when you swap hardware? It is the unit, not the environment. Three good cuts turn eight suspects into one.#
Eight suspects, three probes, one left. Each probe lands at the midpoint of what remains, which is rarely the easiest place to reach and always the most informative.
Back to the four packets that arrived twenty milliseconds apart. The first cut was the capture itself: the packets left on time, so the sender and the network were eliminated in one move. The second cut was the print statements in the receive loop, which were removed, and the numbers changed: the second packet now arrived 1.5 milliseconds after the first, and the third and fourth arrived 15.8 milliseconds later. A cleaner symptom, and a real one. The third cut was inside the receiver, timestamping each stage of the path from the kernel ring to the application, until the 15.8 milliseconds landed in a single function: a pure ring-index lookahead with no system calls in it, the last place anyone would have looked and the only place the evidence allowed.
The lesson. Each probe was placed where it would split the remaining suspects, and each one made the symptom sharper instead of vaguer. A stall in a function with no system calls is a strange place to end up, which is exactly why guessing would never have gone there.
Rank your hypotheses like an engineer, not a fan.#
List every explanation that fits all the evidence, not just your favorite. A hypothesis that ignores one observation is wrong, however elegant.
Rank by probability times cost-to-test. A likely cause with a five-minute test goes first. An unlikely cause needing a week of setup goes last, even if it is more interesting.
Write the disqualifying observation for each. Before you run a test, state what result would prove the hypothesis wrong. If no result could disqualify it, it is a hunch rather than a hypothesis.
Can you say, in one sentence, what your next test will tell you regardless of which way it comes out? If the answer only helps when the test “works,” you are guessing rather than experimenting.
And from the first minute: keep a written log. What you tried, what you observed, timestamped. Intermittent problems get solved by correlation across days of notes rather than by memory. The log is also the raw material for every status report you will owe later in this chapter.
Cut along the boundaries you built, probe at the midpoint, and rank hypotheses by probability times cost-to-test, with the disqualifying result written down first.
The full system has too many variables to reason about at once. A test solution, a harness, a stub, a bench setup, a stripped-down replica, isolates one question and answers it definitively. Each answer is a box checked, and checked boxes stay settled in meetings.
One hypothesis per test. Change one variable; hold everything else fixed.
Smallest setup that produces a definitive answer. A 50-line harness beats a full build.
Reproduce the failure before attempting the fix. If you cannot make it fail on demand, you cannot prove you fixed it.
Cheapest disqualifying test first. Spend an hour eliminating a suspect before spending a week confirming one.
Record the result either way. A hypothesis eliminated is progress.
Keep the harness. It is tomorrow’s regression test.
The checkbox discipline. Maintain the hypothesis list as a literal checklist. Every entry carries a status: untested, eliminated (with the evidence), or confirmed (with the evidence). This stops you from re-testing what is already settled, it makes progress visible to anyone who asks, and it forces honesty, because an entry cannot move to “eliminated” on a feeling.#
A receiver was losing lock for 2 to 5 seconds every 10 to 20 minutes, but only above 40 C ambient. The team’s first week went to rebuilding firmware and reseating boards: random action, no evidence, no progress. The reset was a ranked hypothesis list that fit all the observations, including the temperature dependence: a reference oscillator drifting with temperature, a supply sagging under thermal load, a firmware timing bug fed by a temperature-dependent sensor path, and a connector expanding intermittently.
Then the test solutions, cheapest disqualifier first. The oscillator alone went into a thermal chamber with a frequency counter logging, a thirty-minute setup. The board ran from a bench supply with current logging. The firmware ran against a simulated sensor feed at fixed temperature. Each rig answered exactly one question.
The lesson. Box one confirmed it: the oscillator drifted past the tracking loop’s pull-in range at 43 C. The other three boxes were eliminated with logged evidence nobody could re-litigate. Two days of directed tests closed what a week of thrashing on the full system could not touch, and the thermal-chamber rig became a permanent acceptance screen for every oscillator after that.
A hypothesis eliminated on Tuesday is a meeting you will never have on Thursday.
One hypothesis per harness, reproduce before you fix, cheapest disqualifier first, and keep the harness as tomorrow’s regression test.
Proving something on the bench is different from fixing the system. The bench proved the mechanism; the system has the interactions. The last mile, from validated test result to deployed fix, is where careless engineers reintroduce the chaos they just spent a week eliminating.
Port the fix under real constraints. Your harness lacked the real timing, the real power budget, the real concurrency, and the real environment. Re-apply the fix inside the actual system and re-run the reproduction case there. If the failure was intermittent, decide up front how long a clean run must be before you call it fixed, statistically rather than hopefully. A failure that struck every twenty minutes needs hours of clean soak, and one good demo proves little.#
Fast solutions come from the method, never from skipping it: a tight problem statement prevents wandering, ranked hypotheses prevent expensive detours, small harnesses turn around in minutes instead of full-build hours, and the log prevents re-doing work you already did. Every shortcut above buys minutes now and costs days later.
I was brought into a codebase whose sample generation was underflowing badly; the team had been working features and had never focused on throughput. The temptation was to patch the hot spots one at a time. Instead I spent two days on analysis, measuring where the cycles went and where the samples stalled, then wrote a short proof of concept that generated samples the straight way, with no allocation and no detours, to confirm the hardware could keep up. Only then came the fix: about 1800 lines, rewritten in under a week, built on what the analysis and the proof had already established.
The lesson. Analysis, then proof of concept, then the fix. The sequence looks slower than diving in and is reliably faster, because every line of the fix was written knowing what it had to do. The usual approach skips the first two steps and spends the saved time on the third, several times over.
Two disciplines keep the speed honest. Timebox every test. If an experiment cannot produce its answer in the time you allotted, that is data too: either the test is wrong-sized or the cause is buried deeper than you ranked it. Take workarounds deliberately. When the schedule forces a patch before the root cause is fixed, that can be the right engineering call, but write down what is temporary, why, and when it comes out. A workaround you chose is engineering. A workaround you drifted into is debt.
Slow is smooth, smooth is fast. The team that spends the first hour writing the problem down finishes the week ahead of the team that spent the first hour rebooting things.
Analysis, then proof of concept, then the fix: the sequence that looks slow is the one that finishes first.
While you are heads-down, everyone above and around you is deciding whether to trust you, help you, or panic. Clear communication is what buys you the room to do the work. Silence buys you a daily status meeting.
Every update is the same four lines, whatever the audience:
What we know. Observed facts only, with numbers. Theories go in line three.
What we have ruled out. The checked boxes, with the evidence. This is what makes progress visible before the answer is found.
What we are testing next. The current hypothesis and the experiment against it.
When we will know more. A specific time for the next update, and then deliver it at that time, even if the update is “the test is still running.”
When the audience is a customer or a program office, the four lines become four slides, and I have used the same four for years: accomplished since the last report, challenges in the way, support needed from the people in the room, and upcoming before the next one. The third slide is the one that earns its keep. It converts a status meeting into a meeting where someone can actually do something.
And a few rules that keep trust once you have it. Separate fact from hypothesis in every sentence: “the oscillator drifts 8 ppm” is a different kind of statement from “we suspect the loop cannot track it,” and your credibility depends on your listeners always knowing which kind they are hearing. No surprises: bad news travels from you, first, with a plan attached. Write for the reader: the program manager needs impact and dates; the engineer next to you needs waveforms and register values. Say what was proven, how, and what soak time remains, rather than “it should work now.” And when you are stuck, say so and ask; stating a problem cleanly to another person solves a remarkable fraction of them before he or she says a word.#
Stabilize and write the problem down. Split it until each piece has a test. Run the cheapest disqualifying tests first and check the boxes in writing. Prove the fix in the real system, beyond the bench. And at every step, tell people what you know, what you have ruled out, what is next, and when you will report again. That is what control looks like before you have the answer.
Take the bug that has been open longest and write its problem statement in observable terms. If you cannot, list the observations you would need, and go get the first one.
Write the four lists, known, assumed, unknown, changed, for that bug. Pick the top item on the assumed list and design the five-minute test that would disqualify it.
Send one status update in the four-line form to whoever is waiting on you, with a time for the next one. Then deliver the next one at that time.
What we know, what we ruled out, what is next, when we will know more: the same four lines buy you the room to do the work.
This chapter on one page
Each line is the sentence to remember from its section. The number takes you back to it.
7.1Every hard problem starts with the same sick feeling, and the method is what you reach for instead of the reboot button.
7.2Stabilize, write the problem statement in observable terms, and fill in the four lists; control arrives with the next step, long before the answer.
7.3Cut along the boundaries you built, probe at the midpoint, and rank hypotheses by probability times cost-to-test, with the disqualifying result written down first.
7.4One hypothesis per harness, reproduce before you fix, cheapest disqualifier first, and keep the harness as tomorrow’s regression test.
7.5The bench proved the mechanism; prove the fix in the real system, soak it for long enough to mean something, and ask where else the same cause lives.
7.6Analysis, then proof of concept, then the fix: the sequence that looks slow is the one that finishes first.
7.7What we know, what we ruled out, what is next, when we will know more: the same four lines buy you the room to do the work.