“It seems fine” is an opinion. Verification is the disciplined act of proving each requirement is met, with evidence you can hand to an auditor or a customer. This chapter is about the difference between testing and proving.
The test suite was green. Three hundred tests, every one passing, every commit, for the better part of a year. The software had never once run on the hardware it was written for. It ran on the author’s laptop, where the timing was whatever the laptop felt like and the peripherals were stubs that always answered. When it finally met the target processor, it failed in the first minute, and the three hundred green tests had nothing to say about why. They had verified, thoroughly and repeatedly, that the software did what the laptop expected.
Two questions get confused here. Verification asks “did we build it right?” Validation asks “did we build the right thing?” The first is internal: does the system meet its specifications? The second is external: do those specifications solve the customer’s problem? You can verify a flawless implementation of requirements the customer never needed, and you can verify, flawlessly, against an environment the system will never see. You need both questions, and you need them asked against reality.
The V model. Needs and requirements decompose down the left arm; each level is answered by a test level climbing the right arm. Validation closes the loop at the top, against the need that started it.
The mental model. Verification is the building inspector checking that the construction matches the plans. Validation is asking the family whether the house fits how they live. The inspector cannot answer the family’s question, and the family cannot answer the inspector’s. Both visits are necessary, and they happen at different times for different reasons.#
Verification
Did we build it right?
Test against specifications. Measure performance against requirements. Check every interface. Verify error handling and edge cases. Document results objectively.
Validation
Did we build the right thing?
Customer acceptance testing. Does it solve the real problem? Is it usable in the operational environment? Does it meet the unstated but obvious needs? The customer signs off.
Verification proves the build matches the spec; validation proves the spec matches the need; and both are only proof when they are run against the real environment.
A team where the first integration test is the entire system. Every bug found requires the full stack standing up to reproduce. Every fix requires the full stack to verify. Every test cycle costs a coordinated team effort. Days of debugging burn on bugs that should have surfaced in seconds. The fix is testing at the right level for the bug you are hunting, with infrastructure built on purpose to catch each kind early.
The principle is simple: each tier introduces exactly one new variable. Tier 1 exercises pure logic with zero dependencies. Tier 2 adds component-to-component integration in a single process. Tier 3 adds real serialization and transport on a local machine. Tier 4 brings up the full software stack. Hardware enters only above Tier 4, as a separate problem with its own tiers. When a test fails at any tier, you know with certainty that the failure was introduced by what changed between that tier and the one below it.
Test isolation by tier: the software pyramid below, hardware integration above.
The pyramid shape is also the cost shape. Tier 1 tests are cheap, fast, and run on every commit. Tier 4 tests are expensive, slow, and run rarely. You want hundreds of tests at the bottom filtering bugs out cheaply before they ever reach the slow tiers. A team that only has Tier 4 tests pays full price to find every bug, including the ones a thirty-second unit test would have caught in isolation.
The names below are general. In any specific project the boundaries land where the architecture has natural seams; what counts as “transport” depends on whether you are dealing with gRPC, MQTT, shared memory, or raw sockets. The principle, each tier adds exactly one new thing, is what carries across.
Tier
What it tests
What it leaves out
Catches
Tier 1 Pure logic
Algorithms, state machines, data structures, message types.
I/O, threading, serialization, sockets.
Math errors, edge cases, contract bugs.
Tier 2 Component contracts
Components calling each other through their public interfaces, in a single process.
Above the software pyramid sits hardware integration, and it deserves to be treated as a separate problem rather than another tier on top. Software tiers test logic; hardware tiers test the physical environment. They are hunting different kinds of bugs, on different cost curves. By the time code reaches dev hardware, state-machine bugs and serialization bugs are already gone. What remains are the questions only hardware can answer.
Hardware tier
What it tests
Catches
HW Tier 1 Software on dev hardware
Verified software running on the target processor. No FPGA logic yet.
The codebase from the opening of this chapter had 220 source files and more than 300 test files, nearly all of it written by one remote engineer, and every one of those tests passed on his laptop. On the real hardware, the software did not work, and the tests could not say why, because nothing in them had ever touched a real clock, a real peripheral, or a real interrupt. Two junior engineers on the team had quietly stopped contributing; they could not find a way in. I deleted all of it and rewrote the application from scratch, with the hardware on the bench from day one and the lower tiers built to be run against it.
The lesson. A test suite that never meets the hardware is a Tier 4 pyramid with the top cut off: a great deal of confidence about the wrong environment. The tiers exist to carry the software toward reality one variable at a time, and a tier that stops short of the target is a tier that proves the laptop.
You will likely run HW Tier 1 as soon as anything compiles for the target, in parallel with continued work in the software pyramid. The lower tiers run continuously, on every change, because they are cheap; the upper tiers run when their cost is justified. The discipline is keeping the lower tiers alive throughout the project, rather than finishing each one before starting the next.
A team had built solid tiered tests during early development: no specific hardware required, fast feedback, clean isolation. When production hardware arrived, support for the development boards was dropped. The reasoning sounded sensible: “the dev hardware is no longer needed, supporting it costs engineer time, drop it.” The savings looked real on the schedule.
Months later, the only way to test anything was to bring it up as an entire system on production hardware, the most expensive, slowest, most contested test environment available. Every bug fix required the full stack standing up. Every regression chased on production silicon. The customer, who was paying for the hardware time, grew frustrated watching everything get pulled into the hardware lab; when I agreed with him and tried to stop it, the attempt was written up as a deviation from the plan and filed as a risk, where it sat without follow-up.
The lesson. Lower tiers get more valuable to keep as the project matures. The cost of maintaining a tier is fixed. The cost of debugging at the wrong level is paid every time a bug appears. By the time someone is willing to revisit the decision, the institutional knowledge to rebuild the tiers is usually gone too.
Maintained tiers are infrastructure. Treat them like the build system, the version control server, the CI pipeline: permanent, owned, and worth investing in across the project lifecycle. The team that keeps the lower tiers running pays the cheap price most of the time and the expensive price rarely.
This is a management decision as much as an engineering one#
Sustaining tiered test infrastructure costs engineer time, and that cost shows up on someone’s budget every quarter. Without management understanding the math, that the cost of the tiers is dwarfed by the cost of debugging at the wrong level, the tiers will erode under schedule pressure. If you are an engineer reading this without that backing, your job is to make the cost case explicitly, with numbers, before the erosion starts. Lifecycle has the shape of that argument.
Every bug is cheapest at the tier where it was born, and most expensive at the tier where it is finally noticed.
Each tier adds exactly one new variable, the lower tiers run on every commit, and keeping them alive is infrastructure rather than overhead.
A requirement names what must be true. A test procedure is the recipe for producing evidence that it is, written so that a different engineer, on a different day, would produce the same evidence. The gap between “we tested it” and “here is the procedure and the data” is the gap between an anecdote and a proof, and the procedure is where that gap closes.
Requirement. SR-001: the system SHALL deliver an alert to the base station within 5 minutes of the onset of a detectable event, for at least 95% of events, measured from the start of a calibrated chainsaw playback at any deployed node to receipt of the alert at the base station. Method: Test.#
Configuration under test. Node firmware 0.4.2 (build hash recorded), gateway firmware 0.4.0, base-station software 0.3.1. Three nodes at surveyed positions N1, N2, N3, with N3 two relay hops from the gateway. Gateway on the mast at the post.#
Equipment. The reference chainsaw recording (file hash recorded). A battery speaker set to 95 dB SPL at 1 m, checked with a sound level meter before each session. The operator’s phone and the base-station computer both synchronized to GPS time, agreement checked to within 1 second.#
Preconditions. Every node has reported a heartbeat within the last 10 minutes. Every node battery reads above 3.5 V. The preserve has no logging activity scheduled.#
Confirm clock agreement between the phone and the base station. Expected: within 1 second.
Position the speaker 50 m from the node under test, facing it.
Start playback and log the start time T0 from the phone.
At the base station, record the time T1 at which the alert for this node appears on the map, along with hop count, confidence, and gateway signal strength from the message log.
Stop playback after 90 seconds. Expected: one alert, from the node under test, with hop count matching its position.
Repeat for 20 trials spread across N1, N2, and N3, over at least two days, at least one of them wet.
Pass criteria. T1 minus T0 is 300 seconds or less in at least 19 of 20 trials. A trial with no alert within 300 seconds counts as a failure.#
Data recorded per trial. Node, T0, T1, latency, hop count, confidence, gateway signal strength, weather, operator initials.#
Witness. The customer’s representative observes at least five trials, including one at N3, and signs the record.#
Notice how much of the procedure exists to make the number mean something: the clock check, the calibrated speaker, the wet day, the two-hop node, the rule for a missing alert. Twenty trials on a dry afternoon at the nearest node would produce a smaller number and a weaker proof. The procedure is where the requirement’s conditions become the test’s conditions, on purpose.
A procedure is a recipe another engineer can follow to produce the same evidence, and the configuration under test is the first line of it.
Every requirement must have a test. Every test must trace to a requirement. If you cannot demonstrate compliance with a procedure, the requirement is unverifiable, and probably poorly written. The requirements traceability matrix from Requirements is where the loop closes: the status column changes from planned to passed one procedure at a time, and the day it is all green is the day the system is verified rather than merely finished.
Retesting and hoping is the amateur move. Root-cause the failure: is the requirement wrong, is the design inadequate, or is the procedure flawed? SR-007 above failed at three hops because the gateway’s duplicate filter dropped a legitimate relay with a wrapped sequence number, which is a design defect found by a test that was written to find exactly that. Fix the actual problem, rerun the procedure, and record both runs. Problem solving is the method for the hunt.
The matrix turns green one procedure at a time, and a failed row gets a root cause before it gets a rerun.
MTBF. Mean time between failures is the average operating time between failures across a population. It is a statistical statement about a fleet, and a weak one about any single unit. An MTBF of 10,000 hours means that, on average, a fleet accumulates 10,000 operating hours per failure; it says nothing like “each unit lasts 10,000 hours.”#
For a constant failure rate, the probability that a unit survives to time t is R(t) = e^(-t / MTBF). At t equal to the MTBF, that is 0.37: a unit has about a one-in-three chance of reaching its own MTBF without a failure. At one tenth of the MTBF, R is 0.90, so one unit in ten has failed by then. These are the numbers to have in your head when a datasheet quotes a large, round MTBF.
Example: estimating MTBF from a test
Test 10 units for 1000 hours each; 2 failures occur. MTBF = (10 units x 1000 hours) / 2 failures = 5000 hours, and the failure rate is 1 / 5000, or 0.0002 failures per hour.
Two failures is a small sample, and the honest statement is a range rather than a point: with so few failures the true MTBF could plausibly be half or double the estimate. Reliability numbers from short tests on small populations deserve wide error bars, and the error bars belong in the report.
What to ask a vendor. How the MTBF was obtained: a demonstrated test over how many unit-hours with how many failures, or a prediction from a parts-count handbook? At what temperature, load, and duty cycle? What counted as a failure? A predicted MTBF at 25 C under half load is a different number from a demonstrated one at the temperature your enclosure actually reaches.#
Derate. Components run at their maximum ratings fail early. Run power components at 50 to 70% of rated power, keep voltages under 80% of the maximum, keep junction temperatures well below the limit, and leave current margin on wires and traces. Derating is the cheapest reliability improvement available, and it is applied one part at a time during design rather than discovered one failure at a time in the field.#
Add redundancy where the function is critical. Dual power supplies in an N+1 arrangement, redundant communication links, a watchdog timer on every processor, redundant storage, and graceful degradation so that the system fails soft rather than hard. Redundancy is expensive and effective: two units in series with reliability 0.95 each give 0.90 together, while two in parallel give 0.9975. The Reference has the formulas.#
An MTBF is a promise about a fleet, delivered to a customer who owns one unit.
A unit has a one-in-three chance of reaching its own MTBF; ask how the number was obtained, derate every part, and add redundancy where the function is critical.
Lab conditions are the one environment the system will never see again. Temperature extremes, vibration, humidity, electromagnetic interference, electrostatic discharge: if the system will meet it in service, it meets it on a test fixture first, where a failure costs a day instead of a field recall. The environmental requirements written in Requirements say which conditions apply; the standards below say how to apply them repeatably.
Environment
What the test does
What it finds
Common references
Thermal
Operating range soak (often -40 C to +85 C), storage range (often -55 C to +125 C), thermal cycling between extremes, power-on soak at each limit.
Timing drift, cracked solder joints, components out of spec at temperature, condensation paths.
MIL-STD-810 methods 501, 502, 503; IEC 60068-2
Vibration and shock
Random vibration profiles for transport and operation, sinusoidal sweeps for resonance, mechanical shock and drop.
Radiated and conducted emissions, susceptibility to external RF, and ESD at connectors and seams.
Noisy regulators, unshielded cables, resets on a static discharge, interference with neighbors.
FCC Part 15; MIL-STD-461; IEC 61000-4-2 (ESD), -3, -4, -6
The order matters a little. Thermal and vibration find mechanical and electrical weaknesses that EMC testing would otherwise blame on shielding, so run them first. And a unit that has been through qualification testing is a different unit afterward; qualify on an engineering model, and keep the acceptance screen for production units lighter and non-destructive.
Thermal and vibration first, EMC after, on an engineering model, because the lab is the one environment the system will never see again.
Manual testing is slow, inconsistent, and resistant to repetition, which means it is skipped under exactly the schedule pressure that makes it necessary. Automated tests run the same way every time, catch regressions the day they are introduced, and free engineers to work on new functionality. They are also the only practical way to keep the lower tiers of the pyramid alive on every commit.
Automate
Unit tests, on every commit, always
Integration tests for the common interfaces
Regression tests that detect broken existing behavior
Performance benchmarks that catch degradation early
Smoke tests on every build
Hardware-in-the-loop runs against mocked and then real boards
Keep manual
Exploratory testing, where the unknown unknowns live
Usability and user experience
Complex environmental testing
Acceptance testing with the customer present
Hardware deserves a word, because “it needs hardware” is the usual excuse for leaving a test manual. Most of it does not. Mock the device at the driver boundary and the whole software pyramid runs on a build server. Then phase in real hardware one board at a time on a bench the pipeline can reach, and the test that used to need an engineer and an afternoon runs at 3 a.m. on every merge.
Automate everything that can run without a person, including the hardware tiers, and keep people for the exploratory and acceptance work that needs judgment.
Two tests get confused with each other, and the confusion costs either rigor or money. The first proves that a release of the design meets its requirements: a witnessed rebuild from source, loaded onto representative hardware, run against the full requirement set, once per release. The second proves that a particular manufactured unit works: a shorter screen, run on every unit that ships, against the subset of requirements that manufacturing can get wrong.
Running the full qualification on every unit is unaffordable and tells you nothing new after the first one. Running only the per-unit screen on a new release is how a software defect reaches every unit in the fleet with a stamp of approval. Keep the two procedures separate, name them differently, and make the traceability matrix say which requirements each one covers.
Take one requirement that is currently marked as tested and find its procedure. If the procedure is a sentence in an email, write the anatomy above around it, starting with the configuration under test.
Count your tests by tier. If the bottom of the pyramid is thinner than the top, pick one Tier 4 test and write the Tier 1 test that would have caught the last bug it found.
Find the MTBF you are quoting to a customer and write down how it was obtained. If the answer is “from the datasheet,” add the test conditions next to it.
Qualify the release once, with a witnessed rebuild; screen every unit with a lighter test; and never let one stand in for the other.
This chapter on one page
Each line is the sentence to remember from its section. The number takes you back to it.
6.1Verification proves the build matches the spec; validation proves the spec matches the need; and both are only proof when they are run against the real environment.
6.2Each tier adds exactly one new variable, the lower tiers run on every commit, and keeping them alive is infrastructure rather than overhead.
6.3A procedure is a recipe another engineer can follow to produce the same evidence, and the configuration under test is the first line of it.
6.4The matrix turns green one procedure at a time, and a failed row gets a root cause before it gets a rerun.
6.5A unit has a one-in-three chance of reaching its own MTBF; ask how the number was obtained, derate every part, and add redundancy where the function is critical.
6.6Thermal and vibration first, EMC after, on an engineering model, because the lab is the one environment the system will never see again.
6.7Automate everything that can run without a person, including the hardware tiers, and keep people for the exploratory and acceptance work that needs judgment.
6.8Qualify the release once, with a witnessed rebuild; screen every unit with a lighter test; and never let one stand in for the other.