Chapter 04 · Building it for real

Integration & testing.

Integration is where assumptions go to die. The discipline is to design for testability before you write code, lock interfaces before you build internals, and stand up integration test points incrementally so the last six weeks of a project aren’t a fire drill.

In this chapter

  1. What integration is, and why it dominates schedules
  2. Design for testability
  3. Lock interfaces first, internals second
  4. Incremental integration
  5. Bring-up priority discipline
  6. Building Without Hardware in Hand
  7. Technology Choices: Popular vs Appropriate
  8. Documentation as Integration Tool

What integration is, and why it dominates schedules.

What it is. Integration is the work of taking subsystems that have been developed independently and making them function together as one system. It includes physical assembly, electrical interconnection, software interaction, timing coordination, and the long debugging tail that always follows when the assumptions of separate teams collide.

Why it dominates schedules. Almost every project that slips, slips here. Subsystems work in isolation. The interfaces drift during development. The first time the whole thing talks to itself end-to-end is two weeks before delivery, and it doesn’t. The team blames itself for being unprepared, but the problem was structural: integration was treated as a phase at the end, not as a discipline practiced from day one.

The mental model. Integration is what happens when assumptions made in different rooms have to agree in one room. The comms team assumed messages were 64 bytes. The avionics team assumed they were 256. Both assumptions were reasonable. Both teams worked competently. The integration is where they discover that one of them has to redo work, and the schedule doesn’t care.

The discipline. Three habits, applied throughout the project rather than at the end:

  • Design for testability before you design for function. Build the test points, the observability, the debug interfaces into the architecture from the first review. Retrofitting them after the design is locked is painful, expensive, and the team that needs them most is the team that gets denied them by schedule pressure.
  • Lock interfaces before internals. The contract between subsystems matters more than what each subsystem does internally. Internal redesigns are cheap if the interface is stable. Interface changes are expensive because they cascade through every subsystem that touches the interface.
  • Integrate incrementally. Do not wait for everything to be done. Stand up the integration test environment as soon as any two subsystems can talk, and verify the basic path on every change after that. The cost of finding a bug at the integration test point is small. The cost of finding the same bug in a fully-stacked system at acceptance is the schedule.
The principle that ties it together. Most projects fail integration not because individual components are bad — they’re usually fine — but because they don’t work together. The discipline of integration engineering is the discipline of thinking about “how will this connect” before “how will this work,” and refusing to defer that question to the schedule’s last six weeks.

Design for testability.

What it is. Designing the system so its internal behavior can be observed, controlled, and verified from the outside. Test points, instrumentation hooks, debug interfaces, controllable state — all the affordances that let an engineer answer “is this part working?” without disassembling the system to find out.

Why it matters. Things break. The question is not whether they will, but whether you’ll be able to diagnose them when they do. Test points let you measure. Observability lets you see. Without them, debugging is guesswork — and guesswork on a complex system is how week-long bug hunts happen on what should be ten-minute fixes.

The trap. Test points are the first thing scheduled out of a tight design. They cost board area, they cost connectors, they cost firmware to expose them. The team that skips them feels like it saved cost. The team that included them spends one tenth as long debugging the first bug, and the savings dwarf the up-front investment within weeks.

Where to add test points.

Power Rails

Every major voltage rail needs measurement access:

  • Voltage test points (before and after regulators)
  • Current sense resistors or Hall sensors
  • Load switches to isolate subsystems
  • Indicator LEDs for quick visual checks

Data Buses

Communication interfaces need probe access:

  • SPI/I2C/UART: expose CLK, MOSI, MISO, CS lines
  • CAN bus: high-side and low-side differential pairs
  • Ethernet: TX/RX pairs accessible for scope probing
  • Logic analyzer headers for multi-signal capture

RF Chains

Radio paths need multiple measurement points:

  • RF test points after PA, before antenna
  • Directional couplers for TX power monitoring
  • RSSI or AGC voltage monitoring for RX
  • LNA bypass for troubleshooting RX issues

Debug Interfaces

Every processor needs debug access:

  • JTAG or SWD exposed (don't bury it)
  • UART console for printf debugging
  • Dedicated debug connector (not shared pins)
  • Boot mode selection (normal vs debug/recovery)

What to Measure

Subsystem Key Measurements Why It Matters
Power Voltage, current, ripple, efficiency Detect brownouts, shorts, thermal issues
Communications Signal integrity, bit error rate, latency Verify protocol timing, catch data corruption
RF TX power, RX sensitivity, SNR, frequency error Ensure link budget margins, diagnose range issues
Timing Clock frequency, jitter, phase noise Real-time systems fail with timing violations
Thermal Component temperatures, gradient mapping Prevent thermal runaway, validate cooling
Built-in diagnostics: Don't just add test points—add self-test routines, health checks, and telemetry hooks. Software should report its own status: "watchdog fired 3 times," "I2C bus timeout on sensor 2," "temperature approaching limit." Make the system tell you what's wrong.

Lock interfaces first, internals second.

The principle. An interface is a contract between two parts of the system. Like any contract, the cost of changing it grows with how many things have come to depend on it. Internal implementations can evolve cheaply — you change one place and the rest of the system doesn’t notice. Interface changes propagate. They cascade through every subsystem on either side, every test that exercises the boundary, every document that describes the connection. The cost of an interface change three months in is roughly the cost of every internal change touching that interface, multiplied by the number of teams that have to redo work.

What to do about it. Lock interfaces early, with a written specification both sides agree to. Then let internals develop in parallel behind those interfaces. The specification is what aerospace and defense projects call an Interface Control Document, or ICD; the form matters less than the discipline of writing the contract down before the development behind it has fossilized.

The diagnostic question. If you change a subsystem’s internals, can you do it without coordinating with another team? If yes, the interface is doing its job. If no, the interface is leaking implementation details and is going to be expensive to maintain.

What to define in an ICD.

Electrical Interfaces

  • Voltage levels (3.3V, 5V, differential)
  • Current draw (max, average, inrush)
  • Connector type and pinout
  • Impedance matching (for high-speed)
  • Grounding and shielding strategy
  • ESD protection requirements

Mechanical Interfaces

  • Mounting hole patterns and spacing
  • Envelope constraints (max dimensions)
  • Connector orientation and access
  • Cable routing and bend radius
  • Thermal interface (heatsink contact)
  • Keep-out zones for other subsystems

Data Interfaces

  • Protocol (UART, SPI, CAN, Ethernet, USB)
  • Baud rate or clock frequency
  • Message formats (packet structure)
  • Timing requirements (setup, hold, latency)
  • Error handling (CRC, retry, timeout)
  • Flow control mechanism

Power Sequencing

  • Startup order (which rails first)
  • Delay requirements between rails
  • Brownout behavior (what happens below threshold)
  • Shutdown sequence (graceful vs emergency)
  • Hot-swap capability (if required)
  • Fault isolation (prevent cascade failures)
Version Control Your ICDs

ICDs are living documents. Track every change, get sign-off from affected teams, and maintain revision history. A single "quick fix" to an interface without updating the ICD causes integration disasters. Treat ICD changes like code changes: review, approve, document.

Incremental integration.

The principle. Integrate subsystems one at a time, in a known order, with a verification step between each addition. Big-bang integration — everything assembled at once, powered on at once — produces a pile of broken parts and no diagnostic signal. When something fails, the failure could be in any of the subsystems or any of the interfaces between them, and the team is debugging in the dark.

Incremental integration produces diagnostic signal. When you add subsystem N to a working assembly of N-1 subsystems, any new failure is, by elimination, caused by what changed. The bug isolates itself. The cost of integrating one subsystem at a time is small. The cost of debugging a fully-stacked assembly without that isolation is large and unpredictable.

Integration steps.

  1. Power-on test: Before anything else, verify power rails come up cleanly. Measure voltages, check for shorts, verify sequencing. There is no point testing data if power is broken.
  2. One interface at a time: Add subsystems sequentially. Computer boots → add sensor → verify I2C communication → add actuator → verify SPI. Isolate failures to specific interfaces.
  3. Known-good baselines: After each successful integration step, save that configuration. If the next step breaks something, you can roll back to the last working state. Version control for hardware integration.
  4. Stub out missing subsystems: Don’t wait for final hardware to start integrating. Use simulators, dev boards, or dummy loads to exercise interfaces early. Replace stubs with real hardware incrementally as it arrives.
  5. Integration checkpoints: Define clear pass/fail criteria at each step. Document what works, what doesn’t, and what workarounds were needed. Lessons learned feed forward into the next step.
The 80/20 rule of integration: 80% of integration time is spent debugging interfaces. Only 20% is actual assembly. Plan accordingly — schedule integration time as if you’re debugging, not building.

Bring-up priority discipline.

When new hardware first comes alive, you have a choice about what order to bring up the layers. The intuitive choice is to enable everything — security, framing, error correction, the high-speed interfaces — on day one, because that’s what the final system needs. The disciplined choice is the opposite. Bring up the dumb thing first. Prove it works. Then add complexity in layers, verifying each one before the next is enabled.

The principle. The first thing you bring up should be the simplest thing that could possibly work. Every layer you add before the basic path is verified is a layer you will have to debug through later, when something fails and you can’t tell which layer is at fault.

Why this matters mechanically: when you debug a failure on a fully-stacked system, you are debugging through every layer between the source of the bug and the symptom you observed. If the symptom is “data didn’t arrive” and the stack is encryption / framing / rate-limiting / transport / hardware, the failure could be in any of five places. Each layer is a place where the test fails silently for a different reason. The longer your stack at first power-on, the more time you spend isolating the failure rather than fixing it.

The right order is the reverse: hardware first, then transport, then framing, then any payload semantics, and only then the production-ready additions like encryption and rate-limiting. Each layer added is one more thing you can independently verify works before turning on the next. When the system is fully stacked and running cleanly, you have certainty about every layer beneath the one you’re currently debugging. That is the only way to make integration tractable.

The temptation to skip this discipline is real. The team feels like they’re “wasting time” bringing up a stripped-down version of something they already plan to ship in a richer form. But the time spent on the dumb path is small. The time saved on later debugging is enormous, because every subsequent failure can be isolated to the layer that changed.

From the field

A ground station bring-up. The first two weeks of the schedule went into setting up encrypted command links and rate-limited high-throughput telemetry channels. The team was proud of the work — modern, secure, future-proof. Week three, they tried to send a basic ping. It didn’t go through. Nobody could tell whether the transmitter was off, the receiver was deaf, the antenna was misaligned, the encryption was misconfigured, or the data path was broken. There were six layers of complexity between “send a byte” and “the byte arrives,” and any one of them could have been wrong.

They spent four days debugging through encrypted, framed, multi-protocol stacks before someone suggested bypassing all of it and sending a raw unframed test signal directly. That worked in fifteen minutes. The actual bug was a cabling issue in the RF front end — which would have surfaced on day one if anyone had checked the dumb path first.

The lesson. Encryption, protocols, framing, rate-limiting — every layer you add before the basic path is verified is a layer you have to debug through later, often under schedule pressure, often without good tools for the layer that’s actually broken. Bring up the dumb thing first. The four days saved is reliably worth more than the perceived professionalism of bringing up the polished stack on day one.

Building Without Hardware in Hand

Reality check: You'll rarely have final hardware when development starts. Deadlines are fixed, but hardware delivery slips. Smart engineers derisk by building incrementally, testing on approximations, and designing for testability. Don't wait for the perfect environment—create strategic stepping stones.

Development Timeline (Without Final Hardware)

1

Weeks 1-4: Simulations & Models

Build software models before touching hardware. Mathematical models for RF link budgets, thermal simulations, power analysis. Software-in-the-loop (SIL) for algorithm development. Remove algorithmic uncertainty before hardware complexity.

Deliverable: Proven algorithms, validated assumptions, identified risks

2

Weeks 5-10: Development Boards / COTS

Use off-the-shelf hardware to validate interfaces early. Arduino, Raspberry Pi, STM32 dev boards—whatever's close enough. Prove software runs on real hardware with interrupts and timing constraints. Test interface protocols (SPI, I2C, UART) with actual devices.

Identify gotchas: Race conditions, buffer overflows, timing violations, real-world noise

3

Weeks 11-16: Breadboard / Proto PCBs

Build functional approximation with target components. Use actual chips, connectors, power supplies you'll fly. Electrical verification: signal integrity, noise, power consumption. Integration testing between subsystems. Iterate quickly—breadboards are disposable.

Find problems now: Before committing to expensive PCB fabrication

4

Weeks 17-24: Engineering Model

First real PCB, actual form factor, representative hardware. Not flight-ready, but electrically and mechanically similar. Full system integration: all subsystems talking. Environmental testing if available (thermal, vibration).

Purpose: Find design flaws before flight hardware commits

5

Week 25+: Flight Hardware

Final hardware arrives—but your software already works. Minimal surprises because you've tested approximations. Focus on qualification testing, not basic functionality. Timeline risk reduced: integration is incremental, not big-bang.

Result: Confidence from proven performance on similar hardware

Key insight: Each stage removes one type of uncertainty. Simulation removes algorithmic uncertainty. Dev boards remove software/hardware interface uncertainty. Breadboards remove electrical uncertainty. Engineering model removes mechanical/thermal uncertainty. By the time you get flight hardware, you've validated everything except the exact components.
Don't follow trends blindly: Just because everyone uses Kafka doesn't mean your system needs it. Analyze your actual requirements. Sometimes the "boring" technology is exactly right. Sometimes you need something niche. The question is always: does this solve my problem better than alternatives?

Common Technology Trade-Offs

Message Queues: Kafka vs Alternatives

Kafka is great when:

  • High throughput (millions of messages/sec)
  • Distributed system with multiple consumers
  • Need message replay and persistence
  • Have ops team to manage cluster

Simpler alternatives when:

  • RabbitMQ: Easier ops, good routing, lower throughput OK
  • Redis Streams: In-memory speed, simpler setup
  • NATS: Lightweight, low latency, embedded use cases
  • Direct socket: Ultimate simplicity for point-to-point

Databases: SQL vs NoSQL vs File

PostgreSQL when:

  • Need transactions and consistency
  • Complex queries and joins
  • Data has clear schema

Alternatives:

  • SQLite: Single node, embedded, zero-config
  • MongoDB: Schema flexibility, document-oriented
  • InfluxDB: Time-series data (telemetry, logs)
  • Flat files: Telemetry dumps, log rotation, simplicity

Networking: REST vs gRPC vs Custom

REST when:

  • Human-readable debugging matters
  • Widely supported clients
  • Request/response pattern sufficient

Alternatives:

  • gRPC: Binary efficiency, streaming, type safety
  • MQTT: Pub/sub, low bandwidth (IoT, embedded)
  • WebSockets: Bidirectional, real-time updates
  • Raw TCP/UDP: Ultimate control and efficiency

Processing: Microservices vs Monolith

Microservices when:

  • Large teams, independent deployments
  • Different scaling needs per service
  • Polyglot requirements (multiple languages)

Monolith when:

  • Small team (< 10 people)
  • Simple deployment preferred
  • Network latency matters
  • Starting new project (defer complexity)
Decision framework: (1) What problem am I solving? (2) What's the simplest solution? (3) Does complexity pay for itself? (4) Can I test/maintain it? Popular technology attracts talent and has good docs, but don't let that override fit-for-purpose analysis.

Documentation as Integration Tool

If it's not documented, it doesn't exist: Integration depends on shared understanding. When subsystems are developed by different teams (or even different people), documentation is the only way to ensure everyone's building compatible pieces. Skimp here and pay during integration.

Essential Integration Documents

  • Interface Control Documents (ICDs): Sacred contracts between subsystems. Version controlled, reviewed, signed-off.
  • Integration procedures: Step-by-step instructions. Not "plug it in and see," but detailed, ordered, verified sequences.
  • Test plans: What to verify at each integration milestone. Clear pass/fail criteria, no ambiguity.
  • Troubleshooting guides: Common failure modes and diagnostic steps. "If X fails, check Y, measure Z."
  • Configuration management: Track hardware revisions, software versions, which combinations work together.
  • Lessons learned log: Capture surprises, workarounds, and root causes for next time.
Update Documentation in Real-Time

Don't wait until after integration to document what you learned. Capture it immediately: "We had to add a 10µF cap on the 3.3V rail to fix SPI glitches." Six months later, you won't remember. Write it down now, in the ICD, with date and initials.