A critical vulnerability required an emergency binary build from source. The fix was ready, the code was tagged, the CI pipeline triggered — and the runner ran out of memory mid-compilation. During an active incident.
The Situation
The upstream project released an emergency patch for a security vulnerability. Every node running this software needed the update immediately. The patch was a source-code change — no pre-built binaries available yet. The only option: build from source, package as a Docker image, deploy.
The CI pipeline had a workflow for this: checkout code, run make, build Docker image, push to registry. It had worked before on smaller builds.
The Failure
c++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[2]: *** [Makefile:576: libbitcoin_wallet_a-wallet.o] Error 1
The compiler was killed by the OOM killer. The GitHub Actions runner had 8GB RAM. The compilation needed more — linking the final binary alone consumed 6GB+ of peak memory.
The build failed 18 minutes in. The vulnerability was still unpatched.
The Workaround
A colleague built the image locally on a machine with 32GB RAM. It worked, but it took 40 minutes and the image had to be manually pushed to the registry. Not repeatable, not automated, and one person became the bottleneck.
The Real Fix
I built an ephemeral development machine tool — scripts that spin up a beefy cloud server on demand:
./spinup.sh # Creates a 16-core, 32GB server
./push.sh # Syncs code to the server
./run.sh # Runs the build remotely
./pull.sh # Pulls the built artifact back
./teardown.sh # Destroys the server
A cloud server with 16 cores and 32GB RAM built the same binary in 5 minutes 35 seconds. Cost: less than $0.50 for the hour.
For CI, I added a dedicated large runner profile (16 cores, 32GB) that only gets used for these heavy builds. Regular builds stay on standard runners.
The Numbers
| Approach | Time | Cost | Reliability |
|---|---|---|---|
| Standard CI runner (8GB) | Failed at 18min | Free | 0% — OOM |
| Laptop build (32GB) | ~40 min | Free | Manual, one person |
| Ephemeral cloud server (32GB) | 5m 35s | ~$0.50 | Automated, repeatable |
| Large CI runner (32GB) | ~8 min | ~$0.15/run | Fully automated |
Takeaway
Standard CI runners can’t compile everything. If your build involves compiling large C/C++ codebases from source (database engines, network daemons, game engines), test the build on a runner with matching specs before you need it in an emergency. Have a fallback: either a dedicated large runner profile or a script that spins up an ephemeral machine. The worst time to discover your CI can’t handle a build is during an active incident.