Teardown Northwind Labs
Home Get a teardown
Production-Readiness Teardown · Sample Report

Infrastructure
teardown, ranked by what breaks first.

A five-day review of deployment, runtime, and operational risk — every finding scored by severity and effort, and mapped to the business event it actually threatens.

Client
Northwind Labs, Inc. (sample)
Profile
Seed · 14 engineers · 9 mo post-launch
Engagement
Fixed-scope · 5 business days
Stack
AWS us-east-1 · EKS · GitHub Actions
IaC / Data
Terraform (partial) · Postgres RDS
Access
Read-only IAM · GitHub read · Screen-share
Passive review only · No production credential custody · No active scanning or penetration testing
Section 01 · Executive Summary

The platform works. That was never the question.

The question this teardown answers is what happens under the three events already on your calendar: a funding announcement, a first enterprise contract, and the traffic that follows both. Across five days I found 19 issues — most ordinary and cheap. Three are not, and they compound.

0
Critical
0
High
0
Medium & Low
0
Fixable This Week
The three findings that matter most
01
Deploys cannot be rolled back. Images are tagged latest, so a bad release can't be reverted without a rebuild — recovery time is a build cycle, not a click.
02
The database is a single point of failure. Single-AZ RDS, no read replica, backups never restore-tested. An untested backup is a hypothesis, not a recovery plan.
03
You are blind during incidents. No structured logging, no tracing, no alerting on user-facing symptoms. You'll learn about outages from customers.
Section 02 · Domain Scorecard

Six domains, benchmarked to your stage.

Each scored 0–10 against what a company at your funding level should have in place. A relative benchmark, not an absolute one — a 6 is fine at seed and a liability at Series B.

Deploys & CI/CD
4/10
Kubernetes & Runtime
5/10
IaC & Environments
3/10
Observability & Incident
2/10
Security Posture
5/10
Cloud Cost Efficiency
6/10
Observability is the lowest score and the highest leverage. Everything else gets cheaper to fix once you can see what's happening.
DomainFindingsSingle most consequential gap
Deploys & CI/CD4No rollback path; mutable image tags
Kubernetes & Runtime3No resource limits on the API deployment
IaC & Environments3~40% of infrastructure created by hand in console
Observability4No alerting on user-facing error rates
Security Posture3Long-lived static IAM keys in CI
Cloud Cost2Idle staging cluster running 24/7
Section 03 · Critical & High Findings

Where one ordinary event becomes a bad day.

Each finding carries a severity rating, an effort estimate, and the business event it maps to. The full register — including ten medium and low items — appears in Appendix A of the delivered report.

FINDING TD-01
No rollback path — images tagged latest
Critical Effort · Low · ~1 day CI/CD
Observed

The GitHub Actions workflow builds and pushes to ECR under a mutable latest tag, then triggers a rolling restart. No immutable artifact is tied to a commit SHA, so a previously-working image isn't addressable once the next build overwrites it.

Consequence

Reverting a bad release requires reverting the commit, rebuilding, and redeploying — 8 to 14 minutes, assuming CI is green and nobody panics. Effective recovery time is closer to 25 minutes.

Business impact: On launch day, a bad deploy costs 25 minutes of downtime instead of 30 seconds — the exact window when new users form their only impression of the product.
FINDING TD-02
Single-AZ database, backups never restore-tested
Critical Effort · Medium · ~3 days Runtime
Observed

Primary Postgres runs as a single-AZ RDS instance. Automated snapshots are enabled with 7-day retention, but no restore has run since the instance was created 11 months ago. No read replica. Connection pooling is app-side only, with a 100-connection ceiling your current peak already reaches.

Consequence

An AZ-level event takes the product fully offline for the duration of a manual restore — realistically 40 to 90 minutes — with the added risk that an untested backup fails to restore cleanly. The single largest concentration of risk in the system.

Business impact: This is the finding that fails an enterprise security review, the one investors' technical diligence surfaces, and the outage that produces a public postmortem.
FINDING TD-03
No alerting on user-facing symptoms
Critical Effort · Medium · ~5 days Observability
Observed

CloudWatch collects infra metrics but nothing is alerted on. Application logs are unstructured plain text with no correlation IDs, making it impossible to trace a single request across the API, worker, and inference services. No tracing. No on-call rotation or escalation path.

Consequence

Mean time to detection is effectively how long it takes a customer to write a support email. Mean time to diagnosis is unbounded, because the data needed to diagnose isn't being captured.

Business impact: During your next incident, the honest answer to "how long has this been broken?" is "we don't know." That answer costs more than the outage when a customer asks it.
FINDING TD-04
Long-lived static IAM keys in CI secrets
High Effort · Low · ~1 day Security
Observed

Deployment authenticates with an IAM user's access key pair — created 11 months ago, never rotated, stored as a GitHub Actions secret. The attached policy grants broader ECR and EKS permissions than the pipeline uses. GitHub OIDC is not configured.

Business impact: A compromised CI account becomes a compromised AWS account. This is question three on every enterprise security questionnaire you're about to receive.
FINDING TD-05
No resource requests or limits on core workloads
High Effort · Low · ~1 day Kubernetes
Observed

The api and worker deployments declare neither requests nor limits. The scheduler can't make informed placement decisions, and a memory leak in either service can starve co-located pods. The HPA is configured but inert without requests to scale against.

Business impact: Under a traffic spike, autoscaling never triggers. The system degrades instead of scaling — precisely when you least want it to.
FINDING TD-06
~40% of production infrastructure exists only in the console
High Effort · High · ~8 days IaC
Observed

Terraform covers the VPC, EKS cluster, and RDS. Security groups, S3 buckets, IAM roles, the ALB, and all CloudWatch config were created by hand. State lives in a local backend committed to the repo — so two engineers applying concurrently corrupt it.

Business impact: The environment can't be reliably reproduced. Disaster recovery becomes manual reconstruction from memory, and an infra hire loses weeks to archaeology.
FINDING TD-09
Staging runs 24/7 at production node sizing
Medium Effort · Low · ~0.5 days Cost
Observed

Staging runs three m5.xlarge nodes continuously, mirroring production. Off-hours utilization is under 4%. Estimated waste is $680–840/month — roughly $9,000 a year.

Business impact: A line on the AWS bill funding nothing. Post-raise, that's runway.
Section 04 · Top Quick Wins

Seven fixes, one week, ordered by leverage.

Each shippable within a week by one engineer, ranked by risk reduced per hour spent. If nothing else from this report gets done, do these.

Tag images by commit SHA and pin deployments
~3 hours · Resolves TD-01 · CI/CD

Tag with ${{ github.sha }} alongside latest and reference the SHA in the manifest. Rollback becomes a one-line change against an image that still exists — the single highest-leverage change in the report.

Enable Multi-AZ on the primary RDS instance
~1 hour + failover window · Partially resolves TD-02 · Runtime

A single console toggle plus a maintenance window. Roughly doubles database cost (~$180/mo) and removes the largest single point of failure. Schedule during your lowest-traffic window.

Perform and document one full restore drill
~4 hours · Resolves TD-02 · Runtime

Restore the latest snapshot to a throwaway instance, run integrity checks against production, record the elapsed time. You now have a verified RTO instead of an assumption — and an artifact to hand an enterprise buyer.

Set resource requests and limits on all workloads
~4 hours · Resolves TD-05 · Kubernetes

Derive baselines from two weeks of CloudWatch metrics: requests at observed p50, limits at ~2× p95. This alone activates the autoscaler you already have and prevents noisy-neighbor starvation.

Replace CI static keys with GitHub OIDC
~4 hours · Resolves TD-04 · Security

Register GitHub's OIDC provider in IAM, scope a role to your repo and branch, delete the key pair. No long-lived credential remains in CI — and it directly answers a security-questionnaire item.

Add four alerts on user-facing symptoms
~6 hours · Partially resolves TD-03 · Observability

5xx rate >1% over 5 min; p95 latency >2s; worker queue depth >500; RDS connections >80. Route to a shared Slack channel with a named weekly owner. Symptom-based alerting correlates with what users actually feel.

Schedule staging shutdown outside business hours
~2 hours · Resolves TD-09 · Cost

A scheduled scaling action to zero from 8pm–7am and weekends. Recovers ~$700/month with zero workflow impact — pays for the Multi-AZ upgrade three times over.

≈ 24 engineer-hoursCombined, these resolve two of three critical findings and materially reduce a third.
Section 05 · 30 / 60 / 90 Roadmap

Each phase makes the next one cheaper.

Observability comes early not because it's urgent alone, but because everything after it is easier to verify once it exists.

Days 1–30 Stop the bleeding
Recoverability & visibility
  • All seven quick wins
  • Structured JSON logging with request correlation IDs
  • Incident-response runbook + named on-call owner
  • Terraform state migrated to S3 with DynamoDB locking
Days 31–60 Build the foundation
Reproducibility & controlled release
  • Import console resources into Terraform; target 90%
  • Distributed tracing across the request path
  • Staging defined by the same modules as production
  • Read replica for analytics & reporting
  • Automated migration gate in the deploy pipeline
Days 61–90 Prepare for scale
Confidence under load & audit
  • Load test to 5× peak; document what breaks first
  • Progressive delivery: canary or blue-green for the API
  • Security baseline docs for enterprise questionnaires
  • Cost allocation tags + monthly spend review
  • Quarterly restore drill as a recurring commitment
Risk Register · What breaks first at 5× traffic
#ComponentFailure modeEst. thresholdMitigation
1Postgres connection poolConnection exhaustion; app-wide failure~2.2× peakPgBouncer + raise ceiling
2API podsNo scale-out; latency then timeouts~2.5× peakQuick win 4 (requests/limits)
3Worker queueUnbounded backlog; growing job latency~3× peakQueue-depth autoscaling
4EKS node groupMax nodes reached; pods stay pending~4× peakRaise ASG maximum
5Inference API quotaUpstream rate limiting; feature degradation~5× peakRequest quota increase now
The database fails first — at a little over twice your current peak. A successful launch week is enough to find that threshold.
Section 06 · What Happens Next

This teardown finds. It doesn't fix.

That boundary is deliberate — it keeps the review honest and the pricing fixed. Here's what most clients do with a report like this one.

Option One
Your team executes it

Section 4 is written to be actionable without me. Roughly 24 hours of engineering work covers the quick wins. One week of async follow-up questions is included at no cost.

Option Two
A quick-wins fix sprint

A fixed-price 1–2 week engagement implementing the Section 4 items directly in your repositories, PRs reviewed by your team. Scoped from this document, so there's nothing left to discover.

Engagement terms

The full teardown fee is credited in full against a fix sprint booked within 30 days. Capacity is limited to two engagements per month. If this report isn't actionable, it's refunded in full — no conversation required.

Book a fix sprint →Schedule the readout call

About this sample. A representative sample of a delivered teardown report, based on an anonymized composite of real engagements. Client name, metrics, and identifying details have been changed. A live engagement produces a report of comparable depth specific to your environment, delivered as a shareable page and PDF export, alongside a recorded walkthrough and a 30-minute readout call.