Capstone: Designing and Validating a Cloud Application

CampusCart's university approves a migration from two manually configured virtual machines to a supported cloud architecture. The existing application serves 5,000 daily users, usually completes 80 requests per second, and reaches 400 requests per second during registration. PostgreSQL stores orders. Product images live in object storage, but lifecycle and recovery rules have not been tested. One administrator account performs deployments. The team has never rehearsed a zone failure.

Jordan, a graduating computer science student joining the platform team, receives a bounded assignment: produce and validate a migration design before any production data moves. The design must preserve order correctness, support registration demand, recover from one availability-zone loss, restrict deployment privilege, preserve audit evidence, and state monthly cost assumptions. A separate CivicPermit review must identify additional government-work questions without claiming compliance from architecture alone.

Jordan cannot complete the assignment by drawing provider icons. The review board requires inspectable artifacts: an inventory, request-path diagram, data classification, threat analysis, capacity experiment, infrastructure definition, recovery transcript, objective specification, cost table, and decision log. Each artifact must name its producer, observation conditions, and unresolved uncertainty.

Prerequisites and earlier lessons

• All prior chapters are prerequisites because the capstone integrates history, networking, virtualization, providers, government boundaries, containers, Kubernetes, edge behavior, and operations.
• Chapter 1 supplies the CampusCart and CivicPermit recurring examples.
• Chapter 12 supplies service objectives, telemetry, incident evidence, least privilege, and cost units.

Learning outcomes

After completing the capstone, the reader can turn requirements into testable architecture properties, select managed or self-operated capabilities with explicit ownership, stage a migration with reversible checkpoints, conduct capacity and recovery tests, and present bounded conclusions. The reader can also identify which public-sector decisions require an authorizing official, security officer, contracting specialist, privacy counsel, or records officer.

Observable evidence and required artifacts

The first artifact is a current-state inventory produced from source repositories, host inspection, network configuration, database catalogs, identity directories, and interviews. Each row names an application executable, data store, external dependency, credential path, network listener, owner, backup, and recovery evidence. An interview statement is an assertion from a person; a configuration export is a direct observation from one software source. Jordan labels the difference.

The second artifact is a request-path diagram. The diagram starts at a browser and names recursive Domain Name System resolution, authoritative answers, edge proxy, origin load balancer, network subnets, application targets, database endpoints, object storage, identity calls, and logging destinations. Every arrow names a protocol, direction, and authorization mechanism. The diagram prevents the word “cloud” from hiding a network or institutional boundary.

The third artifact is a data classification table. CampusCart order rows contain names, contact details, payment-provider references, products, and timestamps. Product images are public. Application logs can accidentally contain personal information. The university data owner and privacy staff produce classification decisions; the engineer records technical locations and flows. CivicPermit adds government records, potentially sensitive personal information, retention schedules, public-records obligations, and jurisdiction-specific controls.

The fourth artifact is an objective and experiment package. A load generator produces timestamped request outcomes under a committed workload definition. Edge and application instrumentation produce success, latency, and dependency evidence. A recovery exercise produces a timeline from injected failure through detected impact and restored objective. A cost export or provider calculator produces priced quantities under a named date and region.

Mechanism: turning requirements into architecture

Start with one user action: place an order. Order completion requires an accepted authenticated request, a valid price decision, one durable database commit, and a response or retry contract that does not create a duplicate order. A request identifier and database uniqueness rule can make a repeated client attempt safe under stated conditions. Object-image availability does not determine order correctness, so image delivery can use a different caching and recovery strategy.

Place stateless application containers across at least two availability zones behind a health-aware load balancer. Run PostgreSQL through a managed high-availability configuration only after reading the exact failover, backup, maintenance, and replication semantics. Store product images in object storage with versioning or another recovery control. Put public static delivery behind a content delivery network. Use workload identity instead of long-lived static cloud keys where supported.

The architecture still needs explicit ownership. The provider operates physical facilities and managed-product internals under contract. The CampusCart team owns application code, identity policy, network exposure, data classification, schema changes, backup requirements, recovery testing, and user objectives. A managed label changes operation distribution; the label does not transfer accountability for configuration or application correctness.

Infrastructure as code gives Jordan a reviewable desired configuration. A change pipeline can validate syntax, policy, and a proposed difference before applying it. A versioned definition supports reconstruction, but the definition does not automatically capture mutable data, external identity state, certificates, or out-of-band changes. Drift detection compares actual provider configuration with declared configuration and reports a difference; an authorized reviewer decides the response.

Migration uses parallel verifiable stages. Build an application image and test it without moving production data. Create networks, identities, storage, and observability. Copy a production-like sanitized dataset. Validate schema and behavior. Measure capacity. Rehearse backup restore. Rehearse one-zone loss. Establish a replication or bounded outage plan for final data movement. Lower relevant Domain Name System time to live before cutover when addresses change. Keep a tested rollback path until correctness and recovery evidence pass acceptance rules.

Working model: capacity, availability, and recovery

Chapter 10 measured 48.75 usable requests per Pod per second after reserve. A 400-request-per-second peak needs nine Pods by capacity. A one-zone-loss requirement changes the placement. With three zones and nine evenly placed Pods, losing one zone leaves six Pods with about 6 × 48.75 = 292.5 acceptable requests per second, below peak.

To preserve 400 requests per second after losing one of three equal zones, each surviving pair needs at least nine Pods. An even placement therefore needs at least 15 Pods, five per zone, because losing five leaves ten. The surviving approximate capacity is 10 × 48.75 = 487.5 requests per second. The model assumes the database, load balancer, network, and remaining nodes accept the work. The model omits autoscaling delay and correlated demand changes. A lower business requirement could permit degraded peak capacity during zone loss; the decision belongs in the objective.

Recovery uses two quantities. A recovery point objective, abbreviated RPO, bounds acceptable data loss measured backward from disruption. A recovery time objective, abbreviated RTO, bounds acceptable time to restore a defined capability. A five-minute RPO and 30-minute RTO do not prove that backups meet them. Restore tests must measure the newest recovered committed order and elapsed time from declaration to usable validation.

Suppose a backup is taken every hour and transaction logs are archived every five minutes. The five-minute archive interval suggests a possible five-minute recovery point under successful complete delivery. The estimate assumes logs are durable, ordered, compatible, and replayable. An empirical restore establishes the achieved point for the tested failure and data set.

Empirical test plan

Use a production-like but non-sensitive dataset. Fix the image digest, database size, request mix, client timeouts, region, node type, and application configuration. Warm the cache according to a documented rule. Increase offered requests in steps through 500 per second. Record offered attempts, completed successes, failure categories, latency distributions, application replicas, processor throttling, memory, database connections, storage operations, and network transfer.

At a stable 400-request-per-second condition, remove one zone's application targets using a safe test mechanism. Preserve the time of injection, load-balancer health changes, connection failures, retries, objective impact, and recovery. The model predicts that 15 properly distributed Pods retain at least ten replicas and sufficient measured application capacity. A result below the objective rejects the architecture under the test even if replica arithmetic passes. Database failover, connection concentration, client retry storms, or uneven traffic can explain the difference.

Restore a database backup into an isolated environment. Validate row counts, constraints, selected checksums, most recent committed order time, application reads, and authorization. Keep the original backup immutable. The exercise rejects the recovery explanation if required logs are missing, replay fails, integrity checks differ, or elapsed time exceeds the objective.

Worked investigation

Initial question. Can CampusCart cut over without losing or duplicating orders?

Operating conditions. The old database accepts writes. A new managed PostgreSQL database receives logical replication. The application has a client-generated order identifier with a unique database constraint. Domain Name System uses a 300-second TTL. The team can place the old application in read-only maintenance mode.

Evidence collected. Replication reports zero unapplied transactions under a controlled write pause. Source and target row counts match for required tables. Checksums over stable key ranges match. A test suite retries 1,000 order requests after simulated response loss and produces 1,000 unique orders. The target database backup restores successfully in 18 minutes. Authorization tests show only the migration identity can perform replication actions.

Interpretation. Matching counts and checksums support copied-data integrity for tested ranges. The unique order identifier directly prevents a second row with the same identifier; the rule does not prevent a caller from creating a new identifier for the same intent. Zero reported replication lag supports cutover readiness only while writes remain paused and monitoring remains accurate.

Proposed intervention and prediction. Announce the bounded maintenance window, stop new writes, wait for confirmed replication completion, switch the application database endpoint, run read and controlled write validations, and direct traffic to the new application. Preserve the old database read-only. The prediction is that repeated requests with the same order identifier return the existing order and that row-count differences remain explained by controlled validation writes.

Observed result. A rehearsal with 50,000 synthetic orders completes in 11 minutes. Twenty deliberately repeated requests create no duplicate rows. Source and target differences equal the documented validation rows. Reverting the application endpoint within the rehearsal window restores the old read-only application.

Bounded conclusion and remaining uncertainty. The procedure preserved tested orders and identifiers during the rehearsal. The procedure does not prove production network behavior, human timing, third-party payment idempotency, or a cutover under unexpected writes. The final plan assigns stop authority, rollback criteria, and reconciliation owners.

Normal case

The review board accepts an architecture only after every requirement maps to a mechanism, an owner, an artifact, and a test. The team deploys by immutable digest, separates environments and identities, reviews infrastructure changes, collects user-facing evidence, rehearses recovery, and records priced units. A post-cutover review compares outcomes with acceptance thresholds and retains the former path until rollback criteria expire.

Failure and edge cases

PostgreSQL logical replication does not replicate schema definition changes, sequence state, or large objects. Roles, extensions, and job-scheduler configuration require separate migration procedures. Cache warming can make a test look faster than a cold deployment. Synthetic traffic can miss request-size and concurrency patterns. A zone label does not prove independent power, network, and dependency paths beyond the provider's published model. A rollback after new production writes can require forward reconciliation rather than a simple endpoint reversal.

Government-work extension: CivicPermit

CivicPermit is a government permit-intake application that stores personally identifiable information, must preserve audit evidence, and is designed to operate across two availability zones. The authorizing organization must define impact, data categories, jurisdiction, retention, incident reporting, personnel access, supply-chain requirements, and approved products. Amazon Web Services GovCloud (US) changes account eligibility, uses the separate aws-us-gov partition, requires partition-specific credentials and endpoints, and can have a different service and feature inventory. A GovCloud region does not make the application compliant. Jordan must map applicable controls to inherited provider evidence, organization procedures, and workload-specific implementation and test evidence.

The CivicPermit package includes a data-flow diagram, control-responsibility matrix, identity review, encryption and key-use design, audit-event inventory, records-retention procedure, recovery exercise, incident route, and authorization decision. Contracting staff verify service terms. Security and privacy officials verify control scope. Records staff verify retention and disclosure duties. Engineers provide technical evidence and do not sign beyond their authority.

Important qualifications

A reference architecture is a hypothesis about behavior under stated conditions. Provider documentation, service agreements, and certifications define inputs and inherited evidence, while workload tests establish configured behavior. No finite test proves absence of every failure. State scope, confidence, omissions, and monitoring that can detect changed assumptions.

Cost comparisons require equal work and equal reliability conditions. A managed database with higher direct price can reduce staffing or recovery risk; those effects need explicit organizational estimates. Multi-cloud designs can add portability but also duplicate identity, network, data, and operational complexity. Select a second provider only for a stated requirement that the added mechanisms satisfy.

Common errors

“The architecture diagram is the design.” A diagram omits acceptance conditions, ownership, data semantics, experiments, and recovery evidence.

“High availability means no maintenance window.” Availability is a measured outcome under stated failure and change conditions. Data migration can require a bounded write pause even when the target runs across zones.

“Infrastructure as code is the backup.” Configuration definitions do not contain application data or prove restoration.

“Passing the rehearsal proves production cutover.” A rehearsal increases confidence under matched conditions. Production demand, data, networks, people, and third parties retain uncertainty.

Knowledge check

Question: Nine Pods are evenly distributed across three zones, and each Pod has 48.75 usable requests-per-second capacity. Can the application sustain 400 requests per second after one zone fails?

Answer: No. One zone removes three Pods, leaving six. The approximate surviving capacity is 292.5 requests per second, below 400. The conclusion also assumes all other dependencies remain capable.

Exercises

Check. Submit a current-state inventory for a small application. Include executable, listener, data store, identity, credential source, backup, owner, and direct evidence for every row.

Practice. Create a provider-neutral request-path diagram and responsibility matrix for CampusCart. Name protocols, directions, failure domains, policy enforcement points, artifact producers, and one test per major claim.

Challenge. Execute the capstone in a safe cloud account or local equivalent. Submit infrastructure definitions, immutable image digest, load-test table, zone-failure timeline, database-restore transcript, security test results, objective calculation, monthly priced-quantity table, rollback plan, and a two-page bounded conclusion. Do not use production personal data.

Authoritative sources and further reading

Required reading

• National Institute of Standards and Technology Special Publication (NIST SP) 800-34 Rev. 1 provides contingency-planning guidance and recovery-plan structure.
• NIST SP 800-53 Rev. 5 supports CivicPermit control mapping and responsibility analysis.

Implementation documentation

• Kubernetes running in multiple zones documents topology considerations and limitations for cluster workloads.
• PostgreSQL logical-replication restrictions states the engine's current exclusions for schema, sequence, and large-object replication; record the production engine version before migration.

Primary provider engineering frameworks

• The official provider architecture frameworks—AWS Well-Architected, Google Cloud Architecture Framework, and Microsoft Azure Well-Architected Framework—supply provider-specific review questions; none substitutes for workload evidence.

Advanced and version-sensitive material

• Provider database backup, replication, and failover documentation is version-sensitive. Record engine version, region, configuration, and access date in the capstone.

Transition to continued study

The capstone completes the introductory path but not professional development. Continue by operating one small application long enough to observe deploys, certificate rotation, capacity change, cost drift, dependency failure, backup restore, and an incident review. Read the exact specifications behind each tool. Repeat experiments after upgrades. Practical cloud competence grows from connecting observed user results to named programs, machines, networks, policies, and institutions under changing conditions.