# Introduction to Cloud Computing # Cloud Computing: A Concrete System Map ## Prerequisites You should know that a browser sends requests to a web server and that a relational database stores rows. Basic familiarity with source code, files, and a command-line shell is enough. No cloud account is required. Readers arriving through a direct link can complete the observations with browser developer tools and `curl`. ## Learning outcomes After this lesson, you will be able to: - follow one web request from a browser to application code, a PostgreSQL database, and object storage; - identify which cloud objects exist in a region, in an availability zone, or at a provider-wide scope; - separate the request path from the management path; - name the evidence that confirms a claimed deployment; and - explain why two availability zones, a compliance label, or a provider product name does not by itself guarantee a desired outcome. ## Opening concrete narrative: one purchase, many machines At 10:03 on registration morning, Maya opens CampusCart, a small university bookstore application, from a laptop in a residence hall. CampusCart serves about 5,000 users per day. The application normally completes 80 web requests each second and can receive 400 requests each second during registration peaks. Maya selects a laboratory manual and presses **Buy**. A spinner remains visible for two seconds, so the campus developer on duty needs to locate the delay. CampusCart has a browser interface, an application programming interface (API), a PostgreSQL database, and an object store that holds book-cover images. The browser does not contact PostgreSQL directly. The Domain Name System (DNS) resolves a public name, a traffic distributor selects an API instance, and the API reads inventory rows from PostgreSQL. A separate object request retrieves the cover image. Each program runs somewhere physical even when the provider console displays only icons. Across town, CivicPermit accepts government permit applications that contain personally identifiable information (PII). CivicPermit must preserve audit evidence and remain available after the loss of one of its two configured availability zones. An operator cannot establish either property by pointing to a diagram. The operator needs deployment configuration, access logs, database replication status, restore results, and a controlled failover exercise. Both investigations begin with the same question: which concrete object handled the action, and which artifact can prove the answer? ## Observable evidence A browser's Network panel produces one row for each observed browser request. The row usually contains the target name, method, status, transferred bytes, and timing phases. The browser clock supplies the client-side timestamps. A 200 status establishes that a Hypertext Transfer Protocol (HTTP) server reported success; the status does not establish that the intended database row was committed. The API program produces an application log when its logging code executes. A useful CampusCart entry contains a generated request identifier, route, response status, elapsed milliseconds, and selected API instance. The load balancer can produce another access-log entry with a listener address, chosen target, and target response time. PostgreSQL can produce a statement log or an audit event, subject to its configuration. The object store produces object-access evidence only when access logging or an audit facility is enabled. Missing logs can therefore mean either “no action occurred” or “the responsible producer was not configured to record the action.” The cloud management interface produces configuration objects and audit events. A deployment listing can establish that two virtual machines were requested in two zones. A provider audit event can identify which authenticated principal changed a firewall rule. Neither artifact proves that a browser request reached both zones or that application data replicated correctly. The following map names the main CampusCart objects and the artifacts that expose them: ```text Student laptop browser Network row | | DNS lookup and encrypted HTTP request v Public name and traffic distributor DNS answer, certificate, load-balancer access log | +-----------------------+ | | v v API instance in zone A API instance in zone B application log application log | | +-----------+-----------+ | PostgreSQL connection v managed PostgreSQL deployment query evidence, transaction rows, replication and backup status Browser ------------------> object store object metadata and access event Engineer -----------------> provider management API configuration and audit event ``` The arrows above represent distinct communications, not a guarantee of one physical route. A provider can change an internal route without changing the public API. ## Mechanism: the request path and the management path Maya's purchase begins when browser code resolves the CampusCart name through the Domain Name System (DNS). The browser then establishes protected communication with the returned endpoint and sends a Hypertext Transfer Protocol (HTTP) request. A load balancer accepts the connection and forwards the request to a healthy API instance. The API validates Maya's input, asks PostgreSQL to change inventory and order rows in a transaction, and returns a response. The cover image can follow a separate path through an object-storage endpoint or a cache. Cloud practitioners call that movement the **data plane**: the provider mechanisms that carry or operate on workload data. The term includes more than network packets. A database read and an object retrieval are data-plane operations because both act on CampusCart's workload data. Earlier, an engineer used a console, command-line client, or deployment tool to ask the provider to create networks, compute instances, database capacity, and access rules. Provider control software authenticated the engineer, checked policy and quota, selected capacity, and stored the requested configuration. Practitioners call those management operations the **control plane**. A control-plane success response means that the provider accepted or completed a requested configuration change according to the named API. The response does not prove that CampusCart now works end to end. The distinction explains an important failure. Existing CampusCart requests might continue while the management API is unavailable because already-created load balancers and API instances still forward traffic. Conversely, a successful deployment can produce an unusable application when a route or credential is wrong. ## Technical terms in context A **cloud resource** is a provider-managed object with an identity and lifecycle, such as a virtual machine, network, database deployment, or object bucket. The surrounding noun matters: a PostgreSQL connection is not the same object as a managed PostgreSQL deployment. An **API** is the defined request and response interface through which one program asks another program to act. A graphical console normally calls provider APIs on the operator's behalf. The National Institute of Standards and Technology (NIST) definition supplies a durable test for whether the surrounding operating arrangement qualifies as cloud computing. A **region** is a provider-defined geographic area used to place cloud objects. An **availability zone** is a provider-defined failure-isolation location within a region. Provider definitions differ in physical detail. Current Google Cloud documentation describes regions as independent geographic areas containing zones; current Microsoft Azure documentation describes an availability zone as a separated group of datacenters with independent power, cooling, and networking; Amazon Web Services (AWS) documents regions and multiple isolated locations called Availability Zones. The durable design idea is failure separation. Exact zone counts, supported products, and physical arrangements remain version-sensitive. Provider product names are coordinates, not concepts. Structured Query Language (SQL) names the query language in the Cloud SQL product brand; the managed deployment includes much more than a language parser. The table maps common concepts to names visible in official documentation as checked on July 20, 2026: | Needed object | AWS | Google Cloud | Microsoft Azure | |---|---|---|---| | virtual machine | Amazon Elastic Compute Cloud (Amazon EC2) instance | Compute Engine virtual machine (VM) instance | Azure Virtual Machine | | isolated virtual network | Amazon Virtual Private Cloud (Amazon VPC) | Virtual Private Cloud network | Azure Virtual Network | | object storage | Amazon Simple Storage Service (Amazon S3) bucket | Cloud Storage bucket | Blob Storage container in a storage account | | managed PostgreSQL | Amazon Relational Database Service (Amazon RDS) for PostgreSQL | Cloud SQL for PostgreSQL | Azure Database for PostgreSQL | Product features and availability change more quickly than the request path described above. Google Cloud is the current official product-family name; Google Cloud Platform (GCP) remains common shorthand but should not be treated as a separate provider. ## Working model: requests in flight CampusCart's **request completion rate** is the number of completed API responses divided by the observation interval, measured in requests per second. Let `lambda` denote that completion rate. Let `W` denote the mean time in seconds from API arrival to API response. Under a stable observation interval, where arrivals and completions are balanced and no large queue is growing, the approximate mean number `L` of requests inside the API is: `L = lambda × W` The units explain the relationship: `requests/second × seconds = requests` At the ordinary 80 completed requests per second and a mean API time of 0.200 seconds, CampusCart has approximately `80 × 0.200 = 16` requests in progress. At 400 completed requests per second with the same mean time, the estimate becomes 80 requests in progress. The working model assumes a stable interval, uses means, and treats the API boundary as the start and end of the measurement. The estimate does not predict a percentile, database connections, browser concurrency, or required server count. A queue that grows during overload violates the stability assumption. Slow database calls can also make `W` rise with load, so multiplying peak arrivals by an old timing value can understate peak concurrency. The calculation gives an immediate evidence check. If an operator claims 400 completed requests per second, 200 milliseconds mean time, and only five API requests in progress, at least one measurement boundary or clock differs. ## Empirical test: expose the public part of the path Use a public test name rather than a production government endpoint. Run the following commands twice and preserve the output. The target Uniform Resource Locator (URL) is intentionally non-production: ```sh dig +noall +answer example.com curl -sS -o /dev/null \ -w 'code=%{http_code} peer=%{remote_ip} dns=%{time_namelookup}s connect=%{time_connect}s tls=%{time_appconnect}s first_byte=%{time_starttransfer}s total=%{time_total}s\n' \ https://example.com/ ``` The Internet Assigned Numbers Authority maintains `example.com` for documentation. `dig` displays DNS resource records produced by the resolver's answer. `curl` produces client-side cumulative timings from its own monotonic timing source where supported. Keep the network, target URL, and command constant; vary only whether the lookup is likely cached by comparing the first and second run. The expected second lookup can be faster, but a shared resolver cache can make the first run fast as well. A different result does not reject DNS caching by itself. Repeatedly slower connection or first-byte phases could instead reflect routing, server work, radio conditions, or measurement noise. The test rejects the simple claim that “all delay is in application code” when a large part of `total` occurs before `first_byte` processing could plausibly finish. Chapter 3 will separate the cumulative fields correctly. ## Worked investigation: did two zones serve CivicPermit? ### Question Did CivicPermit continue to accept permit submissions when one availability zone became unavailable? ### Conditions The staging deployment has one API instance in zone A and one in zone B, one zonal traffic distributor, and a PostgreSQL primary in zone A. A test client submits one uniquely numbered synthetic permit each second for five minutes. Synthetic names contain no PII. Operators block the zone-A API health endpoint after minute two; they do not alter the database. ### Evidence collected The test client produces 300 timestamped result rows. Load-balancer access logs identify the selected API target. API logs identify the zone and synthetic permit number. PostgreSQL rows establish committed submissions. The deployment listing confirms two requested API placements. During the intervention, 296 client rows report 201, while four report 503. After health detection, every successful API log names zone B. PostgreSQL contains 296 unique rows. ### Interpretation The target and API logs establish that zone B accepted work after zone A's API became unhealthy. The four 503 responses show a short interruption during health detection. The PostgreSQL evidence establishes committed synthetic submissions, not merely HTTP responses. The test did not remove the zone-A database, so the exercise did not represent complete zone-A loss. ### Intervention or prediction Before the run, the team predicted that API traffic would move to zone B after a health-check interval and that all database writes would continue because the database remained reachable. ### Result Traffic moved as predicted, but four requests failed before the traffic distributor stopped selecting the unhealthy target. The observed success fraction was `296 / 300 = 98.67%` during this deliberately disruptive five-minute interval. ### Bounded conclusion Collected evidence supports only the claim that the staging traffic distributor redirected API requests after the simulated API health failure. Collected evidence does not support the broader claim that CivicPermit survives loss of zone A. ### Remaining uncertainty The exercise left database failover, object storage, identity, DNS, audit-log delivery, and real zone isolation untested. Provider maintenance could also fail differently from the health-endpoint block. ## Normal case, failure case, and qualifications In the normal CampusCart case, DNS returns a reachable endpoint, protected HTTP negotiation succeeds, the traffic distributor selects a healthy API instance, PostgreSQL commits the order, and each evidence producer emits a correlated artifact. A request identifier connects the browser response to load-balancer and API entries without placing customer secrets in logs. A common edge case appears when two API instances exist but both depend on one zonal database. A zone failure can then leave healthy API programs unable to complete purchases. Two-zone compute placement is necessary for that design goal but not sufficient. Capacity also matters: one remaining zone must have enough healthy compute and database capacity to carry the intended failure load. Government deployment names require another qualification. AWS GovCloud (US), where “US” means United States, uses isolated regions with different region names, endpoints, and Amazon Resource Name partitions. Azure Government uses separate endpoints and can differ from global Azure in product and feature availability. Google Cloud Assured Workloads applies predefined control packages and monitors specified policy violations. None of those offerings transfers CivicPermit's legal, data-classification, access-control, or application-design responsibility to the provider. Compliance scope must be verified for the exact product, region, configuration, and assessment date. ## Common errors - **“The console icon is the running application.”** The icon represents a control-plane object. Request evidence must come from the data path and the workload. - **“Two zones mean high availability.”** Correlated dependencies, insufficient remaining capacity, and missing failover logic can still stop the application. - **“A 200 response proves the purchase.”** The HTTP status reports the server's response semantics. A database row or domain event is needed to establish the intended commit. - **“The cloud has no physical location.”** Provider abstractions hide many hardware details, but processors, disks, cables, buildings, and personnel still exist in governed locations. - **“Managed means no customer responsibility.”** The provider operates a larger part of the stack, while the customer still controls data, identities, allowed access, application behavior, and many configurations. ## Knowledge check **Question:** A deployment listing shows one CivicPermit API instance in each of two zones. Which claim does the listing directly establish, and which additional evidence would support continued operation after a zone failure? **Answer:** The listing directly establishes the requested or current placement reported by the provider control plane. A controlled failover run needs client results, traffic-distributor target logs, API logs, committed database evidence, and database failover status. The additional artifacts connect placement to actual request completion. ## Exercises ### Check Draw the CampusCart request path with separate arrows for the API call and cover-image retrieval. Label each endpoint and write one evidence artifact beside every arrow. Submit the annotated diagram and one sentence that distinguishes direct observation from inference. ### Practice Run the `dig` and `curl` commands five times at least ten seconds apart. Preserve the raw output in a text file. Calculate the median `total` time in seconds, name the clock and program that produced each timing, and explain why the DNS answer does not reveal every machine that handled the request. ### Challenge Write a two-zone CivicPermit test plan with one synthetic request per second for ten minutes. Include unique identifiers, controlled conditions, the exact failure action, expected artifacts, a success criterion, a stop condition, and a list of claims the test cannot establish. Submit the plan plus a table schema for the evidence. Do not use real PII. ## Authoritative sources and further reading ### Required reading - [NIST Special Publication 800-145, *The NIST Definition of Cloud Computing*](https://doi.org/10.6028/NIST.SP.800-145) defines the durable essential characteristics, deployment arrangements, and service models used throughout this book. - [NIST Special Publication 800-146, *Cloud Computing Synopsis and Recommendations*](https://doi.org/10.6028/NIST.SP.800-146) explains opportunities, limitations, performance, reliability, and security considerations for organizational decisions. ### Implementation documentation - [Google Cloud: Regions and zones](https://cloud.google.com/compute/docs/regions-zones) identifies the scope and failure placement of current Compute Engine objects. - [Microsoft Azure: Regions overview](https://learn.microsoft.com/en-us/azure/reliability/regions-overview) explains current region, geography, and zone relationships and warns that resiliency options vary. - [AWS GovCloud (US) User Guide: What is AWS GovCloud (US)?](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/whatis.html) identifies the isolated regions and the continuing shared-responsibility requirement. - [Azure Government: Compare Azure Government and global Azure](https://learn.microsoft.com/en-us/azure/azure-government/compare-azure-government-global-azure) documents endpoint, configuration, and feature differences that application code must account for. - [Google Cloud: Assured Workloads overview](https://cloud.google.com/assured-workloads/docs/overview) describes control packages, violation monitoring, and the scope of regulated-workload guardrails. All implementation pages above were checked on July 20, 2026. Provider names, region inventories, compliance authorizations, and feature availability are version-sensitive; verify the current page and the exact target environment before making a deployment decision. ## Next-chapter transition The system map shows what exists and which artifacts make a cloud claim testable. The next chapter explains how time-sharing, packet networks, virtual machines, distributed datacenters, and metered interfaces developed into the operating arrangement now called cloud computing. --- # Computing Before the Public Cloud ## Prerequisites Read [Cloud Computing: A Concrete System Map](./01-orientation.md), or be able to distinguish a browser request path from provider management operations. Basic knowledge of processors, memory, and files is sufficient. Historical machine details are introduced where needed. ## Learning outcomes After this lesson, you will be able to: - explain which operational problems batch processing, time-sharing, virtualization, and networked computing addressed; - distinguish remote hosting from cloud computing by observable operating characteristics; - connect public-cloud APIs and metering to earlier technical and institutional mechanisms; - calculate utilization and provisioned server-hours with explicit assumptions; and - use primary historical artifacts without treating product launch dates as the invention of a single idea. ## Opening concrete narrative: waiting for the computer In 1966, a university researcher carries a deck of punched cards to a computing center. An operator groups the deck with other submitted jobs. The expensive computer reads jobs, runs each program, and sends printed output back later. A missing comma can cost the researcher an entire submission cycle. The researcher does not directly control the processor while the job runs. Another researcher sits at a terminal connected to a time-sharing computer. The terminal sends short interactions while a supervisor program switches the processor among users. Each person receives quick enough responses to edit, compile, and run programs interactively. The central processor still executes only a limited amount of work per second; the supervisor changes who can use that capacity and when. CampusCart faces a related allocation problem decades later. Its application programming interface (API) normally completes 80 requests per second but can receive 400 requests per second during registration. Buying enough servers for the peak leaves processors underused during ordinary weeks. Buying only ordinary capacity creates a queue during the peak. Cloud computing did not remove the allocation problem. Cloud control software made standardized capacity available through network APIs, shortened acquisition time, and attached measurements to billing and policy. ## Observable historical evidence A historical claim needs an artifact produced near the event. A machine manual records commands and operating assumptions. A research paper records the authors' design and reported results. An announcement letter establishes what a vendor offered on a date, not how every customer used the offering. Source code can establish implemented logic, while an operator log can establish a particular run. R. J. Creasy's 1981 International Business Machines (IBM) paper, *The Origin of the VM/370 Time-Sharing System*, reports that the Compatible Time-Sharing System (CTSS) became operational in 1961 and influenced the IBM Control Program (CP)/Cambridge Monitor System (CMS) design. “VM” identifies the Virtual Machine product family in VM/370. The paper describes operational CP-40 and CMS work in 1966 and the later VM/370 line. IBM's preserved timeline records the official VM/370 announcement on August 2, 1972. Creasy's paper and IBM's timeline support a sequence of named projects. The two historical artifacts do not support the claim that IBM invented every form of virtualization or that present cloud instances are unchanged VM/370 machines. Public product announcements provide later landmarks. Amazon Web Services (AWS) announced Simple Storage Service (Amazon S3) on March 13, 2006 and an Elastic Compute Cloud (Amazon EC2) beta on August 24, 2006. Google's April 7, 2008 App Engine announcement called the offering a preview and listed quotas. Microsoft announced a Windows Azure technology preview on October 27, 2008. Each announcement exposes the original scope and language. Current product behavior must come from current documentation rather than an old announcement. ## Mechanism: four developments that accumulated ### Sharing expensive processors Batch operation kept a processor supplied with queued jobs and reduced manual setup between runs. Time-sharing added rapid scheduling, protected memory, per-user files, and interactive terminals. A scheduler allowed one user's program to execute for a short interval and then gave another program access. Protection mechanisms prevented one ordinary program from freely overwriting another user's memory. Practitioners use **time-sharing** for the allocation of computing time among multiple users or programs so that each can make progress. Time-sharing is not cloud computing. A department time-sharing machine can lack on-demand self-service, network access through standard mechanisms, rapid elasticity, provider pooling across customers, or measured consumption as defined by the National Institute of Standards and Technology (NIST). ### Presenting more than one machine IBM's CP/CMS work separated a control program from user environments. The control program scheduled processors, mapped memory, and presented virtual devices. Each guest environment could behave as though it controlled a machine even while physical capacity was shared. Practitioners call the mediating software a **hypervisor** and the presented computer a **virtual machine** (VM). Chapter 4 explains the mechanism in detail. Virtualization improved isolation, compatibility, and allocation, but virtualization alone also does not create a cloud. A laptop can run three VMs without any remote API, metering, elasticity, or provider organization. ### Connecting independently operated computers Packet networking divided communication into bounded messages that could traverse interconnected links. The early Advanced Research Projects Agency Network (ARPANET) and later Internet protocols allowed programs on separately administered computers to exchange data. Client/server applications and the World Wide Web made remote programs broadly reachable. Domain Name System (DNS) names let administrators change addresses without changing the human-facing name. Networking made remote access possible. Remote access alone did not standardize capacity acquisition. A hosting company could install one rented physical server after a support ticket and charge a fixed monthly fee. Such hosting was useful, but the customer might wait days for installation and might not receive a programmable mechanism for adding or releasing capacity. ### Operating warehouses of computers through software Large Internet applications required operators to treat many ordinary computers as a coordinated fleet. Automated installation, monitoring, replication, failure detection, load distribution, and distributed storage converted machine failures from rare exceptional events into expected operational inputs. Provider engineering joined that fleet automation with tenant isolation, identity, policy, public APIs, usage measurement, and commercial or institutional rules. NIST's 2011 definition identifies five essential cloud characteristics: on-demand self-service, broad network access, pooled provider capacity, rapid elasticity, and measured service. The phrase **cloud computing** therefore names an operating model, not one hardware invention. The “measured service” term in NIST is historical terminology; a clearer local description is that provider mechanisms meter a named quantity such as instance-seconds, stored byte-months, or completed requests over a defined interval. ## A causal timeline | Period or date | Primary artifact | Mechanism made visible | Important limit | |---|---|---|---| | 1961–1964 | CTSS accounts in Creasy's later origin paper | interactive time-sharing and per-user work | one institutional computer was not an elastic public cloud | | 1966 | reported CP-40 and CMS operation | a control program presented protected virtual machines | specialized hardware and local operations remained necessary | | August 2, 1972 | IBM VM/370 announcement preserved by IBM | a supported commercial virtual-machine product | product access and capacity acquisition were not modern web APIs | | 1980s–1990s | Internet standards and web specifications | interoperable communication across organizations | remote servers could still be manually provisioned | | March and August 2006 | Amazon S3 and EC2 announcements | storage and VM capacity requested through web interfaces with usage charges | the initial offerings had far fewer regions, instance types, and durability options than current products | | April and October 2008 | Google App Engine and Windows Azure announcements | provider-managed application platforms entered broader developer use | both announcements described previews, not current production scope | | September 2011 | NIST Special Publication 800-145 | a durable vocabulary for characteristics and service models | the vocabulary classifies deployments; it does not certify quality or security | Several streams overlap, so the rows must not be read as a single line of invention. Universities, governments, standards organizations, hardware firms, telecommunications operators, and Internet companies contributed different mechanisms under different names. ## Technical terms in context: arrangements that are not synonyms **Batch processing** names the queued execution arrangement that reduced operator setup between jobs. **Time-sharing** names rapid allocation of processor time among interactive users or programs. Both arrangements can improve use of an expensive computer, but neither term specifies network self-service or rapid capacity acquisition. **Remote hosting** names the operation of a customer's workload on another organization's computers. **Utility computing** emphasizes measured consumption and charging that resembles an institutional utility. **Grid computing** usually joins capacity from multiple administrative or technical domains for coordinated work. Historical authors used the labels in overlapping ways, so a practitioner should state the observed acquisition, isolation, network, elasticity, and measurement mechanisms rather than infer them from one label. **Cloud computing** names the broader operating arrangement defined by NIST's characteristics. Virtual machines, web APIs, metering, and fleet automation often implement that arrangement, but no one mechanism is sufficient by itself. ## Working model: utilization and provisioned time Let one CampusCart API server safely complete 100 requests per second under the stated workload. Assume requests divide evenly and that one server can carry the normal 80 requests per second. Four servers can carry the 400-request-per-second registration peak. Ignore redundancy for the moment so the allocation effect remains visible. Define server utilization for this capacity-planning example as completed request capacity used divided by available request capacity during the same interval. The ordinary utilization of four purchased servers is: `80 requests/second ÷ (4 × 100 requests/second) = 0.20 = 20%` Peak utilization becomes: `400 requests/second ÷ 400 requests/second = 1.00 = 100%` The peak calculation leaves no failure or latency headroom. A practical design would target less than 100%, use measured latency and queue depth, and include redundant capacity. Now assume CampusCart needs ordinary capacity for 8,752 hours each year and peak capacity for eight registration hours. Four owned servers provide `4 × 8,760 = 35,040` server-hours of annual capacity. A hypothetical perfectly elastic allocation uses one baseline server for 8,760 hours plus three extra servers for eight hours: `8,760 + (3 × 8) = 8,784 server-hours` The ratio `8,784 ÷ 35,040 ≈ 25.1%` describes demanded server-hours divided by provisioned server-hours under these assumptions. The calculation does not compare money. Cloud prices can include different processors, storage, network transfer, support, and managed operations. Scaling also takes time, capacity can be unavailable, and a real two-zone design needs more than the minimum request capacity. The useful prediction is narrower: when short peaks are much larger than the baseline and additional capacity can arrive before the queue becomes harmful, acquiring capacity for shorter intervals can reduce idle provisioned time. ## Empirical test: observe time-sharing on a current computer The following Linux or macOS shell experiment runs two processor-intensive commands at the same time. Use a non-production machine. Preserve the shell transcript and the timing output. ```sh /usr/bin/time -p sh -c 'yes > /dev/null & p1=$!; yes > /dev/null & p2=$!; sleep 5; kill "$p1" "$p2"; wait "$p1" "$p2" 2>/dev/null || true' ``` Open a second terminal during the five-second interval and observe the two `yes` programs with `top` or `ps`. The shell starts both programs, the operating-system scheduler assigns processor time, and `top` samples processor accounting. The controlled condition is a five-second wall-clock interval. The varied condition can be one worker versus two workers. On a single available logical processor, two workers should share the available processor time. On a multi-core machine, both can run simultaneously, so record the reported logical processor count. A result where both workers accumulate processor time supports scheduling and accounting; the result does not reproduce CTSS or a cloud. Thermal throttling, other programs, container quotas, and different `top` conventions are alternative explanations for unexpected percentages. A run where one worker receives no processor time for the entire interval would reject the simple fairness expectation and require investigation of priority, affinity, suspension, or measurement. ## Worked investigation: should CampusCart buy for the peak? ### Initial question Would on-demand API capacity materially reduce CampusCart's idle provisioned time compared with four permanently allocated servers? ### Operating conditions One server safely completes 100 requests per second. CampusCart needs 80 requests per second for 8,752 hours and 400 requests per second for eight announced hours. Additional instances become healthy in three minutes. Registration demand rises from 80 to 400 requests per second over ten minutes. Storage and database capacity are held constant. ### Evidence collected Last year's timestamped load-balancer log summary shows the eight-hour peak and the ten-minute ramp. A repeatable load test establishes the 100-request-per-second safe limit for the tested server image. Deployment events from staging show three minutes from request to health-check success. ### Interpretation The ten-minute demand ramp is longer than the observed three-minute acquisition interval. An operator can therefore request extra instances before demand reaches the one-server limit if an alert or schedule acts early enough. The annual server-hour calculation predicts much less idle compute allocation. The database remains an untested bottleneck. ### Proposed intervention or prediction Keep one baseline instance and schedule three additional instances ten minutes before registration opens. Predict 8,784 allocated server-hours per year under the simplified schedule and no compute queue during the tested peak. ### Observed or calculated result The staging replay reaches 400 completed requests per second with four healthy instances and a 58% processor busy fraction on each instance. The replay's 95th-percentile API time remains 240 milliseconds. The annual allocation calculation remains 8,784 server-hours if staging behavior transfers to production. ### Bounded conclusion The historical demand profile and staging replay support scheduled temporary compute allocation for this eight-hour peak. The evidence does not establish a lower bill or successful production behavior. ### Remaining uncertainty Production request mixtures, provider capacity, zone loss, database locks, network limits, object retrieval, scaling failures, and price changes remain outside the test. A flash crowd faster than three minutes can still overload the baseline instance. ## Normal case, failure case, and qualifications The normal on-demand case has a known demand ramp, a tested machine image, sufficient provider quota, available zone capacity, a health check, and an automated release action. Provider metering produces usage records after allocation. CampusCart operators compare those provider records with deployment events and workload measurements. A limiting case occurs when demand jumps to 400 requests per second in ten seconds while additional instances need three minutes. Requests queue or fail before new capacity becomes healthy. Pre-scaling, spare capacity, admission control, caching, or a different architecture must cover that interval. “Elastic” describes the ability to change allocation; the word does not mean instantaneous or unlimited. CivicPermit's history can lead to a different allocation decision. The government application carries personally identifiable information, has audit obligations, and requires operation across two availability zones. Procurement time, authorization scope, data residency, evidence retention, and failure capacity can dominate the simple server-hour ratio. Cloud acquisition changes who operates each mechanism; the change does not remove the agency's duty to verify the complete arrangement. Historical labels also require care. “Utility computing,” “grid computing,” “hosting,” and “application service provider” described overlapping arrangements. NIST's characteristics offer a useful test, but real products can sit near category edges. Private clouds can meet the characteristics without selling capacity to the public. Public provider use can also include dedicated hardware. ## Common errors - **“Cloud computing began on one launch date.”** Public launches are important commercial landmarks, while the required mechanisms developed across decades and institutions. - **“Time-sharing and cloud computing are synonyms.”** Time-sharing allocates processor time. Cloud computing adds a broader operational arrangement including network access, self-service, pooling, elasticity, and measurement. - **“Virtualization created elasticity.”** Virtualization makes capacity easier to partition, but control software, images, networking, health detection, quota, and institutional processes create usable elasticity. - **“Twenty-percent utilization means the owned servers are waste.”** Redundancy, latency targets, growth, failure headroom, and fixed-cost constraints can justify idle capacity. The percentage is evidence for a decision, not the decision itself. - **“Pay per use always costs less.”** Unit prices, transfer charges, support, engineering labor, commitments, and demand shape determine total cost. ## Knowledge check **Question:** A hosting company installs a dedicated server after a customer submits a ticket, charges one fixed monthly fee, and provides network access. Which NIST cloud characteristics are directly visible, and which are missing or unproved? **Answer:** Broad network access is visible. The description does not establish on-demand self-service, rapid elasticity, pooled provider capacity, or usage measurement. Dedicated hardware does not disqualify a broader cloud arrangement, but the stated ticket workflow and fixed allocation do not establish the remaining characteristics. ## Exercises ### Check Create a five-row evidence table for CTSS, CP-40, VM/370, Amazon EC2 beta, and Google App Engine preview. For each row, preserve a source Uniform Resource Locator (URL), artifact producer, date, directly supported claim, and one unsupported claim. Submit the table. ### Practice Repeat the utilization calculation for a safe server capacity of 125 requests per second, a normal demand of 80 requests per second, and a peak of 400 requests per second lasting 24 hours each year. Choose the minimum integer server count for the peak, state a headroom qualification, and submit all arithmetic with units. ### Challenge Run the scheduler experiment with one, two, and four workers for five seconds each. Preserve raw timing and `top` or `ps` samples. Report logical processor count, processor time per worker, wall time, and at least two confounding factors. Write a bounded conclusion that does not call the local computer a cloud. ## Authoritative sources and further reading ### Required reading - [R. J. Creasy, *The Origin of the VM/370 Time-Sharing System*](https://www.ibm.com/support/pages/zvm/history/50th/vm370ori.pdf) is a primary engineering account connecting CTSS influence, CP/CMS design, and operational experience. - [IBM z/VM history timeline](https://www.ibm.com/support/pages/zvm/history/timeline.html) preserves dates and original IBM VM/370 artifacts, including the August 2, 1972 announcement. - [NIST Special Publication 800-145](https://doi.org/10.6028/NIST.SP.800-145) supplies the explicit characteristics used to distinguish cloud computing from adjacent arrangements. ### Primary product records - [AWS: Announcing Amazon S3](https://aws.amazon.com/about-aws/whats-new/2006/03/13/announcing-amazon-s3---simple-storage-service/) records the March 13, 2006 storage interface announcement. - [AWS: Announcing Amazon EC2 beta](https://aws.amazon.com/about-aws/whats-new/2006/08/24/announcing-amazon-elastic-compute-cloud-amazon-ec2---beta/) records the original beta's on-demand capacity and charging claims. - [Google: Introducing Google App Engine](https://cloudplatform.googleblog.com/2008/04/introducing-google-app-engine-our-new.html) records preview status, initial quotas, and the managed application-platform scope on April 7, 2008. - [Microsoft: Windows Azure announcement](https://news.microsoft.com/source/2008/10/27/microsoft-unveils-windows-azure-at-professional-developers-conference/) records the October 27, 2008 technology preview and original platform framing. ### Advanced reference - [Gerald J. Popek and Robert P. Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures”](https://doi.org/10.1145/361011.361073) gives the classic formal treatment of virtual-machine monitors. Chapter 4 translates the relevant distinction into operational evidence. Historical sources establish behavior and claims at their publication dates. Old quotas, prices, product limits, and product names are not current implementation documentation. ## Next-chapter transition Time-sharing and virtualization explain how operators divide machines. Public cloud use also depends on communication between independently located programs. The next chapter follows a request through DNS, addresses, routes, transport security, traffic distribution, and private cloud networks. --- # Networking Foundations for Cloud Applications ## Prerequisites Read [Cloud Computing: A Concrete System Map](./01-orientation.md) and [Computing Before the Public Cloud](./02-before-cloud.md), or know that browser requests and provider management requests follow different paths. You should recognize an Internet Protocol (IP) address and a web Uniform Resource Locator (URL). Advanced routing knowledge is not required. ## Learning outcomes After this lesson, you will be able to: - follow a cloud request through name resolution, routing, transport, encryption, Hypertext Transfer Protocol (HTTP), traffic distribution, and application handling; - read a Classless Inter-Domain Routing (CIDR) prefix and calculate its address count; - distinguish a subnet route from a traffic-authorization rule; - decompose client timing evidence without adding cumulative fields incorrectly; - design a controlled network investigation with a rejectable explanation; and - account for Domain Name System caching, Hypertext Transfer Protocol version 3 (HTTP/3), overlapping addresses, request limits, and other important exceptions. ## Opening concrete narrative: a slow registration request At 8:00:01 on registration morning, CampusCart receives a burst of purchases. The bookstore application serves about 5,000 daily users, normally completes 80 application programming interface (API) requests per second, and can receive 400 requests per second at this peak. A student in a campus residence sees a 2.4-second delay before the first response byte. Another student on a wired laboratory computer receives the same page in 180 milliseconds. The on-call engineer cannot act on “the network is slow.” The student's browser first converts `campuscart.example` into connection information. Packets leave the laptop, cross the campus network and an Internet provider, reach a cloud traffic distributor, and then cross a virtual cloud network to an API program. The API can wait for a PostgreSQL query before producing a response. Several independently operated queues and programs contribute time. CivicPermit adds a stricter constraint. Permit submissions contain personally identifiable information (PII), and the application must retain audit evidence. A network packet capture can itself contain names, tokens, or document content. The operator must use synthetic data, restrict capture access, and collect only the bytes or metadata necessary for the question. The investigation begins by naming every communication endpoint and every artifact producer. ## Observable evidence along the path A Domain Name System (DNS) resolver produces an answer containing resource records, such as an A record for an Internet Protocol version 4 (IPv4) address or an AAAA record for an Internet Protocol version 6 (IPv6) address. `dig` displays the resolver response it receives. An answer establishes the returned name data and time-to-live field; the answer does not establish which destination a browser ultimately used or which physical machine accepted the request. `curl` produces client-side timing fields. `time_namelookup` ends after name lookup, `time_connect` ends after the selected transport connection, `time_appconnect` ends after a security handshake where applicable, `time_starttransfer` ends when the first response byte arrives, and `time_total` ends when transfer finishes. Each value is cumulative from the beginning of the operation. The fields must be subtracted to estimate separate phases. A packet-capture program such as `tcpdump` records packets visible at the capture interface. The operating-system capture clock timestamps each observed packet. A capture on the laptop cannot directly establish packets on the load balancer-to-API connection because the load balancer terminates one connection and creates another. Encryption also hides HTTP content from an ordinary network capture while leaving some sizes, timing, and addressing visible. A cloud flow-log facility can produce metadata for accepted or rejected traffic at selected virtual network interfaces. Exact fields and sampling behavior depend on the provider and configuration. A load-balancer access log can identify the selected target and target-response timing. An API log can identify application work. PostgreSQL evidence can identify a query or committed transaction. Correlation requires a safe request identifier because separate clocks can be offset. ## Mechanism: from a name to an application response ### Name resolution The browser asks a resolver for information about the requested name. The resolver can answer from a cache while the record's time to live remains valid, or it can query other DNS servers. DNS separates a stable application name from changeable connection addresses. A low time to live can shorten some cache retention, but resolvers and clients still have implementation behavior and in-flight connections. ### Addressing and forwarding An IP address identifies an interface for packet delivery within an addressing context. A router examines a destination address and selects a matching route. CIDR notation writes an address plus the number of leading prefix bits. The IPv4 prefix `10.20.4.0/24` fixes 24 of 32 address bits and leaves eight variable bits, so the block contains `2^8 = 256` addresses. Cloud providers reserve some addresses in many subnet products; 256 is the mathematical block size, not necessarily the assignable host count. A **subnet** is an address prefix attached to a defined network placement. In Amazon Web Services (AWS), for example, a subnet belongs to one Availability Zone and associates with a route table. A **route table** contains destination prefixes and next-hop targets. When several routes match, IP forwarding normally uses the longest matching prefix. A default IPv4 route, `0.0.0.0/0`, matches every IPv4 destination but loses to any more-specific route. A route says where matching packets should go. A **security group** or provider firewall rule says which traffic is authorized at a protected interface or workload. AWS security groups are stateful, while AWS network access control lists have different stateless rule behavior. Google Cloud virtual-network firewall rules and Azure network security groups have their own scopes and defaults. Product names do not erase the distinction between forwarding and authorization. ### Transport, encryption, and HTTP Transmission Control Protocol (TCP) uses sequence numbers, acknowledgments, retransmission, and flow control to present an ordered byte stream between two endpoints. A port number helps the receiving host select a listening program. User Datagram Protocol (UDP) presents datagrams without TCP's ordered byte-stream mechanism. Transport Layer Security (TLS) authenticates the server under the chosen certificate and parameters, establishes keys, and protects later application bytes for confidentiality and integrity. As of July 20, 2026, Request for Comments (RFC) 9846 is the current TLS 1.3 specification and obsoletes RFC 8446. RFC 9846 retains the TLS 1.3 version number and is backward compatible with RFC 8446, while tightening requirements and clarifying the earlier text. A successful TLS handshake authenticates the presented identity under the client's trust rules; the handshake does not prove that the application is authorized to release a permit. Hypertext Transfer Protocol (HTTP) defines request methods, target identifiers, fields, content, and response status semantics. HTTP/1.1 and HTTP/2 commonly operate over TCP with TLS for an `https` URL. HTTP/3 operates over QUIC, which uses UDP, so the `https` scheme does not guarantee TCP port 443. A traffic distributor can terminate the client connection and use a different HTTP version or protection arrangement on the API side. The uppercase label HTTPS names HTTP carried with TLS under the `https` URL scheme. HTTPS protects the connection under the negotiated TLS properties; the label does not specify one HTTP version or one transport protocol. ### Traffic distribution and private paths The public traffic distributor checks target health and selects an API destination according to configured behavior. A private API address need not be reachable from the public Internet. The traffic distributor can have public reachability on one side and private virtual-network reachability on the other. CampusCart's API opens a separate connection to PostgreSQL. A database subnet route, name, port rule, database authentication rule, and TLS configuration can all affect that connection. A browser cannot infer those server-side conditions from the public address alone. ## Technical terms in context: configured communication boundaries An **endpoint** is one named side of a communication, such as the browser address and port or the traffic distributor address and port. A **network flow** is a set of packets grouped by identifiers such as source address, destination address, transport protocol, and ports. Provider flow logs summarize selected flow metadata; they are delayed or sampled under some products and are not packet captures. A **virtual private cloud (VPC)** is a provider network object with configured addressing, routing, and policy. Provider scope differs: an AWS VPC is regional and contains zonal subnets, a Google Cloud VPC is global with regional subnets that span zones, and an Azure Virtual Network is regional with subnets that span zones. A generic architecture diagram must therefore state the provider before a “subnet per zone” claim. A **load balancer** is a traffic distributor that selects among eligible destinations. Layer 4 and Layer 7 products observe different protocol information, and one provider can offer several products under a load-balancing portfolio. A **private address** is not globally routed on the public Internet under the relevant address rules; private addressing does not encrypt or authorize traffic. ## Working model: a lower-bound latency budget Let each timing quantity use milliseconds. For a fresh HTTPS operation over TCP and TLS 1.3, a useful approximate decomposition is: `T_total = T_DNS + T_TCP + T_TLS + T_server_wait + T_transfer` The terms represent elapsed time, not packet counts. A sample `curl` run reports these cumulative values: | Cumulative field | Value | |---|---:| | name lookup | 0.012 s | | TCP connect | 0.032 s | | TLS complete | 0.055 s | | first byte | 0.133 s | | total | 0.140 s | Convert each field to milliseconds and subtract adjacent endpoints: - `T_DNS = 12 ms` - `T_TCP = 32 - 12 = 20 ms` - `T_TLS = 55 - 32 = 23 ms` - `T_server_wait = 133 - 55 = 78 ms` - `T_transfer = 140 - 133 = 7 ms` The sum is `12 + 20 + 23 + 78 + 7 = 140 ms`. The model assumes a fresh connection, one chosen address, no proxy timing hidden before `curl`, and a response small enough that transfer is simple. `T_server_wait` includes travel to the server, traffic-distributor work, API work, database work, and travel of the first response byte. The field is not “application processor time.” Connection reuse can make TCP and TLS setup disappear from a later request. HTTP/3 has different transport setup. Parallel address attempts, redirects, DNS-over-HTTPS, radio scheduling, packet loss, and congestion can change the phases. Bandwidth and latency also name different quantities. **Network throughput** is delivered bits divided by the observation interval, measured in bits per second. A 1-megabyte response contains approximately 8 megabits when “megabyte” means 1,000,000 bytes. At an achieved 20 megabits per second, its ideal serialization time is `8 megabits ÷ 20 megabits/second = 0.4 seconds`. Round-trip latency affects handshakes, acknowledgments, and request-response dependencies even when the link can carry many bits per second. ## Empirical test: decompose a real request Run the following command against the reserved documentation site. Preserve at least ten output rows. ```sh curl -V for n in 1 2 3 4 5 6 7 8 9 10; do curl -sS -o /dev/null \ -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} first=%{time_starttransfer} total=%{time_total} ip=%{remote_ip} code=%{http_code}\n' \ https://example.com/ done ``` Keep the target, client machine, and network access constant. Vary only the repetition number. Preserve `curl -V` because supported HTTP versions depend on the client build. Compute the five phase increments for each row. Record whether the remote address changes. The expected evidence may show a shorter cached DNS phase after the first operation. Each `curl` invocation can still create a new connection, so the experiment does not test browser connection reuse. The explanation “DNS dominates the delay” is rejected for a row where DNS contributes 2 milliseconds to a 500-millisecond total. The evidence cannot identify the remaining cause by itself. A slow first-byte increment can reflect API work, database waiting, proxy queueing, packet loss, or distant routing. Run a controlled server-side correlation before assigning responsibility. Optional `traceroute` or `tracert` output can reveal replies from some routing hops. A missing hop reply does not establish a broken path because routers can forward application traffic without answering diagnostic probes. A returned hop name does not establish physical location. ## Worked investigation: why do large CivicPermit uploads fail? ### Initial question Why does a 2-mebibyte synthetic Portable Document Format (PDF) upload succeed while a 12-mebibyte synthetic PDF upload fails? One mebibyte (MiB) is 1,048,576 bytes. ### Operating conditions The staging CivicPermit endpoint uses the production-like traffic distributor, two API instances in separate availability zones, and synthetic documents with no PII. The client uses one wired host and the same authentication role. Operators perform five 2 MiB uploads and five 12 MiB uploads. The configured edge request limit is believed to be 10 MiB. ### Evidence collected `curl` verbose output shows TLS completion for every operation. Each 2 MiB upload returns 201 after about 600 milliseconds. Each 12 MiB upload returns HTTP 413 after about 45 milliseconds. Traffic-distributor access logs contain all ten request identifiers and identify the distributor as the response producer for the 413 entries. API logs contain only the five small-upload identifiers. Packet counters show transmitted bytes, but no repeated retransmission pattern is present in the client capture. ### Interpretation HTTP 413 means that the responding HTTP intermediary or origin rejected the request content as too large under its policy. The absence of large-upload identifiers from API logs, combined with distributor-generated 413 entries, places the rejection before the API program. The fast, repeatable response contradicts a simple packet-loss explanation. ### Proposed intervention or prediction After a security and storage review, raise the staging distributor's explicit body limit from 10 MiB to 16 MiB. Predict that 12 MiB uploads will reach an API instance and return 201, while 17 MiB uploads will still return 413 at the distributor. ### Observed result Five 12 MiB uploads return 201, and correlated API logs plus object metadata show five stored synthetic objects of the expected byte length. Five 17 MiB uploads return 413 and remain absent from API logs. ### Bounded conclusion The configured distributor body limit caused the tested staging failures. The intervention changed the threshold in the predicted direction. ### Remaining uncertainty The test does not establish safe production capacity, malware scanning behavior, interrupted-upload recovery, user radio performance, audit completeness, or a maximum permitted document size under agency policy. Another upstream intermediary could apply a separate limit outside staging. ## Normal case, failure cases, and qualifications In the normal CampusCart path, DNS returns usable addresses, the client selects a supported transport, routing tables provide next hops, authorization rules allow the intended ports, TLS validates the CampusCart name, HTTP reaches a healthy target, and the API reaches PostgreSQL through a separate private path. Each communication has two named endpoints and a configured return path. An address-planning edge case appears when the campus network and cloud network both use `10.20.0.0/16`. A virtual private network cannot select a unique destination from the overlapping prefixes without translation or renumbering. Both prefixes are valid private address space; the conflict comes from using the same addresses on two connected administrative sides. A routing failure can coexist with an apparently correct firewall rule. Allowing TCP destination port 5432 toward PostgreSQL does not create a route to the database subnet. Conversely, a valid route does not authorize the traffic. DNS can also return the intended address while TLS fails because the certificate identity does not match the requested name. Network Address Translation (NAT) is another qualification. NAT rewrites address or port information at a defined inside/outside connection point. NAT can provide outbound reachability for privately addressed instances, but NAT is not a general inbound firewall and does not provide application authentication. Stateful translation tables can exhaust ports under large connection counts. DNS commonly uses UDP for an initial ordinary query, but current DNS operation also uses TCP and encrypted transport arrangements. A diagnostic rule that blocks or tests only UDP port 53 does not describe every resolver path. ## Common errors - **Adding cumulative `curl` fields.** `time_connect` already includes name-lookup time. Subtract adjacent values to estimate phase increments. - **Calling every delay “latency.”** Name the measured interval: DNS lookup, connection establishment, first-byte time, full transfer, API execution, or database wait. - **Treating a subnet as a security control.** A subnet assigns and places an address prefix. Routes and authorization rules perform different actions. - **Treating one packet capture as the full path.** A traffic distributor terminates one connection and creates another, so client capture evidence stops at that connection endpoint. - **Assuming ping failure means application failure.** Internet Control Message Protocol replies can be blocked while HTTPS remains reachable. - **Assuming HTTPS hides all metadata.** TLS protects application content and integrity under its guarantees, while addresses, some names, sizes, timing, and endpoint behavior can remain observable. ## Knowledge check **Question:** An AWS route table sends `0.0.0.0/0` to a NAT gateway and `10.20.4.0/24` to a local target. Which route matches destination `10.20.4.17`, and does the match prove that TCP port 5432 is allowed? **Answer:** The `/24` route matches because longest-prefix selection favors it over `/0`. The route establishes a next-hop choice only. A security group, network access control list, host firewall, and database listener can still deny or fail the connection. ## Exercises ### Check For `10.30.8.0/23`, calculate the number of IPv4 addresses, the first address, and the last address. Submit the binary or power-of-two calculation. Mark the mathematical block size separately from any provider-specific assignable count. ### Practice Run the ten-request `curl` test. Preserve raw rows in a text file and create a table of phase increments in milliseconds. Report the median and maximum total time, the program and clock source, one observation, one inference, and two alternative explanations. ### Challenge Design a private address plan for CampusCart with two availability zones. Include separate API and database subnet prefixes inside one stated VPC prefix, route destinations and targets, allowed communication pairs, and a non-overlap check against a campus prefix of `10.20.0.0/16`. Submit a diagram, a route table, and a firewall-intent table. The evidence must name both endpoints for every allowed flow. ## Authoritative sources and further reading ### Required standards - [RFC 9499, *DNS Terminology*](https://www.rfc-editor.org/info/rfc9499) is the current Best Current Practice terminology source for resolvers, authoritative servers, zones, and DNS answers; the document is terminology, not a complete beginner tutorial. - [RFC 1034, *Domain Names—Concepts and Facilities*](https://www.rfc-editor.org/info/rfc1034) explains the DNS namespace, resolvers, authoritative servers, caching, and resource records. Later RFCs update specific behavior, so use the RFC Editor's “Updated by” list for implementation work. - [RFC 7766, *DNS Transport over TCP—Implementation Requirements*](https://www.rfc-editor.org/info/rfc7766) establishes why “DNS uses only UDP” is incorrect. - [RFC 4632, *Classless Inter-domain Routing (CIDR)*](https://www.rfc-editor.org/info/rfc4632) documents prefix aggregation and longest-match forwarding for IPv4. - [RFC 8200, *Internet Protocol, Version 6 (IPv6) Specification*](https://www.rfc-editor.org/info/rfc8200) defines IPv6 and identifies later updates on its status page. - [RFC 9293, *Transmission Control Protocol (TCP)*](https://www.rfc-editor.org/info/rfc9293) is the current base TCP specification and replaces RFC 793 for that role. - [RFC 9110, *HTTP Semantics*](https://www.rfc-editor.org/info/rfc9110) defines version-independent HTTP methods, fields, content, and status meanings. - [RFC 9846, *The Transport Layer Security (TLS) Protocol Version 1.3*](https://www.rfc-editor.org/info/rfc9846) is the current TLS 1.3 specification as checked July 20, 2026; it obsoletes RFC 8446 while retaining version 1.3. ### Implementation documentation - [AWS: Subnet route tables](https://docs.aws.amazon.com/vpc/latest/userguide/subnet-route-tables.html) documents current route targets and longest-prefix behavior in Amazon VPC. - [AWS: VPC infrastructure security](https://docs.aws.amazon.com/vpc/latest/userguide/infrastructure-security.html) compares current security-group and network-access-control-list behavior. - [Google Cloud: VPC overview](https://cloud.google.com/vpc/docs/overview) documents the global network and regional, zone-spanning subnet scope. - [Azure: Virtual networks frequently asked questions (FAQ)](https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-faq) documents the regional virtual-network and zone-spanning subnet scope. - [Google Cloud: Regions and zones](https://cloud.google.com/compute/docs/regions-zones) identifies current regional and zonal placement constraints that address design must respect. Standards status, browser transport selection, provider defaults, reserved subnet addresses, route targets, and firewall behavior are version-sensitive. Verify the status page and the exact provider documentation before applying a rule to production. ## Next-chapter transition Networking explains how CampusCart reaches a compute endpoint and how CivicPermit separates public and private communication. The next chapter opens that endpoint and explains how hypervisors, virtual machines, containers, and managed service models divide a physical computer and divide operational responsibility. --- # Virtualization and Cloud Service Models ## Prerequisites Read [Networking Foundations for Cloud Applications](./03-networking.md), or understand that an application endpoint has an address, routes, and authorization rules. You should know the roles of a processor, memory, an operating system, and an application program. No hypervisor or container experience is required. ## Learning outcomes After this lesson, you will be able to: - explain how a hypervisor presents processors, memory, and devices to a virtual machine; - distinguish a virtual machine from an operating-system container through kernel ownership and observable artifacts; - compare Infrastructure as a Service, Platform as a Service, and Software as a Service without assuming that every product fits one category perfectly; - calculate processor demand and failure headroom with explicit units; - inspect a local execution environment for virtualization and control-group evidence; and - explain why isolation, portability, multiple zones, and managed operation each require separate tests. ## Opening concrete narrative: a machine that is not a machine CampusCart's developer asks a cloud management application programming interface (API) for a four-virtual-processor Linux instance with 16 gibibytes of memory. The provider returns an identifier, an address, and a “running” status. The developer connects through Secure Shell (SSH) and sees processors, memory, a disk, and a network interface. The developer never sees a technician install a physical server. Another CampusCart developer packages the API in a container image and runs two containers on that instance. Each container shows its own file tree and process identifiers. Both containers report the same Linux kernel release because both use the instance's kernel. A virtual machine can instead run a separate guest kernel because a hypervisor presents a virtual hardware interface. CivicPermit uses two availability zones because the permit intake application must continue after one-zone loss. An architecture diagram shows one virtual machine in each zone. The PostgreSQL database, however, has only one zonal primary and no tested standby. Compute isolation and geographic placement cannot repair an application dependency that remains in the failed zone. The investigation needs evidence at three different divisions: physical host versus virtual machine, host kernel versus container process, and provider operation versus customer operation. ## Observable evidence Inside a Linux virtual machine, `lscpu` can report a hypervisor vendor or a virtualization flag when the platform exposes one. `systemd-detect-virt` checks several known indicators. `/sys/class/dmi/id/` can contain firmware identity strings. Each artifact is produced by the guest kernel or user-space program from interfaces the hypervisor chooses to expose. A missing vendor string does not prove bare metal. The Linux path `/proc/self/status` exposes the current process identifiers and other kernel-maintained values. Namespace links under `/proc/self/ns/` identify the process's namespace memberships. `/proc/self/cgroup` exposes control-group membership. The Linux kernel produces these virtual files when a process reads them. Two processes with different process-identifier namespace links have different views of process identifiers, but the difference alone does not establish strong security isolation. A cloud instance description is a control-plane artifact. The provider API produces fields for instance type, image, zone, disks, and network interfaces. A guest operating-system log establishes what the guest observed. Provider host placement and maintenance details can remain intentionally hidden. The reported virtual central processing units (vCPUs) are guest execution interfaces, not direct proof of dedicated physical cores. Container runtime commands produce another view. `docker inspect` reports configuration and runtime metadata known to the Docker daemon. A container image manifest identifies ordered filesystem content by cryptographic digest. The Open Container Initiative (OCI) Image Specification defines interoperable manifests, indexes, layers, and configuration. An image digest establishes selected content identity; the digest does not establish that the content is safe, authorized, or currently running. ## Mechanism: virtual machines A physical processor implements an instruction set and privilege rules. An operating-system kernel expects to control privileged operations such as page tables, interrupts, and devices. A **hypervisor** mediates that control so multiple guest operating systems can use one physical host. The hypervisor schedules vCPUs onto physical processor execution time. The hypervisor and processor translation hardware map guest memory addresses through additional mappings to physical memory. Virtual device mechanisms translate guest disk and network actions into host or dedicated device actions. When a guest attempts a protected operation, processor hardware can transfer control to the hypervisor, which validates or emulates the action before the guest continues. Practitioners call the resulting execution environment a **virtual machine** (VM). A VM has a virtual hardware contract on which a guest kernel runs. Popek and Goldberg's 1974 paper gave a formal framework for virtual-machine monitors, including equivalence, control, and efficiency properties under their modeled architecture. Modern processors and hypervisors add hardware assistance and implementation techniques beyond that historical machine model, but the control distinction remains useful. Kernel-based Virtual Machine (KVM) provides a concrete current example. The Linux KVM API uses `/dev/kvm` and input/output control operations to create a VM, virtual processors, and virtual devices. User-space software supplies memory and device emulation while kernel and processor facilities execute guest work. The KVM application binary interface has been stable since Linux 2.6.22, but capabilities must be queried rather than guessed from a kernel version. Virtualization permits consolidation and different guest kernels. Virtualization does not make memory, processor time, or network capacity infinite. Host scheduling contention can delay a guest. A physical host failure can stop several VMs. Provider placement controls determine whether replicas share a failure domain. ## Mechanism: containers A Linux container normally consists of ordinary processes that share the host's Linux kernel. **Namespaces** give a selected process group distinct views of kernel objects such as process identifiers, mounts, host names, user identifiers, or network interfaces. A **control group**, written `cgroup` in Linux documentation, organizes processes hierarchically and controls or accounts for resources such as processor time and memory. The container runtime prepares a filesystem, namespace memberships, credentials, capability limits, cgroup placement, and an initial process. The kernel still schedules the resulting threads. Container processes make system calls to the host kernel; they do not boot an independent Linux kernel inside the ordinary container. Docker is one toolset that builds images and asks a runtime to start containers. Docker is not a synonym for a container, and a container is not a miniature VM. On macOS and Windows, Docker Desktop commonly uses a Linux VM because Linux containers require a Linux kernel. The visible stack can therefore be physical laptop, desktop hypervisor, Linux VM, container runtime, and container process. Container isolation depends on correct namespace, cgroup, capability, system-call, filesystem, device, and runtime configuration. Docker documentation states that containers have no resource constraints by default unless limits are configured. A process view can be isolated while processor consumption remains effectively unlimited within host scheduling. ## Technical terms in context: service models divide control The National Institute of Standards and Technology (NIST) describes three cloud service models. The categories name which computing capability the provider exposes and which parts the customer controls. **Infrastructure as a Service (IaaS)** gives the customer processing, storage, networks, and other fundamental capacity on which the customer can run arbitrary software. The provider operates the physical infrastructure and virtualization mechanism. The customer normally controls the guest operating system, deployed applications, and much network configuration. Amazon Elastic Compute Cloud (Amazon EC2) from Amazon Web Services (AWS), Google Compute Engine, and Azure Virtual Machines are current examples, subject to each product's exact terms. **Platform as a Service (PaaS)** gives the customer a provider-supported application deployment environment. The customer deploys applications with supported languages, libraries, or tools while the provider operates the underlying network, servers, operating systems, and storage mechanisms. Google App Engine's original preview made this division visible, although current platform products offer many additional options. **Software as a Service (SaaS)** gives the customer use of the provider's running application, usually through a browser or API. The provider operates the application and underlying stack. The customer still manages authorized users, data use, configuration, and integration choices within the product's interface. The following responsibility map is deliberately simplified: | Operational object | On premises | IaaS | PaaS | SaaS | |---|---|---|---|---| | buildings, hardware, host network | customer | provider | provider | provider | | hypervisor or host kernel | customer | provider | provider | provider | | guest operating system | customer | customer | provider | provider | | language runtime and scaling mechanism | customer | customer | mostly provider | provider | | application code | customer | customer | customer | provider | | tenant configuration, identities, and lawful data use | customer | customer | customer | shared through product controls | “Managed PostgreSQL” sits on a spectrum near PaaS: the provider patches database software and manages selected backup or failover mechanisms, while the customer designs schemas, queries, roles, retention, connectivity, and restore validation. NIST's categories are useful responsibility questions, not labels that settle every product detail. Serverless functions and managed container platforms can also combine properties. ## Working model: processor demand and zone failure Define processor demand per request as vCPU-seconds consumed by the API divided by completed requests during the same controlled load test. Assume the tested CampusCart request mix uses 0.012 vCPU-seconds per completed request. At 80 completed requests per second, expected processor demand is: `80 requests/second × 0.012 vCPU-seconds/request = 0.96 vCPU` At 400 completed requests per second, expected demand is: `400 × 0.012 = 4.8 vCPU` Assume each four-vCPU VM has a target ceiling of 60% sustained processor use to preserve latency headroom. Each VM then contributes `4 vCPU × 0.60 = 2.4 vCPU` of planned capacity. Two VMs, one per zone, provide `2 × 2.4 = 4.8 vCPU`, exactly the planned peak under ordinary operation. Loss of either zone leaves only 2.4 vCPU, corresponding to `2.4 ÷ 0.012 = 200` requests per second. Two-zone placement therefore fails the 400-request-per-second zone-loss requirement. Four VMs, two per zone, provide 9.6 vCPU normally. Loss of one zone leaves 4.8 vCPU, corresponding to the tested 400 requests per second at the 60% ceiling. The arithmetic supports a compute-capacity claim under the assumptions. The working model assumes linear scaling, equal request mixtures, even load distribution, independent zone placement, and no database or network limit. Processor demand can rise under cache misses, garbage collection, encryption, or retries. A vCPU is a scheduled execution abstraction, not necessarily one dedicated physical core. The calculation predicts neither cost nor response-time percentiles. ## Empirical test: inspect the execution boundary On Linux, run the following read-only commands and preserve the output: ```sh uname -a lscpu 2>/dev/null | sed -n '1,30p' systemd-detect-virt 2>/dev/null || true cat /proc/self/cgroup ls -l /proc/self/ns/ grep -E '^(Name|Pid|NSpid|Cpus_allowed_list|Mems_allowed_list):' /proc/self/status ``` The controlled object is one shell process. The commands observe the kernel release, reported processors, virtualization hints, cgroup membership, namespace identities, and allowed processors or memory nodes. A cgroup v2 line commonly begins `0::`; cgroup v1 or hybrid arrangements show different controller fields. The installed kernel and distribution documentation control interpretation. If Docker is already installed and an approved local image is present, compare host and container evidence without pulling new content: ```sh docker image ls --digests docker run --rm --network none APPROVED_LOCAL_IMAGE sh -c \ 'uname -a; cat /proc/self/cgroup; ls -l /proc/self/ns/' ``` Keep the host, image digest, runtime version, and command constant. Vary only host execution versus container execution. Linux containers should report the same kernel release as their Linux host, while namespace identifiers or cgroup paths can differ. Docker Desktop can insert a Linux VM, so the relevant “host” for kernel sharing is the hidden Linux VM rather than the macOS or Windows kernel. A differing kernel release between an ordinary Linux host and its purported direct Linux container would reject the simple shared-kernel explanation and suggest a VM boundary, a remote runtime, or a mistaken comparison. Equal kernel strings do not prove isolation or common physical hardware. ## Worked investigation: CivicPermit's untested database boundary ### Initial question Will CivicPermit continue to commit synthetic permit submissions after the loss of either configured availability zone? ### Operating conditions Staging has two four-vCPU API VMs, one in each zone. A managed PostgreSQL primary resides in zone A with automated backups but no configured cross-zone standby. A test client submits one synthetic permit per second. Each request carries a unique identifier and no personally identifiable information (PII). Operators use the provider's approved fault-injection procedure to make the zone-A API and database endpoints unavailable after two steady minutes. ### Evidence collected The deployment export confirms API placements and the single-zone database setting. Before intervention, 120 client requests return 201 and produce 120 unique database rows. After intervention, the traffic distributor selects the zone-B API. The next 90 client requests return 503. Zone-B API logs show database connection timeouts. No new database rows appear. Audit-log delivery continues to a regional destination. ### Interpretation The zone-B VM and API process remain available, but every submission depends on the unavailable zone-A database. The traffic switch proves compute-path failover only. Automated backup configuration provides recoverable copies under its own policy; a backup does not act as a live database endpoint. ### Proposed intervention or prediction Configure the managed PostgreSQL product's supported multi-zone standby, validate synchronous or provider-documented replication behavior, and add bounded connection retries with idempotent request identifiers. Predict that a repeated test will show a finite database failover interval, followed by one committed row per accepted identifier and no duplicates. ### Observed result In the repeated staging exercise, 71 requests return 503 during a 71-second failover interval. The next 229 return 201 through the zone-B API. PostgreSQL contains 349 unique rows: 120 before failure plus 229 after failover. No duplicate identifier appears. ### Bounded conclusion The tested multi-zone database configuration and idempotent write path restored staging commits after 71 seconds under the selected fault. The original single-zone database was the dependency that defeated the earlier two-zone claim. ### Remaining uncertainty A region-wide event, simultaneous host failures, replication lag under peak load, exhausted retry queues, backup corruption, identity failure, and provider control-plane impairment remain untested. CivicPermit's required recovery time and recovery point must be compared with the measured result by the responsible agency. ## Normal case, failure cases, and qualifications In the normal IaaS case, the provider places a VM, the hypervisor schedules vCPUs and maps memory, the guest kernel starts, and the customer's API process listens on an authorized interface. Instance metrics, guest logs, and request evidence agree about useful work. A container can package that API while the VM supplies its Linux kernel. A **noisy-neighbor** edge case occurs when several guests or containers contend for shared processor, cache, storage, or network capacity. Provider isolation prevents one tenant from directly controlling another tenant's guest, but scheduling contention can still increase timing variance. Dedicated-host options change some sharing conditions, not every downstream dependency. Memory pressure produces a different failure. A container without a suitable limit can consume host memory until the Linux out-of-memory mechanism kills a process. A hard limit can instead cause the container workload to be killed when it exceeds the configured ceiling. Processor overcommit can degrade time gradually, while unsafe memory allocation can cause abrupt termination. Live migration and provider maintenance are implementation-specific qualifications. A provider can move or restart selected VMs according to product policy. Local ephemeral disks can have a different lifecycle from persistent block volumes. Read the current machine-family and maintenance documentation for the exact region and instance type. ## Common errors - **“A vCPU is a dedicated core.”** A vCPU is the processor execution interface presented to a guest. Dedication, simultaneous multithreading, scheduling, and overcommit depend on the product and host arrangement. - **“A container includes its own kernel.”** An ordinary Linux container shares the host Linux kernel. A VM or sandboxed runtime can add another kernel boundary. - **“An image digest proves security.”** A digest identifies content bytes. Provenance, signature policy, vulnerability review, runtime configuration, and authorization require separate evidence. - **“PaaS removes operations.”** Provider staff and software perform more operations, while customers still operate code, data, identity, configuration, testing, and incident response within the exposed interface. - **“Two zones provide zone-failure capacity.”** Surviving instances and every required dependency must carry the failure load. The processor calculation and database exercise show two separate ways the claim can fail. - **“Backups provide immediate availability.”** A backup supports restoration. A live standby and a tested failover path address a different recovery interval. ## Knowledge check **Question:** A CampusCart API container and its Linux VM host report the same `uname -r` kernel release. Which mechanism explains the match, and which claim remains unproved? **Answer:** An ordinary Linux container uses the host Linux kernel, so the shared kernel release is expected. The match does not prove secure isolation, correct cgroup limits, identical filesystem content, or common physical-host placement. ## Exercises ### Check Create a responsibility table for CampusCart on IaaS, a managed application platform, and a purchased bookstore SaaS product. Include hardware, guest patching, runtime patching, application code, database schema, user access, backups, and restore tests. Submit one evidence artifact for every customer-owned row. ### Practice Run the read-only Linux inspection commands on a VM, Linux host, or approved lab environment. Preserve raw output. Annotate which program or kernel interface produced each line, which boundary the line exposes, and one claim the line cannot establish. ### Challenge Recalculate CampusCart's zone-failure capacity when a request consumes 0.016 vCPU-seconds, each VM has eight vCPUs, the planned ceiling is 55%, and peak demand is 600 requests per second. Choose an integer VM count per zone that survives one-zone loss. Submit equations with units, normal and failure utilization, assumptions, and one empirical load test that could reject linear scaling. ## Authoritative sources and further reading ### Required reading - [NIST Special Publication 800-145, *The NIST Definition of Cloud Computing*](https://doi.org/10.6028/NIST.SP.800-145) provides the authoritative IaaS, PaaS, and SaaS capability definitions used here. - [Popek and Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures”](https://doi.org/10.1145/361011.361073) is the primary formal paper on classic virtual-machine monitor properties and their modeled architectural conditions. ### Implementation documentation - [Linux kernel: KVM API](https://docs.kernel.org/virt/kvm/api.html) describes the stable file-descriptor and input/output-control interface used to create and operate KVM guests. Query KVM capabilities rather than relying on a version-number guess. - [Linux kernel: Control Group v2](https://docs.kernel.org/admin-guide/cgroup-v2.html) is the rolling authoritative description of cgroup v2 hierarchy, controllers, delegation, and observable files. Match the page to the installed kernel when behavior matters. - [Open Container Initiative Image Specification](https://specs.opencontainers.org/image-spec/) defines current interoperable image manifests, indexes, filesystem layers, and configuration. The rolling site showed a November 2025 publication state when checked July 20, 2026. - [Docker: Resource constraints](https://docs.docker.com/engine/containers/resource_constraints/) documents current processor and memory controls and warns that containers have no constraints by default. - [Google Cloud: Compute Engine overview](https://cloud.google.com/compute/docs/overview) identifies Compute Engine as an IaaS offering with VMs using KVM; the page was last updated July 17, 2026 when checked. - [AWS: Nested virtualization on EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-ec2-nested-virtualization.html) provides a current concrete diagram of physical infrastructure and Nitro hypervisor, a guest instance, and nested guests. Supported instance types and restrictions are version-sensitive. ### Advanced reference - [International Business Machines (IBM): Overview of the z/VM Control Program](https://www.ibm.com/docs/en/zvm/7.4.0?topic=zvm-overview-control-program-cp) shows a current, explicit mapping of virtual processors, memory, and devices in a long-running VM lineage. The linked page is for z/VM 7.4.0; use the documentation matching an installed release. Provider VM features, container runtime defaults, Linux interfaces, cgroup mode, instance maintenance behavior, and managed-platform responsibility divisions change over time. Record product, version, region, date, and configuration with every operational claim. ## Next-chapter transition The first four chapters established the cloud application map, the historical mechanisms that produced it, the network path between programs, and the compute abstractions beneath those programs. The next part can now examine provider identities, permissions, storage, databases, and the core mechanisms that turn raw infrastructure into a production cloud application. --- # AWS Core Platform and the Web Request Path ## Prerequisites and outcomes **Prerequisites.** Read the earlier lessons on Internet Protocol (IP) addressing, Domain Name System (DNS) resolution, Transmission Control Protocol (TCP), Transport Layer Security (TLS), and virtual machines. The networking lesson supplies the distinction between a DNS name, an IP address, and a TCP port. The virtualization lesson explains why a virtual machine can fail while the physical cloud platform continues to operate. A direct-link reader should know that a Hypertext Transfer Protocol (HTTP) request contains a method, a path, headers, and sometimes a body. **After this chapter, you can:** - follow one browser request through a typical Amazon Web Services (AWS) deployment; - distinguish a Region, an Availability Zone, a virtual private cloud, and a subnet; - connect DNS, load balancing, compute, a relational database, and object storage to evidence that each product produces; - calculate a first capacity target with explicit units and headroom; and - investigate a slow or failed request without treating an architecture diagram as proof of runtime behavior. Product names and console layouts in this chapter are current as of **July 2026**. The request stages, fault-isolation reasoning, and measurement methods are durable. Region inventories, default settings, quotas, metric interfaces, and product features are version-sensitive and require a documentation check before a production change. ## A purchase during registration week At 9:02 a.m. on registration Monday, Maya opens the CampusCart university bookstore in a browser from a residence hall. CampusCart serves about 5,000 users on an ordinary day. The application normally completes 80 HTTP requests each second and reaches 400 completed HTTP requests each second during registration peaks. Maya selects a calculus text, chooses **Buy**, and expects one order row to appear in PostgreSQL while the cover image remains available from object storage. CampusCart runs in one AWS Region. An Application Load Balancer has network interfaces in two Availability Zones. Stateless application instances run in private subnets in both zones. An Amazon Relational Database Service (Amazon RDS) PostgreSQL database stores orders. An Amazon Simple Storage Service (Amazon S3) bucket stores book images. Amazon Route 53 hosts the public DNS record for `shop.campuscart.example`. Maya reports a 6-second pause followed by an HTTP `503 Service Unavailable` response. The incident responder needs to locate the wait. A high-level diagram cannot establish whether Maya waited for DNS, TLS, the load balancer, application code, or PostgreSQL. Runtime artifacts can narrow the question because different programs create evidence at different stages. ## Evidence before explanation Maya's browser developer tools produce a network entry when the browser starts and completes the request. Common fields include the URL, HTTP status, transferred bytes, and timing phases such as DNS lookup, connection establishment, TLS negotiation, time to first response byte, and content download. The browser clock supplies the local timestamps. A long “waiting for server response” interval establishes that the browser had sent the request and had not yet received the first response byte. The browser entry alone does not identify which server-side operation consumed the interval. The Application Load Balancer can produce an access-log line after a request finishes when access logging is enabled. Fields can identify the client address, target address, request line, load-balancer status, target status, bytes, and separate request-processing, target-processing, and response-processing times. The load balancer produces the line, not Maya's browser and not the application. A target-processing time near 5.8 seconds supports an inference that most of the delay occurred after forwarding to an application target. The field does not prove that PostgreSQL caused the delay because application code, an outbound call, or local CPU contention can consume the same interval. The CampusCart application produces a structured log entry when its request handler runs. A useful entry contains a generated request identifier, route name, selected instance identifier, Availability Zone, database-call duration in milliseconds, outcome, and timestamp. Amazon RDS and PostgreSQL produce database logs and performance observations according to the enabled settings. Amazon CloudWatch receives provider-produced measurements and application-published measurements. A CloudWatch measurement needs a name, unit, timestamp, and identifying dimensions; a statistic such as an average or percentile summarizes observations over a stated period. AWS documentation in July 2026 describes both OpenTelemetry ingestion and the classic CloudWatch metric format. The ingestion interface can change while the need to identify the measurement producer and aggregation remains constant. Three statements must remain separate: 1. **Observation:** the load-balancer log reports 5.8 seconds of target processing and an HTTP 503 response. 2. **Inference:** a target accepted the request but did not produce a successful response promptly. 3. **Assumption to test:** the target waited for an exhausted PostgreSQL connection pool. ## How the request moves Route 53 answers the browser's DNS query with addresses associated with the Application Load Balancer alias target. A recursive DNS resolver can cache the answer until the DNS time to live expires. The browser then establishes a TCP connection and negotiates TLS with a load-balancer listener configured for HTTPS, commonly TCP port 443. The listener evaluates ordered rules against request properties such as the host and URL path. A matching rule selects a target group. The target group contains registered instance or IP targets and a health-check configuration. Each load-balancer node sends periodic health-check requests. A target becomes eligible only after registration and a successful initial health check. The node normally forwards application requests only to targets considered healthy. An application target receives the HTTP request on the target group's configured protocol and port. CampusCart validates Maya's session, checks inventory, begins a PostgreSQL transaction, writes the order, and commits. A later browser request can retrieve the cover image from an S3 URL. S3 is not a mounted disk in this design. S3 accepts operations against object keys through an application programming interface (API), and CampusCart must authorize those operations separately from database access. The network placement names now have concrete referents. An AWS **Region** is a geographic area that AWS operates separately from other Regions. An **Availability Zone** is an isolated location inside a Region; AWS describes zones in a Region as connected by low-latency, high-bandwidth, redundant networking. A **virtual private cloud (VPC)** is a customer-configured virtual network within AWS. A **subnet** is an IP-address range tied to one Availability Zone. Each subnet has one associated route table, directly or through the VPC's main route table, and each route names a destination address range and a next target. A subnet is not “public” because of its name. Public reachability depends on routes, gateways, public addresses, and packet filters on both sides of the traffic path. CampusCart's load balancer uses subnets with routes appropriate for Internet-facing traffic. Application targets use private subnets and receive traffic from the load balancer according to their security-group rules. The database permits PostgreSQL traffic from the application targets rather than from arbitrary Internet addresses. An Amazon Elastic Compute Cloud (Amazon EC2) Auto Scaling group maintains a minimum, desired, and maximum instance count from a launch template. Scaling policies can change desired capacity from measurements or schedules. New capacity requires launch, boot, application initialization, target registration, and successful health checking before the load balancer can use the target. Autoscaling therefore changes future capacity; autoscaling does not make an already queued request finish immediately. ## A capacity model with units and assumptions CampusCart's team load-tests one initialized application instance. With a controlled request mix, one instance completes 45 requests per second while its 95th-percentile response time remains below 300 milliseconds. “Requests per second” means successful, representative CampusCart requests completed by the instance divided by the observed seconds. The registration target is 400 completed requests per second. The team limits planned utilization to 60 percent so that retries, uneven distribution, and short bursts have room. Let: - \(L\) = forecast arrival load, in requests per second; - \(C\) = measured per-instance capacity, in requests per second; - \(u\) = planned utilization fraction, dimensionless; and - \(N\) = required healthy application instances. Each instance contributes \(C u\) planned requests per second. Therefore the fleet must satisfy \(N C u \ge L\), which gives: \[ N = \left\lceil \frac{L}{C u} \right\rceil = \left\lceil \frac{400\ \text{requests/s}} {(45\ \text{requests/s/instance})(0.60)} \right\rceil = 15\ \text{instances}. \] Two-zone placement suggests at least 8 instances in one zone and 7 in the other during normal operation. A stricter requirement to survive either zone with the same 60-percent target needs enough remaining instances in one zone to serve all 400 requests per second. The stricter assumption leads to 15 instances **per zone**, or 30 total. Capacity planning therefore depends on the stated failure requirement, not only on peak traffic. The calculation assumes the load-test request mix matches registration traffic, PostgreSQL and downstream capacity do not bind first, instances have equal capacity, and new targets are already warm. The calculation omits connection limits, queue growth, zone imbalance, cache behavior, and service quotas. Omitted factors matter when the database saturates, when traffic spikes faster than instance readiness, or when one zone fails. ## Make the mechanism visible Run an empirical test in a nonproduction environment. Hold the application build, instance type, database dataset, object size, region, and request mix constant. Increase offered load in steps of 40 requests per second every two minutes. Preserve the load generator's completed-request count and latency distribution, load-balancer request and target times, healthy target count, application database-wait duration, and PostgreSQL connection count. Expected evidence shows nearly proportional completed throughput while spare capacity exists. Latency and errors should rise when a limiting resource saturates. A step increase in healthy targets should raise the sustainable completion rate after the boot and health-check delay. Evidence that would reject the application-capacity explanation includes flat application CPU, no queue growth, and a database connection count already at its limit. Alternative explanations include a load-generator limit, network packet loss, an artificial health endpoint that reports success while dependencies fail, or a different request mix during production. ## Worked investigation: registration 503 responses **Initial question.** Why did 4.1 percent of checkout requests return HTTP 503 between 09:02 and 09:06? **Operating conditions.** CampusCart received 390 to 430 requests per second. Eight application instances were healthy across two Availability Zones. A scaling policy requested six more instances after a CPU alarm. Each application instance allowed 20 PostgreSQL connections, and the database accepted at most 160 application connections. **Evidence collected.** The browser entry showed a successful connection followed by 6 seconds waiting for a response. Load-balancer logs placed most time in target processing. Load-balancer measurements kept eight healthy targets until 09:05. Application entries with the same request identifiers reported database-pool waits near 5 seconds. PostgreSQL observations reported 160 active or reserved application connections. Auto Scaling activity history reported that six instances launched at 09:03 but did not become healthy until 09:05. **Interpretation.** Existing handlers consumed the database pool before replacement capacity became ready. Extra instances could not help before registration. Extra instances also would request more database connections, so compute scaling alone could move the limit to PostgreSQL. **Proposed intervention and prediction.** A scheduled scaling action will place 15 warm instances before registration begins. A bounded checkout queue and shorter database transactions will reduce simultaneous connection occupancy. The predicted result is fewer pool waits, provided that measured PostgreSQL capacity supports the resulting transaction volume. **Observed or calculated result.** A replay with 15 pre-warmed instances and optimized transaction duration completed 418 requests per second. The 95th-percentile checkout time was 410 milliseconds, and HTTP 5xx responses were 0.2 percent. PostgreSQL connection usage peaked at 142. **Bounded conclusion.** Warm application capacity and shorter transactions removed the reproduced connection-pool bottleneck at the tested request mix through 418 completed requests per second. **Remaining uncertainty.** The replay did not simulate an Availability Zone failure, an S3 slowdown, a payment-provider slowdown, or traffic above 430 requests per second. Production success still requires alarms, quota checks, and a zonal-failure test. ## Normal operation, edge behavior, and qualifications During normal operation, Route 53 returns the load-balancer alias, TLS terminates at a listener, a listener rule selects the CampusCart target group, and a healthy instance completes the request. The application writes the order to PostgreSQL and returns an HTTP response. Multiple zones reduce dependence on one location only when targets, capacity, and relevant data paths actually use multiple zones. Several edge cases weaken a simplified diagram: - Application Load Balancer health checks are periodic samples, not proofs that every application operation works. A shallow `/health` route can succeed while PostgreSQL access fails. - AWS documents a fail-open behavior when all registered targets in a target group are unhealthy: the load balancer can route to all targets regardless of health. Practitioners must not assume that “all unhealthy” means “no forwarding.” - A DNS client can retain an address until its cache entry expires. DNS failover therefore does not move every client at one instant. - Auto Scaling replaces impaired instances and changes desired count, but readiness consumes time. Scheduled capacity or a higher minimum can be necessary for a known peak. - An Amazon RDS Multi-AZ deployment improves database availability through standby or multi-instance arrangements described by the selected deployment type. Multi-AZ does not remove application reconnection behavior, failover time, logical corruption, or the need for tested backups. - AWS Regions isolate regional infrastructure. AWS does not automatically replicate every regional resource to another Region. Cross-Region recovery requires explicit design and testing. ## Common reasoning errors **“A private subnet cannot communicate with the Internet.”** Subnet labels do not decide reachability. Route tables, gateways, addresses, security groups, network access control lists, and return paths jointly determine packet delivery. **“The load balancer makes the application highly available.”** The load balancer selects available targets; the application still needs healthy targets, sufficient capacity, correct zonal placement, and available dependencies. **“Average latency proves users are fast.”** An average can hide a slow minority. Preserve a latency distribution and a stated percentile over a stated interval. **“Four hundred requests per second requires ten instances because each instance handled forty.”** A division without headroom assumes perfect balancing and no failures. The capacity test must also represent the production request mix and dependency limits. **“CloudWatch caused an API action because a graph changed at the same time.”** A metric graph establishes a time relationship. CloudTrail activity, scaling history, and application evidence are needed to support a causal sequence. ## Knowledge check **Question.** A target reports healthy every 30 seconds, but checkout requests time out while the static home page works. Which evidence should the responder collect next, and why does health status not settle the question? **Answer.** Correlate load-balancer access entries, application request entries, database-pool wait measurements, and PostgreSQL observations with a request identifier and time window. The health request may exercise only a simple route. A successful sample proves that the sampled route responded under its health-check conditions; the sample does not prove that checkout's database path had capacity. ## Evidence exercises ### Check Draw Maya's request path from browser DNS query through Route 53, the Application Load Balancer, an application target, and PostgreSQL. Label every protocol boundary, port where known, artifact producer, and location scope. Submit the diagram and a four-row table that states one direct observation and one unsupported inference for each artifact. ### Practice Use a local HTTP server and a load generator, or an approved cloud sandbox. Hold payload and code constant. Increase concurrency in at least five steps. Preserve the exact command, start and end timestamps, completed requests per second, error count, and 50th- and 95th-percentile latency. Submit a graph and a causal explanation that identifies the first observed limit. Do not claim an AWS product limit from a local test. ### Challenge Create two capacity calculations for CampusCart at 400 completed requests per second: normal two-zone operation and survival of either zone. Use the measured 45 requests per second per instance and a 60-percent planned utilization. Add one database-capacity constraint of your choice with explicit units. Submit assumptions, equations, rounded instance counts, and a test that could disprove the selected database limit. ## Authoritative sources and further reading ### Required reading - [AWS Regions and Availability Zones](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions-availability-zones.html) identifies geographic and fault-isolation scopes and warns that regional resources are not automatically replicated. - [Application Load Balancer introduction](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html) names listeners, rules, target groups, targets, and health checks used in the request path. - [Amazon VPC subnet route tables](https://docs.aws.amazon.com/vpc/latest/userguide/subnet-route-tables.html) establishes the destination-and-target mechanism for subnet routing. ### Implementation and version-sensitive documentation - [Application Load Balancer health checks](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html) documents current thresholds, target states, and fail-open behavior. - [Amazon EC2 Auto Scaling overview](https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html) documents minimum, maximum, desired capacity, replacement, and load-balancer registration. - [Amazon CloudWatch metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html) documents metric producers and the July 2026 ingestion choices; revisit the page because interfaces and limits change. - [Amazon RDS Multi-AZ DB instance deployments](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) describes current database deployment and failover behavior. ### Advanced reference - [AWS Well-Architected Reliability Pillar](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html) connects quotas, testing, recovery objectives, and fault isolation to production design. ## Transition CampusCart's request path gives concrete meanings to regional placement, routing, managed load balancing, compute, object storage, and a managed database. The next chapter keeps the same application and request so that provider vocabulary cannot hide architectural differences. Google Cloud and Microsoft Azure offer comparable capabilities, but their resource hierarchies, network scopes, load-balancer choices, and identity attachment points are not interchangeable names. --- # Comparing Google Cloud and Microsoft Azure ## Prerequisites and outcomes **Prerequisites.** Read the preceding Amazon Web Services (AWS) request-path chapter and the earlier networking lessons. The AWS chapter supplies concrete meanings for Region, Availability Zone, Domain Name System (DNS) answer, load-balancer listener, application target, object, and relational transaction. A reader arriving directly should understand that a cloud request crosses separately observable DNS, transport, application, and data stages through protocols such as Hypertext Transfer Protocol (HTTP). **After this chapter, you can:** - translate an application architecture among Amazon Web Services (AWS), Google Cloud, and Microsoft Azure without claiming that similarly named products behave identically; - compare organization and billing hierarchies, geographic scopes, request-routing products, compute choices, managed PostgreSQL, object storage, identity, and observability; - build and test a provider-neutral latency budget with milliseconds as the unit; and - identify which comparison claims require a current product, region, quota, and pricing check. Provider catalogs change frequently. Product names, supported regions, feature tiers, defaults, and quotas in this lesson are current as of **July 2026**. The durable subject is the method: name a workload requirement, locate the provider control that implements the requirement, collect evidence, and test the resulting path. ## The same bookstore, a different procurement decision CampusCart's university is renewing a hosting agreement. One architecture group proposes Google Cloud because another research unit already uses Google identity and billing. A second group proposes Azure because the university manages workforce accounts in Microsoft Entra ID. The existing AWS deployment serves 5,000 daily users, normally completes 80 HTTP requests per second, and reaches 400 completed HTTP requests per second during registration. PostgreSQL stores orders, while object storage holds book images. The procurement team asks, “Which provider has the equivalent of the current AWS products?” A catalog-matching answer would be misleading. A Google Cloud global external Application Load Balancer can use a single anycast IP address with backends in multiple regions. An AWS Application Load Balancer is a regional endpoint. Azure Front Door, Azure Application Gateway, and Azure Load Balancer operate at different network scopes and protocol levels. All products distribute traffic, but the traffic entry point, supported protocols, failure scope, health mechanism, and billing dimensions differ. The engineering team therefore traces one checkout on each proposed platform. The team also records the administrative parent of every deployed object, the geographic location, the identity used by application code, and the producer of every operational artifact. A translation becomes credible only when the translated deployment produces the required behavior under measurement. ## What the engineers can observe The browser produces the same initial evidence on all three providers: DNS timing, connection timing, Transport Layer Security (TLS) timing, HTTP status, bytes, and time to first response byte. Provider choice does not change what the browser directly establishes. Provider choice changes the managed endpoints that receive the packets and the artifacts available inside the provider network. Google Cloud Load Balancing can produce request logs in Cloud Logging when logging is configured for the selected load-balancer type and backend. A log entry can identify the forwarding or URL-map decision, backend, status, latency, and request properties. Google Cloud's load-balancer implementation produces the entry. The CampusCart application must separately produce an application request identifier and database timing if the team needs an end-to-end correlation. Azure Front Door or Application Gateway can send access and health-probe categories to Azure Monitor destinations through configured diagnostic settings. The chosen Azure product produces those entries. Fields and category names depend on product and diagnostic schema. Azure Resource Health and Azure Activity Log answer different questions: platform health describes provider-side resource health, while the Activity Log describes control-plane actions against Azure resources. Neither artifact is a substitute for the application's own checkout entry. An inventory export supplies administrative evidence. Google Cloud Resource Manager represents an organization at the root, optional folders below it, projects below folders or the organization, and product resources inside projects. Policies attached above a project can be inherited downward. Azure Resource Manager places management groups above subscriptions and resource groups within subscriptions; Azure resources reside in resource groups. Microsoft Entra tenants supply identity context, but a tenant is not the same object as a subscription. AWS Organizations groups AWS accounts under a management account and optional organizational units. The three hierarchies solve related governance problems with different parent types and lifecycle rules. ## Translate requirements, not names The following table is a starting map, not a declaration of equivalence. | Workload need | AWS example | Google Cloud example | Azure example | Comparison question | |---|---|---|---|---| | Administrative isolation and billing scope | AWS account in AWS Organizations | Google Cloud project under a folder or organization | Azure subscription under a management group | Where do quotas, invoices, inherited policy, and blast radius attach? | | Private IP network | Amazon Virtual Private Cloud (VPC) | Virtual Private Cloud network | Azure virtual network | Is the network regional or global, and what does a subnet's region or zone mean? | | Public DNS | Amazon Route 53 | Cloud DNS | Azure DNS | Which authoritative name servers answer, and how is health-based routing implemented? | | HTTP entry and routing | Application Load Balancer; other AWS edge products for global entry | Global or regional external Application Load Balancer | Azure Front Door or Application Gateway | Where does TLS terminate, what is the routing scope, and which backends are supported? | | Virtual-machine fleet | Amazon Elastic Compute Cloud Auto Scaling group | Compute Engine managed instance group | Azure Virtual Machine Scale Sets | How are health, image rollout, placement, warm-up, and scale-in controlled? | | Managed application platform | AWS App Runner, Elastic Beanstalk, containers, or functions depending on requirements | Cloud Run, App Engine, containers, or functions depending on requirements | Azure App Service, Container Apps, containers, or functions depending on requirements | Which runtime and network responsibilities remain with the customer? | | Managed PostgreSQL | Amazon Relational Database Service for PostgreSQL or Aurora PostgreSQL-Compatible Edition | Cloud SQL for PostgreSQL or another selected database | Azure Database for PostgreSQL | What topology, failover, backup, connection, extension, and version behavior applies? | | Object storage | Amazon Simple Storage Service (S3) | Cloud Storage | Azure Blob Storage | What are the consistency, retention, replication-location, authorization, and egress rules? | | Workload identity | AWS Identity and Access Management role credentials | Attached service account or workload identity mechanism | Managed identity or workload identity mechanism | Which principal obtains a short-lived token, and which policy authorizes the action? | | Operations evidence | CloudWatch, CloudTrail, and product logs | Cloud Monitoring, Cloud Logging, and Cloud Audit Logs | Azure Monitor, Activity Log, and product diagnostic logs | Which producer creates each measurement or event, and what collection must be enabled? | Several nearby terms require care. A Google Cloud project is not merely a folder for resources. A project is the fundamental application programming interface (API), billing-association, and permission context for many Google Cloud operations. An Azure resource group is a lifecycle and management container inside a subscription; an Azure resource can belong to only one resource group at a time. An AWS account supplies a strong administrative and billing isolation scope and can be governed through AWS Organizations. Moving a design among providers therefore requires a governance translation before product deployment begins. Network scope also differs. A Google Cloud VPC network is a global resource, while its subnets are regional. An Amazon VPC belongs to one AWS Region, and its subnets belong to Availability Zones. An Azure virtual network belongs to a region, while availability-zone support depends on the resource type and region. A copied diagram that labels every box “VPC” hides those scope differences and can lead to incorrect routing or recovery assumptions. ## One checkout path on each platform An AWS regional path can use Route 53, an Application Load Balancer in two zones, application targets, Amazon Relational Database Service (Amazon RDS) for PostgreSQL, and Amazon Simple Storage Service (Amazon S3). A Google Cloud path can use Cloud DNS, a global or regional external Application Load Balancer, a managed instance group or Cloud Run backend, Cloud SQL for PostgreSQL, and Cloud Storage. An Azure path can use Azure DNS, Azure Front Door for global application entry or Application Gateway for regional web routing, Virtual Machine Scale Sets or App Service, Azure Database for PostgreSQL, and Blob Storage. Each path still performs causal steps. A resolver obtains an address. A client opens a transport connection. A frontend accepts TLS and selects a backend. Application code authenticates Maya, obtains database connectivity, executes a transaction, and returns a response. Object retrieval follows a separately authorized path. Provider products implement those actions at different scopes; no brand removes the need to measure the actions. The word **global** needs a named object. A global Google Cloud load balancer can present a global frontend and use backends in multiple regions for supported configurations. A globally distributed frontend does not automatically make CampusCart's single-region PostgreSQL database multi-region. Azure Front Door can direct traffic among regional origins, but a single regional database remains a regional dependency. Route 53 can direct clients among regional AWS endpoints, but DNS routing alone does not copy orders between databases. ## A latency budget with units and assumptions CampusCart sets a 95th-percentile objective of 500 milliseconds for a warm checkout request. The percentile means that at least 95 percent of qualifying checkout requests complete in 500 milliseconds or less during the measured interval. The team decomposes the browser-observed duration into serial stages for a first approximation: \[ T_{checkout}=T_{dns}+T_{connect}+T_{frontend}+T_{app}+T_{db}+T_{response}. \] The variables use milliseconds per checkout. The connection stage includes Transmission Control Protocol (TCP) and TLS. A measured candidate budget is: | Stage | Budget | |---|---:| | DNS lookup | 15 ms | | TCP and TLS connection | 55 ms | | Provider frontend routing | 20 ms | | Application work excluding database | 90 ms | | PostgreSQL transaction | 170 ms | | First response byte through body completion | 60 ms | | **Estimated total** | **410 ms** | The 90-millisecond margin between 410 and 500 milliseconds is not spare capacity in a universal sense. The model assumes a fresh DNS and transport path, serial stages, a warm application target, one database transaction, a controlled response size, and no retry. Reused connections can remove much of the DNS and connection work. Parallel calls can make addition overestimate elapsed time. Queuing and retries can make the estimate underestimate tail latency. Clock disagreement prevents safe subtraction between unrelated hosts unless the evidence supplies a correlation method and synchronized time. The budget predicts where a provider migration can spend time without violating the objective. A candidate platform that adds 80 milliseconds of application-to-database network time consumes most of the margin. The calculation does not rank providers. Only measurements from equivalent regions, datasets, request mixes, compute sizes, database tiers, and connection policies support a comparison. ## An empirical cross-provider test Deploy the same minimal checkout-shaped application in approved nonproduction projects or subscriptions. Use the same application build, response bytes, synthetic dataset size, PostgreSQL schema, connection-pool limit, client region, and load schedule. Choose approximately comparable compute and database capacity, then disclose every mismatch. Increase offered traffic from 40 to 400 requests per second in ten steps. “Offered requests per second” means requests started by the generator divided by test seconds; “completed requests per second” means responses completed by the generator divided by the same interval. Preserve browser or load-generator timings, frontend access entries, backend request entries, database duration, error codes, healthy-backend count, and cost-export identifiers. Expected evidence should show that frontend time remains a minority stage for a warm regional request and that application or database queuing grows near a capacity limit. A result in which one provider is faster can reject equality under the tested configuration, but the result cannot establish a universal provider ranking. Alternative explanations include unequal instance processors, database storage tiers, network tiers, cold starts, quota throttling, log-sampling differences, and a client that is geographically closer to one region. ## Worked investigation: a false portability assumption **Initial question.** Why does the Google Cloud CampusCart prototype return fast image responses but intermittently time out during order creation at 400 requests per second? **Operating conditions.** The prototype uses a global external Application Load Balancer, backends in two regions, one Cloud SQL for PostgreSQL instance in a single region, and Cloud Storage for images. The load generator runs from two geographic locations. Application builds and response sizes match. **Evidence collected.** Frontend request entries identify healthy backends in both application regions. Application entries show that requests sent to the region without the database wait longer for database connections and network responses. Database measurements show a connection ceiling during the peak. Image requests avoid PostgreSQL and remain fast. Resource inventory confirms that the database exists only in the first region. **Interpretation.** The global frontend distributed application requests farther than the single-region database topology could support efficiently. Additional application backends increased concurrent database demand but did not add database write capacity. **Proposed intervention and prediction.** Constrain write-serving application backends to the database region for the current architecture, or design and validate an explicit multi-region data strategy with conflict, consistency, recovery, and cost requirements. The constrained design predicts lower database network time and fewer connection timeouts, with less regional failure tolerance than a valid multi-region data design. **Observed or calculated result.** A controlled replay with write traffic pinned to the database region completes 397 requests per second, reduces 95th-percentile checkout latency from 1,420 to 465 milliseconds, and reduces checkout timeouts from 3.7 percent to 0.3 percent. Image delivery remains below its previous 95th-percentile latency. **Bounded conclusion.** Regional alignment removed the reproduced cross-region database penalty under the tested load. The result does not establish regional disaster recovery or provider superiority. **Remaining uncertainty.** The test does not measure a regional outage, database failover, cross-region data recovery, long-duration cost, or client populations outside the two generator locations. ## Normal cases, failure cases, and qualifications A normal translation keeps a browser-facing frontend, stateless application capacity in more than one failure location, a managed PostgreSQL topology that matches the availability objective, and object storage with explicit access rules. Provider-native identities give application code narrowly scoped access without embedding long-lived access keys in source code. Failure and limiting cases require product-specific reading: - A global frontend can stay reachable while every configured origin or backend is unhealthy. - A zone-redundant frontend does not prove that the selected database tier or application plan is zone redundant. - Autoscaling can reach a subscription, project, regional processor, address, or database-connection quota. - Managed platforms reduce operating-system work but can restrict runtimes, networking, connection behavior, or startup time. - Provider health checks can mark a shallow route healthy while a required dependency fails. - Logs can be disabled, sampled, retained for too little time, or stored outside the required jurisdiction. - A product can exist in a provider catalog but be unavailable in the required region or excluded from a compliance authorization boundary. Current documentation must settle current product behavior. Google Cloud's load-balancing overview distinguishes global and regional load balancers and network tiers. Azure documentation distinguishes Front Door, Application Gateway, and Azure Load Balancer. Region and zone availability changes, so a 2026 table cannot serve as a permanent procurement fact. ## Common reasoning errors **“Cloud Run equals Lambda equals Azure Functions.”** Similar billing or execution characteristics do not establish equivalent event sources, maximum durations, concurrency, networking, identity, or regional availability. Start from the CampusCart execution requirement and verify each behavior. **“A project, account, and resource group are the same container.”** The three objects occupy different hierarchy levels and carry different billing, policy, quota, and lifecycle effects. **“Global load balancing makes the application global.”** A global traffic frontend does not replicate application state or PostgreSQL writes. **“Managed means the provider owns every failure.”** The provider operates more of the runtime, while the customer still owns application code, identities, data choices, configuration, capacity selections, and recovery validation. **“The lowest single test latency identifies the best cloud.”** A credible comparison controls geography, compute, database capacity, warm-up, request mix, network tier, and statistical interval. One sample establishes only one observed request. ## Knowledge check **Question.** CampusCart moves an AWS architecture diagram to Google Cloud by renaming an Amazon VPC to a Google Cloud VPC and each Availability Zone subnet to a Google Cloud zonal subnet. Which assumption is wrong? **Answer.** An Amazon VPC is regional and an Amazon VPC subnet belongs to an Availability Zone. A Google Cloud VPC network is global and a Google Cloud subnet is regional. The team must redesign and verify routing, policy, and failure scope rather than rename the boxes. ## Evidence exercises ### Check Choose the CampusCart request path and produce a three-provider table with one row for DNS, HTTP frontend, compute, PostgreSQL, object storage, workload identity, request evidence, and control-plane evidence. For every cell, include a direct official-document URL and a one-sentence scope qualification. Preserve the table as Markdown or comma-separated values. ### Practice Measure a public test endpoint or an approved sandbox from one client location. Collect at least 30 uncached connection samples and 30 reused-connection samples. Preserve the command or browser export, Coordinated Universal Time timestamps, DNS milliseconds, connection milliseconds, time to first byte, status, and response bytes. Submit the distributions and explain which differences are observed versus inferred. Do not compare provider quality unless equivalent workloads were controlled. ### Challenge Write a migration decision report for CampusCart. State requirements for 400 completed requests per second, 500-millisecond 95th-percentile checkout latency, two failure locations, PostgreSQL semantics, and object-image delivery. Compare one AWS, one Google Cloud, and one Azure design. Submit architecture diagrams, current region and quota evidence, a latency budget, an identity path, a failure test, and five unresolved product-specific questions. ## Authoritative sources and further reading ### Required reading - [Google Cloud resource hierarchy](https://docs.cloud.google.com/resource-manager/docs/cloud-platform-resource-hierarchy) explains organizations, folders, projects, product resources, parentage, and policy inheritance. - [Google Cloud Load Balancing overview](https://docs.cloud.google.com/load-balancing/docs/load-balancing-overview) distinguishes internal and external, global and regional, proxy and passthrough, and network-tier behavior. - [Azure Resource Manager overview](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/overview) explains Azure management scopes and resource groups. - [Azure load balancing and content delivery](https://learn.microsoft.com/en-us/azure/networking/load-balancer-content-delivery/) routes readers among Front Door, Application Gateway, and Azure Load Balancer instead of treating them as one product. ### Implementation and version-sensitive documentation - [Google Cloud global, regional, and zonal resources](https://cloud.google.com/compute/docs/regions-zones/global-regional-zonal-resources) is useful for current location scope; recheck the page during design. - [Azure regions and availability zones](https://learn.microsoft.com/en-us/azure/reliability/regions-overview) describes the Azure geographic hierarchy and current zone concepts. - [Azure subscription and service limits](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits) documents changing default and maximum quotas that can invalidate a scale plan. - [AWS Regions and Availability Zones](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions-availability-zones.html) supplies the AWS comparison baseline. ### Security and responsibility reference - [Google Cloud shared responsibility and shared fate](https://docs.cloud.google.com/architecture/framework/security/shared-responsibility-shared-fate) explains which security work remains with a customer across infrastructure, platform, software, and function offerings. - [Microsoft shared responsibility in the cloud](https://learn.microsoft.com/en-us/azure/security/fundamentals/shared-responsibility) distinguishes customer and Microsoft work by service category. ## Transition Product translation exposes a deeper common mechanism. Every provider must decide whether a principal may act, store bytes with stated protection and consistency, recover from failure, and charge for measured consumption. The next chapter studies identity, data protection, reliability, observability, and cost as connected operating mechanisms rather than independent product menus. --- # Identity, Data Protection, Reliability, and Cost ## Prerequisites and outcomes **Prerequisites.** Read the provider request-path chapters and the earlier lessons on public-key cryptography, Hypertext Transfer Protocol (HTTP), databases, and processes. The Amazon Web Services (AWS) chapter explains a Region, an Availability Zone, a role, an object-storage request, and a PostgreSQL transaction. The provider-comparison chapter explains why product names do not establish equivalent behavior. A direct-link reader should know that application code calls cloud application programming interfaces (APIs) and that those calls can succeed, fail authorization, time out, or incur measured usage. **After this chapter, you can:** - distinguish authentication from authorization and calculate effective access through allow, intersection, and explicit-deny rules; - explain how encryption, object versioning, replication, database standby capacity, and backups address different failure modes; - define recovery point objective and recovery time objective with units; - connect metrics, logs, audit events, and billing line items to the program that produces each artifact; and - create a monthly cost estimate whose assumptions can be tested against a provider bill. Examples use Amazon Web Services (AWS) names when exact policy or artifact behavior matters. Google Cloud and Microsoft Azure provide corresponding identity, key-management, audit, monitoring, backup, and billing capabilities, but their evaluation rules and schemas require provider-specific verification. Product interfaces and prices are current only when checked; this chapter's July 2026 date is not a price guarantee. ## A permit attachment becomes unreadable CivicPermit accepts government permit applications. A submitted application contains personally identifiable information (PII), a PostgreSQL row, and attachment objects such as site plans. The application runs across two Availability Zones. The operations team expects either zone to handle intake if the other zone becomes unavailable, and the agency requires an audit trail for administrative changes. During a controlled zone-failure exercise, application capacity in the second zone starts receiving traffic. New permit rows reach PostgreSQL, but attachment downloads return HTTP 500. The API retries each failed object read three times. Debug logging is temporarily enabled, so every attempt writes a large structured entry. The exercise lasts 40 minutes. The security engineer needs to answer four questions. Which principal attempted the read? Did an authorization rule deny the read? Did the attachment bytes still exist? How much recovery and diagnostic activity did the provider meter? A single “access denied” screenshot cannot answer all four questions. ## Records and measurements with named producers The CivicPermit API produces an application entry for each download attempt. Useful fields include a correlation identifier, permit identifier represented by a non-PII internal value, operation, workload-role name, object key, attempt number, duration in milliseconds, and error class. The application must exclude raw PII and tokens because centralizing secrets in diagnostic storage creates another disclosure path. AWS CloudTrail produces an event for supported account activity. A CloudTrail event can identify `eventTime`, `eventSource`, `eventName`, `awsRegion`, request parameters, principal information within `userIdentity`, source address, and an error code. AWS documents `eventTime` as the completion time reported by the host serving the API endpoint. Management events, data events, network activity events, and Insights events are distinct categories. Event history provides recent management events, but object-level data activity requires the relevant data-event collection. Therefore absence from default event history does not prove that an object read never happened. Amazon CloudWatch receives AWS-produced measurements and customer-published measurements. A measurement such as object request count, application error count, database connections, or log-ingestion bytes has a producer, unit, timestamp, and identifying dimensions or labels. A 5-minute average cannot establish a one-second spike. A counter of failed calls cannot establish which principal made them unless another artifact supplies identity. AWS Cost Explorer and billing exports derive line items from provider metering and pricing rules. Cost Explorer commonly shows usage through the previous day, and upstream billing data can arrive or change later. A current-month number is therefore an estimate, not an instantaneous packet counter. Resource tags can help allocate charges only when the provider supports the tag for that line item, the tag is activated where required, and the resource was tagged during the measured use. ## Identity as a request decision An operator signs in as a human identity. Application code should normally use a workload identity rather than copied human credentials. The identity system **authenticates** a principal by establishing which human or workload is making the request. The provider then **authorizes** a specific action against a specific target by evaluating request context and applicable policies. Authentication can succeed while authorization denies `kms:Decrypt` against one key. AWS Identity and Access Management (IAM) documentation provides a useful concrete policy mechanism. Within the applicable evaluation context, identity-based and resource-based permissions can contribute allows. A permissions boundary limits an identity policy through intersection. AWS Organizations service control policies can further limit permissions for organization members. An explicit deny that matches the request overrides an allow. An ordered request evaluation is safer than one set equation. First identify the exact principal, action, target, account relationship, request conditions, and policy types. Next check every applicable policy for an explicit deny; a matching explicit deny ends the request with denial. Without an explicit deny, locate an applicable allow and then determine whether a permissions boundary, session policy, or organization service control policy limits the principal under the documented case. Resource-based grants behave differently for an Identity and Access Management user Amazon Resource Name, role Amazon Resource Name, and role-session Amazon Resource Name, so a generic intersection formula would give some beginners the wrong result. AWS's current policy-evaluation and permissions-boundary documentation remains the authority for a real request. In CivicPermit, the zone-two workload obtains temporary credentials for `CivicPermitRuntimeRole-v2`. The object-storage policy permits reading the attachment. The encryption key policy names only the old role, `CivicPermitRuntimeRole-v1`. The object request can locate ciphertext while the decrypt operation is denied. “Object exists” and “principal may recover plaintext” are separate claims. Short-lived role credentials reduce the exposure of a copied secret because credentials expire and can be issued to a runtime identity. Temporary credentials do not repair an overbroad role. Least privilege means granting the actions and targets required for the workload under stated conditions, then testing both intended success and intended denial. ## Data protection addresses named failures Transport Layer Security (TLS) protects data moving between named endpoints while the connection uses an approved configuration. Storage encryption protects persisted bytes according to the chosen storage and key arrangement. Neither mechanism prevents an authorized application from writing incorrect content or returning PII to the wrong authenticated user. Envelope encryption supplies a common causal path. A key-management product protects a key-encryption key. The storage or application path uses a data key to encrypt attachment bytes, stores ciphertext plus required metadata, and protects the data key under the key-encryption key. A later read needs ciphertext, encryption metadata, and an authorized decrypt operation. Deleting or disabling the required key can make intact ciphertext unrecoverable. Key policy, rotation, backup, and deletion controls therefore belong in the data-recovery design. Amazon S3 currently provides strong read-after-write consistency for successful object `PUT` and `DELETE` operations in all AWS Regions. A successful write followed by a read returns the written object under the documented conditions. Consistency does not mean immortality. An authorized delete can remove the current object. Versioning can preserve prior versions, Object Lock can apply retention controls, and replication can copy objects according to configured rules. Each control addresses a different event. A database standby improves service continuity when the primary database instance or location fails. A standby is not an independent historical backup when application code corrupts every replicated row. A backup preserves an earlier recovery point. A restored backup can still be useless if nobody has tested credentials, schema compatibility, key access, network routes, and application startup. The **recovery point objective (RPO)** is the maximum acceptable time between the latest usable recovery point and the disruption. The unit is time, such as 5 minutes of accepted permit updates. The **recovery time objective (RTO)** is the maximum acceptable delay between disruption and restored capability, also measured in time. An RPO of 5 minutes does not promise zero lost records. An RTO of 30 minutes does not state how fresh the restored data will be. Both objectives come from mission impact and require a test. For CivicPermit, two Availability Zones address a zonal infrastructure failure only when intake capacity, routing, database behavior, key access, and dependencies work from either zone. Two instances placed in one zone do not meet the same failure condition. Two-zone replication also does not address a Region-wide disruption or a destructive credential used in both zones. ## Reliability needs observable targets Availability is a completed-capability fraction over a stated interval, not a product adjective. If CivicPermit defines one qualifying minute as available only when a resident can submit a valid permit and receive a durable receipt, then: \[ Availability = \frac{Qualifying\ available\ minutes}{Total\ qualifying\ minutes}. \] For a 30-day month containing 43,200 minutes, a 99.9-percent internal availability target permits 43.2 unavailable minutes under that definition. Maintenance exclusions, partial failures, and measurement locations must be stated. A provider service-level agreement can use a different formula and can specify eligibility conditions or service credits. An architecture should use the workload's mission measure rather than silently substituting a provider contract measure. Reliability evidence includes a request success ratio, latency distribution, healthy-target count, replication condition, backup completion, restore duration, and application-level receipt verification. The load balancer produces target health. The backup product produces a job status. The application produces a durable-receipt test. A successful backup job establishes that a backup artifact was created under the product's checks; only a restore exercise tests whether CivicPermit can use the artifact within its RTO. ## A monthly workload ledger Cloud cost begins as measured quantity multiplied by a price for that quantity. Use a consistent billing interval and preserve dimensions such as region, tier, operation class, purchase commitment, and data-transfer direction. The following ledger is an **illustrative teaching estimate**, not an AWS, Google Cloud, or Azure quote. Assumptions for one 730-hour month: - six application instances run continuously at an assumed $0.12 per instance-hour; - a two-zone managed PostgreSQL configuration is represented by an assumed $1.20 per database-hour; - attachments occupy 2,048 gigabyte-months at an assumed $0.023 per gigabyte-month; - Internet egress is 500 gigabytes at an assumed $0.09 per gigabyte; - diagnostic ingestion is 120 gigabytes at an assumed $0.50 per ingested gigabyte; and - taxes, support, free allowances, request charges, discounts, replicas, backup overage, inter-zone transfer, and tiered pricing are omitted. The estimate is: \[ Cost = Q_{compute}P_{compute}+Q_{db}P_{db}+Q_{storage}P_{storage} +Q_{egress}P_{egress}+Q_{logs}P_{logs}. \] | Item | Quantity | Assumed price | Calculated cost | |---|---:|---:|---:| | Application compute | 6 instances × 730 h = 4,380 instance-hours | $0.12/instance-hour | $525.60 | | Managed PostgreSQL | 730 database-hours | $1.20/database-hour | $876.00 | | Attachment storage | 2,048 GB-month | $0.023/GB-month | $47.10 | | Internet egress | 500 GB | $0.09/GB | $45.00 | | Diagnostic ingestion | 120 GB | $0.50/GB | $60.00 | | **Estimated total** | | | **$1,553.70** | The ledger predicts which quantities deserve attention. Cutting compute hours by 10 percent saves an estimated $52.56 under the assumptions. Cutting attachment storage by 10 percent saves about $4.71. The result does not imply that compute optimization is safe; six instances might be required for zone-failure capacity. Reliability and cost share the same configuration, so cost removal needs a failure and load test. ## Make the mechanisms visible Use a nonproduction identity, object, key, and database. Hold application build, object bytes, region, runtime placement, and policy documents constant. First allow the workload identity to store and retrieve one uniquely hashed test object. Preserve the object checksum, audit event, application correlation identifier, and request result. Then add a narrowly scoped deny for decrypting only the test key, wait for the documented policy propagation behavior, and repeat the read. Expected evidence shows successful authentication, a denied decrypt action, intact ciphertext, and an application error. Remove the test deny through the approved change path and confirm recovery. A result in which plaintext remains readable would reject the claim that the selected key controls that object's read path. Alternative explanations include cached plaintext, a different key, an unlogged data-event category, a policy that did not match the request, or application access through another principal. Extend the test by deleting the current version of a disposable versioned object and restoring the prior version. Record recovery-point age in minutes and restoration time in minutes. Compare both observations with stated RPO and RTO. Never perform the exercise against production PII. ## Worked investigation: denied decrypts and diagnostic cost **Initial question.** Why did attachment downloads fail only after zone-two takeover, and why did diagnostic ingestion increase during the exercise? **Operating conditions.** CivicPermit ran across two Availability Zones. Zone two started revision `v2`, which assumed `CivicPermitRuntimeRole-v2`. Each download attempted one object read plus as many as three retries. Debug entries averaged 18 kilobytes per attempt. The exercise sent 25 attachment downloads per second for 40 minutes. **Evidence collected.** Application entries showed four attempts per user download and a key-decrypt authorization error. CloudTrail data collection showed object `GET` attempts against the expected key, while key-management events showed `AccessDenied` for the `v2` role. Object inventory and checksums showed that ciphertext remained. Deployment history identified the role-name change. Monitoring showed no database saturation. Billing usage later reported increased request and log-ingestion quantities. **Interpretation.** The deployment changed the principal identifier without changing the key policy. Retries repeated a deterministic authorization failure. Debug entries multiplied the metered log bytes without improving availability. **Proposed intervention and prediction.** Use a stable workload role referenced through infrastructure configuration, validate decrypt permission from both zones before shifting traffic, stop retries for deterministic authorization denial, and return a bounded application error without PII. The prediction is one denied attempt during a forced-denial test, followed by a successful read after the policy repair, with at least 75 percent fewer failure entries than the four-attempt behavior. **Observed or calculated result.** The original exercise produced \(25 \times 2{,}400 \times 4 = 240{,}000\) attempt entries. At 18 kilobytes each, the application produced about 4.32 decimal gigabytes before provider framing and metadata. The repaired replay produced one entry per denied download, about 1.08 decimal gigabytes under the same simplifying assumptions, and successful reads resumed after the approved key-policy change. **Bounded conclusion.** A mismatched key policy caused the reproduced zone-two read failures. Removing pointless retries reduced application-produced diagnostic bytes by 75 percent for the controlled failure path. **Remaining uncertainty.** The calculation omits provider-added log bytes, compression, ingestion pricing tiers, policy propagation delay under other conditions, and a full regional recovery. The exercise also does not establish that production audit retention meets agency obligations. ## Normal cases, edge cases, and qualifications A normal request uses a temporary workload credential, obtains an allow for the exact object and decrypt action, reads ciphertext, recovers plaintext in the authorized path, and returns a response. Audit collection preserves the supported activity category. Monitoring aggregates operational measurements. Billing later attributes measured quantities according to the provider's rules. Important limiting cases include: - An explicit deny can override a broad allow. More attached policies do not necessarily create more access. - Cross-account or cross-project access can require permission on both sides of the administrative boundary. - Policy changes can take time to propagate. Immediate success or failure during a change needs a documented propagation qualification. - Replication can faithfully copy malicious deletion or corrupt application writes unless retention or historical recovery supplies another point. - Encryption can protect stored bytes while logs, caches, snapshots, or exports expose PII through separate paths. - A successful failover can exceed the RTO. An automatic action is not necessarily a timely action. - Current-month cost can change after delayed metering, credits, refunds, or invoice adjustments. - Tags do not allocate untaggable shared charges automatically. ## Common reasoning errors **“Authenticated means authorized.”** Authentication names a principal. Authorization evaluates one requested action, target, and context. **“Encrypted means backed up.”** Encryption changes readable plaintext into protected ciphertext. A backup preserves recoverable content at another recovery point. Losing the only ciphertext or required key still loses access. **“Multi-zone means zero downtime.”** Zonal placement reduces one class of dependency. Detection, routing, capacity, database behavior, credentials, and application recovery still consume time and can fail. **“Eleven nines of object durability means every object is always available.”** Durability concerns loss over time under the provider's design objective. Availability concerns successful access during an interval. Authorized deletion, policy denial, regional access failure, and application defects are separate events. **“The bill equals instances multiplied by hours.”** Database, storage, operations, data transfer, monitoring, addresses, support, commitments, and taxes can contribute. Each charge needs a provider meter and a pricing dimension. ## Knowledge check **Question.** A backup job reports success every hour, and CivicPermit has a 15-minute RPO. Does the job schedule prove the objective? **Answer.** No. An hourly backup schedule can already exceed a 15-minute RPO unless another recovery mechanism supplies more frequent usable points. Job success also does not prove restorability. Measure the age of the latest usable recovery point during a failure and test a restore, including keys, configuration, dependencies, and application verification. ## Evidence exercises ### Check Write a six-row authorization table for one CivicPermit attachment. Include principal, action, target, identity allow, resource allow, limiting policy, explicit deny, and expected decision. Submit the table plus two test requests: one intended allow and one intended deny. Preserve sanitized audit evidence for both. ### Practice Create a disposable versioned object in an approved sandbox or reproduce the mechanism with a local versioned store. Save a checksum, overwrite the object, remove the current version, and recover the original. Submit commands, artifact producer, timestamps, version identifiers, checksums, recovery-point age, restoration duration, and a statement of what the test omits. ### Challenge Build a monthly workload ledger for CampusCart or CivicPermit from an official current calculator and pricing pages. Include compute hours, database topology, gigabyte-months, request operations, backup storage, data transfer by direction, and diagnostic ingestion. Preserve the calculator date, region, tier, currency, formulas, and exported estimate. Compare the estimate with one later billing export and explain every material variance without including account secrets or PII. ## Authoritative sources and further reading ### Required reading - [AWS IAM policy evaluation logic](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html) documents current unions, intersections, and explicit-deny behavior that qualify the simplified set expression. - [Amazon S3 data consistency model](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel) distinguishes strong read-after-write behavior from durability and recovery controls. - [AWS Well-Architected recovery objectives](https://docs.aws.amazon.com/wellarchitected/latest/framework/rel_planning_for_recovery_objective_defined_recovery.html) gives the authoritative AWS meanings of recovery time objective and recovery point objective. ### Implementation and artifact documentation - [CloudTrail event record contents](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-record-contents.html) identifies event fields, timestamp origin, optional values, and error fields. - [Understanding CloudTrail events](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-events.html) distinguishes management, data, network activity, and Insights events and warns that categories differ from default collection. - [Amazon S3 data protection](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html) explains versioning, Object Lock, replication, durability, and availability as separate mechanisms. - [AWS Cost Explorer](https://docs.aws.amazon.com/cost-management/latest/userguide/ce-exploring-data.html) explains estimate freshness and cost views; prices still require current pricing pages or a calculator. ### Cross-provider reference - [Google Cloud shared responsibility and shared fate](https://docs.cloud.google.com/architecture/framework/security/shared-responsibility-shared-fate) explains retained customer duties for access and data. - [Microsoft shared responsibility](https://learn.microsoft.com/en-us/azure/security/fundamentals/shared-responsibility) explains retained customer duties across infrastructure, platform, and software offerings. - [Azure shared responsibility for reliability](https://learn.microsoft.com/en-us/azure/reliability/concept-shared-responsibility) explains customer responsibility for tier choice, backups, application resilience, and service-level-agreement conditions. ## Transition CivicPermit's technical controls create evidence that an authorization and risk decision can inspect. Government work adds institutional requirements: information categorization, control selection, assessment, authorization, continuous monitoring, personnel restrictions, and contract scope. The next chapter explains how AWS GovCloud (US), provider certifications, and inherited controls can support that work without automatically making an agency workload compliant. --- # AWS for United States Government Work ## Prerequisites and outcomes **Prerequisites.** Read the chapters on the Amazon Web Services (AWS) request path, provider comparison, and identity, data protection, reliability, and cost. The identity lesson explains workload principals, audit events, encryption keys, recovery point objective (RPO), and recovery time objective (RTO). A direct-link reader should know that a cloud provider operates some controls while a customer configures and operates other controls. **After this chapter, you can:** - explain what AWS GovCloud (US) Regions change and what they do not change; - distinguish Federal Risk and Authorization Management Program (FedRAMP) evidence for a cloud offering from an agency authorization to operate for an information system; - connect information categorization, control responsibility, implementation evidence, assessment, authorization, and continuous monitoring; - calculate audit-storage volume from event counts and record sizes; and - reject claims that a Region, certification, cryptographic endpoint, or product name automatically makes a workload compliant. The lesson describes engineering and evidence practices, not legal advice. Statutes, regulations, contracts, agency policy, data categories, authorization packages, and product scope change. Counsel, contracting officials, privacy officials, security officers, data owners, assessors, and the agency authorizing official make decisions within their authority. Version-sensitive statements in this lesson were checked against official sources in **July 2026**. ## A premature compliance claim CivicPermit accepts permit applications containing personally identifiable information (PII), address data, site-plan attachments, payment references, and staff decisions. The agency requires auditable administrative changes and continued intake after failure of one Availability Zone. A contractor proposes an Application Load Balancer, application capacity in two zones, Amazon Relational Database Service (Amazon RDS) for PostgreSQL, Amazon Simple Storage Service (Amazon S3), AWS Key Management Service (AWS KMS), and AWS CloudTrail. A design review slide says, “Deployment in AWS GovCloud (US) is FedRAMP High compliant.” The sentence joins three separate decisions. AWS has provider-operated offerings and controls within defined authorization boundaries. The contractor builds and configures a particular CivicPermit workload. An agency authorizing official decides whether to accept risk for the agency information system and its use of the cloud offering. The reviewer asks for evidence. The team must identify CivicPermit's information, authorization boundary, impact category, selected services and features, control owners, data flows, access rules, audit coverage, contingency tests, assessment results, and unresolved risks. A Region name supplies useful platform facts, but the name cannot supply those workload records. ## Evidence that exists before an authorization decision An information inventory is produced by the agency data owner and CivicPermit team during design and updated when data flows change. Each row should name an information type, source, destination, purpose, sensitivity or regulatory handling basis, retention, and approved recipients. A row directly establishes the team's declared handling design. The row does not prove that production traffic follows the design. A data-flow diagram is produced by the architecture team from deployed routes, interfaces, Domain Name System (DNS) names, application behavior, and interviews. Each arrow must name both endpoints, protocol, data category, authentication method, encryption arrangement, and artifact producer. A line from the browser to “cloud” hides the customer-to-provider boundary, the Internet path, and the exact termination endpoint. An infrastructure inventory is produced from provider application programming interfaces (APIs) or infrastructure-as-code evaluation. The inventory should include account, partition, Region, Availability Zone where applicable, service, feature, Amazon Resource Name (ARN), encryption setting, network attachment, tags, and creation evidence. AWS GovCloud (US) ARNs begin with `arn:aws-us-gov`, while standard AWS partition ARNs begin with `arn:aws`. The different string changes policies, templates, and integrations that previously assumed `arn:aws`. AWS supplies certification and compliance artifacts for provider-controlled offerings through the applicable FedRAMP process and AWS channels such as AWS Artifact. The current AWS “services in scope” page lists services inside particular assurance-program boundaries and includes an update date. A check mark for one service and boundary is evidence about that provider offering. The mark does not establish that every feature, third-party product, customer configuration, or external connection in CivicPermit belongs to the same boundary. The customer produces implementation evidence. Examples include identity policies, key policies, configuration snapshots, change approvals, training records, incident exercises, vulnerability findings, backup reports, restore results, access reviews, CloudTrail events, application audit entries, and plans of action and milestones. Each artifact needs a producer, collection interval, integrity protection, retention, and link to the control implementation. A screenshot without account, time, query, and source cannot support repeatable assessment. ## What AWS GovCloud (US) changes AWS GovCloud (US) consists of isolated AWS Regions intended for sensitive government-related and regulated workloads. The Region codes are `us-gov-west-1` and `us-gov-east-1`. AWS documentation describes physical and logical separation, GovCloud-specific endpoints, distinct credentials, and administration of the AWS boundary by vetted United States citizens under the documented conditions. AWS GovCloud documentation also describes use for Controlled Unclassified Information (CUI) and unclassified data. The word **partition** names an AWS administrative and naming separation larger than one Region. AWS GovCloud resources use the `aws-us-gov` partition. Standard AWS credentials cannot access AWS GovCloud resources, and GovCloud credentials cannot access standard AWS resources. A GovCloud account is associated one-to-one with a standard AWS account for billing, support, and account purposes, but the credentials remain distinct. An AWS Organization in the standard partition is separate from an AWS Organization created in the GovCloud partition. The separate partition affects automation. CivicPermit templates must use GovCloud Region codes, endpoints, ARNs, service principals, certificate options, and product availability. A continuous-integration runner in the standard partition does not gain GovCloud access because both accounts appear on one invoice. The team must create an approved identity path and network path, then verify that build artifacts and diagnostic content remain within allowed locations. Federal Information Processing Standards (FIPS) endpoints expose provider endpoints using cryptographic modules under the documented validation context. Endpoint selection does not prove that CivicPermit's browser code, third-party library, local file handling, database client, or export job uses an approved module correctly. The National Institute of Standards and Technology (NIST) Cryptographic Module Validation Program explicitly warns that module validation does not assure that a product correctly uses the embedded module. Product availability also differs. A service available in standard AWS can be absent in GovCloud, present with fewer features, or documented with fields that must not contain export-controlled content. Current service documentation must be checked before selection. AWS support documentation, for example, warns users not to put export-controlled data in support cases because support-case handling has its own boundary considerations. ## The institutional mechanism from data to authorization Federal Information Processing Standards Publication 199 (FIPS 199) categorizes federal information and information systems according to potential impact on confidentiality, integrity, and availability. The responsible organization identifies information types and analyzes mission harm. Choosing GovCloud first cannot replace categorization because the category and governing requirements determine which protections and authorization path are necessary. NIST Special Publication (SP) 800-37 Revision 2 describes the Risk Management Framework (RMF) as a lifecycle: prepare, categorize, select, implement, assess, authorize, and monitor. NIST SP 800-53 Revision 5 supplies a catalog of security and privacy controls. Control baselines and tailoring decisions come from the applicable framework and organizational requirements, not from copying every catalog entry into a spreadsheet. During **selection**, the team states the required control outcome and planned responsibility. During **implementation**, AWS may provide an inherited control, CivicPermit may provide a customer control, or both parties may implement different portions of a shared control. During **assessment**, an assessor examines evidence and tests whether the implementation satisfies the selected requirement. During **authorization**, the authorizing official uses the security plan, assessment, remediation information, mission need, and risk analysis to make a risk decision. During **monitoring**, provider and customer evidence supports continued awareness and future decisions. FedRAMP standardizes reusable security assessment, certification, and continuous-monitoring evidence for cloud service offerings used by federal agencies. Current 2026 FedRAMP guidance distinguishes the certified cloud service offering from the agency information system that uses the offering. The agency still evaluates its data, configuration, integrations, customer-operated controls, and risk. An agency authorization to operate (ATO) therefore applies to the agency information system and its use of external cloud capabilities; provider certification is evidence for that decision, not a transfer of the decision. ## Shared responsibility as a control statement AWS describes security **of** the cloud as AWS responsibility for the infrastructure that runs AWS products. Security **in** the cloud remains with the customer according to the selected product. Amazon Elastic Compute Cloud (Amazon EC2) customers patch and configure guest operating systems and applications. More abstracted products move operating-system work to AWS, while customers still classify data, manage identities, configure access, secure application code, choose retention, and verify recovery. A useful control statement names five fields: 1. **Required outcome:** what protection or evidence must exist. 2. **Provider implementation:** which AWS capability or inherited control contributes. 3. **Customer implementation:** which CivicPermit configuration or procedure contributes. 4. **Evidence:** which artifact, producer, interval, and query demonstrate operation. 5. **Gap owner:** who must resolve a missing or ineffective portion. For audit accountability, AWS can operate CloudTrail infrastructure and document provider controls. CivicPermit must select relevant event categories, protect destinations, avoid PII leakage, retain evidence for the required period, review alerts, synchronize application identifiers, and test retrieval. Calling the entire audit control “inherited” would hide customer work. Calling the entire control “customer-owned” would ignore provider-operated infrastructure and reusable assessment evidence. Certification scope also has levels. The selected AWS service must be in the relevant program boundary. The selected feature and Region must match current scope. CivicPermit must configure the feature correctly. External software-as-a-service telemetry, contractor laptops, identity providers, code repositories, payment processors, and support workflows can cross the authorization boundary. A complete data-flow and responsibility analysis must include those connections. ## A storage model for audit evidence CivicPermit estimates audit storage before setting retention. The team forecasts 12,000 management events per day at an average serialized size of 1.8 kilobytes and 600,000 application or data events per day at 2.2 kilobytes. The retention requirement used for the exercise is 365 days. The team keeps two protected copies and adds 20 percent for indexes and metadata. Using decimal units for the estimate: \[ \begin{aligned} \text{DailyBytes} &=(12{,}000\times1.8\ \text{kB})+(600{,}000\times2.2\ \text{kB}) \\ &=1{,}341.6\ \text{MB/day}. \end{aligned} \] \[ \begin{aligned} \text{StoredBytes} &=1{,}341.6\ \text{MB/day}\times365\ \text{days}\times2\times1.20 \\ &=1{,}175{,}241.6\ \text{MB}, \end{aligned} \] or about 1.18 decimal terabytes. The event rate means completed serialized events divided by the collection interval in seconds or days; the model uses daily completed-event counts. The calculation assumes constant volume, stated average serialized sizes, two full copies, no compression, and 20-percent overhead. The result predicts initial capacity and a cost input. The result omits query scans, archive retrieval, growth, burst delivery, provider framing, cryptographic metadata, retention locks, backups, duplicates, and failed delivery. Measured exports must replace guessed averages. Compliance cannot be reduced to retaining 1.18 terabytes because audit content, access, integrity, review, and response matter as much as volume. ## Test the evidence path Use an authorized nonproduction GovCloud account with synthetic data. Hold the trail configuration, destination, identity, Region, and query constant. Perform one labeled management action and one labeled object data action. Preserve the command, synthetic resource identifier, Coordinated Universal Time interval, request identifier, principal, source endpoint, CloudTrail configuration, delivered object checksum, and query result. Expected evidence shows the management event in the configured collection and the data event only when the relevant data-event selector is enabled. Disable only the nonproduction selector through the approved change method, repeat the data action, and verify the documented difference. A result in which the event remains can reject the selector explanation, or it can indicate another trail, event data store, duplicate delivery, or delayed record. Restore the configuration and verify current state. The test makes collection visible. The test does not establish every selected control, full retention, incident response, or legal sufficiency. A separate restore or retrieval exercise should measure how long an authorized investigator needs to find a one-year-old event, validate integrity, and relate the event to an application action. ## Worked investigation: can CivicPermit use the claimed boundary? **Initial question.** Does the proposed GovCloud architecture provide enough scoped implementation and evidence for CivicPermit to enter formal assessment? **Operating conditions.** CivicPermit handles agency-defined moderate-impact PII in this scenario. The design uses two Availability Zones in one GovCloud Region, an Application Load Balancer, application instances, Amazon RDS for PostgreSQL, S3, KMS, and CloudTrail. A third-party error-reporting agent sends diagnostic payloads to a commercial software-as-a-service endpoint outside GovCloud. **Evidence collected.** The team obtains the current AWS services-in-scope listing and applicable authorization package through approved channels. Infrastructure inventory confirms `aws-us-gov` ARNs and GovCloud endpoints. Data-flow testing shows that the error agent sends stack context outside the proposed boundary. Identity review finds broad administrator access for contractor accounts. Audit testing finds management events but no configured S3 object data events. A restore test succeeds in 47 minutes against a 60-minute RTO. A 28-row scoped responsibility matrix marks 9 outcomes as provider-inherited, 11 as shared, and 8 as customer-operated for this review sample. **Interpretation.** The named AWS products can contribute relevant authorized capabilities, but CivicPermit has an external diagnostic flow and missing customer implementations. Two-zone placement and an in-scope service list do not resolve the gaps. **Proposed intervention and prediction.** Remove or reconfigure external diagnostics so approved sanitized evidence remains in the assessed boundary. Replace standing broad access with federated, role-based, time-bounded administration. Enable and validate required object data-event collection. Attach artifacts to every matrix row and run an incident exercise. The prediction is that assessors can reproduce 27 of 28 sampled outcomes, with the remaining incident-response exercise explicitly scheduled rather than silently marked complete. **Observed or calculated result.** After remediation, controlled network testing finds no connection to the former telemetry endpoint. Intended object actions appear in the configured audit destination with correlated application identifiers. Access tests show that ordinary contractor roles receive a denial for administrator actions. Twenty-seven matrix rows contain current reproducible evidence; one row remains open for the scheduled exercise. **Bounded conclusion.** The remediated sample is better prepared for formal assessment and no longer has the reproduced external diagnostic flow. The evidence does **not** declare CivicPermit compliant, FedRAMP certified, or authorized to operate. Only the designated assessment and authorization process can reach the applicable decisions. **Remaining uncertainty.** The sample does not cover every control, product feature, supply-chain dependency, privacy requirement, continuous-monitoring interval, contract term, or later configuration change. The agency must evaluate the open item and the complete authorization package. ## Normal operation, edge cases, and qualifications A normal government-cloud deployment starts with categorized information and an explicit authorization boundary. The team selects services and features in the required Region and current program scope, maps provider and customer responsibilities, configures identities and evidence collection, tests normal and failure behavior, supplies assessment evidence, resolves findings, and supports an authorizing official's risk decision. Important edge cases include: - PII does not by itself prove that GovCloud is legally required or sufficient. Agency requirements, impact, contracts, and applicable law determine the handling design. - GovCloud eligibility and U.S.-person requirements do not automatically enforce the customer's workforce access decisions. The customer controls customer identities and content access. - A service listed in GovCloud can have different features from the standard partition. A service available in GovCloud can still be outside a particular assurance-program boundary. - A FedRAMP High provider boundary does not convert an incorrectly configured customer workload into a compliant workload. - A FIPS endpoint does not validate every cryptographic path or correct module use. - Two Availability Zones do not provide Region-wide disaster recovery. The two GovCloud Regions require explicit replication, routing, recovery procedures, and testing where a cross-Region objective applies. - Resource names, log messages, support cases, and metadata fields can have handling restrictions even when payload storage is approved. - An ATO is a risk decision with scope and conditions. An ATO is not a guarantee that no incident will occur. ## Common reasoning errors **“GovCloud equals compliance.”** GovCloud supplies documented platform characteristics and eligible offerings. CivicPermit still needs correct scope, configuration, customer controls, assessment, authorization, and monitoring. **“AWS has a certification, so every AWS product is covered.”** Assurance boundaries list specific offerings, Regions, and current status. Features and external dependencies require individual verification. **“Shared responsibility means every failed control belongs partly to AWS.”** A control can be inherited, shared through distinct implementations, or customer-operated. The responsibility matrix must name the exact outcome and evidence. **“Encryption at rest satisfies all PII requirements.”** PII handling also involves collection, use, access, disclosure, retention, deletion, audit, incident response, privacy analysis, and external flows. **“An audit log proves continuous monitoring.”** Continuous monitoring requires defined collection, review, analysis, response, reporting, and update intervals. Unread events in storage supply potential evidence, not completed monitoring work. ## Knowledge check **Question.** CivicPermit uses only AWS services that show a current check mark under FedRAMP High in the AWS scope table. Can the project manager state that CivicPermit has an ATO? **Answer.** No. The check marks provide scoped provider-offering evidence. The agency must evaluate CivicPermit's information, configuration, external connections, customer controls, assessment findings, and risk. The authorizing official issues or withholds the agency information system's ATO through the applicable process. ## Evidence exercises ### Check Create a ten-row responsibility matrix for CivicPermit covering identity lifecycle, privileged access, encryption key control, object audit events, application audit entries, vulnerability response, backup, restore, incident response, and personnel training. For each row, name provider implementation, customer implementation, artifact producer, collection interval, assessor test, and gap owner. Submit the matrix and mark unknowns explicitly. ### Practice Build a synthetic data-flow diagram for one permit submission. Include the resident browser, DNS, Transport Layer Security (TLS) endpoint, application target, PostgreSQL, object storage, key management, audit destination, administrator workstation, code pipeline, support path, and every external product. Submit the diagram plus a table of protocols, data types, partitions, Regions, principals, and evidence. Highlight any flow that leaves the proposed authorization boundary. ### Challenge Use official July 2026 sources to prepare a mock authorization evidence index. Base the index on FIPS 199 categorization reasoning and 15 selected NIST SP 800-53 control outcomes. Link each outcome to provider package evidence or customer-produced evidence without copying restricted package content. Include one normal test, one intended-denial test, one zone-failure test, one restore test, and one one-year audit-retrieval plan. Submit unresolved risks and never label the mock workload compliant or authorized. ## Authoritative sources and further reading ### Required government sources - [FIPS 199: Security Categorization](https://csrc.nist.gov/pubs/fips/199/final) establishes impact-based categorization for confidentiality, integrity, and availability. - [NIST SP 800-37 Revision 2](https://csrc.nist.gov/pubs/sp/800/37/r2/final) defines the Risk Management Framework lifecycle and the roles of assessment, authorization, and continuous monitoring. - [NIST SP 800-53 Revision 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final) supplies the security and privacy control catalog; the NIST page also identifies current release updates. - [FedRAMP 2026: Using Rev5 Certification Packages](https://www.fedramp.gov/2026/agencies/use/packages/rev5/) distinguishes reusable provider certification evidence from an agency's authorization of its information system. ### AWS GovCloud implementation documentation - [What is AWS GovCloud (US)?](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/whatis.html) documents intended workload categories, U.S.-citizen administration of the AWS boundary, and shared responsibility. - [AWS GovCloud compared with standard Regions](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/govcloud-differences.html) identifies partition isolation, credentials, endpoints, billing association, and feature differences. - [GovCloud Amazon Resource Names](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/using-govcloud-arns.html) documents `arn:aws-us-gov` and the two Region codes. - [GovCloud service endpoints](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/using-govcloud-endpoints.html) documents current endpoint selection and FIPS qualifications; revisit the page before implementation. - [Services in AWS GovCloud Regions](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/using-services.html) lists product-specific availability, differences, and export-controlled-content warnings. ### Compliance and cryptographic scope - [AWS services in scope for FedRAMP](https://aws.amazon.com/compliance/services-in-scope/FedRAMP/) provides a dated, version-sensitive table of service status within specific provider boundaries. - [AWS shared responsibility model](https://aws.amazon.com/compliance/shared-responsibility-model/) distinguishes provider infrastructure work from customer configuration, data, identity, and application work. - [NIST Cryptographic Module Validation Program](https://csrc.nist.gov/Projects/Cryptographic-Module-Validation-Program) provides current validation status and transition information. - [NIST CMVP frequently asked questions](https://csrc.nist.gov/Projects/Cryptographic-Module-Validation-Program/FAQs) explains why module validation does not assure correct product use of a validated module. ## Transition Part II has followed a request from DNS to application and data, translated the design across three providers, connected operations to identity and recovery, and placed government-cloud choices inside an authorization process. The next part can now introduce containers and Kubernetes without presenting them as an alternative to these foundations. A container still uses provider identity, networks, storage, evidence, capacity, cost meters, and an authorization boundary. --- # Containers and Docker Maya, a new CampusCart developer, receives a bug report on Monday morning. The bookstore application formats a receipt correctly on her macOS laptop but fails in the Linux test environment because a native image library is missing. Registration week starts in four days. Maya needs a repeatable way to move the application code, its language runtime, and its operating-system libraries from a developer machine to a cloud machine. Maya creates a Dockerfile in the CampusCart source repository. A continuous-integration runner reads that file, downloads a named base image, installs locked dependencies, copies the application, and writes a content-addressed image to a registry. A test machine downloads the same image and starts a container. The receipt now has the same native library in both environments, although the macOS laptop and Linux test machine still use different kernels. The incident gives Maya concrete objects to inspect: a Dockerfile produced by a developer, image layers produced by the builder, a manifest stored by the registry, a Linux process started by the container runtime, and log bytes written by that process. Containers solve a packaging and execution problem. Containers do not create a separate computer, do not preserve data automatically, and do not remove the need to understand networking or operating-system limits. ## Prerequisites and earlier lessons - Chapter 3, **Networking for Cloud Applications**, supplies Internet Protocol addresses, ports, Domain Name System resolution, and request paths. - Chapter 4, **Virtualization and Cloud Service Models**, distinguishes a virtual machine from a physical host. - The reader should understand a process, a filesystem, an environment variable, and a software dependency. ## Learning outcomes After completing the chapter, the reader can distinguish an image from a container, explain how Linux namespaces and control groups affect a containerized process, predict which bytes survive container replacement, inspect image and runtime evidence, and decide when a container is a useful deployment unit. The reader can also explain why a container is not a security boundary equivalent to a separate physical machine. ## Observable evidence The command `docker image inspect campuscart-api:1.0` asks the Docker Engine daemon for image metadata. The daemon returns JavaScript Object Notation (JSON) fields that name the image identifier, architecture, operating system, configuration, and ordered root-filesystem layers. A registry and a local daemon identify image content by cryptographic digest. Matching digests directly establish matching content; a shared tag such as `latest` does not establish matching content because an operator can move the tag. The command `docker container inspect campuscart-api` asks the daemon for a running container description. Fields such as `State.Pid`, `NetworkSettings.Networks`, `Mounts`, and `Config.Image` report a host process identifier, network attachment, mounted storage, and source image. The Docker daemon produces the description from runtime metadata. The description establishes the daemon's configured view. A separate command such as `ps` on the Linux host can confirm that the named process exists. The command `docker logs campuscart-api` retrieves bytes captured by the configured logging driver from the container's standard output and standard error streams. The application produces those bytes. Docker transports or stores them. A line with Hypertext Transfer Protocol (HTTP) status 500 establishes that the application wrote that line; the line alone does not prove which request failed unless the application included a request identifier and the operator correlates the identifier with other evidence. ## Mechanism and terminology in context An **image** is an immutable, ordered description of filesystem layers plus execution configuration. A **container** is a running or stopped instance created from an image. The distinction matters because a registry stores images while a runtime creates containers. Replacing a container can preserve the image and remove the container's writable layer. On Linux, the runtime asks the kernel to create namespaces for selected views of operating-system objects. A process identifier namespace gives the containerized program a process-number view. A mount namespace gives the program a filesystem-mount view. A network namespace gives the program network interfaces, routes, and ports that differ from the host view. Namespaces change visibility; namespaces do not emulate a new kernel. Linux **control groups**, commonly written as cgroups, account for and constrain processor time, memory, and other quantities for groups of processes. A memory limit can cause an out-of-memory termination when the CampusCart process requests more memory than the cgroup permits. A processor quota can cause the kernel scheduler to withhold processor time after the process consumes its allowed share. The application still competes for the physical host's processors and memory buses. Image layers use a content-addressed, stackable representation. Docker storage drivers present read-only image layers and a thin writable container layer as one filesystem view. A modification to an existing image file can cause a copy-on-write operation: the storage driver copies the file into the writable layer before changing the copy. A volume mounts storage outside the disposable writable layer. Therefore, a PostgreSQL database that writes durable rows belongs on managed database storage or a deliberately managed volume, not only in the container layer. Container networking commonly attaches a network namespace to a virtual bridge. The host forwards packets between a virtual interface and the outside network, sometimes translating addresses. Publishing host port 8080 to container port 3000 creates a forwarding rule; the application still listens on port 3000 inside its network namespace. A cloud load balancer may send packets to the host or directly to a container-aware target, depending on the platform. Docker is a product and toolchain that builds, distributes, and runs containers. The Open Container Initiative (OCI) publishes image and runtime specifications. A Kubernetes kubelet calls a runtime service through version 1 of the Container Runtime Interface (CRI); runtime services such as containerd or CRI-O can then use OCI image and runtime conventions. OCI compatibility alone does not implement the CRI. Kubernetes therefore does not require Docker Engine on each node. Product names and interfaces must remain separate from the underlying image and process mechanisms. ## Working model: startup transfer time Suppose the CampusCart image contains five compressed layers. A target node already has four layer digests and lacks a 180-megabyte application layer. The registry-to-node path sustains 60 megabytes per second, and decompression plus filesystem preparation consumes 1.4 seconds. A first approximation is: `startup preparation time = missing compressed bytes / transfer throughput + local preparation time` The predicted preparation time is `180 MB / 60 MB/s + 1.4 s = 4.4 s`. The numerator counts only missing compressed bytes because matching layer digests allow reuse. The denominator measures completed registry bytes divided by transfer seconds. The model assumes stable throughput, no registry throttling, enough disk space, and no concurrent pulls. The model omits Domain Name System lookup, Transport Layer Security negotiation, image signature checks, queueing, and application initialization. The omitted lookup, negotiation, verification, queueing, and initialization times matter when the image is small or the registry path is congested. Layer reuse changes the prediction. A careless Dockerfile that installs dependencies after copying frequently changed source can invalidate a large layer on every build. Moving locked dependency installation before source copy can keep the dependency layer reusable. Layer order affects transfer work, but excessive layering can add metadata and make builds harder to understand. ## Empirical test Build two images from the same CampusCart commit. Image A copies source before installing dependencies. Image B installs dependencies from the lockfile before copying source. Prepare a test node whose cache contains only the recorded base and dependency layer digests. Change one source comment without changing dependencies, rebuild both images, remove the rebuilt image reference without pruning cached content, and pull again. Keep the recorded cache baseline, base image, registry, network path, and node constant. Record `docker history`, image digests, bytes transferred, and elapsed pull time for each rebuild. The mechanism predicts that Image B transfers fewer bytes because the dependency layer digest remains present. A result with identical missing bytes would reject the proposed cache benefit for this build, perhaps because the build tool included changing metadata or because both dependency layers were invalidated. Network congestion and registry-side compression are alternative explanations for elapsed-time differences, so transferred bytes provide stronger evidence than time alone. ## Worked investigation **Initial question.** Why did one of four CampusCart application containers lose uploaded product images after a restart? **Operating conditions.** An Amazon Elastic Container Service (Amazon ECS) service runs four tasks from image digest `sha256:7a…` behind one load balancer. Each task contains one application container that receives product-image uploads at `/app/uploads`. One container exceeded a 512-megabyte memory limit, and the ECS service replaced its stopped task at 14:07 Coordinated Universal Time. Object storage is configured for catalog thumbnails but not for raw uploads. **Evidence collected.** The Amazon ECS control plane produces a service event and stopped-task description that report the terminated task and its replacement. A node diagnostic collector preserved `docker inspect` output for the stopped container before the cleanup policy removed its metadata; the preserved output lists no volume at `/app/uploads`. The image manifest contains no uploaded files. Load-balancer access logs show three successful uploads routed to the old container before 14:07. The replacement container returns HTTP status 404 for those paths. **Interpretation.** Direct observations establish a replacement and the absence of an external mount. The image manifest establishes the original immutable content, not later writes. The bounded inference is that uploads entered the old container's writable layer and disappeared when that layer was removed. The evidence does not establish whether a backup copied the files elsewhere. **Proposed intervention and prediction.** Change the upload path so the application writes each object to versioned object storage, records the returned object key in PostgreSQL, and reports success only after both operations complete or a compensating action is queued. A replacement container should then read the same object by key. **Observed result.** A staging run uploads 100 known files, forcibly removes every application container, and retrieves all 100 files through replacement containers. Object-storage audit logs contain 100 successful writes and 100 later reads. Checksums before and after replacement match. **Bounded conclusion and uncertainty.** External object storage removed dependence on the container writable layer under the tested replacement conditions. The experiment does not prove cross-region durability, correct authorization, or recovery from a database commit that succeeds after an object write fails. A later transaction-design exercise must test those cases. ## Normal case CampusCart builds one image per reviewed commit, pins deployment manifests to a digest, injects environment-specific configuration at startup, writes request logs to standard output, and stores durable data in PostgreSQL and object storage. Several containers use the same image digest. Each container can be removed without losing application data. Health checks prevent the load balancer from routing requests before the application is ready. ## Failure and edge cases A container can pass a process-alive check while returning incorrect answers because a liveness check establishes only narrow evidence. A read-only root filesystem can break software that assumes `/tmp` or a home directory is writable. A container built for one processor architecture may not execute on another architecture without a matching image manifest or emulation. A root user inside a container can retain dangerous kernel capabilities. A tag can move between digests. A secret copied during an image build can remain recoverable from a layer even after a later layer deletes the file. ## Important qualifications Containers improve environmental consistency only for content actually captured by the image and declared configuration. The host kernel, processor architecture, runtime, attached storage, and network policy remain external dependencies. Reproducible builds require pinned inputs and a controlled build procedure. Image scanning reports known findings under a scanner's database and configuration; a clean report does not prove absence of vulnerable or malicious code. Isolation strength depends on kernel configuration, runtime settings, capabilities, seccomp filters, mandatory access controls, and workload behavior. A virtual machine generally provides a separate guest kernel and a different isolation boundary. Neither boundary removes the need for least privilege or patching. ## Common errors **“A container is a small virtual machine.”** The reasoning fails because the containerized process normally uses the host kernel. An image supplies user-space files, not a guest kernel. **“The image contains the running application data.”** The reasoning confuses immutable image layers with a container's writable layer and external mounts. Inspect each mount and write path. **“The same tag means the same release.”** A tag is a mutable name. A digest identifies content. **“A low processor limit gives a process a slower processor.”** A quota usually changes how much scheduler time the process receives during an interval. The physical processor does not necessarily change clock frequency. ## Knowledge check **Question:** A container writes 20 megabytes to `/var/cache/app`, has no volume at that path, and is then removed. A new container starts from the same image digest. Which bytes are guaranteed to appear in the new container? **Answer:** Only bytes present in the immutable image layers are guaranteed by the stated conditions. The removed container's 20-megabyte writable-layer change is not part of the image and is not preserved by a volume. ## Exercises **Check.** Run a disposable container, record its image digest and host process identifier, write a named file into the writable layer, and remove the container. Start a new container from the digest. Submit the commands, relevant inspection fields, and evidence about the file. **Practice.** Build the two Dockerfile orders from the empirical test. Submit both Dockerfiles, `docker history` output, changed-layer digests, bytes transferred, and a short causal explanation. Keep the base image and dependency lockfile constant. **Challenge.** Draw CampusCart's browser-to-container packet path for a local Docker bridge and for a cloud load balancer. Name both sides of every address translation and port mapping. Submit the diagrams plus packet or connection evidence from `curl`, `ss`, and an appropriate capture tool. Remove credentials and personal data. ## Authoritative sources and further reading ### Required reading - [Docker overview](https://docs.docker.com/get-started/docker-overview/) supplies the official image, container, daemon, registry, namespace, and execution walkthrough. - [Open Container Initiative Image Format Specification](https://github.com/opencontainers/image-spec) defines interoperable image manifests and layers. ### Implementation documentation - [Docker storage drivers](https://docs.docker.com/engine/storage/drivers/) explains writable layers and copy-on-write behavior; implementation details depend on the selected driver. - [Docker storage](https://docs.docker.com/engine/storage/) distinguishes container layers, volumes, bind mounts, and temporary mounts. - [Kubernetes Container Runtime Interface](https://kubernetes.io/docs/concepts/architecture/cri/) defines the kubelet-to-runtime interface and current v1 requirement. ### Primary standards and kernel reference - [Open Container Initiative Runtime Specification](https://github.com/opencontainers/runtime-spec) defines the runtime configuration and lifecycle model. - [Linux kernel control group v2 documentation](https://docs.kernel.org/admin-guide/cgroup-v2.html) is the primary kernel reference for resource distribution and accounting. ### Advanced and version-sensitive material Runtime socket configuration, image-store implementation, cgroup mode, and available isolation controls vary by operating system, Docker release, Kubernetes release, and runtime service. Record exact versions in exercise evidence. ## Transition A container packages one program and its user-space dependencies. Production applications require repeated placement, health evaluation, network identity, updates, and recovery across many containers and machines. Chapter 10 examines how Kubernetes represents desired workload conditions and continuously reconciles observed cluster conditions toward them. --- # Kubernetes Architecture and Operations At 08:42 on registration day, CampusCart receives 390 checkout requests per second. Elena, the engineer on call, sees two application instances stop answering. The checkout address still resolves, and most students continue receiving responses. Elena opens a terminal and runs `kubectl get pods`, `kubectl get deployment`, and `kubectl get events` against the bookstore's managed Kubernetes cluster. The command output shows eight requested application replicas, six ready Pods, two terminating Pods, and two newly pending Pods. A Deployment object still says eight replicas should exist. The cluster scheduler has not yet placed the pending Pods because the available worker nodes lack requested memory. A node autoscaler asks the cloud provider for another virtual machine. Four minutes later, the kubelet on that machine reports the new Pods ready, and the Service begins sending connections to them. The recovery involves stored objects and repeated control loops, not one central program that directly executes every instruction. A user submits a desired condition through the Kubernetes application programming interface. Controllers compare stored desired conditions with observed conditions. A scheduler chooses nodes for unscheduled Pods. A kubelet on each selected node asks a container runtime to start containers. Network and storage plugins connect external capabilities. Each participant produces different evidence. ## Prerequisites and earlier lessons - Chapter 3 supplies packet paths, addresses, ports, load balancing, and Domain Name System behavior. - Chapter 4 supplies virtual machines, regions, zones, and shared responsibility. - Chapter 9 supplies images, containers, control groups, volumes, and process evidence. - The reader should understand JavaScript Object Notation, Hypertext Transfer Protocol requests, and eventual completion of asynchronous work. ## Learning outcomes After completing the chapter, the reader can follow a Kubernetes object from an application programming interface request to a running container, distinguish a Pod, Deployment, and Service, interpret common `kubectl` evidence, calculate basic replica capacity, and diagnose a pending or unready workload without assuming that self-healing preserves correctness. The reader can also state which cluster responsibilities remain with an application team when a cloud provider operates the control plane. ## Observable evidence `kubectl` is a client. The command serializes a request and sends it to the Kubernetes application programming interface server identified by the active kubeconfig context. The server authenticates the caller, authorizes the requested action, applies admission rules, validates the object, and reads or writes persistent cluster data. A successful command establishes that the server accepted or returned a representation at a particular time. The output does not prove that every controller has completed subsequent work. `kubectl get deployment campuscart-api -o yaml` returns the Deployment's desired replica count, Pod template, labels, current status, and metadata. The application programming interface server produces the response from stored object data. `metadata.generation` changes when the desired specification changes. `status.observedGeneration` reports the generation processed by the relevant controller. A lower observed value supplies direct evidence that reconciliation has not caught up. `kubectl get events --sort-by=.metadata.creationTimestamp` returns Event objects written by cluster participants. A scheduler can write `FailedScheduling`; a kubelet can write image-pull or mount failures. The selected sort field orders Event-object creation, not necessarily every underlying occurrence; aggregated Event series can report separate event and last-observed times. Kubernetes Events are best-effort supplemental evidence. Event retention and aggregation are implementation settings, so absence of an old event does not prove that the event never occurred. Node-agent logs, container logs, metrics, and cloud-provider activity records may supply additional evidence. `kubectl describe pod` combines object fields and related events into a human-readable report. A container status with `state.waiting.reason: ImagePullBackOff` establishes that the kubelet is delaying another pull attempt. The reason does not by itself distinguish a nonexistent digest, denied registry credential, or unreachable registry. The detailed message and an independent registry test refine the conclusion. ## Mechanism and terminology in context A Kubernetes **Pod** is the smallest ordinary scheduling unit. One Pod can contain one or more containers that share a network namespace and declared storage volumes. Containers in one Pod reach one another through localhost. The shared fate is important: Kubernetes schedules the Pod to one node and replaces the Pod as a unit. A Pod address is not a durable application address because replacement can create a new Pod with a new address. A **Deployment** declares a desired template and replica count for replaceable Pods. The Deployment controller manages ReplicaSets, and a ReplicaSet controller creates or deletes Pods to approach the requested count. During an update, old and new ReplicaSets can coexist according to surge and unavailable limits. The controller does not copy a running process. The controller creates new Pods from the new template and later removes old Pods. A conventional selector-backed **ClusterIP Service** gives selected Pods a stable virtual address, and cluster Domain Name System software normally gives the Service a stable name. The Service selector matches Pod labels. EndpointSlice objects can list ready, unready, serving, and terminating endpoints; the ordinary data path selects endpoints considered ready for new traffic. A node networking implementation or another data-plane implementation directs connections addressed to the virtual address toward an endpoint. Headless Services do not receive the usual virtual address, and ExternalName Services return a Domain Name System alias without selecting Pods. A Service does not inspect whether an application response is semantically correct. The control plane includes the application programming interface server, a data store commonly implemented by etcd, a scheduler, and controller managers. The scheduler selects a node for an unbound Pod by filtering impossible placements and scoring feasible nodes. The scheduler writes a binding; the scheduler does not start the container. A kubelet on the chosen node observes the assigned Pod and calls a Container Runtime Interface implementation. Container, network, and storage interfaces let different implementations supply execution, connectivity, and volumes. Kubernetes uses **reconciliation**: a controller repeatedly compares desired object fields with observed cluster conditions and performs bounded actions to reduce a difference. Reconciliation is asynchronous. A successful Deployment update means the desired object was accepted, not that the rollout is healthy. `kubectl rollout status`, readiness, application metrics, and user-facing probes provide later evidence. Kubernetes self-healing can restart a failed container, replace a failed Pod, and stop routing through an unready endpoint. The mechanism restores declared quantity and health conditions. The mechanism cannot restore data that existed only in a deleted writable layer, determine that a wrong price is incorrect, or make two zones independent when both depend on one database. ## Working model: replica capacity CampusCart normally completes 80 requests per second and reaches 400 requests per second during registration. A controlled load test shows that one Pod sustains 65 completed requests per second while keeping the 95th-percentile latency below 300 milliseconds. The team reserves 25 percent of theoretical Pod capacity for measurement error, uneven distribution, and short spikes. `usable requests per Pod per second = 65 × (1 - 0.25) = 48.75` `required replicas = ceiling(400 / 48.75) = 9` The numerator counts completed CampusCart requests during one second. The denominator counts acceptable-latency completions by one Pod during one second after the reserve. The model assumes requests resemble the load test, dependencies can accept the combined work, Pods receive their requested processor and memory, and traffic distribution is sufficiently even. The model omits startup time, zone failures, retries, long-lived connections, and database contention. An availability requirement may add replicas and topology constraints beyond the capacity result. A common Horizontal Pod Autoscaler input such as processor utilization is an indirect signal. If a checkout waits on PostgreSQL, request latency can rise while processor utilization remains low. A request-concurrency or queue-length signal may describe demand better, but the signal must have bounded cardinality, stable collection, and a tested relationship to safe capacity. ## Empirical test Deploy CampusCart to a disposable cluster with three replicas and fixed Pod requests. Send a reproducible mix of catalog and checkout traffic. Increase offered requests from 50 to 250 per second in 25-request steps while preserving database contents, client concurrency rules, image digest, node type, and region. Record completed requests per second, error fraction, latency percentiles, Pod processor time, memory, restarts, and database connections. The capacity model predicts near-linear completed work before a shared dependency saturates. A plateau below 146 acceptable requests per second for three replicas rejects the isolated-Pod estimate under cluster conditions. Database connection saturation, network limits, load-generator capacity, or throttled processor time provide alternative explanations. Repeat one point after adding replicas. An unchanged completion ceiling implicates a shared dependency more strongly than Pod count. ## Worked investigation **Initial question.** Why do four new CampusCart Pods remain pending after a replica increase from six to ten? **Operating conditions.** The cluster has three worker nodes. Each node offers 4 gibibytes of schedulable memory after operating-system and cluster reservations. Existing Pods request a combined 3.4 gibibytes on each node. Each new CampusCart Pod requests 1 gibibyte. Pod anti-affinity prefers but does not require different nodes. **Evidence collected.** Deployment status reports ten desired replicas and six available replicas. Four Pod specifications have no `spec.nodeName`. Scheduler Event objects report `0/3 nodes are available: 3 Insufficient memory`. Node descriptions show the allocatable and already requested quantities. Actual memory-use metrics show only 2.1 gibibytes per node. **Interpretation.** The missing node name and scheduler event directly establish a placement failure caused by requested, not observed, memory. Kubernetes schedules against requests because current use can change after placement. Low observed use does not create schedulable requested capacity. **Proposed intervention and prediction.** Review measured high-percentile memory under representative load, set a justified 700-mebibyte request while retaining a 1.2-gibibyte limit, and add one node before registration peaks. Each existing node can then place exactly one additional 700-mebibyte Pod under the stated 4-gibibyte allocation: three new Pods place on the old nodes, and the fourth waits until the new node is ready. The new node also supplies failure and rollout margin. The change must not rely on memory compression or swapping unless the node policy explicitly supports and tests those mechanisms. **Observed result.** A staging rollout with the revised request schedules ten Pods across four nodes. A load test at 400 offered requests per second completes 397 per second, reports a 0.75 percent application-error fraction, and keeps peak Pod memory below 610 mebibytes. The team separately defines a degraded one-node-loss objective of at least 99 percent successful requests at 300 offered requests per second. Draining a node that hosts three Pods leaves seven ready replicas and completes 298 requests per second during the failure-mode test. **Bounded conclusion and uncertainty.** Requested-memory pressure explained the pending Pods under the recorded scheduler conditions. Revised requests and one node enabled the tested replica count. The experiment does not establish safety for a memory leak, a whole-zone loss, or a database limit. Production alerts must compare working-set growth, out-of-memory events, and user-facing results. ## Normal case The application team submits a Deployment pinned to an image digest, a Service selecting the Deployment's labels, resource requests based on measurements, and readiness and startup probes. The managed provider operates the application programming interface server and etcd. The team still owns the workload manifest, access policy, image security, application correctness, data design, and user-facing objectives. A controlled rollout creates new Pods, admits ready endpoints, and removes old Pods within stated surge and unavailability bounds. ## Failure and edge cases A readiness probe can succeed before the application can complete a checkout because a shallow endpoint checks only the local event loop. A liveness probe can amplify an overloaded dependency by repeatedly restarting callers. A PodDisruptionBudget constrains voluntary disruption; the object does not prevent involuntary node failure. A two-replica Deployment can still place both Pods in one zone unless topology rules and node supply prevent that placement. A `CrashLoopBackOff` label describes delayed restart behavior, not one root cause. A network policy has effect only when the installed networking implementation enforces the policy. A Secret object is not automatically encrypted in every cluster configuration, and base64 representation is not encryption. Cluster administrator access can bypass many workload controls. Managed Kubernetes reduces control-plane operation but does not remove version upgrades, application programming interface deprecations, node images, add-ons, cost, or incident response. ## Important qualifications The official documentation viewed in July 2026 presents Kubernetes version 1.36 as current and retains versioned documentation for earlier releases. Clusters commonly lag the newest release. Always read the documentation matching the server version returned by `kubectl version`, and check provider-supported version and deprecation schedules. Declarative objects do not make every operation idempotent. An application can receive a repeated request during a retry or rollout. The application must provide appropriate request identifiers, transactions, or deduplication. Kubernetes availability depends on external identity, registry, network, storage, Domain Name System, and cloud-provider mechanisms. ## Common errors **“A Deployment runs containers.”** The reasoning collapses several actors. The Deployment controller manages ReplicaSets; the scheduler binds Pods; the kubelet and container runtime start containers. **“A Service is a cloud load balancer.”** A Kubernetes Service is an application programming interface object with several types. A `LoadBalancer` Service can cause provider integration to create an external load balancer, while a `ClusterIP` Service remains internal. **“Current memory use determines scheduling.”** The default scheduler primarily considers declared requests and constraints. Current use can influence other controllers but does not erase admitted requests. **“Self-healing makes the application reliable.”** Replacement restores declared infrastructure conditions. Application data, dependency independence, correct probes, safe retries, and tested failure behavior determine the user result. ## Knowledge check **Question:** The application programming interface server accepts a Deployment update, but `status.observedGeneration` remains lower than `metadata.generation`. What does the evidence establish? **Answer:** The server accepted and stored a newer desired specification, while the Deployment controller has not yet reported processing that generation. The evidence does not yet establish a failed rollout or a healthy rollout. ## Exercises **Check.** Create a local learning cluster, submit a two-replica web Deployment, and preserve the Deployment, ReplicaSet, Pod, Service, and EndpointSlice YAML. Annotate which actor created each object and which fields link the objects. **Practice.** Delete one Pod while watching Pods and Events with timestamps. Submit the command transcript and a causal timeline from deletion request through replacement readiness. Identify direct observations and inferences separately. **Challenge.** Reproduce the capacity experiment with a safe test application. Submit the load-generator configuration, offered and completed requests per second, latency percentiles, Pod requests, node allocatable quantities, and a graph or table. State the first shared bottleneck and one result that would reject your explanation. ## Authoritative sources and further reading ### Required reading - [Kubernetes components](https://kubernetes.io/docs/concepts/overview/components/) names control-plane and node participants; use the documentation version matching the cluster. - [Kubernetes Pods](https://kubernetes.io/docs/concepts/workloads/pods/) defines the scheduling unit, shared context, and Pod lifecycle. - [Kubernetes Service](https://kubernetes.io/docs/concepts/services-networking/service/) defines stable network exposure and EndpointSlices. ### Implementation documentation - [Kubernetes self-healing](https://kubernetes.io/docs/concepts/architecture/self-healing/) states the recovery behaviors and their scope. - [Kubernetes API reference v1.36](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/) supplies exact fields matching the current documentation version cited in July 2026. ### Primary engineering reference - [etcd documentation](https://etcd.io/docs/) explains the consistent key-value store commonly used for Kubernetes data. ### Advanced and version-sensitive material Kubernetes feature gates, application programming interface fields, deprecations, and managed-provider support schedules change by release. Match documentation to `kubectl version` server output rather than assuming the newest website version. ## Transition Kubernetes controls workloads inside and around clusters. A public request still needs global name resolution, Internet routing, Transport Layer Security termination, denial-of-service protection, caching, and an origin path. Chapter 11 follows a request through Cloudflare's authoritative Domain Name System and edge proxy, then connects those mechanisms back to cloud networks and Kubernetes origins. --- # Cloudflare, Domain Name System, and the Network Edge Nia, a student in Singapore, opens `store.campuscart.example` during registration. CampusCart's origin application runs in a cloud region in Virginia. Nia's browser first needs an Internet Protocol address. If the recursive Domain Name System resolver has no fresh cached answer, the resolver follows referrals and ultimately asks an authoritative nameserver for the domain. The returned address leads Nia's packets to a nearby Cloudflare location rather than directly to the Virginia origin. Cloudflare terminates Nia's Transport Layer Security connection, evaluates configured security rules, and checks an edge cache. A catalog image has a fresh cached response, so Cloudflare returns the image without contacting Virginia. A checkout request is not cached. Cloudflare creates or reuses another encrypted connection to the origin load balancer, which forwards the request to the CampusCart application. Nia can observe the resolved address, certificate, Hypertext Transfer Protocol response headers, timings, and path changes. The CampusCart team can observe authoritative Domain Name System records, Cloudflare request logs or analytics, cache status, origin load-balancer logs, and application request identifiers. No single artifact proves the whole path. Each institution produces evidence about the portion that the institution operates. ## Prerequisites and earlier lessons - Chapter 3 supplies packets, routing, Domain Name System resolution, Transport Layer Security, ports, and load balancing. - Chapter 5 supplies cloud regions, load balancers, object storage, and origin applications. - Chapter 10 supplies Services and external exposure for a Kubernetes origin. - The reader should understand that one Hypertext Transfer Protocol request can produce several dependent network exchanges. ## Learning outcomes After completing the chapter, the reader can distinguish authoritative and recursive Domain Name System roles, explain proxied and direct records, follow a request through an anycast edge and an origin, compute a cache hit ratio and a latency budget, interpret cache and certificate evidence, and diagnose common failures without treating Cloudflare as a replacement for origin security or reliability. ## Observable evidence `dig store.campuscart.example` asks the configured recursive resolver for a Domain Name System answer. The response names the status, answer records, time to live, responding resolver, and query duration. The recursive resolver produces the response from its cache or from iterative queries. An answer establishes what that resolver returned at that moment. The answer does not prove that every resolver has the same cached value. `dig +trace` performs iterative referrals from root servers toward authoritative nameservers. The transcript identifies delegations and final authoritative data under the tool's network conditions. A corporate firewall, split-horizon configuration, or local override can make an application's actual resolution path differ. Compare the application environment rather than assuming a laptop transcript represents a production container. `curl -v https://store.campuscart.example/catalog/42` shows connection addresses, Transport Layer Security negotiation, certificate names, request headers, response status, and response headers. Cloudflare can add a `CF-Cache-Status` header describing cache handling for the response. `HIT` supplies evidence that a Cloudflare cache served an eligible stored response in the observed exchange. The header does not establish that every location has the object or that a later request will hit. An origin load-balancer access log is produced when the origin receives a request. A Cloudflare edge response with no matching origin entry can support a cache-hit inference when identifiers and time windows align. Absence alone is weak evidence because log delivery can be delayed, sampled, filtered, or misconfigured. A request identifier forwarded end to end gives a stronger correlation key than timestamps alone. ## Mechanism and terminology in context The **Domain Name System**, abbreviated DNS, maps names to typed data through a distributed hierarchy. A stub resolver in an application or operating system asks a recursive resolver. The recursive resolver follows or caches referrals and asks an authoritative nameserver for the final zone data. The authoritative nameserver stores the domain owner's configured records. The recursive resolver and authoritative nameserver occupy different sides of the query relationship and can cache different information. A DNS **time to live**, abbreviated TTL, states how many seconds a resolver may cache a record before refreshing it. Changing an authoritative record does not recall copies already cached under the previous TTL. A 300-second TTL bounds compliant positive caching under ordinary conditions but does not guarantee every client changes within exactly five minutes because applications and intermediaries can add behavior. For a Cloudflare-proxied address record, authoritative DNS returns Cloudflare anycast addresses rather than the origin address. **Anycast** means multiple network locations announce reachability for the same address prefix through the Border Gateway Protocol. Internet routing selects a reachable path according to routing policy and topology. “Nearest” therefore means a routing result, not necessarily minimum geographic distance or minimum measured latency. At the selected edge location, Cloudflare acts as a **reverse proxy**. The browser creates a connection to Cloudflare, and Cloudflare creates a logically separate connection toward the configured origin when needed. The browser-to-edge and edge-to-origin connections have distinct source addresses, certificates, timing, and failure conditions. Origin software must recover the intended client information from authenticated proxy headers and must prevent untrusted callers from forging those values. A **content delivery network**, abbreviated CDN, stores eligible response bytes at distributed cache locations. A cache key commonly includes the scheme, hostname, path, and selected query or header values under product configuration. The edge checks freshness and request rules. A fresh match can be served without an origin round trip. A miss or stale response can cause an origin fetch, validation, or configured stale response. Cache-control directives are protocol inputs. `Cache-Control: public, max-age=300` makes shared caching permissible for five minutes under normal interpretation. Personalized checkout responses should normally be private or non-cacheable. A cache rule that ignores authentication or cookies can expose one user's response to another user. Cloudflare settings can override origin directives, so the effective policy includes both configurations. Cloudflare can also evaluate a web application firewall rule, rate limit, bot rule, or distributed denial-of-service mitigation before forwarding. The edge has no automatic knowledge of CampusCart's business semantics. A valid but abusive checkout pattern may pass a generic rule, while an overly broad rule may block legitimate registration traffic. Rule events, sampled requests, application outcomes, and controlled tests must be compared. Cloudflare is not synonymous with “the cloud.” Cloudflare operates a global network and offers DNS, proxy, compute, storage, and security products. An origin can run on Amazon Web Services, Google Cloud, Microsoft Azure, another provider, or private hardware. The edge and origin remain separate administrative and network domains even when one vendor supplies both types of capability. ## Working model: cache effect on origin traffic and latency During a five-minute catalog test, browsers make 60,000 eligible image requests. Cloudflare serves 48,000 from cache and sends 12,000 to the origin. `cache hit ratio = cache-served eligible requests / all eligible requests = 48,000 / 60,000 = 0.80` The origin request reduction is 48,000 requests during 300 seconds, or 160 avoided origin requests per second on average. The completed quantity is avoided origin requests; the interval is 300 seconds. The ratio applies only to requests defined as eligible in the measurement. Suppose measured edge hits have a 35-millisecond sample mean, while measured origin-path responses have a 125-millisecond sample mean: 35 milliseconds for the browser-to-edge portion plus 82 milliseconds for the edge-to-origin round trip and origin work plus 8 milliseconds for remaining edge processing. A weighted mean approximation is `0.80 × 35 ms + 0.20 × 125 ms = 53 ms`. The calculation predicts a population mean from path-specific sample means, not a percentile. The model assumes stable traffic mix, independent samples, and comparable object sizes. The model omits connection reuse, cache revalidation, tiered caching, queueing, loss, and regional variation. Percentiles cannot be combined with the same weighted arithmetic without distribution data. ## Empirical test Choose one versioned public image and one authenticated checkout endpoint in a staging zone. Purge the image once. Send a controlled request from one test location, record DNS answers, connection addresses, response headers, edge timing, origin log entry, and request identifier. Repeat the identical image request ten times while holding URL, query, headers, and deployment constant. Then change one cache-key input deliberately. The mechanism predicts an initial miss followed by hits under a cacheable policy, plus a new miss when the effective key changes. The checkout endpoint should not return a shared hit. A repeated miss rejects the expected cache behavior for the tested location and inputs. Possible causes include a non-cacheable directive, cookie, rule override, oversized response, purge, location change, or cache eviction. Inspect configuration and edge status rather than assuming one cause. ## Worked investigation **Initial question.** Why do some CampusCart users receive the former origin address for up to ten minutes after a migration? **Operating conditions.** The team changes an unproxied DNS A record from `192.0.2.10` to `192.0.2.20` at 15:00. The old record had a 600-second TTL. The old origin remains reachable but serves a maintenance page. Cloudflare authoritative nameservers immediately answer the new address after the change. **Evidence collected.** An authoritative query directed to a Cloudflare nameserver returns the new address and a 600-second TTL at 15:01. A recursive resolver that queried at 14:59 returns the old address with 472 seconds remaining at 15:01. Browser connection evidence shows a user connecting to the old address. The old origin access log contains the user's request. **Interpretation.** The authoritative answer establishes current zone data. The recursive answer directly establishes a still-valid cached copy of former data. The remaining TTL explains why the resolver does not yet ask the authoritative nameserver. The user's connection and old-origin log establish use of the former address. “DNS propagation” is an imprecise label; retained caches, not a broadcast moving slowly between all DNS servers, explain the recorded case. **Proposed intervention and prediction.** Keep the old origin serving or forwarding valid responses for at least the former TTL plus operational margin. For future planned changes, lower the TTL sufficiently early, wait for the old TTL to expire, perform the address change, and later raise the TTL. The prediction is that resolvers refreshing after the preparatory interval retain the old value only for the lower TTL. **Observed result.** The old origin remains available for 20 minutes. Queries to six controlled recursive resolvers all return the new address after their recorded remaining TTL reaches zero. No further maintenance responses appear after 15:10. **Bounded conclusion and uncertainty.** Positive DNS caching under the prior 600-second TTL caused the observed split during the migration. The evidence does not cover every recursive resolver or application cache. Keeping both origins correct during the transition provides stronger safety than assuming uniform resolver behavior. ## Normal case CampusCart delegates authoritative DNS to Cloudflare, enables proxying for the public hostname, restricts origin ingress to authenticated or expected proxy traffic, uses end-to-end Transport Layer Security, and forwards a validated request identifier. Versioned public assets have explicit long-lived cache directives. HTML receives a shorter policy. Checkout responses are not stored in a shared cache. Edge and origin logs share enough identifiers for bounded correlation. ## Failure and edge cases An origin certificate can expire while browser-to-edge Transport Layer Security remains valid, causing an edge-to-origin failure. A cached object can remain available during an origin outage, which helps static reads but can hide origin failure from superficial monitoring. A cache key that omits a language, authorization, or content-negotiation input can return wrong bytes. Purging one URL may not remove variants under other keys. Anycast route changes can move two consecutive connections to different edge locations. Internet routing incidents can create longer paths without any application deployment. A proxied record hides the ordinary origin address from DNS answers but does not remove an origin address already published elsewhere. A distributed denial-of-service network can absorb large traffic volumes, yet an attacker who reaches the origin directly can bypass the proxy unless origin controls prevent access. ## Important qualifications Cloudflare product limits, cache eligibility rules, default TTL behavior, and log availability are version-sensitive. Consult the current product documentation and the active account plan. The architectural principles—separate recursive and authoritative DNS roles, distinct proxy connections, cache keys, freshness, and origin controls—remain more durable than one product default. Cache ratio alone does not measure user experience. A high ratio for tiny fast objects can coexist with a slow checkout. An edge location close to a user can still depend on a distant origin for dynamic responses. Measure the browser result, edge processing, origin work, and dependency time separately. ## Common errors **“DNS sends the request to the server.”** DNS supplies typed name data. The client uses an address to create later network packets. **“A low TTL changes every client immediately.”** A new TTL affects responses that contain it. Resolvers can retain a prior answer for its prior remaining TTL. **“Anycast always selects the geographically closest location.”** Border Gateway Protocol policy and reachable topology select a path. Geography is not the protocol input. **“Cloudflare eliminates the origin.”** A cache hit can avoid one origin request. Dynamic misses, revalidation, storage, application logic, and data still require an origin unless the application has been deliberately implemented on edge compute and storage products. ## Knowledge check **Question:** A browser receives `CF-Cache-Status: HIT`, and no matching origin request appears in the immediate log search. What can the engineer conclude? **Answer:** The header supports the observation that Cloudflare served a cached response for the exchange. The missing origin entry is consistent with no origin fetch but needs qualification because origin logs can be delayed, filtered, sampled, or correlated incorrectly. ## Exercises **Check.** Query one domain through the default recursive resolver and directly against an authoritative nameserver. Submit the two commands, answer sections, TTL values, and an annotation naming who produced each answer. **Practice.** Serve a versioned static file through a test Cloudflare zone. Preserve two `curl -v` transcripts, edge cache-status headers, an origin access-log excerpt, and a causal explanation for the first and second results. Do not include authentication tokens. **Challenge.** Build a migration runbook for an origin-address change with a 3,600-second initial TTL. Submit a timeline that includes the preparatory TTL change, verification queries, coexistence interval, rollback condition, and evidence required before old-origin removal. Test the timing with two controlled resolvers if you own an appropriate domain. ## Authoritative sources and further reading ### Required reading - [Cloudflare DNS concepts](https://developers.cloudflare.com/dns/concepts/) distinguishes authoritative nameservers, records, proxy status, and resolver roles. - [Request for Comments (RFC) 9111: HTTP Caching](https://www.rfc-editor.org/rfc/rfc9111) defines cache stores, freshness, validation, and directives independently of one vendor. ### Implementation documentation - [Cloudflare DNS time to live](https://developers.cloudflare.com/dns/manage-dns-records/reference/ttl/) documents current TTL behavior and proxied-record qualifications. - [Cloudflare CDN reference architecture](https://developers.cloudflare.com/reference-architecture/architectures/cdn/) explains the provider's anycast proxy, caching, tiered cache, and origin relationship. - [Cloudflare cache responses](https://developers.cloudflare.com/cache/concepts/cache-responses/) defines the current `CF-Cache-Status` values used in the chapter's empirical evidence. ### Primary protocol standards - [RFC 1034: Domain Names—Concepts and Facilities](https://www.rfc-editor.org/rfc/rfc1034) and [RFC 1035](https://www.rfc-editor.org/rfc/rfc1035) provide foundational DNS protocol specifications; later RFCs update details. - [RFC 4271: Border Gateway Protocol 4](https://www.rfc-editor.org/rfc/rfc4271) specifies the interdomain routing protocol underlying anycast route announcements. ### Advanced and version-sensitive material Cloudflare cache eligibility, header behavior, plan limits, logging, and optional routing products change independently. Record the active zone configuration, account plan, test location, and documentation access date. ## Transition The request path now includes applications, clusters, cloud networks, DNS, and an edge proxy. Operators need measurements that connect user results with the responsible program and machine, plus decision rules for safe change and incident response. Chapter 12 develops production evidence, service objectives, security controls, cost models, and an investigation procedure. --- # Production Observability, Reliability, Security, and Cost At 10:16 on a Tuesday, CampusCart's checkout success fraction falls from 99.95 percent to 97.8 percent. Priya, the engineer on call, receives an alert generated from completed checkout responses during a five-minute window. Processor use looks ordinary. A deployment ended twelve minutes earlier. PostgreSQL connection-wait time rose at the same moment as the errors. Priya opens a dashboard, a deployment record, sampled distributed request traces, application logs, and database metrics. Each artifact has a producer and scope. The browser or synthetic client measures the user path. The application emits spans and logs. The database exporter reads server counters. The deployment platform records the image digest and time. Priya must not combine timestamps into a causal claim until identifiers, clocks, and mechanisms support the relationship. Production practice turns desired outcomes into measured conditions, compares observations with explicit objectives, and changes the application under controlled risk. Reliability, security, and cost share the same request path but answer different questions. Reliability asks whether intended work completes within stated conditions. Security asks whether only authorized actions and information flows occur. Cost analysis asks which priced quantities the architecture consumes during an interval. ## Prerequisites and earlier lessons - Chapters 3 and 4 supply latency, throughput, failure domains, and shared responsibility. - Chapters 5 through 8 supply provider identity, storage, audit, reliability, cost, and public-sector constraints. - Chapters 9 through 11 supply container, Kubernetes, edge, and origin evidence. - The reader should understand percentages, percentiles, identifiers, and controlled experiments. ## Learning outcomes After completing the chapter, the reader can define a measurable service level indicator and objective, calculate an error budget, distinguish logs, metrics, and distributed traces, conduct a bounded incident investigation, connect least privilege to observable authorization decisions, and calculate a basic workload cost model. The reader can also explain why one dashboard, one average, one certification, or one monthly invoice cannot establish production health. ## Observable evidence A **log** is an event entry produced by a named program or device. A CampusCart application log can contain a timestamp, severity, request identifier, route, result code, and error category. The application clock and logging library produce the timestamp and serialized fields. A log directly establishes what the instrumented code wrote; the entry does not automatically establish what the user received or what an uninstrumented dependency did. A **metric** is a runtime measurement aggregated or reported with time and attributes. An OpenTelemetry counter can add one for each completed checkout. A histogram can group observed checkout durations. The application instrumentation produces measurements, an exporter transmits them, and a backend aggregates them. A five-minute success fraction must name numerator and denominator: successful eligible checkouts divided by all eligible completed checkouts during those five minutes. A distributed request **trace** represents a request's path as spans. Each instrumented library or application creates spans for bounded operations and propagates trace context through request headers. A span reports an operation name, start and end times, attributes, status, and relationships. A missing span can mean no operation occurred, context failed to propagate, sampling omitted it, or instrumentation failed. Trace evidence is powerful but incomplete. A cloud audit event is produced by a provider control plane when a caller performs a recorded management or data action under the product's audit configuration. Fields commonly include caller identity, action, target, time, source context, and result. Audit retention, enabled data events, and delivery delay matter. An audit log cannot record an action performed outside the instrumented control plane. A cost and usage line item is produced by provider billing software from metered priced quantities. The line can name account, product, region, usage type, amount, price, and credits. The bill establishes charged quantities under provider rules. The bill does not allocate business value or reveal an application's causal need for the quantity without tags, workload mapping, and engineering evidence. ## Mechanism and terminology in context A **service level indicator**, abbreviated SLI, is a carefully defined quantitative measure of user-relevant behavior. CampusCart can define availability as successful eligible checkout responses divided by all eligible checkout attempts measured at the public edge. Eligibility rules might exclude explicitly blocked abuse but include application errors and timeouts. A **service level objective**, abbreviated SLO, is a target for that indicator over an interval, such as 99.9 percent successful eligible checkouts over 30 days. An SLO differs from a **service level agreement**, abbreviated SLA. An agreement includes consequences when a target is missed. A private engineering target without contractual consequences is an objective, not automatically an agreement. Teams use objectives to decide whether reliability work or change risk deserves priority. An **error budget** is the permitted unsuccessful fraction implied by an objective. For 99.9 percent success, the unsuccessful fraction is 0.1 percent. If CampusCart receives 2,000,000 eligible checkout attempts in 30 days, the request error budget is `2,000,000 × 0.001 = 2,000` unsuccessful attempts. A time-based approximation answers a different question and can mislead when traffic varies. The interactive model below uses minutes for intuition and states its assumption. Alerting converts a measurement into an operator interruption. An alert should describe user risk and link to evidence, not merely report a high internal counter. A fast error-budget consumption rule can detect a severe incident quickly. A slower rule can detect a small persistent loss. Multiple windows reduce noise while retaining urgency. Incident response is an evidence-preserving control loop. The responder declares scope, stabilizes user impact, records a timeline, tests hypotheses, and communicates bounded conclusions. Rollback is an intervention, not proof that one code change was the root cause. A rollback that restores service strengthens a causal explanation when the image digest is the controlled difference, but concurrent load or dependency recovery can still explain the result. Security applies identity and policy at every privileged action. **Least privilege** means a principal receives only actions, targets, conditions, and duration required for a task. Authentication establishes an identity claim under a mechanism. Authorization evaluates whether that principal can perform one action on one target under current policy. Network reachability and authorization are separate: a blocked packet prevents the request from reaching an identity check, while an allowed packet does not grant database permission. Encryption in transit protects bytes between named endpoints under certificate and key assumptions. Encryption at rest protects stored representation under a key-management and access model. Neither mechanism guarantees correct authorization before decryption. Key-use audit events, rotation tests, denied requests, and recovery exercises provide operational evidence. Cost has causal drivers. Compute charges can follow allocated instance time, requested container capacity, or execution duration. Storage can charge stored byte-months, operations, retrieval, and transfer. Network transfer can depend on direction and location. Managed products can add per-request or provisioned-capacity charges. An engineer maps every line to a workload object and an interval before proposing optimization. ## Working model: request error budget and cost CampusCart has a 99.9 percent monthly checkout-success objective and expects 2,000,000 eligible attempts. The permitted unsuccessful attempts are 2,000. Suppose an incident causes 440 failures in 22 minutes. The incident consumed `440 / 2,000 = 22 percent` of the monthly request error budget. The burn fraction has no time unit; the burn rate must name an interval or baseline. The incident average is 20 unsuccessful checkouts per minute during 22 minutes. For cost, suppose nine application replicas request 0.5 virtual processor and 0.75 gibibyte each for 730 hours. A platform price model charges $0.04 per requested virtual-processor hour and $0.005 per requested gibibyte-hour. The monthly approximation is: `processor cost = 9 × 0.5 × 730 × $0.04 = $131.40` `memory cost = 9 × 0.75 × 730 × $0.005 = $24.64` The $156.04 sum models only requested application compute. The model assumes constant replicas and prices, and it omits nodes, control plane, database, storage, requests, logs, support, discounts, taxes, and transfer. Provider calculators and actual billing data must replace example prices for a real decision. Lowering requests can reduce charge only when the billing product meters that quantity or when lower requests allow fewer paid nodes. ## Empirical test Instrument a staging checkout with one request identifier propagated through edge, application, and database-client operations. Send 1,000 controlled attempts and force the dependency call for every fiftieth attempt to return a terminal error. Disable automatic retries for this experiment so every injected dependency failure remains visible as one client failure. Keep image digest, database contents, client timeout, and request mix constant. Record client outcomes, edge status, application counters, trace sampling rule, structured logs, database errors, and timestamps. The mechanism predicts exactly 20 injected client failures and correlation between those failures and named dependency spans. A server-side success indicator that still reports 100 percent rejects the indicator's ability to represent user results. A difference can arise because the injector miscounts, sampling omits failed traces, eligibility filters remove cases, or identifiers fail to propagate. A second experiment can enable one retry deliberately and predict which client failures disappear while dependency-error evidence remains. ## Worked investigation **Initial question.** Did the 10:04 CampusCart release cause the checkout-objective violation observed at 10:16? **Operating conditions.** A rolling update replaced eight Pods with image digest B. Image A had served the previous day. Traffic remained between 75 and 90 completed requests per second. PostgreSQL has a 120-connection limit. Image B changes the connection pool from 10 to 25 connections per Pod. **Evidence collected.** The deployment record reports digest B reaching all eight Pods at 10:04. PostgreSQL metrics show established application sessions rising toward the 120-session limit as new Pods overlap old Pods. Application connection-pool metrics show callers waiting before they obtain a PostgreSQL session. Application histograms show connection-acquisition latency increasing. Failed traces contain database-acquisition spans ending at a two-second timeout. Edge logs report matching Hypertext Transfer Protocol (HTTP) status 503 responses. A configuration diff names the pool change. **Interpretation.** The deployment, configuration, database, trace, and edge evidence align on a mechanism: rollout overlap allowed more potential application connections than the database limit, callers waited, and the application returned 503 after its acquisition timeout. Timing alone would be insufficient; the changed limit and wait spans provide causal support. **Proposed intervention and prediction.** Roll back to image A, then cap total connection demand below the database limit with per-Pod pools and rollout surge included. The immediate prediction is that waiting connections and 503 responses fall after old image-B Pods stop. The durable prediction requires a load test across maximum replica and surge counts. **Observed result.** Rollback completes at 10:23. Connection waits return near zero by 10:24, and checkout success returns above 99.9 percent in the next five-minute window. A staging test with a 10-connection pool, ten replicas, and two surge Pods permits demand for 120 application sessions. Two existing administrative sessions leave capacity for only 118 application sessions under the 120-session database limit. The team lowers the pool to eight, which bounds the twelve rollout Pods at 96 application sessions and leaves 24 sessions for administration, migrations, and short overlap. **Bounded conclusion and uncertainty.** Image B's pool setting caused the recorded connection pressure under the rollout conditions, and rollback restored the measured path. The investigation does not prove that every 503 shared the same cause. The final design still depends on database work duration and connection release correctness. ## Normal case CampusCart defines user-facing indicators at the edge and corroborates them with application and dependency evidence. Structured fields use bounded cardinality; raw user identifiers do not become metric labels. Every deployment records an immutable image digest. Alerts use error-budget consumption and include a runbook. Workload identities receive narrow permissions. Secrets come from an audited secret store. Cost reports map priced quantities to owners and environments. ## Failure and edge cases An average latency can improve while a small user group experiences timeouts. Percentiles retain distribution information but still require population and window definitions. Metric labels containing raw request or user identifiers can create unbounded series and high memory or billing cost. A trace sampler can omit rare failures. Synchronized clocks can still differ enough to confuse a short event order. An administrator can grant permission through several policy paths, so reviewing one role does not establish effective privilege. A blocked public endpoint can remain reachable through an internal route. Encryption keys can be available to an overly broad runtime identity. A cost decrease can come from lower demand rather than an optimization. Compare normalized completed work and service outcomes. ## Important qualifications OpenTelemetry specifications and language implementations evolve independently. The metrics signal is stable, while newer signals can have different maturity by language and release. Read the current specification and implementation status. Telemetry can contain personal or secret data; collection, retention, redaction, and access need explicit controls. An objective is a product and engineering decision, not a provider guarantee copied from a product page. Dependency agreements do not automatically compose into application availability. Two dependencies that each report 99.9 percent availability can fail together or at different times. Test the actual request path and model correlation. ## Common errors **“Processor use is normal, so the application is healthy.”** The reasoning ignores blocked dependencies, authorization failures, network loss, and semantic errors. **“The trace proves the whole request path.”** The trace represents instrumented and sampled operations with propagated context. Missing spans and client-side behavior require separate evidence. **“99.9 percent means 43.2 minutes of downtime.”** The time conversion assumes a 30-day interval and uniform time-based measurement. A request-based objective can produce a different error budget. **“A lower bill proves an optimization.”** Demand, credits, commitments, and omitted work can lower charges. Normalize cost by a relevant completed quantity while confirming reliability and security constraints. ## Knowledge check **Question:** CampusCart completes 500,000 eligible checkout attempts during a week under a 99.95 percent success objective. How many unsuccessful attempts fit the objective? **Answer:** The unsuccessful fraction is 0.05 percent, or 0.0005. The request error budget is `500,000 × 0.0005 = 250` unsuccessful eligible attempts for the stated week. ## Exercises **Check.** Define one CampusCart availability indicator. Submit the numerator, denominator, measurement point, interval, eligibility rules, artifact producer, and one user failure the indicator misses. **Practice.** Add a request identifier to a three-operation test application. Preserve one client transcript, correlated structured logs, and a distributed trace. Annotate direct observations and inferences. Remove secrets and personal information. **Challenge.** Conduct a safe failure injection in a disposable environment. Submit the hypothesis, controlled condition, changed condition, objective impact, timeline, rollback trigger, telemetry limitations, and bounded conclusion. Include a cost estimate for the test using named priced quantities. ## Authoritative sources and further reading ### Required reading - [Google Site Reliability Engineering: Service Level Objectives](https://sre.google/sre-book/service-level-objectives/) provides the primary engineering treatment of indicators, objectives, agreements, percentiles, and error budgets. - [National Institute of Standards and Technology Special Publication (NIST SP) 800-53 Rev. 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final) catalogs security and privacy controls without asserting that one product configuration satisfies them. ### Implementation documentation - [OpenTelemetry concepts](https://opentelemetry.io/docs/concepts/) and [signals](https://opentelemetry.io/docs/concepts/signals/) define current telemetry categories and context. - [OpenTelemetry metrics specification](https://opentelemetry.io/docs/specs/otel/metrics/) supplies the normative programming and data model. ### Primary engineering report The Google Site Reliability Engineering chapter above documents an operating method developed from production practice. Apply the method to CampusCart's measured user path rather than copying example thresholds. ### Advanced and version-sensitive material - [FinOps Framework](https://www.finops.org/framework/) describes a current operating framework for cloud-cost accountability; provider bills and contracts remain authoritative for actual charges. - Provider pricing pages and calculators are version-sensitive implementation documentation. Record the region, date, unit, commitment, and tax assumptions used in any decision. ## Transition Production judgment combines every earlier mechanism. The final chapter asks the reader to migrate CampusCart and design CivicPermit through evidence-producing stages. The capstone requires architecture, experiments, failure analysis, security decisions, cost calculations, and a deployment review rather than one provider-product checklist. --- # Capstone: Designing and Validating a Cloud Application CampusCart's university approves a migration from two manually configured virtual machines to a supported cloud architecture. The existing application serves 5,000 daily users, usually completes 80 requests per second, and reaches 400 requests per second during registration. PostgreSQL stores orders. Product images live in object storage, but lifecycle and recovery rules have not been tested. One administrator account performs deployments. The team has never rehearsed a zone failure. Jordan, a graduating computer science student joining the platform team, receives a bounded assignment: produce and validate a migration design before any production data moves. The design must preserve order correctness, support registration demand, recover from one availability-zone loss, restrict deployment privilege, preserve audit evidence, and state monthly cost assumptions. A separate CivicPermit review must identify additional government-work questions without claiming compliance from architecture alone. Jordan cannot complete the assignment by drawing provider icons. The review board requires inspectable artifacts: an inventory, request-path diagram, data classification, threat analysis, capacity experiment, infrastructure definition, recovery transcript, objective specification, cost table, and decision log. Each artifact must name its producer, observation conditions, and unresolved uncertainty. ## Prerequisites and earlier lessons - All prior chapters are prerequisites because the capstone integrates history, networking, virtualization, providers, government boundaries, containers, Kubernetes, edge behavior, and operations. - Chapter 1 supplies the CampusCart and CivicPermit recurring examples. - Chapter 12 supplies service objectives, telemetry, incident evidence, least privilege, and cost units. ## Learning outcomes After completing the capstone, the reader can turn requirements into testable architecture properties, select managed or self-operated capabilities with explicit ownership, stage a migration with reversible checkpoints, conduct capacity and recovery tests, and present bounded conclusions. The reader can also identify which public-sector decisions require an authorizing official, security officer, contracting specialist, privacy counsel, or records officer. ## Observable evidence and required artifacts The first artifact is a current-state inventory produced from source repositories, host inspection, network configuration, database catalogs, identity directories, and interviews. Each row names an application executable, data store, external dependency, credential path, network listener, owner, backup, and recovery evidence. An interview statement is an assertion from a person; a configuration export is a direct observation from one software source. Jordan labels the difference. The second artifact is a request-path diagram. The diagram starts at a browser and names recursive Domain Name System resolution, authoritative answers, edge proxy, origin load balancer, network subnets, application targets, database endpoints, object storage, identity calls, and logging destinations. Every arrow names a protocol, direction, and authorization mechanism. The diagram prevents the word “cloud” from hiding a network or institutional boundary. The third artifact is a data classification table. CampusCart order rows contain names, contact details, payment-provider references, products, and timestamps. Product images are public. Application logs can accidentally contain personal information. The university data owner and privacy staff produce classification decisions; the engineer records technical locations and flows. CivicPermit adds government records, potentially sensitive personal information, retention schedules, public-records obligations, and jurisdiction-specific controls. The fourth artifact is an objective and experiment package. A load generator produces timestamped request outcomes under a committed workload definition. Edge and application instrumentation produce success, latency, and dependency evidence. A recovery exercise produces a timeline from injected failure through detected impact and restored objective. A cost export or provider calculator produces priced quantities under a named date and region. ## Mechanism: turning requirements into architecture Start with one user action: place an order. Order completion requires an accepted authenticated request, a valid price decision, one durable database commit, and a response or retry contract that does not create a duplicate order. A request identifier and database uniqueness rule can make a repeated client attempt safe under stated conditions. Object-image availability does not determine order correctness, so image delivery can use a different caching and recovery strategy. Place stateless application containers across at least two availability zones behind a health-aware load balancer. Run PostgreSQL through a managed high-availability configuration only after reading the exact failover, backup, maintenance, and replication semantics. Store product images in object storage with versioning or another recovery control. Put public static delivery behind a content delivery network. Use workload identity instead of long-lived static cloud keys where supported. The architecture still needs explicit ownership. The provider operates physical facilities and managed-product internals under contract. The CampusCart team owns application code, identity policy, network exposure, data classification, schema changes, backup requirements, recovery testing, and user objectives. A managed label changes operation distribution; the label does not transfer accountability for configuration or application correctness. Infrastructure as code gives Jordan a reviewable desired configuration. A change pipeline can validate syntax, policy, and a proposed difference before applying it. A versioned definition supports reconstruction, but the definition does not automatically capture mutable data, external identity state, certificates, or out-of-band changes. Drift detection compares actual provider configuration with declared configuration and reports a difference; an authorized reviewer decides the response. Migration uses parallel verifiable stages. Build an application image and test it without moving production data. Create networks, identities, storage, and observability. Copy a production-like sanitized dataset. Validate schema and behavior. Measure capacity. Rehearse backup restore. Rehearse one-zone loss. Establish a replication or bounded outage plan for final data movement. Lower relevant Domain Name System time to live before cutover when addresses change. Keep a tested rollback path until correctness and recovery evidence pass acceptance rules. ## Working model: capacity, availability, and recovery Chapter 10 measured 48.75 usable requests per Pod per second after reserve. A 400-request-per-second peak needs nine Pods by capacity. A one-zone-loss requirement changes the placement. With three zones and nine evenly placed Pods, losing one zone leaves six Pods with about `6 × 48.75 = 292.5` acceptable requests per second, below peak. To preserve 400 requests per second after losing one of three equal zones, each surviving pair needs at least nine Pods. An even placement therefore needs at least 15 Pods, five per zone, because losing five leaves ten. The surviving approximate capacity is `10 × 48.75 = 487.5` requests per second. The model assumes the database, load balancer, network, and remaining nodes accept the work. The model omits autoscaling delay and correlated demand changes. A lower business requirement could permit degraded peak capacity during zone loss; the decision belongs in the objective. Recovery uses two quantities. A **recovery point objective**, abbreviated RPO, bounds acceptable data loss measured backward from disruption. A **recovery time objective**, abbreviated RTO, bounds acceptable time to restore a defined capability. A five-minute RPO and 30-minute RTO do not prove that backups meet them. Restore tests must measure the newest recovered committed order and elapsed time from declaration to usable validation. Suppose a backup is taken every hour and transaction logs are archived every five minutes. The five-minute archive interval suggests a possible five-minute recovery point under successful complete delivery. The estimate assumes logs are durable, ordered, compatible, and replayable. An empirical restore establishes the achieved point for the tested failure and data set. ## Empirical test plan Use a production-like but non-sensitive dataset. Fix the image digest, database size, request mix, client timeouts, region, node type, and application configuration. Warm the cache according to a documented rule. Increase offered requests in steps through 500 per second. Record offered attempts, completed successes, failure categories, latency distributions, application replicas, processor throttling, memory, database connections, storage operations, and network transfer. At a stable 400-request-per-second condition, remove one zone's application targets using a safe test mechanism. Preserve the time of injection, load-balancer health changes, connection failures, retries, objective impact, and recovery. The model predicts that 15 properly distributed Pods retain at least ten replicas and sufficient measured application capacity. A result below the objective rejects the architecture under the test even if replica arithmetic passes. Database failover, connection concentration, client retry storms, or uneven traffic can explain the difference. Restore a database backup into an isolated environment. Validate row counts, constraints, selected checksums, most recent committed order time, application reads, and authorization. Keep the original backup immutable. The exercise rejects the recovery explanation if required logs are missing, replay fails, integrity checks differ, or elapsed time exceeds the objective. ## Worked investigation **Initial question.** Can CampusCart cut over without losing or duplicating orders? **Operating conditions.** The old database accepts writes. A new managed PostgreSQL database receives logical replication. The application has a client-generated order identifier with a unique database constraint. Domain Name System uses a 300-second TTL. The team can place the old application in read-only maintenance mode. **Evidence collected.** Replication reports zero unapplied transactions under a controlled write pause. Source and target row counts match for required tables. Checksums over stable key ranges match. A test suite retries 1,000 order requests after simulated response loss and produces 1,000 unique orders. The target database backup restores successfully in 18 minutes. Authorization tests show only the migration identity can perform replication actions. **Interpretation.** Matching counts and checksums support copied-data integrity for tested ranges. The unique order identifier directly prevents a second row with the same identifier; the rule does not prevent a caller from creating a new identifier for the same intent. Zero reported replication lag supports cutover readiness only while writes remain paused and monitoring remains accurate. **Proposed intervention and prediction.** Announce the bounded maintenance window, stop new writes, wait for confirmed replication completion, switch the application database endpoint, run read and controlled write validations, and direct traffic to the new application. Preserve the old database read-only. The prediction is that repeated requests with the same order identifier return the existing order and that row-count differences remain explained by controlled validation writes. **Observed result.** A rehearsal with 50,000 synthetic orders completes in 11 minutes. Twenty deliberately repeated requests create no duplicate rows. Source and target differences equal the documented validation rows. Reverting the application endpoint within the rehearsal window restores the old read-only application. **Bounded conclusion and remaining uncertainty.** The procedure preserved tested orders and identifiers during the rehearsal. The procedure does not prove production network behavior, human timing, third-party payment idempotency, or a cutover under unexpected writes. The final plan assigns stop authority, rollback criteria, and reconciliation owners. ## Normal case The review board accepts an architecture only after every requirement maps to a mechanism, an owner, an artifact, and a test. The team deploys by immutable digest, separates environments and identities, reviews infrastructure changes, collects user-facing evidence, rehearses recovery, and records priced units. A post-cutover review compares outcomes with acceptance thresholds and retains the former path until rollback criteria expire. ## Failure and edge cases PostgreSQL logical replication does not replicate schema definition changes, sequence state, or large objects. Roles, extensions, and job-scheduler configuration require separate migration procedures. Cache warming can make a test look faster than a cold deployment. Synthetic traffic can miss request-size and concurrency patterns. A zone label does not prove independent power, network, and dependency paths beyond the provider's published model. A rollback after new production writes can require forward reconciliation rather than a simple endpoint reversal. ## Government-work extension: CivicPermit CivicPermit is a government permit-intake application that stores personally identifiable information, must preserve audit evidence, and is designed to operate across two availability zones. The authorizing organization must define impact, data categories, jurisdiction, retention, incident reporting, personnel access, supply-chain requirements, and approved products. Amazon Web Services GovCloud (US) changes account eligibility, uses the separate `aws-us-gov` partition, requires partition-specific credentials and endpoints, and can have a different service and feature inventory. A GovCloud region does not make the application compliant. Jordan must map applicable controls to inherited provider evidence, organization procedures, and workload-specific implementation and test evidence. The CivicPermit package includes a data-flow diagram, control-responsibility matrix, identity review, encryption and key-use design, audit-event inventory, records-retention procedure, recovery exercise, incident route, and authorization decision. Contracting staff verify service terms. Security and privacy officials verify control scope. Records staff verify retention and disclosure duties. Engineers provide technical evidence and do not sign beyond their authority. ## Important qualifications A reference architecture is a hypothesis about behavior under stated conditions. Provider documentation, service agreements, and certifications define inputs and inherited evidence, while workload tests establish configured behavior. No finite test proves absence of every failure. State scope, confidence, omissions, and monitoring that can detect changed assumptions. Cost comparisons require equal work and equal reliability conditions. A managed database with higher direct price can reduce staffing or recovery risk; those effects need explicit organizational estimates. Multi-cloud designs can add portability but also duplicate identity, network, data, and operational complexity. Select a second provider only for a stated requirement that the added mechanisms satisfy. ## Common errors **“The architecture diagram is the design.”** A diagram omits acceptance conditions, ownership, data semantics, experiments, and recovery evidence. **“High availability means no maintenance window.”** Availability is a measured outcome under stated failure and change conditions. Data migration can require a bounded write pause even when the target runs across zones. **“Infrastructure as code is the backup.”** Configuration definitions do not contain application data or prove restoration. **“Passing the rehearsal proves production cutover.”** A rehearsal increases confidence under matched conditions. Production demand, data, networks, people, and third parties retain uncertainty. ## Knowledge check **Question:** Nine Pods are evenly distributed across three zones, and each Pod has 48.75 usable requests-per-second capacity. Can the application sustain 400 requests per second after one zone fails? **Answer:** No. One zone removes three Pods, leaving six. The approximate surviving capacity is 292.5 requests per second, below 400. The conclusion also assumes all other dependencies remain capable. ## Exercises **Check.** Submit a current-state inventory for a small application. Include executable, listener, data store, identity, credential source, backup, owner, and direct evidence for every row. **Practice.** Create a provider-neutral request-path diagram and responsibility matrix for CampusCart. Name protocols, directions, failure domains, policy enforcement points, artifact producers, and one test per major claim. **Challenge.** Execute the capstone in a safe cloud account or local equivalent. Submit infrastructure definitions, immutable image digest, load-test table, zone-failure timeline, database-restore transcript, security test results, objective calculation, monthly priced-quantity table, rollback plan, and a two-page bounded conclusion. Do not use production personal data. ## Authoritative sources and further reading ### Required reading - [National Institute of Standards and Technology Special Publication (NIST SP) 800-34 Rev. 1](https://csrc.nist.gov/pubs/sp/800/34/r1/final) provides contingency-planning guidance and recovery-plan structure. - [NIST SP 800-53 Rev. 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final) supports CivicPermit control mapping and responsibility analysis. ### Implementation documentation - [Kubernetes running in multiple zones](https://kubernetes.io/docs/setup/best-practices/multiple-zones/) documents topology considerations and limitations for cluster workloads. - [PostgreSQL logical-replication restrictions](https://www.postgresql.org/docs/current/logical-replication-restrictions.html) states the engine's current exclusions for schema, sequence, and large-object replication; record the production engine version before migration. ### Primary provider engineering frameworks - The official provider architecture frameworks—[AWS Well-Architected](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html), [Google Cloud Architecture Framework](https://cloud.google.com/architecture/framework), and [Microsoft Azure Well-Architected Framework](https://learn.microsoft.com/azure/well-architected/)—supply provider-specific review questions; none substitutes for workload evidence. ### Advanced and version-sensitive material - Provider database backup, replication, and failover documentation is version-sensitive. Record engine version, region, configuration, and access date in the capstone. ## Transition to continued study The capstone completes the introductory path but not professional development. Continue by operating one small application long enough to observe deploys, certificate rotation, capacity change, cost drift, dependency failure, backup restore, and an incident review. Read the exact specifications behind each tool. Repeat experiments after upgrades. Practical cloud competence grows from connecting observed user results to named programs, machines, networks, policies, and institutions under changing conditions. --- # Source catalog The textbook favors official specifications, official documentation, original papers, and primary engineering reports. Version-sensitive entries name the relevant release or access date. # Foundations Source Catalog Research check date: July 20, 2026. The catalog favors standards, official documentation, original papers, and primary product records. “Rolling” means that the publisher can update the page without issuing a new stable edition; recheck rolling pages before implementation. ## Cloud definitions and orientation | Source | Status or date | Use in the manuscript | Currency note | |---|---|---|---| | [NIST SP 800-145, *The NIST Definition of Cloud Computing*](https://doi.org/10.6028/NIST.SP.800-145) | Final, September 2011 | Five essential characteristics; IaaS, PaaS, SaaS; deployment arrangements | Durable classification, not a security certification or implementation guide | | [NIST SP 800-146, *Cloud Computing Synopsis and Recommendations*](https://doi.org/10.6028/NIST.SP.800-146) | Final, May 2012 | Benefits, limitations, reliability, performance, and security qualifications | Product details are historical; principles remain useful | | [Google Cloud Compute Engine: Regions and zones](https://cloud.google.com/compute/docs/regions-zones) | Rolling | Regional and zonal placement | Zone inventory and machine availability change | | [Microsoft Azure: Regions overview](https://learn.microsoft.com/en-us/azure/reliability/regions-overview) | Rolling | Regions, geographies, availability zones, data residency | Exact resiliency choices vary by region and product | | [AWS GovCloud (US): What is AWS GovCloud (US)?](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/whatis.html) | Rolling | Isolated government regions and shared responsibility | Compliance scope, endpoints, and product availability change | | [AWS GovCloud (US): Using the regions](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/using-govcloud.html) | Rolling | Distinct `us-gov-*` names, endpoints, and partitions | Check the exact target partition | | [Azure Government comparison](https://learn.microsoft.com/en-us/azure/azure-government/compare-azure-government-global-azure) | Rolling | Endpoint, feature, and configuration differences | Government and global feature parity changes | | [Google Cloud Assured Workloads overview](https://cloud.google.com/assured-workloads/docs/overview) | Rolling; control-package names changed June 2025 | Regulated-workload guardrails and violation monitoring | A control package does not transfer customer compliance responsibility | ## Historical primary records | Source | Status or date | Use in the manuscript | Qualification | |---|---|---|---| | [R. J. Creasy, *The Origin of the VM/370 Time-Sharing System*](https://www.ibm.com/support/pages/zvm/history/50th/vm370ori.pdf) | IBM engineering paper, 1981 | CTSS influence, CP-40/CMS, design motives, operational lineage | Retrospective primary engineering account | | [IBM z/VM history timeline](https://www.ibm.com/support/pages/zvm/history/timeline.html) | Official archive | VM/370 announcement date and linked artifacts | Timeline is a vendor-preserved selection | | [Popek and Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures”](https://doi.org/10.1145/361011.361073) | *Communications of the ACM*, July 1974 | Formal virtual-machine monitor properties | Applies to its modeled architectures; modern mechanisms extend the setting | | [AWS: Amazon S3 announcement](https://aws.amazon.com/about-aws/whats-new/2006/03/13/announcing-amazon-s3---simple-storage-service/) | March 13, 2006 | Public object-storage landmark | Original claims do not describe current S3 scope | | [AWS: Amazon EC2 beta announcement](https://aws.amazon.com/about-aws/whats-new/2006/08/24/announcing-amazon-elastic-compute-cloud-amazon-ec2---beta/) | August 24, 2006 | On-demand VM capacity and usage charging landmark | Beta scope, prices, and limits are historical | | [Google: App Engine introduction](https://cloudplatform.googleblog.com/2008/04/introducing-google-app-engine-our-new.html) | April 7, 2008 preview | Managed application-platform landmark and original quotas | Preview behavior is not current App Engine documentation | | [Microsoft: Windows Azure announcement](https://news.microsoft.com/source/2008/10/27/microsoft-unveils-windows-azure-at-professional-developers-conference/) | October 27, 2008 technology preview | Azure platform landmark | Original branding and scope are historical | ## Internet standards and measurement | Source | Current role | Use in the manuscript | Important update or exception | |---|---|---|---| | [RFC 9499, *DNS Terminology*](https://www.rfc-editor.org/info/rfc9499) | Best Current Practice 219, March 2024 | Current resolver, authoritative-server, zone, and answer terminology | Obsoletes RFC 8499; terminology guide, not a complete tutorial | | [RFC 1034](https://www.rfc-editor.org/info/rfc1034) and [RFC 1035](https://www.rfc-editor.org/info/rfc1035) | Foundational DNS architecture and format | Names, resource records, resolvers, caching | Both have many later updates; DNS uses more than one transport | | [RFC 7766, *DNS Transport over TCP*](https://www.rfc-editor.org/info/rfc7766) | Standards Track | Correction to “DNS uses only UDP” | Encrypted DNS adds other transport arrangements | | [RFC 4632, *CIDR*](https://www.rfc-editor.org/info/rfc4632) | Best Current Practice 122 | Prefix arithmetic, aggregation, longest match | Provider-reserved subnet addresses are separate product rules | | [RFC 8200, *IPv6*](https://www.rfc-editor.org/info/rfc8200) | Internet Standard 86 | IPv6 base mechanism | RFC status page lists later updates | | [RFC 9293, *TCP*](https://www.rfc-editor.org/info/rfc9293) | Internet Standard 7, August 2022 | Reliable ordered byte-stream mechanism | Obsoletes RFC 793 as the base TCP specification | | [RFC 9000, *QUIC*](https://www.rfc-editor.org/info/rfc9000) | Proposed Standard | Secure multiplexed transport over UDP datagrams | Corrects the inference that every UDP-based application is unreliable | | [RFC 9110, *HTTP Semantics*](https://www.rfc-editor.org/info/rfc9110) | Internet Standard 97, June 2022 | Version-independent HTTP meaning | Pair with RFC 9112, 9113, or 9114 for HTTP/1.1, HTTP/2, or HTTP/3 framing | | [RFC 9846, *TLS 1.3*](https://www.rfc-editor.org/info/rfc9846) | Proposed Standard, July 2026 | Current TLS 1.3 handshake and record protection | Obsoletes RFC 8446 while retaining TLS 1.3 and backward compatibility | | [RFC 5136, *Defining Network Capacity*](https://www.rfc-editor.org/info/rfc5136) | Informational | Capacity, available capacity, utilization, and interval terms | Definitions support measurement; the RFC does not promise achievable application throughput | | [curl manual](https://curl.se/docs/manpage.html) | Rolling official tool documentation | Cumulative timing fields and negotiated protocol evidence | Record `curl -V`; supported protocols depend on the build | ## Current cloud networking documentation | Source | Use | Version-sensitive point | |---|---|---| | [AWS VPC and subnet basics](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-subnet-basics.html) | A VPC spans a region and an AWS subnet belongs to one Availability Zone | Reserved addresses, IPv6 options, and defaults change | | [AWS subnet route tables](https://docs.aws.amazon.com/vpc/latest/userguide/subnet-route-tables.html) | Destination/target routes and longest-prefix selection | Available targets change | | [AWS security groups](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-security-groups.html) | Stateful interface-level allow rules | Multiple attached groups aggregate; exact managed features change | | [Google Cloud VPC overview](https://cloud.google.com/vpc/docs/overview) | Global VPC with regional subnets spanning zones | Scope differs from AWS and Azure terminology | | [Google Cloud firewall rules](https://cloud.google.com/firewall/docs/firewalls) | Stateful network-level allow and deny behavior enforced per instance | Hierarchical and policy features require current documentation | | [Azure virtual-network FAQ](https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-faq) | Regional virtual networks and zone-spanning subnets | Scope differs from AWS subnet placement | | [Azure network security groups](https://learn.microsoft.com/en-us/azure/virtual-network/network-security-groups-overview) | Ordered stateful five-tuple rules | Existing tracked connections can outlive a rule removal | ## Virtualization and containers | Source | Status | Use | Currency note | |---|---|---|---| | [Linux kernel KVM API](https://docs.kernel.org/virt/kvm/api.html) | Rolling kernel documentation; stable ABI since Linux 2.6.22 | VM, vCPU, and virtual-device creation interfaces | Query `KVM_CAP_*` capabilities; do not infer from version alone | | [Linux kernel cgroup v2 guide](https://docs.kernel.org/admin-guide/cgroup-v2.html) | Rolling kernel documentation | Hierarchical process organization, controllers, observable files | Match documentation to the installed kernel and cgroup mode | | [OCI Image Specification](https://specs.opencontainers.org/image-spec/) | Standards Track; site showed November 2025 publication | Manifests, indexes, content descriptors, layers, configuration | Record the selected specification release for implementation | | [Docker resource constraints](https://docs.docker.com/engine/containers/resource_constraints/) | Rolling official documentation | Default lack of constraints and CPU/memory limit behavior | Kernel and cgroup support affect available controls | | [Google Compute Engine overview](https://cloud.google.com/compute/docs/overview) | Rolling; last updated July 17, 2026 when checked | Current IaaS classification and KVM statement | Machine families, availability, and objectives change | | [AWS nested virtualization](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-ec2-nested-virtualization.html) | Rolling | Physical/Nitro, guest, and nested-guest layers | Supported types, processors, and regions change | | [IBM z/VM 7.4 Control Program overview](https://www.ibm.com/docs/en/zvm/7.4.0?topic=zvm-overview-control-program-cp) | Versioned product documentation | Virtual processor, storage, and device mapping | Applies to z/VM 7.4.0; select matching docs for other releases | ## Source-use rules for later revisions 1. Use RFC Editor status pages rather than an old RFC number from memory. The July 2026 replacement of RFC 8446 by RFC 9846 demonstrates why status checks matter. 2. Record provider, product, region or partition, documentation date, and configuration for a version-sensitive claim. 3. Treat launch announcements as evidence of what was announced, not evidence of current behavior. 4. Treat provider configuration listings as management-plane evidence, not proof of end-to-end request completion. 5. Preserve units, observation intervals, artifact producers, and untested alternatives with every empirical claim. --- # Part II Source Catalog **Research cutoff:** July 20, 2026. The catalog records the authoritative sources used for Chapters 5–8. Official documentation is still versioned evidence rather than a permanent fact. Recheck every region, product, quota, price, compliance scope, endpoint, cryptographic validation, and policy behavior before a production or authorization decision. ## Source-selection method Provider documentation is authoritative for current provider behavior. NIST publications and FedRAMP material are primary government sources for the federal risk and authorization discussion. Provider compliance pages establish the provider's described offering and scope; they do not establish a customer's legal obligations or authorize a customer workload. Each entry has one of two stability labels: - **Durable mechanism:** the source supports a concept that changes slowly, although wording and implementation can evolve. - **Version-sensitive:** the source describes current product inventory, defaults, limits, interfaces, assurance scope, or program procedure that must be checked again. ## Chapter 5: AWS core platform and request path 1. [AWS Regions and Availability Zones](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions-availability-zones.html) **Use:** Defines Regions, Availability Zones, regional resources, and explicit replication responsibility. **Stability:** Durable geographic and isolation mechanism; Region and zone inventory is version-sensitive. 2. [AWS Availability Zones](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-availability-zones.html) **Use:** Documents zone names, stable zone identifiers, and 2026 account-mapping qualifications. **Stability:** Version-sensitive because mappings, constrained zones, and inventory change. 3. [Routing Route 53 traffic to an Elastic Load Balancing load balancer](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-elb-load-balancer.html) **Use:** Supports the DNS alias-to-load-balancer request stage. **Stability:** Durable DNS mechanism with version-sensitive console and address-family details. 4. [Application Load Balancer introduction](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html) **Use:** Defines listeners, rules, target groups, registered targets, and multi-zone traffic distribution. **Stability:** Durable request mechanism; supported features are version-sensitive. 5. [Application Load Balancer target health checks](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html) **Use:** Documents periodic checks, thresholds, target states, and fail-open behavior when all targets are unhealthy. **Stability:** Version-sensitive implementation documentation. 6. [Application Load Balancer target groups](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html) **Use:** Explains protocol, port, health, draining, algorithms, and routing-failover details. **Stability:** Version-sensitive product behavior. 7. [Amazon VPC subnet route tables](https://docs.aws.amazon.com/vpc/latest/userguide/subnet-route-tables.html) **Use:** Establishes that each subnet has an associated route table and that a route maps a destination to a target. **Stability:** Durable routing mechanism. 8. [Amazon EC2 Auto Scaling overview](https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html) **Use:** Defines minimum, desired, and maximum capacity, instance replacement, policies, and load-balancer integration. **Stability:** Durable capacity-control mechanism; current features are version-sensitive. 9. [Amazon RDS Multi-AZ DB instance deployments](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) **Use:** Qualifies standby and failover behavior for the selected managed database topology. **Stability:** Version-sensitive because engines, deployment types, and behavior change. 10. [Amazon CloudWatch metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html) **Use:** Identifies AWS-produced and customer-published measurements and the OpenTelemetry and classic metric paths documented in July 2026. **Stability:** Version-sensitive interface and limit documentation; measurement units and aggregation remain durable concepts. 11. [AWS Well-Architected Reliability Pillar](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html) **Use:** Connects capacity, failure management, quotas, recovery, and testing. **Stability:** Durable engineering guidance with periodically revised recommendations. ## Chapter 6: Google Cloud and Azure comparison 1. [Google Cloud resource hierarchy](https://docs.cloud.google.com/resource-manager/docs/cloud-platform-resource-hierarchy) **Use:** Defines organization, folder, project, product resource, parentage, and policy inheritance. **Stability:** Durable hierarchy with version-sensitive creation and eligibility details. 2. [Google Cloud Resource Manager overview](https://docs.cloud.google.com/resource-manager/docs/resource-manager-overview) **Use:** Confirms projects as fundamental organizing entities and explains governance services. **Stability:** Durable organizational mechanism. 3. [Google Cloud Load Balancing overview](https://docs.cloud.google.com/load-balancing/docs/load-balancing-overview) **Use:** Distinguishes global and regional, internal and external, proxy and passthrough, Layer 4 and Layer 7, and network tiers. **Stability:** Version-sensitive product portfolio. 4. [Google Cloud global, regional, and zonal resources](https://cloud.google.com/compute/docs/regions-zones/global-regional-zonal-resources) **Use:** Supports the distinction between a global Google Cloud VPC network and regional subnets and products. **Stability:** Durable scope model with version-sensitive product examples. 5. [Google Cloud shared responsibility and shared fate](https://docs.cloud.google.com/architecture/framework/security/shared-responsibility-shared-fate) **Use:** Identifies provider and customer security work across infrastructure, platform, software, and function offerings. **Stability:** Durable responsibility concept; provider program descriptions can evolve. 6. [Azure Resource Manager overview](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/overview) **Use:** Explains Azure resource management, resource groups, subscriptions, and management scopes. **Stability:** Durable hierarchy and deployment mechanism. 7. [Azure load balancing and content delivery](https://learn.microsoft.com/en-us/azure/networking/load-balancer-content-delivery/) **Use:** Distinguishes Azure Front Door, Application Gateway, and Azure Load Balancer use cases. **Stability:** Version-sensitive product portfolio. 8. [Azure regions and availability zones](https://learn.microsoft.com/en-us/azure/reliability/regions-overview) **Use:** Defines Azure geographies, regions, and zone concepts. **Stability:** Durable geographic mechanism with version-sensitive support lists. 9. [Azure subscription and service limits](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits) **Use:** Demonstrates that limits attach to specific scopes and that some quotas require provider approval. **Stability:** Highly version-sensitive; never copy values into long-lived capacity assumptions. 10. [Microsoft shared responsibility in the cloud](https://learn.microsoft.com/en-us/azure/security/fundamentals/shared-responsibility) **Use:** Distinguishes customer, shared, and Microsoft work across infrastructure, platform, and software offerings. **Stability:** Durable responsibility model with periodically revised product examples. ## Chapter 7: identity, data, reliability, and cost 1. [AWS IAM policy evaluation logic](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html) **Use:** Documents authentication, request context, allow unions, limiting intersections, and explicit denies. **Stability:** Durable policy mechanism; new policy types and special cases are version-sensitive. 2. [AWS policy evaluation for same-account requests](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic_policy-eval-basics.html) **Use:** Provides concrete identity-policy, resource-policy, and explicit-deny examples. **Stability:** Durable worked mechanism subject to current IAM rules. 3. [Amazon S3 data consistency model](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel) **Use:** Establishes current strong read-after-write behavior, atomic single-key updates, and the conditions around successful writes. **Stability:** Provider guarantee that must remain tied to current documentation. 4. [Data protection in Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html) **Use:** Separates durability, availability, versioning, Object Lock, and replication. **Stability:** Durable distinctions with version-sensitive classes and objectives. 5. [CloudTrail record contents](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-record-contents.html) **Use:** Identifies event fields, timestamp origin, optionality, regions, source addresses, user agents, and errors. **Stability:** Version-sensitive schema; the page currently documents event format versioning. 6. [Understanding CloudTrail events](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-events.html) **Use:** Distinguishes management, data, network activity, and Insights events and their collection behavior. **Stability:** Version-sensitive event categories and defaults. 7. [CloudTrail event history](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html) **Use:** Documents the current recent-management-event view and explains why absence from event history is not proof that other activity did not occur. **Stability:** Version-sensitive retention and interface behavior. 8. [AWS recovery-objective guidance](https://docs.aws.amazon.com/wellarchitected/latest/framework/rel_planning_for_recovery_objective_defined_recovery.html) **Use:** Defines RTO and RPO and connects both objectives to business impact and testing. **Stability:** Durable recovery terminology. 9. [AWS understanding availability needs](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/understanding-availability-needs.html) **Use:** Separates data-plane and control-plane availability and recommends workload-specific objectives. **Stability:** Durable architecture guidance. 10. [AWS Cost Explorer](https://docs.aws.amazon.com/cost-management/latest/userguide/ce-exploring-data.html) **Use:** Documents current cost views, estimates, and data freshness qualifications. **Stability:** Version-sensitive billing interface; no textbook number should replace a current invoice or price page. 11. [Azure shared responsibility for reliability](https://learn.microsoft.com/en-us/azure/reliability/concept-shared-responsibility) **Use:** Confirms that customers select tiers, configure and verify backups, meet service-level-agreement conditions, and design resilient applications. **Stability:** Durable responsibility concept with version-sensitive product examples. ## Chapter 8: AWS for government work 1. [FIPS 199: Standards for Security Categorization](https://csrc.nist.gov/pubs/fips/199/final) **Use:** Primary federal standard for categorizing information and information systems by potential confidentiality, integrity, and availability impact. **Stability:** Durable normative standard; check NIST for superseding publications or policy. 2. [NIST SP 800-37 Revision 2](https://csrc.nist.gov/pubs/sp/800/37/r2/final) **Use:** Primary source for the Risk Management Framework lifecycle, assessment, authorization, and continuous monitoring. **Stability:** Durable normative guidance; check NIST for revisions. 3. [NIST SP 800-53 Revision 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final) **Use:** Primary security and privacy control catalog. The NIST page identifies later minor releases and supplemental material. **Stability:** Version-sensitive release; control concepts remain durable. 4. [FedRAMP Authorization Act: agency roles and responsibilities](https://www.fedramp.gov/2026/authority/law/agencies/) **Use:** Identifies the current statutory responsibilities of agencies and the limits of reusable provider authorization evidence. **Stability:** Version-sensitive legal and policy reference. 5. [FedRAMP 2026: Using a Certified Cloud Service](https://www.fedramp.gov/2026/agencies/use/) **Use:** Primary current guidance distinguishing reusable provider certification material from an agency information-system risk decision and ATO. **Stability:** Highly version-sensitive program procedure. 6. [FedRAMP Rev. 5 agency authorization](https://www.fedramp.gov/rev5/agency-authorization/) **Use:** Describes the agency authorization path and the need to define a cloud offering's authorization boundary. **Stability:** Version-sensitive program process. 7. [What is AWS GovCloud (US)?](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/whatis.html) **Use:** Documents AWS's intended workload categories, isolation, U.S.-citizen administration of the AWS boundary, FIPS endpoints, and customer access responsibility. **Stability:** Version-sensitive provider description. 8. [AWS GovCloud compared with standard AWS Regions](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/govcloud-differences.html) **Use:** Documents credentials, endpoints, authentication separation, billing association, network isolation, and service differences. **Stability:** Version-sensitive implementation and account documentation. 9. [AWS GovCloud sign-up](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/getting-started-sign-up.html) **Use:** Documents current eligibility and separate-organization requirements. **Stability:** Highly version-sensitive eligibility and onboarding process. 10. [AWS GovCloud standard-account linking](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/getting-started-standard-account-linking.html) **Use:** Establishes the one-to-one standard-account association and separate AWS Organizations structures. **Stability:** Version-sensitive account mechanism. 11. [Accessing AWS GovCloud](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/accessing-govcloud.html) **Use:** Documents distinct credentials and current access methods and warns that cryptographic-library choices still matter. **Stability:** Version-sensitive access documentation. 12. [AWS GovCloud ARNs](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/using-govcloud-arns.html) **Use:** Documents the `aws-us-gov` partition string and Region codes. **Stability:** Durable naming mechanism, subject to current provider documentation. 13. [AWS GovCloud service endpoints](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/using-govcloud-endpoints.html) **Use:** Documents current control-plane endpoints and FIPS 140-3 qualifications. **Stability:** Highly version-sensitive endpoint list. 14. [Services in AWS GovCloud Regions](https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/using-services.html) **Use:** Product-by-product differences, availability, and export-controlled-content guidance. **Stability:** Highly version-sensitive catalog. 15. [AWS services in scope for FedRAMP](https://aws.amazon.com/compliance/services-in-scope/FedRAMP/) **Use:** Dated provider table for service status within the Moderate and High boundaries. The researched page reported a June 18, 2026 update. **Stability:** Highly version-sensitive compliance scope. 16. [AWS shared responsibility model](https://aws.amazon.com/compliance/shared-responsibility-model/) **Use:** Provider source for infrastructure responsibility and customer duties that vary by selected product. **Stability:** Durable responsibility division with version-sensitive examples. 17. [NIST Cryptographic Module Validation Program](https://csrc.nist.gov/Projects/Cryptographic-Module-Validation-Program) **Use:** Primary source for active validation status, federal use, and the FIPS 140-2 to FIPS 140-3 transition. **Stability:** Highly version-sensitive validation and transition status. 18. [NIST CMVP frequently asked questions](https://csrc.nist.gov/Projects/Cryptographic-Module-Validation-Program/FAQs) **Use:** Primary clarification that validation of a module does not assure correct product use of the module. **Stability:** Durable qualification with version-sensitive program details. ## Known research limitations - Provider documentation states provider behavior and scope; independent assessors and agency officials still evaluate CivicPermit. - Pricing examples in Chapter 7 are explicitly invented teaching assumptions. No example price should be used for procurement. - Restricted FedRAMP authorization-package content was not accessed or reproduced. Chapter 8 teaches how to index approved evidence without claiming access to restricted material. - Legal conclusions about PII, CUI, International Traffic in Arms Regulations, criminal justice information, tax information, defense impact levels, or agency-specific overlays depend on facts outside this textbook. The chapter names the decision process and relevant specialists instead of giving a legal guarantee. --- # Part III and IV sources ## Containers and image execution - [Docker documentation](https://docs.docker.com/) is the current product reference for image builds, runtime inspection, storage, and networking. The manuscript separates Docker behavior from OCI standards. - [Open Container Initiative image specification](https://github.com/opencontainers/image-spec) and [runtime specification](https://github.com/opencontainers/runtime-spec) are the primary interoperability specifications. - [Linux kernel control group v2](https://docs.kernel.org/admin-guide/cgroup-v2.html) is the primary kernel documentation for cgroup accounting and control. ## Kubernetes - [Kubernetes documentation](https://kubernetes.io/docs/) supplies the current and versioned architecture, workload, networking, storage, security, and operations references. The current site exposed version 1.36 in July 2026; readers must select the version matching their cluster. - [Kubernetes API reference v1.36](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/) supplies exact fields matching the current documentation version cited in July 2026. - [etcd documentation](https://etcd.io/docs/) describes the consistent store commonly used by the Kubernetes control plane. ## Network edge and protocols - [Cloudflare DNS documentation](https://developers.cloudflare.com/dns/) and [CDN reference architecture](https://developers.cloudflare.com/reference-architecture/architectures/cdn/) supply current product behavior and architectural claims. - [RFC 1034](https://www.rfc-editor.org/rfc/rfc1034), [RFC 1035](https://www.rfc-editor.org/rfc/rfc1035), [RFC 9111](https://www.rfc-editor.org/rfc/rfc9111), and [RFC 4271](https://www.rfc-editor.org/rfc/rfc4271) provide protocol foundations for DNS, HTTP caching, and Border Gateway Protocol routing. ## Production practice and capstone - [OpenTelemetry specifications](https://opentelemetry.io/docs/specs/otel/) define telemetry data and application programming interfaces. Signal and language maturity remain version-sensitive. - [Google Site Reliability Engineering](https://sre.google/sre-book/table-of-contents/) is a primary engineering report for service objectives, monitoring, error budgets, and incident practice. - [NIST Special Publications](https://csrc.nist.gov/publications/sp) supply authoritative United States government guidance for controls and contingency planning. Applicability depends on the organization and authorization context. - [AWS Well-Architected](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html), [Google Cloud Architecture Framework](https://cloud.google.com/architecture/framework), and [Microsoft Azure Well-Architected Framework](https://learn.microsoft.com/azure/well-architected/) provide provider-specific architecture review material.