Production Observability, Reliability, Security, and Cost

At 10:16 on a Tuesday, CampusCart's checkout success fraction falls from 99.95 percent to 97.8 percent. Priya, the engineer on call, receives an alert generated from completed checkout responses during a five-minute window. Processor use looks ordinary. A deployment ended twelve minutes earlier. PostgreSQL connection-wait time rose at the same moment as the errors.

Priya opens a dashboard, a deployment record, sampled distributed request traces, application logs, and database metrics. Each artifact has a producer and scope. The browser or synthetic client measures the user path. The application emits spans and logs. The database exporter reads server counters. The deployment platform records the image digest and time. Priya must not combine timestamps into a causal claim until identifiers, clocks, and mechanisms support the relationship.

Production practice turns desired outcomes into measured conditions, compares observations with explicit objectives, and changes the application under controlled risk. Reliability, security, and cost share the same request path but answer different questions. Reliability asks whether intended work completes within stated conditions. Security asks whether only authorized actions and information flows occur. Cost analysis asks which priced quantities the architecture consumes during an interval.

Prerequisites and earlier lessons

• Chapters 3 and 4 supply latency, throughput, failure domains, and shared responsibility.
• Chapters 5 through 8 supply provider identity, storage, audit, reliability, cost, and public-sector constraints.
• Chapters 9 through 11 supply container, Kubernetes, edge, and origin evidence.
• The reader should understand percentages, percentiles, identifiers, and controlled experiments.

Learning outcomes

After completing the chapter, the reader can define a measurable service level indicator and objective, calculate an error budget, distinguish logs, metrics, and distributed traces, conduct a bounded incident investigation, connect least privilege to observable authorization decisions, and calculate a basic workload cost model. The reader can also explain why one dashboard, one average, one certification, or one monthly invoice cannot establish production health.

Observable evidence

A log is an event entry produced by a named program or device. A CampusCart application log can contain a timestamp, severity, request identifier, route, result code, and error category. The application clock and logging library produce the timestamp and serialized fields. A log directly establishes what the instrumented code wrote; the entry does not automatically establish what the user received or what an uninstrumented dependency did.

A metric is a runtime measurement aggregated or reported with time and attributes. An OpenTelemetry counter can add one for each completed checkout. A histogram can group observed checkout durations. The application instrumentation produces measurements, an exporter transmits them, and a backend aggregates them. A five-minute success fraction must name numerator and denominator: successful eligible checkouts divided by all eligible completed checkouts during those five minutes.

A distributed request trace represents a request's path as spans. Each instrumented library or application creates spans for bounded operations and propagates trace context through request headers. A span reports an operation name, start and end times, attributes, status, and relationships. A missing span can mean no operation occurred, context failed to propagate, sampling omitted it, or instrumentation failed. Trace evidence is powerful but incomplete.

A cloud audit event is produced by a provider control plane when a caller performs a recorded management or data action under the product's audit configuration. Fields commonly include caller identity, action, target, time, source context, and result. Audit retention, enabled data events, and delivery delay matter. An audit log cannot record an action performed outside the instrumented control plane.

A cost and usage line item is produced by provider billing software from metered priced quantities. The line can name account, product, region, usage type, amount, price, and credits. The bill establishes charged quantities under provider rules. The bill does not allocate business value or reveal an application's causal need for the quantity without tags, workload mapping, and engineering evidence.

Mechanism and terminology in context

A service level indicator, abbreviated SLI, is a carefully defined quantitative measure of user-relevant behavior. CampusCart can define availability as successful eligible checkout responses divided by all eligible checkout attempts measured at the public edge. Eligibility rules might exclude explicitly blocked abuse but include application errors and timeouts. A service level objective, abbreviated SLO, is a target for that indicator over an interval, such as 99.9 percent successful eligible checkouts over 30 days.

An SLO differs from a service level agreement, abbreviated SLA. An agreement includes consequences when a target is missed. A private engineering target without contractual consequences is an objective, not automatically an agreement. Teams use objectives to decide whether reliability work or change risk deserves priority.

An error budget is the permitted unsuccessful fraction implied by an objective. For 99.9 percent success, the unsuccessful fraction is 0.1 percent. If CampusCart receives 2,000,000 eligible checkout attempts in 30 days, the request error budget is 2,000,000 × 0.001 = 2,000 unsuccessful attempts. A time-based approximation answers a different question and can mislead when traffic varies. The interactive model below uses minutes for intuition and states its assumption.

Alerting converts a measurement into an operator interruption. An alert should describe user risk and link to evidence, not merely report a high internal counter. A fast error-budget consumption rule can detect a severe incident quickly. A slower rule can detect a small persistent loss. Multiple windows reduce noise while retaining urgency.

Incident response is an evidence-preserving control loop. The responder declares scope, stabilizes user impact, records a timeline, tests hypotheses, and communicates bounded conclusions. Rollback is an intervention, not proof that one code change was the root cause. A rollback that restores service strengthens a causal explanation when the image digest is the controlled difference, but concurrent load or dependency recovery can still explain the result.

Security applies identity and policy at every privileged action. Least privilege means a principal receives only actions, targets, conditions, and duration required for a task. Authentication establishes an identity claim under a mechanism. Authorization evaluates whether that principal can perform one action on one target under current policy. Network reachability and authorization are separate: a blocked packet prevents the request from reaching an identity check, while an allowed packet does not grant database permission.

Encryption in transit protects bytes between named endpoints under certificate and key assumptions. Encryption at rest protects stored representation under a key-management and access model. Neither mechanism guarantees correct authorization before decryption. Key-use audit events, rotation tests, denied requests, and recovery exercises provide operational evidence.

Cost has causal drivers. Compute charges can follow allocated instance time, requested container capacity, or execution duration. Storage can charge stored byte-months, operations, retrieval, and transfer. Network transfer can depend on direction and location. Managed products can add per-request or provisioned-capacity charges. An engineer maps every line to a workload object and an interval before proposing optimization.

Working model: request error budget and cost

CampusCart has a 99.9 percent monthly checkout-success objective and expects 2,000,000 eligible attempts. The permitted unsuccessful attempts are 2,000. Suppose an incident causes 440 failures in 22 minutes. The incident consumed 440 / 2,000 = 22 percent of the monthly request error budget. The burn fraction has no time unit; the burn rate must name an interval or baseline. The incident average is 20 unsuccessful checkouts per minute during 22 minutes.

For cost, suppose nine application replicas request 0.5 virtual processor and 0.75 gibibyte each for 730 hours. A platform price model charges $0.04 per requested virtual-processor hour and $0.005 per requested gibibyte-hour. The monthly approximation is:

processor cost = 9 × 0.5 × 730 × $0.04 = $131.40

memory cost = 9 × 0.75 × 730 × $0.005 = $24.64

The $156.04 sum models only requested application compute. The model assumes constant replicas and prices, and it omits nodes, control plane, database, storage, requests, logs, support, discounts, taxes, and transfer. Provider calculators and actual billing data must replace example prices for a real decision. Lowering requests can reduce charge only when the billing product meters that quantity or when lower requests allow fewer paid nodes.

Empirical test

Instrument a staging checkout with one request identifier propagated through edge, application, and database-client operations. Send 1,000 controlled attempts and force the dependency call for every fiftieth attempt to return a terminal error. Disable automatic retries for this experiment so every injected dependency failure remains visible as one client failure. Keep image digest, database contents, client timeout, and request mix constant. Record client outcomes, edge status, application counters, trace sampling rule, structured logs, database errors, and timestamps.

The mechanism predicts exactly 20 injected client failures and correlation between those failures and named dependency spans. A server-side success indicator that still reports 100 percent rejects the indicator's ability to represent user results. A difference can arise because the injector miscounts, sampling omits failed traces, eligibility filters remove cases, or identifiers fail to propagate. A second experiment can enable one retry deliberately and predict which client failures disappear while dependency-error evidence remains.

Worked investigation

Initial question. Did the 10:04 CampusCart release cause the checkout-objective violation observed at 10:16?

Operating conditions. A rolling update replaced eight Pods with image digest B. Image A had served the previous day. Traffic remained between 75 and 90 completed requests per second. PostgreSQL has a 120-connection limit. Image B changes the connection pool from 10 to 25 connections per Pod.

Evidence collected. The deployment record reports digest B reaching all eight Pods at 10:04. PostgreSQL metrics show established application sessions rising toward the 120-session limit as new Pods overlap old Pods. Application connection-pool metrics show callers waiting before they obtain a PostgreSQL session. Application histograms show connection-acquisition latency increasing. Failed traces contain database-acquisition spans ending at a two-second timeout. Edge logs report matching Hypertext Transfer Protocol (HTTP) status 503 responses. A configuration diff names the pool change.

Interpretation. The deployment, configuration, database, trace, and edge evidence align on a mechanism: rollout overlap allowed more potential application connections than the database limit, callers waited, and the application returned 503 after its acquisition timeout. Timing alone would be insufficient; the changed limit and wait spans provide causal support.

Proposed intervention and prediction. Roll back to image A, then cap total connection demand below the database limit with per-Pod pools and rollout surge included. The immediate prediction is that waiting connections and 503 responses fall after old image-B Pods stop. The durable prediction requires a load test across maximum replica and surge counts.

Observed result. Rollback completes at 10:23. Connection waits return near zero by 10:24, and checkout success returns above 99.9 percent in the next five-minute window. A staging test with a 10-connection pool, ten replicas, and two surge Pods permits demand for 120 application sessions. Two existing administrative sessions leave capacity for only 118 application sessions under the 120-session database limit. The team lowers the pool to eight, which bounds the twelve rollout Pods at 96 application sessions and leaves 24 sessions for administration, migrations, and short overlap.

Bounded conclusion and uncertainty. Image B's pool setting caused the recorded connection pressure under the rollout conditions, and rollback restored the measured path. The investigation does not prove that every 503 shared the same cause. The final design still depends on database work duration and connection release correctness.

Normal case

CampusCart defines user-facing indicators at the edge and corroborates them with application and dependency evidence. Structured fields use bounded cardinality; raw user identifiers do not become metric labels. Every deployment records an immutable image digest. Alerts use error-budget consumption and include a runbook. Workload identities receive narrow permissions. Secrets come from an audited secret store. Cost reports map priced quantities to owners and environments.

Failure and edge cases

An average latency can improve while a small user group experiences timeouts. Percentiles retain distribution information but still require population and window definitions. Metric labels containing raw request or user identifiers can create unbounded series and high memory or billing cost. A trace sampler can omit rare failures. Synchronized clocks can still differ enough to confuse a short event order.

An administrator can grant permission through several policy paths, so reviewing one role does not establish effective privilege. A blocked public endpoint can remain reachable through an internal route. Encryption keys can be available to an overly broad runtime identity. A cost decrease can come from lower demand rather than an optimization. Compare normalized completed work and service outcomes.

Important qualifications

OpenTelemetry specifications and language implementations evolve independently. The metrics signal is stable, while newer signals can have different maturity by language and release. Read the current specification and implementation status. Telemetry can contain personal or secret data; collection, retention, redaction, and access need explicit controls.

An objective is a product and engineering decision, not a provider guarantee copied from a product page. Dependency agreements do not automatically compose into application availability. Two dependencies that each report 99.9 percent availability can fail together or at different times. Test the actual request path and model correlation.

Common errors

“Processor use is normal, so the application is healthy.” The reasoning ignores blocked dependencies, authorization failures, network loss, and semantic errors.

“The trace proves the whole request path.” The trace represents instrumented and sampled operations with propagated context. Missing spans and client-side behavior require separate evidence.

“99.9 percent means 43.2 minutes of downtime.” The time conversion assumes a 30-day interval and uniform time-based measurement. A request-based objective can produce a different error budget.

“A lower bill proves an optimization.” Demand, credits, commitments, and omitted work can lower charges. Normalize cost by a relevant completed quantity while confirming reliability and security constraints.

Knowledge check

Question: CampusCart completes 500,000 eligible checkout attempts during a week under a 99.95 percent success objective. How many unsuccessful attempts fit the objective?

Answer: The unsuccessful fraction is 0.05 percent, or 0.0005. The request error budget is 500,000 × 0.0005 = 250 unsuccessful eligible attempts for the stated week.

Exercises

Check. Define one CampusCart availability indicator. Submit the numerator, denominator, measurement point, interval, eligibility rules, artifact producer, and one user failure the indicator misses.

Practice. Add a request identifier to a three-operation test application. Preserve one client transcript, correlated structured logs, and a distributed trace. Annotate direct observations and inferences. Remove secrets and personal information.

Challenge. Conduct a safe failure injection in a disposable environment. Submit the hypothesis, controlled condition, changed condition, objective impact, timeline, rollback trigger, telemetry limitations, and bounded conclusion. Include a cost estimate for the test using named priced quantities.

Authoritative sources and further reading

Required reading

• Google Site Reliability Engineering: Service Level Objectives provides the primary engineering treatment of indicators, objectives, agreements, percentiles, and error budgets.
• National Institute of Standards and Technology Special Publication (NIST SP) 800-53 Rev. 5 catalogs security and privacy controls without asserting that one product configuration satisfies them.

Implementation documentation

• OpenTelemetry concepts and signals define current telemetry categories and context.
• OpenTelemetry metrics specification supplies the normative programming and data model.

Primary engineering report

The Google Site Reliability Engineering chapter above documents an operating method developed from production practice. Apply the method to CampusCart's measured user path rather than copying example thresholds.

Advanced and version-sensitive material

• FinOps Framework describes a current operating framework for cloud-cost accountability; provider bills and contracts remain authoritative for actual charges.
• Provider pricing pages and calculators are version-sensitive implementation documentation. Record the region, date, unit, commitment, and tax assumptions used in any decision.

Transition

Production judgment combines every earlier mechanism. The final chapter asks the reader to migrate CampusCart and design CivicPermit through evidence-producing stages. The capstone requires architecture, experiments, failure analysis, security decisions, cost calculations, and a deployment review rather than one provider-product checklist.