Identity, Data Protection, Reliability, and Cost

Prerequisites and outcomes

Prerequisites. Read the provider request-path chapters and the earlier lessons on public-key cryptography, Hypertext Transfer Protocol (HTTP), databases, and processes. The Amazon Web Services (AWS) chapter explains a Region, an Availability Zone, a role, an object-storage request, and a PostgreSQL transaction. The provider-comparison chapter explains why product names do not establish equivalent behavior. A direct-link reader should know that application code calls cloud application programming interfaces (APIs) and that those calls can succeed, fail authorization, time out, or incur measured usage.

After this chapter, you can:

• distinguish authentication from authorization and calculate effective access through allow, intersection, and explicit-deny rules;
• explain how encryption, object versioning, replication, database standby capacity, and backups address different failure modes;
• define recovery point objective and recovery time objective with units;
• connect metrics, logs, audit events, and billing line items to the program that produces each artifact; and
• create a monthly cost estimate whose assumptions can be tested against a provider bill.

Examples use Amazon Web Services (AWS) names when exact policy or artifact behavior matters. Google Cloud and Microsoft Azure provide corresponding identity, key-management, audit, monitoring, backup, and billing capabilities, but their evaluation rules and schemas require provider-specific verification. Product interfaces and prices are current only when checked; this chapter's July 2026 date is not a price guarantee.

A permit attachment becomes unreadable

CivicPermit accepts government permit applications. A submitted application contains personally identifiable information (PII), a PostgreSQL row, and attachment objects such as site plans. The application runs across two Availability Zones. The operations team expects either zone to handle intake if the other zone becomes unavailable, and the agency requires an audit trail for administrative changes.

During a controlled zone-failure exercise, application capacity in the second zone starts receiving traffic. New permit rows reach PostgreSQL, but attachment downloads return HTTP 500. The API retries each failed object read three times. Debug logging is temporarily enabled, so every attempt writes a large structured entry. The exercise lasts 40 minutes.

The security engineer needs to answer four questions. Which principal attempted the read? Did an authorization rule deny the read? Did the attachment bytes still exist? How much recovery and diagnostic activity did the provider meter? A single “access denied” screenshot cannot answer all four questions.

Records and measurements with named producers

The CivicPermit API produces an application entry for each download attempt. Useful fields include a correlation identifier, permit identifier represented by a non-PII internal value, operation, workload-role name, object key, attempt number, duration in milliseconds, and error class. The application must exclude raw PII and tokens because centralizing secrets in diagnostic storage creates another disclosure path.

AWS CloudTrail produces an event for supported account activity. A CloudTrail event can identify eventTime, eventSource, eventName, awsRegion, request parameters, principal information within userIdentity, source address, and an error code. AWS documents eventTime as the completion time reported by the host serving the API endpoint. Management events, data events, network activity events, and Insights events are distinct categories. Event history provides recent management events, but object-level data activity requires the relevant data-event collection. Therefore absence from default event history does not prove that an object read never happened.

Amazon CloudWatch receives AWS-produced measurements and customer-published measurements. A measurement such as object request count, application error count, database connections, or log-ingestion bytes has a producer, unit, timestamp, and identifying dimensions or labels. A 5-minute average cannot establish a one-second spike. A counter of failed calls cannot establish which principal made them unless another artifact supplies identity.

AWS Cost Explorer and billing exports derive line items from provider metering and pricing rules. Cost Explorer commonly shows usage through the previous day, and upstream billing data can arrive or change later. A current-month number is therefore an estimate, not an instantaneous packet counter. Resource tags can help allocate charges only when the provider supports the tag for that line item, the tag is activated where required, and the resource was tagged during the measured use.

Identity as a request decision

An operator signs in as a human identity. Application code should normally use a workload identity rather than copied human credentials. The identity system authenticates a principal by establishing which human or workload is making the request. The provider then authorizes a specific action against a specific target by evaluating request context and applicable policies. Authentication can succeed while authorization denies kms:Decrypt against one key.

AWS Identity and Access Management (IAM) documentation provides a useful concrete policy mechanism. Within the applicable evaluation context, identity-based and resource-based permissions can contribute allows. A permissions boundary limits an identity policy through intersection. AWS Organizations service control policies can further limit permissions for organization members. An explicit deny that matches the request overrides an allow.

An ordered request evaluation is safer than one set equation. First identify the exact principal, action, target, account relationship, request conditions, and policy types. Next check every applicable policy for an explicit deny; a matching explicit deny ends the request with denial. Without an explicit deny, locate an applicable allow and then determine whether a permissions boundary, session policy, or organization service control policy limits the principal under the documented case. Resource-based grants behave differently for an Identity and Access Management user Amazon Resource Name, role Amazon Resource Name, and role-session Amazon Resource Name, so a generic intersection formula would give some beginners the wrong result. AWS's current policy-evaluation and permissions-boundary documentation remains the authority for a real request.

In CivicPermit, the zone-two workload obtains temporary credentials for CivicPermitRuntimeRole-v2. The object-storage policy permits reading the attachment. The encryption key policy names only the old role, CivicPermitRuntimeRole-v1. The object request can locate ciphertext while the decrypt operation is denied. “Object exists” and “principal may recover plaintext” are separate claims.

Short-lived role credentials reduce the exposure of a copied secret because credentials expire and can be issued to a runtime identity. Temporary credentials do not repair an overbroad role. Least privilege means granting the actions and targets required for the workload under stated conditions, then testing both intended success and intended denial.

Data protection addresses named failures

Transport Layer Security (TLS) protects data moving between named endpoints while the connection uses an approved configuration. Storage encryption protects persisted bytes according to the chosen storage and key arrangement. Neither mechanism prevents an authorized application from writing incorrect content or returning PII to the wrong authenticated user.

Envelope encryption supplies a common causal path. A key-management product protects a key-encryption key. The storage or application path uses a data key to encrypt attachment bytes, stores ciphertext plus required metadata, and protects the data key under the key-encryption key. A later read needs ciphertext, encryption metadata, and an authorized decrypt operation. Deleting or disabling the required key can make intact ciphertext unrecoverable. Key policy, rotation, backup, and deletion controls therefore belong in the data-recovery design.

Amazon S3 currently provides strong read-after-write consistency for successful object PUT and DELETE operations in all AWS Regions. A successful write followed by a read returns the written object under the documented conditions. Consistency does not mean immortality. An authorized delete can remove the current object. Versioning can preserve prior versions, Object Lock can apply retention controls, and replication can copy objects according to configured rules. Each control addresses a different event.

A database standby improves service continuity when the primary database instance or location fails. A standby is not an independent historical backup when application code corrupts every replicated row. A backup preserves an earlier recovery point. A restored backup can still be useless if nobody has tested credentials, schema compatibility, key access, network routes, and application startup.

The recovery point objective (RPO) is the maximum acceptable time between the latest usable recovery point and the disruption. The unit is time, such as 5 minutes of accepted permit updates. The recovery time objective (RTO) is the maximum acceptable delay between disruption and restored capability, also measured in time. An RPO of 5 minutes does not promise zero lost records. An RTO of 30 minutes does not state how fresh the restored data will be. Both objectives come from mission impact and require a test.

For CivicPermit, two Availability Zones address a zonal infrastructure failure only when intake capacity, routing, database behavior, key access, and dependencies work from either zone. Two instances placed in one zone do not meet the same failure condition. Two-zone replication also does not address a Region-wide disruption or a destructive credential used in both zones.

Reliability needs observable targets

Availability is a completed-capability fraction over a stated interval, not a product adjective. If CivicPermit defines one qualifying minute as available only when a resident can submit a valid permit and receive a durable receipt, then:

For a 30-day month containing 43,200 minutes, a 99.9-percent internal availability target permits 43.2 unavailable minutes under that definition. Maintenance exclusions, partial failures, and measurement locations must be stated. A provider service-level agreement can use a different formula and can specify eligibility conditions or service credits. An architecture should use the workload's mission measure rather than silently substituting a provider contract measure.

Reliability evidence includes a request success ratio, latency distribution, healthy-target count, replication condition, backup completion, restore duration, and application-level receipt verification. The load balancer produces target health. The backup product produces a job status. The application produces a durable-receipt test. A successful backup job establishes that a backup artifact was created under the product's checks; only a restore exercise tests whether CivicPermit can use the artifact within its RTO.

A monthly workload ledger

Cloud cost begins as measured quantity multiplied by a price for that quantity. Use a consistent billing interval and preserve dimensions such as region, tier, operation class, purchase commitment, and data-transfer direction. The following ledger is an illustrative teaching estimate, not an AWS, Google Cloud, or Azure quote.

Assumptions for one 730-hour month:

• six application instances run continuously at an assumed $0.12 per instance-hour;
• a two-zone managed PostgreSQL configuration is represented by an assumed $1.20 per database-hour;
• attachments occupy 2,048 gigabyte-months at an assumed $0.023 per gigabyte-month;
• Internet egress is 500 gigabytes at an assumed $0.09 per gigabyte;
• diagnostic ingestion is 120 gigabytes at an assumed $0.50 per ingested gigabyte; and
• taxes, support, free allowances, request charges, discounts, replicas, backup overage, inter-zone transfer, and tiered pricing are omitted.

The estimate is:

| Item | Quantity | Assumed price | Calculated cost | |---|---:|---:|---:| | Application compute | 6 instances × 730 h = 4,380 instance-hours | $0.12/instance-hour | $525.60 | | Managed PostgreSQL | 730 database-hours | $1.20/database-hour | $876.00 | | Attachment storage | 2,048 GB-month | $0.023/GB-month | $47.10 | | Internet egress | 500 GB | $0.09/GB | $45.00 | | Diagnostic ingestion | 120 GB | $0.50/GB | $60.00 | | Estimated total | | | $1,553.70 |

The ledger predicts which quantities deserve attention. Cutting compute hours by 10 percent saves an estimated $52.56 under the assumptions. Cutting attachment storage by 10 percent saves about $4.71. The result does not imply that compute optimization is safe; six instances might be required for zone-failure capacity. Reliability and cost share the same configuration, so cost removal needs a failure and load test.

Make the mechanisms visible

Use a nonproduction identity, object, key, and database. Hold application build, object bytes, region, runtime placement, and policy documents constant. First allow the workload identity to store and retrieve one uniquely hashed test object. Preserve the object checksum, audit event, application correlation identifier, and request result. Then add a narrowly scoped deny for decrypting only the test key, wait for the documented policy propagation behavior, and repeat the read.

Expected evidence shows successful authentication, a denied decrypt action, intact ciphertext, and an application error. Remove the test deny through the approved change path and confirm recovery. A result in which plaintext remains readable would reject the claim that the selected key controls that object's read path. Alternative explanations include cached plaintext, a different key, an unlogged data-event category, a policy that did not match the request, or application access through another principal.

Extend the test by deleting the current version of a disposable versioned object and restoring the prior version. Record recovery-point age in minutes and restoration time in minutes. Compare both observations with stated RPO and RTO. Never perform the exercise against production PII.

Worked investigation: denied decrypts and diagnostic cost

Initial question. Why did attachment downloads fail only after zone-two takeover, and why did diagnostic ingestion increase during the exercise?

Operating conditions. CivicPermit ran across two Availability Zones. Zone two started revision v2, which assumed CivicPermitRuntimeRole-v2. Each download attempted one object read plus as many as three retries. Debug entries averaged 18 kilobytes per attempt. The exercise sent 25 attachment downloads per second for 40 minutes.

Evidence collected. Application entries showed four attempts per user download and a key-decrypt authorization error. CloudTrail data collection showed object GET attempts against the expected key, while key-management events showed AccessDenied for the v2 role. Object inventory and checksums showed that ciphertext remained. Deployment history identified the role-name change. Monitoring showed no database saturation. Billing usage later reported increased request and log-ingestion quantities.

Interpretation. The deployment changed the principal identifier without changing the key policy. Retries repeated a deterministic authorization failure. Debug entries multiplied the metered log bytes without improving availability.

Proposed intervention and prediction. Use a stable workload role referenced through infrastructure configuration, validate decrypt permission from both zones before shifting traffic, stop retries for deterministic authorization denial, and return a bounded application error without PII. The prediction is one denied attempt during a forced-denial test, followed by a successful read after the policy repair, with at least 75 percent fewer failure entries than the four-attempt behavior.

Observed or calculated result. The original exercise produced $25 \times 2{,}400 \times 4 = 240{,}000$ attempt entries. At 18 kilobytes each, the application produced about 4.32 decimal gigabytes before provider framing and metadata. The repaired replay produced one entry per denied download, about 1.08 decimal gigabytes under the same simplifying assumptions, and successful reads resumed after the approved key-policy change.

Bounded conclusion. A mismatched key policy caused the reproduced zone-two read failures. Removing pointless retries reduced application-produced diagnostic bytes by 75 percent for the controlled failure path.

Remaining uncertainty. The calculation omits provider-added log bytes, compression, ingestion pricing tiers, policy propagation delay under other conditions, and a full regional recovery. The exercise also does not establish that production audit retention meets agency obligations.

Normal cases, edge cases, and qualifications

A normal request uses a temporary workload credential, obtains an allow for the exact object and decrypt action, reads ciphertext, recovers plaintext in the authorized path, and returns a response. Audit collection preserves the supported activity category. Monitoring aggregates operational measurements. Billing later attributes measured quantities according to the provider's rules.

Important limiting cases include:

• An explicit deny can override a broad allow. More attached policies do not necessarily create more access.
• Cross-account or cross-project access can require permission on both sides of the administrative boundary.
• Policy changes can take time to propagate. Immediate success or failure during a change needs a documented propagation qualification.
• Replication can faithfully copy malicious deletion or corrupt application writes unless retention or historical recovery supplies another point.
• Encryption can protect stored bytes while logs, caches, snapshots, or exports expose PII through separate paths.
• A successful failover can exceed the RTO. An automatic action is not necessarily a timely action.
• Current-month cost can change after delayed metering, credits, refunds, or invoice adjustments.
• Tags do not allocate untaggable shared charges automatically.

Common reasoning errors

“Authenticated means authorized.” Authentication names a principal. Authorization evaluates one requested action, target, and context.

“Encrypted means backed up.” Encryption changes readable plaintext into protected ciphertext. A backup preserves recoverable content at another recovery point. Losing the only ciphertext or required key still loses access.

“Multi-zone means zero downtime.” Zonal placement reduces one class of dependency. Detection, routing, capacity, database behavior, credentials, and application recovery still consume time and can fail.

“Eleven nines of object durability means every object is always available.” Durability concerns loss over time under the provider's design objective. Availability concerns successful access during an interval. Authorized deletion, policy denial, regional access failure, and application defects are separate events.

“The bill equals instances multiplied by hours.” Database, storage, operations, data transfer, monitoring, addresses, support, commitments, and taxes can contribute. Each charge needs a provider meter and a pricing dimension.

Knowledge check

Question. A backup job reports success every hour, and CivicPermit has a 15-minute RPO. Does the job schedule prove the objective?

Answer. No. An hourly backup schedule can already exceed a 15-minute RPO unless another recovery mechanism supplies more frequent usable points. Job success also does not prove restorability. Measure the age of the latest usable recovery point during a failure and test a restore, including keys, configuration, dependencies, and application verification.

Evidence exercises

Check

Write a six-row authorization table for one CivicPermit attachment. Include principal, action, target, identity allow, resource allow, limiting policy, explicit deny, and expected decision. Submit the table plus two test requests: one intended allow and one intended deny. Preserve sanitized audit evidence for both.

Practice

Create a disposable versioned object in an approved sandbox or reproduce the mechanism with a local versioned store. Save a checksum, overwrite the object, remove the current version, and recover the original. Submit commands, artifact producer, timestamps, version identifiers, checksums, recovery-point age, restoration duration, and a statement of what the test omits.

Challenge

Build a monthly workload ledger for CampusCart or CivicPermit from an official current calculator and pricing pages. Include compute hours, database topology, gigabyte-months, request operations, backup storage, data transfer by direction, and diagnostic ingestion. Preserve the calculator date, region, tier, currency, formulas, and exported estimate. Compare the estimate with one later billing export and explain every material variance without including account secrets or PII.

Authoritative sources and further reading

Required reading

• AWS IAM policy evaluation logic documents current unions, intersections, and explicit-deny behavior that qualify the simplified set expression.
• Amazon S3 data consistency model distinguishes strong read-after-write behavior from durability and recovery controls.
• AWS Well-Architected recovery objectives gives the authoritative AWS meanings of recovery time objective and recovery point objective.

Implementation and artifact documentation

• CloudTrail event record contents identifies event fields, timestamp origin, optional values, and error fields.
• Understanding CloudTrail events distinguishes management, data, network activity, and Insights events and warns that categories differ from default collection.
• Amazon S3 data protection explains versioning, Object Lock, replication, durability, and availability as separate mechanisms.
• AWS Cost Explorer explains estimate freshness and cost views; prices still require current pricing pages or a calculator.

Cross-provider reference

• Google Cloud shared responsibility and shared fate explains retained customer duties for access and data.
• Microsoft shared responsibility explains retained customer duties across infrastructure, platform, and software offerings.
• Azure shared responsibility for reliability explains customer responsibility for tier choice, backups, application resilience, and service-level-agreement conditions.

Transition

CivicPermit's technical controls create evidence that an authorization and risk decision can inspect. Government work adds institutional requirements: information categorization, control selection, assessment, authorization, continuous monitoring, personnel restrictions, and contract scope. The next chapter explains how AWS GovCloud (US), provider certifications, and inherited controls can support that work without automatically making an agency workload compliant.