Kubernetes Architecture and Operations

At 08:42 on registration day, CampusCart receives 390 checkout requests per second. Elena, the engineer on call, sees two application instances stop answering. The checkout address still resolves, and most students continue receiving responses. Elena opens a terminal and runs kubectl get pods, kubectl get deployment, and kubectl get events against the bookstore's managed Kubernetes cluster.

The command output shows eight requested application replicas, six ready Pods, two terminating Pods, and two newly pending Pods. A Deployment object still says eight replicas should exist. The cluster scheduler has not yet placed the pending Pods because the available worker nodes lack requested memory. A node autoscaler asks the cloud provider for another virtual machine. Four minutes later, the kubelet on that machine reports the new Pods ready, and the Service begins sending connections to them.

The recovery involves stored objects and repeated control loops, not one central program that directly executes every instruction. A user submits a desired condition through the Kubernetes application programming interface. Controllers compare stored desired conditions with observed conditions. A scheduler chooses nodes for unscheduled Pods. A kubelet on each selected node asks a container runtime to start containers. Network and storage plugins connect external capabilities. Each participant produces different evidence.

Prerequisites and earlier lessons

• Chapter 3 supplies packet paths, addresses, ports, load balancing, and Domain Name System behavior.
• Chapter 4 supplies virtual machines, regions, zones, and shared responsibility.
• Chapter 9 supplies images, containers, control groups, volumes, and process evidence.
• The reader should understand JavaScript Object Notation, Hypertext Transfer Protocol requests, and eventual completion of asynchronous work.

Learning outcomes

After completing the chapter, the reader can follow a Kubernetes object from an application programming interface request to a running container, distinguish a Pod, Deployment, and Service, interpret common kubectl evidence, calculate basic replica capacity, and diagnose a pending or unready workload without assuming that self-healing preserves correctness. The reader can also state which cluster responsibilities remain with an application team when a cloud provider operates the control plane.

Observable evidence

kubectl is a client. The command serializes a request and sends it to the Kubernetes application programming interface server identified by the active kubeconfig context. The server authenticates the caller, authorizes the requested action, applies admission rules, validates the object, and reads or writes persistent cluster data. A successful command establishes that the server accepted or returned a representation at a particular time. The output does not prove that every controller has completed subsequent work.

kubectl get deployment campuscart-api -o yaml returns the Deployment's desired replica count, Pod template, labels, current status, and metadata. The application programming interface server produces the response from stored object data. metadata.generation changes when the desired specification changes. status.observedGeneration reports the generation processed by the relevant controller. A lower observed value supplies direct evidence that reconciliation has not caught up.

kubectl get events --sort-by=.metadata.creationTimestamp returns Event objects written by cluster participants. A scheduler can write FailedScheduling; a kubelet can write image-pull or mount failures. The selected sort field orders Event-object creation, not necessarily every underlying occurrence; aggregated Event series can report separate event and last-observed times. Kubernetes Events are best-effort supplemental evidence. Event retention and aggregation are implementation settings, so absence of an old event does not prove that the event never occurred. Node-agent logs, container logs, metrics, and cloud-provider activity records may supply additional evidence.

kubectl describe pod combines object fields and related events into a human-readable report. A container status with state.waiting.reason: ImagePullBackOff establishes that the kubelet is delaying another pull attempt. The reason does not by itself distinguish a nonexistent digest, denied registry credential, or unreachable registry. The detailed message and an independent registry test refine the conclusion.

Mechanism and terminology in context

A Kubernetes Pod is the smallest ordinary scheduling unit. One Pod can contain one or more containers that share a network namespace and declared storage volumes. Containers in one Pod reach one another through localhost. The shared fate is important: Kubernetes schedules the Pod to one node and replaces the Pod as a unit. A Pod address is not a durable application address because replacement can create a new Pod with a new address.

A Deployment declares a desired template and replica count for replaceable Pods. The Deployment controller manages ReplicaSets, and a ReplicaSet controller creates or deletes Pods to approach the requested count. During an update, old and new ReplicaSets can coexist according to surge and unavailable limits. The controller does not copy a running process. The controller creates new Pods from the new template and later removes old Pods.

A conventional selector-backed ClusterIP Service gives selected Pods a stable virtual address, and cluster Domain Name System software normally gives the Service a stable name. The Service selector matches Pod labels. EndpointSlice objects can list ready, unready, serving, and terminating endpoints; the ordinary data path selects endpoints considered ready for new traffic. A node networking implementation or another data-plane implementation directs connections addressed to the virtual address toward an endpoint. Headless Services do not receive the usual virtual address, and ExternalName Services return a Domain Name System alias without selecting Pods. A Service does not inspect whether an application response is semantically correct.

The control plane includes the application programming interface server, a data store commonly implemented by etcd, a scheduler, and controller managers. The scheduler selects a node for an unbound Pod by filtering impossible placements and scoring feasible nodes. The scheduler writes a binding; the scheduler does not start the container. A kubelet on the chosen node observes the assigned Pod and calls a Container Runtime Interface implementation. Container, network, and storage interfaces let different implementations supply execution, connectivity, and volumes.

Kubernetes uses reconciliation: a controller repeatedly compares desired object fields with observed cluster conditions and performs bounded actions to reduce a difference. Reconciliation is asynchronous. A successful Deployment update means the desired object was accepted, not that the rollout is healthy. kubectl rollout status, readiness, application metrics, and user-facing probes provide later evidence.

Kubernetes self-healing can restart a failed container, replace a failed Pod, and stop routing through an unready endpoint. The mechanism restores declared quantity and health conditions. The mechanism cannot restore data that existed only in a deleted writable layer, determine that a wrong price is incorrect, or make two zones independent when both depend on one database.

Working model: replica capacity

CampusCart normally completes 80 requests per second and reaches 400 requests per second during registration. A controlled load test shows that one Pod sustains 65 completed requests per second while keeping the 95th-percentile latency below 300 milliseconds. The team reserves 25 percent of theoretical Pod capacity for measurement error, uneven distribution, and short spikes.

usable requests per Pod per second = 65 × (1 - 0.25) = 48.75

required replicas = ceiling(400 / 48.75) = 9

The numerator counts completed CampusCart requests during one second. The denominator counts acceptable-latency completions by one Pod during one second after the reserve. The model assumes requests resemble the load test, dependencies can accept the combined work, Pods receive their requested processor and memory, and traffic distribution is sufficiently even. The model omits startup time, zone failures, retries, long-lived connections, and database contention. An availability requirement may add replicas and topology constraints beyond the capacity result.

A common Horizontal Pod Autoscaler input such as processor utilization is an indirect signal. If a checkout waits on PostgreSQL, request latency can rise while processor utilization remains low. A request-concurrency or queue-length signal may describe demand better, but the signal must have bounded cardinality, stable collection, and a tested relationship to safe capacity.

Empirical test

Deploy CampusCart to a disposable cluster with three replicas and fixed Pod requests. Send a reproducible mix of catalog and checkout traffic. Increase offered requests from 50 to 250 per second in 25-request steps while preserving database contents, client concurrency rules, image digest, node type, and region. Record completed requests per second, error fraction, latency percentiles, Pod processor time, memory, restarts, and database connections.

The capacity model predicts near-linear completed work before a shared dependency saturates. A plateau below 146 acceptable requests per second for three replicas rejects the isolated-Pod estimate under cluster conditions. Database connection saturation, network limits, load-generator capacity, or throttled processor time provide alternative explanations. Repeat one point after adding replicas. An unchanged completion ceiling implicates a shared dependency more strongly than Pod count.

Worked investigation

Initial question. Why do four new CampusCart Pods remain pending after a replica increase from six to ten?

Operating conditions. The cluster has three worker nodes. Each node offers 4 gibibytes of schedulable memory after operating-system and cluster reservations. Existing Pods request a combined 3.4 gibibytes on each node. Each new CampusCart Pod requests 1 gibibyte. Pod anti-affinity prefers but does not require different nodes.

Evidence collected. Deployment status reports ten desired replicas and six available replicas. Four Pod specifications have no spec.nodeName. Scheduler Event objects report 0/3 nodes are available: 3 Insufficient memory. Node descriptions show the allocatable and already requested quantities. Actual memory-use metrics show only 2.1 gibibytes per node.

Interpretation. The missing node name and scheduler event directly establish a placement failure caused by requested, not observed, memory. Kubernetes schedules against requests because current use can change after placement. Low observed use does not create schedulable requested capacity.

Proposed intervention and prediction. Review measured high-percentile memory under representative load, set a justified 700-mebibyte request while retaining a 1.2-gibibyte limit, and add one node before registration peaks. Each existing node can then place exactly one additional 700-mebibyte Pod under the stated 4-gibibyte allocation: three new Pods place on the old nodes, and the fourth waits until the new node is ready. The new node also supplies failure and rollout margin. The change must not rely on memory compression or swapping unless the node policy explicitly supports and tests those mechanisms.

Observed result. A staging rollout with the revised request schedules ten Pods across four nodes. A load test at 400 offered requests per second completes 397 per second, reports a 0.75 percent application-error fraction, and keeps peak Pod memory below 610 mebibytes. The team separately defines a degraded one-node-loss objective of at least 99 percent successful requests at 300 offered requests per second. Draining a node that hosts three Pods leaves seven ready replicas and completes 298 requests per second during the failure-mode test.

Bounded conclusion and uncertainty. Requested-memory pressure explained the pending Pods under the recorded scheduler conditions. Revised requests and one node enabled the tested replica count. The experiment does not establish safety for a memory leak, a whole-zone loss, or a database limit. Production alerts must compare working-set growth, out-of-memory events, and user-facing results.

Normal case

The application team submits a Deployment pinned to an image digest, a Service selecting the Deployment's labels, resource requests based on measurements, and readiness and startup probes. The managed provider operates the application programming interface server and etcd. The team still owns the workload manifest, access policy, image security, application correctness, data design, and user-facing objectives. A controlled rollout creates new Pods, admits ready endpoints, and removes old Pods within stated surge and unavailability bounds.

Failure and edge cases

A readiness probe can succeed before the application can complete a checkout because a shallow endpoint checks only the local event loop. A liveness probe can amplify an overloaded dependency by repeatedly restarting callers. A PodDisruptionBudget constrains voluntary disruption; the object does not prevent involuntary node failure. A two-replica Deployment can still place both Pods in one zone unless topology rules and node supply prevent that placement. A CrashLoopBackOff label describes delayed restart behavior, not one root cause.

A network policy has effect only when the installed networking implementation enforces the policy. A Secret object is not automatically encrypted in every cluster configuration, and base64 representation is not encryption. Cluster administrator access can bypass many workload controls. Managed Kubernetes reduces control-plane operation but does not remove version upgrades, application programming interface deprecations, node images, add-ons, cost, or incident response.

Important qualifications

The official documentation viewed in July 2026 presents Kubernetes version 1.36 as current and retains versioned documentation for earlier releases. Clusters commonly lag the newest release. Always read the documentation matching the server version returned by kubectl version, and check provider-supported version and deprecation schedules.

Declarative objects do not make every operation idempotent. An application can receive a repeated request during a retry or rollout. The application must provide appropriate request identifiers, transactions, or deduplication. Kubernetes availability depends on external identity, registry, network, storage, Domain Name System, and cloud-provider mechanisms.

Common errors

“A Deployment runs containers.” The reasoning collapses several actors. The Deployment controller manages ReplicaSets; the scheduler binds Pods; the kubelet and container runtime start containers.

“A Service is a cloud load balancer.” A Kubernetes Service is an application programming interface object with several types. A LoadBalancer Service can cause provider integration to create an external load balancer, while a ClusterIP Service remains internal.

“Current memory use determines scheduling.” The default scheduler primarily considers declared requests and constraints. Current use can influence other controllers but does not erase admitted requests.

“Self-healing makes the application reliable.” Replacement restores declared infrastructure conditions. Application data, dependency independence, correct probes, safe retries, and tested failure behavior determine the user result.

Knowledge check

Question: The application programming interface server accepts a Deployment update, but status.observedGeneration remains lower than metadata.generation. What does the evidence establish?

Answer: The server accepted and stored a newer desired specification, while the Deployment controller has not yet reported processing that generation. The evidence does not yet establish a failed rollout or a healthy rollout.

Exercises

Check. Create a local learning cluster, submit a two-replica web Deployment, and preserve the Deployment, ReplicaSet, Pod, Service, and EndpointSlice YAML. Annotate which actor created each object and which fields link the objects.

Practice. Delete one Pod while watching Pods and Events with timestamps. Submit the command transcript and a causal timeline from deletion request through replacement readiness. Identify direct observations and inferences separately.

Challenge. Reproduce the capacity experiment with a safe test application. Submit the load-generator configuration, offered and completed requests per second, latency percentiles, Pod requests, node allocatable quantities, and a graph or table. State the first shared bottleneck and one result that would reject your explanation.

Authoritative sources and further reading

Required reading

• Kubernetes components names control-plane and node participants; use the documentation version matching the cluster.
• Kubernetes Pods defines the scheduling unit, shared context, and Pod lifecycle.
• Kubernetes Service defines stable network exposure and EndpointSlices.

Implementation documentation

• Kubernetes self-healing states the recovery behaviors and their scope.
• Kubernetes API reference v1.36 supplies exact fields matching the current documentation version cited in July 2026.

Primary engineering reference

• etcd documentation explains the consistent key-value store commonly used for Kubernetes data.

Advanced and version-sensitive material

Kubernetes feature gates, application programming interface fields, deprecations, and managed-provider support schedules change by release. Match documentation to kubectl version server output rather than assuming the newest website version.

Transition

Kubernetes controls workloads inside and around clusters. A public request still needs global name resolution, Internet routing, Transport Layer Security termination, denial-of-service protection, caching, and an origin path. Chapter 11 follows a request through Cloudflare's authoritative Domain Name System and edge proxy, then connects those mechanisms back to cloud networks and Kubernetes origins.