Resource Orchestration and State Synchronization in Edge Operating Systems

Research work by Aygul Nuryyeva student of International University for the Humanities and Development. Instructor Nurmyrat Allayarov.

How operating systems that span many machines decide where work runs, and how they keep shared state correct when the network between cloud and edge keeps breaking.

What is a distributed OS, and what is an edge OS?

A distributed operating system makes a collection of independent computers appear to the user as a single coherent system, often called a single system image. Tanenbaum and van Steen separate three designs. A true distributed OS is tightly integrated with one global resource manager (Amoeba, Sprite, Plan 9, V, Mach-based systems). A network OS leaves each node with its own OS and shares through services such as NFS or SSH. Middleware is a layer on top of heterogeneous OSes that provides distribution transparency (CORBA, gRPC, Kubernetes, Mesos, Ray).

Today the distributed-OS role is mostly played by a cluster manager paired with a coordination store: Kubernetes with etcd, Borg with Chubby, Mesos with ZooKeeper, Nomad with Raft and Consul. This is why Kubernetes is often called "the datacenter OS", the same framing Mesosphere used for DC/OS.

The classic responsibilities

Resource management and scheduling of CPU, memory, accelerators, network and storage across nodes.
Naming and discovery of services, files and processes.
Communication: RPC, messaging, streams, shared-memory abstractions.
Synchronization and coordination: locks, leases, barriers, leader election, consensus.
Consistency and replication of state.
Fault tolerance: detect, mask, recover.
Security: authentication, authorization, isolation, attestation.
Transparency of access, location, migration, replication, concurrency and failure.

What changes at the edge

"Edge" means compute placed near data sources or users: gateways, base stations (MEC), street cabinets, factory floors, vehicles and small on-premises clusters. Related terms include fog computing (Bonomi et al., 2012), cloudlets (Satyanarayanan et al., 2009), ETSI Multi-access Edge Computing and the "computing continuum".

Compared with a datacenter, the edge brings geo-distribution and WAN latency, intermittent connectivity, heterogeneous hardware (ARM, x86, RISC-V, GPUs, NPUs, FPGAs, microcontrollers), tight RAM, energy and thermal budgets, weak physical security, thousands of small clusters instead of a few large ones, user mobility, and no administrator on site.

At the edge, network partitions are normal operation, not an exception.

An edge OS therefore needs five properties: a lightweight control plane, autonomy when disconnected from the cloud, eventual or causal state sync with the cloud and peer sites, latency- and locality-aware placement, and secure isolation through microVMs, WebAssembly, hardened containers or TEEs.

System models and failure models

Timing models

In a synchronous system, message delay, clock drift and processing time have known bounds. In an asynchronous system there are no bounds at all, so a crashed node cannot be told apart from a slow one. The partially synchronous model (Dwork, Lynch and Stockmeyer, 1988) assumes bounds exist but are unknown, or hold only after an unknown Global Stabilization Time. Paxos, Raft and PBFT target this model: safety always, liveness only when the network behaves.

Failure models, weakest adversary first

Crash-stop: a node halts and never returns.
Crash-recovery: a node halts and may restart with persistent state.
Omission: a node fails to send or receive some messages.
Timing failures: responses arrive too late, which matters a lot at the edge.
Byzantine: arbitrary or malicious behaviour (Lamport, Shostak, Pease, 1982); tolerating f faults needs 3f+1 replicas, fewer with signatures or trusted hardware.
Network failures: loss, duplication, reordering, partition, and asymmetric partition where A reaches B but B cannot reach A.
Gray failures (Huang et al., HotOS 2017): partial, hard-to-detect degradation, very common in practice.

Failure detectors

Chandra and Toueg (1996) classified unreliable failure detectors by completeness and accuracy; the eventually-strong detector ◇S is the weakest one that solves consensus with a correct majority. In practice systems use heartbeats with timeouts, the phi-accrual detector (Hayashibara et al., 2004) in Cassandra and Akka, and SWIM (Das, Gupta, Motivala, 2002) for gossip-based membership in HashiCorp Serf, Memberlist and Consul.

Fundamental limits

FLP impossibility (Fischer, Lynch, Paterson, 1985): in a purely asynchronous system, no deterministic protocol can guarantee consensus if even one process may crash. Real protocols sidestep it with randomization, failure detectors or timeouts.

CAP, conjectured by Brewer in 2000 and proved by Gilbert and Lynch in 2002: during a network partition a system must choose between linearizable consistency and availability. Because edge–cloud partitions are frequent, many edge designs choose availability with eventual consistency across sites and use consensus only inside a small local cluster. PACELC (Abadi, 2012) extends this: if partitioned, choose A or C; else, choose latency or consistency.

Other results worth knowing are the Two Generals problem, the 3f+1 Byzantine bound, the 2f+1 bound for crash-tolerant consensus, and the CALM theorem (Hellerstein and Alvaro): monotonic programs, which only accumulate facts, can be made consistent without coordination. CALM is the theoretical basis for CRDTs and coordination-free edge sync.

**Fig. 2** The consistency hierarchy (after Viotti and Vukolić, 2016). Edge systems usually keep the left end inside one site and the right end between sites.

Time, ordering, and causality

Lamport clocks (CACM, 1978) define the happened-before relation; logical timestamps give a partial order that can be extended to a total order.
Vector clocks (Fidge; Mattern, 1988) capture causality exactly, but their size grows with the number of nodes, a real problem at edge scale. Dotted version vectors and interval tree clocks reduce the cost.
Hybrid Logical Clocks (Kulkarni et al., 2014) combine physical and logical time with bounded drift; CockroachDB and YugabyteDB use them.
TrueTime in Google Spanner (2012) uses GPS and atomic clocks to expose an uncertainty interval; commit-wait yields external consistency. Most edge hardware cannot do this, which shows that accurate time is itself a resource to orchestrate.
NTP, PTP (IEEE 1588), chrony synchronize clocks in practice. Edge devices often sync poorly or not at all, so designs must tolerate skew.
Distributed snapshots (Chandy and Lamport, 1985) capture a consistent global state without stopping the system, and underlie checkpointing in Apache Flink.

Consensus and replicated state machines

State machine replication (Schneider, 1990) runs a deterministic state machine on every replica and feeds it the same totally ordered log of commands. Consensus decides the contents of each log slot.

**Fig. 3** Raft log replication. An entry is committed once a majority stores it, so slow or partitioned followers do not block progress. They catch up from the leader later.

Crash-fault-tolerant protocols

Viewstamped Replication (Oki and Liskov, 1988; revisited 2012).
Paxos (Lamport, 1998; "Paxos Made Simple", 2001) and Multi-Paxos for logs. Variants: Fast Paxos, Flexible Paxos (2016), leaderless EPaxos (SOSP 2013) for geo-replication, and wide-area WPaxos.
Raft (Ongaro and Ousterhout, USENIX ATC 2014): leader election, log replication, safety, membership change and snapshot-based log compaction. Implementations include etcd/raft, HashiCorp Raft, Dragonboat, raft-rs in TiKV, Consul and CockroachDB.
Zab, ZooKeeper's atomic broadcast (2011), and chain replication (van Renesse and Schneider, OSDI 2004) with its CRAQ extension for reads from any replica.

Byzantine-fault-tolerant protocols

PBFT (Castro and Liskov, OSDI 1999), Zyzzyva, Aardvark, HotStuff (PODC 2019), Tendermint, and MinBFT/CheapBFT, which use trusted hardware to bring the replica count down to 2f+1. Edge nodes are physically exposed, so BFT or TEE-assisted protocols using Intel SGX or ARM TrustZone are a legitimate research direction.

Coordination services and reconfiguration

Chubby (OSDI 2006), ZooKeeper (ATC 2010), etcd as the Kubernetes source of truth, and Consul/Nomad (Raft plus SWIM gossip) offer consensus as a service. Because edge nodes join and leave constantly, dynamic membership matters: Raft joint consensus, Vertical Paxos, Stoppable Paxos and non-voting learner members are the main techniques.

Geo-aware consensus

A single Raft group stretched across WAN-connected edges has very poor latency. The usual patterns are a small Raft group per site with asynchronous replication to the cloud, hierarchical consensus with local and regional leaders, leaderless or per-key quorums (EPaxos, Dynamo-style), and sharded consensus groups such as Spanner's Paxos groups, CockroachDB ranges and TiKV regions (Multi-Raft).

Fault-tolerant state synchronization beyond consensus

Consensus needs a quorum online and at least one round trip per decision. On a flaky edge uplink that means unavailability. Many sync problems, such as device twins, configuration, telemetry and user documents, can tolerate convergence after the fact.

Eventual consistency and its toolbox

Dynamo (SOSP 2007): consistent hashing, sloppy quorums, hinted handoff, vector clocks, read repair and Merkle-tree anti-entropy. Descendants include Riak, Cassandra and Voldemort.
Bayou (SOSP 1995): weakly connected replicas, tentative versus committed writes, application-specific conflict resolution. It was an early mobile design and is still very relevant to the edge.
Coda (1990s): disconnected operation for file systems through hoarding, emulation and reintegration.
Session guarantees (Terry et al., 1994): read-your-writes, monotonic reads, monotonic writes, writes-follow-reads.

CRDTs: conflict-free replicated data types

Shapiro, Preguiça, Baquero and Zawirski (2011) formalized two families. State-based CRDTs merge by taking the join in a semilattice, an operation that is commutative, associative and idempotent, so duplicated or reordered messages are harmless. Operation-based CRDTs send commuting operations over reliable causal delivery. Delta-state CRDTs (Almeida, Shoker, Baquero, 2018) ship only the changed part.

**Fig. 4** A grow-only counter (G-Counter). Each replica only increments its own slot. Merging takes the element-wise maximum, so both sites reach the same value without any coordination.

Common types include G-Counter, PN-Counter, G-Set, 2P-Set, OR-Set, LWW-Register, MV-Register, and RGA, Logoot, LSEQ and Treedoc for sequences. Collaborative editing uses Automerge (Kleppmann and Beresford, 2017), Yjs and Diamond Types. Production systems include Riak DT, Redis Enterprise CRDB, Akka Distributed Data, AntidoteDB, Azure Cosmos DB multi-master and Anna KVS.

Limit. Not every invariant can be expressed this way. "Bank balance never below zero" still needs coordination (see Bailis et al., coordination avoidance and I-confluence). Tombstones and vector-clock metadata also grow without bound and need garbage collection, which is important on constrained edge nodes.

Gossip, logs, and migration

Epidemic protocols (Demers et al., PODC 1987) spread state through anti-entropy and rumor mongering, converging in O(log N) rounds with push, pull or push-pull exchange. They power membership (SWIM, Serf), state dissemination (Cassandra) and aggregation, with Merkle trees making large diffs cheap.

Append-only logs such as Kafka, Pulsar, NATS JetStream and Redpanda form the backbone of edge-to-cloud pipelines; exactly-once behaviour comes from idempotent producers with transactions or idempotent consumers. Change data capture tools like Debezium replicate databases.

Running workloads move with CRIU checkpoint/restore, pre-copy live VM migration (Clark et al., NSDI 2005) and post-copy migration; at the edge this enables "follow-me" services that migrate containers between MEC sites as users move. Flink barrier snapshots, Erlang/OTP supervision trees and Akka cluster sharding add further patterns, and event sourcing with sagas (Garcia-Molina and Salem, 1987) avoids distributed two-phase commit for long-lived transactions. Where cross-node transactions are needed, the reference designs are Paxos Commit, Spanner, Calvin, Percolator, FaRM and FoundationDB.

A tiered design for edge state

The pattern below appears, with variations, in most production edge platforms.

Local strongly consistent core. A small Raft, etcd or SQLite-with-WAL group on the site holds local control-plane state.
Local application data. A CRDT, LWW or causal store that works offline first.
Edge to cloud. Asynchronous, batched, idempotent and resumable replication with versioning, a conflict policy, backpressure and compression.
Cloud to edge. Declarative desired state, signed and versioned, applied by a local reconciler.

Reconciliation loops that compare desired state with observed state are naturally idempotent and tolerate lost messages.

Resource orchestration: theory

Given jobs with resource demands and constraints (CPU, RAM, GPU, bandwidth, latency, affinity, data locality, deadlines, priority, energy) and a set of heterogeneous nodes, the orchestrator decides where and when each job runs and how much it receives, continuously and under failures. This is multi-dimensional bin packing and is NP-hard, so real systems rely on heuristics, scoring functions and incremental rescheduling.

Objectives

Utilization and packing efficiency; makespan and throughput; latency, tail latency and SLO compliance; fairness; cost, energy and carbon; reliability through spreading across failure domains; and data locality, which at the edge means moving compute to data.

Fairness

Dominant Resource Fairness (Ghodsi et al., NSDI 2011) equalizes each user's dominant share and is strategy-proof, Pareto-efficient, envy-free and sharing-incentive; Mesos and YARN implement it. Related work includes max-min fairness, lottery and stride scheduling, Quincy's min-cost-flow scheduling (SOSP 2009), Tetris multi-resource packing (SIGCOMM 2014), and Themis, Gandiva, Tiresias and AntMan for GPU clusters.

Scheduler architectures

Architecture	Idea	Examples
Monolithic	One scheduler holds all placement logic.	Borg, early Hadoop JobTracker
Two-level	A resource manager offers resources to framework schedulers.	Mesos (NSDI 2011), YARN (SoCC 2013)
Shared state	Several schedulers work on a shared cell state with optimistic concurrency.	Omega (EuroSys 2013), Nomad
Distributed	Decentralized sampling, e.g. power of two random choices.	Sparrow (SOSP 2013), Apollo (OSDI 2014), Hawk, Eagle
Flow-based	Placement solved as a min-cost flow problem.	Quincy, Firmament (OSDI 2016)
Federated	Many clusters under one policy layer.	Mercury, Karmada, Open Cluster Management, Liqo

**Fig. 5** The Kubernetes scheduling framework. Filter plugins remove infeasible nodes, score plugins rank the rest, and the binding is written back to the API server. Edge-aware schedulers usually add latency or data-locality scores.

Placement algorithms

First-fit, best-fit, worst-fit and first-fit-decreasing; filter-then-score pipelines; the power of two random choices (Mitzenmacher); ILP, MILP and constraint programming with Google OR-Tools; metaheuristics such as genetic algorithms, simulated annealing and particle swarm, which are common in fog papers but often evaluated only in simulators; and learning-based schedulers such as DeepRM, Decima (SIGCOMM 2019), RL-based offloading and Gavel (OSDI 2020). Learned schedulers need explainability, safety and a fallback policy. Priorities and preemption appear as Borg priority bands, Kubernetes PriorityClass and PodDisruptionBudgets, gang scheduling (Volcano, Kueue) and HPC backfilling in Slurm.

Autoscaling and isolation

Horizontal, vertical and cluster autoscaling (HPA, VPA, Cluster Autoscaler, Karpenter) build on feedback control, PID, model predictive control and queueing theory such as M/M/c and Little's Law; Google's Autopilot (EuroSys 2020) is the reference for vertical scaling, and KEDA and Knative add event-driven scale-to-zero. Isolation is part of orchestration: cgroups v2, namespaces, CPU quotas and cpusets, NUMA, PSI pressure signals and eBPF; noisy-neighbour control through Heracles, Parties and Intel RDT; and sandboxes from runc containers through gVisor, Kata, Firecracker microVMs (NSDI 2020) and unikernels to WebAssembly.

Real schedulers and orchestrators

Borg, Omega, Kubernetes. Verma et al. (EuroSys 2015) describe Borg's cells, jobs, tasks, allocs, priorities and quota, and a Paxos-replicated BorgMaster. Burns et al., "Borg, Omega, and Kubernetes" (ACM Queue, 2016), is the best short account of the lessons learned.

Kubernetes internals. The control plane consists of kube-apiserver (REST plus watch), etcd, kube-scheduler and kube-controller-manager running reconcile loops. Each node runs kubelet, kube-proxy and a CRI runtime such as containerd or CRI-O, with CNI networking and CSI storage. A declarative API plus level-triggered reconciliation gives idempotent convergence, backed by resourceVersion optimistic concurrency, Lease-based leader election and informer caches. The resource model includes requests and limits, QoS classes, taints and tolerations, affinity, topology spread, device plugins and Dynamic Resource Allocation. Node failure flows from missed lease heartbeats to a NotReady taint to eviction after tolerationSeconds.

Mesos, YARN, Nomad, Swarm. Nomad is a single binary with Raft and optimistic scheduling, a good fit for small edge clusters. Docker Swarm also uses Raft-based managers.

Edge-oriented orchestrators

Platform	Approach	State and autonomy
K3s	CNCF-certified lightweight Kubernetes, single binary, built for ARM and IoT.	SQLite, etcd or dqlite backend; full cluster runs on the site.
KubeEdge	CloudCore in the cloud, EdgeCore on nodes; MQTT device management.	Local SQLite metadata, device twins, cloud–edge sync over WebSocket or QUIC.
OpenYurt	Extends unmodified Kubernetes with node pools and tunnels.	YurtHub caches API responses so nodes survive disconnection.
Nomad	Single binary scheduler with optimistic concurrency.	Raft servers per region; clients keep running allocations.
AWS IoT Greengrass v2	Components deployed from signed manifests.	Local shadow, stream manager, offline queue.
Azure IoT Edge	Modules described by deployment manifests.	Module twins, offline mode, reconcile on reconnect.

Others include k0s, MicroK8s, MicroShift and Talos Linux; SuperEdge, Baetyl, EdgeX Foundry, Eclipse ioFog, Akraino, Open Horizon and Akri; serverless edge platforms such as Cloudflare Workers (V8 isolates), Fastly Compute (WebAssembly), OpenFaaS, Knative and Fermyon Spin; and research systems such as Fogify, FogBus and Osmotic Computing. Shi et al., "Edge Computing: Vision and Challenges" (IEEE IoT Journal, 2016), introduced the EdgeOS concept. For comparison, HPC and data-centre managers include Slurm, PBS, LSF, Flux, Ray (OSDI 2018) and Dask.

Edge-specific challenges

Orchestration

Scale of sites: thousands of one- to five-node clusters, so the central control plane must scale in cluster count, not only node count.
Autonomy: each site keeps scheduling and healing when its uplink drops, using a local cache of desired state and a local leader.
Heterogeneity: Node Feature Discovery, device plugins, GPU sharing (MIG, MPS, time-slicing), multi-arch OCI images.
Locality: placement by RTT, bandwidth and data gravity; "follow the user" migration; anycast and GSLB routing.
Footprint: etcd memory and watch fan-out are costly, hence SQLite, dqlite and kine.
Networks: NAT and flapping links call for WireGuard, Tailscale or Nebula tunnels, MQTT, QUIC and reverse tunnels, since edges often cannot accept inbound connections.
Updates: staged OTA with A/B partitions (Mender, RAUC, SWUpdate, Uptane), health-checked rollback, delta updates and signed artifacts (TUF, Sigstore, in-toto, SLSA).
Observability: local buffering, edge-side aggregation, OpenTelemetry collectors, Prometheus remote-write with a WAL; send summaries rather than raw data.

State synchronization

Choose availability for the data plane and consistency for the local control plane.
Conflict policies range from LWW (vulnerable to clock skew) to CRDTs, semantic merge, human review, and "cloud wins" or "edge wins" per field.
Bandwidth is saved with delta sync, compression and content-addressed storage.
Split-brain is prevented with fencing tokens, leases with bounded drift, epoch numbers and a witness node; two nodes plus a witness is a very common edge topology.
Device twins with desired and reported state (AWS IoT shadow, Azure IoT twin, KubeEdge twin) amount to a small versioned two-register protocol.
Data sovereignty and GDPR constrain where state may be replicated.

Fault tolerance, security, real time

Edges suffer correlated failures (regional power or ISP outages), SD-card wear, missing ECC memory, thermal throttling and brown-outs, which call for journaling, fsync discipline, checksumming file systems and hardware watchdogs. Recovery objectives (RTO and RPO) should be set per tier, and safety-critical loops, such as industrial control at under 10 ms, stay local. Security relies on zero trust and mutual TLS with SPIFFE/SPIRE identities, TPM 2.0 with secure and measured boot, remote attestation (Keylime, Veraison), confidential computing (SGX, SEV-SNP, TDX, ARM CCA) and admission only after attestation. Hard real-time work belongs to PREEMPT_RT, Xenomai, partitioning hypervisors such as Jailhouse and ACRN, seL4 mixed-criticality scheduling, AUTOSAR Adaptive or ROS 2 with DDS; Kubernetes is suitable only for the soft real-time tier.

Edge operating systems and runtimes

Device OSes for constrained nodes include FreeRTOS, Zephyr, RIOT, Contiki-NG, Mbed OS, NuttX and ThreadX, offering small footprints, deterministic scheduling, OTA, low-power states and stacks such as lwIP, 6LoWPAN, Thread, BLE, LoRaWAN and CoAP.

Gateway and edge-server OSes are mostly Linux based: Yocto, Buildroot, Ubuntu Core, Fedora IoT, Flatcar, Talos Linux, balenaOS, Bottlerocket and Red Hat Device Edge. Immutable, image-based OSes with atomic updates are a major trend because they make OS-level fault tolerance much simpler.

Research OS designs connect back to distributed-OS ideas. Barrelfish's multikernel (SOSP 2009) treats one multicore machine as a distributed system of cores with replicated state. seL4 is formally verified; Theseus, Tock and Hubris explore Rust-based structure; LegoOS (OSDI 2018) splits the kernel across disaggregated hardware; Unikraft and MirageOS are library OSes; and eBPF, sched_ext, io_uring and ghOSt (SOSP 2021) make the kernel itself extensible.

Runtime	Start-up and overhead	Isolation	Trade-off
Containers	Fastest start, smallest overhead	Weakest (shared kernel)	Best ecosystem
gVisor / Kata	Moderate overhead	Stronger	Compatibility gaps for some syscalls
MicroVMs	About 100–200 ms start, small footprint	Strong (hardware virtualization)	Needs virtualization support
Unikernels	Tiny and fast	Strong	Hard to debug, limited ecosystem
WebAssembly	Millisecond start, tiny	Sandboxed by design	Language and WASI ABI limits

Historical distributed operating systems

The ideas behind today's edge platforms are decades old. Amoeba (Vrije Universiteit) used capabilities and a processor-pool model. Sprite (UC Berkeley) offered a single system image, the log-structured file system and process migration. The V System (Stanford) pioneered efficient IPC and process groups. Plan 9 (Bell Labs) made everything a file with per-process namespaces and the 9P protocol, ideas that resurfaced in Linux namespaces and containers. Mach, Chorus, Locus, MOSIX, Kerrighed, Inferno, Emerald and Popcorn Linux explored migration and heterogeneity.

The lesson is that a perfect single system image is expensive and leaky. Waldo et al., "A Note on Distributed Computing" (1994), argued that local and remote calls are not the same; the end-to-end argument and the fallacies of distributed computing point the same way. Modern systems expose distribution in their APIs through timeouts, retries and idempotency instead of hiding it.

Case studies

Kubernetes and etcd. If etcd loses quorum the API becomes unavailable, yet running pods keep running because the data plane is independent of the control plane. Plain kubelet would mark a disconnected edge node NotReady and evict its pods; KubeEdge and OpenYurt add local caches to prevent this.
Google Spanner. Paxos per shard, 2PC across groups, TrueTime for external consistency. It shows the cost and the benefit of global strong consistency.
Amazon Dynamo and DynamoDB. Dynamo chose availability with vector clocks and sloppy quorums; DynamoDB now runs Multi-Paxos per partition (ATC 2022) with LWW across regions in global tables.
CockroachDB, TiKV, YugabyteDB. Multi-Raft ranges, HLC, leaseholders, follower reads for low-latency stale reads and geo-partitioning for data residency.
Cassandra. Gossip with phi-accrual detection, tunable consistency levels, hinted handoff and Merkle-tree repair.
Flink and Kafka. Barrier snapshots for exactly-once state; Kafka replaced ZooKeeper with its own Raft (KRaft).
Cloudflare. Workers on V8 isolates, Durable Objects as strongly consistent single-writer actors placed near users, eventually consistent KV, and Quicksilver for global configuration.
Uptane. Secure OTA for vehicles with role separation and threshold signatures.
Offline-first apps. Firestore offline persistence, CouchDB and PouchDB revision trees, Couchbase Lite, Ditto and Automerge-repo.

Giving each piece of state one authoritative owner, whether an actor, a partition or a single writer, avoids multi-writer conflicts at the edge.

Testing, verification, and benchmarking

Fault injection. Jepsen tests consistency claims as a black box, with the Knossos and Elle checkers, and has published analyses of etcd, Consul, MongoDB and CockroachDB. Chaos engineering tools include Chaos Monkey, Chaos Mesh, LitmusChaos, Toxiproxy, and tc netem or iptables for latency, loss and partitions. Deterministic simulation testing is used by FoundationDB, TigerBeetle and Antithesis.

Formal methods. TLA+ and TLC (used at AWS, Newcombe et al., CACM 2015, and for Raft's own specification), the P language, Ivy, Verdi, IronFleet and Alloy, plus lineage-driven fault injection.

Simulation and testbeds. iFogSim, EdgeCloudSim, YAFS, PureEdgeSim, SimGrid, ns-3 and Mininet; KinD and k3d for local clusters; kubemark for scheduler scale tests; Raspberry Pi and Jetson clusters, FIT IoT-LAB, CloudLab and Chameleon for real hardware. Public traces come from Google (2011, 2019), Alibaba, Azure VMs and Functions, and Microsoft Philly and Helios.

Metrics to report

Scheduling latency at p50, p95 and p99; throughput in pods per second; utilization and fragmentation; SLO violation rate; failover time; data lost at recovery; replica convergence time; bytes transferred; conflict rate; staleness; availability in nines; control-plane CPU and memory on edge hardware; energy per request; and cost.

Reproducible experiments. Five inexpensive experiments cover the topic well: Raft failover time versus election timeout under injected latency; writes on both sides of a partition between two edge sites, comparing LWW and CRDT results after healing; default Kubernetes scoring versus a latency-aware plugin measured at p95; control-plane footprint of K3s, kubeadm Kubernetes and KubeEdge EdgeCore on a 1–2 GB node; and a cut uplink with a node restart to check whether workloads recover from the local cache.

Open research problems

Scheduling across the continuum with conflicting objectives (latency, energy, cost, carbon, privacy) and stale global knowledge.
Control planes for 10⁴–10⁶ tiny clusters on intermittent links, without losing declarative semantics.
Consistency-aware placement: co-locating compute with the owner of its state, and deciding when to migrate state and when to migrate compute.
Coordination-avoiding programming models for the edge: CALM analysis tools, Bloom, Lasp and local-first frameworks.
CRDTs with bounded metadata and safe garbage collection under dynamic membership and long disconnection.
Consensus with witnesses and asymmetric networks; two-node clusters; consensus over lossy low-power wireless.
Trustworthy edge: BFT/TEE hybrids, attestation-aware scheduling, confidential containers.
Learned schedulers that are safe, explainable and robust to distribution shift.
A unified abstraction for CPU, GPU, NPU, FPGA and MCU, including sharing, preemption and migration of accelerator state.
Energy- and carbon-aware orchestration for battery-powered nodes.
Real benchmarks and public edge traces; most studies still rely on simulators and synthetic load.
Verified and deterministically tested edge control planes.
Gray-failure detection and automated remediation.
The WebAssembly component model as a portable, secure edge execution unit (wasmCloud, SpinKube, runwasi).
OTA delivery to millions of devices with strong rollback and supply-chain guarantees.

Glossary

Anti-entropy: Background process that compares and repairs replica state.
Autonomy (edge): Ability of an edge site to operate without the cloud.
BFT: Byzantine fault tolerance.
Causal consistency: If A may have caused B, every replica sees A before B.
CRDT: Replicated data type that converges without coordination.
DRF: Dominant Resource Fairness.
Fencing token: Monotonic number used to reject stale leaders or lock holders.
Gossip: Randomized peer-to-peer state dissemination.
HLC: Hybrid Logical Clock.
Idempotent: Applying an operation many times equals applying it once.
Lease: Time-bounded lock or ownership grant.
Linearizability: Operations appear atomic in real-time order.
MEC: Multi-access Edge Computing (ETSI telecom edge).
Merkle tree: Hash tree used to compare large data sets efficiently.
Quorum: Subset of replicas whose agreement is sufficient.
Reconciliation: Loop that moves observed state toward desired state.
RPO / RTO: Recovery Point and Recovery Time Objectives.
Sloppy quorum: Use of non-home nodes to stay available (Dynamo).
SMR: State machine replication.
Split-brain: Two partitions that both believe they are primary.
Tombstone: Deletion marker kept for replication correctness.
Vector clock: Per-node counters that capture causal history.
WAL: Write-ahead log.
Witness: Lightweight node that votes in a quorum without storing data.

Selected references

Foundations

Lamport, L. (1978). Time, Clocks, and the Ordering of Events in a Distributed System. CACM.

Chandy, K. M., Lamport, L. (1985). Distributed Snapshots: Determining Global States of Distributed Systems. ACM TOCS.

Fischer, M., Lynch, N., Paterson, M. (1985). Impossibility of Distributed Consensus with One Faulty Process. JACM.

Lamport, L., Shostak, R., Pease, M. (1982). The Byzantine Generals Problem. ACM TOPLAS.

Dwork, C., Lynch, N., Stockmeyer, L. (1988). Consensus in the Presence of Partial Synchrony. JACM.

Chandra, T., Toueg, S. (1996). Unreliable Failure Detectors for Reliable Distributed Systems. JACM.

Gilbert, S., Lynch, N. (2002). Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services. SIGACT News.

Schneider, F. (1990). Implementing Fault-Tolerant Services Using the State Machine Approach: A Tutorial. ACM Computing Surveys.

Waldo, J. et al. (1994). A Note on Distributed Computing. Sun Microsystems Labs.

Consensus and replication

Lamport, L. (1998). The Part-Time Parliament. ACM TOCS.

Ongaro, D., Ousterhout, J. (2014). In Search of an Understandable Consensus Algorithm. USENIX ATC.

Castro, M., Liskov, B. (1999). Practical Byzantine Fault Tolerance. OSDI.

Yin, M. et al. (2019). HotStuff: BFT Consensus with Linearity and Responsiveness. PODC.

Moraru, I., Andersen, D., Kaminsky, M. (2013). There Is More Consensus in Egalitarian Parliaments. SOSP.

Burrows, M. (2006). The Chubby Lock Service for Loosely-Coupled Distributed Systems. OSDI.

Hunt, P. et al. (2010). ZooKeeper: Wait-free Coordination for Internet-scale Systems. USENIX ATC.

Eventual consistency and CRDTs

DeCandia, G. et al. (2007). Dynamo: Amazon's Highly Available Key-value Store. SOSP.

Terry, D. et al. (1995). Managing Update Conflicts in Bayou. SOSP.

Demers, A. et al. (1987). Epidemic Algorithms for Replicated Database Maintenance. PODC.

Shapiro, M., Preguiça, N., Baquero, C., Zawirski, M. (2011). Conflict-free Replicated Data Types. SSS.

Kleppmann, M., Beresford, A. (2017). A Conflict-Free Replicated JSON Datatype. IEEE TPDS.

Hellerstein, J., Alvaro, P. (2020). Keeping CALM: When Distributed Consistency is Easy. CACM.

Kleppmann, M. et al. (2019). Local-First Software. Onward!

Viotti, P., Vukolić, M. (2016). Consistency in Non-Transactional Distributed Storage Systems. ACM Computing Surveys.

Transactions and geo-replication

Corbett, J. et al. (2012). Spanner: Google's Globally-Distributed Database. OSDI.

Thomson, A. et al. (2012). Calvin: Fast Distributed Transactions for Partitioned Database Systems. SIGMOD.

Taft, R. et al. (2020). CockroachDB: The Resilient Geo-Distributed SQL Database. SIGMOD.

Scheduling and cluster management

Verma, A. et al. (2015). Large-scale Cluster Management at Google with Borg. EuroSys.

Burns, B. et al. (2016). Borg, Omega, and Kubernetes. ACM Queue.

Schwarzkopf, M. et al. (2013). Omega: Flexible, Scalable Schedulers for Large Compute Clusters. EuroSys.

Hindman, B. et al. (2011). Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center. NSDI.

Ghodsi, A. et al. (2011). Dominant Resource Fairness. NSDI.

Ousterhout, K. et al. (2013). Sparrow: Distributed, Low Latency Scheduling. SOSP.

Rzadca, K. et al. (2020). Autopilot: Workload Autoscaling at Google. EuroSys.

Agache, A. et al. (2020). Firecracker: Lightweight Virtualization for Serverless Applications. NSDI.

Edge and fog computing

Satyanarayanan, M. et al. (2009). The Case for VM-Based Cloudlets in Mobile Computing. IEEE Pervasive Computing.

Bonomi, F. et al. (2012). Fog Computing and Its Role in the Internet of Things. MCC Workshop.

Shi, W. et al. (2016). Edge Computing: Vision and Challenges. IEEE Internet of Things Journal.

Satyanarayanan, M. (2017). The Emergence of Edge Computing. IEEE Computer.

Hong, C.-H., Varghese, B. (2019). Resource Management in Fog/Edge Computing: A Survey. ACM Computing Surveys.

Taleb, T. et al. (2017). On Multi-Access Edge Computing. IEEE COMST.

OS research and verification

Baumann, A. et al. (2009). The Multikernel: A New OS Architecture for Scalable Multicore Systems. SOSP.

Klein, G. et al. (2009). seL4: Formal Verification of an OS Kernel. SOSP.

Shan, Y. et al. (2018). LegoOS: A Disseminated, Distributed OS for Hardware Resource Disaggregation. OSDI.

Newcombe, C. et al. (2015). How Amazon Web Services Uses Formal Methods. CACM.

Huang, P. et al. (2017). Gray Failure: The Achilles' Heel of Cloud-Scale Systems. HotOS.

Alquraan, A. et al. (2018). An Analysis of Network-Partitioning Failures in Cloud Systems. OSDI.

Books

Tanenbaum, A., van Steen, M. Distributed Systems, 4th ed.

Kleppmann, M. Designing Data-Intensive Applications. O'Reilly.

Coulouris, G. et al. Distributed Systems: Concepts and Design.

Cachin, C., Guerraoui, R., Rodrigues, L. Introduction to Reliable and Secure Distributed Programming.