ADR-002: Kameo Actors for Federation Orchestration

Date: 2024-03-10

Status: Accepted

Context

SynDB federation delegates queries to multiple remote nodes simultaneously. A federated query must fan out Flight gRPC calls, enforce per-node timeouts, retry transient failures, and aggregate partial results into a single response stream.

Implementing this with raw tokio::spawn and channels leads to scattered state, ad-hoc cancellation logic, and difficult-to-test concurrency patterns. We need a structured concurrency model that encapsulates per-query state and lifecycle.

Decision

Use the kameo actor framework for federation query orchestration. Each federated query spawns a coordinator actor that in turn spawns per-node worker actors. Workers issue Flight gRPC calls and stream results back to the coordinator via typed messages.

• Actor mailboxes provide natural back-pressure.
• Supervision trees handle worker failures without crashing the coordinator.
• Actor state is private and mutation-free from the caller’s perspective.

Consequences

Positive:

• Clean separation of concerns: each actor owns its state and lifecycle.
• Message-passing eliminates shared mutable state across concurrent operations.
• Supervision and timeout semantics are built into the framework rather than hand-rolled.
• Actors are straightforward to unit-test in isolation by sending messages directly.

Negative:

• Adds a runtime dependency on the kameo crate and its executor integration.
• Actor mailbox overhead exists, though it is negligible for the federation workload (tens of messages, not millions).
• Developers must learn the actor model; it is less familiar than plain async/await to most Rust programmers.