Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Federated Query Across Institutions

SynDB’s federation layer connects independently operated nodes into a single query surface. A researcher on the hub can query datasets hosted at any participating institution as if they were local — without moving data or coordinating per-dataset access.

Scenario

The FlyBrain Institute hosts a Drosophila optic-lobe reconstruction. The Cortex Lab hosts a mouse V1 dataset. Both run SynDB nodes.

A researcher on the hub wants to compare synapse densities between the two species. With federation, this is a single query.

Architecture

┌─────────────────────────────────────┐
│               Hub                   │
│  ┌──────────────────────────────┐   │
│  │  Single API endpoint         │   │
│  │  syql.flybrain.org/v1/syql   │   │
│  └──────┬──────────────┬────────┘   │
│         │              │            │
┌─────────▼──────┐  ┌────▼────────────┐
│  FlyBrain Node │  │  Cortex Lab     │
│  (drosophila)  │  │  (mouse V1)     │
│  ClickHouse    │  │  ClickHouse     │
│  S3            │  │  S3             │
└────────────────┘  └─────────────────┘

Prerequisites

  • A hub node with federation enabled
  • Two or more remote clusters registered with the hub
  • The remote clusters must be reachable from the hub over Arrow Flight (TCP)

Walkthrough

1. Register a cluster

On the hub, register each participating site:

syndb federation register \
  --name flybrain \
  --api-url https://flybrain.example.com \
  --flight-addr flybrain.example.com:50051

2. Verify cluster health

syndb federation list
syndb federation health flybrain

Healthy clusters return dataset catalog entries, version info, and latency.

3. Query across clusters

The SCOPE federation directive in SyQL tells the planner to scatter the query to all available clusters:

syndb query run "
  SCOPE federation
  FROM synapses
  WHERE species = 'drosophila' OR species = 'mouse'
  GROUP BY species
"

The planner:

  1. Sends the scatter statement to every cluster
  2. Each cluster executes locally and streams partial results back
  3. The gather stage merges partials (summing counts, re-aggregating measures)

4. Query a specific cluster

If you only need data from one remote site, use SCOPE remote:

syndb query run "
  SCOPE remote
  FROM synapses LIMIT 100
"

The planner routes to the first healthy cluster.

5. Monitor federation

syndb federation health     # per-cluster status
syndb federation routes     # active routing table

Data ownership

Each cluster retains full ownership of its data. The hub never stores remote data — it only caches catalog metadata and routing information. Queries execute on the remote cluster’s infrastructure and the hub streams results through.

Troubleshooting

SymptomLikely causeCheck
Query returns only local resultsRemote cluster unreachablesyndb federation health <name>
Timeout on large queriesRemote cluster under-provisionedIncrease timeout in K8sDriverConfig
“Cluster not found”Registration expiredRe-register the cluster

Next steps