Fallback Routing Protocols for University Research & Lab Inventory Automation

Fallback routing protocols are the operational backbone that keeps a university research automation platform compliant when its primary ingestion path degrades or fails. When a sponsor portal returns 503s, a campus network partition isolates a lab subnet, or the central data store rejects a write, these deterministic workflows preserve data continuity, grant lifecycle tracking, and laboratory equipment telemetry without losing a single record or silently writing a duplicate. This guide is anchored to the foundational principles in Core Architecture & Policy Mapping for Research Grants: fallback is not a network contingency bolted on after the fact, it is a structured compliance mechanism that holds institutional service-level agreements and federal funding mandates intact during infrastructure degradation.

University administrators, research compliance officers, Python automation developers, and lab managers depend on this subsystem when the happy path is unavailable. It inherits the idempotency and policy contracts established in the Grant Lifecycle Architecture Design, and it routes any record it cannot deliver into the same quarantine queue used across the platform — so a transient outage never becomes a compliance gap.

Problem framing #

A naive ingestion client treats an upstream error as a fatal exception: it logs a stack trace, drops the batch, and moves on. In grant administration that behaviour is a chain-of-custody breach. A 90-second sponsor-portal outage during a fiscal-year-close window can strand dozens of award amendments; a flaky lab-subnet link can lose the calibration tag that proves a $2M instrument was within tolerance when an experiment ran. The records are gone, and no audit can reconstruct what was never persisted.

The specific gap this layer closes is delivery under degradation. Three contracts, implemented in the rest of this page, hold the line:

• Zero data loss. Every payload that passes validation is either committed to the central store or durably persisted to a local fallback tier; nothing is dropped because an endpoint was briefly unreachable.
• Zero silent duplication. A deterministic idempotency key drives every write, so a record that is retried, cached, and later reconciled lands exactly once — even if the original request actually succeeded before the timeout fired.
• Strict schema enforcement at the edge. A malformed record is quarantined before it consumes a retry budget, so a single bad payload never poisons the fallback path for the rest of the batch.

Policy constraints #

Fallback routing operates inside the same regulatory envelope as the primary path — degradation does not relax the rules. Federal and institutional mandates require unbroken, cryptographically verifiable audit trails, and the routing engine must satisfy them even while a tier is down. The regulatory matrix codified in the University Policy Mapping Frameworks governs every routing decision.

Operational boundary. Policy dictates what must survive an outage, how long a cached record may wait before it becomes a reportable delay, and which roles may drain or replay the fallback tiers. Implementation handles the mechanical retry, caching, and reconciliation. Encryption-at-rest, chain-of-custody logging, and retention schedules for the fallback cache itself are governed by the Security Boundary Configuration; no fallback tier may store a record outside those controls, regardless of how urgent the recovery is.

Data schema & field mapping #

Every record that enters the routing engine is normalized to a single canonical envelope before any tier touches it. The system-owned fields (idempotency_key, correlation_id, routed_at, tier) are what make a cached record safe to replay; the payload fields carry the compliance-bearing content that must survive unmutated.

The idempotency_key is a SHA-256 hash over the canonicalized payload (sorted keys, no whitespace), which is what lets the same record traverse primary → cache → reconciliation and still write exactly once. Records that fail edge validation never receive a key; they are quarantined with their raw source payload intact for later replay.

Implementation #

The fallback architecture operates across three deterministic tiers: primary API ingestion, a secondary edge cache, and tertiary quarantine (dead-letter) storage. When upstream services return 5xx errors, connection timeouts, or payload rejections, the system intercepts the batch, validates it synchronously, and redirects it down the appropriate tier. The engine balances throughput with stability using exponential backoff with randomized jitter to prevent a thundering-herd recovery storm when a campus-wide network partition heals.

Figure: three deterministic tiers — primary API, retried with jittered backoff, then edge cache that later drains to the central store.

Idempotent fallback router #

The following module is an idempotent fallback router. It guarantees that repeated executions produce identical state changes, preventing duplicate grant submissions or inventory double-counting. It uses correlation identifiers, deterministic hashing, and atomic local cache writes.

import hashlib
import json
import logging
import os
import time
import uuid
from pathlib import Path
from typing import Any

from pydantic import BaseModel, ValidationError
import requests

# Structured logging feeds the NIST SP 800-53 AU audit trail.
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
)
logger = logging.getLogger("fallback_router")


class LabTelemetryPayload(BaseModel):
    """Strict canonical envelope for lab equipment and chemical-inventory records."""

    asset_id: str
    calibration_date: str
    status: str
    compliance_tag: str
    submission_timestamp: str


class IdempotentFallbackRouter:
    """Primary API ingestion with deterministic failover to a local cache tier.

    Idempotency is guaranteed by cryptographic payload hashing and atomic
    file writes, so the same payload is safe to route any number of times.
    """

    def __init__(
        self,
        primary_endpoint: str,
        cache_directory: str = "/var/cache/research_fallback",
        quarantine_directory: str = "/var/cache/research_quarantine",
    ) -> None:
        self.primary_endpoint = primary_endpoint
        self.cache_dir = Path(cache_directory)
        self.quarantine_dir = Path(quarantine_directory)
        for d in (self.cache_dir, self.quarantine_dir):
            d.mkdir(parents=True, exist_ok=True)

    def _compute_idempotency_key(self, payload: dict[str, Any]) -> str:
        """Deterministic SHA-256 over the canonicalized payload."""
        canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
        return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

    def _is_already_processed(self, idempotency_key: str) -> bool:
        """True if this key is already durably persisted in the cache tier."""
        return (self.cache_dir / f"{idempotency_key}.json").exists()

    def _atomic_write(self, directory: Path, name: str, record: dict[str, Any]) -> None:
        """Write-then-rename so a crash never leaves a half-written record."""
        temp_path = directory / f"{name}.tmp"
        final_path = directory / f"{name}.json"
        try:
            with open(temp_path, "w", encoding="utf-8") as f:
                json.dump(record, f)
            os.replace(temp_path, final_path)  # atomic on POSIX
        except OSError as exc:
            logger.critical("Atomic write failed for %s: %s", name, exc)
            raise

    def _quarantine(self, payload: dict[str, Any], reason: str, correlation_id: str) -> None:
        """Tier 3: route a malformed/undeliverable record to dead-letter storage."""
        name = uuid.uuid4().hex
        self._atomic_write(
            self.quarantine_dir,
            name,
            {
                "source_payload": payload,
                "rejection_reason": reason,
                "correlation_id": correlation_id,
                "tier": "quarantine",
                "routed_at": time.time(),
            },
        )
        logger.warning("Quarantined record %s (corr_id=%s): %s", name, correlation_id, reason)

    def route_with_fallback(
        self,
        payload: dict[str, Any],
        max_retries: int = 3,
        base_delay: float = 1.0,
    ) -> bool:
        """Attempt primary ingestion; fail over to the cache tier on 5xx/timeout.

        Strictly idempotent: safe to call repeatedly with an identical payload.
        """
        correlation_id = str(uuid.uuid4())

        # Edge validation BEFORE consuming any retry budget.
        try:
            LabTelemetryPayload(**payload)
        except ValidationError as ve:
            self._quarantine(payload, f"schema_validation: {ve}", correlation_id)
            return False

        idempotency_key = self._compute_idempotency_key(payload)
        if self._is_already_processed(idempotency_key):
            logger.info("Key %s already cached; skipping (idempotent).", idempotency_key)
            return True

        headers = {"Idempotency-Key": idempotency_key, "X-Correlation-ID": correlation_id}
        delay = base_delay
        for attempt in range(1, max_retries + 2):
            try:
                response = requests.post(
                    self.primary_endpoint, json=payload, headers=headers, timeout=5.0
                )
                # 4xx is a permanent client error: do not retry, do not cache.
                if 400 <= response.status_code < 500:
                    self._quarantine(payload, f"client_error_{response.status_code}", correlation_id)
                    return False
                if response.status_code < 400:
                    logger.info("Primary ingest ok (attempt %d, corr_id=%s)", attempt, correlation_id)
                    return True
                logger.warning("Server error %d on attempt %d", response.status_code, attempt)
            except requests.RequestException as exc:
                logger.warning("Network exception on attempt %d: %s", attempt, exc)

            if attempt <= max_retries:
                jitter = time.time() % 0.3
                sleep_time = delay + jitter
                logger.info("Backing off %.2fs before retry %d", sleep_time, attempt + 1)
                time.sleep(sleep_time)
                delay *= 2.0

        # Tier 1 exhausted: durably persist to the tier-2 cache for reconciliation.
        logger.warning("Primary exhausted; caching (corr_id=%s)", correlation_id)
        self._atomic_write(
            self.cache_dir,
            idempotency_key,
            {
                "idempotency_key": idempotency_key,
                "correlation_id": correlation_id,
                "payload": payload,
                "tier": "cache",
                "routed_at": time.time(),
            },
        )
        return True

Three properties matter for compliance. The 4xx branch quarantines instead of retrying, because a permanent client error will never succeed and must not consume the retry budget. The cache write is atomic (os.replace), so a process kill mid-write can never leave a torn record. And the idempotency key is computed once from the canonical payload, so the same record cannot be cached twice even if it is submitted from two workers concurrently.

Off-peak reconciliation drain #

A cached record is not yet compliant — it is pending. A scheduled reconciliation job drains the cache into the immutable ledger during off-peak hours, using a SQLAlchemy upsert keyed on the idempotency key so a record that actually committed before its timeout fired is reconciled in place rather than duplicated.

from datetime import datetime, timezone
from pathlib import Path
import json

from sqlalchemy import create_engine
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session

from models import RoutedRecord  # SQLAlchemy model mapping the canonical envelope


def reconcile_cache(cache_directory: str, database_url: str) -> dict[str, int]:
    """Drain the tier-2 cache into the immutable ledger, exactly once per key."""
    engine = create_engine(database_url, future=True)
    cache_dir = Path(cache_directory)
    drained, skipped = 0, 0

    with Session(engine) as session:
        for cached in sorted(cache_dir.glob("*.json")):
            record = json.loads(cached.read_text(encoding="utf-8"))
            payload = record["payload"]

            stmt = pg_insert(RoutedRecord).values(
                idempotency_key=record["idempotency_key"],
                correlation_id=record["correlation_id"],
                asset_id=payload["asset_id"],
                calibration_date=payload["calibration_date"],
                status=payload["status"],
                compliance_tag=payload["compliance_tag"],
                submission_timestamp=payload["submission_timestamp"],
                reconciled_at=datetime.now(timezone.utc),
            )
            # ON CONFLICT DO NOTHING: a key already in the ledger means the
            # original request committed despite the timeout — never double-post.
            stmt = stmt.on_conflict_do_nothing(index_elements=["idempotency_key"])
            result = session.execute(stmt)

            if result.rowcount:
                drained += 1
            else:
                skipped += 1
            cached.unlink()  # remove only after the ledger write is staged

        session.commit()

    return {"drained": drained, "skipped_duplicates": skipped}

Integration points #

The router sits between the ingestion edge and the platform’s systems of record, so its tiers must speak the same envelope as everything around it.

• Grant portals (NIH eRA Commons, NSF Research.gov). Polled award amendments flow in through API Polling & Portal Integration; when a portal rate-limits or returns 503, the router caches the amendment rather than dropping the poll cursor.
• ERP / finance. Reconciled records upsert into the financial ledger by idempotency key, so a cached budget amendment posts once even after a replay.
• LIMS and lab inventory. Calibration and chemical-inventory telemetry from the Equipment Calibration & Lab Inventory Tracking domain is routed with compliance_tag priority, keeping OSHA/EPA logging unbroken during a subnet outage.
• Async workers. When the platform runs ingestion through a broker, the cache-and-reconcile pattern composes with the retry semantics described in Async Processing & Queue Management.

A reconciliation run emits a structured summary other systems can consume:

{
  "run_id": "0d6b2f9e-1c4a-4f7e-9c2b-7a1e5f0c3d88",
  "window": "2026-06-28T02:00:00Z/2026-06-28T02:14:31Z",
  "drained": 142,
  "skipped_duplicates": 7,
  "quarantine_depth": 3,
  "oldest_cached_age_seconds": 5230
}

Verification & audit #

Confirming a fallback episode resolved correctly is a deterministic check, not a judgement call.

1. Count parity. The cache drained + skipped_duplicates count must equal the number of files the cache held before the run. A shortfall means a file failed to parse and should appear in quarantine.
2. Reproduce the audit hash. Recompute SHA-256 over the canonical payload for a sampled ledger row; it must equal the stored idempotency_key. A mismatch proves the payload was mutated in transit — a chain-of-custody failure.
3. Correlation trace. Query the centralized log platform by X-Correlation-ID to reconstruct a record’s full journey across primary, cache, and reconciliation tiers — the trace the Grant Lifecycle Architecture Design maps to fiscal-year deadlines and audit windows.
4. Quarantine review. Compliance officers review the dead-letter queue on a fixed cadence; every entry carries its raw source payload, rejection reason, correlation id, and timestamp, so a missing grant submission or calibration log is reconcilable rather than lost.

The reconciliation job is itself idempotent: re-running it over an already-drained window produces drained: 0 and changes no ledger state, which is the property an auditor replays to confirm the pipeline is deterministic.

Failure modes & recovery #

Recovery requires strict separation of duties: compliance officers define retention and audit thresholds, developers maintain the idempotent routing logic and schema validators, and lab managers verify equipment telemetry accuracy after a drain. This tripartite boundary keeps fallback routing a predictable, auditable, and federally compliant component of the research automation stack.

Frequently asked questions #

Related #

• Parent guide: Core Architecture & Policy Mapping for Research Grants
• Grant Lifecycle Architecture Design — the state transitions and audit windows fallback reconciles against
• University Policy Mapping Frameworks — the regulatory matrix that bounds every routing decision
• Security Boundary Configuration — encryption and credential scoping for the fallback tiers
• Async Processing & Queue Management — broker-level retry semantics the cache pattern composes with