Skip to content

ADR-004: Mercury Outbox

Date Author Status
2026-05-09 Fabian Beyerlein Accepted

Context

All application messages that Mercury sends to the cloud need to be delivered in a durable manner. Currently, if Mercyry tries to send a message and it fails (either because network offline, session expired or handshake in progress, etc.) the message is lost forever. As these messages can be business critical operations like order confirmations or other transactional actions, message loss must be avoided.

Decision

Mercury's Tunnel module implements a outbox pattern. Instead of sending every message directly to the gRPC stream, it is first written to the outbox (Storage module based on etcd-io/bbolt). Then, a background process picks up a batch of pending messages, processes them and marks them as in-flight.

Later, Nexus sends an ACK message back to Mercury. Based on the ACK status, Mercury either marks the message as processed, marks it as dead-lettered or marks it for retry.

Updated Message Envelope: 

message AgentMessageEnvelope {
  string                    message_id     = 1; // globally unique, stable across retries
  string                    correlation_id = 2;
  uint64                    sequence       = 3;
  google.protobuf.Timestamp timestamp      = 4;
  string                    message_type   = 5;
  bytes                     payload        = 6;
  map<string, string>       metadata       = 7;
  uint32                    attempt        = 8;

  oneof control {
    TunnelAck        ack       = 100;
    Pong             pong      = 101;
    HeartbeatRequest heartbeat = 102;
  }
}

message CloudMessageEnvelope {
  string                    message_id       = 1; // globally unique, stable across retries
  string                    correlation_id   = 2;
  uint64                    sequence         = 3;
  google.protobuf.Timestamp timestamp        = 4;
  string                    message_type     = 5;
  bytes                     payload          = 6;
  map<string, string>       metadata         = 7;
  uint32                    attempt          = 8;
  string                    targent_agent_id = 99;

  oneof control {
    TunnelAck         ack       = 100;
    Ping              ping      = 101;
    HeartbeatResponse heartbeat = 102;
    Reconnect         reconnect = 103;
  }
}

Tunnel ACK 

message TunnelAck {
  string    message_id = 1;
  AckStatus status     = 2;
  string    reason     = 3;
}

enum AckStatus {
  ASUnspecified         = 0;
  ASAccepted            = 1;
  ASRetryableError      = 2;
  ASPermanentlyRejected = 3;
}

Outbox Record 

message OutboxRecord {
  string                             message_id    = 1;
  OutboxRecordStatus                 status        = 2;
  uint32                             attempts      = 3;
  bytes                              payload       = 4;
  google.protobuf.Timestamp          enqueued_at   = 5;
  google.protobuf.Timestamp          updated_at    = 6;
  optional google.protobuf.Timestamp next_retry_at = 7;
}

enum OutboxRecordStatus {
  ORSUnspecified = 0;
  ORSPending     = 1;
  ORSInFlight    = 2;
  ORSRetryable   = 3;
  ORSDeadLetter  = 4;
  ORSAcked       = 5;
}