STP - Shoal Transport Protocol

STP is a lightweight transport-layer library for inter-process communication within the Shoal system.

What it Does

STP defines transport primitives and reference behavior for:

• Message delivery between Shoal components
• Connection management
• Optional congestion control
• Training coordination (heartbeats, progress, checkpoint metadata)
• Gradient synchronization signals

Training Communication Types

STP extends for training pipeline with:

• CommunicationType::training_progress - Pod reports TrainingMetrics (step, loss, throughput, GPU memory)
• CommunicationType::checkpoint_uploaded - Pod notifies Hatchery of checkpoint save
• CommunicationType::resource_report - Pod reports available VRAM, storage, mode state
• CommunicationType::training_control - Hatchery sends Start/Pause/Resume/Abort/CheckpointNow

How to Use

Core - Communication Values

The core defines a CommunicationValue with:

• Communication Type - Identifies the type of message
• Sender ID - Optional identifier of sender
• Receiver ID - Optional identifier of receiver
• Message ID - Optional unique message identifier
• Data Value - The payload (Container, Array, Boolean, Number, String, or Null)

DataValues are constructed from DataTypes (1-byte arbitrary mappings). See core/src/data_types.rs for standard mappings.

Native - WTransport Communication

The native crate provides inter-process communication via WTransport:

Host - Accepts client connections:

let host = Host::new(addr);
while let Ok((sender, receiver)) = host.next().await {
    // Handle connected client
}

Client - Connects to host:

let client = Client::new();
let (sender, receiver) = client.connect(addr).await;

Connection - Split into Sender/Receiver pairs for bidirectional messaging.

API

The api/ crate provides Axum server integration for STP communication.

Web

The web/ crate provides JavaScript/Node.js implementations.

Architecture

stp/
├── core/      # Protocol logic (pure Rust, platform-agnostic)
├── native/    # WTransport integration
├── api/       # Axum HTTP server
└── web/       # JavaScript/Node client