oxspeak_server/.junie/plans/replace-udp-with-rustrtc.md

sessionId

sessionId
session-260912-101455-1pxv

Requirements

Overview & Goals

Replace the current placeholder raw-UDP voice relay (src/udp/*) with a real WebRTC media stack built on the rustrtc crate (already declared in Cargo.toml). The server becomes a centralized SFU (Selective Forwarding Unit): each connected client opens one PeerConnection with the server, the server decrypts/receives that client's audio (and later video) track and re-encodes/forwards it to every other member of the same voice Channel.

Scope

In Scope

• Remove src/udp/server.rs, src/udp/router.rs raw-socket logic (metrics module is kept/adapted).
• Add a voice module hosting: PeerConnection lifecycle management via rustrtc, SDP offer/answer exchange, ICE candidate exchange, and SFU-style track forwarding scoped by channel_id.
• Reuse the existing WebSocket gateway (GatewayManager / RealtimeRouter / ws_on_message in src/routes/gateway/mod.rs) as the signaling transport — add a Voice namespace to the existing JSON event envelope (GatewayEvent { namespace, action, content }) instead of introducing a new endpoint, per explicit request to avoid breaking the current structure.
• Reuse channel::ChannelType::Voice, channel_user, and the JOIN_VOICE / SPEAK permission bits already defined in src/permissions.rs to authorize who may join/publish in a voice channel.
• Update config.toml / NetworkConfig so the single udp_port continues to be the one UDP port used, now bound through rustrtc's ice_udp_mux (single-port ICE) instead of a raw UdpSocket.
• Update AppMetrics/core::mod::App::run wiring so the new voice service starts/stops alongside the HTTP server, replacing UdpServer.

Out of Scope

• TURN/STUN server configuration (kept minimal per user's answer — LAN/local network scenario for now).
• Video/screen-share specific handling (structure will allow it later, but only audio forwarding is implemented now).
• Client-side (frontend) implementation details beyond the signaling contract needed by the server.

User Stories

• As a user with JOIN_VOICE permission on a voice channel, I want to connect and have my microphone audio heard by other members of that same channel.
• As a user already in a voice channel, I want to hear every other member's audio mixed/forwarded by the server without opening a direct connection to each peer.
• As a server operator, I want the previous ad-hoc UDP relay removed and replaced by a maintained WebRTC stack (rustrtc) so voice traffic is encrypted (SRTP) and NAT-traversal (ICE) works out of the box.

Functional Requirements

• A client sends an SDP offer through the gateway (namespace: "Voice"), scoped to a channel_id; the server validates channel membership/permissions, creates a rustrtc::PeerConnection, and returns an SDP answer through the same gateway connection.
• ICE candidates generated by either side are exchanged as additional Voice gateway events.
• Once connected, the server subscribes the client's inbound audio track to every other PeerConnection currently joined to the same voice channel (SFU fan-out), and adds new joiners' tracks to previously-connected peers as they arrive.
• When a user disconnects (gateway socket closes) or leaves the voice channel, their PeerConnection is closed and their track is removed from all other peers' forwarding sets.
• Existing UdpMetrics-style counters (packets/bytes in/out, errors) are preserved in spirit, adapted to whatever counters rustrtc exposes (or wrapped manually) for the periodic reporter.

Technical Design

Current Implementation

• src/udp/server.rs: binds a raw tokio::net::UdpSocket on NetworkConfig.udp_port, loops on recv_from, and blindly forwards every datagram to all peers registered in a RoutingTable under a hard-coded "default" channel — there is no real protocol, no per-voice-channel isolation, no encryption.
• src/udp/router.rs: RoutingTable maps ChannelId -> Vec<SocketAddr>; join/leave/routing_table_mut() exist but are never called anywhere in the codebase — dead placeholder API.
• src/udp/metrics.rs: atomic counters (packets_received, bytes_sent, …) + spawn_reporter logging every interval. Plugged into AppMetrics.udp (src/metrics/mod.rs).
• src/core/mod.rs: App::build/App::run construct UdpServer::new(&config.network, udp_metrics) and tokio::spawn(udp_server.run()), joined into the same tokio::select!/shutdown-broadcast pattern as HttpServer.
• src/routes/gateway/mod.rs: GatewayManager keeps ConnectionKey{user_id, connection_id} -> GatewayClient{sender: mpsc::UnboundedSender<Message>}; events are pushed as JSON GatewayEvent{namespace, action, content}. GatewayClient::on_message currently just logs incoming text — this is the extension point for voice signaling.
• src/services/realtime_registry.rs: already maintains channel_id -> HashSet<user_id> membership computed from computed_permission (READ_CHANNEL) — reusable to know who is allowed in a channel, but voice-specific "currently connected to voice" state does not exist yet and must be tracked separately (WebSocket presence in a channel != actively broadcasting audio).
• src/permissions.rs: JOIN_VOICE, SPEAK, STREAM, MOVE_OTHERS, DISCONNECT_OTHERS, MANAGE_VOICE_CHANNEL bits already defined but unused by any voice logic today.
• src/models/channel.rs: ChannelType::Voice variant already exists.

Key Decisions

1. Topology: centralized SFU (confirmed by user) — one rustrtc::PeerConnection per connected client per voice channel; the server forwards each publisher's decoded RTP track to every other subscriber in that channel. No client-to-client PeerConnections.
2. Signaling transport: existing WebSocket gateway (confirmed by user, to avoid restructuring) — SDP offer/answer and ICE candidates are carried as new Voice namespace events inside the current GatewayEvent envelope, handled in GatewayClient::on_message (currently a stub) and dispatched to a new VoiceService.
3. Single UDP port reuse — rustrtc's RtcConfiguration will be set up with ice_udp_mux = true and bound to the existing NetworkConfig.udp_port, so the media/ICE traffic keeps using the same single port previously owned by the raw UdpServer, minimizing config/infra changes (firewall rules, config.toml stay compatible).
4. No TURN/STUN for now (confirmed by user) — RtcConfiguration.ice_servers left empty/default; can be added later via config.toml without further architecture changes.
5. Voice presence tracked separately from RealtimeRegistry — a new lightweight in-memory registry (VoiceRoom per channel_id) tracks which PeerConnections are actively publishing/subscribing in a voice channel, since being subscribed to gateway events (RealtimeRegistry) is not the same as being connected to voice media.

Proposed Changes

• Replace the src/udp module with a new src/voice module:
  • voice/mod.rs: public API, exports VoiceService.
  • voice/service.rs: VoiceService owns a rustrtc::RtcConfiguration (built from NetworkConfig), a map channel_id -> VoiceRoom, and methods handle_offer(user_id, channel_id, sdp) -> answer_sdp, handle_ice_candidate(...), leave(user_id, channel_id).
  • voice/room.rs: VoiceRoom holds HashMap<user_id, Arc<PeerConnection>> for one channel; implements track fan-out — on receiving a remote track from peer A, it calls into every other peer's PeerConnection to add/forward that track (rustrtc PeerConnection API, per its Usage example: create connection, set_remote_description, create_answer, set_local_description, subscribe to on_track).
  • voice/metrics.rs: keep counters analogous to today's UdpMetrics (packets/bytes/errors), fed by hooks around track forwarding, still exposed through AppMetrics.udp (renamed AppMetrics.voice where feasible) and the existing reporter::spawn_reporter cadence.
• Extend GatewayEvent handling: add "Voice" namespace with actions like offer, answer, ice-candidate, leave; GatewayClient::on_message parses these and calls VoiceService through AppState, then pushes the answer/ICE-candidate response back on the same mpsc::UnboundedSender<Message> used for all other gateway events (no protocol reinvention).
• Permission check on offer: verify the user has ChannelPermission::JOIN_VOICE (and SPEAK to publish) on the target channel, reusing the same computed_permission lookups already used by RealtimeRegistry/ http/permissions.rs.
• Update src/core/mod.rs: drop UdpServer::new / udp_server.run() / udp_shutdown_tx; construct VoiceService instead and store it in AppState (alongside gateway, services) so gateway handlers can call it; no UDP socket bind/run task is spawned by App::run anymore — rustrtc manages its own I/O internally once configured.
• Update src/config.rs: keep NetworkConfig.udp_port (renamed conceptually to "voice/media port" in comments) feeding rustrtc's ice_udp_mux_port; update DEFAULT_CONFIG_TOML comment accordingly.
• Remove src/udp/ directory entirely once voice/ fully replaces its responsibilities; update Cargo.toml/module declarations (lib.rs) accordingly.

Data Models / Contracts

// Gateway JSON contract additions (namespace = "Voice")
{ "namespace": "Voice", "action": "offer", "content": { "channel_id": Uuid, "sdp": String } }
{ "namespace": "Voice", "action": "answer", "content": { "channel_id": Uuid, "sdp": String } }
{ "namespace": "Voice", "action": "ice-candidate", "content": { "channel_id": Uuid, "candidate": String } }
{ "namespace": "Voice", "action": "leave", "content": { "channel_id": Uuid } }

pub struct VoiceService {
    config: rustrtc::RtcConfiguration,
    rooms: RwLock<HashMap<Uuid, VoiceRoom>>, // channel_id -> room
}

impl VoiceService {
    pub async fn handle_offer(&self, user_id: Uuid, channel_id: Uuid, sdp: String) -> anyhow::Result<String>;
    pub async fn handle_ice_candidate(&self, user_id: Uuid, channel_id: Uuid, candidate: String) -> anyhow::Result<()>;
    pub fn leave(&self, user_id: Uuid, channel_id: Uuid);
}

Architecture Diagram

graph TD
    ClientA[Client A - browser] -- WebSocket gateway: Voice offer/answer/ICE --> Gateway[GatewayManager / on_message]
    ClientB[Client B - browser] -- WebSocket gateway: Voice offer/answer/ICE --> Gateway
    Gateway -- dispatch --> VoiceService
    VoiceService -- creates/owns --> RoomA[VoiceRoom per channel_id]
    RoomA -- PeerConnection A --> PCA[rustrtc PeerConnection A]
    RoomA -- PeerConnection B --> PCB[rustrtc PeerConnection B]
    ClientA == RTP/SRTP media, single UDP port ==> PCA
    ClientB == RTP/SRTP media, single UDP port ==> PCB
    PCA -- forward audio track --> PCB
    PCB -- forward audio track --> PCA

Risks

• rustrtc is a young/fast-moving crate (frequent point releases per the benchmark notes found) — API surface for track-forwarding/SFU usage should be validated against the pinned 0.3.133 docs during implementation; if a needed primitive (e.g., explicit track re-publishing helper) is missing, a thin adapter layer will be needed inside voice/room.rs.
• Moving from "no real protocol" to full SDP/ICE negotiation is a larger surface than the previous placeholder; permission checks (JOIN_VOICE/SPEAK) must be enforced before any PeerConnection is created to avoid unauthorized channel joins.
• Since signaling now flows through the same WebSocket used for all other realtime events, malformed/large SDP payloads must not block the GatewayClient message loop — the voice service calls will be dispatched without blocking other event types.

Delivery Steps

* Step 1: Scaffold the voice module and RtcConfiguration from NetworkConfig

A new src/voice module exists with a VoiceService capable of building a rustrtc::RtcConfiguration from the app's network config, replacing the old src/udp module's role in core/mod.rs.

• Create src/voice/mod.rs, src/voice/service.rs, src/voice/room.rs, src/voice/metrics.rs.
• Port UdpMetrics-style counters into voice/metrics.rs (packets/bytes in/out, errors), keeping the Metrics/ MetricsSnapshot trait implementations used by crate::metrics::reporter.
• Build RtcConfiguration/RtcConfigurationBuilder in VoiceService::new(&NetworkConfig, metrics) using ice_udp_mux = true and ice_udp_mux_port = network.udp_port, no ICE servers configured.
• Remove src/udp/server.rs and src/udp/router.rs; delete unused RoutingTable (dead code confirmed unused elsewhere).
• Update src/core/mod.rs (App::build/App::run) to construct VoiceService instead of UdpServer, store it on AppState, and drop the now-unneeded udp_shutdown_tx/udp_handle spawn/join wiring.
• Update src/metrics/mod.rs (AppMetrics) to reference the new voice metrics type.

Step 2: Implement per-channel PeerConnection lifecycle in VoiceRoom

VoiceService/VoiceRoom can accept an SDP offer for a given channel, create a rustrtc::PeerConnection, and return an SDP answer, tracking connections per channel_id.

• Implement VoiceRoom (channel_id -> HashMap<user_id, Arc<PeerConnection>>).
• Implement VoiceService::handle_offer(user_id, channel_id, sdp): looks up/creates the room, creates a PeerConnection via rustrtc, calls set_remote_description/create_answer/set_local_description, returns the answer SDP.
• Implement VoiceService::handle_ice_candidate and VoiceService::leave, closing and removing the PeerConnection from its room.
• Enforce ChannelPermission::JOIN_VOICE (and SPEAK for publishing) before creating a PeerConnection, reusing the computed_permission lookup pattern from src/services/realtime_registry.rs.

Step 3: Wire SFU track forwarding between peers in the same channel

Audio published by one connected client in a voice channel is forwarded by the server to every other client connected to the same channel.

• On PeerConnection::on_track for a given user's connection, register the inbound track on the owning VoiceRoom.
• For every other PeerConnection already in that VoiceRoom, add/forward the new track (SFU fan-out) using rustrtc 's track/MediaCapabilities APIs.
• When a new peer joins an existing room, subscribe it to all tracks already being forwarded by other room members.
• On leave/disconnect, remove the peer's published track from all other peers' forwarding sets and close its PeerConnection.

Step 4: Expose Voice signaling through the existing WebSocket gateway

Clients can perform the full offer/answer/ICE handshake over the existing gateway WebSocket connection using a new Voice namespace, with no new HTTP/WS endpoint introduced.

• Extend GatewayClient::on_message in src/routes/gateway/mod.rs to parse GatewayEvent{namespace: "Voice", action, content} messages (offer, answer, ice-candidate, leave).
• Dispatch parsed messages to VoiceService (accessed via AppState) and send the resulting answer/ICE-candidate events back through the client's existing mpsc::UnboundedSender<Message>.
• On gateway disconnect (GatewayClient::on_disconnect), call VoiceService::leave for any channel the user was actively connected to in voice.
• Update config.toml's DEFAULT_CONFIG_TOML comment for udp_port to reflect its new role as the rustrtc ICE/media mux port.