Fluxer Deployment Guide: Microservices, SeaweedFS S3, and WebRTC Voice via LiveKit SFU

Fluxer Deployment Guide: Microservices, SeaweedFS S3, and WebRTC Voice via LiveKit SFU

In the realm of operational and sensitive communications infrastructure, relying on third-party commercial SaaS platforms (such as Discord, Slack, or Microsoft Teams) introduces significant vulnerabilities regarding data sovereignty, regulatory adherence (GDPR/NIS2), and upstream service disruption.

Fluxer delivers a high-performance, self-hosted platform for real-time messaging, community coordination, and low-latency voice/video communication. It implements an isolated microservice architecture engineered for scalable on-premises deployments: a relational database (PostgreSQL 16), high-speed in-memory caching (Valkey/Redis), S3-compatible object storage (SeaweedFS) for persistent avatars and attachments, and a real-time media engine (LiveKit SFU).

This guide documents the complete deployment procedure from scratch, focusing on an uncompromising enterprise firewall traversal architecture: zero non-standard open ports. The entire platform operates exclusively on port 443 (primary WebRTC media on 443/UDP, with web and WebRTC fallback multiplexed on 443/TCP using Nginx Stream L4 with SNI Preread).

Prerequisites and Scenario Parameters

To ensure all configuration files, .env templates, and commands are unambiguous and immediately testable, this guide uses standardized documentation placeholders adhering to IETF standards (RFC 2606 for domains and RFC 5737 for IP blocks).

Anyone implementing this stack must replace these example values with their actual network parameters before booting the containers:

Architecture and Microservice Topology

All backend components operate inside an isolated Docker bridge network. Internal databases, file storage, and secondary daemon ports are never exposed to the public Internet. External clients see exclusively port 443 (TCP and UDP) along with port 80 (for permanent HTTP redirection):

flowchart TD
    subgraph Clients ["Client Applications"]
        Desktop["Desktop Client / Web App"]
        Mobile["Mobile Client"]
    end

    subgraph Perimeter ["Anti-Firewall Perimeter Frontend (Port 443 Only)"]
        StreamRouter["Nginx Stream L4 Router\n(Port 443/TCP with ssl_preread)"]
        NginxHTTP["Nginx Internal HTTPS Virtual Host\n(Internal Port :8443 TCP)"]
        LiveKit["LiveKit SFU Server (Bridge Mode)\n(Port 443/UDP Primary Media)\n(Internal Port :7881 TCP Fallback)"]
    end

    subgraph CoreStack ["Fluxer Application Suite (Private Network)"]
        CaddyEdge["Internal Caddy Ingress\n(:8080)"]
        Gateway["Fluxer Gateway & API\n(REST and WebSockets)"]
        Auth["Authentication & Session Module"]
        VoiceCtrl["Fluxer LiveKit Controller"]
        Search["Meilisearch Engine\n(Full-Text Indexing)"]
    end

    subgraph DataPersistence ["Persistence & Storage Layer"]
        Postgres["PostgreSQL 16\n(Guilds, Channels, Roles, Permissions)"]
        Valkey["Valkey 7.2\n(Session Cache and Pub/Sub Bus)"]
        Seaweed["SeaweedFS Filer + Volume\n(S3-Compatible Object Storage)"]
    end

    Desktop -->|Web & WSS: fluxer.example.org:443 TCP| StreamRouter
    Mobile -->|Web & WSS: fluxer.example.org:443 TCP| StreamRouter
    Desktop -.->|WebRTC Primary: :443 UDP| LiveKit
    Mobile -.->|WebRTC Primary: :443 UDP| LiveKit
    Desktop -.->|WebRTC Fallback: turn.example.org:443 TCP| StreamRouter
    Mobile -.->|WebRTC Fallback: turn.example.org:443 TCP| StreamRouter

    StreamRouter -->|SNI: fluxer.example.org| NginxHTTP
    StreamRouter -->|SNI: turn.example.org| LiveKit
    NginxHTTP -->|Proxy Pass :8080 TCP| CaddyEdge

    CaddyEdge --> Gateway
    CaddyEdge --> Auth
    CaddyEdge --> Search
    
    Gateway --> Valkey
    Gateway --> Postgres
    Gateway --> VoiceCtrl
    VoiceCtrl -->|RPC Token Minting| LiveKit
    Gateway --> Seaweed

Anti-Firewall Architecture: 443/UDP and 443/TCP via SNI Preread

In corporate, banking, or public administration environments, network firewalls and Deep Packet Inspection (DPI) appliances enforce strict policies: all ports except 80 and 443 are dropped. Demanding that users connect to non-standard ports (such as 7881 or dynamic WebRTC UDP pools like 50000:60000) causes outright connection failure.

To overcome this, the architecture unifies all ingress traffic onto port 443:

1. Primary Transport on 443/UDP (Simulating QUIC/HTTP-3): Because the Linux kernel maintains independent socket namespaces for TCP and UDP, the LiveKit container (operating in bridge mode) directly publishes 443:443/udp. Perimeter firewalls handle this traffic identically to standard HTTP/3 (QUIC) flows, allowing ultra-low latency voice streams through.
2. L4 Multiplexing on 443/TCP via Nginx ssl_preread: On port 443/TCP, the Nginx stream module listens with ssl_preread on. Nginx inspects the initial TLS ClientHello packet and extracts the SNI hostname without decrypting the payload:
   • If the SNI matches the web domain (fluxer.example.org), the TCP stream forwards to the local HTTPS engine at 127.0.0.1:8443 handling TLS, web app assets, and persistent WebSockets.
   • If the SNI matches the WebRTC fallback endpoint (turn.example.org), the TCP stream routes directly at Layer 4 to LiveKit's internal TCP media socket (127.0.0.1:7881).
3. Operational Result: Externally, only port 443 exists. Even inside restrictive networks that drop 100% of outbound UDP packets, WebRTC client engines automatically switch to TCP fallback and reach the server cleanly over standard 443/TCP.

1. Kernel Optimization and Socket Buffers

High-density WebRTC media streams and sustained WebSocket sessions require tuned kernel socket buffers to avoid packet drops under load.

Create /etc/sysctl.d/99-fluxer-performance.conf:

cat << 'EOF' | sudo tee /etc/sysctl.d/99-fluxer-performance.conf
# High-capacity UDP buffers for LiveKit SFU WebRTC
net.core.rmem_max = 5000000
net.core.rmem_default = 1048576
net.core.wmem_max = 5000000
net.core.wmem_default = 1048576

# TCP backlog sizing for concurrent connection bursts
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 8192

# Enable IP forwarding for Docker bridge networks
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF

sudo sysctl --system

Install Docker Engine and Docker Compose v2:

sudo apt-get update && sudo apt-get install -y ca-certificates curl gnupg lsb-release
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo systemctl enable --now docker

2. Directory Tree and Configuration Layout

Set up /opt/fluxer and the persistent storage volume mounts:

sudo mkdir -p /opt/fluxer/data/{postgres,valkey,seaweedfs/volume,seaweedfs/filerldb2,meili}
sudo mkdir -p /opt/fluxer/config
cd /opt/fluxer

LiveKit Configuration (/opt/fluxer/config/livekit.yaml)

Define the single-port UDP 443 transport and internal TCP 7881 fallback socket (reached internally by Nginx):

PUBLIC_IP=$(curl -s ifconfig.me)
echo "Detected Public IP: $PUBLIC_IP"

Configure /opt/fluxer/config/livekit.yaml:

# /opt/fluxer/config/livekit.yaml
port: 7880
bind_addresses:
  - "0.0.0.0"

rtc:
  udp_port: 443
  tcp_port: 7881
  use_external_ip: false
  # CRITICAL CONFIGURATION NOTE:
  # Enter your server's real public IP address here (e.g. from curl ifconfig.me).
  # DO NOT leave 203.0.113.10: LiveKit announces this IP to WebRTC clients!
  node_ip: 203.0.113.10

turn:
  enabled: false

keys:
  fluxer: 4b47a61d0ff70bbc9d3dc92d87ac3912be77a42db3dc2c11c37ca2a49833ac9b

logging:
  level: info

3. Environment Variables (/opt/fluxer/.env)

Configure /opt/fluxer/.env replacing the placeholders with your production parameters:

cat << 'EOF' > /opt/fluxer/.env
# Domain and Public Endpoints
FLUXER_DOMAIN=fluxer.example.org
FLUXER_PUBLIC_URL=https://fluxer.example.org
FLUXER_API_URL=https://fluxer.example.org/api

# Authentication and Session Secrets (Generate with: openssl rand -hex 32)
JWT_SECRET=c8d4f9b2a7e1358d601f2e4b89ca0517d3b9e4a816c20573e8fa912567bcde34
COOKIE_SECRET=7f12e89d604b1c5a932e48fa017bd6c39a852e147f0bc63d8912e54a7bc901ef

# Relational Database: PostgreSQL 16
POSTGRES_DB=fluxer
POSTGRES_USER=fluxer
POSTGRES_PASSWORD=SuperSecretPostgresPassword2026!
POSTGRES_HOST=postgres
POSTGRES_PORT=5432

# In-Memory Cache: Valkey
VALKEY_HOST=valkey
VALKEY_PORT=6379

# Object Storage: SeaweedFS S3
SEAWEEDFS_FILER=seaweedfs:8888
SEAWEEDFS_S3_ENDPOINT=http://seaweedfs:8333
S3_ACCESS_KEY=fluxer_storage_key
S3_SECRET_KEY=fluxer_storage_super_secret_key_2026
S3_BUCKET=fluxer-media

# Search Indexer: Meilisearch
MEILI_MASTER_KEY=MeiliSearchSecureMasterKey2026!
MEILI_HOST=http://meilisearch:7700

# LiveKit SFU Integration Parameters (Consistent with livekit.yaml)
FLUXER_LIVEKIT_URL=https://fluxer.example.org/livekit
FLUXER_LIVEKIT_INTERNAL_URL=http://livekit:7880
FLUXER_LIVEKIT_API_KEY=fluxer
FLUXER_LIVEKIT_API_SECRET=4b47a61d0ff70bbc9d3dc92d87ac3912be77a42db3dc2c11c37ca2a49833ac9b
FLUXER_LIVEKIT_UDP_PORT=443
EOF

4. Multi-Container Orchestration (docker-compose.yaml)

All containers run inside the isolated fluxer-net bridge network. LiveKit publishes only 443/udp publicly, while 7880 (signaling) and 7881 (fallback ICE) bind strictly to 127.0.0.1 for Nginx:

# /opt/fluxer/docker-compose.yaml
services:
  # Internal Ingress Caddy
  edge:
    image: caddy:2-alpine
    container_name: fluxer-edge
    restart: unless-stopped
    ports:
      - "127.0.0.1:8080:8080"
    volumes:
      - ./config/Caddyfile:/etc/caddy/Caddyfile:ro
    networks:
      - fluxer-net
    depends_on:
      - api

  # Core API & Gateway
  api:
    image: ghcr.io/fluxerapp/server:latest
    container_name: fluxer-api
    restart: unless-stopped
    env_file: .env
    networks:
      - fluxer-net
    depends_on:
      postgres:
        condition: service_healthy
      valkey:
        condition: service_started
      seaweedfs:
        condition: service_started
      livekit:
        condition: service_started

  # Relational Database: PostgreSQL 16
  postgres:
    image: postgres:16-alpine
    container_name: fluxer-postgres
    restart: unless-stopped
    environment:
      POSTGRES_DB: ${POSTGRES_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - ./data/postgres:/var/lib/postgresql/data
    networks:
      - fluxer-net
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
      interval: 5s
      timeout: 5s
      retries: 5

  # In-Memory Cache: Valkey 7.2
  valkey:
    image: valkey/valkey:7.2-alpine
    container_name: fluxer-valkey
    restart: unless-stopped
    command: valkey-server --save 60 1 --loglevel warning
    volumes:
      - ./data/valkey:/data
    networks:
      - fluxer-net

  # Media Storage: SeaweedFS (Filer and S3 Endpoint)
  seaweedfs:
    image: chrislusf/seaweedfs:latest
    container_name: fluxer-seaweedfs
    restart: unless-stopped
    command: "server -dir=/data -s3 -filer -volume.max=0"
    volumes:
      - ./data/seaweedfs/volume:/data
      - ./data/seaweedfs/filerldb2:/data/filerldb2
    networks:
      - fluxer-net

  # Search Engine: Meilisearch
  meilisearch:
    image: getmeili/meilisearch:v1.6
    container_name: fluxer-meilisearch
    restart: unless-stopped
    environment:
      MEILI_MASTER_KEY: ${MEILI_MASTER_KEY}
      MEILI_NO_ANALYTICS: "true"
    volumes:
      - ./data/meili:/meili_data
    networks:
      - fluxer-net

  # LiveKit SFU WebRTC Media Server (Anti-Firewall Bridge Mode)
  livekit:
    image: livekit/livekit-server:latest
    container_name: fluxer-livekit
    restart: unless-stopped
    command: ["--config", "/etc/livekit.yaml"]
    volumes:
      - ./config/livekit.yaml:/etc/livekit.yaml:ro
    ports:
      - "127.0.0.1:7880:7880"   # Internal signaling to Nginx
      - "443:443/udp"           # Primary WebRTC Media (External)
      - "127.0.0.1:7881:7881"   # Internal TCP fallback to Nginx Stream
    networks:
      - fluxer-net

networks:
  fluxer-net:
    name: fluxer-net
    driver: bridge

5. Nginx Configuration: Stream L4 SNI Preread & Virtual Host

The Nginx setup operates across two layers: L4 stream routing in nginx.conf and HTTPS termination in sites-available.

5.1 The stream Block in /etc/nginx/nginx.conf

Add the stream block at the root level of /etc/nginx/nginx.conf (outside the http { } block):

# Add to /etc/nginx/nginx.conf at root level
stream {
    # SNI Map: Route incoming TLS traffic based on the client's Server Name Indication
    map $ssl_preread_server_name $backend_pool {
        turn.example.org    127.0.0.1:7881; # LiveKit WebRTC TCP Fallback
        default             127.0.0.1:8443; # Nginx Local HTTPS Virtual Host
    }

    server {
        listen 443;
        listen [::]:443;
        proxy_pass $backend_pool;
        ssl_preread on;
        proxy_connect_timeout 10s;
        proxy_timeout 24h;
    }
}

5.2 The HTTPS Virtual Host in /etc/nginx/sites-available/fluxer.example.org.conf

Because external 443/TCP is managed by the stream module, the HTTPS virtual host listens internally on 127.0.0.1:8443:

# /etc/nginx/sites-available/fluxer.example.org.conf
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

# HTTP Port 80 Redirect
server {
    listen 80;
    server_name fluxer.example.org turn.example.org;
    return 301 https://$host$request_uri;
}

# Internal HTTPS Virtual Host (Receives from Nginx Stream at 127.0.0.1:8443)
server {
    listen 127.0.0.1:8443 ssl http2;
    server_name fluxer.example.org;

    ssl_certificate /etc/letsencrypt/live/fluxer.example.org/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/fluxer.example.org/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    client_max_body_size 100M;

    # LiveKit SFU WebSocket Signaling
    location ^~ /livekit/ {
        proxy_pass http://127.0.0.1:7880/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_read_timeout 86400s;
        proxy_send_timeout 86400s;
    }

    # Primary Fluxer Ingress
    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
        proxy_read_timeout 3600s;
    }
}

Enable the configuration and restart Nginx:

sudo ln -sf /etc/nginx/sites-available/fluxer.example.org.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl restart nginx

6. Functional Verification and Testing

Start the full stack:

cd /opt/fluxer && sudo docker compose up -d

Verify the single-port ingress points:

# 1. Verify standard HTTPS Ingress on 443/TCP
curl -I https://fluxer.example.org/_health
# Expected: HTTP/2 200 OK

# 2. Verify LiveKit WebSocket signaling endpoint
curl -I https://fluxer.example.org/livekit/
# Expected: HTTP/1.1 200 OK (or WebSocket 400 Bad Request if unauthenticated)

# 3. Test Primary UDP 443 Socket Ingress
nc -zvu fluxer.example.org 443
# Expected: Connection to fluxer.example.org 443 port [udp/https] succeeded!

# 4. Test L4 SNI Preread Routing over standard 443/TCP
openssl s_client -connect turn.example.org:443 -servername turn.example.org
# Expected: Connection established to LiveKit TCP backend without reset

7. Disaster Recovery and Scheduled Maintenance

Schedule an automated backup task to archive databases and object storage:

#!/usr/bin/env bash
set -euo pipefail
BACKUP_DIR="/backup/fluxer/$(date +%F)"
mkdir -p "$BACKUP_DIR"

# Consistent PostgreSQL Logical Dump
docker exec -t fluxer-postgres pg_dumpall -U fluxer > "$BACKUP_DIR/fluxer_db.sql"

# SeaweedFS Volume and Metadata Snapshot
rsync -aAX /opt/fluxer/data/seaweedfs/filerldb2/ "$BACKUP_DIR/seaweedfs_meta/"
rsync -aAX /opt/fluxer/data/seaweedfs/volume/ "$BACKUP_DIR/seaweedfs_data/"

# Archive Compression
tar -czvf "$BACKUP_DIR.tar.gz" -C "$BACKUP_DIR" .
rm -rf "$BACKUP_DIR"

Combining container isolation via bridge networking, primary media on 443/UDP, and Layer 4 SNI Preread multiplexing on standard 443/TCP delivers an enterprise-resilient voice collaboration infrastructure that bypasses strict perimeter firewalls without exposing arbitrary non-standard ports.

Discussion & Support

Discussion and support channel. To discuss configurations or other topics, join us on fluxer.dreadful.work.