Traffic Stats: Adaptive QoS with nftables, TC-Cake, and TimescaleDB

Written on: 2026-06-21 Download PDF

Congested Network, Skyrocketing Latency, and Useless Routers

You know when someone on your local network starts a backup to the cloud, or starts downloading a heavy file, and suddenly your VoIP call crackles, DNS replies after seconds, and ping on online games shoots up to 600ms? Well, that annoying phenomenon is called bufferbloat, and it is because commercial routers pile up packets in their internal buffers until they burst, instead of discarding or routing them intelligently.

The standard cure for this disease is applying QoS (Quality of Service) queues to limit bandwidth. But if you have an unstable connection, like FWA or a 4G/5G SIM where the actual speed changes constantly based on the weather and cell load, a static configuration is useless.

To address this issue, an Adaptive QoS system was developed to continuously monitor network latency and dynamically adjust bandwidth limits on the firewall.

This document details the low-level implementation using nftables, TC-Cake, and TimescaleDB, optimized to maintain high throughput while avoiding unnecessary disk write wear.

The source code and deployment scripts are available at:
https://git.dreadful.work/errno0x0d/traffic-stats

The Architecture

The setup is split across two nodes to avoid overloading the firewall with databases or graphical interfaces:

The Firewall: Does the dirty work. It collects counters for individual IPs on the local network via nftables, measures RTT (latency) to the internet, adapts shaping queues, and sends logs to the database.
The Centralized Server (antares): Hosts the TimescaleDB database (PostgreSQL with time-series extensions) in a Docker container and exposes a Flask web dashboard with real-time graphs.

Network Interfaces (And How Not to Get Lost Among System Names)

If you read the configuration files, you will find a mess of physical names like enp1s0, enp9s0, and so on. Let's make sense of it by explaining what each interface in our system represents:

wan (in our case enp1s0): The physical interface of the firewall connected directly to the external network (internet) via the ISP's modem. This is where we apply upload shaping.
lan_trust (in our case enp9s0): Our main, trusted local network. This is where the devices whose bandwidth consumption we want to monitor reside.
fe (enp10s0) and be (enp8s0): Physical interfaces for the Frontend and Backend DMZs, respectively.
wg0 / wt0: Virtual interfaces dedicated to VPNs (WireGuard/NetBird).
ifb0 (Intermediate Functional Block): This is a crucial virtual interface. Linux, by its nature, cannot perform shaping (i.e., limiting bandwidth) on incoming (ingress) traffic efficiently. To bypass this limitation, we create ifb0, redirect all incoming traffic from wan (enp1s0) into it, and then apply download shaping rules on ifb0.

Firewall Setup: TC-Cake and nftables

1. DSCP Tagging with nftables

Before limiting bandwidth, we must tell Linux which packets are important and which can wait. We do this by setting DSCP (DiffServ) marks in the firewall:

table ip filter {
    chain forward {
        type filter hook forward priority filter; policy accept;

        # Absolute priority to VoIP (our local PBX)
        ip saddr 192.168.152.3 ip dscp set ef
        ip daddr 192.168.152.3 ip dscp set ef

        # Priority interactive traffic and DNS
        tcp dport 22 ip dscp set cs2
        udp dport 53 ip dscp set cs6

        # Low priority traffic (bulk downloads or non-priority IPs)
        ip saddr { 192.168.152.60, 192.168.152.61 } ip dscp set cs1
    }
}

2. TC (Traffic Control) Rules

The net-admin Python daemon executes these commands at boot to initialize the shaping queues based on the Cake algorithm:

# Activate the IFB interface for download
ip link set dev ifb0 up

# Clean up any old configurations on the physical WAN
tc qdisc del dev enp1s0 root 2>/dev/null
tc qdisc del dev enp1s0 ingress 2>/dev/null

# Configure Cake in Upload on the WAN
tc qdisc add dev enp1s0 root cake bandwidth 45Mbit besteffort triple-ack ack-filter nat rtt 100ms

# Mirror incoming WAN traffic to ifb0
tc qdisc add dev enp1s0 handle ffff: ingress
tc filter add dev enp1s0 parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0

# Configure Cake in Download on ifb0
tc qdisc del dev ifb0 root 2>/dev/null
tc qdisc add dev ifb0 root cake bandwidth 220Mbit besteffort triple-ack wash nat rtt 100ms

The daemon calculates latency dynamically: if it detects congestion, it updates the bandwidth limits (bandwidth XXXMbit) on enp1s0 and ifb0, lowering them until finding a stable point.

How to Prevent Frying the Server's SSD or SD Card

If you installed the database on a cheap SD card or SSD, writing traffic logs for every single local IP every 10 seconds will kill the disk in a few months due to write wear.

To prevent this, we implemented two critical optimizations.

Optimization 1: synchronous_commit = off

In TimescaleDB's docker-compose.yml on antares, we disabled synchronous commit writes to the physical disk:

services:
  timescaledb:
    image: timescale/timescaledb:latest-pg16
    container_name: traffic_timescaledb
    command: ["postgres", "-c", "synchronous_commit=off"]

With synchronous_commit=off, PostgreSQL confirms the write to the client as soon as the data is in RAM, flushing it to disk in cumulative blocks every few seconds. If power cuts out, you lose the last few moments of logs (irrelevant for traffic statistics), but SSD wear drops drastically.

Optimization 2: Midnight Aggregation and 15-Day Retention

Keeping high-resolution logs recorded every 10 seconds for months makes no sense. Raw logs in the device_logs table are kept for only 15 days. Every night at midnight, a Python script (daily_aggregate.py) aggregates the total bytes consumed by each IP during the day and writes the summary to the device_daily_stats table.

Cleaning up old raw logs on TimescaleDB is not done using a slow DELETE (which would lock the table and spike disk I/O), but by directly dropping time partitions (chunks) via TimescaleDB:

SELECT drop_chunks('device_logs', INTERVAL '15 days');

This keeps the log table lean, and Flask dashboard queries respond in under 50ms, even on limited hardware.

Discussion & Support

Discussion and support channel. To discuss configurations or other topics, join us on fluxer.dreadful.work.