Lemmy: Mitigating SSRF Filtering and Media Handling

Lemmy: Mitigating SSRF Filtering and Media Handling

Deploying a federated Lemmy instance in isolated or custom-routed network environments requires careful configuration of proxy and container networking layers.

In topologies with private domains or internal aliases, interactions between containerized services and Lemmy's security constraints can lead to unexpected validation issues, particularly when saving and caching user avatars via Pictrs. This article examines the infrastructure design and how to resolve issues caused by Lemmy's anti-SSRF protections.

Base Architecture

The recommended setup deploys the core application stack (Lemmy, Lemmy-UI, Pictrs, PostgreSQL) within an isolated Docker network, fronted by an Nginx reverse proxy.

Routing is handled through the proxy tier:

1. The ingress endpoint receives traffic on port 80 and forwards it to 127.0.0.1:10080.
2. The docker-compose.yml publishes the Nginx container port on 127.0.0.1:10080.
3. Nginx inside the Docker network inspects and routes requests: API calls to lemmy:8536, frontend requests to lemmy-ui:1234.

We add a few configuration parameters (like defining LEMMY_UI_LEMMY_EXTERNAL_HOST and the hostname in the hjson file with our magical .onion address) and we seem to be done. The instance is online, posts work, federation works (if we configured it). Everything is beautiful.

Until a user decides to change their avatar.

The Phantom Avatar Bug (InvalidUrl)

The user uploads their photo, hits "Save", and the server rejects it. If you dig into the logs of the Lemmy container, you will find a nice 400 Bad Request error followed by a less-than-flattering InvalidUrl generated by the save_user_settings function.

What on earth is going on?

When saving an avatar, Lemmy doesn't receive the file itself, but rather the image URL generated by Pictrs. The Lemmy backend takes this URL (http://your-domain.onion/pictrs/image/...) and tries to connect to it to validate it.

But the Lemmy container has no native access to the Tor network to resolve .onion domains. If we were clever, we would have added a "network alias" in our docker-compose so that our .onion domain resolves internally to the IP of the Nginx container. Problem solved, right?

No. This is where Lemmy's anti-SSRF (Server-Side Request Forgery) protection kicks in. Lemmy uses an internal function called validate_link_ip(). This function takes the IP just resolved from the domain and checks if it's a private IP address (10.x.x.x, 192.168.x.x, 172.16.x.x). Since internal Docker networks typically use ranges like 172.28.0.x, Lemmy sees a private IP, raises its shields, and spits out InvalidUrl, refusing to process the image. Lemmy is literally protecting itself from itself.

There is no environment variable to disable this check. There is no whitelist. Either we change the Rust source code and recompile everything, or we are screwed.

The CGNAT Hack: Tricking the Sentinel

Cryptography and security teach us that when we can't break down the door, we need to understand how the lock thinks.

We dug into Lemmy's source code, specifically the v4_is_invalid() function used to block IPs. This function checks that the IP is not private (is_private()), loopback, multicast, etc.

But do you know what they forgot to include in the blacklist? The CGNAT (Carrier-Grade NAT) address space 100.64.0.0/10.

This address space is normally used by ISPs for NATing customers. For Lemmy's Rust function, a 100.64.x.x address is, for all intents and purposes, a normal, perfectly valid public IP address.

The solution is of absolute elegance: simply force Docker to use this address space for our stack's network.

In our docker-compose.yml, instead of letting Docker choose random subnets in the 172.x.x.x family, we define the default network at the bottom of the file like this:

networks:
  default:
    ipam:
      config:
        - subnet: 100.64.0.0/24

Restart the stack with docker compose down and docker compose up -d.

What happens now?

1. Nginx (our internal proxy with the .onion alias) will receive an IP like 100.64.0.2.
2. Lemmy Backend tries to save the avatar, resolving our .onion domain using Docker's DNS.
3. Docker's DNS replies: "100.64.0.2".
4. Lemmy's anti-SSRF sentinel checks the IP, doesn't find it among the private ones, and says: "Oh, this is a public IP! No SSRF danger. Proceed!"
5. Lemmy connects to Nginx, validates the image, and the avatar is magically saved.

We fooled the security system by exploiting one of its own implementation oversights, without touching a single line of Rust code and keeping the instance isolated and secure behind Tor.

Simple, clean, hacker-style.

[May the Talpone be with you]

Discussion & Support

Discussion and support channel. To discuss configurations or other topics, join us on fluxer.dreadful.work.