Published on Dec 30, 2022
Using “soft-unicast”, Cloudflare manages the egress traffic of its servers. As part of soft-unicast, multiple servers share a single IPv4 address for their egress traffic, while the response packets are redirected to the appropriate physical server. Through this scalable and cost-effective solution, Cloudflare can provide a wide range of products that require tagged egress IP addresses.
Cloudflare splits an egress IP address across servers based on port ranges. For each egress IP address, each server owns a small portion of the available source ports. Cloudflare is able to share one IP address among 31 servers with a port slice of 2048 ports. The system implemented methods to reuse the egress ports efficiently in order to prevent the possibility of running out of ports. In order to ensure that return packets are routed to the correct machine, Cloudflare has customized Unimog, its L4 XDP-based load balancer, to be aware of this technique.
A router with Source-NAT can traditionally share an IP address between multiple servers. The number of egress IPs Cloudflare requires prevents it from using stateful firewalls and NATs at the router level. Additionally, Cloudflare has chosen to avoid installing a distributed NAT for undisclosed reasons.
It is usually only possible to route subnets on the public Internet with a granularity of /24 or 256 IP addresses. This would result in a waste of IP space for Cloudflare. In order to improve the utilization of Cloudflare’s IP space, it has deployed the egress IP addresses as anycast addresses, which it typically uses for ingress traffic. Cloudflare customized Unimog to forward packets over its backbone network to the appropriate data center.
By using this design, an IP address can be used to identify a data center, while an IP address and port range can be used to identify a specific computer. As a result, it behaves almost as a unicast.
In the past, Cloudflare only used anycast for its ingress traffic. It was able to take care of the “last mile” route over its backbone network by customising its L4 load balancer.
