Cloudflare Mesh on VyOS

Published on • 7 min read •

Introduction

Cloudflare has been increasing the capabilities of their Zero Trust product over the years, initially network connectivity was formed via local reverse proxies (cloudflared) that would tunnel back to the Cloudflare infrastructure, eventually this gained the ability to route traffic to a cloudflared instance however it was more of a proxy rather than routing.

This proxy nature has made it difficult to facilitate bidirectional routing, such as something behind cloudflared initiating an IP connection with a WARP user or another cloudflared instance, additionally it was impossible to track the origin of the traffic as everything would have the IP address of the cloudflared instance.

MagicWAN was the enterprise offering to solve this issue using IPSEC or GRE tunnels, but this was not a cost effective option if you just need a basic VPN solution. Eventually Cloudflare enabled using the WARP client as a bidirectional routing solution, this would require running a Linux device and routing traffic to the device.

That was until this week when Cloudflare dropped their official Cloudflare Mesh Container image, primarily aimed to be used in a Docker/Kubernetes environment, but could we use it on VyOS to add a full VPN into Cloudflare?

Configuring VyOS

Taking the Cloudflare suggested Kubernetes configuration, it was surprisingly easy to translate to the native container support offered by VyOS.

By using the container we can natively integrate Cloudflare into VyOS without building a custom image, this will hopefully enable seamless upgrades between VyOS versions.

set container name cf-mesh allow-host-networks
set container name cf-mesh capability 'net-admin'
set container name cf-mesh capability 'net-raw'
set container name cf-mesh device tun destination '/dev/net/tun'
set container name cf-mesh device tun source '/dev/net/tun'
set container name cf-mesh disable
set container name cf-mesh environment MESH_NODE_TOKEN value '<REDACTED>'
set container name cf-mesh image 'cloudflare/mesh'

The sysctls were not configured as they conflict with the allow-host-networks, with the later providing the required by running as the host essentially.

Did it work? How does it look?

Committing the changes, we check show interfaces however there is no new interface… but this is happening outside of an expected VyOS configuration, lets check with the underlying Linux side… ip link and ip addr surprisingly it was that simple to make it work!

vyos@ROUTER:~$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode 
...
12: CloudflareWARP: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1300 qdisc mq state UNKNOWN mode DEFAULT group default qlen 500
    link/none
vyos@ROUTER:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
...
12: CloudflareWARP: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1300 qdisc mq state UNKNOWN group default qlen 500
    link/none
    inet 100.96.0.1/32 scope global CloudflareWARP
       valid_lft forever preferred_lft forever
    inet6 2606:4700:cf1:2000::1/128 scope global deprecated
       valid_lft forever preferred_lft 0sec
    inet6 2606:4700:cf1:1000::2/128 scope global
       valid_lft forever preferred_lft forever

A quick look at the logs however reveals some new firewall rule hits, it appears the Cloudflare Mesh client is having issues performing its self DNS checks, to quickly fix this we will add a firewall rule to permit all input from the lo loopback interface (eg. traffic the router itself generated)

Aug 09 09:57:33 kernel: [ipv4-INP-filter-default-D]IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.2.3 LEN=73 TOS=0x00 PREC=0x00 TTL=64 ID=3559 DF PROTO=UDP SPT=57586 DPT=53 LEN=53

Is it usable in VyOS?

We know it setup the CloudflareWARP interface, but is this usable by VyOS? Lets add some static routes to route the expected 100.96.0.0/12 back

On Cloudflare Mesh side we will also add a route for our internal LAN range to go through this mesh device.

vyos@ROUTER# set protocols static route 100.96.0.0/12 interface CloudflareWARP
[edit]
vyos@ROUTER# commit
[edit]
vyos@ROUTER# run show ip route
Codes: K - kernel route, C - connected, L - local, S - static,
       R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric, t - Table-Direct,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

IPv4 unicast VRF default:
S>* 0.0.0.0/0 [210/0] via 123.123.123.123, eth3, weight 1, 20:44:04
...
S>* 100.96.0.0/12 [1/0] is directly connected, CloudflareWARP, weight 1, 00:00:05
L * 100.96.0.1/32 is directly connected, CloudflareWARP, weight 1, 00:00:05
C>* 100.96.0.1/32 is directly connected, CloudflareWARP, weight 1, 00:00:05

Behind the scenes I also updated the various firewall rules which referenced the interface by name as well.

Doing a ping from a remote workstation running a WARP client, we can successfully access resources from LAN behind VyOS and the source IP is being retained through this flow!

Where to now?

Whilst this is technically possible it may not be production ready and more testing is needed.

Due to the lack of VyOS integration it appears during a reboot the VyOS systems detect the missing or invalid interface and anything referencing CloudflareWARP is removed (eg. firewall / static routing) this is ultimately a show stopper for almost any deployment, but possibly could be worked around by utilizing IP addresses instead of interfaces.

Ideally VyOS could also add support to handle “externally created” network interfaces, this could either automatic or as simple as set interfaces external <name> to raise the interface to the VyOS systems.

For my own interests I will continue to explore ways to refine this implementation.

network   vyos   cloudflare   mesh