WireGuard#

As WireGuard is new in 3.x, this document will try to dive into some more detail regarding how it works.

Configuration#

WireGuard in eduVPN / Let’s Connect! has a lot less toggles than OpenVPN so should be easier to configure. For all configuration options, see:

• Portal Config
• Profile Config

Comparison with OpenVPN#

There are a number of differences between OpenVPN and WireGuard. Most of them are summed up in this blog post.

What to Use?#

It is possible to configure profiles to support both OpenVPN and WireGuard simultaneously. We recommend to use WireGuard whenever possible and disable OpenVPN.

There are currently three (known) reasons why OpenVPN is still required, all related to macOS/iOS:

1. The macOS/iOS application does not (yet) support WireGuard over TCP (#532)
2. DNS “search domains” do not work with the macOS/iOS application when using “split tunnel” VPN profiles (#524)
3. On some installations of macOS a WireGuard connection will not resume after suspend in some scenarios (#527)

For new deployments, since 2024-11-19, we deploy WireGuard by default, but have OpenVPN+TCP as a fallback option for macOS/iOS until WireGuard+TCP works there. If you are using “split tunnel” profiles with DNS search domains you MAY want to switch to OpenVPN as a default VPN protocol depending on the importance of properly supporting macOS/iOS.

The configuration template for new installations can be found here.

IP Management#

When using WireGuard, as opposed to using OpenVPN, the IP address the VPN client will use needs to be determined before starting the VPN “connection”. There is no in-protocol negotiation. WireGuard configuration files therefore contain the IP address that the client will use. This means that the IP address will be reserved for the duration of the session, by default 90 days. This is the case for both configuration file downloads through the portal and when using the API.

If there is enough IPv4 space available to assign to clients this is not a problem. If a /24 prefix is available for 10 clients, this will typically suffice. However, if you do not have so much IP space available, for example when using public IP addresses, additional measures need to be taken to make sure the IP space does not get quickly depleted.

A number of measures were taken to avoid that, and improved upon during the 3.x release:

1. Limit the number of per user manual VPN configuration file downloads through the portal (default = 3);
2. Limit the number of per user VPN connections when using the eduVPN/Let’s Connect! apps (default = 3);
3. Reclaim IP allocations by the eduVPN/Let’s Connect! apps after they are considered “gone” (default = after 72 hours).

The tooling for monitoring and optionally alerting based on server utilization have also improved to show the WireGuard IP allocations.

The defaults should suffice for most deployments, however you can tweak them if necessary, follow the links in the list above for more information about each of them.

IP Prefix Changes#

Changing the client IP prefix used for WireGuard clients is a bit more tricky than with OpenVPN. The main difference between OpenVPN and WireGuard is that with OpenVPN the IP address assigned to the client is decided on connect time, but with WireGuard it is decided ahead of time, i.e. when downloading the configuration through the portal, or when the VPN app calls the /connect API endpoint.

When (completely) changing the IP prefix for WireGuard, all current configurations will be deleted when applying the changes and clients will stop working. When extending the prefix, existing clients will keep working.

In the first scenario. Users of the VPN client applications will need to manually disconnect and then connect again to restart the VPN connection as the old one will be “dead”. Users that manually downloaded a VPN configuration file through the portal will need to download a new one, the old one will no longer work.

NOTE: when running vpn-maint-apply-changes in a scenario with separate controller and node(s), make sure you ALSO run the vpn-maint-apply-changes script on the controller in order to delete the assigned WireGuard IPs where they no longer match the new configuration.

MTU#

NOTE: since 2024-11-19 the MTU for new deployments of the VPN server is 1392, see the announcement.

NOTE: the analysis below might be incomplete, or incorrect at points, it is a very complex topic! Suggestions for improvements and corrections are very welcome!

We noticed some VPN client connection issues in the field when PPPoE and/or DS-Lite is used. This manifests itself as connections hanging (indefinitely) when trying to browse the web, start an SSH session, or try to open your IMAP mailbox. For once it is not DNS, but MTUs!

This problem appears most prevalent on Linux. It is unclear why Windows and macOS do not suffer, or suffer less, from connection hangs. We suspect it is because they implemented some mitigation for Path MTU Discovery (PMTUD) issues, e.g. they may have implemented RFC 4821.

There is an easy mitigation you can apply on a Linux VPN client as well that solves the most immediate problem. The rest of this section will explain how it can be fixed in a, what we hope, more sustainable way.

When?#

We observed that connection hanging occurs reliably when the following holds:

1. The network connection of the VPN client has MTU of 1500;
2. Somewhere on the path, to the VPN server, the MTU is reduced;
3. The MTU becomes low enough to not fit a WireGuard packet anymore.

The following table shows when there will be issues expected with the default WireGuard MTU (which will be 1420 when the network connection has an MTU of 1500):

• The “PMTU” column indicates the maximum MTU on the path between VPN client and server;
• The “Connection” column indicates whether the VPN connection was established over IPv4 or IPv6;
• The “Works?” column indicates whether the default WireGuard MTU of 1420 works with this type of connection;
• The “Max WireGuard MTU” column is the highest WireGuard MTU setting that still works without expecting MTU issues.

Determine PMTU#

If you have access to a VPN client that shows issues when connecting to WireGuard you can test the Path MTU (PMTU) using tracepath. For example the PMTU without VPN connection could be like this:

$ tracepath -4 -n dns.quad9.net
 1?: [LOCALHOST]                      pmtu 1500
 1:  192.168.178.1                                         1.272ms 
 1:  192.168.178.1                                         2.525ms 
 2:  192.0.0.2                                             2.580ms pmtu 1460

...

 7:  9.9.9.9                                              26.348ms !H
     Resume: pmtu 1460 

Note, that for IPv6 the PMTU does not need to be the same:

$ tracepath -6 -n dns.quad9.net
 1?: [LOCALHOST]                        0.010ms pmtu 1500

...

 8:  2620:fe::9                                           12.946ms !A
     Resume: pmtu 1500 

Here we see the PMTU for IPv4 is 1460, and for IPv6 1500. If the VPN client would always connect over IPv6, there would not be a problem, but unfortunately that can’t always be guaranteed. This connection uses DS-Lite to wrap IPv4 in IPv6 packets. This has a 40 byte overhead, and thus reduces the effective MTU to 1460.

In the table above we see that WireGuard’s MTU can be 1400 at most in the scenario where the VPN connection is established over IPv4, which is not enough to fit WireGuard’s default MTU of 1420.

Setting the MTU#

NOTE: setting the MTU only works in vpn-user-portal >= 3.4.0.

NOTE: this is EXPERIMENTAL, we MAY decide to make this the default at some point and remove (or rename) the configuration toggle! Please provide feedback if you have ideas about this.

Based on the table above, we can set the MTU to the maximum value that still is expected to work on the networks used by the VPN clients. If you don’t know what to choose, take 1392.

You can set the option setMtu like this in /etc/vpn-user-portal/config.php:

'WireGuard' => [

    // ... other WireGuard options

    'setMtu' => 1392,
],

The MTU configuration flag will be used by both on the server and the client.

If the client is still using a configuration file with MTU configuration, the firewall’s “TCP MSS Clamping” will take care of making that client work.

Once you set the setMtu option, the changes need to be applied:

$ sudo vpn-maint-apply-changes

By default the firewall that is installed when you deployed your VPN server is already configured to enable TCP MSS Clamping, so nothing needs to be done here.

Mitigation on Linux Client#

If a server solution is currently not possible, one can also create a file on the VPN client, e.g. /etc/sysctl.d/70-vpn-mtu.conf, should work on at least Debian / Ubuntu and Fedora / EL:

net.ipv4.tcp_mtu_probing = 1
net.ipv4.tcp_base_mss = 1024

Then run the below command and reconnect to the (WireGuard) VPN:

$ sudo sysctl --system

References#

We found the following resources very useful for understanding MTU, TCP MSS Clamping and PMTUD.

• Header / MTU sizes for Wireguard
• What is MTU?
• Path MTU discovery in practice
• What is MSS (maximum segment size)?

WireGuard over TCP#

NOTE: ONLY available in vpn-user-portal >= 3.6.0, and not yet with all eduVPN / Let’s Connect! applications.

In order to work around networks that break UDP, or where lowering the MTU is not sufficient, we also implemented WireGuard over TCP.

The higest success rate will be obtained by tunneling the traffic over tcp/443 which is used by HTTPS.

Enabling WireGuard over TCP/HTTPS requires a couple of steps:

1. Install and configure ProxyGuard
2. Change the Portal configuration
3. Change the Web Server Configuration

The following eduVPN / Let’s Connect! applications support it:

ProxyGuard#

Install and enable proxyguard-server:

Fedora / EL#

$ sudo dnf -y install proxyguard-server
$ sudo systemctl enable --now proxyguard-server

You can modify the configuration in /etc/sysconfig/proxyguard-server if needed, for example in case you modified the default WireGuard port. Do not forget to restart proxyguard-server after modifying this file.

As an example: if you changed the WireGuard UDP port to 443, you MUST set TO=127.0.0.1:443 in the /etc/sysconfig/proxyguard-server file.

Debian / Ubuntu#

$ sudo apt -y install proxyguard-server

On Debian/Ubuntu the service is automatically started and enabled.

You can modify the configuration in /etc/default/proxyguard-server if needed, for example in case you modified the default WireGuard port. Do not forget to restart proxyguard-server after modifying this file.

As an example: if you changed the WireGuard UDP port to 443, you MUST set TO=127.0.0.1:443 in the /etc/default/proxyguard-server file.

Portal Configuration#

Set the enableProxy configuration field to true in the WireGuard section in /etc/vpn-user-portal/config.php, look here for all configuration options pertaining to WireGuard (over TCP).

Web Server Configuration#

In order to update the web server configuration, you modify /etc/httpd/conf.d/vpn.example.org.conf (on Fedora/EL), or /etc/apache2/sites-available/vpn.example.org.conf (on Debian/Ubuntu). Replace vpn.example.org in the file name above with your own server’s name.

Add the following line inside the <VirtualHost *:443> section:

ProxyPassMatch "^/proxyguard/vpn.example.org$" "http://127.0.0.1:51820/" upgrade=UoTLV/1

We came up with the “protocol” UoTLV/1 which stands for “UDP over TCP Length Value, Version 1”. “Length Value” refers to the intitial (two) bytes in the protocol that indicates the length.

On Debian/Ubuntu you need to enable the proxy_http Apache module:

$ sudo a2enmod proxy_http

Don’t forget to restart the HTTP server: systemctl restart httpd on Fedora/EL, and systemctl restart apache2 on Debian/Ubuntu.

NOTE the full proxy URL will be extended to include the host name of the VPN node, i.e. the value of hostName, this is important when running your VPN service on multiple nodes. In the example above the full proxy URL will be https://vpn.example.org/proxyguard/vpn.example.org.

NOTE: the node host name in the URL will allow you to redirect traffic to the appropriate node in multi node setups, e.g.:

ProxyPassMatch "^/proxyguard/n1.vpn.example.org$" "http://10.0.0.5:51820/" upgrade=UoTLV/1
ProxyPassMatch "^/proxyguard/n2.vpn.example.org$" "http://10.0.0.6:51820/" upgrade=UoTLV/1

NOTE: you can also set proxyUrl to point to a dedicated proxy. If proxyUrl is for example https://proxy.vpn.example.org the full proxy URL would become https://proxy.vpn.example.org/vpn.example.org.

Testing#

In order to test whether Apache and ProxyGuard are configured correctly, you can quickly check to see if you get the expected output:

NOTE: this is NOT an error, but expected!

$ curl https://vpn.example.org/vpn.example.org
the 'Connection' header is not 'Upgrade', got: 'Keep-Alive'

Changing WireGuard Port#

This section combines the information found in various other sections regarding updating the WireGuard port. By default WireGuard works over udp/51820. You MAY want to change that to another port, for example udp/443, which is what we’ll use below as an example. Feel free to choose any port you want.

We need to change the port in a number of places:

1. Portal Configuration (/etc/vpn-user-portal/config.php);
2. Firewall (/etc/nftables.conf on Debian/Ubuntu, or /etc/sysconfig/nftables.conf on Fedora/EL);
3. ProxyGuard (/etc/default/proxyguard-server on Debian/Ubuntu, or /etc/sysconfig/proxyguard-server on Fedora/EL).

To change the port in the portal configuration, look here for more details, but in essence: update, or set listenPort inside the WireGuard section to 443.

To change the firewall, modify the firewall as instructed here, but to change the WireGuard port, you need to modify the udp dport { 1194, 51820 } accept line inside the “input chain”. Make sure you keep existing ports if there are any, but replace 51820 by 443, e.g.: udp dport { 443, 1194 } accept. Do NOT forget to restart the firewall.

To update the ProxyGuard configuration, it is important that you set/update the TO field in the configuration file, e.g. add the line TO=127.0.0.1:443 to the configuration file and restart ProxyGuard with sudo systemctl restart proxyguard-server.