Multi Node Deployments#
Setting up multiple VPN nodes is one part of making the VPN service “High Available”. The other is setting up a redundant portal. A complete overview of the options can be found here.
Setting up multiple nodes allows the “load” of the VPN service to be distributed over multiple (virtual) servers and avoid (extensive) downtime when one of the nodes goes down.
In this document we’ll show how to deploy a VPN service that has one controller and three nodes. One node is located in Amsterdam, two in Frankfurt. This will demonstrate all possible configuration scenarios.
Here, the VPN clients choosing “Amsterdam” will always connect to the
same node (ams1.vpn.example.org
), clients choosing “Frankfurt” will end up on
one of the two nodes, either fra1.vpn.example.org
or fra2.vpn.example.org
,
randomly selected when connecting to the VPN.
Of course your setup can also start with one controller and one node!
NOTE: the algorithm deciding which node the VPN client will connect to is currently very simple. It randomly picks one from the list of VPN nodes that is up. In the future we envision also considering the “load” of the node before deciding.
NOTE: currently, but we hope to fix this, if a node goes down, the user will have to manually disconnect/connect in order for a new node to be picked for the client to connect to.
Requirements#
In this scenario we require 4 machines (or VMs) running Debian >= 12. Ideally, nodes all have the same specifications.
All four need to be set up with static IP configurations and working DNS. Make sure all works properly before starting the setup. For example, our test deployment uses:
Role | DNS Host | IPv4 | IPv6 |
---|---|---|---|
Controller | vpn.example.org |
192.0.2.10 |
2001:db8::10 |
Node 0 (ams1) | ams1.vpn.example.org |
192.0.2.20 |
2001:db8::20 |
Node 1 (fra1) | fra1.vpn.example.org |
192.0.2.30 |
2001:db8::30 |
Node 2 (fra2) | fra2.vpn.example.org |
192.0.2.40 |
2001:db8::40 |
We will use NAT for IPv4 and IPv6 client traffic.
Perform these steps on the hosts:
$ curl -L -O https://codeberg.org/eduVPN/deploy/archive/v3.tar.gz
$ tar -xzf v3.tar.gz
$ cd deploy
Controller#
On the controller host:
$ sudo -s
# ./deploy_debian_controller.sh
Make note of the user credentials that are printed at the end, you can use them to test your server!
After the controller is installed, make sure you’ll get a valid TLS
certificate, for example using the included lets_encrypt_debian.sh
script:
$ sudo -s
# ./lets_encrypt_debian.sh
Now visit your site at https://vpn.example.org/. Make sure there is no TLS error and you can login with the credentials you noted before.
Configuration#
We’ll modify /etc/vpn-user-portal/config.php
and remove the existing
“default” profile and replace it with our ams
and fra
profiles:
'ProfileList' => [
// Our Node in Amsterdam
[
'profileId' => 'ams',
'displayName' => 'Amsterdam',
'hostName' => 'ams1.vpn.example.org',
'wRangeFour' => '172.23.114.0/24',
'wRangeSix' => 'fd36:2246:1d09:3014::/64',
'defaultGateway' => true,
'dnsServerList' => ['9.9.9.9', '2620:fe::9'],
'nodeUrl' => 'http://ams1.vpn.example.org:41194',
'onNode' => 0,
],
// Our Nodes in Frankfurt
[
'profileId' => 'fra',
'displayName' => 'Frankfurt',
'hostName' => ['fra1.vpn.example.org', 'fra2.vpn.example.org'],
'wRangeFour' => ['10.61.60.0/24', '10.7.192.0/24'],
'wRangeSix' => ['fd85:f1d9:20b7:b74c::/64', 'fd89:79cb:b63c:717e::/64'],
'defaultGateway' => true,
'dnsServerList' => ['9.9.9.9', '2620:fe::9'],
'nodeUrl' => ['http://fra1.vpn.example.org:41194', 'http://fra2.vpn.example.org:41194'],
'onNode' => [1, 2],
],
],
If you want to generate your own random IPv4 and IPv6 prefixes to avoid
“collisions” for use in the wRangeFour
and wRangeSix
configuration options,
you can use /usr/libexec/vpn-user-portal/generate-prefix
:
Address Family | Command |
---|---|
IPv4 | /usr/libexec/vpn-user-portal/generate-prefix -4 |
IPv6 | /usr/libexec/vpn-user-portal/generate-prefix -6 |
See Profile Config for an explanation of what all the configuration options mean exactly.
Next, modify the <Files node-api.php>
section in
/etc/apache2/conf-available/vpn-user-portal.conf
by adding the IP addresses
of the nodes, make sure Require ip
lists all the IP addresses, if you want
to allow only 1 address use the /32
prefix (IPv4) or /128
(IPv6):
<Files node-api.php>
<RequireAny>
Require local
# When using separate VPN node(s) running (vpn-server-node),
# add the IP address(es) of the node(s) here
Require ip 192.0.2.0/24
Require ip 2001:db::/32
</RequireAny>
</Files>
NOTE: if you modify vpn-user-portal.conf
file, on Debian/Ubuntu, you MAY
be notified during upgrades of vpn-user-portal
about a changed configuration
file. In that case choose to KEEP your current configuration file, or manually
merge your changes!
Restart Apache:
$ sudo systemctl restart apache2
For each node you want to add you need to generate a new “Node Key”. By default
we have one for node 0 (ams1.vpn.example.org
), but we need keys for node 1
and 2 (the two nodes in Frankfurt):
$ sudo /usr/libexec/vpn-user-portal/generate-secrets --node 1
$ sudo /usr/libexec/vpn-user-portal/generate-secrets --node 2
Copy the generated node.0.key
, node.1.key
and node.2.key
to the
respective nodes. We’ll put them in the correct place in the next section.
Nodes#
The instructions below will be only shown for Node 0, but they are identical for Node 1 and 2… You have to perform these instructions three times.
Make sure you can reach the API endpoint from the node(s):
$ curl https://vpn.example.org/vpn-user-portal/node-api.php
{"error":"authentication required"}
This error is expected as no secret was provided. This just makes sure the controller is configured correctly and allows requests from the nodes.
Next, it is time to install the software. You should have downloaded and unpacked the archive already, see Requirements.
$ cd documentation-3
$ sudo -s
# ./deploy_debian_node.sh
On your node, modify /etc/vpn-server-node/config.php
, set apiUrl
,
nodeNumber
and profileIdList
:
Node 0#
<?php
return [
'apiUrl' => 'https://vpn.example.org/vpn-user-portal/node-api.php',
'nodeNumber' => 0,
'profileIdList' => ['ams'],
];
Node 1#
<?php
return [
'apiUrl' => 'https://vpn.example.org/vpn-user-portal/node-api.php',
'nodeNumber' => 1,
'profileIdList' => ['fra'],
];
Node 2#
<?php
return [
'apiUrl' => 'https://vpn.example.org/vpn-user-portal/node-api.php',
'nodeNumber' => 2,
'profileIdList' => ['fra'],
];
Next, modify /etc/default/vpn-daemon
:
LISTEN=:41194
Restart the daemon:
$ sudo systemctl restart vpn-daemon
Copy the node.0.key
, node.1.key
and node.2.key
on their respective nodes
to /etc/vpn-server-node/keys/node.key
.
NOTE: make sure it only contains the secret and not a trailing return. If you are using copy/paste using this:
$ echo -n 'SECRET' | sudo tee /etc/vpn-server-node/keys/node.key
NOTE: the sticky bit on the folder /etc/vpn-server-node
makes sure that
the permissions are correct by default. Make sure that the file can be read by
the group, nogroup
on Debian / Ubuntu.
The firewall also requires tweaking, open it to allow traffic from the controller to the node’s VPN daemon:
In /etc/nftables.conf
, remove the comments in front of these lines and update
the IP addresses:
ip saddr 192.0.2.10/32 tcp dport 41194 accept
ip6 saddr 2001:db8::10/128 tcp dport 41194 accept
Restart the firewall:
$ sudo systemctl restart nftables
Now you are ready to apply the changes and this should work without error on all your nodes:
$ sudo vpn-maint-apply-changes
Make sure you repeat these steps on Node 1 and 2 as well!
NOTE: if you also run the HA Portal, you MUST synchronize
the /var/lib/vpn-user-portal
folder again between the n portals!
Now is the time to test everything. Go to the portal at
https://vpn.example.org/
, download a configuration and test it with your
VPN client. Login with an account that has “Admin” privileges, and make sure
you see your client(s) under “Connections” in the portal when connected.
Maintenance Mode#
NOTE: available with vpn-user-portal >= 3.8.0 and vpn-daemon >= 3.1.0
To reduce the impact to client VPN connections when maintenance is scheduled for a node, it can be (temporary) put into “maintenance mode”. This prevents the portal/controller from considering that node for new VPN connections. Existing connections to that node are not disturbed. Once a sufficient number of connections are moved off the node, the node can be updated and/or rebooted.
The portal/controller considers a node in “maintenance mode” when the file
/run/vpn-daemon/maintenance-mode
is present on the particular node. Removing
this file, or rebooting the system, will take the node out of
“maintenance mode” and will be considered again for (new) client connections.
Installing Updates#
The scenario for installing updates in multi node deployments is a bit different from single server installations. That is, if you want to do it “nicely”. The steps involve:
- Stop the node(s)
- Update the controller (+ maybe reboot)
- Update the node(s) (+ maybe reboot)
- Start the node(s)
We have some scripts that make this easy for you as part of the vpn-maint-remote project.
NOTE: make sure you can SSH to the controller and all node(s) and run
commands using sudo
on the machines without requiring a password.
If you install the script(s) on the system that you use to manage your VPN servers, you can run through the above steps with ease. This works on Linux and macOS.
$ git clone https://git.sr.ht/~fkooman/vpn-maint-remote
$ cd vpn-maint-remote
Create a file server.list
with the following content:
CONTROLLERS="
vpn.example.org
"
NODES="
ams1.vpn.example.org
fra1.vpn.example.org
fra2.vpn.example.org
"
Now you can run the script the update your controller and node(s),
there’s no need for root
permissions:
$ bin/vpn-maint-update-system-multi
If you also want to reboot your systems, e.g. in case of kernel or system library updates:
$ bin/vpn-maint-update-system-multi --reboot
TLS#
NOTE: these instructions are for Debian, not (yet) for Fedora or Enterprise Linux!
When everything works properly using HTTP, you SHOULD switch to HTTPS for communication between controller and node(s). Without TLS there is no encryption and no authentication. Enabling TLS will fix this.
We’ll create a tiny CA and issue server certificates for the node(s) and a client certificate for the controller. We use vpn-ca which is already installed on your controller, but you can also download and install it on your own system.
The CA, the server and client certificates will all be valid for 10 years from the moment of generation. Otherwise the defaults are 5 years for the CA and 1 year for both the client and server certificate, which is not great for the communication channel between portal and node(s).
$ vpn-ca -init-ca -not-after $(date -d "+10 years" +%FT%T%:z) -name "Management CA" -domain-constraint .vpn.example.org
$ vpn-ca -server -not-after CA -name node-a.vpn.example.org
$ vpn-ca -server -not-after CA -name node-b.vpn.example.org
...
$ vpn-ca -client -not-after CA -name vpn-daemon-client
Copy ca.crt
, node-X.vpn.example.org.crt
and node-X.vpn.example.org.key
to
the respective node(s). Prepare the directory to store the files:
$ sudo mkdir -p /etc/ssl/vpn-daemon/private
$ sudo mkdir -p /etc/systemd/system/vpn-daemon.service.d
Store the certificates/keys in the following locations (note the
private
folder for the key):
File | Location |
---|---|
ca.crt |
/etc/ssl/vpn-daemon/ca.crt |
node-X.vpn.example.org.crt |
/etc/ssl/vpn-daemon/server.crt |
node-X.vpn.example.org.key |
/etc/ssl/vpn-daemon/private/server.key |
Now, enable System and Service Credentials
by writing the following content to
/etc/systemd/system/vpn-daemon.service.d/credentials.conf
:
[Service]
LoadCredential=ca.crt:/etc/ssl/vpn-daemon/ca.crt
LoadCredential=server.crt:/etc/ssl/vpn-daemon/server.crt
LoadCredential=server.key:/etc/ssl/vpn-daemon/private/server.key
Make sure to reload the systemd
daemon and restart vpn-daemon
:
$ sudo systemctl daemon-reload
$ sudo systemctl restart vpn-daemon
Repeat this on all your nodes.
On your controller(s), create the directory for storing the files:
$ sudo mkdir -p /etc/vpn-user-portal/keys/vpn-daemon
Copy the ca.crt
, vpn-daemon-client.crt
and vpn-daemon-client.key
to
/etc/vpn-user-portal/keys/vpn-daemon
and modify the nodeUrl
option(s)
in the profile configuration in /etc/vpn-user-portal/config.php
to use
https://
instead of http://
.
Viewing the portal “Info” page should show your node(s) as green and have the lock icon visible. Now you are all good!
If there are any problems, review the vpn-daemon
log on your node(s):
$ sudo journalctl -t vpn-daemon
If your daemon is running properly, you can try curl
from your controller(s)
to verify the TLS connection can be established:
$ curl \
--cacert /etc/vpn-user-portal/keys/vpn-daemon/ca.crt \
--cert /etc/vpn-user-portal/keys/vpn-daemon/vpn-daemon-client.crt \
--key /etc/vpn-user-portal/keys/vpn-daemon/vpn-daemon-client.key \
https://node-a.vpn.example.org:41194/i/node