04 July 2022 / Last updated: 30 Jun 2023

Sunsetting vpn.balena-cloud.com and Introducing Cloudlink

November 1st 2023 update

Server shutting down in phases

As mentioned below, the certificate for old VPN will be December 10th.
We announced a shutdown on November 10th, but we are changing plans to give some extra time to last minute users that are still migrating.
We will perform a one week shutdown from November 2nd to 8th. This will help identify what devices are losing VPN and need upgrade.
After November 8th, the following milestone is December 10th. That is a certificate expiration and balena will shut down the servers definitely.

June 30th 2023 update

The summary of the situation is as follows:

Devices running resinOS/balenaOS version < 2.14.0 need an OS update.
Devices running v2.14.0 < balenaOS < v2.61.2 need a reboot or an OS update.
Devices running ESR releases v2019.10 <= balenaOS < v2021.01.0 need a reboot.

NOTE: This is valid for Active and Inactive devices

Service availability

On August 1st 2023 we will perform a temporary manual shutdown of the services to allow user for testing and impact assesment. This will last for a week, from August 1, 2023 to August 7, 2023.
On November 10th 2023 we will start shuting down servers. Service discontinuation is to be expected indefinitely.
On December 10th 2023 the certificate expires. At this point, access to vpn.balena-cloud.com will be closed forever and will be out of our hands.

We have introduced Cloudlink (cloudlink.balena-cloud.com) as a replacement for balena VPN (vpn.balena-cloud.com) on May 07, 2022. This is because balena VPN will no longer be usable on December 10, 2023 due to its CA Certificate expiring and additionally because we have found that there may be confusion with the name and its function.

Sunsetting vpn.balena-cloud.com and introducing Cloudlink

Read on to learn more.

How does this affect fleets and devices?

This shouldn’t change anything on the device, just the connection instance that the device is connected to. This means that OpenVPN is still used as an underlying technology to Cloudlink and components such as open-balena-vpn will still be around. Additionally, the majority of the users should not have experienced any problems with the deployment as devices have a mechanism to automatically transfer to the new connection (os-config).

However, as of our current investigation, there could be 3 causes that a device may have problems connecting to Cloudlink – (1) Devices with old balenaOS versions (<= 2.14.0) may not have os-config, (2) Devices with balenaOS versions > 2.14.0 and < 2.61.2 (ESR versions 2019.10 to 2021.01.0) will have a mechanism to update but will need a power cycle or reboot to do so and (3) your firewall may be blocking cloudlink.balena-cloud.com and your devices can’t connect to this endpoint, resulting in a Heartbeat only state. If your devices are in this situation, please follow the guidelines below in detail.

Introducing Cloudlink

On May 07, 2022, we introduced Cloudlink as a replacement to the balena VPN connection. This is the connection that provides features like SSH access, the public device URL, and the web terminal in balenaCloud. Although balena VPN (vpn.balena-cloud.com) is still up and running, we will be sunsetting this in the near future and will be using Cloudlink (cloudlink.balena-cloud.com) from then on. This is because of 2 main reasons:

1. Expiring CA Certificate

balena VPN(vpn.balena-cloud.com) will become unusable on December 10, 2023 because that will be when the CA Certificate will expire. The CA Certificate is required for any of our connection instances to validate the devices in our platform. When this expires, any device that is still connecting to that subdomain on the date of expiry will effectively be unable to use features like SSH access, Public device URLs and the web terminal in balenaCloud.

2. The primary purpose of Cloudlink (prev. balena VPN)

The primary purpose of this connection isn't to act as a VPN, but as a mechanism for connecting to the cloud. We still use OpenVPN as an underlying technology for Cloudlink to provide features like SSH access, Device Public URLs, and the web terminal. Components and services such as the open-balena-vpn or the openvpn.service are still present and unchanged on the device. Nevertheless, we have faced instances in the past where there may be confusion about expected functions that a VPN usually has because of the name "balena VPN" and domain vpn.balena-cloud.com. Hence, we have abstracted it from the name of the technology and we have chosen a name that reflects its main function. This also allows us the flexibility to change the underlying stack to provide us the same or even allow us more features in the future.

The change over from vpn.balena-cloud.com to cloudlink.balena-cloud.com isn't triggered by a host update but rather from a tool that runs on the device every 24hrs – os-config.

Checking if your device is on Cloudlink

If you want to check if your devices are on Cloudlink, you have a few options depending on your preference.

1. Checking openvpn logs

Take a look at the addresses of cloudlink.balena-cloud.com using nslookup and check it against the connected address that your device’s openVPN service is on.

root@<uuid>:~# nslookup cloudlink.balena-cloud.com
Server:    127.0.0.2
Address 1: 127.0.0.2 a5fa2e0

Name:      cloudlink.balena-cloud.com
Address 1: 34.226.166.12 ec2-34-226-166-12.compute-1.amazonaws.com
Address 2: 52.7.228.224 ec2-52-7-228-224.compute-1.amazonaws.com
Address 3: 3.225.166.106 ec2-3-225-166-106.compute-1.amazonaws.com
Address 4: 2600:1f18:6600:7f01:dc24:54f2:d95f:abc0
Address 5: 2600:1f18:6600:7f02:bda3:7af0:e21:425b
Address 6: 2600:1f18:6600:7f00:b18f:40:ea2:57ca

root@<uuid>:~# journalctl --unit openvpn | grep Peer
Jun 23 14:39:02 a5fa2e0 openvpn[1833]: Thu Jun 23 14:39:02 2022 [vpn.balena-cloud.com] Peer Connection Initiated with [AF_INET]3.225.166.106:443

2. Check `lsof` output

Get the process ID of the openvpn process and then check the lsof output of the process to see the IP address of the instance openvpn is connected to.

root@85b8907:~# ps -ef | grep openvpn
openvpn  1118637       1  0 17:22 ?        00:00:01 /usr/sbin/openvpn --writepid /run/openvpn/openvpn.pid --cd /etc/openvpn/ --config /etc/openvpn/openvpn.conf --

root@85b8907:~# lsof -ai -p 1118637
COMMAND     PID    USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
openvpn 1118637 openvpn    3u  IPv4 6489640      0t0  TCP 85b8907:53288->ec2-3-225-166-106.compute-1.amazonaws.com:https (ESTABLISHED)

3. Query the API config endpoint

Query our API config endpoint to get the configuration details and then check it against the openvpn files on the device (/etc/openvpn/openvpn.conf and /etc/openvpn/ca.crt).

curl --silent -X GET "https://api.balena-cloud.com/os/v1/config" -H "Content-Type: application/json" | jq .

Who could be affected and what should be done?

The majority of our existing users shouldn't have experienced the change directly as we have mechanisms (os-config) that allow the smooth transition between connection instances. If your fleet has not experienced any issue with connectivity for a long period of time since May 07, 2022, this means that the mechanism has worked correctly and your devices are now connected to Cloudlink.

Old resinOS/balenaOS versions

Some devices may still be using the old instance because of old resinOS/balenaOS versions. This can be due to a number of factors but the most obvious one is that the device is on the resinOS/balenaOS version <= 2.14.0. These versions won't have os-config which unfortunately means with a balenaOS version this old (or older), the update to the new connection portal (cloudlink) won't work out of the box. Your devices should not have experienced any downtime related to this because we are keeping our vpn.balena-cloud.com subdomain alive until the certificate expires December 10th, 2023. To prevent unusable connectivity in the future, your best recourse is to update your balenaOS version on the devices that are still with balenaOS version <= 2.14.0.

v2.14.0 < balenaOS < v2.61.2 (ESR versions prior to 2021.01.0)

We have found that on these OS versions, the device does not update the VPN configuration (through os-config) once each single day through a systemd timer, but only on device reboot. The relevant meta-balena log which speaks about introducing the daily timer is found here: https://github.com/balena-os/meta-balena/blob/master/CHANGELOG.md#v2612. That means that devices that run on v2.14.0 < balenaOS < v2.61.2 (ESR versions prior to 2021.01.0) will need a reboot or power cycle to update to the new connection instance (Cloudlink). If you notice that your device may be on the old instances and are above the version 2.61.0 (ESR versions prior to 2021.01.0), we will be in contact with you to investigate and help with the situation.

Experiencing “Heartbeat only” status on device

Additionally, if your devices are experiencing "Heartbeat only" statuses for extended periods of time, you should check if the local network is blocking or using a static access list (whitelist) that doesn't have cloudlink.balena-cloud.com. As suggested in our network requirements, a wildcard that includes all subdomains (*.balena-cloud.com) would solve it if this is the case you are experiencing.

Doublecheck network access lists

There may be vendors or local networks with problems with a wildcarded access list, so we would like to reiterate that balenaCloud is a multi-tenant SaaS built on top of other multi-tenant SaaS products, like AWS and DockerHub to name a few. Due to the nature of these types of multi-tenant cloud environments, and because we regularly add new features which require new endpoints to the platform, we are not able to provide an unchanging FQDN list for access listing purposes.

It would be possible today to perform a network capture of regular traffic in your device's networks, which would produce a list of the endpoints that your devices are using today, but we do not guarantee that this will remain static as your device's consumption of features and the platform itself shift and change over time. For this reason, we provide a list of a few specific domains, and the wildcard *.balena-cloud.com. Some customers solve these problems by placing balena devices in a DMZ, exempting their DHCP range from certain types of firewalling, or installing the root certificate for a MITM proxy within balenaOS to allow for traffic sniffing before it leaves the network, instead of FQDN access listing. BalenaOS also supports RedSocks for proxy configuration.

Feedback is welcome

Please sound off in the comments below (our Forums) if you have any questions or concerns.

You can find out what other features we have in progress, let us know about what you'd like to see us developing and upvote features on our public roadmap.