The server likely failed to get the public address. If you attempted to restart the servers it will start with no public address and if it can’t access the service to get the public IP it will be unset. Open the host settings and go through each host and update the public address.
Also if that got synced to the clients and there are no other hosts to fallback to it will be unable to sync the back to the correct IP. That will require the profile to be manually imported to correct it. You should avoid using only IP addresses and configure DNS names. I will make changes to the server to prevent it from syncing an empty value in the future.
FYI, restarting pritunl servers fixed user connection timeout issue and its now working fine
Thanks
@zach can you tell us what the root cause of this outage?
currently is normal. but we cannot guarantee something like this is happen again in the future.
The South Asia Outage post has all the available information and will be updated if any additional information is found.
I could not determine what the cause of the issue was. It appears to have been limited to some Oracle Cloud IP ranges for the us-ashburn-1 data center routing to the South Asia regions limited to AWS ap-south-1 and Google Cloud asia-south1. I know this because I started an AWS instance in ap-south-1 and noticed some of the other servers I have running in Oracle us-ashburn-1 were working and some were not. Additionally the Status Monitoring checks for multiple regions including Asia and it did not have any issues so it couldn’t have been all of Asia.
It was likely a BGP routing misconfiguration for those Oracle Cloud IP ranges in the South Asia data center regions. These types of outages have happened in the past where IP ranges become unavailable in specific regions due to BGP routing problems.
I think the limited information on the issue is because Oracle Cloud already represents a small share of data center providers and traffic between us-ashburn-1 and South Asia is an even smaller share.