When some of our users (different geographical locations and ISP) try to connect to their usual Pritunl profile, they pass the Okta SSO authentications successfully, but are disconnected from the VPN immediately after. The same behavior happens every time they retry.
Other users of the same profile are totally fine and we couldn’t find the difference between these two groups yet.
There are no logs on the Pritunl server side.
A tcpdump captured on the Pritunl server seems to show the TLS handshake ending with a TCP RST. There is no traffic between the server and the client afterwards.
The Pritunl client version being used is v1.3.4262.38
The Pritunl server is running v1.32.4258.38 (e8414a) on Ubuntu 22.04.5
Most clients are running on MacOS 15.x versions
We tried lowering the “Connection MTU” to 1200 on the server, but it didn’t helped.
We also tried to recreate our VPN server on AlmaLinux 9.5 to benefit from more recent packages, but the same users are facing the same issue on this server too.
The Pritunl server had not been modified for some time when the first user started reporting this issue, around two weeks ago.
It’s failing on the pre-connection authentication which is done with a web request to the Pritunl web server port. It wouldn’t be effected by MTU or any settings with the VPN. Check for errors in the server and verify the CPU or memory are not overloaded. Also run sudo journalctl -u pritunl -n 5000 and look for errors coming directly from the web server process.
Running sudo pritunl set app.web_systemd true and sudo systemctl restart pritunl may help if there are resource issues on the server. Then run sudo journalctl -u pritunl-web -n 5000 and check for errors from the web server.
When an impacted user tries to connect there is no log showing for pritunl or pritunl-web. We tested that with app.web_systemd to false and true and there is no difference.
I can confirm that the server isn’t overloaded.
We also noticed that if impacted people connect to another VPN service first and then connect to their Pritunl profile the connection is established successfully.
In that code after a connection that requires single sign-on is sent to the background service the CLI client will immediately begins a loop to get the single sign-on authorization URL from the background service. It will check every 100ms until it get’s the URL. That shouldn’t cause any issues and it’s also only done with the CLI. The GUI application has an event stream that is used to notify the GUI.
There was a possible context leak in the code where the connection reset error is occurring. Fixed in pritunl-client-electron:56e1994. I had left a TODO message about it but never fixed it. But I don’t think it would produce that error and the context leak is likely not possible with the current usage of the code. I did try to reproduce any possibly race conditions and it still wouldn’t return connection reset by peer.
Also run the lsof commands listed in the server debugging documentation. Check for excessive threads or open connections. Do this shortly after the issue occurs.
There may be an issue with the web server thread adjustments made about 6 months ago. First run sudo pritunl get app.request_queue_size if this returns 50 it is the newer release with the higher values. If it is revert it to the previous values by running the commands below then restart the Pritunl service. If it isn’t 50 you should try updating the server because there was a resource issue specifically with WireGuard connections which were overloading the server due to excessive RSA validation of every WireGuard ping request.
sudo pritunl set app.request_queue_size 10
sudo pritunl set app.request_accepted_queue_size 50
sudo pritunl set app.request_thread_count 250
sudo pritunl set app.request_max_thread_count 50
sudo systemctl restart pritunl
Running the lsof commands and monitoring the number of open files during the connection attempts didn’t showed any noticeable change. We ran these checks before and after changing the app.request_* values you mentioned (swapping the values for the last two settings).
We also ran py-spy and didn’t saw anything special there.
Just in case, we also scaled up our Pritunl instance to c7a.large host in AWS. The machine is mostly idle.
In order to mitigate the problem for our users we cloned our VPN server into another temporary one, on the same host, where the Single Sign-On Authentication parameter is disabled.
The users reporting the issues are the only one authorized to download the profile using this temporary server. A few more people reported the problem today, even if they didn’t changed anything voluntarily since last week, when the usual profile was still working normally for them.
We believe there might be some local update impacting their Pritunl client behavior somehow, when the SSO auth is enabled. We’re still trying to figure out what exactly since no one in my team can reproduce the issue.
I will look into reproducing the issue. The difference between a single sign-on connection and the other connections is the web request before the VPN connection. The other connections will directly connect to the VPN server. With single sign-on and several other authentication modes a web request to /key/ovpn is sent first to complete the authentication which then provides a token. This token is then used for the OpenVPN connection. WireGuard always uses authentication with web requests.
You should verify there is nothing interfering with that web server. Try running the command below in a PowerShell terminal to simulate a POST with JSON data that is similar to the client request. This should return a 401 error with auth_invalid. If there is a load balancer or any kind of web application firewall those may filter requests that don’t appear to come from a web browser.
I can confirm that nothing interferes with the web server. It is running on a publicly available host without any load balancer or anything else in front of it.
I asked several users to run the curl command you provided and they all got the same output:
{"error": "auth_invalid", "error_msg": "Authentication credentials are not valid."}%
