Pritunl client re-connection issue (wg mode)

s2504s · August 2, 2023, 10:20am

Hi!
I am using the Pritunl Enterprise version and yesterday I faced an issue with reconnecting on the client side:
What I did:

Stopped the VPN hosts(for redundancy, I am using two VPN hosts)
Restored Mongo database from nightly backup
Started both VPN hosts
Attached both to the Servers
Restarted Servers

What is the issue:
All Linux (and several Windows) pritunl-clients could not reconnect to the server - looped restart occurred

Logs from LINUX client

[2023-08-02 09:53:49][ERRO] ▶ profile: Failed to start system profile ◆ profile_id="hpwjhpin2znfeqex"
profile: Request wg returned empty data
ORIGINAL STACK TRACE:
github.com/pritunl/pritunl-client-electron/service/profile.(*Profile).startWg
	/go/src/github.com/pritunl/pritunl-client-electron/service/profile/profile.go:3891 +0x9eb5f0
github.com/pritunl/pritunl-client-electron/service/profile.(*Profile).Start
	/go/src/github.com/pritunl/pritunl-client-electron/service/profile/profile.go:1291 +0x9dab2c
github.com/pritunl/pritunl-client-electron/service/profile.SyncSystemProfiles.func1
	/go/src/github.com/pritunl/pritunl-client-electron/service/profile/utils.go:406 +0x9efe64
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x468960

[2023-08-02 09:53:51][INFO] ▶ profile: Connecting ◆ device_auth=false ◆ disable_gateway=false ◆ dynamic_firewall=false ◆ force_dns=false ◆ mode="wg" ◆ profile_id="hpwjhpin2znfeqex" ◆ reconnect=true ◆ sso_auth=false
[2023-08-02 09:53:51][INFO] ▶ profile: Disconnecting ◆ profile_id="hpwjhpin2znfeqex"
[2023-08-02 09:53:52][INFO] ▶ profile: Disconnected ◆ profile_id="hpwjhpin2znfeqex"
[2023-08-02 09:53:52][ERRO] ▶ profile: Failed to start system profile ◆ profile_id="hpwjhpin2znfeqex"

Logs from Windows client

[2023-7-10 17:2:40][ERROR] Request:  Client error ssl=false hostname=127.0.0.1 port=9770 method=GET path=/profile
Error: connect ECONNREFUSED 127.0.0.1:9770
Error: connect ECONNREFUSED 127.0.0.1:9770
    at __node_internal_captureLargerStackTrace (node:internal/errors:477:5)
    at __node_internal_exceptionWithHostPort (node:internal/errors:655:12)
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16)
[2023-7-10 17:2:40][ERROR] Profiles: Status error
RequestError: Request:  Client error ssl=false hostname=127.0.0.1 port=9770 method=GET path=/profile
Error: connect ECONNREFUSED 127.0.0.1:9770
Error: connect ECONNREFUSED 127.0.0.1:9770
    at __node_internal_captureLargerStackTrace (node:internal/errors:477:5)
    at __node_internal_exceptionWithHostPort (node:internal/errors:655:12)
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16)
[2023-7-10 17:2:46][INFO] Profiles: Updating profile 'b0bab32e45e2ea3e'

How to get it fixed (Linux and Windows clients):

Stop the profile - pritunl-client stop ID
Remove the client profile - pritunl-client remove ID
Add the profile again - pritunl-client add pritunl://.....
Start the profile - pritunl-client start ID --mode=wg

Additional info:

All OpenVPN (not Pritunl) clients (vpn-mode) that use Pritunl config restarted automatically and did not face this issue
I have the client with the endless loop if it is needed for debugging
I tried to use the latest pritunl-client for fixing the issue, but no luck
On the Linux clinet I ran the tcpdump and restarted the profile - no connections were made by pritunl client to pritunl server, seems like this is a client issue
Pritunl Client v1.3.3484.2
Pritunl Server v1.32.3602.80 b11bda

I assume that this issue is related to the pritunl-client that works in WG mode
Please feel free to ask any questions

Thanks in advance!

s2504s · August 2, 2023, 4:53pm

This is very interesting. This is content from the file /var/lib/pritunl-client/pritunl-client.json:

{
	"id": "hpwjhpin2znfeqex",
	"name": "",
	"wg": true,
	"last_mode": "wg",
	"organization_id": "6xxxxxxxc7",
	"organization": "servers",
	"server_id": "646xxxxxx9f6fed",
	"server": "servers",
	"user_id": "64xxxxxd40d1c4b7b",
	"user": "blackbox",
	"pre_connect_msg": "",
	"dynamic_firewall": false,
	"device_auth": false,
	"disable_gateway": false,
	"force_dns": false,
	"sso_auth": false,
	"password_mode": "",
	"token": false,
	"token_ttl": 3600,
	"disabled": false,
	"sync_time": 1690872475,
	"sync_hosts": [],
	"sync_hash": "xxxxxxxxxxxxxxxxxxxxxxxxx",
	"sync_secret": "xxxxxxxxxxxxxxxxxxxxxxxxxxx",
	"sync_token": "xxxxxxxxxxxxxxxxxxxxxxxxx",
	"server_public_key": ["-----BEGIN RSA PUBLIC KEY-----", "MIICEAAQ==", "-----END RSA PUBLIC KEY-----"],
	"server_box_public_key": "XXXXXX=",
	"registration_key": "",
	"ovpn_data": "setenv UV_ID d66008f...."
	"password": ""
}

Looks like there are no sync_hosts in this configuration and there is no remote in the ovpn_data value.
Or pritunl-service does not know how to update its configuration either connect to remote because this field is absent?

zach · August 3, 2023, 1:24pm

This may be an issue reaching the Pritunl public IP servers to automatically detect the host IP address. Open the host settings in the hosts tab with a enterprise subscription or the top right settings without a subscription to verify the host public address is set.

s2504s · August 3, 2023, 1:29pm

Public address is set - I add this within the server startup

pritunl set host.public_address ${PRITUNL_SERVER_ADVERT}
pritunl set app.reverse_proxy true
pritunl start

s2504s · August 3, 2023, 1:52pm

I have not checked the client code yet, could you please tell me - does the client update the profile configuration file? If yes, then something went wrong after synchronization finished and the client did not get sync_hosts and put an empty list in the configuration

zach · August 3, 2023, 3:30pm

There is no host.public_address setting available from the CLI options. The host address can only be set in the web console. Set the public address field in the top right settings.

s2504s · August 3, 2023, 3:34pm

Do you mean this one? It is set already

zach · August 3, 2023, 3:38pm

Verify that the host is attached to the server and try downloading the profile archive format and open it with a text editor to check for issues with the remote or sync host value.

s2504s · August 3, 2023, 3:41pm

I wrote how to resolve this issue:

Stop the profile - pritunl-client stop ID
Remove the client profile - pritunl-client remove ID
Add the profile again - pritunl-client add pritunl://.....
Start the profile - pritunl-client start ID --mode=wg

If I do these steps, the issue will be fixed. But I do not want to re-create profile each time

I want the client to avoid updating the sync_hosts field in config if the list of hosts is not received from the Pritunl server

s2504s · August 7, 2023, 10:02am

Dear @zach
Let me clarify this situation - we are an enterprise company and we bought the Pritunl Enterprise license. For now, this situation looks like - we faced an issue, reported it here, and provided a lot of information, we are ready to provide any help that you want, but you think that everything is OK and this is not a bug. It’s very strange to me, but it’s your way of doing business.
I would like to hear your opinion on this in order to decide on the next steps.

Thanks in advance

zach · August 7, 2023, 3:32pm

I would need to be able to reproduce the issue to fix it. Typically with issues like this I will wait for multiple reports to confirm there is a problem with the software.

If you have the public address field set to the default IP address this will leave the host to automatically detect the public address on a regular internal. It’s possible the update is failing temporarily causing the field to be unset and then synced to the client. If the public address field is set to a domain it shouldn’t change on a client sync.

You can run sudo pritunl user.conf_sync false to disable configuration sync.

s2504s · August 7, 2023, 4:13pm

I genuinely appreciate your assistance. Disabling synchronization looks like a good workaround. I have an idea of how to reproduce this issue. I will come back if I have new information

Have a great day!

s2504s · August 8, 2023, 8:35am

Hi @zach
It looks like I have reproduced this issue

How to reproduce it:

Run the server with WG support
Connect the client with WG mode
Make sure that client is connected to the server
Detached all hosts from the Server
After the next client reconnection iteration, Pritunl server returns empty sync_hosts
Client saves this empty param to the configuration

image510×597 25.6 KB

Could you please add some restrictions to avoid saving empty array for sync_hosts?

zach · August 8, 2023, 3:01pm

If all hosts are removed it should return an empty sync hosts. The sync hosts is the list of hosts attached to a server. Hosts shouldn’t be removed from a server until new hosts are added.