Excessive WebSocket 401 Logs in Operational Environment

Pritunl Pritunl Zero
1

Hello,

I am experiencing a recurring issue in our Pritunl Zero environment. While everything is 100% operational for end-users, I am seeing constant errors in the logs like the example below:

proxy: WebSocket dial error 401
websocket: bad handshake
ORIGINAL STACK TRACE:
github.com/pritunl/pritunl-zero/proxy.(*webSocket).ServeHTTP
	/go/src/github.com/pritunl/pritunl-zero/proxy/ws.go:298 +0x12d2b8d
github.com/pritunl/pritunl-zero/proxy.(*Proxy).ServeHTTP
	/go/src/github.com/pritunl/pritunl-zero/proxy/proxy.go:266 +0x12c946f
github.com/pritunl/pritunl-zero/router.(*Router).ServeHTTP
	/go/src/github.com/pritunl/pritunl-zero/router/router.go:97 +0x12ef4d8
net/http.serverHandler.ServeHTTP
	/usr/local/go/src/net/http/server.go:2938 +0x71114d
net/http.(*conn).serve
	/usr/local/go/src/net/http/server.go:2009 +0x70d033
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x46dde0

These logs appear frequently, even though the environment is running normally, and they are causing unnecessary growth in our MongoDB cluster, impacting costs and monitoring.

I noticed there is a recent release (v1.0.3265.55) that seems to address WebSocket and X-Forwarded headers issues. Could you confirm if this update would resolve the problem, or should I take any additional actions?

Looking forward to your guidance. Thank you!

2

Some WebSocket implementations will handle authorization in a way that won’t work through Pritunl Zero. It could be a CSRF protection, in the External Domains set the Host field to the domain that the internal service would expect to be running on. If it still doesn’t work disable Allow WebSockets for that service.

3

Hello Zarch, how are you?
Quick question: wouldn’t disabling this function cause downtime in the solution, as it would block WebSocket communication? Or did I misunderstand the functionality of this feature? Currently, this function is enabled.

4

If the WebSocket requests don’t work it disabling WebSockets won’t add any additional problems. It’s possible it is working and that is the first request that gets sent on an expired session but that shouldn’t generate a lot of logs.

5

Zach, thanks for the quick reply!

Here’s the issue and why I suspect it might be a bug: the environment is fully operational, and users are using the application normally. However, I noticed that the error logs mostly appear in groups of the same IPs — for example, three logs from one IP, three from another, then it repeats the first, and so on. It looks like a looping pattern, which doesn’t seem to align with the expected behavior.

6

Open the Chrome Developer Tools and check the Network tab for request errors. I’ve done a lot of testing with the new WebSocket proxy code and I also use the same code for web VNC in Pritunl Cloud. I haven’t seen any issues and the new code fixed previous issues with the web VNC. Open the Nodes tab and verify the version 1.0.3324.37 is running.

7

Good morning, Zach!

I followed your instructions and opened the Developer Tools in Chrome. Below is an example of the logs from the Network > Response tab.
I noticed that the version of the nodes running is 1.0.2981.59.

Let me know if you need any additional details or further checks!

"logs": [
        {
            "id": "6798dd76ce0fd592ae064003",
            "level": "error",
            "timestamp": "2025-01-28T13:36:54.847Z",
            "message": "proxy: Serve error",
            "stack": "proxy: WebSocket dial error 401\nwebsocket: bad handshake\nORIGINAL STACK TRACE:\ngithub.com/pritunl/pritunl-zero/proxy.(*webSocket).ServeHTTP\n\t/go/src/github.com/pritunl/pritunl-zero/proxy/ws.go:298 +0x12d2b8d\ngithub.com/pritunl/pritunl-zero/proxy.(*Proxy).ServeHTTP\n\t/go/src/github.com/pritunl/pritunl-zero/proxy/proxy.go:266 +0x12c946f\ngithub.com/pritunl/pritunl-zero/router.(*Router).ServeHTTP\n\t/go/src/github.com/pritunl/pritunl-zero/router/router.go:97 +0x12ef4d8\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938 +0x71114d\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009 +0x70d033\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650 +0x46dde0\n",
            "fields": {
                "client": "<client_ip>"
            }

Note that <client_ip> was used to obfuscate the user’s IP address for privacy reasons.

8

Upgrading will likely fix the issue. Older versions would drop data from the WebSocket during the connection initialization which can cause the connection to fail.

9

Thanks, Zach!
That makes sense. I’ll proceed with the upgrade and see if it resolves the issue. Appreciate the insight!

10

Hi Zach,

Thanks for your suggestion! I have updated all Pritunl Zero nodes to version 1.0.3324.37, but I’m still seeing excessive log generation. These logs seem related to Rancher, which runs behind Pritunl, but the application is working fine. The issue is that they are taking up a lot of space in my MongoDB Atlas storage.

Is there a way to ignore these events or prevent them from being logged?

One possibility I considered (but I’m not sure if it works) is modifying the Proxy Headers to support WebSockets by adding:

{
  "Upgrade": "websocket",
  "Connection": "Upgrade"
}

Or manually configuring a proxy in /etc/pritunl-zero-proxy.json:

{
  "proxies": [
    {
      "name": "Rancher Proxy",
      "listen_addr": "0.0.0.0",
      "upstream": "https://rancher.seudominio.com",
      "proxy_headers": {
        "Upgrade": "websocket",
        "Connection": "Upgrade"
      }
    }
  ]
}

I’m not sure if this approach would help reduce unnecessary logging, or if there is a better way to configure Pritunl Zero to ignore these events. Do you have any recommendations?

Thanks again for your help, I really appreciate it.

11

Logging for WebSocket requests will be disabled by default in the next release. The command sudo pritunl-zero set router debug_websocket true will enable the logging.

12

Hi Zach,

Thanks for the update! That’s great to hear.

Is there a way to manually disable WebSocket logging in the current version before the next release? Also, do you have an estimated timeline for the upcoming release?

Appreciate your help!

13

It would require modifying the code. The repository currently has a lot of uncommited code related to adding paging to all of the tabs in the web console so installing directly from the repository won’t work.

You can run the commands below in the MongoDB shell to expire the documents after 10 minutes. This will remain until the Pritunl Zero server is restarted which would replace the index.

db.logs.dropIndex("timestamp_1")
db.logs.createIndex({ "timestamp": 1 }, { expireAfterSeconds: 600 })
14

Thanks for all, Zach!