Websocket proxy has chance to lose messages from backend

Hi, @zach,

I have a websocket server protected by pritunl-zero. The websocket server will send out a message as soon as a connection is established. It seems pritunl-zero has chance to lose the first message.

The issue can be easily reproduced with this example websocket server implementation.

As you can see the server sends server says: hello! you're connected after connection is established.

By clicking the “open” button then “close”, we can check the message received by client. If everything works fine, we should expect the “hello! you’re connected” message is received everytime after clicking “open” button. If we repeat the test multiple times, we can observe the initial message is lost.

After checking the pritunl-zero source code, I think there is a race condition. After we created the backConn, and before we start to copy the bytes received from backConn to frontConn, the data sent by backConn should be buffered by the websocket.Conn, and with current implementation the data won’t be forwarded to frontConn.

I have another piece of golang code to simulate pritunl-zero. By changing the forwarding logic from

_, _ = io.Copy(w.front.UnderlyingConn(), w.back.UnderlyingConn())

to

for {
			msgType, msg, err := w.back.ReadMessage()
			if err != nil {
				m := websocket.FormatCloseMessage(websocket.CloseNormalClosure, fmt.Sprintf("%v", err))
				if e, ok := err.(*websocket.CloseError); ok {
					if e.Code != websocket.CloseNoStatusReceived {
						m = websocket.FormatCloseMessage(e.Code, e.Text)
					}
				}
				slog.Error("WebSocket back read error", slog.Any("error", err))
				_ = w.front.WriteMessage(websocket.CloseMessage, m)
				break
			}

			err = w.front.WriteMessage(msgType, msg)
			if err != nil {
				slog.Error("WebSocket front write error", slog.Any("error", err))
				break
			}
		}

the issue is fixed.

Could you take a look? If it makes sense, could you provide a fix?

I have added this code and it does fix the issue. I noticed in the UnderlyingConn() comment that it was deprecated and replaced with NetConn(). That has the comment below warning that directly writing or reading it will corrupt the WebSocket connection.

// NetConn returns the underlying connection that is wrapped by c.
// Note that writing to or reading from this connection directly will corrupt the
// WebSocket connection.

I believe some change was made to this WebSocket library that prevented directly using the underlying connection. I also use this code to proxy WebSocket VNC connections in Pritunl Cloud and initially it worked without any issues but more recently it would randomly fail to open the connection. This change fixed that issue.

Thanks @zach for the quick response and fix, really appreciate it.

Do we have plan to release a new version to include the fix?

It’s a blocker for us to adopt pritunl-zero in our production system. If no such plan, is there any doc about building the .deb package locally?

I’m finishing some testing and will publish a release in the next hour. The software can be built from source using the commands below. Go has cache servers and GOPROXY=direct may need to be added before go install if the cache is outdated but this is significantly slower.

sudo dnf -y install go git
go install github.com/pritunl/pritunl-zero@latest
sudo cp -f ~/go/bin/pritunl-zero /usr/bin/pritunl-zero
sudo systemctl restart pritunl-zero

Thanks. I’m able to build a binary from source locally.

But I think I’m unable to build a private release in the form of deb or rpm package locally?

Just curious. I can wait for the next release.

I’ve started the build and it will be on the repository in about 10 minutes. There may still be some issues because I believe ReadMessage() only captures data messages. It’s possible some of the control messages are not getting proxied but I don’t see any issues with the WebSocket tests I’ve done.

The source for the packages in the pritunl/pritunl-pacur repository, this uses a custom build system pacur/pacur.

1 Like