Pritunl crashing each day

Pritunl
1

Hi there!

Now I have two instances of pritunl. One of this work fine, but second crashing mongodb each day.

I reinstall pritunl and mongodb - it’s work, but next day got same error.

Any suggestions how to fix it?

systemctl status pritunl
● pritunl.service - Pritunl Daemon
     Loaded: loaded (/etc/systemd/system/pritunl.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-03-30 10:17:48 MSK; 22h ago
   Main PID: 458333 (pritunl)
      Tasks: 95 (limit: 1100)
     Memory: 49.3M
     CGroup: /system.slice/pritunl.service
             ├─458333 /usr/lib/pritunl/usr/bin/python3 /usr/lib/pritunl/usr/bin/pritunl start
             └─469316 pritunl-web

Mar 31 09:17:28 HOSTNAME pritunl[458333]:   File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1617, in _retryable_>
Mar 31 09:17:28 HOSTNAME pritunl[458333]:   File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1511, in _cmd
Mar 31 09:17:28 HOSTNAME pritunl[458333]:   File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/server.py", line 133, in run_operation
Mar 31 09:17:28 HOSTNAME pritunl[458333]:   File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/helpers.py", line 180, in _check_command_re>
Mar 31 09:17:28 HOSTNAME pritunl[458333]: pymongo.errors.OperationFailure: error processing query: ns=pritunl.messagesTree: $and
Mar 31 09:17:28 HOSTNAME pritunl[458333]:     _id $gt ObjectId('64260dd8bd85efced8d09df9')
Mar 31 09:17:28 HOSTNAME pritunl[458333]:     channel $in [ "client" "client_links" "hosts" "instance" "port_forwarding" "queue" "servers" "setting" ">
Mar 31 09:17:28 HOSTNAME pritunl[458333]: Sort: {}
Mar 31 09:17:28 HOSTNAME pritunl[458333]: Proj: {}
Mar 31 09:17:28 HOSTNAME pritunl[458333]:  tailable cursor requested on non capped collection, full error: {'ok': 0.0, 'errmsg': 'error processing que>

systemctl status mongod
● mongod.service - MongoDB Database Server
     Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-03-29 15:40:47 MSK; 1 day 17h ago
       Docs: https://docs.mongodb.org/manual
   Main PID: 331 (mongod)
     Memory: 258.7M
     CGroup: /system.slice/mongod.service
             └─331 /usr/bin/mongod --config /etc/mongod.conf

Mar 30 10:27:34 HOSTNAME systemd[1]: mongod.service: Current command vanished from the unit file, execution of the command list won't be resumed.
Warning: journal has been rotated since unit was started, output may be incomplete.

cat /var/log/pritunl.log:

[evening-meadow-6929][2023-03-31 09:23:56,871][ERROR] Error in listener thread
Traceback (most recent call last):
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/runners/listener.py", line 20, in listener_thread
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/helpers.py", line 51, in _wrapped
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/messenger.py", line 124, in subscribe
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/cursor.py", line 1280, in next
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/cursor.py", line 1195, in _refresh
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/cursor.py", line 1078, in __send_message
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1515, in _run_operation
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1617, in _retryable_read
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1511, in _cmd
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/server.py", line 133, in run_operation
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/helpers.py", line 180, in _check_command_response
pymongo.errors.OperationFailure: error processing query: ns=pritunl.messagesTree: $and
    _id $gt ObjectId('64260dd8bd85efced8d09df9')
    channel $in [ "client" "client_links" "hosts" "instance" "port_forwarding" "queue" "servers" "setting" "subscription" "system" "tokens" "vxlan" ]
Sort: {}
Proj: {}
 tailable cursor requested on non capped collection, full error: {'ok': 0.0, 'errmsg': 'error processing query: ns=pritunl.messagesTree: $and\n    _id $gt ObjectId(\'64260dd8bd85efced8d09df9\')\n    channel $in [ "client" "client_links" "hosts" "instance" "port_forwarding" "queue" "servers" "setting" "subscription" "system" "tokens" "vxlan" ]\nSort: {}\nProj: {}\n tailable cursor requested on non capped collection', 'code': 2, 'codeName': 'BadValue'}
2

The capped collection option needs to be set when creating the collection. This issue can occur if the repair database is done while a Pritunl host is connected to the database. This will cause the collection to be created without the capped option.

You will need to stop all Pritunl hosts connected to the database then either delete the messages collection or run sudo pritunl destroy-secondary to remove all cache data from the database.

3 
root@v1457674:~# service pritunl stop
root@v1457674:~# pritunl destroy-secondary
Destroying secondary database...
Secondary database destroyed
root@v1457674:~# service pritunl start
root@v1457674:~# service pritunl status
● pritunl.service - Pritunl Daemon
     Loaded: loaded (/etc/systemd/system/pritunl.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-03-31 10:26:01 MSK; 4s ago
   Main PID: 563949 (pritunl)
      Tasks: 1 (limit: 1100)
     Memory: 207.3M
     CGroup: /system.slice/pritunl.service
             └─563949 /usr/lib/pritunl/bin/python /usr/lib/pritunl/bin/pritunl start

Mar 31 10:26:01 v1457674.hosted-by-vdsina.ru systemd[1]: Started Pritunl Daemon.

After this I got running pritunl but database was clean.
And i got Use "sudo pritunl default-password" to get default login information on frontend.

4

That command will only remove cache collections. Verify the correct database URI is configured in /etc/pritunl.conf

5 
root@v1457674:~# cat /etc/pritunl.conf
{
    "debug": false,
    "bind_addr": "0.0.0.0",
    "port": 443,
    "log_path": "/var/log/pritunl.log",
    "temp_path": "/tmp/pritunl_%r",
    "local_address_interface": "auto",
    "mongodb_uri": "mongodb://localhost:27017/pritunl"
}

Seems database was corrupted but not recovered. How I can debug why this corrupted?

6

Check sudo cat /var/log/mongodb/mongod.log for errors.

7

It’s contain 2gb log. I try grep most of similar errors, but not see critical.

root@v1457674:~# cat /var/log/mongodb/mongod.log | grep Error | grep -v SSLHandshakeFailed | grep -v ProtocolError
{"t":{"$date":"2023-02-07T18:27:16.216+03:00"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn32","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Connection reset by peer"},"remote":"xxx.xxx.xxx.xxx:54648","connectionId":32}}
{"t":{"$date":"2023-02-20T20:01:10.057+03:00"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn186","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"Broken pipe"},"remote":"xxx.xxx.xxx.xxx:57256","connectionId":186}}
{"t":{"$date":"2023-02-22T02:30:09.077+03:00"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn215","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Connection reset by peer"},"remote":"xxx.xxx.xxx.xxx:39858","connectionId":215}}
{"t":{"$date":"2023-02-27T16:27:56.192+03:00"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn361","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"Broken pipe"},"remote":"xxx.xxx.xxx.xxx:40502","connectionId":361}}
{"t":{"$date":"2023-02-27T20:14:51.247+03:00"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn385","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"Broken pipe"},"remote":"xxx.xxx.xxx.xxx:34406","connectionId":385}}
{"t":{"$date":"2023-03-08T22:01:19.784+03:00"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn582","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Connection reset by peer"},"remote":"xxx.xxx.xxx.xxx:48740","connectionId":582}}
{"t":{"$date":"2023-03-16T21:14:16.269+03:00"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn760","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"Broken pipe"},"remote":"xxx.xxx.xxx.xxx:37600","connectionId":760}}
{"t":{"$date":"2023-03-26T21:46:17.236+03:00"},"s":"W",  "c":"NETWORK",  "id":5746600, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":"Transport endpoint is not connected"}}
{"t":{"$date":"2023-03-27T20:29:04.439+03:00"},"s":"W",  "c":"NETWORK",  "id":5746600, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":"Transport endpoint is not connected"}}
{"t":{"$date":"2023-03-29T21:12:34.250+03:00"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn57","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"Broken pipe"},"remote":"xxx.xxx.xxx.xxx:47360","connectionId":57}}