[ERROR] Instance doc lost, stopping server. Check datetime settings

blueglowey · May 12, 2025, 10:29am

Hi,

I’m having issues with VPN instances repeatedly restarting causing client reconnects. When looking at the log file I see the following error for different VPN server instances on the Pritunl server several times a day.

[thriving-forest-9169][2025-05-11 16:41:00,513][WARNING] Removing instance doc
  server_id      = "67ed374617b505acd725ee94"
  instance_id    = "682013e6c94726328edb72e6"
  cur_timestamp  = "2025-05-11 16:41:00.500098"
  ttl_timestamp  = "2025-05-11 16:40:30.500098"
  ping_timestamp = "2025-05-11 16:39:55.870000"
[thriving-forest-9169][2025-05-11 16:41:00,571][INFO] Recovering server state
  server_id      = "67ed374617b505acd725ee94"
  prefered_hosts = ["ae928187adbf440184e662be89250926"]
[thriving-forest-9169][2025-05-11 16:41:00,641][INFO] Starting vpn server
  server_id        = "67ed374617b505acd725ee94"
  instance_id      = "6820d31cc94726328edc7d04"
  instances        = []
  instances_count  = 0
  route_count      = 0
  network          = "10.0.31.0/24"
  network6         = "fd00:a001:f00::/64"
  dynamic_firewall = false
  route_dns        = false
  device_auth      = false
  host_id          = "ae928187adbf440184e662be89250926"
  host_address     = <snip>
  host_address6    = <snip>
  host_networks    = <snip>
  cur_timestamp    = "2025-05-11 16:41:00.641150"
  libipt           = false
[thriving-forest-9169][2025-05-11 16:41:01,637][ERROR] Instance doc lost, stopping server. Check datetime settings
  server_id     = "67ed374617b505acd725ee94"
  instance_id   = "682013e6c94726328edb72e6"
  cur_timestamp = "2025-05-11 16:41:01.632152"
Traceback (most recent call last):
  File "/usr/lib/pritunl/usr/lib/python3.9/threading.py", line 937, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/pritunl/usr/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/usr/lib/pritunl/usr/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/helpers.py", line 42, in _wrapped
    for _ in call(*args, **kwargs):
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/server/instance.py", line 1120, in _keep_alive_thread
    logger.error(
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/logger/__init__.py", line 55, in error
    kwargs['traceback'] = traceback.format_stack()
[thriving-forest-9169][2025-05-11 16:41:05,697][WARNING] Stopping duplicate instance, check date time sync
  server_id   = "67ed374617b505acd725ee94"
  instance_id = "682013e6c94726328edb72e6"

I’m running

Ubuntu 20.04 LTS
pritunl v1.32.3805.95
mongodb v4.4.3 (Single Server, local instance)

System time is correct and updated via NTP. There are not any mongo errors in journalctl around the times the VPN server instances are restarted by pritunl.

Are there any other log files I can look at for a reason as to what would cause for errors relating to datetime sync?

Any pointers would be most appreciated.

Cheers

James

zach · May 12, 2025, 2:25pm

If the database is on the local system it is likely locking up long enough for the timeout to occur. Run sudo pritunl set vpn server_ping_ttl 90 to increase the timeout. Also verify there are no resource usage issues on the server. Check sudo dmesg -Tkw for errors.

blueglowey · May 13, 2025, 4:17pm

Zach,

Thanks for the pointers. There was nothing in dmesg and the system had free resources but i increased the RAM and the error appeared to have cleared without increasing the ttl value.

Out of interest are there any recommended system requirements for various connected clients/server instances? I would imagine it varies with client throughput.

Cheers

James

blueglowey · May 14, 2025, 1:02pm

I’ve been keeping an eye on the main log and general system resources and things continue to look stable. The only error I did see in /var/log/pritunl.log is

[thriving-forest-9169][2025-05-14 10:33:36,461][ERROR] Exception on /event/68247180a1a9c916b9165928 [GET]                                                                                                                                     
Traceback (most recent call last):                                                                                                                                                                                                            
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 2190, in wsgi_app                                                                                                                                                
    response = self.full_dispatch_request()                                                                                                                                                                                                   
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1486, in full_dispatch_request                                                                                                                                   
    rv = self.handle_user_exception(e)                                                                                                                                                                                                        
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1484, in full_dispatch_request                                                                                                                                   
    rv = self.dispatch_request()                                                                                                                                                                                                              
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1469, in dispatch_request                                                                                                                                        
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)                                                                                                                                                                  
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/auth/app.py", line 10, in _wrapped                                                                                                                                           
    return call(*args, **kwargs)                                                                                                                                                                                                              
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/handlers/event.py", line 28, in event_get                                                                                                                                    
    return utils.jsonify(event.get_events(                                                                                                                                                                                                    
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/event.py", line 25, in get_events                                                                                                                                            
    for event in messenger.subscribe('events', cursor_id=cursor,                                                                                                                                                                              
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/helpers.py", line 51, in _wrapped                                                                                                                                            
    for value in call(*args, **kwargs):                                                                                                                                                                                                       
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/messenger.py", line 141, in subscribe                                                                                                                                        
    for doc in cursor:                                                                                                                                                                                                                        
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/cursor.py", line 1280, in next                                                                                                                                               
    if len(self.__data) or self._refresh():                                                                                                                                                                                                   
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/cursor.py", line 1195, in _refresh                                                                                                                                           
    self.__send_message(q)                                                                                                                                                                                                                    
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/cursor.py", line 1078, in __send_message
    response = client._run_operation(
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1515, in _run_operation
    return self._retryable_read(
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1617, in _retryable_read
    return func(session, server, sock_info, secondary_ok)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1511, in _cmd
    return server.run_operation(
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/server.py", line 133, in run_operation
    _check_command_response(first, sock_info.max_wire_version)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pymongo/helpers.py", line 180, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: Executor error during find command :: caused by :: CollectionScan died due to position in capped collection being deleted. Last seen record id: RecordId(49072072), full error: {'ok': 0.0, 'errmsg': 'Execut
or error during find command :: caused by :: CollectionScan died due to position in capped collection being deleted. Last seen record id: RecordId(49072072)', 'code': 136, 'codeName': 'CappedPositionLost'}

Is this another mongo error related to resources?

Cheers

James

zach · May 14, 2025, 8:57pm

That can happen from a tab left open with the admin console. It is the event stream that indicates to the web app to refresh components of the page when changes occur. If there are continued issues with that collection run sudo pritunl clear-message-cache.

The server should have at least 2gb RAM and CPU types with higher core speeds will perform best.