Server start times out

aluffy · June 2, 2023, 3:00pm

Our servers will not restart successfully. The VPN server is configured and allocated to two hosts in a non-replicated manner. The Pritunl servers are installed per the instructions. We are using Oracle 8.7 on AWS. When we start the pritunl service, we can get a server to start and it’s stable until we restart the vpn server within Pritunl UI. The logs on the server are the following:

[restless-stars-2745][2023-06-02 14:49:39,777][ERROR] Exception on /server/6464f8cb35b75d3f5646b875/operation/start [PUT]
Traceback (most recent call last):
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/auth/app.py", line 10, in _wrapped
    return call(*args, **kwargs)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/handlers/server.py", line 1351, in server_operation_put
    svr.start()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/server/server.py", line 1656, in start
    raise ServerStartError('Server start timed out', {
pritunl.exceptions.ServerStartError: Server start timed out. {'server_id': ObjectId('6464f8cb35b75d3f5646b875')}
[restless-stars-2745][2023-06-02 14:50:30,493][ERROR] Exception on /server/6464f8cb35b75d3f5646b875/operation/start [PUT]
Traceback (most recent call last):
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/auth/app.py", line 10, in _wrapped
    return call(*args, **kwargs)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/handlers/server.py", line 1351, in server_operation_put
    svr.start()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/server/server.py", line 1656, in start
    raise ServerStartError('Server start timed out', {
pritunl.exceptions.ServerStartError: Server start timed out. {'server_id': ObjectId('6464f8cb35b75d3f5646b875')}
[restless-stars-2745][2023-06-02 14:50:30,493][ERROR] Exception on /server/6464f8cb35b75d3f5646b875/operation/start [PUT]
Traceback (most recent call last):
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/auth/app.py", line 10, in _wrapped
    return call(*args, **kwargs)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/handlers/server.py", line 1351, in server_operation_put
    svr.start()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/server/server.py", line 1656, in start
    raise ServerStartError('Server start timed out', {
pritunl.exceptions.ServerStartError: Server start timed out. {'server_id': ObjectId('6464f8cb35b75d3f5646b875')}

Thanks,
Andy

zach · June 4, 2023, 6:34pm

This can happen if the host ID is deleted or reset causing the hosts to be recreated. Verify that the hosts that are marked as online are the same hosts attached to the server and check that no offline hosts are attached to servers. If a server has no online hosts attached the startup will timeout.

aluffy · June 7, 2023, 2:01am

Hey Zach,
Thanks for the reply. I did verify that all attached hosts are online appear to be accurate in the mongo database as well. Still no success in getting the server to restart.

zach · June 7, 2023, 6:55pm

It could be caused by issues with iptables or the servers having too many routes. These issues should show other error messages. Also check /var/log/pritunl.log for database connection errors which would not be shown in the web console logs.

aluffy · June 8, 2023, 7:22pm

No luck. Database looks stable and in checking the logs, it’s the same error.

[restless-stars-2745][2023-06-08 19:16:45,384][ERROR] Exception on /server/648227b68c51fa26c76944fc/operation/start [PUT]
Traceback (most recent call last):
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/auth/app.py", line 10, in _wrapped
    return call(*args, **kwargs)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/handlers/server.py", line 1351, in server_operation_put
    svr.start()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/server/server.py", line 1656, in start
    raise ServerStartError('Server start timed out', {
pritunl.exceptions.ServerStartError: Server start timed out. {'server_id': ObjectId('648227b68c51fa26c76944fc')}
[restless-stars-2745][2023-06-08 19:16:45,384][ERROR] Exception on /server/648227b68c51fa26c76944fc/operation/start [PUT]
Traceback (most recent call last):
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/auth/app.py", line 10, in _wrapped
    return call(*args, **kwargs)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/handlers/server.py", line 1351, in server_operation_put
    svr.start()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/server/server.py", line 1656, in start
    raise ServerStartError('Server start timed out', {
pritunl.exceptions.ServerStartError: Server start timed out. {'server_id': ObjectId('648227b68c51fa26c76944fc')}
[restless-stars-2745][2023-06-08 19:18:48,307][ERROR] Exception on /server/648227b68c51fa26c76944fc/operation/start [PUT]
Traceback (most recent call last):
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/auth/app.py", line 10, in _wrapped
    return call(*args, **kwargs)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/handlers/server.py", line 1351, in server_operation_put
    svr.start()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/server/server.py", line 1656, in start
    raise ServerStartError('Server start timed out', {
pritunl.exceptions.ServerStartError: Server start timed out. {'server_id': ObjectId('648227b68c51fa26c76944fc')}
[restless-stars-2745][2023-06-08 19:18:48,307][ERROR] Exception on /server/648227b68c51fa26c76944fc/operation/start [PUT]
Traceback (most recent call last):
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/auth/app.py", line 10, in _wrapped
    return call(*args, **kwargs)
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/handlers/server.py", line 1351, in server_operation_put
    svr.start()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/server/server.py", line 1656, in start
    raise ServerStartError('Server start timed out', {
pritunl.exceptions.ServerStartError: Server start timed out. {'server_id': ObjectId('648227b68c51fa26c76944fc')}

Tried deleting the server and creating a new server. If I restart the Pritunl service, the server will start once, and then die if I try to restart the server without restarting the service first.

I’ve tried it with selinux disabled and have confirmed all iptables and firewalld services have been removed and stopped.

aluffy · June 8, 2023, 7:25pm

Also, server only has one route defined.

zach · June 8, 2023, 9:08pm

If the disk image was cloned this could cause both hosts to use the same ID creating inconsistencies in the database. Delete the file /var/lib/pritunl/pritunl.uuid and run sudo systemctl restart pritunl.

zahornyak · September 18, 2023, 11:52am

Have the same error. Using single-node pritunl on docker. After server restart it can not start again. It can start only after restart of the container.

[thawing-forest-9678][2023-09-18 11:39:11,650][ERROR] Exception on /server/6436b69017707b585f32e0e0/operation/start [PUT]
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python3.10/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/lib/python3.10/site-packages/pritunl-1.32.3602.80-py3.10.egg/pritunl/auth/app.py", line 10, in _wrapped
    return call(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/pritunl-1.32.3602.80-py3.10.egg/pritunl/handlers/server.py", line 1351, in server_operation_put
    svr.start()
  File "/usr/lib/python3.10/site-packages/pritunl-1.32.3602.80-py3.10.egg/pritunl/server/server.py", line 1656, in start
    raise ServerStartError('Server start timed out', {
pritunl.exceptions.ServerStartError: Server start timed out. {'server_id': ObjectId('6436b69017707b585f32e0e0')}

zach · September 21, 2023, 1:52am

The /var/lib/pritunl/pritunl.uuid file stores the ID of the host. If this file is not preserved a new host will be created when the container restarts. That host will not be attached to any of the servers and the servers will time out on start due to being unable to find an online host that is attached. You will need to reattach the host to the servers and delete the unused offline hosts.