Crash in setup inside group_file.py

I am attempting to launch a new pritunl host in AWS and launched via autoscaling groups (exactly 1 replica). But for the most part the setup follows the installation instructions for an oracle linux host. The AMI in derived from the one identified at: Installation that then includes the scripts in this section (minus install of mongodb).
The launch-template does freeze the node-id to ensure new ASG instances look exactly like the previous ones launched by this ASG and the launch template also sets the MongoDBURI. The primary difference is that I have my mongodb cluster on different machines.

I can access the web console. But was unable to register the domain name via let’s encrypt. In diagnosing this, I found the following:

The service does not seem to open the host on port 80 to handle let’s encrypt challenges. The service is started using oracle linux systemd. In fact, due to the exception, In think it failed to finish the setup.

I have found this error displayed from pritunl logs:

Traceback (most recent call last):
  File "/usr/bin/pritunl", line 33, in <module>
    sys.exit(load_entry_point('pritunl==1.32.3805.95', 'console_scripts', 'pritunl')())
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/__main__.py", line 446, in main
    from pritunl import setup
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/setup/__init__.py", line 2, in <module>
    from pritunl.setup.clean import setup_clean
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/setup/clean.py", line 1, in <module>
    from pritunl import utils
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/utils/__init__.py", line 1, in <module>
    from pritunl.utils.cert import *
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/utils/cert.py", line 2, in <module>
    from pritunl.utils.misc import check_output_logged, get_temp_path
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/utils/misc.py", line 4, in <module>
    from pritunl import settings
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/settings/__init__.py", line 22, in <module>
    sys.modules[__name__] = Settings()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/settings/settings.py", line 24, in __init__
    self._init_modules()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/settings/settings.py", line 128, in _init_modules
    group_cls.load()
  File "/usr/lib/pritunl/usr/lib/python3.9/site-packages/pritunl/settings/group_file.py", line 17, in load
    os.chmod(self.path, 0o600)
PermissionError: [Errno 1] Operation not permitted: '/etc/pritunl.conf'

Suggesting that part of the startup script does not have priviliege to chmod the config file. But I am not sure where to look to understand what startup scripts are being run by the service.

The troubleshooting data for my installation appears as such:

sh-4.4$ pritunl version
pritunl v1.32.3805.95
sh-4.4$ printf "dh:\n" && sudo df -h
dh:
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.8G     0  1.8G   0% /dev/shm
tmpfs           1.8G  8.4M  1.8G   1% /run
tmpfs           1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/nvme0n1p1   10G  5.4G  4.7G  54% /
tmpfs           357M     0  357M   0% /run/user/0
sh-4.4$ printf "free:\n" && sudo free -l -m
free:
              total        used        free      shared  buff/cache   available
Mem:           3567         354        2430           8         782        2985
Low:           3567        1136        2430
High:             0           0           0
Swap:             0           0           0
sh-4.4$ printf "pritunl lsof: " && sudo lsof -p `pgrep -x pritunl` | wc -l
pritunl lsof: sudo: lsof: command not found
0
sh-4.4$ printf "pritunl-web lsof: " && sudo lsof -p `pgrep -x pritunl-web` | wc -l
pritunl-web lsof: sudo: lsof: command not found
0
sh-4.4$ printf "pritunl-dns lsof: " && sudo lsof -p `pgrep -x pritunl-dns` | wc -l
pritunl-dns lsof: sudo: lsof: command not found
0
sh-4.4$ printf "pritunl limits:\n" && sudo cat /proc/`pgrep -x pritunl`/limits
pritunl limits:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             14119                14119                processes
Max open files            500000               500000               files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       14119                14119                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
sh-4.4$ printf "pritunl-web limits:\n" && sudo cat /proc/`pgrep -x pritunl-web`/limits
pritunl-web limits:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             14119                14119                processes
Max open files            500000               500000               files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       14119                14119                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
sh-4.4$ printf "pritunl-dns limits:\n" && sudo cat /proc/`pgrep -x pritunl-dns`/limits
pritunl-dns limits:
cat: /proc//limits: No such file or directory
sh-4.4$ printf "limits:\n" && sudo sh -c "ulimit -Hn; ulimit -Sn"
limits:
64000
64000
sh-4.4$ sudo netstat -tulpn | grep pritunl
tcp6       0      0 :::443                  :::*                    LISTEN      1009/pritunl-web
sh-4.4$ printf "pritunl-http:\n" && curl -I http://localhost/check
pritunl-http:
curl: (7) Failed to connect to localhost port 80: Connection refused
sh-4.4$ printf "pritunl-internal:\n" && curl -I http://localhost:9756/check
pritunl-internal:
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8; charset=utf-8
Content-Length: 2
Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Robots-Tag: noindex
Strict-Transport-Security: max-age=31536000; includeSubDomains
Date: Wed, 03 Jul 2024 11:28:49 GMT
Server:

sh-4.4$ printf "pritunl-https:\n" && curl --insecure https://localhost
pritunl-https:
<!doctype html>
<html lang=en>
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to the target URL: <a href="https/localhost/login">https/localhost/login</a>. If not, click the link.

Do you have any idea on how I trace the setup being done by the systemd service to understand what privileges are required?

This may be an SELinux issue. If you are locking files, it will need to be able to set the permission on that file which is done to ensure it is correctly restricted to root only.

This turned out to be a crash in the pritunl logs command itself due to MY privileges being insufficient. The content was not coming from any log files.

Because I was running this under the standard ec2-user, it did not have rights to view the config. I should have been running this as:

sudo pritunl logs