ElasticSearch document
What is the link of ElasticSearch setup for pritunl zero?
we have ELK server ready. Need to understand how Pritul will call ELK and what setup is required on ELK side and if there any script or setup setup ?
The server https address with port needs to be set in the settings. If authentication is configured the username and password also need to be set. Pritunl Zero uses the OpenSearch Go Library this should work with the latest ElasticSearch release. Below is the OpenSearch configuration used for testing.
#################################
# setup
#################################
sudo dnf -y update
sudo dnf -y install dnf-automatic
sudo sed -i 's/^upgrade_type =.*/upgrade_type = default/g' /etc/dnf/automatic.conf
sudo sed -i 's/^download_updates =.*/download_updates = yes/g' /etc/dnf/automatic.conf
sudo sed -i 's/^apply_updates =.*/apply_updates = yes/g' /etc/dnf/automatic.conf
sudo systemctl enable --now dnf-automatic.timer
sudo tee /etc/security/limits.conf << EOF
* hard nofile 500000
* soft nofile 500000
root hard nofile 500000
root soft nofile 500000
EOF
sudo tee /etc/sysctl.d/10-maxmap.conf << EOF
vm.max_map_count=262144
EOF
sudo sysctl -w vm.max_map_count=262144
#################################
# gpg
#################################
wget https://artifacts.opensearch.org/publickeys/opensearch.pgp
gpg --import opensearch.pgp
rm opensearch.pgp
gpg --edit-key C5B7498965EFD1C2924BA9D539D319879310D3FC
trust
5
quit
#################################
# opensearch
#################################
wget https://artifacts.opensearch.org/releases/bundle/opensearch/1.3.2/opensearch-1.3.2-linux-x64.tar.gz
wget https://artifacts.opensearch.org/releases/bundle/opensearch/1.3.2/opensearch-1.3.2-linux-x64.tar.gz.sig
gpg --verify opensearch-1.3.2-linux-x64.tar.gz.sig opensearch-1.3.2-linux-x64.tar.gz
sudo useradd opensearch
sudo mkdir /var/lib/opensearch
sudo mkdir /var/log/opensearch
sudo chown opensearch:opensearch /var/lib/opensearch
sudo chown opensearch:opensearch /var/log/opensearch
sudo systemctl stop opensearch
sudo systemctl stop opensearch-dashboards
tar xf opensearch-1.3.2-linux-x64.tar.gz
sudo rm -rf /usr/local/opensearch-orig
sudo mv /usr/local/opensearch /usr/local/opensearch-orig
sudo mv opensearch-1.3.2 /usr/local/opensearch
sudo chown -R opensearch:opensearch /usr/local/opensearch
sudo sed -i 's/^-Xms.*/-Xms9g/g' /usr/local/opensearch/config/jvm.options
sudo sed -i 's/^-Xmx.*/-Xmx9g/g' /usr/local/opensearch/config/jvm.options
sudo sed -i 's/^ reserved:.*/ reserved: false/g' /usr/local/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml
sudo sed -i 's/^ reserved:.*/ reserved: false/g' /usr/local/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml
sudo -u opensearch -g opensearch /usr/local/opensearch/opensearch-tar-install.sh -Ecluster.name=silicon.red -Enode.name=opensearch1 -Ehttp.host=0.0.0.0 -Ediscovery.type=single-node -Epath.data=/var/lib/opensearch -Epath.logs=/var/log/opensearch
sudo tee /etc/systemd/system/opensearch.service << EOF
[Unit]
Description=Opensearch
[Service]
User=opensearch
Group=opensearch
WorkingDirectory=/usr/local/opensearch
ExecStart=/bin/bash /usr/local/opensearch/opensearch-tar-install.sh -Ecluster.name=silicon.red -Enode.name=opensearch1 -Ehttp.host=0.0.0.0 -Ediscovery.type=single-node -Epath.data=/var/lib/opensearch -Epath.logs=/var/log/opensearch
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable opensearch
sudo systemctl restart opensearch
curl -XGET --insecure -u 'admin:admin' 'https://localhost:9200'
#################################
# opensearch auth
#################################
sudo -u opensearch -g opensearch JAVA_HOME=/usr/local/opensearch/jdk sh /usr/local/opensearch/plugins/opensearch-security/tools/hash.sh
sudo -u opensearch -g opensearch JAVA_HOME=/usr/local/opensearch/jdk /usr/local/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cd "/usr/local/opensearch/plugins/opensearch-security/securityconfig" -icl -key "/usr/local/opensearch/config/kirk-key.pem" -cert "/usr/local/opensearch/config/kirk.pem" -cacert "/usr/local/opensearch/config/root-ca.pem" -
#################################
# opensearch dashboard
#################################
wget https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/1.3.2/opensearch-dashboards-1.3.2-linux-x64.tar.gz
wget https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/1.3.2/opensearch-dashboards-1.3.2-linux-x64.tar.gz.sig
gpg --verify opensearch-dashboards-1.3.2-linux-x64.tar.gz.sig opensearch-dashboards-1.3.2-linux-x64.tar.gz
sudo systemctl stop opensearch
sudo systemctl stop opensearch-dashboards
tar xf opensearch-dashboards-1.3.2-linux-x64.tar.gz
sudo rm -rf /usr/local/opensearch-dashboards-orig
sudo mv /usr/local/opensearch-dashboards /usr/local/opensearch-dashboards-orig
sudo mv opensearch-dashboards-1.3.2 /usr/local/opensearch-dashboards
sudo chown -R opensearch:opensearch /usr/local/opensearch-dashboards
sudo sed -i 's/^opensearch.password:.*/opensearch.password: "####kibanaserver_password####"/g' /usr/local/opensearch-dashboards/config/opensearch_dashboards.yml
sudo -u opensearch -g opensearch /usr/local/opensearch-dashboards/bin/opensearch-dashboards -H 0.0.0.0
sudo tee /etc/systemd/system/opensearch-dashboards.service << EOF
[Unit]
Description=Opensearch Dashboards
[Service]
User=opensearch
Group=opensearch
WorkingDirectory=/usr/local/opensearch-dashboards
ExecStart=/bin/sh /usr/local/opensearch-dashboards/bin/opensearch-dashboards -H 0.0.0.0
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable opensearch-dashboards
sudo systemctl restart opensearch-dashboards
Getting below error. The VM memory is 16 GB
-XX:MaxDirectMemorySize=4831838208, -Dopensearch.path.home=/usr/local/opensearch, -Dopensearch.path.conf=/usr/local/opensearch/config, -Dopensearch.distribution
.type=tar, -Dopensearch.bundled_jdk=true]
[2022-05-19T19:06:37,463][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin opensearch-security due to jar hell
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:182) ~[opensearch-1.3.2.jar:1.3.2]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169) ~[opensearch-1.3.2.jar:1.3.2]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100) ~[opensearch-1.3.2.jar:1.3.2]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-1.3.2.jar:1.3.2]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-1.3.2.jar:1.3.2]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135) ~[opensearch-1.3.2.jar:1.3.2]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101) ~[opensearch-1.3.2.jar:1.3.2]
Caused by: java.lang.IllegalStateException: failed to load plugin opensearch-security due to jar hell
The errors could be caused by an incorrect installation.
Adjust the memory allocation for the JVM if the 9gb in the example is too much.
sudo sed -i 's/^-Xms.*/-Xms2g/g' /usr/local/opensearch/config/jvm.options
sudo sed -i 's/^-Xmx.*/-Xmx2g/g' /usr/local/opensearch/config/jvm.options
Opensearch is runnung but still getting below error in the logs
search: Failed to check elastic index
cannot retrieve information from OpenSearch
ORIGINAL STACK TRACE:
Search · · GitHub).CreateIndex
Below are the opensearch status
[ztadmin@pritunlvm ~]$ sudo systemctl status opensearch-dashboards
● opensearch-dashboards.service - Opensearch Dashboards
Loaded: loaded (/etc/systemd/system/opensearch-dashboards.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2022-05-20 01:39:30 UTC; 6min ago
Main PID: 18659 (node)
Tasks: 11
Memory: 102.0M
CGroup: /system.slice/opensearch-dashboards.service
└─18659 /usr/local/opensearch-dashboards/bin/…/node/bin/node /usr/local/opensearch-dashboards/bin/…/src/cli/dist -H 0.0.0.0
May 20 01:45:12 pritunlvm sh[18
we are getting below error still
search: Failed to check elastic index
cannot retrieve information from OpenSearch
ORIGINAL STACK TRACE:
Search · · GitHub).CreateIndex
/go/src/github.com/pritunl/pritunl-zero/search/index.go:96 +0xf467f9
Search · · GitHub).UpdateIndexes
Openserarch is working as per instructions
[2022-05-24T14:04:58,503][INFO ][o.o.s.s.ConfigHelper ] [opensearch1] Index .opendistro_security already contains doc with id whitelist, skipping update.
[2022-05-24T14:04:58,504][INFO ][o.o.s.s.ConfigHelper ] [opensearch1] Will update ‘audit’ with /usr/local/opensearch/plugins/opensearch-security/securitycon
fig/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2022-05-24T14:04:58,521][INFO ][o.o.s.s.ConfigHelper ] [opensearch1] Index .opendistro_security already contains doc with id audit, skipping update.
[2022-05-24T14:04:58,679][INFO ][stdout ] [opensearch1] [FINE] No subscribers registered for event class org.opensearch.security.securityconf.
DynamicConfigFactory$NodesDnModelImpl
[2022-05-24T14:04:58,680][INFO ][stdout ] [opensearch1] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberE
vent
[2022-05-24T14:04:58,680][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Auditing on REST API is enabled.
[2022-05-24T14:04:58,680][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
[2022-05-24T14:04:58,680][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Auditing on Transport API is enabled.
[2022-05-24T14:04:58,681][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
[2022-05-24T14:04:58,681][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Auditing of request body is enabled.
[2022-05-24T14:04:58,681][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Bulk requests resolution is disabled during request auditing.
[2022-05-24T14:04:58,681][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Index resolution is enabled during request auditing.
[2022-05-24T14:04:58,681][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Sensitive headers auditing is enabled.
[2022-05-24T14:04:58,681][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Auditing requests from kibanaserver users is disabled.
[2022-05-24T14:04:58,684][WARN ][o.o.s.a.r.AuditMessageRouter] [opensearch1] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGE
S, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTER
NAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2022-05-24T14:04:58,684][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Auditing of external configuration is disabled.
[2022-05-24T14:04:58,684][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Auditing of internal configuration is enabled.
[2022-05-24T14:04:58,685][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Auditing only metadata information for read request is enabled.
[2022-05-24T14:04:58,685][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Auditing will watch {} for read requests.
[2022-05-24T14:04:58,685][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Auditing read operation requests from kibanaserver users is disabled.
[2022-05-24T14:04:58,685][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Auditing only metadata information for write request is enabled.
[2022-05-24T14:04:58,685][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Auditing diffs for write requests is disabled.
[2022-05-24T14:04:58,685][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Auditing write operation requests from kibanaserver users is disabled.
[2022-05-24T14:04:58,685][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Auditing will watch for write requests.
[2022-05-24T14:04:58,686][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] .opendistro_security is used as internal security index.
[2022-05-24T14:04:58,686][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch1] Internal index used for posting audit logs is null
[2022-05-24T14:04:58,686][INFO ][o.o.s.c.ConfigurationRepository] [opensearch1] Hot-reloading of audit configuration is enabled
[2022-05-24T14:04:58,686][INFO ][o.o.s.c.ConfigurationRepository] [opensearch1] Node ‘opensearch1’ initialized
[2022-05-24T14:05:57,848][INFO ][o.o.i.i.ManagedIndexCoordinator] [opensearch1] Performing move cluster state metadata.
[2022-05-24T14:05:57,850][INFO ][o.o.i.i.MetadataService ] [opensearch1] ISM config index not exist, so we cancel the metadata migration job.
[2022-05-24T14:06:57,849][INFO ][o.o.i.i.ManagedIndexCoordinator] [opensearch1] Cancel background move metadata process.
[2022-05-24T14:06:57,851][INFO ][o.o.i.i.ManagedIndexCoordinator] [opensearch1] Performing move cluster state metadata.
[2022-05-24T14:06:57,851][INFO ][o.o.i.i.MetadataService ] [opensearch1] Move metadata has finished.
[2022-05-24T14:09:57,687][INFO ][o.o.j.s.JobSweeper ] [opensearch1] Running full sweep
after some more updates I am getting below error now
9:09:36 AM
error
search: Failed to create elastic index
response: “{“statusCode”:404,“error”:“Not Found”,“message”:“Not Found”}”
status_code: 404
9:09:36 AM
info
search: Create index
index: “zero-requests-2022-05-25”
what is missing?