Can't get replicated setup to work

apologies for the long post, but I wanted to capture as much info from our setup as possible

We’re trying to setup a replicated environment and having some issues getting it work

2x AWS ec2 instances running Rocky 8.10 for the pritunl hosts running pritunl-1.32.4057.36-1.el8.oraclelinux.x86_64.rpm

  • the instances have an IAM role attached that grants them the AmazonVPCFullAccess policy
  • the instances are on the same subnet to eliminate any cross subnet routing issues for now
  • security groups are wide open for testing (allow access to all ports tcp/udp from anywhere)
  • source/destination check is disabled for both instances
  • selinux is disabled at the OS (for testing purposes)
  • firewalld / iptables services are not installed

in the pritunl UI, each host has the following configured:

  • Public Address
  • Local Address
  • Sync Address
    • this is a domain that resolves to both public IPs of the pritunl hosts and is the same domain that the clients use to connect to pritunl

the pritunl vpn server is configured as follows:

  • Replication Count: 2
  • VXLan Routing checked
  • Inter-Client Routing checked
  • the route for the VPN server’s network has Cloud Route Advertise enabled
  • all routes are non-NAT

The two hosts are attached to the VPN Server.
The VPN server is started and the two hosts come online.

The VPN client is started and attaches to one of the two hosts (repeated connects does show it will use one or the other so that works).

However:
if the AWS routes managed by pritunl are over pritunl-host1’s ENI and the pritunl client is connected to pritunl-host1 all destinations are reachable EXCEPT being to (e.g.) ssh to pritun-host2 via private ip (the vpn server has a route to the subnet for the hosts)

if the AWS routes managed by pritunl are over pritunl-host1’s ENI and the pritunl client is connected to pritunl-host2 all destinations are blocked EXCEPT being able to ssh to priutnl-host2

We’re stumped as to why this happening - any thoughts?

ah, we found the issue - net.ipv4.conf.all.rp_filter was enabled (even though all the others, default and interface specific ones, were disabled