We are trying to build a configuration with 2 Pritunl servers on OCI, accessing on-premises servers through an OCI IPSEC gateway
We need to manage multiple population types and access rules, so we will use multiple VPN ranges ; NAT is disabled on the on-premises range.
VXLAN is enabled on both servers to allow them to share the VPN range, and it works : clients connected on 2 different servers can see each other
OCI route advertisement is configured and works too, however it also publishes the VXLAN range 100.97.162.0/24 ; I’m not sure if this is useful/normal ?
Now the issue : when connected on the server NOT advertised, we can’t open an SSH session to our on-premises server ; however it works OK on the advertised server
I believe it’s an issue of asymmetric routing : on the non-advertised server, packets go out directly through the gateway, but they come back through the advertised server and the VXLAN, causing the issue
To solve it, I added a rule on the non-advertised server to forward all the packets, sent to the on-premises server and originating from the VPN range, to the advertised server through the VXLAN interface
In that way, sent and received packets follow the same routes
Here is what I used (on-premises 192.168.3.0/24, VPN range 171.7.100.0/24) sudo ip route add table 162 192.168.3.0/24 via 100.97.162.1
sudo ip rule add table 162 from 172.17.100.0/24
And it works!
Could you please add an option somewhere to automate this in the route advertisement module ?
The code for this has been added to the repository and will be included in the next release. It will have support for both IPv4 and IPv6. All non-NAT routes will be routed to the server selected for the advertised routes with a routing table for each server. When a failover occurs all tables will be updated to the next selected host.
hi, we are using 1.32.4469.94 and this doesn’t seem to be implemented, no routes are added and if the client connects to the server that doesn’t have the return advertised route, pings don’t work(as expected)
Hm, we’ve apparently hit a regression caused by a changeset seemingly related to the current discussion:
More specifically, after upgrading from pritunl-1.32.4400.99-1.el8.oraclelinux.x86_64 to 1.32.4465.96 or to the latestpritunl-1.34.4681.89-1.el8.oraclelinux.x86_64,
connections from a client on one Pritunl VPN server to a client on another Pritunl VPN server stopped working for some combinations of corresponding Pritunl hosts.
connected to VPN server p-a-1, served by Pritunl host pritunl3
VPN IP 10.99.101.70
VPN client box
connected to VPN server c-1, served by Pritunl host pritunl2
VPN IP 172.28.0.158
Now, for example, if I try to SSH from laptop to box
I get No route to host error.
If I do a traceroute to the same IP, I see an apparent routing loop:
laptop $ traceroute 172.28.0.158
traceroute to 172.28.0.158 (172.28.0.158), 64 hops max, 40 byte packets
1 10.99.101.1 (10.99.101.1) 51.604 ms 43.538 ms 43.862 ms
2 100.97.241.2 (100.97.241.2) 44.221 ms 43.486 ms 46.976 ms
3 100.97.241.3 (100.97.241.3) 45.064 ms 43.677 ms 48.192 ms
4 100.97.241.2 (100.97.241.2) 45.250 ms 44.254 ms 48.094 ms
5 100.97.241.3 (100.97.241.3) 44.742 ms 45.264 ms 46.327 ms
6 * 100.97.241.2 (100.97.241.2) 55.804 ms 52.973 ms
7 100.97.241.3 (100.97.241.3) 45.993 ms 45.953 ms 45.931 ms
8 100.97.241.2 (100.97.241.2) 52.977 ms 46.212 ms 47.832 ms
9 100.97.241.3 (100.97.241.3) 47.931 ms 47.081 ms *
10 100.97.241.2 (100.97.241.2) 133.679 ms 310.067 ms 183.703 ms
11 100.97.241.3 (100.97.241.3) 171.040 ms 177.282 ms 176.706 ms
12 100.97.241.2 (100.97.241.2) 175.573 ms 175.616 ms 176.904 ms
13 100.97.241.3 (100.97.241.3) 175.650 ms 174.899 ms 176.866 ms
14 100.97.241.2 (100.97.241.2) 177.796 ms 176.300 ms *
15 100.97.241.3 (100.97.241.3) 53.884 ms 48.509 ms 48.492 ms
16 100.97.241.2 (100.97.241.2) 49.755 ms 49.516 ms 48.827 ms
<abbreviated>
If I try to SSH again and run tcpdump on pritunl2, I get more confirmations of the routing loop (repeating in and out, decreasing TTL):
[root@pritunl2 ~]# ip route get 172.28.0.158
172.28.0.158 dev tun8 src 172.28.0.1 uid 0
cache
[root@pritunl2 ~]# ip rule | grep 10.99.101.
32758: from 10.99.101.0/24 lookup 106
[root@pritunl2 ~]# ip route list table 106 | grep 172.28
172.28.0.0/16 via 100.97.241.2 dev pxlan241
Routes on pritunl3:
[root@pritunl3 ~]# ip route get 172.28.0.158
172.28.0.158 via 100.97.129.2 dev pxlan129 src 100.97.129.3 uid 0
cache
Manually removing the source routing rules (as in ip rule del from 10.99.101.0/24 lookup 106) helps restore connectivity right away.
I can share more details if needed (IPs, subnets, routes, routing rules, etc.).
We could be doing something wrong,
but things were working pretty well for like 1.5 years in this setup.
With regards to system configuration, we only disable rp_filter to workaround a similar issue with asymmetric routing.
(also had to configure a bit more permissive Security Groups on AWS side since connection tracking there doesn’t work properly due to the same asymmetric routing)
I think one apparent difference from @ituser is that in our case both source and destination IPs live on VPNs, which are managed by 2 different Pritunl VPN servers running on the same set of hosts.
@ituser , did you guys try to disable rp_filter instead of configuring source routing?
Do you have linked servers configured linking those two servers in Pritunl, it likely shouldn’t be for that use case?
Also another option would be to disable VXLan routing in the both the server settings if all the servers are on the same VPC it should have layer 2 on AWS. This would remove the tables and make the routing more optimal.
I will add a server option to make the tables optional in the next release in case there are any configurations that won’t work.
The Pritunl EC2 instances (pritunl1, pritunl2, and pritunl3) live in the same VPC, each in its own Availability Zone and thus in its own subnet.
The VPN server p-a-1 cloud advertises its own IP range (10.99.101.0/24).
It also has a route defined in Pritunl for the IP range of c-1 (172.28.0.0/16),
which gets pushed down to VPN clients.
Similarly, the VPN server c-1 cloud advertises its own IP range (172.28.0.0/16).
And also has a route defined in Pritunl for the IP range of p-a-1 (10.99.101.0/24).
This way VPN clients connected to p-a-1 and to c-1 can talk to each other.
Also another option would be to disable VXLan routing in the both the server settings if all the servers are on the same VPC it should have layer 2 on AWS. This would remove the tables and make the routing more optimal.
Pritunl Link is separate from the server links. The server links would be shown in the servers tab the links would be displayed below the list of hosts in the server.
If you don’t have source/dest checking disabled that can cause issues. If the traffic is only going to VPN clients it would remain on the VXLan but for that configuration there would be not use for the VPC routes. The security group will also need to allow all traffic from the VPN virtual networks for both servers.
And no, we don’t have these configured either.
The servers in question share the same 3 hosts.
The documentation ( Server Links | Pritunl VPN | Pritunl Documentation ) is not very clear on what this feature does or how it works,
but looking at the code,
it seems like it uses openvpn to, well, link multiple disjoint servers together.
As far as I can tell, the code has checks preventing creation of links between servers running on the same host,
so it thankfully wouldn’t have allowed us to misconfigure this anyway.
Sorry for the confusion.
I’ve forgot that we actually have source/dest checks disabled already indeed.
And yeah, you’re right that VPN clients won’t be able to talk to resources in the VPC without source/dest checks being disabled given that we don’t have NAT either.
Thinking about this a bit more,
I suppose replicated servers and VXLAN stuff required disabled (rp_filter=0) or loose (rp_filter=2) Reverse Path Filter anyway,
so this couldn’t have been be the culprit.
What could have been the problem though are security groups (i.e. firewall) and their connection tracking on OCI side.
As @ituser pointed out, due to asymmetric routing, return packets follow a different path and can’t be correlated to an original flow.
And thus, they are seen as an attempt of the on-premise server to establish a new connection to some high random port number on the Pritunl hosts and thus get blocked by the security group(s).
A global server option should certainly work great for us.
However, I now think that a per-route option could be a bit more flexible.
This way, we’d be able to disable ip rule creation for routes to other VPN subnets
(since they cause the forwarding loops) and to enable ip rule creation for routes towards VPC and on-prem resources (and then use more strict security groups on AWS side).
