On MacOS, when the client is connected to the server in WireGuard mode and disconnects, sometimes tainted DNS configuration remains on the client’s host. Depending on the configuration that remains, this can prevent the host from resolving any DNS query while disconnected from the VPN. Connecting in OpenVPN mode does not suffer from the same issue.
Here’s how I reproduce the issue and what I see in scutil and networksetup output.
How to reproduce
I’ve done my best to document a method for reproducing the issue we are seeing.
The method doesn’t work every time, I suspect because the root cause of the
issue is a race condition.
0. Initial state
Disconnected from VPN. Connected to WiFi. All client settings are at the default
(all toggles are off).
$ networksetup -getdnsservers Wi-Fi
There aren't any DNS Servers set on Wi-Fi.
$ sudo scutil --dns
DNS configuration
resolver #1
search domain[0] : home
nameserver[0] : 2a01:cb00:534:c600:d6f8:29ff:fe44:8bf0
nameserver[1] : 192.168.1.1
if_index : 14 (en0)
flags : Request A records, Request AAAA records
reach : 0x00020002 (Reachable,Directly Reachable Address)
Removed irrelevant resolvers (.local and .arpa domains).
DNS configuration (for scoped queries)
resolver #1
search domain[0] : home
nameserver[0] : 2a01:cb00:534:c600:d6f8:29ff:fe44:8bf0
nameserver[1] : 192.168.1.1
if_index : 14 (en0)
flags : Scoped, Request A records, Request AAAA records
reach : 0x00020002 (Reachable,Directly Reachable Address)
$ sudo scutil
> open
> show State:/Network/Global/DNS
<dictionary> {
SearchDomains : <array> {
0 : home
}
ServerAddresses : <array> {
0 : 2a01:cb00:534:c600:d6f8:29ff:fe44:8bf0
1 : 192.168.1.1
}
__CONFIGURATION_ID__ : Default: 0
__FLAGS__ : 6
__IF_INDEX__ : 14
__ORDER__ : 0
}
show State:/Network/Global/IPv4
<dictionary> {
PrimaryInterface : en0
PrimaryService : 1692E22E-ADBB-4FB7-8349-72BE1809C202
Router : 192.168.1.1
}
> show State:/Network/Service/1692E22E-ADBB-4FB7-8349-72BE1809C202/DNS
<dictionary> {
DomainName : home
SearchDomains : <array> {
0 : home
}
ServerAddresses : <array> {
0 : 192.168.1.1
1 : 2a01:cb00:534:c600:d6f8:29ff:fe44:8bf0
}
}
> list .*Pritunl.*
no keys.
> quit
1. Connected to VPN
Connected to VPN in WireGuard mode. Connected to WiFi. DNS resolution works as expected.
$ networksetup -getdnsservers Wi-Fi
10.254.0.7
10.254.1.9
10.254.2.12
8.8.8.8
$ sudo scutil --dns
DNS configuration
resolver #1
search domain[0] : home
nameserver[0] : 10.254.0.7
nameserver[1] : 10.254.1.9
nameserver[2] : 10.254.2.12
nameserver[3] : 8.8.8.8
flags : Request A records, Request AAAA records
reach : 0x00000002 (Reachable)
Removed irrelevant resolvers (.local and .arpa domains).
DNS configuration (for scoped queries)
resolver #1
search domain[0] : home
nameserver[0] : 10.254.0.7
nameserver[1] : 10.254.1.9
nameserver[2] : 10.254.2.12
nameserver[3] : 8.8.8.8
if_index : 14 (en0)
flags : Scoped, Request A records, Request AAAA records
reach : 0x00000002 (Reachable)
$ sudo scutil
> open
> show State:/Network/Global/DNS
<dictionary> {
SearchDomains : <array> {
0 : home
}
ServerAddresses : <array> {
0 : 10.254.0.7
1 : 10.254.1.9
2 : 10.254.2.12
3 : 8.8.8.8
}
__CONFIGURATION_ID__ : Default: 0
__FLAGS__ : 6
__ORDER__ : 0
}
> show State:/Network/Service/1692E22E-ADBB-4FB7-8349-72BE1809C202/DNS
<dictionary> {
DomainName : home
SearchDomains : <array> {
0 : home
}
ServerAddresses : <array> {
0 : 192.168.1.1
1 : 2a01:cb00:534:c600:d6f8:29ff:fe44:8bf0
}
}
> show Setup:/Network/Service/1692E22E-ADBB-4FB7-8349-72BE1809C202/DNS
<dictionary> {
ServerAddresses : <array> {
0 : 10.254.0.7
1 : 10.254.1.9
2 : 10.254.2.12
3 : 8.8.8.8
}
}
> list .*Pritunl.*
no keys.
> quit
2. Disconnected from VPN uncleanly
We noticed the issue happened often when users closed their laptop and went home. We found that we could reproduce the issue by toggling WiFi and the VPN connection in quick succession. We eventually end up in a state where the client is disconnected from the VPN, but the VPN’s DNS configuration remains.
The Pritunl service attempts to restore DNS (watch DNS is enabled on the client) to what it was before but fails. It looks like this is because the State:/Network/Global/IPv4 scutil key does not exist when the host is disconnected from WiFi. The service tries twice and then falls back to “Reseting networking”, which doesn’t solve the issue.
At this point, DNS resolution is broken. How hard it’s broken depends on the DNS servers that were configured by the VPN. If they’re not reachable, DNS resolution will fail. If they’re reachable, DNS resolution may work.
$ networksetup -getdnsservers Wi-Fi
10.254.0.7
10.254.1.9
10.254.2.12
8.8.8.8
$ sudo scutil --dns
DNS configuration
resolver #1
search domain[0] : home
nameserver[0] : 10.254.0.7
nameserver[1] : 10.254.1.9
nameserver[2] : 10.254.2.12
nameserver[3] : 8.8.8.8
flags : Request A records, Request AAAA records
reach : 0x00000002 (Reachable)
Removed irrelevant resolvers (.local and .arpa domains).
DNS configuration (for scoped queries)
resolver #1
search domain[0] : home
nameserver[0] : 10.254.0.7
nameserver[1] : 10.254.1.9
nameserver[2] : 10.254.2.12
nameserver[3] : 8.8.8.8
if_index : 14 (en0)
flags : Scoped, Request A records, Request AAAA records
reach : 0x00000002 (Reachable)
$ sudo scutil
> open
> show State:/Network/Global/DNS
<dictionary> {
SearchDomains : <array> {
0 : home
}
ServerAddresses : <array> {
0 : 10.254.0.7
1 : 10.254.1.9
2 : 10.254.2.12
3 : 8.8.8.8
}
__CONFIGURATION_ID__ : Default: 0
__FLAGS__ : 6
__ORDER__ : 0
}
> show State:/Network/Service/1692E22E-ADBB-4FB7-8349-72BE1809C202/DNS
<dictionary> {
DomainName : home
SearchDomains : <array> {
0 : home
}
ServerAddresses : <array> {
0 : 192.168.1.1
1 : 2a01:cb00:534:c600:d6f8:29ff:fe44:8bf0
}
}
> show Setup:/Network/Service/1692E22E-ADBB-4FB7-8349-72BE1809C202/DNS
<dictionary> {
ServerAddresses : <array> {
0 : 10.254.0.7
1 : 10.254.1.9
2 : 10.254.2.12
3 : 8.8.8.8
}
}
> list .*Pritunl.*
no keys.
> quit
Investigation
I did my best to investigate the issue. Here is what I found.
The Pritunl service’s logs:
[2023-12-04 13:14:18][INFO] ▶ profile: Disconnecting ◆ profile_id="bf4fc47f6dd108b5"
[2023-12-04 13:14:18][INFO] ▶ profile: Disconnected ◆ profile_id="bf4fc47f6dd108b5"
[2023-12-04 13:14:20][INFO] ▶ utils: Restore DNS
[2023-12-04 13:14:43][ERRO] ▶ utils: Failed to find primary service from scutil ◆ output="No such key"
[2023-12-04 13:14:43][WARN] ▶ watch: Failed to restore DNS
utils: Failed to find primary service from scutil
ORIGINAL STACK TRACE:
github.com/pritunl/pritunl-client-electron/service/utils.GetScutilService
/Users/apple/go/src/github.com/pritunl/pritunl-client-electron/service/utils/utils.go:343 +0x102689de0
github.com/pritunl/pritunl-client-electron/service/utils.RestoreScutilDns
/Users/apple/go/src/github.com/pritunl/pritunl-client-electron/service/utils/utils.go:365 +0x102689f23
github.com/pritunl/pritunl-client-electron/service/watch.dnsWatch
/Users/apple/go/src/github.com/pritunl/pritunl-client-electron/service/watch/watch.go:162 +0x10276b317
runtime.goexit
/opt/homebrew/Cellar/go/1.21.3/libexec/src/runtime/asm_arm64.s:1197 +0x102238593
[2023-12-04 13:14:48][INFO] ▶ utils: Restore DNS
[2023-12-04 13:15:11][ERRO] ▶ utils: Failed to find primary service from scutil ◆ output="No such key"
[2023-12-04 13:15:11][ERRO] ▶ watch: Failed to restore DNS, resetting network
utils: Failed to find primary service from scutil
ORIGINAL STACK TRACE:
github.com/pritunl/pritunl-client-electron/service/utils.GetScutilService
/Users/apple/go/src/github.com/pritunl/pritunl-client-electron/service/utils/utils.go:343 +0x102689de0
github.com/pritunl/pritunl-client-electron/service/utils.RestoreScutilDns
/Users/apple/go/src/github.com/pritunl/pritunl-client-electron/service/utils/utils.go:365 +0x102689f23
github.com/pritunl/pritunl-client-electron/service/watch.dnsWatch
/Users/apple/go/src/github.com/pritunl/pritunl-client-electron/service/watch/watch.go:162 +0x10276b317
runtime.goexit
/opt/homebrew/Cellar/go/1.21.3/libexec/src/runtime/asm_arm64.s:1197 +0x102238593
[2023-12-04 13:15:11][INFO] ▶ utils: Reseting networking
Shutting Pritunl down, uninstalling it, and rebooting the host do not resolve the issue.
Using the client’s “Reset networking” button resets the DNS configuration to how it was before the issue happened. This lead me to the client’s utils.ClearDns function. The function uses networksetup to clear DNS from all network devices. I discovered that when the client disconnects from the VPN uncleanly, some network services keep the DNS configuration set by the client:
# Only a sample of the network services on the host.
$ networksetup -getdnsservers Wi-Fi
10.254.0.7
10.254.1.9
10.254.2.12
8.8.8.8
$ networksetup -getdnsservers "USB 10/100 LAN"
10.254.0.7
10.254.1.9
10.254.2.12
8.8.8.8
$ networksetup -getdnsservers "Thunderbolt Bridge"
There aren't any DNS Servers set on Thunderbolt Bridge.
When the VPN disconnects, it usually resets the DNS servers of all network services. When it does not, the DNS servers left behind lead to the corrupt DNS configuration we observe.
There seems to be a race condition somewhere. I tried to look at the client code to understand where, but I couldn’t figure out how the client handled DNS differently for OpenVPN and WireGuard modes.
Once the issue has happened, Pritunl can interract with the tainted DNS configuration when using other modes or profiles. This leads to very weird network behavior. For instance, if connected to another profile in OpenVPN mode, Pritunl’s “DNS Watch” feature will fight with the system and the host will use the corrupt DNS settings for a few seconds before Pritunl changes them back. Pritunl will also store the corrupt DNS settings in its Restore scutil key, so it will put the settings back when the user disconnects.
May be related to [DNS setting stays on WiFi after disconnect Mac Client](DNS setting stays on WiFi after disconnect Mac Client).