You would need to remove the Google Admin Email and Google JSON Private Key from the top right settings. This will use only OAuth to authenticate the user. When using the Google Workspace API only one workspace can be used. It would also be best to configure device authentication with this configuration.
You would need to use the groups mode to do that and it would only work with one Google Workspace. To do this delete all the organizations and create one organization. Set this organization as the default single sign-on organization in the top right settings then attach the organization to all servers. Then run the commands below. In each of the server settings add the groups that will be able to access that server. This can result in larger usage of IP address pools. Every user that is attached to a server will have a static IP assigned even if a group is not matched. The server virtual network subnet size should allow for this. For SAML the attribute groups is used to set a comma separated list of groups.
sudo pritunl set app.sso_azure_mode '"groups"'
sudo pritunl set app.sso_authzero_mode '"groups"'
sudo pritunl set app.sso_google_mode '"groups"'