Sudden licensing or subscription issues?

Hi All,

All of our premium instances have suddenly become unlicensed and the subscription page is showing a blank screen/no details. Nothing was changed on these servers, and restarts etc have proven ineffective in correcting the issue. Any advice? Logs also don’t show any relevant errors that would indicate anything.

Screenshot from 2023-03-24 17-03-22

I’m having the same issue with 2 of my servers hosted on Linode!

Ours are on AWS. We’ve also tried to temporarily drop the firewalls in relevant places to see if that was the issue. No luck at all. So looks like it’s not a cloud specific issue.

I even rebuilt my servers from scratch - same issue

Same for our infrastructure. All premium features was gone

We’re experiencing the same behaviour. All enterprise features are disabled.

I’m also having the same issue. The subscription information screen is blank and features are restricted.

We have the same problem, the subscription dialog is blank and the Yubikey authentication does not work.

The database had an issue causing the subscription CSS styles which are loaded from the server to not to load correctly. These are cached so you may need to click Subscription in the top right to refresh the server then reload the page a few times to correct the issue.

1 Like

Thanks for the update Zach. Now all is working a treat. Thank you :slight_smile:. However, I did have one question, assuming you have a post-mortem for this issue. Is there anything planned to mitigate this issue in future? Specifically, I’m thinking is there some way to store license details locally on a given set of servers and if license authentication fails for one reason or another it retries for X number of days before failing entirely? Perhaps some kind of alert in the GUI or logs that indicate a license auth failure? (Ok that’s 2 questions sorry!).

Whilst we are merely only a premium subscriber right now, I have been on enterprise in the past at another org (and loved it) and we are looking to upgrade to enterprise in my current org. But I need an answer to the powers that be as to what will prevent this issue in future, since as you can imagine this can be critical infrastructure depending on it’s use case.

This wasn’t an issue with the subscription state on the server, there are alerts that I have configured if an issue occurs with that to fix it quickly. To protect the subscription features some of the CSS styles for features in the subscription are downloaded from the subscription server. This is then loaded by the web browser. The format of this was changed for the upcoming v1.32 release and this change was inadvertently applied to some stable releases. The stable releases couldn’t load the new CSS styles which caused the issues. This would only be noticed when logging into the administrator web console. The subscription state on the server was uneffected. All other features continued to function including non-administrator users authenticating with single sign-on in the web console. Because the impact was limited it went unnoticed for a while.

I see, this makes more sense. Given what you said about subscription state of the servers remaining untouched/still working. Would this extend to TOTP Auth?

The reason I mention this is during the outage window, we had users that couldn’t authenticate with 2fa on clients. We had to give them a pin to allow them to login to get around the issue. These pins have been revoked now given the issue is resolved and they were able to login again with their existing tokens. I would chalk this up to user error on their end, except these are users that have historically not had issues logging in previously using 2fa. Could this be a potential edge case? Again thank you for taking the time to follow up on this. It’s much appreciated :slight_smile:

If you’re referring to Google Authenticator that feature is available in the free version. Any issues on the server would either be unrelated or caused by attempting to fix the web console issue. This CSS file is only loaded after logging into the administrator web console. The login page, all the single sign-on page and user profile pages do not load any CSS files. If a license was removed by an administrator in an attempt to fix the display issue that would have disable the subscription features.

The most common issue with the Google Authenticator is a time mismatch (timezone does not effect it) between the server and client authenticator device. Both use the time to calculate the passcode. There is a 90 second window for the authenticator codes.