Incident on Sept 4th

[Lauren]

Some of you may notice that our license server was down on the night of Sept 4th. It was due to hard disk failure and the server was unable to reboot.

We worked the whole night to restore the service to another server. License service was restored on 9/5 2:30am EDT.

Usually the LSWS/LSLB should not be affected if it cannot connect to the license server. However when we were restoring the service, one script was not migrated over, ( this is human error! at 2am in the morning during labor day long weekend) and it caused bad data returned. We found out the problem in half an hour. But if a server happened to access license server at that time, it will fail and stop working. We also lost the data for new sign ups on that day as the database backup was from prior day. We manually recovered all the data, but still required clients to register a new license key.

Most of our clients are not affected or aware of this incident. (I'm surprised that no one post on this.) For those who did get affected, we sincerely apologize for it.

Most affected servers were recovered by their own support staff, we can tell from the log that most affected servers got good license key on 9/5 morning.

If your server still have problem and cannot be brought up by restarting. You can go to client area->my products->product detail->release license.
then register a new key by going to installed directory like /usr/local/lsws/bin
run ./lshttpd -r

We have resolved all the issues reported by email. If you still have issues, please create a ticket and we'll treat as first priority.

Lessons learned:
1. We'll improve our code, license system error should not affect LSWS or LSLB.
2. We'll improve our infrastructure to have HA set up on the main site and DR fail-over.

We are sorry for the trouble and will work harder to improve our products and services.

Lauren