Status update: 09/18/2020
Recently, we experienced a catastrophic failure that has resulted in
loss of data. Our primary data storage system experienced a 'double fault'
condition as a result of two physically malfunctioning hard drives within a
storage pool of 4, resulting in that data pool becomming completely
inaccessible. This failed pool held certain criticial data including hosted
websites, and internal services such as our billing systems, and the virtual
machine images that run some of our public facing services, as well as
backups and snapshots.
Here are the details of the events:
9/5/2020 – One disk in the storage pool failed. Due to redundancies, dual controllers and other high availability design features, all data remained online and accessible. A replacement disk is scheduled for installation.
9/8/2020 – Another drive fails, and unfortunately, it happens to be exactly the wrong disk out of the remaining set, resulting in that storage pool becomming inaccessible, despite all advance precautions.
Data recovery was attempted with a recovery service that was prepared to repair the disks and the electronics, but the damage to these was greater than anticipated and the recovery effort was unsuccessful. A further recovery effort has been initiated by the drive manufacturer itself now which has taken the disks to their recovery lab in Ohio. We have been promised a 15 day turn around, while not as fast as we would like, it is the last line of recovery possible.
Furthermore, we also determined that our off-site cloud backup provider, also failed, leaving us without our last line of defense. Due to uncommunicated policy changes and other issues, our cloud backup provider ceased backing up certain kinds of data apparently without notice, and further, retroactively removed the previously backed up data in conformance with their revised policies, resulting in a complete and total loss due to omissions in the backups of the backups.
We have scrambled to restore the services that we can, and email and dns continues to function normally. However, customer websites are still offline and may not be recoverable, as well as our former webmail service. We understand this hardship and are continuing to work to recover this data if possible, however, it may be some weeks if at all. We are reccomending at this time that customers move their web hosting and we will assist you in changing your dns or other settings as necessary. If and when a final recovery is made, we will bring that data online and make it accessible to our customers.
Due to the loss of billing data, in the comming weeks, we anticipate re-enrolling our current customers and we will be contacting you to verify your services and billing details. We will post an update to this regarding the process as it becomes available.