Affecting Other - allegiance.wa-us.vmho.st

• 06/11/2025 18:29 - 06/11/2025 00:00
• Last Updated 06/12/2025 04:34

The "Allegiance" node in our Liberty Lake, Washington location is undergoing hardware maintenance to replace RAM in the machine. This should not take more than 30 minutes.

EDIT: Datacenter staff promptly replaced the RAM, but the server is failing to boot properly. Diagnosing this new, unexpected and unplanned issue now.

---

Unfortunately, the situation is more serious and what was supposed to be a simple, quick window of maintenance has turned into something severe.

The VM host node, "Allegiance" was originally provisioned with only 128GB of RAM, whereas our other hostnodes are provisioned with 256GB of RAM. With all other server specs for new deployments being 1:1 matches for consistency, we decided to take this server offline to perform a quick maintenance to upgrade the RAM so that this server could operate to the same capacity as others. The server, otherwise, would be under utilized and only capable of serving half it's potential.

After the maintenance was completed, the server failed to boot. Upon review, it appears to have been an issue with the NVMe RAID array (2 disk, RAID-1). What we believe has happened is that sometime between initial deployment of the hardware node in March of 2025 and recent time is that one of the two NVMe drives died. All components were tested and working as expected upon deployment. It's believed that the second drive was beginning to suffer a similar fate, where the data was intact and there but failed to re-initiatlize on boot.

Data restoration is currently being attempted, but I've been given a low estimate of recovery by the techs working on this.

Since we've been in business, we've never had a data loss event like this, so this is a first for us. We're working hard on recovery, but failing that, we will extend all services by a month (You get a free month) after reprovisioning your account. If you created manual snapshots, these are stored offsite and should be recoverable.

Updates will follow here.