• 05/25/2023 14:53
• Last Updated 05/25/2023 15:53

The NL-01 VPS node in the Netherlands is currently down. We are working with the datacenter to bring it back as it's not responding to remote commands. Internal monitoring doesn't indicate a hardware failure of any sort at this time, however we will continue to investigate and will bring this back up as soon as possible.

Update 1: Internal monitoring showed a large spike in CPU temperature right before the server went offline. The datacenter is reviewing it now and will take appropriate action. More updates to come as they're known to us.

Update 2: Datacenter states the CPU fan has failed and are replacing it. Should be resolved soon.

Update 3: The datacenter has replaced the CPU cooler, the VPS node has been booted back up and is now once again reachable. We'll continue to monitor for any signs of potential failure or overheating.

Affecting Other - CDA Network

• 05/16/2023 19:36 - 05/17/2023 07:09
• Last Updated 05/16/2023 19:37

We are aware of a service impacting issue in our USA location in Idaho and are investigating and reviewing it.

Affecting System - VPS Control Panel

• 04/01/2023 08:11 - 04/01/2023 21:39
• Last Updated 04/01/2023 21:39

We're aware of impossibly slow OS installs / reinstalls for most VPS nodes. OS templates are stored on the master server which is outside our network, with a 3rd party, and we're experiencing pretty terrible network conditions right now. This may delay or prevent OS reinstalls for VPS customers

We host some of our public facing infrastructure outside of our own network so that not all eggs are in the same basket. Things like our website mirrors, portal, VPS control, DNS cluster, etc get spread out so that in the event of network issues on our end, main points of contact and access aren't impacted by it.

We will be re-doing the setup in the near future, but for now, we wait for them to fix their network issues.

EDIT: It's not fully resolved, but greatly improved. The issue is on Lumen's network and we have faith that fully performance will be restored soon.

Affecting Other - IDAHO USA KVM VPS

• 01/30/2023 00:00 - 01/31/2023 15:27
• Last Updated 01/29/2023 01:45

We are aware of network issues impacting the stability of connectivity in our Idaho, USA location. On Monday (01/30/2023) we will replacing a switch to correct the issues currently being experienced.

Affecting System - VPS Control Panel

• 04/05/2022 03:57 - 04/05/2022 06:48
• Last Updated 04/05/2022 04:02

Be advised that we are undergoing some planned maintenance of the customer facing VPS control panel. This upgrade moves the system onto dedicated hardware for some increased performance as well as on a network offering superior protection against DDoS attacks. This update also allows us to increase the number of offered OS templates and ISOs available for your use.

During this period you may encounter issues logging into your VPS control panel or see SSL errors. This should all be resolved soon, most of this is caused by the DNS updating.

Contact us via the helpdesk if you encounter any issues.

Affecting Other - NL-02

• 01/01/2022 11:45
• Last Updated 01/01/2022 15:21

UPDATE: 3:20PM EST

It would appear at this time that replacing the motherboard did the trick. The node has been stable since the emergency maintenance and as such this issue is being closed. We are still monitoring the situation and still plan to have additional capacity within the next 24 hours to accommodate migrations if need be.

Thanks for your patience if you were impacted by this today, or previous stability issues on the NL-02 node in the past month.

UPDATE: 11:42AM EST

Within the last hour the motherboard has been replaced and early observations show things have stabilized. All VMs are online, no data has been lost, and we are still monitoring the node for further issues. Furthermore, we will have additional capacity within the next 24 hours to accommodate a migration of all containers if needed if it proves that the motherboard replacement does not solve the issue.

Not exactly how we wanted to kick off the New Year. ( -_-)

(Downgrading priority to Medium)

09:54AM EST

The Netherlands based KVM host node NL-02 is experiencing service impacting issues. We are awaiting for the delivery and setup of a new hardware node and will undergo emergency maintenance to bring the server to a stable condition. In previous weeks we have completed a thorough checklist of tests, including replacing RAM, yet the issue persists. Datacenter staff are scheduling a replacement of the motherboard.

Right now the physical hardware node is rebooting sporadically which is preventing emergency migration efforts to other available nodes. Once service is stabilized we will contact impacted customers for options on moving forward, which include migrating to a new host node.

Thank you for patience.

Affecting System - nl-02.incogvps.com

• 12/08/2021 08:00 - 12/10/2021 13:43
• Last Updated 12/09/2021 16:41

UPDATE: 12/09/2021: Scheduling a memtest tonight which will cause some expected downtime. Will share results when the test is complete.

UPDATE: 12/08/2021: A review and test of the hardware has shown no obvious defects. A new hardware node will become available in the next week and we may migrate all VMs from NL-02 to the new node. We are reviewing options with our server provider to possible replace the PSU and do a more thorough check of the physical hardware after we have migrated customers off of it. Thank you for your patience.

On 12/04/2021 we rebooted the NL-02 host node after a period of software maintenance to diagnose an issue causing the containers on the host node to reboot every few days. After review, it appeared the kernel was crashing, so some updates were applied that we believed would fix this issue and we scheduled a reboot to apply the changes in hopes it would resolve the concern moving forward.

On 12/07/2021, the issue appeared to be unresolved. We have scheduled a full hardware check on the host node that will require some planned, scheduled downtime.

This scheduled hardware check will occur tomorrow, 12/08/2021 sometime after 8AM Eastern Standard Time. The estimated time of completion should be 30 - 120 minutes

Affecting System - nl-02.incogvps.com

• 12/04/2021 08:00 - 12/04/2021 08:07
• Last Updated 12/04/2021 08:04

Please be advised that we will be performing a purposeful and scheduled reboot of the NL-02 VPS node to correct an issue that was causing containers on the node to reboot unexpectedly. This scheduled rebooting of the hardware node should correct the issue, with no unexpected reboots of the virtual containers within moving forward being the desired and expected result.

UPDATE:

• Reboot issued at 08:00AM EST
• Server back online at 08:04AM EST
• All containers have come back online.

We will continue to monitor the node to ensure that the updates prevent the original issue that this maintenance period aims to correct.

Affecting System - VPS Control Panel

• 09/30/2021 20:00
• Last Updated 10/01/2021 06:40

UPDATE: Maintenance has been re-scheduled to a future date and has not yet been completed. We will re-announce when this is to be completed.

When: 09/30/2021
Duration: 2 hours~
Why?:

In an effort to constantly improve our ability to serve you we are migrating our VPS Control panel ( https://control.incogvps.com:4083 ) to a new server. This new server will allow us to:

• Enable Full Disk Encryption for the VPS Control Panel / VPS Master. This VPS master server communicates with all VPS slave servers and our billing portal to provision new servers in all of our locations. This is also how you access and manage your VPS from the web control panel. We are enabling Full Disk Encryption as an extra effort to protect the privacy of our customers in key areas. This prevents data from being able to be accessed in the event that a hard drive is removed from the server.
• Enable IPv6 support for control panel access. Currently the control panel is only available over IPv4 due to a networking limitation where it is currently installed.
• More powerful hardware will help ensure that resources are always available no matter how many concurrent tasks are in queue.
• Additional storage means we can offer even more OS Templates.

Potential Issues:

• After the migration is complete you may need to flush your DNS cache as you may find that the URL to the control panel does not resolve or does not resolve properly.
• Possible SSL errors, see above.
• Existing rDNS / PTR records may temporarily not respond properly or at all. After the migration is completed we will have to update our DNS cluster to connect to the new master server.

Affecting Other - DNS

• 08/26/2021 13:00 - 08/26/2021 14:30
• Last Updated 08/26/2021 14:28

We were alerted of an issue with our PowerDNS setup that syncs user submitted rDNS / PTR records from the Virtualizor Control Panel for VPS hostnames with our small DNS cluster. We believe the issue is now resolved.

Some users may be required to recreate their rDNS records via their control panel.

Customers using our Finland location will still need to ticket us for manual creation of their rDNS records. We own the IP's in our Netherlands location which allows us to automate the process, whereas in Finland they're leased. Shoot us a ticket and we'll be happy to get you setup.

Affecting Other - VPN Network

• 07/02/2021 06:00 - 07/02/2021 06:10
• Last Updated 07/02/2021 06:19

A configuration change on our behalf temporarily made this location unavailable. The issue was reported immediately via automatic alerting. The issue was found, corrected and resolved in about 10 minutes of time. During this time this VPN location/node was unavailable. Service has returned to normal now.