The NL-01 VPS node in the Netherlands is currently down. We are working with the datacenter to bring it back as it's not responding to remote commands. Internal monitoring doesn't indicate a hardware failure of any sort at this time, however we will continue to investigate and will bring this back up as soon as possible.
Update 1: Internal monitoring showed a large spike in CPU temperature right before the server went offline. The datacenter is reviewing it now and will take appropriate action. More updates to come as they're known to us.
Update 2: Datacenter states the CPU fan has failed and are replacing it. Should be resolved soon.
Update 3: The datacenter has replaced the CPU cooler, the VPS node has been booted back up and is now once again reachable. We'll continue to monitor for any signs of potential failure or overheating.
Affecting Other - CDA Network
We are aware of a service impacting issue in our USA location in Idaho and are investigating and reviewing it.
Affecting System - VPS Control Panel
We're aware of impossibly slow OS installs / reinstalls for most VPS nodes. OS templates are stored on the master server which is outside our network, with a 3rd party, and we're experiencing pretty terrible network conditions right now. This may delay or prevent OS reinstalls for VPS customers
We host some of our public facing infrastructure outside of our own network so that not all eggs are in the same basket. Things like our website mirrors, portal, VPS control, DNS cluster, etc get spread out so that in the event of network issues on our end, main points of contact and access aren't impacted by it.
We will be re-doing the setup in the near future, but for now, we wait for them to fix their network issues.
EDIT: It's not fully resolved, but greatly improved. The issue is on Lumen's network and we have faith that fully performance will be restored soon.
Affecting Other - IDAHO USA KVM VPS
We are aware of network issues impacting the stability of connectivity in our Idaho, USA location. On Monday (01/30/2023) we will replacing a switch to correct the issues currently being experienced.
Affecting System - VPS Control Panel
Be advised that we are undergoing some planned maintenance of the customer facing VPS control panel. This upgrade moves the system onto dedicated hardware for some increased performance as well as on a network offering superior protection against DDoS attacks. This update also allows us to increase the number of offered OS templates and ISOs available for your use.
During this period you may encounter issues logging into your VPS control panel or see SSL errors. This should all be resolved soon, most of this is caused by the DNS updating.
Contact us via the helpdesk if you encounter any issues.
Affecting Other - NL-02
UPDATE: 3:20PM EST
It would appear at this time that replacing the motherboard did the trick. The node has been stable since the emergency maintenance and as such this issue is being closed. We are still monitoring the situation and still plan to have additional capacity within the next 24 hours to accommodate migrations if need be.
Thanks for your patience if you were impacted by this today, or previous stability issues on the NL-02 node in the past month.
UPDATE: 11:42AM EST
Within the last hour the motherboard has been replaced and early observations show things have stabilized. All VMs are online, no data has been lost, and we are still monitoring the node for further issues. Furthermore, we will have additional capacity within the next 24 hours to accommodate a migration of all containers if needed if it proves that the motherboard replacement does not solve the issue.
Not exactly how we wanted to kick off the New Year. ( -_-)
(Downgrading priority to Medium)
09:54AM EST
The Netherlands based KVM host node NL-02 is experiencing service impacting issues. We are awaiting for the delivery and setup of a new hardware node and will undergo emergency maintenance to bring the server to a stable condition. In previous weeks we have completed a thorough checklist of tests, including replacing RAM, yet the issue persists. Datacenter staff are scheduling a replacement of the motherboard.
Right now the physical hardware node is rebooting sporadically which is preventing emergency migration efforts to other available nodes. Once service is stabilized we will contact impacted customers for options on moving forward, which include migrating to a new host node.
Thank you for patience.
Affecting System - nl-02.incogvps.com
UPDATE: 12/09/2021: Scheduling a memtest tonight which will cause some expected downtime. Will share results when the test is complete.
UPDATE: 12/08/2021: A review and test of the hardware has shown no obvious defects. A new hardware node will become available in the next week and we may migrate all VMs from NL-02 to the new node. We are reviewing options with our server provider to possible replace the PSU and do a more thorough check of the physical hardware after we have migrated customers off of it. Thank you for your patience.
On 12/04/2021 we rebooted the NL-02 host node after a period of software maintenance to diagnose an issue causing the containers on the host node to reboot every few days. After review, it appeared the kernel was crashing, so some updates were applied that we believed would fix this issue and we scheduled a reboot to apply the changes in hopes it would resolve the concern moving forward.
On 12/07/2021, the issue appeared to be unresolved. We have scheduled a full hardware check on the host node that will require some planned, scheduled downtime.
This scheduled hardware check will occur tomorrow, 12/08/2021 sometime after 8AM Eastern Standard Time. The estimated time of completion should be 30 - 120 minutes
Affecting System - nl-02.incogvps.com
Please be advised that we will be performing a purposeful and scheduled reboot of the NL-02 VPS node to correct an issue that was causing containers on the node to reboot unexpectedly. This scheduled rebooting of the hardware node should correct the issue, with no unexpected reboots of the virtual containers within moving forward being the desired and expected result.
UPDATE:
We will continue to monitor the node to ensure that the updates prevent the original issue that this maintenance period aims to correct.
Affecting System - VPS Control Panel
UPDATE: Maintenance has been re-scheduled to a future date and has not yet been completed. We will re-announce when this is to be completed.
When: 09/30/2021
Duration: 2 hours~
Why?:
In an effort to constantly improve our ability to serve you we are migrating our VPS Control panel ( https://control.incogvps.com:4083 ) to a new server. This new server will allow us to:
Potential Issues:
Affecting Other - DNS
We were alerted of an issue with our PowerDNS setup that syncs user submitted rDNS / PTR records from the Virtualizor Control Panel for VPS hostnames with our small DNS cluster. We believe the issue is now resolved.
Some users may be required to recreate their rDNS records via their control panel.
Customers using our Finland location will still need to ticket us for manual creation of their rDNS records. We own the IP's in our Netherlands location which allows us to automate the process, whereas in Finland they're leased. Shoot us a ticket and we'll be happy to get you setup.
Affecting Other - VPN Network
A configuration change on our behalf temporarily made this location unavailable. The issue was reported immediately via automatic alerting. The issue was found, corrected and resolved in about 10 minutes of time. During this time this VPN location/node was unavailable. Service has returned to normal now.
[ Return To Portal Index ]
[ © 2020 - 2023 IncogNET LLC - All Rights Reserved ]