Affecting Other - Network

• 01/31/2025 09:38 - 01/31/2025 17:10
• Last Updated 01/31/2025 09:40

An ongoing DDoS attack is impacting our Kansas City location. While we have mitigation in place, it would seem that attack traffic is still leaking to it's target servers. We are working on mitigating this and restoring service.

Affecting Other - VPS Nodes

• 01/30/2025 13:21 - 01/31/2025 05:02
• Last Updated 01/30/2025 17:11

UPDATE: Things are stabilizing now. You may experience some higher latency than usual which will be temporary.

UPDATE: The time is now 5PM EST (01/30/2025) and our PA POP is still experiencing high latency and packet loss as we mitigate an attack on this location.

We are investigating a report / alert of inaccessible services in our PA and MO location. Will update this shortly.

Affecting Other - Kansas City, Missouri (All Services)

• 12/05/2024 04:00 - 12/05/2024 04:44
• Last Updated 12/05/2024 04:50

EDIT: After 37 minutes, the service impacting part of the maintenance has been completed. Other maintenance continues that should not impact end-user connectivity.

Starting at about 4AM EST, planned network maintenance in our Kansas City POP will begin. This will impact only customers with a dedicated server in Kansas City or VPN customers with an active Kansas City Wireguard profile. Yesterday we sent notice via email to those who would be impacted by this.

Estimated time is one hour. Some service disruption will be experienced during this period. Updates will be posted here.

Affecting Other - Network

• 11/28/2024 00:00 - 11/28/2024 10:08
• Last Updated 11/28/2024 05:55

An upstream ISP has had a reported fiber cut that is impacting network performance for our Liberty Lake, Washington location. As a result, sporadic network stability has been observed over the last several hours causing uptime monitors and other alerts to sound. We are awaiting a fix for this, which is outside of IncogNET's control, for performance to stabilize and return to normal.

Affecting Other - Network

• 11/06/2024 09:48
• Last Updated 11/06/2024 13:24

FINAL UPDATE (11/06/2024 @ 1:15PM EST):

Should be resolved now. All monitors are indicating everything is back online. The datacenter has replaced a linecard on their end, and chatter indicates that their other customers are online at this time as well.

---

UPDATE (11/06/2024 @ 11:30AM EST):

"The network engineering team has identified the issue as a line card failure, which is affecting the main upstream connection. They are actively working on resolving it. We’ll share an estimated time for repair (ETR) as soon as possible. Thank you for your patience."

---

Original message below:

We are aware of an issue in our Liberty Lake, Washington location. After review, it appears to be an issue with our datacenter provider and is impacting all of their tenants and customers, not just IncogNET. We are awaiting an update from them to restore connectivity to normal.

Affecting Other - Network / Facility Maintenance

• 09/30/2024 15:01 - 10/01/2024 09:17
• Last Updated 09/30/2024 15:11

Sorry for the late notice. We received an email from our datacenter yesterday and it was only discovered after investigating what appeared (originally) to be a network related outage in our Allentown, PA location. Had we had more notice, we would have certainly relayed it to our customers who are impacted by this.

Details of the notice below:

We are writing to inform you of a planned facility maintenance that will affect your services in Pennsylvania with us. Our team is working diligently to ensure minimal disruption to your services. We will be physically migrating our hardware into our new facility. This will allow us to add stock and will additionally bring our PA network up to speed with our other locations as the move will enable us to add both NTT and Lumen as well as a physical POP at 60 Hudson, NY. This has been in the works for awhile, but due to so many third parties being involved to make the move as seamless as possible we've been beholden to outside timetables and confirmations.

I do apologize for the short notice and some of you may already have received this e-mail notice on September 23rd, but a large portion of our affected customers did not so we're sending it to all PA service holders again.

Maintenance Window:

• Start Time: September 30th, 2024 at approximately 1PM EST
• Expected End Time: September 30th, 2024 8PM EST

Affected Services:

• All services in Allentown, PA (TEK)

What to Expect:

• Temporary unavailability of the affected services during the maintenance period.

Updates:

• We will post an update to our status page when the move is complete, and if you are still experiencing any issues you should open a ticket after that.

We apologize for any inconvenience this may cause and will be working diligently to complete everything as quickly as possible.

 

We (IncogNET) do apologize for this interruption in service and will await updates and progress from the datacenter.

Affecting Other - Kansas City, Missouri (All Services)

• 09/05/2024 10:00 - 09/06/2024 13:35
• Last Updated 09/06/2024 13:36

LATEST UPDATE: Maintenance is now complete. The hardware has been migrated and all devices appear to have come online after being moved.

UPDATE: Yesterday's delayed maintenance is now taking place. This is service impacting as hardware is being physically migrated and are disconnected from power and networking during this move. ETA: 2 hours.

UPDATE: Maintenance was unable to be completed today and will resume tomorrow.

We are undergoing maintenance in our Kansas City, Missouri, USA location that will impact service for all services offered from this POP. This includes dedicated server clients and VPN customers.

What is happening?

We will be physically migrating our hardware to a new rack within the same data center. The new rack is located two rows away from the current one, and the estimated time for this process is approximately two hours.

Why is this necessary?

This migration is part of our efforts to accommodate planned growth at this location. It will provide us with additional space and more control over our network and infrastructure, allowing us to bring this location to the same standards as our other POPs.

Affecting Other - EU-NL-08

• 08/13/2024 03:15
• Last Updated 08/13/2024 04:17

Update: Services restored and all VMs back online. A reboot of the physical node was required. We're continuing to monitor the host node and try to determine what may be causing the virtnetwork to sometimes fail. Marking this issue as resolved now.

---

The VPS node EU-NL-08's virtual network is down. Currently working to debug now. This is impacting all Virtual Servers on this host node. More details to come.

Affecting System - EU-NL-07

• 07/03/2024 21:23
• Last Updated 07/03/2024 23:28

(Updates shown newest first, oldest last)

FINAL UPDATE:

All VMs should be coming back online.

There was a hardware issue causing a voltage discrepancy preventing the server from booting. A new server was built and the drives were migrated over. At a glance, the server appears to be healthy and operating normally. We'll continue to monitor for issues (though the old hardware provided no indication of potential failure). If your VPS is still down, please contact us. They're all currently booting back up one by one at this time.

UPDATE:

"Unfortunately, all power-related troubleshoots have so far proven unproductive. This means there is a myriad of possible hardware-related causes for the voltage error and refusal to power on."

A new server is being built currently and the drives will be swapped over to it. More updates to follow.

---

UPDATE:

"Checking the server's remote console, we notice the server is powered down. System powerup commands however do not seem to work and we found a voltage-related error log recorded around 00:02 CEST."

More updates to follow.

---

The VPS node EU-NL-07 has been reported as down. Our internal monitoring detected this shortly more than an hour ago.

We've had to reach out to the datacenter staff to review this from their end.

Will provide additional updates as they become available.

Affecting Other - Network

• 11/29/2023 17:23
• Last Updated 11/29/2023 20:50

We received notification from our internal monitoring several minutes ago of a network issue in our Allentown, Pennsylvania location. We are currently failing over to our backup circuit which will route traffic through our Liberty Lake, Washington location. Until the issue is resolved, additional latency may be experienced since traffic is now currently being hauled through a non-ideal path while the issue with the local circuit is under review.

UPDATE: The issue was upstream, and has since been resolved. We're now routing traffic as normal again, so latency and speeds should return to normal now. Closing this issue out now.

Affecting System - EU-NL-08

• 11/15/2023 01:00 - 11/15/2023 02:00
• Last Updated 11/24/2023 10:49

UPDATE 11/24/2023:

This has occurred again so we are having the datacenter replace the power cable and PDU port that this server is connected to. Will update shortly.

UPDATE:

The PDU cable has been replaced and ran to another PDU within the same rack. All VMs are coming back up right now.

----

The EU-NL-08 VPS node was reported by internal monitoring as offline. Upon review, it appeared as powered off with the datacenter. It was promptly powered back on and each VPS is now currently in the state of coming back online.

We are investigating this further, including reaching out to the datacenter to have them review their end. Internal monitoring on our end show no hardware concerns or resource constraints that'd force a physical node to power off. At a glance, we observe no high temps, the CPU idles at about 35% under normal constant use, no storage or array alerts, etc.

Will report back when there is more.

UPDATE:

The iDRAC logs didn't show any sort of power outage, and the PDU port that the server is connected to only shows a 'power on' entry (us powering the server back on) in it's logs, no 'power off'. Datacenter staff checked to make sure the power cable was firmly connected, and it is. We may schedule a PDU cable replacement and have it ran to a different PDU in the same rack in the chance that either of these are or were defective.

Affecting Server - nl-01.incoghost.com

• 10/20/2023 05:14 - 10/20/2023 06:37
• Last Updated 10/20/2023 06:37

Be advised that we are migrating shared hosting clients in the Netherlands to new hardware. This is part of a larger network and infrastructure upgrade in this location. We will post any related updates regarding this migration here.

EDIT: Migration is complete, with our monitoring showing 1 hour and 5 minutes of downtime.

Affecting Other - Network

• 10/13/2023 19:20 - 10/14/2023 04:20
• Last Updated 10/14/2023 04:20

Hello,

We've experienced a couple short outages in our US-West (Liberty Lake, Washington) location today. Our datacenter is currently replacing some network hardware as this issue is at their core and impacting multiple customers, not just us. We will update this once we have received an update from them with an incident report.

From our datacenter:

INCIDENT REPORT

Description: Router hardware issues and emergency repairs on October 13, 2023 at approximately the following times:

1. 11:03PST until 11:04PST
2. 15:12PST until 15:13PST
3. 16:12PST until 16:24PST
4. 16:28PST until 16:32PST
   
   

Root cause: Our master routing engine within edge1.LBLKWA experienced an unplanned reboot, accompanied by an unusual error code at 11:03PST. Subsequently, the backup routing engine assumed control, and by 11:04PST network traffic resumed following the re-establishment of BGP sessions. To diagnose and resolve this unexpected issue, our NOC immediately engaged with a Juniper JTAC representative. The advice received was to pursue two critical actions: a firmware upgrade on both routing engines and the replacement of the problematic routing engine and control board. Notably, the error message exhibited by the router was abnormal and had not been seen by JTAC.

Resolution: Our NOC diligently executed the recommended measures, which included the firmware upgrades and the replacement of the malfunctioning routing engine and additional control board in our core router. During these procedures, there was a series of brief outages as the mastership role transitioned between the master and backup routing engines, leading to momentary disruptions in BGP sessions which subsequently re-established. After the repairs were completed network traffic stability and core redundancy within our edge infrastructure were re-established.

We understand that this incident caused inconveniences to our customers, and we do sincerely apologize for the disruptions it caused. We will immediately restock our spare/replacement critical hardware and continue to actively monitor for any signs of trouble.

Affecting System - DNS (NS1/NS2)

• 06/28/2023 20:00 - 06/28/2023 22:20
• Last Updated 06/28/2023 22:36

On 06/28/2023 both the NS1 and NS2 nameserver failed, resulting in a DNS outage for our shared web hosting customers in all locations.

There was a delayed response our review of this due to a mishap with an internal notification system.

When discovered, it appeared that both our individual nameservers that are used by our shared hosting customers had ran out of disk space. The same level of monitoring used on our other in-production servers that would typically alert us of this was not applied to these name servers, as the monitoring we had before was to only check that their IPs were responding to ping (which they both were).

Once this was resolved, we re-synced the DNS records/zones from the individual shared hosting servers in the US and EU which allowed domain names to resolve again.

We’ve added some additional monitoring and safeguards to prevent this from ever occurring again as well as created new monitoring rules for alerts.

Impacted customers can contact us for service credit.

• 05/25/2023 14:53
• Last Updated 05/25/2023 15:53

The NL-01 VPS node in the Netherlands is currently down. We are working with the datacenter to bring it back as it's not responding to remote commands. Internal monitoring doesn't indicate a hardware failure of any sort at this time, however we will continue to investigate and will bring this back up as soon as possible.

Update 1: Internal monitoring showed a large spike in CPU temperature right before the server went offline. The datacenter is reviewing it now and will take appropriate action. More updates to come as they're known to us.

Update 2: Datacenter states the CPU fan has failed and are replacing it. Should be resolved soon.

Update 3: The datacenter has replaced the CPU cooler, the VPS node has been booted back up and is now once again reachable. We'll continue to monitor for any signs of potential failure or overheating.

Affecting Other - CDA Network

• 05/16/2023 19:36 - 05/17/2023 07:09
• Last Updated 05/16/2023 19:37

We are aware of a service impacting issue in our USA location in Idaho and are investigating and reviewing it.

Affecting System - VPS Control Panel

• 04/01/2023 08:11 - 04/01/2023 21:39
• Last Updated 04/01/2023 21:39

We're aware of impossibly slow OS installs / reinstalls for most VPS nodes. OS templates are stored on the master server which is outside our network, with a 3rd party, and we're experiencing pretty terrible network conditions right now. This may delay or prevent OS reinstalls for VPS customers

We host some of our public facing infrastructure outside of our own network so that not all eggs are in the same basket. Things like our website mirrors, portal, VPS control, DNS cluster, etc get spread out so that in the event of network issues on our end, main points of contact and access aren't impacted by it.

We will be re-doing the setup in the near future, but for now, we wait for them to fix their network issues.

EDIT: It's not fully resolved, but greatly improved. The issue is on Lumen's network and we have faith that fully performance will be restored soon.

Affecting Other - IDAHO USA KVM VPS

• 01/30/2023 00:00 - 01/31/2023 15:27
• Last Updated 01/29/2023 01:45

We are aware of network issues impacting the stability of connectivity in our Idaho, USA location. On Monday (01/30/2023) we will replacing a switch to correct the issues currently being experienced.

Affecting System - VPS Control Panel

• 04/05/2022 03:57 - 04/05/2022 06:48
• Last Updated 04/05/2022 04:02

Be advised that we are undergoing some planned maintenance of the customer facing VPS control panel. This upgrade moves the system onto dedicated hardware for some increased performance as well as on a network offering superior protection against DDoS attacks. This update also allows us to increase the number of offered OS templates and ISOs available for your use.

During this period you may encounter issues logging into your VPS control panel or see SSL errors. This should all be resolved soon, most of this is caused by the DNS updating.

Contact us via the helpdesk if you encounter any issues.

Affecting Other - NL-02

• 01/01/2022 11:45
• Last Updated 01/01/2022 15:21

UPDATE: 3:20PM EST

It would appear at this time that replacing the motherboard did the trick. The node has been stable since the emergency maintenance and as such this issue is being closed. We are still monitoring the situation and still plan to have additional capacity within the next 24 hours to accommodate migrations if need be.

Thanks for your patience if you were impacted by this today, or previous stability issues on the NL-02 node in the past month.

UPDATE: 11:42AM EST

Within the last hour the motherboard has been replaced and early observations show things have stabilized. All VMs are online, no data has been lost, and we are still monitoring the node for further issues. Furthermore, we will have additional capacity within the next 24 hours to accommodate a migration of all containers if needed if it proves that the motherboard replacement does not solve the issue.

Not exactly how we wanted to kick off the New Year. ( -_-)

(Downgrading priority to Medium)

09:54AM EST

The Netherlands based KVM host node NL-02 is experiencing service impacting issues. We are awaiting for the delivery and setup of a new hardware node and will undergo emergency maintenance to bring the server to a stable condition. In previous weeks we have completed a thorough checklist of tests, including replacing RAM, yet the issue persists. Datacenter staff are scheduling a replacement of the motherboard.

Right now the physical hardware node is rebooting sporadically which is preventing emergency migration efforts to other available nodes. Once service is stabilized we will contact impacted customers for options on moving forward, which include migrating to a new host node.

Thank you for patience.

Affecting System - nl-02.incogvps.com

• 12/08/2021 08:00 - 12/10/2021 13:43
• Last Updated 12/09/2021 16:41

UPDATE: 12/09/2021: Scheduling a memtest tonight which will cause some expected downtime. Will share results when the test is complete.

UPDATE: 12/08/2021: A review and test of the hardware has shown no obvious defects. A new hardware node will become available in the next week and we may migrate all VMs from NL-02 to the new node. We are reviewing options with our server provider to possible replace the PSU and do a more thorough check of the physical hardware after we have migrated customers off of it. Thank you for your patience.

On 12/04/2021 we rebooted the NL-02 host node after a period of software maintenance to diagnose an issue causing the containers on the host node to reboot every few days. After review, it appeared the kernel was crashing, so some updates were applied that we believed would fix this issue and we scheduled a reboot to apply the changes in hopes it would resolve the concern moving forward.

On 12/07/2021, the issue appeared to be unresolved. We have scheduled a full hardware check on the host node that will require some planned, scheduled downtime.

This scheduled hardware check will occur tomorrow, 12/08/2021 sometime after 8AM Eastern Standard Time. The estimated time of completion should be 30 - 120 minutes

Affecting System - nl-02.incogvps.com

• 12/04/2021 08:00 - 12/04/2021 08:07
• Last Updated 12/04/2021 08:04

Please be advised that we will be performing a purposeful and scheduled reboot of the NL-02 VPS node to correct an issue that was causing containers on the node to reboot unexpectedly. This scheduled rebooting of the hardware node should correct the issue, with no unexpected reboots of the virtual containers within moving forward being the desired and expected result.

UPDATE:

• Reboot issued at 08:00AM EST
• Server back online at 08:04AM EST
• All containers have come back online.

We will continue to monitor the node to ensure that the updates prevent the original issue that this maintenance period aims to correct.

Affecting System - VPS Control Panel

• 09/30/2021 20:00
• Last Updated 10/01/2021 06:40

UPDATE: Maintenance has been re-scheduled to a future date and has not yet been completed. We will re-announce when this is to be completed.

When: 09/30/2021
Duration: 2 hours~
Why?:

In an effort to constantly improve our ability to serve you we are migrating our VPS Control panel ( https://control.incogvps.com:4083 ) to a new server. This new server will allow us to:

• Enable Full Disk Encryption for the VPS Control Panel / VPS Master. This VPS master server communicates with all VPS slave servers and our billing portal to provision new servers in all of our locations. This is also how you access and manage your VPS from the web control panel. We are enabling Full Disk Encryption as an extra effort to protect the privacy of our customers in key areas. This prevents data from being able to be accessed in the event that a hard drive is removed from the server.
• Enable IPv6 support for control panel access. Currently the control panel is only available over IPv4 due to a networking limitation where it is currently installed.
• More powerful hardware will help ensure that resources are always available no matter how many concurrent tasks are in queue.
• Additional storage means we can offer even more OS Templates.

Potential Issues:

• After the migration is complete you may need to flush your DNS cache as you may find that the URL to the control panel does not resolve or does not resolve properly.
• Possible SSL errors, see above.
• Existing rDNS / PTR records may temporarily not respond properly or at all. After the migration is completed we will have to update our DNS cluster to connect to the new master server.

Affecting Other - DNS

• 08/26/2021 13:00 - 08/26/2021 14:30
• Last Updated 08/26/2021 14:28

We were alerted of an issue with our PowerDNS setup that syncs user submitted rDNS / PTR records from the Virtualizor Control Panel for VPS hostnames with our small DNS cluster. We believe the issue is now resolved.

Some users may be required to recreate their rDNS records via their control panel.

Customers using our Finland location will still need to ticket us for manual creation of their rDNS records. We own the IP's in our Netherlands location which allows us to automate the process, whereas in Finland they're leased. Shoot us a ticket and we'll be happy to get you setup.

Affecting Other - VPN Network

• 07/02/2021 06:00 - 07/02/2021 06:10
• Last Updated 07/02/2021 06:19

A configuration change on our behalf temporarily made this location unavailable. The issue was reported immediately via automatic alerting. The issue was found, corrected and resolved in about 10 minutes of time. During this time this VPN location/node was unavailable. Service has returned to normal now.