Resolved issues
Gateway migration
Apr 4 2024
Our upstream gateway is being migrated to new hardware, this means it will go down for a few minutes today.
ISP network maintenance
Mar 13 2024
Today an engineer from our ISP will be working on the uplink line so expect some service interruptions for our main network and core servers.
Debian 12 upgrade
Mar 2 2024
This weekend we will be updating to Debian 12. Expect some breakages and at least one server reboot.
IRC server updates
Oct 7 2023
We will be updating our IRC servers to a new version of UnrealIRCd. We expect active connections to be interrupted as well as a few minutes of downtime for the IRC network. All other services are unaffected.
ISP network maintenance
Oct 5 2023
Our main ISP will be conducting network maintenance which may result in a temporary degradation or drop of our core services during the day.
ISP network maintenance
May 17 2023
Our main ISP will be conducting network maintenance which may result in a temporary degradation or drop of our core services during the day.
Uplink maintenance
Nov 25 2022
Our main uplink will be down for a short while for maintenance. We expect everything to be back online in less than an hour.
RAID Array issues
Sept 06 2022
We are having issues with our main storage array on the server. The filesystem is currently read-only and a great number of files are unable to be read properly. Naturally we are extremely concerned about this turn of events and are doing all we can to diagnose and resolve the issue.
Update: There is indeed a very serious issue with the storage array. Many files are missing or partially zeroed out. We are most likely looking at restoring from last nights backups.
Update: The server has been been re-provisioned running on new drives for the storage array and a backup from the night preceding the incident has been restored.
Update: The restore has been completed. Due to compounding issues, some databases had to be restored from an older backup.
ISP outage
Jun 21 2022
Our ISP is experiencing an outage. Their current estimate is the outage will last a few hours.
Networking equipment OS upgrades
Jan 18 2022
We will be performing full OS upgrades on our networking equipment today. Expect a few short network interruptions as a result of this.
Main server replacement
Jul 26 2021
At the start of the month our main server suffered some damage after a power outage and we have been running on spare hardware ever since. This Monday (July 26th) we are planning to replace it with the new permanent hardware setup. Expect half an hour or so of downtime while we make the switchover.
Main server down
Jul 8 2021
After a brief power outage earlier today our main server is unfortunately failing to boot. We are investigating the issue and preparing spare hardware in case the main server cannot be repaired in a timely manner.
Update; We have identified the power backplane has failed in the server and are working to commission replacement hardware.
Update; We are now running on spare hardware and will be doing so for at least the next few days.
Planned ISP network maintenance
Nov 24 2020
Our ISP is doing planned maintenance on the 24th of Nov. The expected downtime is around 10 mintues. This will impact most services with the exception of our IRC network and this page.
Upstream network outage
Nov 16 2020
There appears to be a major outage at one of our upstream providers. We are monitoring the situation but we cannot do much besides wait for them to fix it.
Gateway outage
Aug 13 2020
An outage at an upstream provider has caused our gateway to go down. Their current estimates say it should be resolved within the hour.
Core network and server move
Jul 22 2020
We will be physically moving our core network and the bulk of our servers to a new location. Expect a t least a few hours of downtime for all our services apart from IRC and this status page.
Network outage
Jun 20 2020
We are experiencing an outage at our ISP. All major services are currently unreachable.
Update; Connectivity appears to be restored.
Network outage
Apr 19 2020
We are working on our networking infrastructure and we're experiencing some difficulties in the process. Unforutnately this is causing intermittent loss of connectivity.
Networking issues
Mar 30 2020
Due to networking issues at OVH you may experience problems connecting to our service. If you have the addresses for any of our Insomnia Midnight Project endpoints, those should not be affected.
Network outage
Mar 11 2020
Our ISP is currently doing maintenance which means we all services besides IRC and this page are currently down. They expect to be up and running again within the hour.
Planned downtime for RAM upgrade
Dec 3 2019
We will be upgrading the server RAM today, so expect an hour or so of downtime in during the course of the day.
Network down
Nov 5 2019
Our uplink is currently down. We have not yet been able to ascertain the cause of the outage but our remote telemetry suggest we may not be able to restore connectivity until we can get an engineer on site.
Main server upgrade
Aug 17 2019
Over the last weeks a few auxiliary systems have already been upgraded to Debian 10, however this weekend we will be upgrading the main server as well. Expect at least a few minutes of downtime during the day when the server reboots into the new OS version.
ISP outage
Jul 30 2019
Our ISP is having issues. Unfortunately we cannot do much other than wait for the problems to be resolved upstream.
Update; Upstream appears to have resolved the issue.
Packet loss
Feb 18 2019
We are having some problems with packet loss to our ISP. The issue is under investigation.
Migrating to new server
Oct 31 2018
We are switching over to our new server. In theory all should go smooth but in practice we expect some services will probably missbehave.
Planned power outages
Oct 9 & 10 2018
Planned maintenance to our electrical systems on October 9th and 10th will result on 8 hours without power on both these days. This means the bulk of our services will be unavailable for the duration of the maintenance.
Switching VPN up-link
Sep 30 2018
We will be switching over to a new VPN tunnel, this means the uplink will be down for a few minutes. The IP address will be moved to the new VPN tunnel so updating DNS records is not needed.
Update; There was a misconfigured port forward for a few hours which prevented people from connecting to our main server over SSH. A fix has been pushed to the offending router.
ISP outage
Sep 27 2018
Currently we cannot connect to our ISP. This appears to be an issue on the ISP side so unfortunately there is not much we can do right now beyond waiting for them to fix it. Updates will be posted here if more information becomes available.
Update; Connectivity restored. Hopefully permanently.
Update; It appears to have failed again. Updates will be posted here.
Update; Uplink has returned. Keeping fingers thoroughly crossed.
Update; Aaaaaaand it's gone. I think it's safe to say our uplink is currently being somewhat unreliable.
Main up-link is down
Sep 25 2018
Overnight our core router broke down. We are in the process of replacing it. Services are expected to be fully restored within a few hours (resolved).
Intermittent service secondary outages
Jul 8 2018
We are currently upgrading our virutalization cluster, meaning some of our secondary services will occasionally go down today. No one service is expected to be down for more than an hour or so at the most.
Main server outage
Jun 5 2018
Our main server is currently down. We are investigating the issue and will update on the issue as soon as further information becomes available.
Update 1; It appears we are dealing with a hardware failure. Either a CPU or a mainboard failure appears to be the direct cause of this outage.
Service unreachable
Mar 29 2018
Our ISP is performing scheduled maintenance on their fiberoptic cables. Downtime is expected to last at until 7am local time (NL/Amsterdam) at the latest.
Main server down
Feb 9 2018
We are having technical issues with our main server. We are currently investigating the problem.
Update: Hardware failure caused this outage. We are currently getting replacements and will be restoring a backup of last night.
Update: Backup is restored and server is mostly operational again. Still working on repairing MySQL databases.
Update: MySQL database recovery is finished. Everything should be back to normal.
Network outage
Jan 24 2018
Our upstream provider where we host our gateway is experiencing some issues. Service should be restored soon.
Network outage
Dec 26 2017
Connectivity has been restored after an outage earlier today. We are still waiting to hear back from our ISP as to the exact cause of the outage.
Network issues
Dec 9 2017
During the early hours of December 8th we suffered a lightning strike which damaged some of our routing equipment. We have now replaced the faulty equipment with a permanent new machine. Everything should be back to normal.
Server down
Sep 7 2017
Never have I beheld such a relentless clusterfuck. While replacing the server for a hardware upgrade, the new machine decided to pretty well kill itself. It fried (at least) one major chip on the motherboard, the RAID controller that was in there, and every drive in the array. As it happens the raid controller and the drives where transplanted from the old machine. I think you see where this is going. Long story short: It will likely be a few days before I have found replacement drives, reinstalled the OS, and restored user data backups.
Virtual machine host down
Aug 29 2017
One of our virtual machine host nodes is currently experiencing some hardware issues. Supplementary services like the wiki and the "manager" command may not work correctly.
Main server degraded performance
Aug 15 2017
Following last weeks hardware failure we are currently still running in degraded performance mode with only half the CPU cores available. Since we need to source some new hardware to permanently fix this, it will likely take a few weeks before we are back to full capacity.
Main server down
Aug 6 2017
Our main server is refusing to power up. We are working on resolving the issue.
Upgrade to Debian 9
Jul 8 2017
We are upgraded to Debian 9 (Stretch) today. The upgrade is finished and no further downtime should occur.
Planned upgrade to Debian 9
Jun 6 2017
We are planning to upgrade to Debian 9 on Saturday the 8th of July. Expect some service downtime for the duration of the upgrade process.