Resolved issues

Networking equipment OS upgrades

Jan 18 2022
We will be performing full OS upgrades on our networking equipment today. Expect a few short network interruptions as a result of this.

Main server replacement

Jul 26 2021
At the start of the month our main server suffered some damage after a power outage and we have been running on spare hardware ever since. This Monday (July 26th) we are planning to replace it with the new permanent hardware setup. Expect half an hour or so of downtime while we make the switchover.

Main server down

Jul 8 2021
After a brief power outage earlier today our main server is unfortunately failing to boot. We are investigating the issue and preparing spare hardware in case the main server cannot be repaired in a timely manner.
Update; We have identified the power backplane has failed in the server and are working to commission replacement hardware.
Update; We are now running on spare hardware and will be doing so for at least the next few days.

Planned ISP network maintenance

Nov 24 2020
Our ISP is doing planned maintenance on the 24th of Nov. The expected downtime is around 10 mintues. This will impact most services with the exception of our IRC network and this page.

Upstream network outage

Nov 16 2020
There appears to be a major outage at one of our upstream providers. We are monitoring the situation but we cannot do much besides wait for them to fix it.

Gateway outage

Aug 13 2020
An outage at an upstream provider has caused our gateway to go down. Their current estimates say it should be resolved within the hour.

Core network and server move

Jul 22 2020
We will be physically moving our core network and the bulk of our servers to a new location. Expect a t least a few hours of downtime for all our services apart from IRC and this status page.

Network outage

Jun 20 2020
We are experiencing an outage at our ISP. All major services are currently unreachable.
Update; Connectivity appears to be restored.

Network outage

Apr 19 2020
We are working on our networking infrastructure and we're experiencing some difficulties in the process. Unforutnately this is causing intermittent loss of connectivity.

Networking issues

Mar 30 2020
Due to networking issues at OVH you may experience problems connecting to our service. If you have the addresses for any of our Insomnia Midnight Project endpoints, those should not be affected.

Network outage

Mar 11 2020
Our ISP is currently doing maintenance which means we all services besides IRC and this page are currently down. They expect to be up and running again within the hour.

Planned downtime for RAM upgrade

Dec 3 2019
We will be upgrading the server RAM today, so expect an hour or so of downtime in during the course of the day.

Network down

Nov 5 2019
Our uplink is currently down. We have not yet been able to ascertain the cause of the outage but our remote telemetry suggest we may not be able to restore connectivity until we can get an engineer on site.

Main server upgrade

Aug 17 2019
Over the last weeks a few auxiliary systems have already been upgraded to Debian 10, however this weekend we will be upgrading the main server as well. Expect at least a few minutes of downtime during the day when the server reboots into the new OS version.

ISP outage

Jul 30 2019
Our ISP is having issues. Unfortunately we cannot do much other than wait for the problems to be resolved upstream.
Update; Upstream appears to have resolved the issue.

Packet loss

Feb 18 2019
We are having some problems with packet loss to our ISP. The issue is under investigation.

Migrating to new server

Oct 31 2018
We are switching over to our new server. In theory all should go smooth but in practice we expect some services will probably missbehave.

Planned power outages

Oct 9 & 10 2018
Planned maintenance to our electrical systems on October 9th and 10th will result on 8 hours without power on both these days. This means the bulk of our services will be unavailable for the duration of the maintenance.

Switching VPN up-link

Sep 30 2018
We will be switching over to a new VPN tunnel, this means the uplink will be down for a few minutes. The IP address will be moved to the new VPN tunnel so updating DNS records is not needed.
Update; There was a misconfigured port forward for a few hours which prevented people from connecting to our main server over SSH. A fix has been pushed to the offending router.

ISP outage

Sep 27 2018
Currently we cannot connect to our ISP. This appears to be an issue on the ISP side so unfortunately there is not much we can do right now beyond waiting for them to fix it. Updates will be posted here if more information becomes available.
Update; Connectivity restored. Hopefully permanently.
Update; It appears to have failed again. Updates will be posted here.
Update; Uplink has returned. Keeping fingers thoroughly crossed.
Update; Aaaaaaand it's gone. I think it's safe to say our uplink is currently being somewhat unreliable.

Main up-link is down

Sep 25 2018
Overnight our core router broke down. We are in the process of replacing it. Services are expected to be fully restored within a few hours (resolved).

Intermittent service secondary outages

Jul 8 2018
We are currently upgrading our virutalization cluster, meaning some of our secondary services will occasionally go down today. No one service is expected to be down for more than an hour or so at the most.

Main server outage

Jun 5 2018
Our main server is currently down. We are investigating the issue and will update on the issue as soon as further information becomes available.
Update 1; It appears we are dealing with a hardware failure. Either a CPU or a mainboard failure appears to be the direct cause of this outage.

Service unreachable

Mar 29 2018
Our ISP is performing scheduled maintenance on their fiberoptic cables. Downtime is expected to last at until 7am local time (NL/Amsterdam) at the latest.

Main server down

Feb 9 2018
We are having technical issues with our main server. We are currently investigating the problem.
Update: Hardware failure caused this outage. We are currently getting replacements and will be restoring a backup of last night.
Update: Backup is restored and server is mostly operational again. Still working on repairing MySQL databases.
Update: MySQL database recovery is finished. Everything should be back to normal.

Network outage

Jan 24 2018
Our upstream provider where we host our gateway is experiencing some issues. Service should be restored soon.

Network outage

Dec 26 2017
Connectivity has been restored after an outage earlier today. We are still waiting to hear back from our ISP as to the exact cause of the outage.

Network issues

Dec 9 2017
During the early hours of December 8th we suffered a lightning strike which damaged some of our routing equipment. We have now replaced the faulty equipment with a permanent new machine. Everything should be back to normal.

Server down

Sep 7 2017
Never have I beheld such a relentless clusterfuck. While replacing the server for a hardware upgrade, the new machine decided to pretty well kill itself. It fried (at least) one major chip on the motherboard, the RAID controller that was in there, and every drive in the array. As it happens the raid controller and the drives where transplanted from the old machine. I think you see where this is going. Long story short: It will likely be a few days before I have found replacement drives, reinstalled the OS, and restored user data backups.

Virtual machine host down

Aug 29 2017
One of our virtual machine host nodes is currently experiencing some hardware issues. Supplementary services like the wiki and the "manager" command may not work correctly.

Main server degraded performance

Aug 15 2017
Following last weeks hardware failure we are currently still running in degraded performance mode with only half the CPU cores available. Since we need to source some new hardware to permanently fix this, it will likely take a few weeks before we are back to full capacity.

Main server down

Aug 6 2017
Our main server is refusing to power up. We are working on resolving the issue.

Upgrade to Debian 9

Jul 8 2017
We are upgraded to Debian 9 (Stretch) today. The upgrade is finished and no further downtime should occur.

Planned upgrade to Debian 9

Jun 6 2017
We are planning to upgrade to Debian 9 on Saturday the 8th of July. Expect some service downtime for the duration of the upgrade process.