Resolved issues

We will be updating our IRC servers to a new version of UnrealIRCd. We expect active connections to be interrupted as well as a few minutes of downtime for the IRC network. All other services are unaffected.

ISP network maintenance

Our main ISP will be conducting network maintenance which may result in a temporary degradation or drop of our core services during the day.

ISP network maintenance

Our main ISP will be conducting network maintenance which may result in a temporary degradation or drop of our core services during the day.

Uplink maintenance

Our main uplink will be down for a short while for maintenance. We expect everything to be back online in less than an hour.

RAID Array issues

We are having issues with our main storage array on the server. The filesystem is currently read-only and a great number of files are unable to be read properly. Naturally we are extremely concerned about this turn of events and are doing all we can to diagnose and resolve the issue.
Update: There is indeed a very serious issue with the storage array. Many files are missing or partially zeroed out. We are most likely looking at restoring from last nights backups.
Update: The server has been been re-provisioned running on new drives for the storage array and a backup from the night preceding the incident has been restored.
Update: The restore has been completed. Due to compounding issues, some databases had to be restored from an older backup.

ISP outage

Our ISP is experiencing an outage. Their current estimate is the outage will last a few hours.

Networking equipment OS upgrades

We will be performing full OS upgrades on our networking equipment today. Expect a few short network interruptions as a result of this.

Main server replacement

At the start of the month our main server suffered some damage after a power outage and we have been running on spare hardware ever since. This Monday (July 26th) we are planning to replace it with the new permanent hardware setup. Expect half an hour or so of downtime while we make the switchover.

Main server down

After a brief power outage earlier today our main server is unfortunately failing to boot. We are investigating the issue and preparing spare hardware in case the main server cannot be repaired in a timely manner.
Update; We have identified the power backplane has failed in the server and are working to commission replacement hardware.
Update; We are now running on spare hardware and will be doing so for at least the next few days.

Planned ISP network maintenance

Our ISP is doing planned maintenance on the 24th of Nov. The expected downtime is around 10 mintues. This will impact most services with the exception of our IRC network and this page.

Upstream network outage

There appears to be a major outage at one of our upstream providers. We are monitoring the situation but we cannot do much besides wait for them to fix it.

Gateway outage

An outage at an upstream provider has caused our gateway to go down. Their current estimates say it should be resolved within the hour.

Core network and server move

We will be physically moving our core network and the bulk of our servers to a new location. Expect a t least a few hours of downtime for all our services apart from IRC and this status page.

Network outage

We are experiencing an outage at our ISP. All major services are currently unreachable.
Update; Connectivity appears to be restored.

Network outage

We are working on our networking infrastructure and we're experiencing some difficulties in the process. Unforutnately this is causing intermittent loss of connectivity.

Networking issues

Due to networking issues at OVH you may experience problems connecting to our service. If you have the addresses for any of our Insomnia Midnight Project endpoints, those should not be affected.

Network outage

Our ISP is currently doing maintenance which means we all services besides IRC and this page are currently down. They expect to be up and running again within the hour.

Planned downtime for RAM upgrade

We will be upgrading the server RAM today, so expect an hour or so of downtime in during the course of the day.

Network down

Our uplink is currently down. We have not yet been able to ascertain the cause of the outage but our remote telemetry suggest we may not be able to restore connectivity until we can get an engineer on site.

Main server upgrade

Over the last weeks a few auxiliary systems have already been upgraded to Debian 10, however this weekend we will be upgrading the main server as well. Expect at least a few minutes of downtime during the day when the server reboots into the new OS version.

ISP outage

Our ISP is having issues. Unfortunately we cannot do much other than wait for the problems to be resolved upstream.
Update; Upstream appears to have resolved the issue.

Packet loss

We are having some problems with packet loss to our ISP. The issue is under investigation.

Migrating to new server

We are switching over to our new server. In theory all should go smooth but in practice we expect some services will probably missbehave.

Planned power outages

Planned maintenance to our electrical systems on October 9th and 10th will result on 8 hours without power on both these days. This means the bulk of our services will be unavailable for the duration of the maintenance.

Switching VPN up-link

We will be switching over to a new VPN tunnel, this means the uplink will be down for a few minutes. The IP address will be moved to the new VPN tunnel so updating DNS records is not needed.
Update; There was a misconfigured port forward for a few hours which prevented people from connecting to our main server over SSH. A fix has been pushed to the offending router.

ISP outage

Currently we cannot connect to our ISP. This appears to be an issue on the ISP side so unfortunately there is not much we can do right now beyond waiting for them to fix it. Updates will be posted here if more information becomes available.
Update; Connectivity restored. Hopefully permanently.
Update; It appears to have failed again. Updates will be posted here.
Update; Uplink has returned. Keeping fingers thoroughly crossed.
Update; Aaaaaaand it's gone. I think it's safe to say our uplink is currently being somewhat unreliable.

Main up-link is down

Overnight our core router broke down. We are in the process of replacing it. Services are expected to be fully restored within a few hours (resolved).

Intermittent service secondary outages

We are currently upgrading our virutalization cluster, meaning some of our secondary services will occasionally go down today. No one service is expected to be down for more than an hour or so at the most.

Main server outage

Our main server is currently down. We are investigating the issue and will update on the issue as soon as further information becomes available.
Update 1; It appears we are dealing with a hardware failure. Either a CPU or a mainboard failure appears to be the direct cause of this outage.

Service unreachable

Our ISP is performing scheduled maintenance on their fiberoptic cables. Downtime is expected to last at until 7am local time (NL/Amsterdam) at the latest.

Main server down

We are having technical issues with our main server. We are currently investigating the problem.
Update: Hardware failure caused this outage. We are currently getting replacements and will be restoring a backup of last night.
Update: Backup is restored and server is mostly operational again. Still working on repairing MySQL databases.
Update: MySQL database recovery is finished. Everything should be back to normal.

Network outage

Our upstream provider where we host our gateway is experiencing some issues. Service should be restored soon.

Network outage

Connectivity has been restored after an outage earlier today. We are still waiting to hear back from our ISP as to the exact cause of the outage.

Network issues

During the early hours of December 8th we suffered a lightning strike which damaged some of our routing equipment. We have now replaced the faulty equipment with a permanent new machine. Everything should be back to normal.

Server down

Never have I beheld such a relentless clusterfuck. While replacing the server for a hardware upgrade, the new machine decided to pretty well kill itself. It fried (at least) one major chip on the motherboard, the RAID controller that was in there, and every drive in the array. As it happens the raid controller and the drives where transplanted from the old machine. I think you see where this is going. Long story short: It will likely be a few days before I have found replacement drives, reinstalled the OS, and restored user data backups.

Virtual machine host down

One of our virtual machine host nodes is currently experiencing some hardware issues. Supplementary services like the wiki and the "manager" command may not work correctly.

Main server degraded performance

Following last weeks hardware failure we are currently still running in degraded performance mode with only half the CPU cores available. Since we need to source some new hardware to permanently fix this, it will likely take a few weeks before we are back to full capacity.

Server maintenance

Gateway migration

ISP network maintenance

Debian 12 upgrade

IRC server updates

ISP network maintenance

ISP network maintenance

Uplink maintenance

RAID Array issues

ISP outage

Networking equipment OS upgrades

Main server replacement

Main server down

Planned ISP network maintenance

Upstream network outage

Gateway outage

Core network and server move

Network outage

Network outage

Networking issues

Network outage

Planned downtime for RAM upgrade

Network down

Main server upgrade

ISP outage

Packet loss

Migrating to new server

Planned power outages

Switching VPN up-link

ISP outage

Main up-link is down

Intermittent service secondary outages

Main server outage

Service unreachable

Main server down

Network outage

Network outage

Network issues

Server down

Virtual machine host down

Main server degraded performance

Main server down

Upgrade to Debian 9

Planned upgrade to Debian 9