Resolved issues

Migrating to new server

Oct 31 2018
We are switching over to our new server. In theory all should go smooth but in practice we expect some services will probably missbehave.

Planned power outages

Oct 9 & 10 2018
Planned maintenance to our electrical systems on October 9th and 10th will result on 8 hours without power on both these days. This means the bulk of our services will be unavailable for the duration of the maintenance.

Switching VPN up-link

Sep 30 2018
We will be switching over to a new VPN tunnel, this means the uplink will be down for a few minutes. The IP address will be moved to the new VPN tunnel so updating DNS records is not needed.
Update; There was a misconfigured port forward for a few hours which prevented people from connecting to our main server over SSH. A fix has been pushed to the offending router.

ISP outage

Sep 27 2018
Currently we cannot connect to our ISP. This appears to be an issue on the ISP side so unfortunately there is not much we can do right now beyond waiting for them to fix it. Updates will be posted here if more information becomes available.
Update; Connectivity restored. Hopefully permanently.
Update; It appears to have failed again. Updates will be posted here.
Update; Uplink has returned. Keeping fingers thoroughly crossed.
Update; Aaaaaaand it's gone. I think it's safe to say our uplink is currently being somewhat unreliable.

Main up-link is down

Sep 25 2018
Overnight our core router broke down. We are in the process of replacing it. Services are expected to be fully restored within a few hours (resolved).

Intermittent service secondary outages

Jul 8 2018
We are currently upgrading our virutalization cluster, meaning some of our secondary services will occasionally go down today. No one service is expected to be down for more than an hour or so at the most.

Main server outage

Jun 5 2018
Our main server is currently down. We are investigating the issue and will update on the issue as soon as further information becomes available.
Update 1; It appears we are dealing with a hardware failure. Either a CPU or a mainboard failure appears to be the direct cause of this outage.

Service unreachable

Mar 29 2018
Our ISP is performing scheduled maintenance on their fiberoptic cables. Downtime is expected to last at until 7am local time (NL/Amsterdam) at the latest.

Main server down

Feb 9 2018
We are having technical issues with our main server. We are currently investigating the problem.
Update: Hardware failure caused this outage. We are currently getting replacements and will be restoring a backup of last night.
Update: Backup is restored and server is mostly operational again. Still working on repairing MySQL databases.
Update: MySQL database recovery is finished. Everything should be back to normal.

Network outage

Jan 24 2018
Our upstream provider where we host our gateway is experiencing some issues. Service should be restored soon.

Network outage

Dec 26 2017
Connectivity has been restored after an outage earlier today. We are still waiting to hear back from our ISP as to the exact cause of the outage.

Network issues

Dec 9 2017
During the early hours of December 8th we suffered a lightning strike which damaged some of our routing equipment. We have now replaced the faulty equipment with a permanent new machine. Everything should be back to normal.

Server down

Sep 7 2017
Never have I beheld such a relentless clusterfuck. While replacing the server for a hardware upgrade, the new machine decided to pretty well kill itself. It fried (at least) one major chip on the motherboard, the RAID controller that was in there, and every drive in the array. As it happens the raid controller and the drives where transplanted from the old machine. I think you see where this is going. Long story short: It will likely be a few days before I have found replacement drives, reinstalled the OS, and restored user data backups.

Virtual machine host down

Aug 29 2017
One of our virtual machine host nodes is currently experiencing some hardware issues. Supplementary services like the wiki and the "manager" command may not work correctly.

Main server degraded performance

Aug 15 2017
Following last weeks hardware failure we are currently still running in degraded performance mode with only half the CPU cores available. Since we need to source some new hardware to permanently fix this, it will likely take a few weeks before we are back to full capacity.

Main server down

Aug 6 2017
Our main server is refusing to power up. We are working on resolving the issue.

Upgrade to Debian 9

Jul 8 2017
We are upgraded to Debian 9 (Stretch) today. The upgrade is finished and no further downtime should occur.

Planned upgrade to Debian 9

Jun 6 2017
We are planning to upgrade to Debian 9 on Saturday the 8th of July. Expect some service downtime for the duration of the upgrade process.