Server Downtime log

Brief network fuckup there, probably a ddos on someone else

 

Temp banned from gmail again yay

 

Forum may be intermittently very slow to load due to steam api lag being exacerbated by high gmod activity (motd etc)

 

Continuing to investigate the forum's php-fpm workers getting stuck, likely some external service timing out but I can't figure out what

 

mfw hard disk failing in my backup server

if shit hits the fan in the next 12 hours then ruh roh

 

Brief network outage (outside my control), unfortunately emptied all servers at a peak time

 

Turned into a 15 minute outage with almost all routing into the bw network dying (at least everything through the primary router in Telehouse North)

Again outside my control and hopefully the last of it

 

The cause of the downtime last night was a line card exploding in one of the insane £80,000 routers underpinning the network

Not all users were being routed through this router hence some people (@SnuggleMuffin) could still access things

 

so it turns out this was actually a deeper issue than i thought - offsite backups have been failing halfway through for 4 months

bloody good thing i checked this

 

tl;dr expect downtime from 02:00-03:00 UTC on the 18th

 

~45 mins of 7 Gbit/s DDoS

UDP traffic to one of the TF2 server ports, so someone's salty about getting banned

 

Was not me. I did not do it.

 

Down for about an hour due to DDoS again

I'll have to come up with a way of killing the gameservers automatically when this happens, because the particular type of DDoS allows them to continue to function with massive lag, which is obviously not a good image for us

 

and again for an hour

mfw chargen udp attack

 

if it is in fact a chargen attack, then it's not even a botnet, just a list of server IPs dumb enough to have chargen listening publicly

 

ddos detection false alarm, all gameservers down for the past 90 mins