Server Downtime log

A huge amount of things are currently broken, probably won't have time to fix until tomorrow

 

Fixed:

- Forums (updated to XF 1.5 also)
- imgkk uploading
- Logging into sourcebans


Still broken:

- Pretty much all my old dodgy PHP scripts which used shit that is removed in PHP 7 - so sprays, homepage widgets, dsock, etc.
- Anything managed by DSM, ie the gameservers
- HLstatsX
- The Kirino bot
- My custom changes to Murmur ([[limit:x]] tag on channels)


On the bright side PHP 7 is a lot faster so I guess all this effort is worth it

 

Fixed:

- Sprays
- DSock core
- Servers sidebar widget
- Gameservers/DSM


Not fixed:

- Mumble sidebar widget (will need rewrite or wait for PHP 7 ICE extension)
- HLstatsX web panel (horrible old code, needs further investigation for PHP 7)
- Murmur custom shit (don't really see the need)
- Kirino bot

 

The issue that initially crashed the system has happened another 3 times and appears to be connected to the gameservers, so they will be down for at least another day

 

Fixed:

- Gameservers, I think
- Remaining parts of DSock
- Kirino bot
- Most of MGE stats

 

Fixed:

- HLstatsX
- Mumble channel limit
- Mumble sidebar widget (holy fucking shit the amount of hoops I had to jump through)

 

annnnd another kernel crash and the tf2 servers are all running twice each and constantly restarting and sapping all the cpu

fml

 

Apart from a hard disk that's been slowly failing for months (need to get off my ass and go to london to replace it), things actually seem ok now

 

It's the made in china 3tb

It's exhibiting the same symptoms as the Samsung drives that were in there ages ago - the intense vibration makes it slightly miss a write now and again, rendering the sector unreadable until it is overwritten with zeroes. According to smart it is not actually reallocating the sector, so its not a bad sector in the traditional sense

To be fair to it, it lasted 3.5 years 24/7 until any corruption started

Conclusion: only use ssds or enterprise hdds in anything with 5 digit rpm fans

 

Literally running on fumes at this point

Definitely will have to replace the ssd too when I get round to doing the hard disk

Thank fuck SSDs don't just fail at random, still 10% rated life left, enough for a few months

 

Was down for 30 mins there for some preventative maintenance for the ssd, which consisted of turning the entire server off for a few mins - seems to have helped, no longer clocking up wear levelling count every 10 mins

 

1% ssd lifetime left

mfw 'a few months' turns out to be < 3 weeks