[00:01:13] <mirahezebots_>	 PROBLEM - bacula2 Puppet on bacula2 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file
[00:02:42] <icinga-miraheze>	 PROBLEM - bacula2 Puppet on bacula2 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file
[00:06:30] <icinga-miraheze>	 RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3777 MB (19% inode=93%);
[00:07:18] <mirahezebots_>	 RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[00:07:56] <icinga-miraheze>	 RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:08:21] <paladox>	 !log ip -6 address add 2604:180:f3::382/64 dev ens3 - bacula2
[00:08:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[00:08:32] <paladox>	 !log ip addr del fe80::f816:3eff:fe31:f5f/64 dev ens3 - bacula2
[00:08:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[00:12:55] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvgsA
[00:12:56] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03aa98dda - Update comment
[00:15:51] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvgGf
[00:15:52] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03a52c1bf - Add db6
[00:19:14] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvgGT
[00:19:16] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03affe1ce - Add db6 part 2
[01:17:54] <paladox>	 Post mortom is at https://meta.miraheze.org/wiki/Postmortem-29022020
[01:17:55] <ZppixBot>	 [ Postmortem-29022020 - Miraheze Meta ] - meta.miraheze.org
[01:33:03] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 0.34, 1.83, 2.01
[01:37:48] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.59, 1.06, 1.62
[02:26:18] <mirahezebots_>	 PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[02:29:34] <mirahezebots_>	 RECOVERY - bacula2 Bacula Phabricator Static on bacula2 is OK: OK: Full, 82043 files, 3.092GB, 2020-02-26 23:42:00 (2.1 days ago)
[06:29:29] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3216 MB (13% inode=94%);
[06:45:16] <mirahezebots_>	 PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-redis]
[06:53:38] <mirahezebots_>	 RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[11:55:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvgzT
[11:55:08] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 0303005f1 - BOT: Updating services config for wikis
[15:44:46] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2642 MB (10% inode=94%);
[16:35:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvgrG
[16:35:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 033120483 - BOT: Updating services config for wikis
[16:40:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvgru
[16:40:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 035a988bd - BOT: Updating services config for wikis
[16:45:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvgri
[16:45:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 0309d85b2 - BOT: Updating services config for wikis
[16:55:25] <Not-5a0a>	 [02DataDump] 07rail01 opened pull request 03#5: Polish i18n - 13https://git.io/JvgrN
[17:58:03] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JvgKA
[17:58:05] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 036f4bdc0 - add wiki.bydorn.de cert
[18:26:24] <icinga-miraheze>	 PROBLEM - wiki.2035.rocks - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.2035.rocks' expires in 15 day(s) (Mon 16 Mar 2020 06:17:50 PM GMT +0000).
[18:26:38] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvg6w
[18:26:40] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 03324d79b - Bot: Update SSL cert for wiki.2035.rocks
[18:36:28] <icinga-miraheze>	 RECOVERY - wiki.2035.rocks - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.2035.rocks' will expire on Fri 29 May 2020 05:26:32 PM GMT +0000.
[19:12:30] <ZppixBot>	 Hello theeagles! If you have any questions, feel free to ask and someone should answer soon.
[19:13:00] <theeagles>	 Today is Leap Day 29 February 2020 ! Time for some tunes to celebrate it it only happens once every 4 years!!!
[19:13:01] <theeagles>	 On a dark desert highway, cool wind in my hair
[19:13:04] <theeagles>	 Warm smell of colitas, rising up through the air
[19:13:12] <theeagles>	 Up ahead in the distance, I saw a shimmering light
[19:13:19] <theeagles>	 My head grew heavy and my sight grew dim
[19:13:25] <theeagles>	 I had to stop for the night.
[19:13:30] <theeagles>	 Welcome to the Hotel California
[19:13:37] <theeagles>	 Such a lovely place (such a lovely place)
[19:13:44] <theeagles>	 Such a lovely face.
[19:13:50] <theeagles>	 Plenty of room at the Hotel California
[19:13:55] <theeagles>	 Any time of year (any time of year) you can find it here
[19:14:00] <theeagles>	 Her mind is Tiffany-twisted, she got the Mercedes bends
[19:14:08] <theeagles>	 She got a lot of pretty, pretty boys, that she calls friends
[19:14:14] <theeagles>	 Today is Leap Day 29 February 2020 ! Time for some tunes to celebrate it it only happens once every 4 years!!!
[19:19:57] <RhinosF1>	 Thanks Voidwalker
[19:20:01] <RhinosF1>	 grumble: ^
[19:20:11] <ZppixBot>	 Hello thebeatles123[m]! If you have any questions, feel free to ask and someone should answer soon.
[19:20:11] <thebeatles123[m]>	 lets sing alonggg
[19:20:14] <thebeatles123[m]>	 We all live in a yellow submarine
[19:20:14] <thebeatles123[m]>	 Yellow submarine, yellow submarine
[19:20:16] <thebeatles123[m]>	 We all live in a yellow submarine
[19:20:16] <thebeatles123[m]>	 Yellow submarine, yellow submarine
[19:24:02] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 0.68, 2.77, 3.01
[19:27:47] <ZppixBot>	 Hello mikejackson[m]! If you have any questions, feel free to ask and someone should answer soon.
[19:27:47] <mikejackson[m]>	 Billie Jean is not my lover
[19:27:51] <mikejackson[m]>	 She's just a girl who claims that I am the one
[19:28:22] <RhinosF1>	 ilbelkyr: any ideas?
[19:32:56] <ZppixBot>	 Hello rick_astley[m]! If you have any questions, feel free to ask and someone should answer soon.
[19:32:57] <rick_astley[m]>	 This is for you Miraheze Staff! Never gonna give you up you're the best!
[19:32:57] <rick_astley[m]>	 We're no strangers to love
[19:33:01] <rick_astley[m]>	 You know the rules and so do I
[19:33:28] <RhinosF1>	 Please do give us up
[19:33:29] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.47, 1.05, 1.99
[19:37:55] <ZppixBot>	 Hello coldplay123[m]! If you have any questions, feel free to ask and someone should answer soon.
[19:37:56] <coldplay123[m]>	 Oh, angel sent from up above
[19:38:00] <coldplay123[m]>	 You know you make my world light up
[19:38:01] <coldplay123[m]>	 When I was down, when I was hurt
[19:38:33] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.17, 0.72, 1.61
[19:43:11] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2905: SING ALONG Reception123 Voidwalker RhinosF1 Zppix !! It's really fun - 13https://git.io/JvgPl
[19:43:17] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2906: SING ALONG Reception123 Voidwalker RhinosF1 Zppix !! It's really fun - 13https://git.io/JvgP8
[19:43:19] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2907: SING ALONG Reception123 Voidwalker RhinosF1 Zppix !! It's really fun - 13https://git.io/JvgP4
[19:43:20] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2908: SING ALONG Reception123 Voidwalker RhinosF1 Zppix !! It's really fun - 13https://git.io/JvgPB
[19:43:22] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2909: SING ALONG Reception123 Voidwalker RhinosF1 Zppix !! It's really fun - 13https://git.io/JvgPR
[19:43:23] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2910: SING ALONG Reception123 Voidwalker RhinosF1 Zppix !! It's really fun - 13https://git.io/JvgP0
[19:43:25] <RhinosF1>	 Oh bloody god
[19:43:25] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2911: SING ALONG Reception123 Voidwalker RhinosF1 Zppix !! It's really fun - 13https://git.io/JvgPE
[19:43:36] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2912: SING ALONG Reception123 Voidwalker RhinosF1 Zppix !! It's really fun - 13https://git.io/JvgPu
[19:43:42] <Voidwalker>	 paladox, ^
[19:43:45] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2913: I wish I found some better sounds no one's ever heard - 13https://git.io/JvgPz
[19:43:49] <Voidwalker>	 gonna mute the bot if that keeps up
[19:43:50] <Reception123>	 I'mm on it
[19:44:38] <paladox>	 oh my
[19:44:53] <RhinosF1>	 paladox: we’ve got it
[19:44:59] <RhinosF1>	 Blocked and reported
[19:47:16] <paladox>	 heh
[19:47:24] <paladox>	 Reception123 delete the issues too!
[19:48:02] <RhinosF1>	 paladox: account has vanished altogether
[19:48:08] <paladox>	 ok
[19:54:20] <ZppixBot>	 Hello ELO_BAND! If you have any questions, feel free to ask and someone should answer soon.
[19:54:23] <ELO_BAND>	 SIng along!
[19:54:24] <ELO_BAND>	 Last train to London, just headin' out,
[19:54:24] <ELO_BAND>	 Last train to London, just leavin' town.
[20:09:45] <ZppixBot>	 Hello sting! If you have any questions, feel free to ask and someone should answer soon.
[20:09:46] <sting>	 Oh, I'm an alien, I'm a legal alien
[20:09:49] <sting>	 \I'm an Englishman in New York
[20:26:39] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2914: HAVE A NICE LEAP DAY RhinosF1 Voidwalker Zppix Reception123 Two Princes - Spin Doctors - 13https://git.io/JvgX8
[20:26:41] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2915: HAVE A NICE LEAP DAY RhinosF1 Voidwalker Zppix Reception123 Two Princes - Spin Doctors - 13https://git.io/JvgX4
[20:26:42] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2916: HAVE A NICE LEAP DAY RhinosF1 Voidwalker Zppix Reception123 Two Princes - Spin Doctors - 13https://git.io/JvgXB
[20:26:44] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2917: HAVE A NICE LEAP DAY RhinosF1 Voidwalker Zppix Reception123 Two Princes - Spin Doctors - 13https://git.io/JvgXR
[20:26:45] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2918: HAVE A NICE LEAP DAY RhinosF1 Voidwalker Zppix Reception123 Two Princes - Spin Doctors - 13https://git.io/JvgX0
[20:26:47] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2919: HAVE A NICE LEAP DAY RhinosF1 Voidwalker Zppix Reception123 Two Princes - Spin Doctors - 13https://git.io/JvgXE
[20:26:48] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2920: HAVE A NICE LEAP DAY RhinosF1 Voidwalker Zppix Reception123 Two Princes - Spin Doctors - 13https://git.io/JvgXz
[20:26:50] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2921: HAVE A NICE LEAP DAY RhinosF1 Voidwalker Zppix Reception123 Two Princes - Spin Doctors - 13https://git.io/JvgXg
[20:26:50] <Voidwalker>	 Reception123, ^
[20:26:51] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2922: HAVE A NICE LEAP DAY RhinosF1 Voidwalker Zppix Reception123 Two Princes - Spin Doctors - 13https://git.io/JvgX2
[20:26:53] <Not-5a0a>	 [02mw-config] 07autoresponder[bot] commented on issue 03#2923: HAVE A NICE LEAP DAY RhinosF1 Voidwalker Zppix Reception123 Two Princes - Spin Doctors - 13https://git.io/JvgXa
[20:29:26] <RhinosF1>	 Voidwalker: can you +z so Reception123 can op and still see to handle while the abuse happens
[20:30:15] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2099 MB (10% inode=93%);
[20:33:52] <Voidwalker>	 +q gonna expire in 5 minutes anyway
[20:35:13] <RhinosF1>	 Voidwalker: see priv channel
[21:04:04] <mirahezebots_>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[21:04:37] <mirahezebots_>	 PROBLEM - test2 MediaWiki Rendering on test2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4214 bytes in 0.045 second response time
[21:05:08] <mirahezebots_>	 PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is CRITICAL: CRITICAL - NGINX Error Rate is 67%
[21:05:12] <mirahezebots_>	 PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4214 bytes in 0.044 second response time
[21:05:13] <mirahezebots_>	 PROBLEM - mw6 MediaWiki Rendering on mw6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4214 bytes in 0.043 second response time
[21:05:31] <mirahezebots_>	 PROBLEM - mw7 MediaWiki Rendering on mw7 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4214 bytes in 0.024 second response time
[21:06:59] <mirahezebots_>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[21:07:29] <mirahezebots_>	 RECOVERY - test2 MediaWiki Rendering on test2 is OK: HTTP OK: HTTP/1.1 200 OK - 18719 bytes in 0.331 second response time
[21:07:33] <paladox>	 Um
[21:07:34] <paladox>	 JohnLewis: can you look at db4 please? I think it’s ran out of space
[21:07:35] <paladox>	 (I’m currently mobile)
[21:07:36] <paladox>	 Or reception123 ^
[21:07:37] <paladox>	 Oh
[21:08:01] <mirahezebots_>	 RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 1%
[21:08:04] <mirahezebots_>	 RECOVERY - mw6 MediaWiki Rendering on mw6 is OK: HTTP OK: HTTP/1.1 200 OK - 18700 bytes in 1.169 second response time
[21:08:08] <mirahezebots_>	 RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 18719 bytes in 0.430 second response time
[21:08:16] <JohnLewis>	 I don't have access
[21:08:20] <Reception123>	 paladox: looks like it's back 
[21:08:23] <mirahezebots_>	 RECOVERY - mw7 MediaWiki Rendering on mw7 is OK: HTTP OK: HTTP/1.1 200 OK - 18719 bytes in 0.394 second response time
[21:08:28] <JohnLewis>	 but are we seriously still not monitoring db4 appropriately?
[21:08:32] <Reception123>	 I don't think it could've ran out that fast 
[21:08:35] <Reception123>	 But I will take a look
[21:10:11] <Voidwalker>	 down to ~3g judging from icinga
[21:10:32] <Reception123>	 !log PURGE BINARY LOGS BEFORE '2020-02-29 12:00'; on db4
[21:10:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:10:39] <Reception123>	 14G now
[21:10:49] <Reception123>	 that's what it was when I moved the last wiki to db5
[23:12:16] <mirahezebots_>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[23:15:07] <mirahezebots_>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online