[00:11:03] <Not-f817>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeZ0U
[00:11:04] <Not-f817>	 [02miraheze/ssl] 07paladox 03db4585a - Remove unrecnations.wiki as domain is not pointing at us
[00:11:06] <Not-f817>	 [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L
[00:11:07] <Not-f817>	 [02ssl] 07paladox opened pull request 03#226: Remove unrecnations.wiki as domain is not pointing at us - 13https://git.io/JeZ0T
[00:11:49] <Not-f817>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JeZ0k
[00:11:51] <Not-f817>	 [02miraheze/ssl] 07paladox 0386db376 - Delete unrecnations.wiki.crt
[00:11:52] <Not-f817>	 [02ssl] 07paladox synchronize pull request 03#226: Remove unrecnations.wiki as domain is not pointing at us - 13https://git.io/JeZ0T
[00:11:58] <Not-f817>	 [02ssl] 07paladox closed pull request 03#226: Remove unrecnations.wiki as domain is not pointing at us - 13https://git.io/JeZ0T
[00:11:59] <Not-f817>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JeZ0L
[00:12:01] <Not-f817>	 [02miraheze/ssl] 07paladox 03f28a6ab - Remove unrecnations.wiki as domain is not pointing at us (#226) * Remove unrecnations.wiki as domain is not pointing at us * Delete unrecnations.wiki.crt
[00:15:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZ0m
[00:15:11] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 03909e890 - BOT: Updating services config for wikis
[00:16:24] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZ0Y
[00:16:26] <Not-f817>	 [02miraheze/puppet] 07paladox 03112eee0 - Remove static-temp
[00:16:56] <Not-f817>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZ03
[00:16:58] <Not-f817>	 [02miraheze/dns] 07paladox 03f27cca8 - Remove glusterfs[12] and static-temp
[00:17:34] <Not-f817>	 [02dns] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbQXl
[00:17:36] <Not-f817>	 [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeZ0G
[00:17:37] <Not-f817>	 [02miraheze/dns] 07paladox 039a47b98 - Remove es related things
[00:17:39] <Not-f817>	 [02dns] 07paladox opened pull request 03#113: Remove es related things - 13https://git.io/JeZ0Z
[00:18:07] <Not-f817>	 [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeZ0c
[00:18:09] <Not-f817>	 [02miraheze/dns] 07paladox 038f31143 - Update config
[00:18:10] <Not-f817>	 [02dns] 07paladox synchronize pull request 03#113: Remove es related things - 13https://git.io/JeZ0Z
[00:18:22] <Not-f817>	 [02dns] 07paladox closed pull request 03#113: Remove es related things - 13https://git.io/JeZ0Z
[00:18:24] <Not-f817>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JeZ0C
[00:18:25] <Not-f817>	 [02miraheze/dns] 07paladox 031a7e431 - Remove es related things (#113) * Remove es related things * Update config
[00:18:49] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): File[unrecnations.wiki],File[unrecnations.wiki_private]
[00:18:56] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): File[unrecnations.wiki],File[unrecnations.wiki_private]
[00:19:10] <Not-f817>	 [02dns] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbQXl
[00:19:12] <Not-f817>	 [02miraheze/dns] 07paladox deleted branch 03paladox-patch-2
[00:19:13] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): File[unrecnations.wiki],File[unrecnations.wiki_private]
[00:19:18] <Not-f817>	 [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbQXl
[00:19:19] <Not-f817>	 [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1
[00:19:28] <Not-f817>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2
[00:19:29] <Not-f817>	 [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS
[00:19:31] <Not-f817>	 [02puppet] 07paladox deleted branch 03revert-1081-paladox-patch-4 - 13https://git.io/vbiAS
[00:19:32] <Not-f817>	 [02miraheze/puppet] 07paladox deleted branch 03revert-1081-paladox-patch-4
[00:19:34] <Not-f817>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[00:19:35] <Not-f817>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[00:19:36] <travis-ci>	 miraheze/dns/paladox-patch-2/8f31143 - paladox The build has errored. https://travis-ci.org/miraheze/dns/builds/590650279
[00:19:54] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): File[unrecnations.wiki],File[unrecnations.wiki_private]
[00:20:18] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[00:20:23] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): File[unrecnations.wiki],File[unrecnations.wiki_private]
[00:20:25] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): File[unrecnations.wiki],File[unrecnations.wiki_private]
[00:23:05] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[00:23:48] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[00:24:54] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[00:24:54] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures
[00:25:06] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[00:25:16] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[00:25:22] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:04:15] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:13:49] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[06:26:18] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2909 MB (12% inode=94%);
[08:26:02] <ZppixBot>	 Hello rain2! If you have any questions, feel free to ask and someone should answer soon.
[08:26:06] <rain2>	 hi
[08:26:12] <rain2>	 how do i rename a page?
[08:35:12] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZ21
[08:35:14] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 0365c129c - BOT: Updating services config for wikis
[08:36:00] <rain2>	 nvm i found the move button!
[08:40:29] <RhinosF1>	 rain2: glad you got an answer, shout if you need more help
[08:40:36] <rain2>	 thanks!
[09:14:34] <RhinosF1>	 .tell JohnLewis Why are so many common extensions locked on Managewiki? Can we consider making some of them on by default or unrestricted?
[09:14:34] <ZppixBot>	 RhinosF1: I'll pass that on when JohnLewis is around.
[09:31:08] <Reception123>	 RhinosF1: what's an example of one?
[09:59:49] <RhinosF1>	 Reception123: docx
[10:01:08] <Reception123>	 RhinosF1: ohhh I thought you meant MediaWiki extensions not file extensions
[10:02:49] <RhinosF1>	 Reception123: yeah file extensions
[10:02:59] <RhinosF1>	 Docx, mp*
[10:03:11] <RhinosF1>	 Are ones off the top of my head I can think of
[10:03:15] <Reception123>	 I guess you should ask paladox then he did those iirc
[10:03:28] <Reception123>	 but yeah, some definitely seem fine like .docx or .pptx or things like that
[10:04:40] <RhinosF1>	 paladox: ^
[10:12:58] <paladox>	 That setting is locked as we doin’t want anyone to change it without staff doing it
[10:20:23] <RhinosF1>	 paladox: is there a need though?
[10:21:01] <paladox>	 Yes
[10:21:12] <paladox>	 We doint want them to enable it freely :)
[11:42:25] <Reception123>	 paladox: yeah, but what's the reason behind that?
[11:42:34] <Reception123>	 isn't it the same thing if we enable it for them?
[12:10:12] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZwH
[12:10:13] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 03373f7db - BOT: Updating services config for wikis
[12:13:39] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%);
[12:58:43] <paladox>	 I discussss this with John, but forgot the reason :(
[12:58:57] <RhinosF1>	 Okay
[13:15:34] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2
[13:16:00] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[13:17:11] <icinga-miraheze>	 PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[13:18:50] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[13:19:13] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[13:20:09] <icinga-miraheze>	 RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24570 bytes in 1.431 second response time
[14:52:09] <RhinosF1>	 !log rhinos@mw1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=amazingreceptionwikiswiki --logwiki=metawiki 'PapercutsExist' 'PaperClip'
[14:52:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:33:27] <icinga-miraheze>	 PROBLEM - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'cyberlaw.ccdcoe.org' expires in 15 day(s) (Mon 14 Oct 2019 03:28:04 PM GMT +0000).
[15:33:41] <Not-f817>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZ6s
[15:33:43] <Not-f817>	 [02miraheze/ssl] 07MirahezeSSLBot 033522daf - Bot: Update SSL cert for cyberlaw.ccdcoe.org
[15:43:29] <icinga-miraheze>	 RECOVERY - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cyberlaw.ccdcoe.org' will expire on Fri 27 Dec 2019 02:33:35 PM GMT +0000.
[15:54:09] <Reception123>	 !log MariaDB [mhglobal]> DELETE FROM renameuser_status WHERE ru_wiki = 'awfultvshowswiki';
[15:54:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:55:03] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=awiki --logwiki=metawiki 'PapercutsExist' 'PaperClip' on mw1
[15:55:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:57:03] <RhinosF1>	 Reception123: have you checked there's no other deleted wikis left in the rename job
[15:57:49] <Reception123>	 !log reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=bakapediawiki --logwiki=metawiki 'PapercutsExist' 'PaperClip' 
[15:57:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:58:03] <Reception123>	 RhinosF1: I'll just see if it gets blocked on any other ones
[15:59:26] <RhinosF1>	 Reception123: if you're doing that then do you know the fix script fixes wikis blocked as non existent dB anyway
[16:05:03] <RhinosF1>	 Reception123: stuck for game breaking wiki
[16:05:09] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZ65
[16:05:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 03e823980 - BOT: Updating services config for wikis
[16:06:35] <Reception123>	 !log MariaDB [mhglobal]> DELETE FROM renameuser_status WHERE ru_wiki = 'gamebreakingwiki';
[16:06:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:06:55] <Reception123>	 !log reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=gamelabwiki --logwiki=metawiki 'PapercutsExist' 'PaperClip'
[16:06:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:07:46] <RhinosF1>	 Reception123: your first step is redundant if it's already stuck - only needed if you're fixing it before it breaks
[16:09:07] <Reception123>	 RhinosF1: well how else can I do it then? that's how I did it the last time at least
[16:09:19] <Reception123>	 or else there's no way to continue renaming for the other wikis
[16:10:29] <RhinosF1>	 Reception123: the fix script fixes missing DBs somehow
[16:11:13] <RhinosF1>	 So you only need to do that if you're going to do it for any deleted wiki before it's gets stuck again
[16:11:24] <Reception123>	 oh ok, last time it didn't
[16:11:50] <RhinosF1>	 It did for me
[16:12:16] <Reception123>	 !log reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=horribleobjectshowswiki --logwiki=metawiki 'PapercutsExist' 'PaperClip'
[16:12:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:13:18] <RhinosF1>	 Reception123: see
[16:13:54] <Reception123>	 yeah, don't know why it didn't last time I did it
[16:14:53] <RhinosF1>	 Reception123: strange, is it easier if you delete the entry for deleted wikis before it fails again or are you just going to keep fixing when it sticks?
[16:27:46] <ZppixBot>	 Hello volarevia! If you have any questions, feel free to ask and someone should answer soon.
[16:37:40] <RhinosF1>	 !log sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=reverseafwuwiki --logwiki=metawiki 'PapercutsExist' 'PaperClip' (attemtpting to fix rename but redis has crashed so failed)
[16:37:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:51:07] <Zppix>	 RhinosF1:  lol it just keeps stalling
[16:52:22] <RhinosF1>	 Zppix: most of it was deleted wikis but redis also decided to have a tantrum
[16:52:38] <RhinosF1>	 And still is
[17:28:49] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[17:30:19] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[17:30:20] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2
[17:30:29] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2
[17:32:20] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24570 bytes in 0.685 second response time
[17:33:09] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[17:33:11] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[17:33:20] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[18:10:22] <icinga-miraheze>	 PROBLEM - db5 Current Load on db5 is WARNING: WARNING - load average: 3.85, 2.87, 1.61
[18:12:52] <icinga-miraheze>	 RECOVERY - db5 Current Load on db5 is OK: OK - load average: 2.33, 2.69, 1.73
[18:20:57] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.57, 8.08, 5.91
[18:23:18] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.11, 7.16, 5.86
[18:25:40] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.72, 6.06, 5.63
[19:37:07] <hispano76_>	 hello
[19:48:20] <RhinosF1>	 Hi hispano76_
[20:28:05] <Zppix>	 NEW RfC fyi https://meta.miraheze.org/wiki/Requests_for_Comment/No_open_proxies_policy
[20:28:06] <ZppixBot>	 [ Requests for Comment/No open proxies policy - Miraheze Meta ] - meta.miraheze.org
[20:55:16] <RhinosF1>	 Zppix: cool
[21:28:05] <Zppix>	 paladox:  just confirming the new UI for gerrit doesn't allow web editing right?
[21:37:26] <paladox>	 yes, until 2.16
[21:43:53] <paladox>	 !log sudo -u www-data php fixStuckGlobalRename.php --wiki=reverseatrociousdeviantswiki --logwiki=metawiki PapercutsExist PaperClip
[21:44:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:45:21] <RhinosF1>	 Zppix: ^
[21:45:28] <Zppix>	 i saw
[21:46:09] <RhinosF1>	 Good
[21:49:54] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2
[21:50:21] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[21:50:28] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[21:52:40] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZSZ
[21:52:42] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb
[21:52:42] <Not-f817>	 [02miraheze/puppet] 07paladox 03eddc0dc - Update php_fpm.pp
[21:52:42] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[21:53:26] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[21:53:33] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[21:55:12] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[21:55:15] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[21:55:44] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[21:59:45] <Not-f817>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[21:59:47] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeZS4
[21:59:48] <Not-f817>	 [02miraheze/puppet] 07paladox 036f67e87 - php: Tweek where the logs are and how they are collected Also add a systemd script to prevent reload if the config is wrong. (All these changes are from https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/mediawiki/php.pp)
[21:59:49] <ZppixBot>	 [ Page not found · GitHub · GitHub ] - github.com
[21:59:52] <Not-f817>	 [02puppet] 07paladox opened pull request 03#1093: php: Tweek where the logs are and how they are collected - 13https://git.io/JeZSB
[22:00:45] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/JeZS0
[22:00:46] <Not-f817>	 [02miraheze/puppet] 07paladox 0385f3301 - Create php-fpm-systemd-override.conf.erb
[22:00:48] <Not-f817>	 [02puppet] 07paladox synchronize pull request 03#1093: php: Tweek where the logs are and how they are collected - 13https://git.io/JeZSB
[22:02:19] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JeZSz
[22:02:21] <Not-f817>	 [02miraheze/puppet] 07paladox 034b47660 - Delete php-fpm-logrotate.conf
[22:02:22] <Not-f817>	 [02puppet] 07paladox synchronize pull request 03#1093: php: Tweek where the logs are and how they are collected - 13https://git.io/JeZSB
[22:05:02] <Not-f817>	 [02puppet] 07paladox closed pull request 03#1093: php: Tweek where the logs are and how they are collected - 13https://git.io/JeZSB
[22:05:03] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/JeZS2
[22:05:05] <Not-f817>	 [02miraheze/puppet] 07paladox 03241c82f - php: Tweek where the logs are and how they are collected (#1093) * php: Tweek where the logs are and how they are collected Also add a systemd script to prevent reload if the config is wrong. (All these changes are from https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/mediawiki/php.pp) * Create php-fpm-systemd-override.conf.erb * Delete
[22:05:05] <Not-f817>	 php-fpm-logrotate.conf
[22:05:06] <ZppixBot>	 [ Page not found · GitHub · GitHub ] - github.com
[22:05:06] <Not-f817>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[22:05:08] <Not-f817>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[22:05:17] <paladox>	 !log depool mw[123] and repool
[22:06:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:06:39] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZSa
[22:06:40] <Not-f817>	 [02miraheze/puppet] 07paladox 032803ef3 - Update php_fpm.pp
[22:09:48] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.89, 7.25, 5.33
[22:10:04] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures
[22:10:11] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures
[22:12:26] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.03, 6.67, 5.40
[22:12:47] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[22:15:00] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZSP
[22:15:02] <Not-f817>	 [02miraheze/puppet] 07paladox 0325d3a86 - php: Send error logs to stderr
[22:20:30] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZSy
[22:20:32] <Not-f817>	 [02miraheze/puppet] 07paladox 03b18ea86 - Revert "php: Send error logs to stderr" This reverts commit 25d3a869470c8d7e7368b49f556a557deba9956e.
[22:21:51] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 12.56, 9.01, 6.59
[22:27:12] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZS7
[22:27:13] <Not-f817>	 [02miraheze/puppet] 07paladox 03dff0cae - Update init.pp
[22:27:31] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.22, 7.52, 6.71
[22:30:03] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.69, 5.87, 6.20
[22:32:41] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
[22:39:12] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZSp
[22:39:13] <Not-f817>	 [02miraheze/puppet] 07paladox 0371241e8 - Update pool.pp
[22:40:10] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZSh
[22:40:11] <Not-f817>	 [02miraheze/puppet] 07paladox 03ad6e7b3 - php: Lower process_control_timeout to 180
[22:42:01] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZ9T
[22:42:03] <Not-f817>	 [02miraheze/puppet] 07paladox 034448f2a - php: Lower max_execution_time to 180
[22:44:15] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZ9k
[22:44:16] <Not-f817>	 [02miraheze/puppet] 07paladox 03a8e0943 - matomo: Remove max_execution_time from matomo let it use the one defined in the php class
[22:45:21] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZ9I
[22:45:23] <Not-f817>	 [02miraheze/puppet] 07paladox 03037969c - php: Remove max_execution_time Use default from php clsss