[00:02:27] <paladox|UKInEU>	 !log drop database piwik on db5; create databse db5 & import from backup. To see if this will reduce the disk space it's using from ~75gb.
[00:02:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[00:05:41] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.499 second response time
[00:06:16] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for misc2 on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.003 second response time
[00:06:30] <icinga-miraheze>	 PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.295 second response time
[00:06:35] <icinga-miraheze>	 PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures
[00:06:35] <icinga-miraheze>	 PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.005 second response time
[00:09:58] <icinga-miraheze>	 PROBLEM - db5 Current Load on db5 is WARNING: WARNING - load average: 3.95, 3.46, 2.86
[00:14:02] <icinga-miraheze>	 PROBLEM - db5 Current Load on db5 is CRITICAL: CRITICAL - load average: 4.90, 4.02, 3.20
[00:16:02] <icinga-miraheze>	 PROBLEM - db5 Current Load on db5 is WARNING: WARNING - load average: 3.03, 3.69, 3.19
[00:22:07] <icinga-miraheze>	 RECOVERY - db5 Current Load on db5 is OK: OK - load average: 1.73, 2.95, 3.06
[00:38:55] <paladox|UKInEU>	 !log root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php rebuildall.php --wiki=csydeswiki
[00:38:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[01:40:05] <Voidwalker>	 paladox|UKInEU, is that script done running?
[01:40:13] <paladox|UKInEU>	 Voidwalker nope
[01:40:23] <paladox|UKInEU>	 it's refreshing links now.
[01:40:51] <Voidwalker>	 all right
[01:53:25] <icinga-miraheze>	 PROBLEM - db5 Disk Space on db5 is WARNING: DISK WARNING - free space: / 17640 MB (9% inode=99%);
[02:05:25] <icinga-miraheze>	 PROBLEM - db5 Disk Space on db5 is CRITICAL: DISK CRITICAL - free space: / 9884 MB (5% inode=99%);
[02:15:49] <paladox|UKInEU>	 Voidwalker it's finished
[02:17:02] <Voidwalker>	 hmm, doesn't seem to have fixed the issue, might have to go to the tech channel and ask for help debugging
[02:17:25] <icinga-miraheze>	 PROBLEM - db5 Disk Space on db5 is WARNING: DISK WARNING - free space: / 16785 MB (9% inode=99%);
[02:37:27] <icinga-miraheze>	 PROBLEM - db5 Disk Space on db5 is CRITICAL: DISK CRITICAL - free space: / 10425 MB (5% inode=99%);
[02:43:25] <icinga-miraheze>	 PROBLEM - db5 Disk Space on db5 is WARNING: DISK WARNING - free space: / 11839 MB (6% inode=99%);
[02:47:25] <icinga-miraheze>	 PROBLEM - db5 Disk Space on db5 is CRITICAL: DISK CRITICAL - free space: / 10092 MB (5% inode=99%);
[02:48:42] <paladox|UKInEU>	 Voidwalker found the cause
[02:48:43] <paladox|UKInEU>	 it's DisplayTitle
[02:48:53] <Voidwalker>	 oh?
[02:49:26] <paladox|UKInEU>	 Voidwalker yeh, debuged it to https://github.com/wikimedia/mediawiki/blob/d0fc5e00e6dfd7b8cc3fa0cc1bb94749654c01cf/includes/page/Article.php#L1115
[02:49:27] <ZppixBot>	 [ mediawiki/Article.php at d0fc5e00e6dfd7b8cc3fa0cc1bb94749654c01cf · wikimedia/mediawiki · GitHub ] - github.com
[02:49:41] <paladox|UKInEU>	 and did some backwords tracking and ended up at https://github.com/wikimedia/mediawiki/blob/master/includes/linker/LinkRenderer.php#L188
[02:49:42] <ZppixBot>	 [ mediawiki/LinkRenderer.php at master · wikimedia/mediawiki · GitHub ] - github.com
[02:49:55] <paladox|UKInEU>	 i then did a search through all extensions for ones that used it
[02:49:59] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[02:51:25] <icinga-miraheze>	 RECOVERY - db5 Disk Space on db5 is OK: DISK OK - free space: / 65064 MB (35% inode=99%);
[02:51:31] <Voidwalker>	 yeah, but this isn't an issue on all wikis
[02:55:58] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:00:23] <paladox|UKInEU>	 Voidwalker wierd, works with the ext now after clicking save on https://csydes.miraheze.org/wiki/C.Syde%27s_Wiki:User_Rights#Autoconfirmed_Users
[03:00:24] <ZppixBot>	 [ C.Syde's Wiki:User Rights - C.Syde's Wiki ] - csydes.miraheze.org
[03:15:33] <icinga-miraheze>	 RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 891 bytes in 0.592 second response time
[03:15:38] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 891 bytes in 0.666 second response time
[03:15:45] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for misc2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 891 bytes in 0.118 second response time
[03:16:32] <icinga-miraheze>	 RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 891 bytes in 0.144 second response time
[03:16:45] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:37:41] <Voidwalker>	 looks like it was disabled
[04:05:20] <paladox|UKInEU>	 Yup
[04:16:19] <Voidwalker>	 uh, the DISPLAYTITLE parser function still works without the extension
[04:17:24] <Voidwalker>	 oh, right, part of it is from MW core
[04:21:25] <icinga-miraheze>	 PROBLEM - db5 Disk Space on db5 is WARNING: DISK WARNING - free space: / 17601 MB (9% inode=99%);
[04:31:25] <icinga-miraheze>	 PROBLEM - db5 Disk Space on db5 is CRITICAL: DISK CRITICAL - free space: / 10834 MB (5% inode=99%);
[04:33:26] <icinga-miraheze>	 RECOVERY - db5 Disk Space on db5 is OK: DISK OK - free space: / 56621 MB (30% inode=99%);
[04:34:38] <paladox|UKInEU>	 !log hack /var/spool/cron/crontabs/root and run /home/paladox/purge-binary.sh every 1 minute.
[04:34:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[04:37:40] <icinga-miraheze>	 PROBLEM - db5 Puppet on db5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures
[06:26:51] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2960 MB (12% inode=94%);
[07:00:13] <RhinosF1>	 .github miraheze/createwiki
[07:00:13] <ZppixBot>	 https://github.com/miraheze/createwiki
[10:01:51] <ZppixBot>	 Hello Suarez2020! If you have any questions, feel free to ask and someone should answer soon.
[10:02:09] <Suarez2020>	 RhinosF1 Lock globally to Suarez2020
[10:02:16] <Suarez2020>	 RhinosF1 Lock globally to Suarez2020
[10:02:17] <Suarez2020>	 RhinosF1 Lock globally to Suarez2020
[10:02:17] <Suarez2020>	 RhinosF1 Lock globally to Suarez2020
[10:03:53] <RhinosF1>	 Reception123: ^
[10:03:56] <RhinosF1>	 grumble: ^
[10:04:47] <RhinosF1>	 Reception123: too wide of a range to be safe blocking without cu
[10:05:04] <RhinosF1>	 PuppyKun, SPF|Cloud: can we cu for on wiki ip block?
[10:05:07] <RhinosF1>	 See -cvt
[10:05:29] <Reception123>	 RhinosF1: hm? 
[10:05:35] <Reception123>	 oh you mean on wiki yeah
[10:05:43] <Reception123>	 I leave that to Stewards anyway so they can have a real check with CU
[10:08:42] <RhinosF1>	 Reception123: we’d need to. I’m not comfortable blocking a range that wide without a proper look
[10:08:55] <Reception123>	 yeah
[11:08:49] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2648 MB (10% inode=94%);
[13:55:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvsZW
[13:55:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03676e127 - BOT: Updating services config for wikis
[14:03:19] <paladox|UKInEU>	 !log remove hack from cron - db5
[14:03:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:05:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvsZz
[14:05:08] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03a4d92b9 - BOT: Updating services config for wikis
[14:06:32] <icinga-miraheze>	 PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[14:12:04] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. lizardfs6
[14:12:39] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. lizardfs6
[14:13:39] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. lizardfs6
[14:16:37] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy
[14:17:39] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy
[14:18:05] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy
[14:22:47] <icinga-miraheze>	 PROBLEM - lizardfs6 Puppet on lizardfs6 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 57 seconds ago with 0 failures
[15:48:07] <paladox|UKInEU>	 !log > MatomoAnalytics::deleteSite( 'davshiqwiki' );
[15:48:12] <paladox|UKInEU>	 !log > MatomoAnalytics::addSite( 'davshiqwiki' );
[15:48:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:48:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:48:30] <paladox|UKInEU>	 !log > MatomoAnalytics::deleteSite( 'foxmaskeventwiki' );
[15:48:35] <paladox|UKInEU>	 !log > MatomoAnalytics::addSite( 'foxmaskeventwiki' );
[15:48:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:48:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:50:07] <paladox|UKInEU>	 !log > MatomoAnalytics::deleteSite( 'loginwiki' );
[15:50:13] <paladox|UKInEU>	 !log > MatomoAnalytics::addSite( 'loginwiki' );
[15:50:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:50:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:20:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvs8U
[17:20:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03209b357 - BOT: Updating services config for wikis
[18:52:10] <RhinosF1>	 Voidwalker: can you cu ips from this mornings lock
[18:55:38] <Voidwalker>	 shoot, lemme pull up the IP from earlier
[18:56:27] <RhinosF1>	 Voidwalker: if you know the ranges, can you irc + global ban em
[19:53:51] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvsuJ
[19:53:52] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03c18a517 - Fix wgManageWikiNamespacesAdditional
[19:53:54] <Not-5a0a>	 [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3
[19:53:56] <Not-5a0a>	 [02mw-config] 07paladox opened pull request 03#2870: Fix wgManageWikiNamespacesAdditional - 13https://git.io/JvsuT
[19:54:03] <Not-5a0a>	 [02mw-config] 07paladox closed pull request 03#2870: Fix wgManageWikiNamespacesAdditional - 13https://git.io/JvsuT
[19:54:04] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvsuk
[19:54:06] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 033070e9f - Fix wgManageWikiNamespacesAdditional (#2870)
[19:54:21] <Not-5a0a>	 [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3
[19:54:23] <Not-5a0a>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1
[19:55:16] <travis-ci>	 miraheze/mw-config/paladox-patch-1/c18a517 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/645168798
[19:55:26] <icinga-miraheze>	 PROBLEM - www.netazar.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.netazar.org' expires in 15 day(s) (Tue 18 Feb 2020 07:52:57 PM GMT +0000).
[19:55:33] <icinga-miraheze>	 PROBLEM - netazar.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.netazar.org' expires in 15 day(s) (Tue 18 Feb 2020 07:52:57 PM GMT +0000).
[19:55:34] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03revert-2870-paladox-patch-1 [+0/-0/±1] 13https://git.io/Jvsut
[19:55:36] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 0357c3681 - Revert "Fix wgManageWikiNamespacesAdditional (#2870)" This reverts commit 3070e9fef0a230e7b3b63063cd50c576525b6e23.
[19:55:37] <Not-5a0a>	 [02mw-config] 07paladox created branch 03revert-2870-paladox-patch-1 - 13https://git.io/vbvb3
[19:55:38] <Not-5a0a>	 [02mw-config] 07paladox opened pull request 03#2871: Revert "Fix wgManageWikiNamespacesAdditional" - 13https://git.io/Jvsuq
[19:55:44] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvsum
[19:55:44] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 037b6ca0a - Bot: Update SSL cert for www.netazar.org
[19:55:44] <Not-5a0a>	 [02mw-config] 07paladox closed pull request 03#2871: Revert "Fix wgManageWikiNamespacesAdditional" - 13https://git.io/Jvsuq
[19:55:46] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvsuO
[19:55:47] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03993ec96 - Revert "Fix wgManageWikiNamespacesAdditional (#2870)" (#2871) This reverts commit 3070e9fef0a230e7b3b63063cd50c576525b6e23.
[19:55:53] <Not-5a0a>	 [02mw-config] 07paladox deleted branch 03revert-2870-paladox-patch-1 - 13https://git.io/vbvb3
[19:55:55] <Not-5a0a>	 [02miraheze/mw-config] 07paladox deleted branch 03revert-2870-paladox-patch-1
[19:57:18] <travis-ci>	 miraheze/mw-config/revert-2870-paladox-patch-1/57c3681 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/645169500
[20:02:45] * hispano76 greetings
[20:03:21] <icinga-miraheze>	 RECOVERY - www.netazar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.netazar.org' will expire on Sat 02 May 2020 06:55:35 PM GMT +0000.
[20:03:32] <icinga-miraheze>	 RECOVERY - netazar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.netazar.org' will expire on Sat 02 May 2020 06:55:35 PM GMT +0000.
[21:41:28] <icinga-miraheze>	 PROBLEM - wiki.lbcomms.co.za - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lbcomms.co.za' expires in 15 day(s) (Tue 18 Feb 2020 09:38:48 PM GMT +0000).
[21:41:42] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvs20
[21:41:43] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 03ef29b86 - Bot: Update SSL cert for wiki.lbcomms.co.za
[21:53:27] <icinga-miraheze>	 RECOVERY - wiki.lbcomms.co.za - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lbcomms.co.za' will expire on Sat 02 May 2020 08:41:35 PM GMT +0000.
[23:02:43] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb
[23:04:41] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[23:16:20] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu]
[23:22:23] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures