[00:00:58] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 1812 MB (7% inode=93%); [00:02:28] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 32.66, 24.69, 17.60 [00:02:57] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2914 MB (12% inode=93%); [00:04:25] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.75, 21.91, 17.44 [00:06:25] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.81, 19.07, 16.93 [01:08:16] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03revert-131-paladox-patch-3 [+0/-0/±1] 13https://git.io/JJ5zQ [01:08:18] [02miraheze/MirahezeMagic] 07paladox 0314180c0 - Revert "Revert "If a wiki dbname is larger then 32, ignore it" (#131)" This reverts commit 472a8acfcbb9798c37c8df88200cf23354209326. [01:08:19] [02MirahezeMagic] 07paladox created branch 03revert-131-paladox-patch-3 - 13https://git.io/fQRGX [01:08:21] [02MirahezeMagic] 07paladox opened pull request 03#142: Revert "Revert "If a wiki dbname is larger then 32, ignore it"" - 13https://git.io/JJ5z7 [01:08:31] [02MirahezeMagic] 07paladox closed pull request 03#142: Revert "Revert "If a wiki dbname is larger then 32, ignore it"" - 13https://git.io/JJ5z7 [01:08:33] [02miraheze/MirahezeMagic] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JJ5z5 [01:08:34] [02miraheze/MirahezeMagic] 07paladox 0395562a2 - Merge pull request #142 from miraheze/revert-131-paladox-patch-3 Revert "Revert "If a wiki dbname is larger then 32, ignore it"" [01:08:36] [02MirahezeMagic] 07paladox deleted branch 03revert-131-paladox-patch-3 - 13https://git.io/fQRGX [01:08:37] [02miraheze/MirahezeMagic] 07paladox deleted branch 03revert-131-paladox-patch-3 [01:09:12] <07EAALKSK> miraheze/MirahezeMagic/revert-131-paladox-patch-3/14180c0 - paladox The build has errored. https://travis-ci.org/miraheze/MirahezeMagic/builds/718096566 [01:09:12] <07IAAYSEU> miraheze/MirahezeMagic/revert-131-paladox-patch-3/14180c0 - paladox The build has errored. https://travis-ci.com/miraheze/MirahezeMagic/builds/179814066 [01:09:12] [ Travis CI - Test and Deploy Your Code with Confidence ] - travis-ci.org [01:09:13] [ Travis CI - Test and Deploy with Confidence ] - travis-ci.com [01:09:30] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJ5zb [01:09:32] [02miraheze/mediawiki] 07paladox 03012d0f3 - Update MirahezeMagic [01:21:39] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:21:41] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:22:19] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:22:41] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:22:50] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:22:59] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:23:21] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:24:39] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [01:24:58] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [01:25:37] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:25:38] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.58, 21.55, 18.07 [01:26:18] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:27:33] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.31, 20.56, 18.12 [01:27:38] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [01:29:22] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:29:26] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.15, 19.59, 18.05 [03:20:16] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ5Vb [03:20:18] [02miraheze/services] 07MirahezeSSLBot 034dcb6a5 - BOT: Updating services config for wikis [03:30:41] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2646 MB (10% inode=93%); [03:50:20] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ5wd [03:50:21] [02miraheze/services] 07MirahezeSSLBot 0375f9ae7 - BOT: Updating services config for wikis [04:23:07] PROBLEM - wiki.warframestat.us - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.warframestat.us' expires in 15 day(s) (Mon 31 Aug 2020 04:18:55 GMT +0000). [04:28:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ5r5 [04:28:08] [02miraheze/ssl] 07MirahezeSSLBot 030741369 - Bot: Update SSL cert for wiki.warframestat.us [04:32:53] PROBLEM - wiki.shaazzz.ir - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.shaazzz.ir' expires in 15 day(s) (Mon 31 Aug 2020 04:25:59 GMT +0000). [04:36:50] RECOVERY - wiki.warframestat.us - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.warframestat.us' will expire on Fri 13 Nov 2020 03:27:58 GMT +0000. [04:37:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ5rp [04:37:44] [02miraheze/ssl] 07MirahezeSSLBot 031162e04 - Bot: Update SSL cert for wiki.shaazzz.ir [04:46:52] RECOVERY - wiki.shaazzz.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.shaazzz.ir' will expire on Fri 13 Nov 2020 03:37:35 GMT +0000. [05:15:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ5oi [05:15:13] [02miraheze/services] 07MirahezeSSLBot 03185aa97 - BOT: Updating services config for wikis [06:10:27] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ56R [06:10:29] [02miraheze/services] 07MirahezeSSLBot 03aba0ed2 - BOT: Updating services config for wikis [06:17:41] It's interesing, spammers have confirmed mail... [06:45:34] RECOVERY - misc1 APT on misc1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:55:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ5XW [07:55:16] [02miraheze/services] 07MirahezeSSLBot 03637cba2 - BOT: Updating services config for wikis [08:00:25] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ5XM [08:00:26] [02miraheze/services] 07MirahezeSSLBot 0336229f3 - BOT: Updating services config for wikis [11:44:41] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.14, 20.75, 17.14 [11:46:36] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.68, 17.51, 16.36 [12:37:13] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [12:38:00] Huh [12:39:13] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 3% [12:50:22] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ55W [12:50:24] [02miraheze/services] 07MirahezeSSLBot 03b65f2f8 - BOT: Updating services config for wikis [12:52:58] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [13:04:44] .op [13:04:45] Please wait... [13:04:52] .deop [13:23:55] PROBLEM - gluster2 APT on gluster2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [13:24:00] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1445 MB (5% inode=93%); [14:21:30] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.81, 19.63, 17.60 [14:23:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.69, 20.68, 18.22 [14:25:20] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.23, 18.74, 17.80 [14:34:55] PROBLEM - jobrunner2 APT on jobrunner2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [14:39:25] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (3 critical updates). [14:56:17] PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [15:05:38] PROBLEM - mw6 APT on mw6 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [15:10:26] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.70, 21.79, 18.31 [15:12:24] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.98, 19.84, 18.04 [15:16:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.30, 21.43, 19.08 [15:18:24] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.95, 20.17, 18.92 [15:45:26] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (3 critical updates). [16:00:59] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.23, 19.89, 18.31 [16:02:53] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.75, 19.06, 18.19 [16:39:17] PROBLEM - services2 APT on services2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:43:47] PROBLEM - phab1 APT on phab1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:00:24] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.20, 20.96, 17.27 [17:02:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.33, 21.65, 17.99 [17:04:25] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.93, 23.75, 19.17 [17:08:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.89, 21.32, 19.34 [17:10:24] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.60, 18.92, 18.67 [17:59:27] PROBLEM - cp9 APT on cp9 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [18:14:34] PROBLEM - puppet2 Current Load on puppet2 is CRITICAL: CRITICAL - load average: 8.38, 5.55, 4.21 [18:15:03] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.00, 5.68, 4.72 [18:16:29] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 1.75, 4.08, 3.84 [18:17:01] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 2.28, 4.40, 4.37 [18:54:14] RhinosF1 around? [18:54:27] I a [18:54:29] I am [18:54:35] [a01a6107c854431411539b50] 2020-08-15 18:53:04: Fatal exception of type "Error" [18:55:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJdU1 [18:55:14] [02miraheze/services] 07MirahezeSSLBot 038f89d92 - BOT: Updating services config for wikis [18:55:36] @MrJaroslavik: where? [18:57:43] Now sorted [18:57:50] Sorry for bother you [18:58:36] PROBLEM - wiki.tallguysfree.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.tallguysfree.com could not be found [19:00:49] > [a01a6107c854431411539b50] 2020-08-15 18:53:04: Fatal exception of type "Error" No bother; those sort of errors are usually indicative of a bug somewhere. For example, an error like that is how we found the "data is too long for column" error in CreateWiki. Do you remember what page that was on? [19:01:49] Cannot post here, racism things on allthetropicwiki [19:02:05] @MrJarsolavik: dm it on discord [19:02:48] @MrJaroslavik Okay, yeah, that's fine. [19:05:17] RECOVERY - wiki.tallguysfree.com - reverse DNS on sslhost is OK: rDNS OK - wiki.tallguysfree.com reverse DNS resolves to cp7.miraheze.org [19:16:19] Lol, my first cross-wiki vandal actions are done 😆 on 4 wikis! [19:17:05] oh okay [19:17:14] RhinosF1 Thanks for #cvt-feed, its very useful! [19:34:13] PROBLEM - puppet2 Current Load on puppet2 is WARNING: WARNING - load average: 7.53, 5.10, 3.94 [19:34:29] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 6.92, 5.36, 4.26 [19:34:48] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.15, 5.78, 4.59 [19:36:09] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 2.31, 4.23, 3.78 [19:36:29] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 3.04, 4.59, 4.13 [19:36:43] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 3.12, 4.75, 4.36 [19:42:37] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 8.90, 6.16, 4.94 [19:46:33] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 3.85, 5.64, 5.07 [19:47:03] @MrJarsolavik: No problem [19:57:14] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.31, 21.05, 17.99 [19:59:08] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.91, 19.75, 17.92 [21:02:09] [02mw-config] 07dmehus opened pull request 03#3207: Update LocalSettings.php - 13https://git.io/JJdtT [21:14:31] @Zppix, can you DM me please? Either Discord or IRC is fine. [21:20:25] [02dns] 07RhinosF1 opened pull request 03#171: depool tools1.mh.wiki from bots website DNS - 13https://git.io/JJdtZ [21:20:33] paladox: ^ [21:21:05] [02dns] 07paladox closed pull request 03#171: depool tools1.mh.wiki from bots website DNS - 13https://git.io/JJdtZ [21:21:09] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/d20ddf4c2b3b...19748e5899e2 [21:21:10] [ Comparing d20ddf4c2b3b...19748e5899e2 · miraheze/dns · GitHub ] - github.com [21:21:11] [02miraheze/dns] 07RhinosF1 0319748e5 - depool tools1.mh.wiki from bots website DNS (#171) [21:21:30] ty paladox [21:43:03] dmehus, @MrJaroslavik: 2020-08-15 21:40:47 mw5 allthetropeswiki: [04d1e11307fd4fcebc7eeea0] Error from line 73 of /srv/mediawiki/w/extensions/Flow/includes/Content/BoardContent.php: Call to a member function getAlphadecimal() on null [21:43:36] https://www.irccloud.com/pastebin/7zXqscDu/ [21:43:36] [ Snippet | IRCCloud ] - www.irccloud.com [21:46:13] @RhinosF1, ah, thanks for the trace. So it's something to do, presumably, with a globalsysop deleting something on a Flow page, from the looks of it. I have no idea what that something is or how to fix it, though. :( [21:48:23] dmehus: the error makes me surprised anyone could delete it but let's test my php skills [21:48:46] oh the actual page never was [21:56:45] @RhinosF1, even local admins? Interesting. [21:58:29] yes [22:04:14] dmehus: if you want you can file a bug upstream for me [22:10:22] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJdq4 [22:10:24] [02miraheze/mw-config] 07RhinosF1 032dcf099 - revert scratch addition to cw blacklist This reverts commit 18d564c9 [22:11:16] @RhinosF1, okay, sure, which WMF project should I use? [22:11:41] dmehus: Structured Discussions and User-RhinosF1 [22:12:22] @RhinosF1 ah, okay [22:15:21] @RhinosF1 Regarding the above commit 2dcf099, I think you will need to merge my outstanding PR (https://github.com/miraheze/mw-config/pull/3207) into your PR (or did you push yours without doing a PR?)? In that case, how should I close my PR, should I just link directly to your commit or is there a push request I should link it to? [22:15:22] [ Update LocalSettings.php by dmehus · Pull Request #3207 · miraheze/mw-config · GitHub ] - github.com [22:17:58] dmehus: I pushed mine directly and completely forgot about that, erm just close it. [22:18:04] [02mw-config] 07RhinosF1 closed pull request 03#3207: Update LocalSettings.php - 13https://git.io/JJdtT [22:18:22] done [22:20:39] @RhinosF1, thanks! :) [22:22:09] https://phabricator.wikimedia.org/T260486 [22:22:10] [ ⚓ T260486 Fatal exception error when deleting a Structured Discussions topic or a post on a topic ] - phabricator.wikimedia.org [22:22:38] seen [22:23:55] [02dns] 07RhinosF1 opened pull request 03#172: Revert "depool tools1.mh.wiki from bots website DNS (#171)" - 13https://git.io/JJdq5 [22:23:59] paladox: ^ [22:24:16] [02dns] 07paladox closed pull request 03#172: Revert "depool tools1.mh.wiki from bots website DNS (#171)" - 13https://git.io/JJdq5 [22:24:18] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJdqF [22:24:19] [02miraheze/dns] 07RhinosF1 039dfa21d - Revert "depool tools1.mh.wiki from bots website DNS (#171)" (#172) This reverts commit 19748e5899e27c35ca2c4e30281aa6e691f18906. [22:24:29] ty [22:26:03] No problem. Was going to Miraheze-Linked column, but you'd already done it, so that's good. [22:27:55] ye