[00:01:15] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [00:01:22] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JsLTr [00:01:24] [02miraheze/DataDump] 07Universal-Omega 03c22e63d - Update ApiViewDumps.php [00:01:33] [02DataDump] 07Universal-Omega synchronize pull request 03#26: T6699: DataDump improvements - 13https://git.io/JsLJl [00:02:30] miraheze/DataDump - Universal-Omega the build passed. [00:03:14] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.08, 19.24, 14.20 [00:04:13] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown) [00:05:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JsLTA [00:05:14] [02miraheze/services] 07MirahezeSSLBot 033381e6f - BOT: Updating services config for wikis [00:05:54] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.82, 5.84, 4.41 [00:07:54] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.30, 5.11, 4.31 [00:09:09] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 34.63, 22.73, 16.98 [00:11:08] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.98, 20.74, 16.92 [00:15:09] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 8.87, 15.60, 15.84 [00:24:11] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.69, 1.77, 1.25 [00:26:11] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.68, 1.35, 1.15 [00:30:01] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [00:39:29] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JsLqq [00:39:31] [02miraheze/DataDump] 07Universal-Omega 03b7bb522 - Update ApiViewDumps.php [00:39:32] [02DataDump] 07Universal-Omega synchronize pull request 03#26: T6699: DataDump improvements - 13https://git.io/JsLJl [00:40:29] miraheze/DataDump - Universal-Omega the build passed. [01:16:48] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JsL3q [01:16:49] [02miraheze/ManageWiki] 07Universal-Omega 038c6c7d4 - Update ManageWikiFormFactoryBuilder.php [01:16:51] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [01:17:03] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [01:17:18] [02ManageWiki] 07Universal-Omega opened pull request 03#274: T6576: Allow Categorising Exemptions from Inactivity by reason - 13https://git.io/JsL3c [01:18:21] miraheze/ManageWiki - Universal-Omega the build passed. [01:25:54] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [01:39:42] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [02:02:23] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [02:37:03] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [02:46:03] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [02:49:29] [02ManageWiki] 07Universal-Omega commented on pull request 03#274: T6576: Allow Categorising Exemptions from Inactivity by reason - 13https://git.io/JsLz2 [03:02:58] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JsL2V [03:02:59] [02miraheze/ManageWiki] 07Universal-Omega 033087640 - Update ManageWikiFormFactoryBuilder.php [03:03:01] [02ManageWiki] 07Universal-Omega synchronize pull request 03#274: T6576: Allow Categorising Exemptions from Inactivity by reason - 13https://git.io/JsL3c [03:04:01] miraheze/ManageWiki - Universal-Omega the build passed. [03:04:12] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JsL2Q [03:04:14] [02miraheze/ManageWiki] 07Universal-Omega 03ae80c78 - Update ManageWikiFormFactoryBuilder.php [03:04:15] [02ManageWiki] 07Universal-Omega synchronize pull request 03#274: T6576: Allow Categorising Exemptions from Inactivity by reason - 13https://git.io/JsL3c [03:04:38] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JsL2N [03:04:40] [02miraheze/ManageWiki] 07Universal-Omega 0359cb5c8 - Update ManageWikiFormFactoryBuilder.php [03:04:42] [02ManageWiki] 07Universal-Omega synchronize pull request 03#274: T6576: Allow Categorising Exemptions from Inactivity by reason - 13https://git.io/JsL3c [03:05:11] miraheze/ManageWiki - Universal-Omega the build passed. [03:05:36] miraheze/ManageWiki - Universal-Omega the build passed. [04:22:27] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [04:24:57] [02puppet] 07Universal-Omega commented on pull request 03#1745: CSP: add iwiki.eu.org mirrors - 13https://git.io/JsLSc [04:52:02] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [04:58:59] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 7.82, 8.03, 3.63 [05:02:57] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.24, 3.72, 2.84 [05:04:55] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.28, 2.59, 2.53 [05:05:48] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [05:14:41] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [05:35:03] PROBLEM - wiki.jumpbound.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.jumpbound.com' expires in 15 day(s) (Sat 29 May 2021 05:29:54 GMT +0000). [05:43:17] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JsLxx [05:43:18] [02miraheze/ssl] 07MirahezeSSLBot 03672001c - Bot: Update SSL cert for wiki.jumpbound.com [05:49:05] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [06:05:36] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/Jstec [06:05:37] [02miraheze/dns] 07Reception123 038e29e82 - Update and rename biblestrength.org to biblestrength.net [06:08:27] [02puppet] 07Universal-Omega opened pull request 03#1753: Map IRC nick to wiki username for MirahezeLogbot - 13https://git.io/JsteF [06:09:12] RECOVERY - wiki.jumpbound.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jumpbound.com' will expire on Wed 11 Aug 2021 04:43:11 GMT +0000. [06:10:10] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/Jstvt [06:10:12] [02miraheze/ssl] 07Reception123 03d40eba1 - rename biblestrength.org to biblestrength.net [06:11:07] [02puppet] 07Reception123 closed pull request 03#1753: Map IRC nick to wiki username for MirahezeLogbot - 13https://git.io/JsteF [06:11:08] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jstvc [06:11:10] [02miraheze/puppet] 07Universal-Omega 0319e5212 - Map IRC nick to wiki username for MirahezeLogbot (#1753) [06:13:25] RECOVERY - mw10 APT on mw10 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:14:30] RECOVERY - mw8 APT on mw8 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:14:50] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [06:15:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jstfv [06:15:13] [02miraheze/services] 07MirahezeSSLBot 039d8572e - BOT: Updating services config for wikis [06:19:14] RECOVERY - test3 APT on test3 is OK: APT OK: 30 packages available for upgrade (0 critical updates). [06:22:28] RECOVERY - jobrunner3 APT on jobrunner3 is OK: APT OK: 31 packages available for upgrade (0 critical updates). [06:30:34] RECOVERY - mw9 APT on mw9 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:32:15] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [06:35:09] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:45:52] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [06:46:02] RECOVERY - jobrunner4 APT on jobrunner4 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:48:18] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 9.98, 4.24, 1.85 [06:50:18] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 2.10, 3.44, 1.87 [06:52:18] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 7.40, 7.50, 3.75 [06:54:36] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [06:55:31] RECOVERY - mw11 APT on mw11 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:56:18] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 0.41, 3.47, 2.93 [06:58:18] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 1.40, 2.65, 2.68 [07:08:11] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [07:23:44] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [07:25:25] PROBLEM - index.pdcommunity.ir - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for index.pdcommunity.ir could not be found [07:32:07] RECOVERY - index.pdcommunity.ir - reverse DNS on sslhost is OK: rDNS OK - index.pdcommunity.ir reverse DNS resolves to cp11.miraheze.org [07:37:20] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [08:07:15] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [08:20:52] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [08:36:26] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 2.78, 3.61, 1.95 [08:38:26] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.50, 2.47, 1.74 [08:55:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jsta3 [08:55:12] [02miraheze/services] 07MirahezeSSLBot 03f883d2a - BOT: Updating services config for wikis [08:55:43] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JstaE [08:55:45] [02miraheze/mediawiki] 07Reception123 039c8d32d - Update AbuseFilter [09:10:19] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [09:10:21] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [09:10:29] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [09:10:46] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [09:11:17] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [09:11:21] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [09:11:47] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [09:14:46] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [09:15:17] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [09:15:20] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [09:16:17] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:23:57] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [09:32:39] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [09:33:46] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [09:34:30] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [10:06:26] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.73, 3.36, 1.86 [10:06:38] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [10:08:27] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.61, 2.71, 1.81 [10:15:19] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [10:25:05] PROBLEM - bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bots.miraheze.wiki could not be found [10:25:07] PROBLEM - www.permanentfuturelab.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.permanentfuturelab.wiki could not be found [10:26:01] PROBLEM - vortekspedia.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vortekspedia.tk could not be found [10:26:04] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mineland.eu could not be found [10:26:05] PROBLEM - puardian.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for puardian.xyz could not be found [10:26:26] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 3.30, 10.26, 5.96 [10:28:54] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [10:31:47] RECOVERY - bots.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - bots.miraheze.wiki reverse DNS resolves to cp11.miraheze.org [10:32:06] RECOVERY - www.permanentfuturelab.wiki - reverse DNS on sslhost is OK: rDNS OK - www.permanentfuturelab.wiki reverse DNS resolves to cp11.miraheze.org [10:32:51] RECOVERY - vortekspedia.tk - reverse DNS on sslhost is OK: rDNS OK - vortekspedia.tk reverse DNS resolves to cp10.miraheze.org [10:32:57] RECOVERY - wiki.mineland.eu - reverse DNS on sslhost is OK: rDNS OK - wiki.mineland.eu reverse DNS resolves to cp10.miraheze.org [10:33:04] RECOVERY - puardian.xyz - reverse DNS on sslhost is OK: rDNS OK - puardian.xyz reverse DNS resolves to cp11.miraheze.org [10:34:27] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.70, 2.61, 3.81 [10:36:26] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.39, 1.86, 3.39 [10:40:27] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 2.22, 2.68, 3.41 [10:42:27] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.41, 2.59, 3.32 [11:29:32] PROBLEM - wiki.park-city.club - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.park-city.club could not be found [11:29:35] PROBLEM - en.pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.pornwiki.org could not be found [11:29:38] PROBLEM - tl.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tl.awiki.org could not be found [11:29:40] PROBLEM - kkutu.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kkutu.wiki could not be found [11:29:42] PROBLEM - www.ferrandalmeida.family - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.ferrandalmeida.family could not be found [11:29:44] PROBLEM - www.permanentfuturelab.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.permanentfuturelab.wiki could not be found [11:29:47] PROBLEM - equestripedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for equestripedia.org could not be found [11:36:21] RECOVERY - wiki.park-city.club - reverse DNS on sslhost is OK: rDNS OK - wiki.park-city.club reverse DNS resolves to cp10.miraheze.org [11:36:25] RECOVERY - en.pornwiki.org - reverse DNS on sslhost is OK: rDNS OK - en.pornwiki.org reverse DNS resolves to cp11.miraheze.org [11:36:27] RECOVERY - equestripedia.org - reverse DNS on sslhost is OK: rDNS OK - equestripedia.org reverse DNS resolves to cp11.miraheze.org [11:36:28] RECOVERY - tl.awiki.org - reverse DNS on sslhost is OK: rDNS OK - tl.awiki.org reverse DNS resolves to cp10.miraheze.org [11:36:33] RECOVERY - www.ferrandalmeida.family - reverse DNS on sslhost is OK: rDNS OK - www.ferrandalmeida.family reverse DNS resolves to cp10.miraheze.org [11:36:35] RECOVERY - kkutu.wiki - reverse DNS on sslhost is OK: rDNS OK - kkutu.wiki reverse DNS resolves to cp10.miraheze.org [11:36:41] RECOVERY - www.permanentfuturelab.wiki - reverse DNS on sslhost is OK: rDNS OK - www.permanentfuturelab.wiki reverse DNS resolves to cp11.miraheze.org [11:51:34] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JsqJJ [11:51:36] [02miraheze/MirahezeMagic] 07translatewiki 035e2f300 - Localisation updates from https://translatewiki.net. [11:51:37] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JsqJU [11:51:38] [ Main page - translatewiki.net ] - translatewiki.net [11:51:39] [02miraheze/ManageWiki] 07translatewiki 0376fd824 - Localisation updates from https://translatewiki.net. [11:51:40] [ Main page - translatewiki.net ] - translatewiki.net [11:51:40] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JsqJT [11:51:42] [02miraheze/WikiDiscover] 07translatewiki 03de78f5f - Localisation updates from https://translatewiki.net. [11:51:45] [ Main page - translatewiki.net ] - translatewiki.net [11:52:38] miraheze/MirahezeMagic - translatewiki the build passed. [11:52:45] miraheze/ManageWiki - translatewiki the build passed. [11:52:48] miraheze/WikiDiscover - translatewiki the build passed. [13:24:02] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 3.46, 5.91, 3.68 [13:27:55] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.86, 3.14, 3.04 [14:18:16] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [14:30:36] [02mw-config] 07R4356th opened pull request 03#3892: Header modification for snapwikiwiki - 13https://git.io/Jsqig [14:31:41] miraheze/mw-config - R4356th the build passed. [14:31:51] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [14:51:58] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.81, 20.38, 15.85 [14:53:57] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.45, 19.19, 15.97 [14:55:04] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.37, 8.69, 6.37 [14:57:43] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.58, 7.35, 5.80 [15:01:03] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.57, 7.76, 6.81 [15:01:33] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.84, 6.69, 5.88 [15:07:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.73, 6.56, 6.60 [15:26:44] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 8.56, 4.53, 2.60 [15:28:17] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [15:32:34] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 2.26, 3.63, 2.92 [15:34:30] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.82, 3.09, 2.80 [15:39:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.66, 9.03, 6.48 [15:40:43] SPF|Cloud: is there a way we can tell what domains MediaWiki app servers are making outgoing HTTP requests too? [15:40:56] Specifically MediaWiki but from us not on client end [15:41:22] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.66, 7.73, 6.33 [15:41:25] WMF are getting new dc and I want to be able to see if we can monitor times from us to respond if we end up hitting it in case of impact [15:41:30] As I think we likely will [15:41:52] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [15:45:23] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.62, 6.38, 6.06 [16:14:25] RhinosF1: the easiest method is https://github.com/miraheze/mw-config/blob/master/LocalSettings.php#L4469 setting the 'http' channel to 'debug' [16:14:25] [ mw-config/LocalSettings.php at master · miraheze/mw-config · GitHub ] - github.com [16:17:59] unfortunately, not all extensions use the built-in http client classes (e.g. https://github.com/miraheze/RottenLinks/blob/master/includes/RottenLinks.php#L6 accesses curl directly) [16:18:00] [ RottenLinks/RottenLinks.php at master · miraheze/RottenLinks · GitHub ] - github.com [16:23:00] SPF|Cloud: might be good to do that at some point then [16:23:15] I'll create a task for it [16:23:41] a grep -r in extensions/ tells me most extensions use the built-in classes [16:24:10] so the 'http' log channel is trustworthy [16:28:35] paladox: any idea about https://phabricator.miraheze.org/T7287 ? [16:28:36] [ ⚓ T7287 /mnt/mediawiki-static failures on mw8; missing thumbnails in places ] - phabricator.miraheze.org [16:28:45] seems like your change in https://phabricator.miraheze.org/T7134#141661 hasn't solved the problem unfortunately :( [16:28:46] [ ⚓ T7134 Puppet cannot remount GlusterFS mount if directory exists ] - phabricator.miraheze.org [16:29:06] SPF|Cloud: nice, would be good to have a dashboard or some nice way to see latency stats [16:29:39] Reception123: no apart from remounting. [16:30:15] it just OOMs and the last time i debugged this https://github.com/gluster/glusterfs/issues/1370 [16:30:16] [ Gluster Client OOMs · Issue #1370 · gluster/glusterfs · GitHub ] - github.com [16:31:28] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [16:32:05] paladox: yeah, though that is the issue, that puppet still doesn't manage to automatically remount the directory [16:32:12] it does after I do sudo umount but not by itself, someone has to intervene [16:32:22] yeh [16:34:01] paladox: would it maybe be because puppet doesn't recognise that the it isn't mounted anymore? And that's why it only mounts itself automatically after someone manually does sudo umount? [16:34:35] It'll be because puppet doesn't unmount and mount it does exactly what you tell it to do which is mount [16:35:01] i don't think it checks for if it's mounted either [16:35:25] ah, so could we fix that? [16:37:11] No i don't think we can. We could try and script that runs every minute i guess. [16:42:52] paladox: yeah that would be nice, as it would be bad if the mount unmounts when no one is around and causes images to lag until someone is active [16:44:42] I think it's worth trying at least [16:45:04] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [16:58:22] paladox: what about check gluster clients [16:58:27] The icinga check currently [16:58:30] That didn't fire [17:04:34] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 7.77, 5.15, 3.05 [17:04:49] I have no idea about that one. It only checks if the process exists. [17:06:30] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.55, 3.63, 2.73 [17:08:27] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.93, 2.78, 2.53 [17:13:12] https://phabricator.miraheze.org/T7297 here you go :) [17:13:13] [ ⚓ T7297 RottenLinks should use HttpRequestFactory ] - phabricator.miraheze.org [17:13:21] [02DataDump] 07Universal-Omega edited pull request 03#26: T6699: DataDump improvements - 13https://git.io/JsLJl [17:14:27] it's not completely true I didn't test it, but someone must verify the function does not return garbage values for URLs [17:15:50] Ty SPF|Cloud [17:16:17] Could we for the logging use a sample for debug but all for error [17:16:59] I think you can find examples of that in LocalSettings.php [17:18:04] afk [17:21:57] PROBLEM - meta.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.gyaanipedia.com could not be found [17:21:58] PROBLEM - erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for erikapedia.com could not be found [17:21:58] PROBLEM - wiki.responsibly.ai - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.responsibly.ai could not be found [17:22:00] PROBLEM - translate.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for translate.petrawiki.org could not be found [17:28:42] RECOVERY - erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - erikapedia.com reverse DNS resolves to cp11.miraheze.org [17:28:44] RECOVERY - wiki.responsibly.ai - reverse DNS on sslhost is OK: rDNS OK - wiki.responsibly.ai reverse DNS resolves to cp10.miraheze.org [17:28:44] RECOVERY - meta.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - meta.gyaanipedia.com reverse DNS resolves to cp11.miraheze.org [17:28:47] RECOVERY - translate.petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - translate.petrawiki.org reverse DNS resolves to cp11.miraheze.org [17:42:48] PROBLEM - services3 APT on services3 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [17:43:58] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [17:46:12] PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates). [17:49:51] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (1 critical updates). [17:50:09] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (2 critical updates). [17:50:12] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 109 packages available for upgrade (1 critical updates). [17:51:20] PROBLEM - gluster3 APT on gluster3 is CRITICAL: APT CRITICAL: 36 packages available for upgrade (1 critical updates). [17:51:25] PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 74 packages available for upgrade (1 critical updates). [17:51:44] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 74 packages available for upgrade (1 critical updates). [17:51:48] PROBLEM - test4 APT on test4 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [17:52:11] PROBLEM - mon2 APT on mon2 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (1 critical updates). [17:53:12] PROBLEM - mem2 APT on mem2 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [17:53:58] PROBLEM - cp10 APT on cp10 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (1 critical updates). [17:53:58] PROBLEM - cp12 APT on cp12 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [17:55:15] PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 50 packages available for upgrade (1 critical updates). [17:55:19] PROBLEM - puppet3 APT on puppet3 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (2 critical updates). [17:57:05] PROBLEM - gluster4 APT on gluster4 is CRITICAL: APT CRITICAL: 36 packages available for upgrade (1 critical updates). [17:57:16] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 41 packages available for upgrade (1 critical updates). [17:57:17] PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates). [18:01:03] PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [18:03:05] PROBLEM - graylog2 APT on graylog2 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (1 critical updates). [18:04:28] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.46, 2.78, 1.75 [18:04:37] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (1 critical updates). [18:04:39] PROBLEM - phab2 APT on phab2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [18:04:43] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (1 critical updates). [18:05:09] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 38 packages available for upgrade (1 critical updates). [18:05:24] PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (1 critical updates). [18:05:32] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (1 critical updates). [18:05:44] PROBLEM - mem1 APT on mem1 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [18:06:06] PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (1 critical updates). [18:06:27] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.17, 2.11, 1.62 [18:06:50] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (1 critical updates). [18:07:40] PROBLEM - cp11 APT on cp11 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [18:08:05] PROBLEM - ldap2 APT on ldap2 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [18:10:09] ^ checking updates [18:11:13] PROBLEM - services4 APT on services4 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [18:13:32] RECOVERY - mw10 APT on mw10 is OK: APT OK: 36 packages available for upgrade (0 critical updates). [18:16:46] !log update libpq5 (sec patch) [18:16:49] RECOVERY - services3 APT on services3 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [18:16:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:17:06] RECOVERY - gluster4 APT on gluster4 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [18:17:12] RECOVERY - mem2 APT on mem2 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [18:17:13] RECOVERY - services4 APT on services4 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [18:17:14] RECOVERY - db13 APT on db13 is OK: APT OK: 49 packages available for upgrade (0 critical updates). [18:17:14] RECOVERY - ns1 APT on ns1 is OK: APT OK: 40 packages available for upgrade (0 critical updates). [18:17:17] RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [18:17:22] RECOVERY - gluster3 APT on gluster3 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [18:17:24] RECOVERY - db12 APT on db12 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [18:17:39] RECOVERY - cp11 APT on cp11 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [18:17:41] RECOVERY - db11 APT on db11 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [18:17:43] RECOVERY - mem1 APT on mem1 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [18:17:51] RECOVERY - test4 APT on test4 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [18:17:52] RECOVERY - cp3 APT on cp3 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [18:17:55] RECOVERY - cp12 APT on cp12 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [18:17:57] RECOVERY - cp10 APT on cp10 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [18:18:01] RECOVERY - ns2 APT on ns2 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [18:18:03] RECOVERY - jobrunner4 APT on jobrunner4 is OK: APT OK: 36 packages available for upgrade (0 critical updates). [18:18:03] RECOVERY - ldap2 APT on ldap2 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [18:18:10] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 108 packages available for upgrade (0 critical updates). [18:18:12] RECOVERY - mon2 APT on mon2 is OK: APT OK: 33 packages available for upgrade (0 critical updates). [18:18:35] RECOVERY - mw9 APT on mw9 is OK: APT OK: 36 packages available for upgrade (0 critical updates). [18:18:44] RECOVERY - mw8 APT on mw8 is OK: APT OK: 36 packages available for upgrade (0 critical updates). [18:18:45] RECOVERY - phab2 APT on phab2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [18:18:50] RECOVERY - mw11 APT on mw11 is OK: APT OK: 36 packages available for upgrade (0 critical updates). [18:19:06] RECOVERY - mail2 APT on mail2 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [18:19:08] RECOVERY - graylog2 APT on graylog2 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [18:19:13] RECOVERY - test3 APT on test3 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [18:19:21] RECOVERY - jobrunner3 APT on jobrunner3 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [18:20:25] RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [18:20:36] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.52, 20.50, 16.86 [18:22:31] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.43, 18.98, 16.70 [18:23:21] RECOVERY - puppet3 APT on puppet3 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [18:23:41] !log upgrade postgresql-client-11 (sec patch) [18:23:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:42] RECOVERY - bacula2 APT on bacula2 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [18:56:21] !log cloud5: echo "none" > /sys/block/sda/queue/scheduler [18:56:24] !log cloud5: echo "none" > /sys/block/sdb/queue/scheduler [18:56:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:09] new i/o scheduler :o [18:59:57] no immediate breakage on cloud5 [19:00:45] what do you think? leave the i/o scheduler as-is for a day (to collect metrics on impact), afterwards try the raid check? [19:02:02] Yeh [19:02:53] meta is slow [19:05:25] is it? [19:05:40] doesn't seem related to the change i made [19:05:45] as it went through to cloud4 [19:06:12] seems to be better now [19:16:23] SPF|Cloud: oh [19:16:31] we have to change it within VMs too apparently [19:16:46] https://www.irccloud.com/pastebin/D6peCDRf/ [19:16:47] [ Snippet | IRCCloud ] - www.irccloud.com [19:17:27] the scheduler on these VMs should be 'none' in the first place...? [19:17:42] if you have mq-deadline on the host, you would be scheduling twice [19:18:05] SPF|Cloud: shall i set it to none? [19:18:33] yep, try that for mediawiki and gluster hosts on cloud5 [19:18:41] ok, doing, thanks! [19:18:55] allows us to compare data between cloud4 and cloud5 [19:19:16] !log mw 10/11: echo "none" > /sys/block/sda/queue/scheduler [19:19:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:59] !log gluster4: echo "none" > /sys/block/sda/queue/scheduler [19:20:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:21:16] !log gluster3: echo "none" > /sys/block/sda/queue/scheduler [19:21:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:21:20] !log mw 8/9: echo "none" > /sys/block/sda/queue/scheduler [19:21:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:28] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JsmrY [19:22:30] [02miraheze/RottenLinks] 07Universal-Omega 03aca94e0 - T5700: Use HttpRequestFactory [19:22:31] [02RottenLinks] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fxBrr [19:22:33] [02RottenLinks] 07Universal-Omega opened pull request 03#32: T5700: Use HttpRequestFactory - 13https://git.io/JsmrO [19:23:17] SPF|Cloud: things feel faster tho could just be a coincidence [19:23:36] miraheze/RottenLinks - Universal-Omega the build passed. [19:23:51] over time, the graphs will tell :) [19:25:22] !log jobrunner 3/4: echo "none" > /sys/block/sda/queue/scheduler [19:25:26] !log test3: echo "none" > /sys/block/sda/queue/scheduler [19:25:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:30:44] i'll also change cp10/11 [19:31:27] !log cp 10/11: echo "none" > /sys/block/sda/queue/scheduler [19:31:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:31:39] if we want to compare data from two cloud nodes, you should adjust one cache proxy only [19:32:05] see my comment from 13 minutes ago [19:33:37] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 2.87, 4.89, 3.11 [19:34:08] > allows us to compare data between cloud4 and cloud5 [19:34:09] SPF|Cloud: that comment? [19:34:09] !log test4: echo "none" > /sys/block/sda/queue/scheduler [19:34:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:29] yes, that comment [19:34:33] okay, cool [19:35:34] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.19, 3.59, 2.84 [19:37:30] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.60, 2.59, 2.56 [19:37:43] SPF|Cloud: ah ok, so should i put mw8/9 back to mq including gluster3? [19:37:50] yes and yes [19:37:57] and cp10 (cloud4?) too [19:38:12] ok [19:38:56] done [19:39:17] !log set back gluster3, cp10, mw8/9 and jobrunner3 to mw-deadline [19:39:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:39:57] !log mon2: echo "none" > /sys/block/sda/queue/scheduler [19:40:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:52] !log mail2: echo "none" > /sys/block/sda/queue/scheduler [19:40:55] !log ldap2: echo "none" > /sys/block/sda/queue/scheduler [19:41:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:41:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:41:28] !log services4: echo "none" > /sys/block/sda/queue/scheduler [19:41:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:41:33] !log phab2: echo "none" > /sys/block/sda/queue/scheduler [19:41:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:34] [02mw-config] 07dmehus opened pull request 03#3893: #T7283 - add protection level to allpediawiki - 13https://git.io/JsmPW [19:50:39] miraheze/mw-config - dmehus the build passed. [19:52:28] [02mw-config] 07Universal-Omega reviewed pull request 03#3893 commit - 13https://git.io/JsmP1 [19:52:29] [02mw-config] 07Universal-Omega reviewed pull request 03#3893 commit - 13https://git.io/JsmPM [19:53:11] [02mw-config] 07dmehus reviewed pull request 03#3893 commit - 13https://git.io/JsmPx [19:53:13] [02mw-config] 07Universal-Omega reviewed pull request 03#3893 commit - 13https://git.io/JsmPp [19:53:26] [02mw-config] 07Universal-Omega reviewed pull request 03#3893 commit - 13https://git.io/JsmPp [19:54:06] [02mw-config] 07dmehus reviewed pull request 03#3893 commit - 13https://git.io/JsmXv [19:54:09] [02mw-config] 07Universal-Omega reviewed pull request 03#3893 commit - 13https://git.io/JsmXf [19:54:11] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 32.37, 11.67, 5.61 [19:54:46] [02mw-config] 07Universal-Omega reviewed pull request 03#3893 commit - 13https://git.io/JsmXI [19:56:07] [02mw-config] 07dmehus reviewed pull request 03#3893 commit - 13https://git.io/JsmXW [19:56:49] [02mw-config] 07Universal-Omega reviewed pull request 03#3893 commit - 13https://git.io/JsmXV [19:57:59] [02mw-config] 07dmehus synchronize pull request 03#3893: #T7283 - add protection level to allpediawiki - 13https://git.io/JsmPW [19:58:18] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 4.69, 4.70, 2.56 [19:58:32] [02mw-config] 07dmehus reviewed pull request 03#3893 commit - 13https://git.io/JsmX5 [19:59:01] [02mw-config] 07dmehus reviewed pull request 03#3893 commit - 13https://git.io/JsmXp [19:59:04] miraheze/mw-config - dmehus the build passed. [19:59:09] [02mw-config] 07Universal-Omega reviewed pull request 03#3893 commit - 13https://git.io/Jsm1v [19:59:53] [02mw-config] 07dmehus synchronize pull request 03#3893: #T7283 - add protection level to allpediawiki - 13https://git.io/JsmPW [20:00:18] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 1.18, 3.42, 2.36 [20:00:33] [02mw-config] 07dmehus reviewed pull request 03#3893 commit - 13https://git.io/Jsm1Z [20:00:42] [02RottenLinks] 07Southparkfan reviewed pull request 03#32 commit - 13https://git.io/Jsm1C [20:00:59] miraheze/mw-config - dmehus the build passed. [20:02:19] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 8.68, 4.33, 2.75 [20:03:51] Reception123: I think I have added comments to all tasks where you have pinged me [20:04:43] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 1.41, 2.85, 2.41 [20:05:09] [02mw-config] 07Universal-Omega reviewed pull request 03#3893 commit - 13https://git.io/Jsm1x [20:06:15] [02mw-config] 07dmehus reviewed pull request 03#3893 commit - 13https://git.io/JsmMT [20:06:45] [02mw-config] 07Universal-Omega reviewed pull request 03#3893 commit - 13https://git.io/JsmMY [20:06:52] SPF|Cloud: thanks! will have a look at your comments [20:06:58] [02mw-config] 07Universal-Omega reviewed pull request 03#3893 commit - 13https://git.io/JsmMY [20:07:57] [02mw-config] 07dmehus reviewed pull request 03#3893 commit - 13https://git.io/JsmMB [20:08:42] no problem, as long as you prioritise the tasks on the MediaWiki (SRE) board properly, we will help [20:09:02] I am aware there's more work to do there :) [20:09:23] afk [20:09:30] [02mw-config] 07Universal-Omega reviewed pull request 03#3893 commit - 13https://git.io/JsmMo [20:09:37] [02mw-config] 07Universal-Omega reviewed pull request 03#3893 commit - 13https://git.io/JsmMo [20:09:39] [02mw-config] 07dmehus synchronize pull request 03#3893: #T7283 - add protection level to allpediawiki - 13https://git.io/JsmPW [20:09:52] [02mw-config] 07Universal-Omega reviewed pull request 03#3893 commit - 13https://git.io/JsmMo [20:10:16] [02mw-config] 07dmehus reviewed pull request 03#3893 commit - 13https://git.io/JsmMM [20:10:45] miraheze/mw-config - dmehus the build passed. [20:11:41] [02mw-config] 07dmehus reviewed pull request 03#3893 commit - 13https://git.io/JsmMh [20:11:55] [02RottenLinks] 07Universal-Omega reviewed pull request 03#32 commit - 13https://git.io/JsmDv [20:12:35] [02RottenLinks] 07Southparkfan reviewed pull request 03#32 commit - 13https://git.io/JsmDL [20:12:48] (almost afk...) [20:13:08] [02RottenLinks] 07Universal-Omega reviewed pull request 03#32 commit - 13https://git.io/JsmDs [20:14:03] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.73, 2.65, 3.91 [20:14:09] [02mediawiki] 07Southparkfan closed pull request 03#1555: Bump extensions/ArticleToCategory2 from `09ffed6` to `9fa7338` - 13https://git.io/J3o1L [20:14:10] [02miraheze/mediawiki] 07Southparkfan pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JsmDu [20:14:12] [02miraheze/mediawiki] 07dependabot[bot] 03b407190 - Bump extensions/ArticleToCategory2 from `09ffed6` to `9fa7338` (#1555) [20:14:17] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/ArticleToCategory2-9fa7338 [20:14:19] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/ArticleToCategory2-9fa7338 - 13https://git.io/vbL5b [20:14:59] [02mediawiki] 07Southparkfan closed pull request 03#1565: Bump extensions/SnapProjectEmbed from `53ffd48` to `cf18475` - 13https://git.io/J3o1D [20:15:00] [02miraheze/mediawiki] 07Southparkfan pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JsmDi [20:15:02] [02miraheze/mediawiki] 07dependabot[bot] 039c6ba4a - Bump extensions/SnapProjectEmbed from `53ffd48` to `cf18475` (#1565) [20:15:07] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/SnapProjectEmbed-cf18475 - 13https://git.io/vbL5b [20:15:08] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/SnapProjectEmbed-cf18475 [20:15:41] [02mediawiki] 07Southparkfan commented on pull request 03#1567: Bump extensions/TwitterTag from `0c2cee6` to `e4c0e39` - 13https://git.io/JsmDy [20:15:55] [02mediawiki] 07Southparkfan commented on pull request 03#1567: Bump extensions/TwitterTag from `0c2cee6` to `e4c0e39` - 13https://git.io/JsmDQ [20:15:56] [02mediawiki] 07dependabot[bot] edited pull request 03#1567: Bump extensions/TwitterTag from `0c2cee6` to `e4c0e39` - 13https://git.io/J3o15 [20:16:37] [02mediawiki] 07dependabot[bot] commented on pull request 03#1567: Bump extensions/TwitterTag from `0c2cee6` to `e4c0e39` - 13https://git.io/JsmDA [20:16:38] [02mediawiki] 07dependabot[bot] edited pull request 03#1567: Bump extensions/TwitterTag from `0c2cee6` to `e4c0e39` - 13https://git.io/J3o15 [20:16:40] [02mediawiki] 07dependabot[bot] closed pull request 03#1567: Bump extensions/TwitterTag from `0c2cee6` to `e4c0e39` - 13https://git.io/J3o15 [20:16:41] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/TwitterTag-e4c0e39 [20:16:43] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/TwitterTag-e4c0e39 - 13https://git.io/vbL5b [20:17:59] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.82, 1.79, 3.28 [20:19:40] RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:20:43] [02RemovePII] 07Universal-Omega closed pull request 03#1: Create SpecialGenerateRandomHash (T7286: phase one) - 13https://git.io/JsJvS [20:20:45] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03master [+6/-0/±0] 13https://git.io/Jsmyr [20:20:46] [02miraheze/RemovePII] 07Universal-Omega 03738036f - Create SpecialGenerateRandomHash (T7286: phase one) (#1) [20:25:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jsm9O [20:25:17] [02miraheze/mw-config] 07Universal-Omega 035429acc - Update GlobalExtensions.php [20:25:19] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [20:25:52] [02mw-config] 07Universal-Omega opened pull request 03#3894: T7286: Deploy RemovePII extension - 13https://git.io/Jsm9Z [20:26:54] miraheze/mw-config - Universal-Omega the build passed. [20:31:42] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 11.65, 10.22, 5.86 [20:31:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JsmH3 [20:31:49] [02miraheze/mw-config] 07Universal-Omega 03fcdfece - Update LocalSettings.php [20:31:50] [02mw-config] 07Universal-Omega synchronize pull request 03#3894: T7286: Deploy RemovePII extension - 13https://git.io/Jsm9Z [20:32:15] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.60, 22.48, 18.24 [20:32:54] miraheze/mw-config - Universal-Omega the build passed. [20:36:37] afk [20:38:01] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 45.61, 26.71, 20.80 [20:40:34] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:41:23] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:42:39] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:42:46] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:43:18] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:43:56] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 11.77, 19.66, 19.88 [20:48:16] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jsm73 [20:48:18] [02miraheze/RottenLinks] 07Universal-Omega 03c4f7c89 - Update extension.json [20:48:19] [02RottenLinks] 07Universal-Omega synchronize pull request 03#32: T5700: Use HttpRequestFactory - 13https://git.io/JsmrO [20:49:19] miraheze/RottenLinks - Universal-Omega the build passed. [20:49:23] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jsm74 [20:49:24] [02miraheze/RottenLinks] 07Universal-Omega 03a14c5e6 - Update CHANGELOG [20:49:26] [02RottenLinks] 07Universal-Omega synchronize pull request 03#32: T5700: Use HttpRequestFactory - 13https://git.io/JsmrO [20:50:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.33, 21.44, 20.39 [20:50:23] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [20:50:34] miraheze/RottenLinks - Universal-Omega the build passed. [20:51:25] [02RottenLinks] 07Universal-Omega closed pull request 03#32: T5700: Use HttpRequestFactory - 13https://git.io/JsmrO [20:51:26] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Jsm7Q [20:51:28] [02miraheze/RottenLinks] 07Universal-Omega 03c5a8212 - T5700: Use HttpRequestFactory (#32) [20:51:29] [02RottenLinks] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fxBrr [20:51:31] [02miraheze/RottenLinks] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [20:51:38] PROBLEM - db11 Puppet on db11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/mnt/dbbackup1-c2] [20:51:56] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 11.89, 17.80, 19.17 [20:52:23] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 11.95, 12.05, 5.59 [20:52:29] miraheze/RottenLinks - Universal-Omega the build passed. [20:52:32] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.82, 20.75, 14.42 [20:53:12] PROBLEM - wikiru.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikiru.wildterra2.com could not be found [20:53:12] PROBLEM - nonbinary.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nonbinary.wiki could not be found [20:54:28] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.34, 18.95, 14.47 [20:58:56] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.45, 1.98, 4.00 [20:59:18] [02mw-config] 07Universal-Omega closed pull request 03#3893: #T7283 - add protection level to allpediawiki - 13https://git.io/JsmPW [20:59:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jsm5p [20:59:21] [02miraheze/mw-config] 07dmehus 03bd5410e - #T7283 - add protection level to allpediawiki (#3893) [21:00:08] RECOVERY - nonbinary.wiki - reverse DNS on sslhost is OK: rDNS OK - nonbinary.wiki reverse DNS resolves to cp10.miraheze.org [21:00:10] RECOVERY - wikiru.wildterra2.com - reverse DNS on sslhost is OK: rDNS OK - wikiru.wildterra2.com reverse DNS resolves to cp11.miraheze.org [21:00:12] miraheze/mw-config - Universal-Omega the build passed. [21:00:18] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 0.98, 3.31, 3.85 [21:00:52] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 9.54, 5.73, 5.19 [21:01:57] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.16, 20.07, 19.08 [21:04:00] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [21:04:19] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.67, 1.89, 3.13 [21:04:34] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:04:35] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:04:45] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [21:05:09] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:05:16] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:05:21] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:05:56] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.48, 17.71, 18.39 [21:12:33] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.48, 2.29, 3.71 [21:13:17] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±1] 13https://git.io/JsmF5 [21:13:18] [02miraheze/mediawiki] 07Universal-Omega 034724db5 - Install RemovePII [21:13:19] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [21:14:28] [02mediawiki] 07Universal-Omega opened pull request 03#1713: Install RemovePII - 13https://git.io/JsmbJ [21:16:26] [02mediawiki] 07Universal-Omega closed pull request 03#1713: Install RemovePII - 13https://git.io/JsmbJ [21:16:27] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+1/-0/±1] 13https://git.io/JsmbG [21:16:29] [02miraheze/mediawiki] 07Universal-Omega 0300e5e8a - Install RemovePII (#1713) [21:16:30] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:16:32] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [21:18:28] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.09, 1.90, 3.16 [21:23:34] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.20, 1.81, 1.30 [21:25:30] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.78, 1.51, 1.26 [21:28:19] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 3.65, 4.02, 2.68 [21:30:18] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.55, 2.70, 2.35 [21:31:09] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 7 minutes ago with 0 failures [21:32:31] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±1] 13https://git.io/Jsmx3 [21:32:32] [02miraheze/mediawiki] 07Universal-Omega 0384daaec - Install RemovePII [21:32:34] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [21:34:45] Crepusculum: rebuild i18n? [21:35:17] RhinosF1: not deployed through mw-config yet. Still testing on test3 [21:35:29] But I will in a second. [21:35:59] Crepusculum: yeah I noticed a missing message on the special page [21:36:56] Yeah I will. [21:37:53] Cool [21:39:57] [02mediawiki] 07Universal-Omega opened pull request 03#1714: Install RemovePII - 13https://git.io/Jsmpm [21:40:07] [02mediawiki] 07Universal-Omega closed pull request 03#1714: Install RemovePII - 13https://git.io/Jsmpm [21:40:08] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+1/-0/±1] 13https://git.io/JsmpY [21:40:10] [02miraheze/mediawiki] 07Universal-Omega 034a5c159 - Install RemovePII (#1714) [21:40:11] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [21:40:13] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:41:19] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:41:21] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:42:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jsmpa [21:42:02] [02miraheze/mw-config] 07Universal-Omega 03b7e0910 - Update extension-list [21:42:03] [02mw-config] 07Universal-Omega synchronize pull request 03#3894: T7286: Deploy RemovePII extension - 13https://git.io/Jsm9Z [21:42:30] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:42:40] uh oh [21:42:46] Crepisculum: ^ [21:43:13] miraheze/mw-config - Universal-Omega the build passed. [21:43:14] s/Crepisculum/Crepusculum [21:43:14] dmehus meant to say: Crepusculum: ^ [21:43:48] dmehus: they're normally transient [21:44:00] Or someone has pushed more than 10 commits in the last 30 minutes [21:44:26] RhinosF1: ack [21:44:39] [02mw-config] 07Universal-Omega closed pull request 03#3894: T7286: Deploy RemovePII extension - 13https://git.io/Jsm9Z [21:44:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Jsmp9 [21:44:42] [02miraheze/mw-config] 07Universal-Omega 03e65081f - Deploy RemovePII extension (#3894) [21:44:43] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:44:45] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [21:44:53] dmehus: it's not the latter but it's very little reason to worry [21:45:18] If puppet is failing, it's probably an indication that it's stopping something happening [21:45:32] Unless it's static that it's moaning about in which case do scream [21:45:38] miraheze/mw-config - Universal-Omega the build passed. [21:46:07] RhinosF1: Oh, I'm not really worried. I just know that it should recover right away as in my experience puppet failing is often the cause of, say, a 503 error [21:46:37] dmehus: puppet failing doesn't cause 503s [21:46:49] well that's not what I meant [21:46:59] That's what it reads as [21:47:05] I just meant that it can be an indication when there is a 503 [21:47:18] i.e. if say a sysadmin merges an extension with a problem [21:47:18] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [21:47:20] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [21:47:31] you'll get a 503 and a Puppet failure [21:47:42] but I know it's not the *cause* of the error [21:47:44] dmehus: you probably won't get a puppet failure [21:47:47] the cause is usually the extension [21:48:17] RhinosF1: not the case with Reception123's recent extension installs/merges a week or two ago though [21:48:18] Puppet just deploys changes, it doesn't care if they work for MediaWiki [21:48:22] so that's what I'm going by [21:48:30] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:48:34] dmehus: I'd say it was likely co-incidence [21:48:58] RhinosF1: okay, I'll ask Reception123 to review this and explain it better to me then [21:49:15] dmehus: I can't remember but puppet doesn't care if MediaWiki is 503ing [21:50:32] It could happen if you made php get stuck in a failed state [21:50:39] But that's unlikely [21:50:40] RhinosF1, I know that, that's not what I'm saying. I'm just saying that often times when there's an issue with an extension that is installed / merged, we'll see a puppet failure. Not every time, but a lot of times. More often than not to be a coincidence [21:51:12] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki && sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* / jbr* / test3 [21:51:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:51:56] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 36.90, 23.82, 18.65 [21:52:04] dmehus: if it's the same time as the Dependabot updates then that's more than likely just the amount being pulled causing errors because git. Completely unrelated to the actual changes. You could have perfect changes fail and broken ones pass that fine. [21:52:15] RhinosF1: ran LC now. [21:53:04] Crepusculum: Looks good to me [21:53:32] RhinosF1: yeah I know they're unrelated, but it's still something that often happens. @Reception123 has just told me it's something to look for when diagnosing an 503 error (i.e., if you just merged an extension or something, then look at that) [21:53:33] Works too [21:53:45] dmehus: okay [21:54:54] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.64, 6.46, 5.26 [21:56:49] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.20, 6.32, 5.35 [21:57:57] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.25, 20.30, 19.18 [21:58:01] RhinosF1: Great! (Also added it to REL1_36 branch by the way) [21:58:20] :) [21:59:08] [02mw-config] 07Universal-Omega closed pull request 03#3892: Header modification for snapwikiwiki - 13https://git.io/Jsqig [21:59:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jsmj0 [21:59:11] [02miraheze/mw-config] 07R4356th 03e01966c - Header modification for snapwikiwiki (#3892) [22:00:06] miraheze/mw-config - Universal-Omega the build passed. [22:01:09] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:05:09] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 3 minutes ago with 0 failures [22:15:05] [02mw-config] 07BukkitIsEnchanted opened pull request 03#3895: LocalSettings.php - 13https://git.io/JsYvC [22:21:11] [02mw-config] 07Universal-Omega edited pull request 03#3895: Add Wikimedia Meta to import sources - 13https://git.io/JsYvC [22:21:33] miraheze/mw-config - Universal-Omega the build passed. [22:38:40] [02mw-config] 07Universal-Omega closed pull request 03#3878: No longer manually set wmgUseExtension in LocalSettings - 13https://git.io/J39sH [22:38:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JsYU9 [22:38:43] [02miraheze/mw-config] 07Universal-Omega 03afdf615 - No longer manually set wmgUseExtension in LocalSettings (#3878) [22:39:44] miraheze/mw-config - Universal-Omega the build passed. [22:45:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-3878-patch-173 [+0/-0/±2] 13https://git.io/JsYTw [22:45:46] [02miraheze/mw-config] 07Universal-Omega 037f68baa - Revert "No longer manually set wmgUseExtension in LocalSettings (#3878)" [22:45:48] [02mw-config] 07Universal-Omega created branch 03revert-3878-patch-173 - 13https://git.io/vbvb3 [22:45:49] [02mw-config] 07Universal-Omega opened pull request 03#3896: Revert "No longer manually set wmgUseExtension in LocalSettings" - 13https://git.io/JsYTo [22:45:54] [02mw-config] 07Universal-Omega closed pull request 03#3896: Revert "No longer manually set wmgUseExtension in LocalSettings" - 13https://git.io/JsYTo [22:45:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JsYT6 [22:45:57] [02miraheze/mw-config] 07Universal-Omega 03fa45f09 - Revert "No longer manually set wmgUseExtension in LocalSettings (#3878)" (#3896) [22:45:59] [02mw-config] 07Universal-Omega deleted branch 03revert-3878-patch-173 - 13https://git.io/vbvb3 [22:46:01] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-3878-patch-173 [22:46:54] miraheze/mw-config - Universal-Omega the build passed. [22:46:56] miraheze/mw-config - Universal-Omega the build passed. [22:48:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JsYTb [22:48:25] [02miraheze/mw-config] 07Universal-Omega 03337b13e - Update LocalExtensions.php [22:48:27] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 6.61, 8.23, 4.11 [22:48:27] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [22:48:57] [02mw-config] 07Universal-Omega opened pull request 03#3897: No longer manually set wmgUseExtension in LocalSettings - 13https://git.io/JsYke [22:49:59] miraheze/mw-config - Universal-Omega the build passed. [22:50:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JsYkm [22:50:08] [02miraheze/mw-config] 07Universal-Omega 03ff3112e - Update LocalSettings.php [22:50:09] [02mw-config] 07Universal-Omega synchronize pull request 03#3897: No longer manually set wmgUseExtension in LocalSettings - 13https://git.io/JsYke [22:51:09] miraheze/mw-config - Universal-Omega the build passed. [22:52:24] [02mw-config] 07JohnFLewis commented on pull request 03#3897: No longer manually set wmgUseExtension in LocalSettings - 13https://git.io/JsYkB [22:53:24] [02mw-config] 07Universal-Omega edited pull request 03#3897: No longer manually set wmgUseExtension in LocalSettings - 13https://git.io/JsYke [22:54:27] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.63, 3.95, 3.54 [22:55:27] [02mw-config] 07Universal-Omega commented on pull request 03#3897: No longer manually set wmgUseExtension in LocalSettings - 13https://git.io/JsYkD [22:55:29] [02mw-config] 07Universal-Omega closed pull request 03#3897: No longer manually set wmgUseExtension in LocalSettings - 13https://git.io/JsYke [22:55:30] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [22:55:32] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [22:56:26] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.24, 2.70, 3.13 [23:00:16] [02mw-config] 07Universal-Omega closed pull request 03#3895: Add Wikimedia Meta to import sources - 13https://git.io/JsYvC [23:00:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JsYIO [23:00:19] [02miraheze/mw-config] 07BukkitIsEnchanted 033809463 - Add Wikimedia Meta to import sources (#3895) [23:01:42] miraheze/mw-config - Universal-Omega the build passed. [23:05:41] !log sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=metawiki --r "[[phab:T7301|Requested]]" /home/universalomega/deletebatch-metawiki.txt [23:05:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:11:08] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [23:18:54] [02CreateWiki] 07Universal-Omega synchronize pull request 03#215: Use textarea type for reason and comment fields - 13https://git.io/J36M3 [23:19:09] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 8 minutes ago with 0 failures [23:19:59] miraheze/CreateWiki - Universal-Omega the build passed. [23:20:10] [02CreateWiki] 07Universal-Omega synchronize pull request 03#215: Use textarea type for reason and comment fields - 13https://git.io/J36M3 [23:21:04] miraheze/CreateWiki - Universal-Omega the build passed. [23:21:52] [02CreateWiki] 07Universal-Omega synchronize pull request 03#215: Use textarea type for reason and comment fields - 13https://git.io/J36M3 [23:22:25] [02CreateWiki] 07Universal-Omega synchronize pull request 03#215: Use textarea type for reason and comment fields - 13https://git.io/J36M3 [23:22:54] miraheze/CreateWiki - Universal-Omega the build passed. [23:23:29] miraheze/CreateWiki - Universal-Omega the build passed. [23:28:09] [02CreateWiki] 07Universal-Omega synchronize pull request 03#215: Use textarea type for reason and comment fields - 13https://git.io/J36M3 [23:29:04] miraheze/CreateWiki - Universal-Omega the build passed. [23:36:48] [02CreateWiki] 07Universal-Omega synchronize pull request 03#215: Use textarea type for reason and comment fields - 13https://git.io/J36M3 [23:37:45] miraheze/CreateWiki - Universal-Omega the build passed. [23:38:16] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [23:38:54] [02CreateWiki] 07Universal-Omega synchronize pull request 03#215: Use textarea type for reason and comment fields - 13https://git.io/J36M3 [23:39:50] miraheze/CreateWiki - Universal-Omega the build passed. [23:40:25] [02CreateWiki] 07Universal-Omega synchronize pull request 03#215: Use textarea type for reason and comment fields - 13https://git.io/J36M3 [23:41:20] miraheze/CreateWiki - Universal-Omega the build passed. [23:43:12] [02CreateWiki] 07Universal-Omega synchronize pull request 03#215: Use textarea type for reason and comment fields - 13https://git.io/J36M3 [23:44:05] miraheze/CreateWiki - Universal-Omega the build passed. [23:51:55] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org