[00:15:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmHV3 [00:15:12] [02miraheze/services] 07MirahezeSSLBot 03975b902 - BOT: Updating services config for wikis [00:28:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.13, 2.57 [00:30:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.75, 2.71, 2.49 [00:35:58] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 166s [00:37:59] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [00:38:15] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:04:14] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [01:04:54] [02mediawiki] 07Universal-Omega closed pull request 03#1327: Bump skins/Citizen from `09b399c` to `b68a2bd` - 13https://git.io/JqKH3 [01:04:56] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JmHXJ [01:04:57] [02miraheze/mediawiki] 07dependabot[bot] 03840ff85 - Bump skins/Citizen from `09b399c` to `b68a2bd` (#1327) [01:04:59] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/skins/Citizen-b68a2bd [01:05:00] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/skins/Citizen-b68a2bd - 13https://git.io/vbL5b [01:09:16] [02mediawiki] 07Universal-Omega closed pull request 03#1316: Bump extensions/Wikibase from `5ebc9d0` to `14df0d2` - 13https://git.io/JqKSb [01:09:17] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JmHX9 [01:09:19] [02miraheze/mediawiki] 07dependabot[bot] 031c021fc - Bump extensions/Wikibase from `5ebc9d0` to `14df0d2` (#1316) [01:09:27] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/Wikibase-14df0d2 [01:09:28] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/Wikibase-14df0d2 - 13https://git.io/vbL5b [01:10:05] [02mediawiki] 07Universal-Omega closed pull request 03#1298: Bump extensions/SyntaxHighlight_GeSHi from `5fecf46` to `75d4fba` - 13https://git.io/JqKD0 [01:10:07] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JmHXx [01:10:09] [02miraheze/mediawiki] 07dependabot[bot] 035cfd035 - Bump extensions/SyntaxHighlight_GeSHi from `5fecf46` to `75d4fba` (#1298) [01:10:10] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/SyntaxHighlight_GeSHi-75d4fba [01:10:11] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/SyntaxHighlight_GeSHi-75d4fba - 13https://git.io/vbL5b [01:23:40] [02mediawiki] 07Universal-Omega closed pull request 03#1320: Bump skins/Medik from `921c3b8` to `8794a72` - 13https://git.io/JqK98 [01:23:42] [02mediawiki] 07dependabot[bot] commented on pull request 03#1320: Bump skins/Medik from `921c3b8` to `8794a72` - 13https://git.io/JmHD6 [01:24:02] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/skins/Medik-8794a72 [01:24:03] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/skins/Medik-8794a72 - 13https://git.io/vbL5b [01:24:45] [02mediawiki] 07Universal-Omega closed pull request 03#1290: Bump extensions/OAuth from `c66d882` to `5a56193` - 13https://git.io/JqKM3 [01:24:47] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JmHDd [01:24:48] [02miraheze/mediawiki] 07dependabot[bot] 03eb7f7a5 - Bump extensions/OAuth from `c66d882` to `5a56193` (#1290) [01:24:50] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/OAuth-5a56193 - 13https://git.io/vbL5b [01:24:51] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/OAuth-5a56193 [01:25:19] [02mediawiki] 07Universal-Omega closed pull request 03#1311: Bump extensions/Comments from `d75fa77` to `c70f7d5` - 13https://git.io/JqKSt [01:25:20] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JmHyv [01:25:22] [02miraheze/mediawiki] 07dependabot[bot] 03e5a99de - Bump extensions/Comments from `d75fa77` to `c70f7d5` (#1311) [01:25:23] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/Comments-c70f7d5 [01:25:25] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/Comments-c70f7d5 - 13https://git.io/vbL5b [01:25:32] [02mediawiki] 07Universal-Omega closed pull request 03#1296: Bump extensions/Translate from `6d5bcca` to `04be1fd` - 13https://git.io/JqKDU [01:25:34] [02mediawiki] 07dependabot[bot] commented on pull request 03#1296: Bump extensions/Translate from `6d5bcca` to `04be1fd` - 13https://git.io/JmHyU [01:25:35] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/Translate-04be1fd [01:25:37] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/Translate-04be1fd - 13https://git.io/vbL5b [01:27:45] [02mediawiki] 07Universal-Omega closed pull request 03#1321: Bump extensions/WikiSEO from `b72a071` to `c0579da` - 13https://git.io/JqK9g [01:27:46] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JmHyK [01:27:48] [02miraheze/mediawiki] 07dependabot[bot] 038161645 - Bump extensions/WikiSEO from `b72a071` to `c0579da` (#1321) [01:27:49] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/WikiSEO-c0579da - 13https://git.io/vbL5b [01:27:51] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/WikiSEO-c0579da [01:30:41] [02mw-config] 07Universal-Omega closed issue 03#467: Do not use GitHub issues for reporting issues! - 13https://git.io/JmHSO [01:31:33] [02CreateWiki] 07Universal-Omega closed issue 03#28: Do not use GitHub issues for reporting issues! - 13https://git.io/JmHSW [01:37:41] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:38:14] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:39:50] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:40:02] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:40:12] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:40:17] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:41:05] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:53:59] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 309s [01:57:58] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 116s [01:59:57] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [02:02:57] !sre `Cannot access the database: Cannot access the database: Unknown error (db11.miraheze.org))` on `metawiki` [02:03:08] Have tried reloading several times, but will try again. [02:03:35] So am I. I don't know why. I don't have PC access anymore [02:03:41] At the moment. [02:03:49] !sre ```Error 503 Backend fetch failed, forwarded for REDACTED, 127.0.0.1 [02:03:49] (Varnish XID 106427923) via cp12.miraheze.org at Sun, 21 Mar 2021 02:03:22 GMT.``` [02:04:00] Universal_Omega, ack and thanks :) [02:04:09] DID THE SERVER CRASH? [02:04:15] Hmmm? [02:04:16] METAWIKI IS DOWN [02:04:18] I'm accessing normally [02:04:22] Seems back now, for me anyways. [02:04:38] Seems to be localized to one cache proxy and db server [02:04:45] But I don't have access to look into the cause at the moment. [02:04:50] yeah [02:05:04] Guess we wait until Reception123 is up in a couple hours. [02:05:20] unless paladox wakes up first, ofc [02:05:24] dmehus: yeah. [02:06:10] PROBLEM - commons.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for commons.gyaanipedia.com could not be found [02:06:23] Universal_Omega, it seems back for me [02:07:05] dmehus: yeah, for me to. Also, do you have a minute for DM (unrelated to this) [02:08:34] Universal_Omega, sure [02:09:00] Thanks! [02:12:11] Hmm? [02:13:01] RECOVERY - commons.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - commons.gyaanipedia.com reverse DNS resolves to cp11.miraheze.org [02:13:04] Load times seem extreme right now. And I have no way to look into it right now... [02:13:47] I think it's localized [02:13:54] everything seems normal on my end [02:14:18] @DarkMatterMan4500: What's the "Hmm?" for? [02:14:44] @Lake: likely is localised. [02:15:29] Meta has been loading for 2 minutes for me. [02:22:24] Universal_Omega, np [02:23:03] Universal_Omega, should we create one or two Phabricator tasks related to the ^ and link to this IRC conversation for the infra team to investigate? [02:23:28] dmehus: just one high priority task. [02:23:40] Universal_Omega, ack, okay, will do [02:24:05] Thanks! [02:30:29] Universal_Omega, https://phabricator.miraheze.org/T7008 [02:30:29] [ ⚓ T7008 Investigate database server/cache proxy issues and extreme load times this evening ] - phabricator.miraheze.org [02:30:34] Feel free to add [02:30:53] Thanks for that dmehus! [02:31:03] Looking now. [02:31:26] np [02:32:45] Looks good to me. That's all the information I'm aware of without access. [02:33:13] I heard there was an error that has since been resolved. [02:34:34] okay, sounds good [03:03:35] Universal_Omega: would be possible to add these flags for Citizen Skin on ManageWiki? [03:03:35] https://cdn.discordapp.com/attachments/808001911868489748/823029038582988810/unknown.png [03:03:43] if it is, I will make a task for it [03:04:08] @Lake: Sure, I can do tomorrow. [03:28:04] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 322s [03:33:58] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [03:49:59] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 273s [04:14:39] PROBLEM - wiki.ddr.red - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ddr.red could not be found [04:18:22] PROBLEM - wiki.ddr.red - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.ddr.red and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [04:38:25] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JmQ8A [04:38:26] [02miraheze/MirahezeMagic] 07Universal-Omega 03e78bdc2 - Fix punctuation on overrides/en.json [04:38:28] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [04:38:29] [02MirahezeMagic] 07Universal-Omega opened pull request 03#236: Fix punctuation on overrides/en.json - 13https://git.io/JmQ8p [04:39:05] [02MirahezeMagic] 07Universal-Omega closed pull request 03#236: Fix punctuation on overrides/en.json - 13https://git.io/JmQ8p [04:39:06] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmQ4U [04:39:08] [02miraheze/MirahezeMagic] 07Universal-Omega 032c8e9b0 - Fix punctuation on overrides/en.json (#236) [04:39:09] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [04:39:11] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [04:39:36] miraheze/MirahezeMagic - Universal-Omega the build passed. [04:40:08] miraheze/MirahezeMagic - Universal-Omega the build passed. [05:11:24] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 304s [06:40:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmQQi [06:40:10] [02miraheze/services] 07MirahezeSSLBot 0333eb8b1 - BOT: Updating services config for wikis [06:49:24] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 2.48, 6.96, 3.72 [06:55:24] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.91, 3.83, 3.28 [06:57:24] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.61, 2.72, 2.94 [09:33:24] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 15.54, 15.42, 6.75 [09:33:40] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.75, 9.71, 5.28 [09:39:24] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.30, 3.34, 3.77 [09:43:24] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.62, 2.25, 3.21 [09:43:24] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 0.36, 2.31, 3.66 [09:45:24] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.20, 1.61, 3.23 [10:01:47] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 6.35, 7.25, 4.36 [10:05:34] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.35, 3.52, 3.47 [10:07:28] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.11, 2.43, 3.07 [10:24:56] dmehus: looks like renames on deleted wikis bug [10:28:34] RhinosF1, oh, has that been a problem for awhile now, and do we have a cause / prior task for that? [10:28:44] dmehus: somewhere will be a task [10:28:50] it's happened before [10:28:57] i think we once tried to fix it [10:29:07] I mean, we do do renames successfully with attached deleted wikis, but not sure why some get stuck and some don't [10:29:35] i'll look in logs in a sec [10:29:41] okay [10:29:51] !log rhinos@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=altuniverseuk1182wiki --logwiki=metawiki KingstonuHull-h1986 KingstonHolm861 for [[phab:T7010|T7010]] [10:29:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:29:58] dmehus: ^ [10:31:35] RhinosF1, ack and thanks [10:31:55] yeah it's weird there's no MediaWiki.org documentation on that script; I only found https://wikitech-static.wikimedia.org/wiki/Stuck_global_renames [10:31:56] [ Stuck global renames - Wikitech-static ] - wikitech-static.wikimedia.org [10:33:19] Does it say which wiki it was stuck on when you query the jobs (`altuniverseuk1182wiki`)? If so, since I knew which deleted wiki that was, I probably could've saved you a step and let you know that was the deleted wiki. Sorry about that :D [10:33:19] dmehus: because we're like the only people who rename users over deleted wikis [10:33:27] yeah [10:33:30] true [10:33:37] dmehus: i looked in the renameuser_status table [10:33:47] RhinosF1, oh, cool :) [10:36:26] dmehus: no errors in logs [10:39:43] dmehus: https://phabricator.miraheze.org/T7011 [10:39:43] [ ⚓ T7011 Renames getting stuck on deleted wikis ] - phabricator.miraheze.org [10:41:15] dmehus: also re https://meta.miraheze.org/wiki/Stewards%27_noticeboard#Request_for_local_%22UserMerge%22_rights_on_TurtleWiki, please don't unmerge accounts [10:41:17] [ Stewards' noticeboard - Miraheze Meta ] - meta.miraheze.org [10:41:31] that would be a mess to clean up in the likely event it went wrong [10:48:05] RhinosF1, ack, and I didn't. As you can see, that started as a usermerge request but through the course of discussion, what the requestor was really after was making their wiki private, which I helped them do [10:49:16] dmehus: yeah, but seriously never unmerge an account without amazingly good reason because stewards can't remerge them and it would cause a wonderfuck of a state. [10:50:43] RhinosF1, yeah [10:51:46] only if something went wrong, though. Would be interesting to test with a test account attached only to Test 3 Wiki and Loginwiki perhaps, though [10:51:55] dmehus: NO [10:52:58] RhinosF1, I'm not saying I would; I'm just saying it'd be interesting to test. I mean, it is a production quality extension and Wikimedia have merged accounts in the past [10:55:46] dmehus: Wikimedia don't merge accounts and haven't in years (https://phabricator.wikimedia.org/T49918#1979870). We do GlobalMerges in rare cases normally when imports are involved. [10:55:48] [ ⚓ T49918 Rename of global (attached) users to existing global usernames ] - phabricator.wikimedia.org [10:56:37] RhinosF1, wait, I didn't think global accounts could be merged as it wasn't scalable or something [10:57:06] Is there a MediaWiki documentation page about that? I'd be interested in reading more about the process [10:57:24] dmehus: it's possible but I wouldn't recommend on 2 in use accounts. Doing it where it's from an import has been done before. [10:57:59] RhinosF1, ack oh okay, thanks :) [11:07:28] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/Jm7de [11:07:29] [02miraheze/mw-config] 07RhinosF1 0374244b1 - LS: fix HTTP timeouts [11:07:31] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [11:07:53] [02mw-config] 07RhinosF1 opened pull request 03#3798: LS: fix HTTP timeouts - 13https://git.io/Jm7dI [11:07:54] [02mw-config] 07RhinosF1 assigned pull request 03#3798: LS: fix HTTP timeouts - 13https://git.io/Jm7dI [11:08:20] Reception123: ^ [11:08:58] miraheze/mw-config - RhinosF1 the build passed. [11:18:06] I'm not familiar with those MediaWiki variables, what's their main purpose? [11:19:43] dmehus: stop pages timing out when Wikimedia sites go down [11:20:23] RhinosF1, ah, okay, yeah, seems reasonable then [11:21:18] dmehus: I think John wants it below 15 at some point [11:21:20] But not yet [11:30:54] RhinosF1, ack, okay, cool :) [11:38:36] RhinosF1: LGTM [12:21:58] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [12:58:32] Reception123: ty [13:05:24] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 5.47, 3.04, 1.65 [13:07:24] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.31, 2.25, 1.53 [13:09:59] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 183s [13:11:57] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 41s [13:15:57] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 213s [13:21:56] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 1s [13:30:53] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 113s [13:32:51] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [13:57:36] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 131s [13:59:37] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 36s [14:01:47] [02mw-config] 07Universal-Omega reviewed pull request 03#3798 commit - 13https://git.io/Jm561 [14:35:35] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 229s [14:43:59] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 226s [14:45:57] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [14:53:30] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 192s [14:57:29] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 65s [15:06:35] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 174s [15:07:25] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.19, 7.12, 3.65 [15:09:52] Universal_Omega: ty for the review [15:10:31] RhinosF1: no problem! [15:10:34] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 82s [15:11:24] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.29, 3.47, 2.94 [15:13:24] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.49, 2.52, 2.66 [15:22:53] [02ssl] 07dmehus opened pull request 03#402: Per request - 13https://git.io/JmdJ2 [15:24:34] [02ssl] 07dmehus commented on pull request 03#402: Per request - 13https://git.io/JmdUJ [15:30:35] [02ssl] 07Reception123 commented on pull request 03#402: Per request - 13https://git.io/JmdTK [15:30:43] RhinosF1: heh, we conflicted [15:30:52] lol [15:30:57] Reception123: yep [15:31:09] I guess you can issue a cert [15:31:18] yeah, I just need to get the cert and then I can merge [15:31:24] Cool [15:32:05] [02ssl] 07dmehus commented on pull request 03#402: Per request - 13https://git.io/JmdkI [15:32:48] dmehus: it depends on what the certificate is issued under [15:32:56] And no that's not a wildcard cert [15:33:15] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/Jmdku [15:33:17] [02miraheze/ssl] 07Reception123 037e0c9b2 - add pornwiki.org cert (for redirect - #402) [15:33:24] RhinosF1, what's the easy way, looking at the GitHub repo, to tell if a cert is a wildcard cert? [15:33:40] dmehus: there isn't [15:33:47] [02ssl] 07Reception123 commented on pull request 03#402: Per request - 13https://git.io/Jmdk1 [15:33:50] [02ssl] 07Reception123 closed pull request 03#402: Per request - 13https://git.io/JmdJ2 [15:33:51] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmdkD [15:33:53] [02miraheze/ssl] 07dmehus 039155abd - Per request (#402) [15:34:16] [02ssl] 07Reception123 commented on pull request 03#402: Per request - 13https://git.io/Jmdkd [15:34:21] [02ssl] 07dmehus commented on pull request 03#402: Per request - 13https://git.io/JmdkN [15:34:54] [02ssl] 07dmehus commented on pull request 03#402: Per request - 13https://git.io/JmdIJ [15:35:38] RhinosF1, ack, okay. I guess I'd have to actually look the actual certificate from the certificate authority? [15:36:47] dmehus: yes [15:38:07] ah [15:38:14] s/ah/ack [15:38:15] dmehus meant to say: ack [15:45:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jmdtx [15:45:10] [02miraheze/services] 07MirahezeSSLBot 038be4801 - BOT: Updating services config for wikis [16:29:40] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 117s [16:31:39] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 2s [16:32:43] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 132s [16:34:43] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 5s [17:09:51] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 124.03 ms [17:11:54] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 117.57 ms [17:23:19] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 130.71 ms [17:58:09] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 105.26 ms [18:18:35] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 3.91, 3.19, 1.66 [18:20:10] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.13, 18.60, 14.60 [18:20:34] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.60, 2.17, 1.47 [18:21:20] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.84, 3.61, 2.09 [18:22:10] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.19, 17.91, 14.84 [18:23:19] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 2.66, 3.40, 2.22 [20:11:39] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.17 ms [20:13:42] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 117.82 ms [20:24:02] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 138.44 ms [20:24:03] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JmFkY [20:24:05] [02miraheze/ManageWiki] 07Universal-Omega 03209d42a - Add new 'interwiki' type [20:24:06] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [20:24:08] [02ManageWiki] 07Universal-Omega opened pull request 03#259: Add new 'interwiki' type - 13https://git.io/JmFk3 [20:25:08] miraheze/ManageWiki - Universal-Omega the build passed. [20:26:05] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 119.45 ms [20:32:18] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 125.68 ms [20:42:13] [02mw-config] 07dmehus opened pull request 03#3799: Remove `$` - 13https://git.io/JmFLF [20:43:16] miraheze/mw-config - dmehus the build passed. [20:44:58] [02mw-config] 07dmehus commented on pull request 03#3799: Remove `$` - 13https://git.io/JmFtW [20:45:34] [02mw-config] 07Universal-Omega closed pull request 03#3799: Remove `$` - 13https://git.io/JmFLF [20:45:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JmFt0 [20:45:37] [02miraheze/mw-config] 07dmehus 0397fa00d - Remove `$` (#3799) [20:45:38] [02mw-config] 07Universal-Omega synchronize pull request 03#3798: LS: fix HTTP timeouts - 13https://git.io/Jm7dI [20:45:58] [02mw-config] 07RhinosF1 commented on pull request 03#3799: Remove `$` - 13https://git.io/JmFtV [20:46:05] [02mw-config] 07dmehus reviewed pull request 03#3798 commit - 13https://git.io/JmFto [20:46:46] miraheze/mw-config - Universal-Omega the build passed. [20:46:49] [02mw-config] 07dmehus commented on pull request 03#3799: Remove `$` - 13https://git.io/JmFtH [20:46:49] Ty dmehus [20:47:01] Feel free to merge my pr Universal_Omega [20:47:53] [02mw-config] 07Universal-Omega commented on pull request 03#3761: Better canned responses - 13https://git.io/JmFtj [20:47:58] [02mw-config] 07Universal-Omega closed pull request 03#3761: Better canned responses - 13https://git.io/JqIWl [20:47:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmFqv [20:48:01] [02miraheze/mw-config] 07R4356th 032da8f99 - Better canned responses (#3761) [20:48:05] RhinosF1, no problem [20:48:13] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.72, 3.33, 2.38 [20:49:01] miraheze/mw-config - Universal-Omega the build passed. [20:50:08] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.78, 2.33, 2.12 [20:51:07] [02mw-config] 07Universal-Omega reviewed pull request 03#3798 commit - 13https://git.io/JmFqV [20:51:39] RhinosF1: ^ [20:51:49] Universal_Omega: interesting [20:51:58] Maybe we check test3wiki [20:52:51] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 106.09 ms [20:56:37] RhinosF1: `Allowed values:(float)` so yes, it has to be a float. [20:56:52] Universal_Omega: ah ok [20:56:56] Will fix later [20:57:02] Need to do a few stuff [20:57:04] And tired [20:57:11] RhinosF1: alright, sounds good, thanks! [20:58:41] [02mw-config] 07Universal-Omega reviewed pull request 03#3798 commit - 13https://git.io/JmFmb [21:05:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JmFYN [21:05:37] [02miraheze/mw-config] 07Universal-Omega 03abc9b0a - Use float with wgHTTPConnectTimeout [21:05:38] [02mw-config] 07Universal-Omega synchronize pull request 03#3798: LS: fix HTTP timeouts - 13https://git.io/Jm7dI [21:06:43] miraheze/mw-config - Universal-Omega the build passed. [21:08:44] RhinosF1: done ^ [21:09:19] Ty [21:09:49] Universal_Omega: feel free to merge [21:09:55] I'll be around for half an hour [21:12:05] Careful, Universal_Omega, RhinosF1 might just have to start calling you as other Personal Assistant. :D [21:12:18] [02mw-config] 07Universal-Omega closed pull request 03#3798: LS: fix HTTP timeouts - 13https://git.io/Jm7dI [21:12:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmFOp [21:12:21] [02miraheze/mw-config] 07RhinosF1 0306c18cd - LS: fix HTTP timeouts (#3798) [21:12:27] [02miraheze/mw-config] 07Universal-Omega deleted branch 03RhinosF1-patch-1 [21:12:28] [02mw-config] 07Universal-Omega deleted branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [21:12:31] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.19 ms [21:13:24] miraheze/mw-config - Universal-Omega the build passed. [21:14:34] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 101.17 ms [21:22:52] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 125.92 ms [21:24:55] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 98.41 ms [21:38:32] [02ManageWiki] 07Universal-Omega closed pull request 03#259: Add new 'interwiki' type - 13https://git.io/JmFk3 [21:38:33] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmFZs [21:38:35] [02miraheze/ManageWiki] 07Universal-Omega 0362f92a7 - Add new 'interwiki' type (#259) [21:38:36] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [21:38:38] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:39:39] miraheze/ManageWiki - Universal-Omega the build passed. [22:01:37] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 139s [22:03:36] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 12s [22:55:23] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 293s [22:57:23] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 10s [23:22:39] [02mw-config] 07dmehus opened pull request 03#3800: Add "author request" as a decline reason - 13https://git.io/JmFEs [23:23:34] [02mw-config] 07Universal-Omega closed pull request 03#3800: Add "author request" as a decline reason - 13https://git.io/JmFEs [23:23:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmFEC [23:23:37] [02miraheze/mw-config] 07dmehus 0387516fa - Add "author request" as a decline reason (#3800) [23:23:43] miraheze/mw-config - dmehus the build passed. [23:23:44] ty, Universal_Omega [23:23:48] you're quick :D [23:23:54] ignore DM [23:24:34] miraheze/mw-config - Universal-Omega the build passed. [23:25:06] Lol, yeah, I did it as soon as I got notification for it. That was essential... [23:35:52] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.43, 5.73, 4.14 [23:37:44] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.79, 6.83, 4.68 [23:37:52] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.34, 6.12, 4.50 [23:39:44] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.55, 6.27, 4.73 [23:58:16] yeah [23:58:37] I'm going to work on the wiki creators' mass message announcing the change that R4356th wanted me to do