[00:00:40] till https://github.com/miraheze/mediawiki/blob/REL1_35/includes/deferred/CdnCacheUpdate.php#L323 all looks good [00:00:40] PROBLEM - cp10 HTTP 4xx/5xx ERROR Rate on cp10 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:00:40] [ mediawiki/CdnCacheUpdate.php at REL1_35 · miraheze/mediawiki · GitHub ] - github.com [00:02:40] PROBLEM - cp10 HTTP 4xx/5xx ERROR Rate on cp10 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:03:29] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 34.14, 24.93, 19.99 [00:04:11] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.92, 5.41, 4.41 [00:06:11] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.88, 5.43, 4.54 [00:07:14] [error] => (curl error: no status set) [00:07:27] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.07, 23.95, 20.76 [00:07:34] yes, that's what I'm seeing here [00:08:51] https://github.com/miraheze/mediawiki/blob/REL1_35/includes/libs/http/MultiHttpClient.php#L286 when dumping $reqs [00:08:52] [ mediawiki/MultiHttpClient.php at REL1_35 · miraheze/mediawiki · GitHub ] - github.com [00:13:30] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.39, 19.15, 19.71 [00:14:24] RECOVERY - gluster2 Current Load on gluster2 is OK: OK - load average: 3.94, 4.83, 4.68 [00:16:55] doing it manually works [00:17:27] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 23.86, 22.07, 20.81 [00:18:17] paladox: https://graylog.miraheze.org/messages/graylog_80/b5b6bfd1-667d-11eb-a875-02000052cc6e that's one of the errors for my purge [00:18:28] lookign [00:18:34] inflicted by my edit, of course [00:19:19] oh i see [00:19:24] i saw that on the command line [00:19:24] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 25.18, 22.83, 21.22 [00:19:28] and it only happens for cp3 [00:19:28] but it still returns 200 [00:19:39] curl -X PURGE --url "/w/index.php?title=User:Southparkfan/Test&action=history" -H "Host: meta.miraheze.org"--proxy 51.77.107.210:81 -H "user-agent: MediaWiki/1.35.1 CdnCacheUpdate" -I [00:20:38] how is that an invalid url? [00:21:22] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.57, 19.89, 20.35 [00:23:33] aha [00:23:34] this could be related actually.. [00:23:40] SPF|Cloud setting the full url works [00:23:42] so: [00:23:44] curl -X PURGE --url "https://meta.miraheze.org/w/index.php?title=User:Southparkfan/Test&action=history" -H "Host: meta.miraheze.org"--proxy 51.77.107.210:81 -H "User-Agent: MediaWiki/1.35.1 CdnCacheUpdate" -H "Connection: Keep-Alive" -H "Proxy-Connection: Keep-Alive" -I [00:23:45] [ Revision history of "User:Southparkfan/Test" - Miraheze Meta ] - meta.miraheze.org [00:24:59] that gives me curl: (56) Received HTTP code 400 from proxy after CONNECT [00:25:30] works for me [00:25:55] not here [00:25:56] SPF|Cloud https://phabricator.miraheze.org/P378 [00:25:57] [ ✎ P378 (An Untitled Masterwork) ] - phabricator.miraheze.org [00:25:58] but [00:26:05] southparkfan@test2:/srv/mediawiki/config$ curl -X PURGE --url "/wiki/User:Southparkfan/Text" -H "Host: meta.miraheze.org" --proxy 51.77.107.210:81 -H "user-agent: MediaWiki/1.35.1 CdnCacheUpdate" -I [00:26:05] curl: (3) URL using bad/illegal format or missing URL [00:26:13] this will fail before we even see traffic at cp6 [00:26:18] so this must be the problem [00:26:27] yeh [00:26:33] setting the full url fixed the problem [00:27:38] your example will need -H 'X-Device: phone-tablet' etc, but yes [00:27:57] result of a curl upgrade? [00:28:42] possibly [00:28:47] aha, it gives the error for cp3, because test2 can't reach cp3 [00:28:56] ah [00:29:00] yeh [00:29:01] guess the firewall rule has not been applied yet [00:29:05] it has [00:29:08] but i automated it [00:29:14] test2 isn't connected to puppet3 [00:29:30] yes and cp6 isn't either [00:29:37] yeh [00:29:37] so if puppet has ran, this should work? [00:29:40] no [00:29:45] cp3 is connected to puppet3 [00:29:57] thus puppetdb used on puppetdb, does not have test2 [00:30:10] understood [00:30:27] i only switched cp3 to puppet3 because it had the rules for mw[4567] already without risk of loosing them. [00:32:22] anyways is there anyway we can test i wonder... [00:32:30] we've all upgraded to buster, hmm [00:35:56] ok, so the problem is: the urls do not contain the hostname [00:36:14] but why ;) [00:39:01] https://github.com/miraheze/mediawiki/blob/REL1_35/includes/deferred/CdnCacheUpdate.php#L307 only sends wfParseUrl( wfExpandUrl( '/wiki/User:Southparkfan/Test', PROTO_INTERNAL ) )['path'], which is /wiki/User:Southparkfan/Test [00:39:02] [ mediawiki/CdnCacheUpdate.php at REL1_35 · miraheze/mediawiki · GitHub ] - github.com [00:39:23] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:39:25] oh SPF|Cloud wgInternalServer [00:39:26] https://github.com/wikimedia/mediawiki/blob/467a4f32b0475fd2840264ada94e3ce87518d22a/includes/GlobalFunctions.php#L497 [00:39:27] [ mediawiki/GlobalFunctions.php at 467a4f32b0475fd2840264ada94e3ce87518d22a · wikimedia/mediawiki · GitHub ] - github.com [00:40:06] ? [00:40:07] hmm [00:40:10] nvm [00:40:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtzeI [00:40:16] [02miraheze/services] 07MirahezeSSLBot 036792447 - BOT: Updating services config for wikis [00:40:19] i was following the traceback [00:40:53] https://github.com/wikimedia/mediawiki/commit/1260bf71da9c5ba8fb25fdcfbab30b128c30010a [00:40:53] [ Migrate CdnCacheUpdate to MultiHttpClient and deprecate SquidPurgeClient · wikimedia/mediawiki@1260bf7 · GitHub ] - github.com [00:41:13] https://github.com/wikimedia/mediawiki/commit/1260bf71da9c5ba8fb25fdcfbab30b128c30010a#diff-1691e99810c6af8b7fda983ba8ceb8529c1085f0626d11241d2427afdc286d59L163 before, the full url was sent, including hostname [00:41:13] [ Migrate CdnCacheUpdate to MultiHttpClient and deprecate SquidPurgeClient · wikimedia/mediawiki@1260bf7 · GitHub ] - github.com [00:41:33] https://github.com/wikimedia/mediawiki/commit/1260bf71da9c5ba8fb25fdcfbab30b128c30010a#diff-1691e99810c6af8b7fda983ba8ceb8529c1085f0626d11241d2427afdc286d59R305 but now, only the path is sent [00:41:33] [ Migrate CdnCacheUpdate to MultiHttpClient and deprecate SquidPurgeClient · wikimedia/mediawiki@1260bf7 · GitHub ] - github.com [00:41:46] when did we deploy 1.35? [00:43:00] > var_dump( wfExpandUrl( '/w/index.php?title=User:Southparkfan/Test&action=history', PROTO_INTERNAL ) ); [00:43:00] string(81) "https://meta.miraheze.org/w/index.php?title=User:Southparkfan/Test&action=history" [00:43:00] [ Revision history of "User:Southparkfan/Test" - Miraheze Meta ] - meta.miraheze.org [00:43:12] SPF|Cloud novemeber i think [00:43:17] yes, that is the correct URL [00:43:30] but nowadays only the path is sent, ie everything but "https://meta.miraheze.org" [00:43:31] [ Meta ] - meta.miraheze.org [00:43:31] oh [00:43:33] actually [00:44:10] october time [00:44:36] let me test a patch [00:44:57] > var_dump( wfParseUrl( wfExpandUrl( '/w/index.php?title=User:Southparkfan/Test&action=history', PROTO_INTERNAL ) ) ); [00:44:57] array(5) { [00:44:57] ["scheme"]=> [00:44:57] string(5) "https" [00:44:57] ["host"]=> [00:44:59] string(17) "meta.miraheze.org" [00:45:00] ["path"]=> [00:45:01] string(12) "/w/index.php" [00:45:02] ["query"]=> [00:45:03] string(43) "title=User:Southparkfan/Test&action=history" [00:45:04] ["delimiter"]=> [00:45:05] string(3) "://" [00:45:06] } [00:45:35] yes? [00:45:57] its using >path and query so hence why it doesn't include full domain [00:46:05] that's what I said [00:46:24] 01:41:13 <+SPF|Cloud> https://github.com/wikimedia/mediawiki/commit/1260bf71da9c5ba8fb25fdcfbab30b128c30010a#diff-1691e99810c6af8b7fda983ba8ceb8529c1085f0626d11241d2427afdc286d59L163 before, the full url was sent, including hostname [00:46:24] 01:41:33 <+SPF|Cloud> https://github.com/wikimedia/mediawiki/commit/1260bf71da9c5ba8fb25fdcfbab30b128c30010a#diff-1691e99810c6af8b7fda983ba8ceb8529c1085f0626d11241d2427afdc286d59R305 but now, only the path is sent [00:46:25] [ Migrate CdnCacheUpdate to MultiHttpClient and deprecate SquidPurgeClient · wikimedia/mediawiki@1260bf7 · GitHub ] - github.com [00:46:26] [ Migrate CdnCacheUpdate to MultiHttpClient and deprecate SquidPurgeClient · wikimedia/mediawiki@1260bf7 · GitHub ] - github.com [00:46:28] sounds like a regression [00:46:57] ohh i se [00:46:58] see [00:47:11] if it was a regression wouldn't upstream have picked it up? [00:47:50] seeing as they use the purger for varnish cache too [00:48:00] Wikimedia used htcp purging [00:48:08] and nowadays some sort of kafka based thing [00:48:18] https://wikitech.wikimedia.org/wiki/Multicast_HTCP_purging [00:48:19] [ Multicast HTCP purging - Wikitech ] - wikitech.wikimedia.org [00:48:46] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.48, 5.58, 4.91 [00:50:19] oh [00:50:53] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.96, 5.74, 5.06 [00:52:50] uh [00:52:52] RECOVERY - gluster2 Current Load on gluster2 is OK: OK - load average: 3.49, 4.84, 4.81 [00:53:03] SPF|Cloud hmm? [00:53:20] https://www.irccloud.com/pastebin/PPXg2g7r/ [00:53:21] [ Snippet | IRCCloud ] - www.irccloud.com [00:53:26] root@cp6:/var/log/varnish# cat varnishncsa.log | grep \"PURGE [00:53:26] 51.77.107.211 - - [04/Feb/2021:00:16:24 +0000] "PURGE http://meta.miraheze.org--proxy/ HTTP/1.1" 200 243 "-" "MediaWiki/1.35.1 CdnCacheUpdate" [00:53:51] that was because you didn't put a space between the url and --proxy :P [00:54:11] oh [00:54:42] you can see in my paste that I get a response from cp7 nginx [00:54:49] curl: (56) Received HTTP code 400 from proxy after CONNECT [00:55:03] mediawiki talks to varnish directly, but nginx responds? [00:56:51] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.46, 5.24, 4.98 [01:01:47] .CONNECT.meta 0x0050: 2e6d 6972 6168 657a 652e 6f72 673a 3434 .miraheze.org:44 0x0060: 3320 4854 5450 2f31 2e31 0d0a 486f 7374 3.HTTP/1.1..Host [01:02:50] RECOVERY - gluster2 Current Load on gluster2 is OK: OK - load average: 4.95, 5.05, 5.03 [01:03:24] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 27.47, 21.18, 18.64 [01:05:19] I'm flabbergasted [01:05:28] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.33, 21.48, 19.13 [01:06:48] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 4.37, 5.34, 5.23 [01:07:25] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 17.82, 19.79, 18.77 [01:07:56] cp3: (curl error: 28) Timeout was reached [01:08:25] * Establish HTTP proxy tunnel to meta.miraheze.org:443 [01:09:09] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:10:14] well, I can reproduce the cache proxy response thing [01:10:33] curl -X PURGE --url "https://meta.miraheze.org/wiki/User:Southparkfan/Test" -H "Host: meta.miraheze.org" --proxy '51.77.107.210:81' -H "User-Agent: MediaWiki/1.35.1 CdnCacheUpdate" -H "Connection: Keep-Alive" -H "Proxy-Connection: Keep-Alive" -H 'X-Device: desktop' [01:10:33] [ User:Southparkfan/Test - Miraheze Meta ] - meta.miraheze.org [01:10:48] curl -X PURGE "w/index.php?title=User:Southparkfan/Test&action=history" -H "Host: meta.miraheze.org" --proxy 51.77.107.210:81 [01:11:05] curl -X PURGE "\/w/index.php?title=User:Southparkfan/Test&action=history" -H "Host: meta.miraheze.org" --proxy 51.77.107.210:81 [01:11:07] SPF|Cloud ^ [01:11:38] does that work? [01:11:40] yeh [01:11:44] good [01:11:47] it returns 200 [01:11:54] but can you check whether it does the content? [01:12:06] that won't purge content obviously [01:12:06] since i did \/ not sure if that breaks something... [01:12:14] why? [01:12:15] content is /wiki/User:Southparkfan/Test :) [01:12:20] ohhh [01:12:22] you're only purging the history page [01:12:22] sorry [01:12:27] the history page [01:12:51] RECOVERY - gluster2 Current Load on gluster2 is OK: OK - load average: 3.40, 4.46, 4.91 [01:13:26] curl -X PURGE "%2Fw%2Findex.php%3Ftitle%3DUser%3ASouthparkfan%2FTest%26action%3Dhistory" -H "Host: meta.miraheze.org" --proxy 51.77.107.210:81 [01:13:27] also works [01:14:53] encoding.. [01:16:33] curl -X PURGE --url "%2Fwiki%2FUser%3ASouthparkfan%2FTest" doesn't return an error, but varnish has trouble with the url [01:17:52] what do you mean trouble? [01:18:09] varnish won't decode the url [01:18:14] oh i see [01:18:20] does \/ that work? [01:18:36] or wait.. [01:18:39] "PURGE http://%2Fwiki%2FUser%3ASouthparkfan%2FTest/ HTTP/1.1" 200 243 "-" "MediaWiki/1.35.1 CdnCacheUpdate" [01:19:24] heh [01:19:33] --url '\/wiki/User:Southparkfan/Test' results in "PURGE http://\\/wiki/User:Southparkfan/Test HTTP/1.1" 200 242 "-" "MediaWiki/1.35.1 CdnCacheUpdate" [01:22:18] what did i miss [01:25:41] purge works now? [01:25:45] huh [01:26:21] does it? [01:26:41] i mean using http:// works but https:// doesn't. But in total neither really work [01:26:48] i tried purging https://meta.miraheze.org/wiki/User:Paladox/test [01:26:50] [ User:Paladox/test - Miraheze Meta ] - meta.miraheze.org [01:26:50] and failed [01:26:56] yep, just found out you shouldn't use https:// [01:27:01] varnish doesn't talk https [01:28:15] my bad, my requests were passed to the backend due to a cookie thing [01:29:10] I think I have a fix [01:30:05] oh? [01:34:20] let's see [01:34:50] BINGO [01:35:23] we were right about self::expand( $url ), because varnish does need the 'meta.miraheze.org/' part [01:35:46] it works for you? [01:35:48] however, self::expand() returns an https:// url, but varnish only understands http:// [01:35:52] i'm reloading the page for my page [01:35:55] and it hasn't cleared [01:36:11] edit your page first? [01:36:31] i did [01:36:42] i've been trying to manually purge it using curl [01:37:08] why do that if you have https://meta.miraheze.org/wiki/User:Paladox/test?action=purge :P [01:37:09] [ User:Paladox/test - Miraheze Meta ] - meta.miraheze.org [01:37:23] oh [01:38:03] it works :D [01:38:35] but I don't understand why this has worked before [01:39:32] $url = str_replace( "\n", '', $url ); [01:39:44] nvm [01:40:00] because wfExpandUrl( $url, PROTO_INTERNAL ) has always returned https:// urls [01:41:22] - 'url' => $urlPath, [01:41:23] + 'url' => str_replace( 'https://', 'http://', self::expand( $url ) ), [01:41:46] i'm going to try something [01:41:56] https://github.com/wikimedia/mediawiki/blob/1260bf71da9c5ba8fb25fdcfbab30b128c30010a/includes/clientpool/SquidPurgeClient.php#L200 [01:41:56] [ mediawiki/SquidPurgeClient.php at 1260bf71da9c5ba8fb25fdcfbab30b128c30010a · wikimedia/mediawiki · GitHub ] - github.com [01:41:59] instead of my str_replace hack, I could https://github.com/miraheze/mediawiki/blob/REL1_35/includes/deferred/CdnCacheUpdate.php#L344 use PROTO_HTTP here instead of PROTO_INTERNAL [01:42:00] [ mediawiki/CdnCacheUpdate.php at REL1_35 · miraheze/mediawiki · GitHub ] - github.com [01:42:26] ? [01:42:45] @SPF|Cloud I wonder if this purge / Varnish purge issue is related to the number of subpages in "Page Information" not updating for a couple days after you delete a given page [01:43:10] eh, https://www.mediawiki.org/wiki/Manual:$wgSquidPurgeUseHostHeader is a thing [01:43:11] [ Manual:$wgSquidPurgeUseHostHeader - MediaWiki ] - www.mediawiki.org [01:43:24] oh [01:43:43] * dmehus has never heard of that MediaWiki variable and is looking [01:44:14] the default of that variable is true, though [01:44:22] @SPF|Cloud says it was deprecated in MW 1.33 [01:44:31] I know [01:44:49] but the existence of that variable is somewhat related here [01:44:53] relevant* [01:44:59] ah [01:45:02] interesting [01:46:40] so, in the past, we used SquidPurgeClient which would perform a PURGE /wiki/User:Southparkfan/Test HTTP/1.1 Host: meta.miraheze.org [01:49:01] wow, things start to make sense [01:49:47] to SquidPurgeClient, MediaWiki sent the full URL, but because of wgSquidPurgeUseHostHeader, https://github.com/wikimedia/mediawiki/blob/REL1_35/includes/clientpool/SquidPurgeClient.php#L202 the request became a PURGE \n Host: [01:49:48] [ mediawiki/SquidPurgeClient.php at REL1_35 · wikimedia/mediawiki · GitHub ] - github.com [01:50:06] you can set the second param to 'http' [01:51:09] hmm nvm [01:51:24] in https://github.com/wikimedia/mediawiki/commit/1260bf71da9c5ba8fb25fdcfbab30b128c30010a, the Wikimedia developers started using curl instead of crafting raw PURGEs for varnish [01:51:25] [ Migrate CdnCacheUpdate to MultiHttpClient and deprecate SquidPurgeClient · wikimedia/mediawiki@1260bf7 · GitHub ] - github.com [01:51:41] however, curl cannot deal with urls that do not contain hostnames, hence it will fail [01:52:13] and that is a regression [01:55:09] SPF|Cloud updated the patch to add a comment [01:55:45] going to push this change to miraheze/mediawiki [01:56:13] ok :) [01:56:20] i can create a patch on gerrit [01:56:29] unless you want to? [01:56:41] As long as you credit me I don't mind :-P [01:57:19] sure [01:58:01] @SPF|Cloud ack [01:59:53] [02miraheze/mediawiki] 07Southparkfan pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtzJn [01:59:55] [02miraheze/mediawiki] 07Southparkfan 038731e01 - CdnCacheUpdate: send full URL for purge + assume HTTP instead of HTTPS Explanation added as a comment. [02:01:00] SPF|Cloud you broke it [02:01:35] shrug [02:02:05] almost right.. [02:02:27] [02miraheze/mediawiki] 07Southparkfan pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtzJR [02:02:29] [02miraheze/mediawiki] 07Southparkfan 039b2c230 - Fix copy mistake [02:03:14] why not just keep 'url' => [02:03:16] no, again [02:03:22] this is still incorrect, argh [02:03:34] * SPF|Cloud fixes again [02:04:32] [02miraheze/mediawiki] 07Southparkfan pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtzJE [02:04:33] [02miraheze/mediawiki] 07Southparkfan 0389e4771 - Writing code is hard [02:04:43] SPF|Cloud https://gerrit.wikimedia.org/r/c/mediawiki/core/+/661546 [02:06:08] cmetz: we believe we have fixed your cache problems [02:06:33] i guess tests may fail... [02:06:44] cool patch [02:08:21] mediawiki is flooding purges to varnish, sounds like we have fixed this [02:08:32] it works :D [02:08:46] good work paladox [02:08:58] and you too! [02:09:01] thanks SPF|Cloud ! [02:09:20] it has been years since I have touched mediawiki [02:09:48] but my skills are just as good as they were back then, if not even better [02:09:53] heh [02:10:41] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [02:11:08] and that fixes files too SPF|Cloud :D [02:11:20] "PURGE http://static.miraheze.org/vsrecommendedgameswiki/b/bd/Pick_Axe_Pete_Odyssey2_screenshot.png HTTP/1.1" 200 242 "-" "MediaWiki/1.35.1 CdnCacheUpdate" [02:11:38] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [02:11:48] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [02:12:13] so our purges were broken since october? [02:12:26] apparently, yeh. [02:12:46] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [02:12:48] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [02:17:33] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtzJH [02:17:35] [02miraheze/mw-config] 07paladox 0371470a5 - Switch over wikis beginning with g to new file storage [02:18:42] miraheze/mw-config - paladox the build passed. [02:21:23] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtzJx [02:21:24] [02miraheze/mw-config] 07paladox 0378f7328 - Switch over wikis beginning with i to new file storage [02:22:25] miraheze/mw-config - paladox the build passed. [02:29:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtzUc [02:29:23] [02miraheze/puppet] 07paladox 03ce79770 - mediawiki: Add location regex block to make some wikis use new file storage [02:29:25] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [02:29:26] [02puppet] 07paladox opened pull request 03#1625: mediawiki: Add location regex block to make some wikis use new file s… - 13https://git.io/JtzUC [02:29:28] [02puppet] 07paladox edited pull request 03#1625: mediawiki: Add location regex block to make some wikis use new file storage - 13https://git.io/JtzUC [02:29:35] [02puppet] 07paladox edited pull request 03#1625: mediawiki: Add location regex block to make some wikis use new file storage - 13https://git.io/JtzUC [02:29:42] .gj paladox [02:29:42] You're doing good work, paladox! [02:29:48] .gj SPF|Cloud [02:29:48] You're doing good work, SPF|Cloud! [02:30:36] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.40, 4.39, 3.24 [02:31:20] sure [02:31:26] and now I need sleep [02:31:34] night [02:31:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtzUB [02:31:39] [02miraheze/puppet] 07paladox 033f4acf8 - Update mediawiki.conf.erb [02:31:40] [02puppet] 07paladox synchronize pull request 03#1625: mediawiki: Add location regex block to make some wikis use new file storage - 13https://git.io/JtzUC [02:32:31] night [02:32:39] RECOVERY - gluster2 Current Load on gluster2 is OK: OK - load average: 4.35, 4.17, 3.30 [02:34:04] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 6.07, 5.41, 3.84 [02:34:42] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [02:34:44] [02puppet] 07paladox closed pull request 03#1625: mediawiki: Add location regex block to make some wikis use new file storage - 13https://git.io/JtzUC [02:34:45] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [02:34:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtzUr [02:34:46] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:34:47] [02miraheze/puppet] 07paladox 0314a1707 - mediawiki: Add location regex block to make some wikis use new file storage (#1625) * mediawiki: Add location regex block to make some wikis use new file storage * Update mediawiki.conf.erb [02:34:49] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [02:34:50] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [02:35:00] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.60, 20.68, 16.54 [02:35:25] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtzUK [02:35:26] [02miraheze/mw-config] 07paladox 03622cb86 - Don't need to use static-new, regex in use to make it use new mount [02:35:42] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:35:54] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:36:04] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 3.97, 4.95, 3.87 [02:36:31] miraheze/mw-config - paladox the build passed. [02:36:57] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 18.50, 20.09, 16.84 [02:38:41] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.79, 5.46, 4.11 [02:40:40] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.30, 5.36, 4.23 [02:42:39] RECOVERY - gluster2 Current Load on gluster2 is OK: OK - load average: 4.45, 5.00, 4.24 [02:45:27] RECOVERY - gluster3 APT on gluster3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [02:46:59] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.21, 4.58, 3.63 [02:48:41] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.84, 5.37, 4.65 [02:50:44] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 7.93, 6.14, 5.01 [02:51:01] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.84, 4.55, 3.87 [03:02:42] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.48, 5.77, 5.54 [03:04:41] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.29, 5.96, 5.63 [03:06:41] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.98, 5.86, 5.63 [03:08:45] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.16, 5.99, 5.70 [03:09:38] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.03, 22.33, 19.62 [03:13:41] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.71, 22.74, 20.52 [03:14:44] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 7.26, 5.66, 4.96 [03:15:40] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.21, 19.97, 19.80 [03:16:43] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.51, 5.45, 4.96 [03:18:40] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 9.30, 6.85, 5.54 [03:22:48] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 4.41, 5.30, 5.65 [03:23:21] PROBLEM - wiki.ripto.gq - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ripto.gq could not be found [03:23:28] PROBLEM - wiki.etn.link - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.etn.link could not be found [03:23:29] PROBLEM - meta.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.gyaanipedia.com could not be found [03:23:29] PROBLEM - wiki.sdbx.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sdbx.dev could not be found [03:28:46] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.50, 5.47, 4.79 [03:30:06] RECOVERY - wiki.ripto.gq - reverse DNS on sslhost is OK: rDNS OK - wiki.ripto.gq reverse DNS resolves to cp6.miraheze.org [03:30:18] RECOVERY - wiki.etn.link - reverse DNS on sslhost is OK: rDNS OK - wiki.etn.link reverse DNS resolves to cp6.miraheze.org [03:30:20] RECOVERY - meta.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - meta.gyaanipedia.com reverse DNS resolves to cp7.miraheze.org [03:30:22] RECOVERY - wiki.sdbx.dev - reverse DNS on sslhost is OK: rDNS OK - wiki.sdbx.dev reverse DNS resolves to cp6.miraheze.org [03:32:41] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.86, 5.91, 5.20 [03:34:49] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.04, 5.20, 5.33 [03:38:36] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.67, 5.96, 5.37 [03:40:47] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.59, 5.54, 5.48 [03:46:26] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.45, 5.80, 6.00 [03:46:28] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.13, 5.82, 5.61 [03:50:26] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.25, 6.14, 5.79 [03:50:28] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 7.60, 6.24, 6.09 [03:50:48] RECOVERY - gluster2 Current Load on gluster2 is OK: OK - load average: 3.73, 4.52, 5.04 [03:56:23] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.38, 5.95, 5.83 [04:02:18] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.33, 6.01, 5.87 [04:06:13] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 3.37, 4.86, 5.74 [04:06:14] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 1.44, 4.37, 5.29 [04:08:11] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 0.21, 2.93, 4.65 [04:10:10] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 2.78, 3.78, 5.10 [04:22:18] PROBLEM - gluster3 Puppet on gluster3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [04:58:53] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [06:02:09] RECOVERY - phab2 APT on phab2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:03:14] RECOVERY - db12 APT on db12 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [06:03:59] RECOVERY - db12 APT on db12 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [06:04:45] RECOVERY - mon1 APT on mon1 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [06:05:05] RECOVERY - services1 APT on services1 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [06:05:51] RECOVERY - mail1 APT on mail1 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [06:08:04] RECOVERY - mon2 APT on mon2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:08:57] RECOVERY - mw4 APT on mw4 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [06:09:41] RECOVERY - rdb4 APT on rdb4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:10:30] RECOVERY - cp6 APT on cp6 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [06:10:58] RECOVERY - gluster4 APT on gluster4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:14:50] RECOVERY - cp10 APT on cp10 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:16:53] RECOVERY - ns2 APT on ns2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:16:53] RECOVERY - mw9 APT on mw9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:17:29] RECOVERY - cloud1 APT on cloud1 is OK: APT OK: 97 packages available for upgrade (0 critical updates). [06:17:36] RECOVERY - puppet3 APT on puppet3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:18:25] RECOVERY - mw5 APT on mw5 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [06:18:44] RECOVERY - puppet3 APT on puppet3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:19:06] RECOVERY - jobrunner2 APT on jobrunner2 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [06:19:20] RECOVERY - db13 APT on db13 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [06:19:22] RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [06:20:17] RECOVERY - db13 APT on db13 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [06:20:31] RECOVERY - rdb1 APT on rdb1 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [06:24:09] RECOVERY - rdb3 APT on rdb3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:25:49] RECOVERY - cp11 APT on cp11 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:27:03]