[00:01:16] hey guys [00:02:59] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.13, 19.90, 15.05 [00:05:00] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 10.69, 16.27, 14.31 [00:08:17] Hello plainoldbarrier! If you have any questions, feel free to ask and someone should answer soon. [02:24:41] [02ssl] 07dmehus commented on pull request 03#365: Add arrowversewiki redirect to dcmultiversewiki - 13https://git.io/JTmne [02:30:19] https://usercontent.irccloud-cdn.com/file/kXm55xYz/image.png [02:30:26] dmehus Are dittos like this really necessary [02:30:52] I also see he decided to approve 14727 which I asked for more information on [02:34:35] @Naleksuh I just wanted to acknowledge your previous message as I was managing the RequestWiki queue. As far as 14727 goes, was that the worldbuilding wiki? I should give you the `wikicreator` tips I gave CircleyDoesExtracter. You may find them helpful [02:35:09] I don't think more than a single response should be done unless there is something critical to mention [05:25:24] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JTmgr [05:25:26] [02miraheze/ssl] 07Reception123 0359ce0d0 - add bharatwiki.online cert [05:45:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTmaC [05:45:13] [02miraheze/services] 07MirahezeSSLBot 03547bf25 - BOT: Updating services config for wikis [05:47:59] !log reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki testwiki --r "Requested - [[phab:T6307|T6307]]" /home/reception/testwikidel.txt [05:48:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:52:55] !log MariaDB [csydeswiki]> UPDATE nl_newsletters SET nl_main_page_id = 2 WHERE nl_name = 'Newsletter2'; [05:52:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:53:29] !log MariaDB [csydeswiki]> UPDATE nl_newsletters SET nl_active = 0 WHERE nl_name = 'Newsletter2'; [05:53:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:55:48] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JTmVW [05:55:49] [02miraheze/dns] 07Reception123 034f6ba2e - add wikibeats.net zone [06:03:21] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JTmV7 [06:03:23] [02miraheze/ssl] 07Reception123 039be3999 - add wikibeats.net cert [06:15:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTmw9 [06:15:13] [02miraheze/services] 07MirahezeSSLBot 034e1aaeb - BOT: Updating services config for wikis [06:21:35] [02mw-config] 07Reception123 closed pull request 03#3289: Add medik configs for lakehubwiki (T6306) - 13https://git.io/JTLKd [06:21:36] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTmrX [06:21:38] [02miraheze/mw-config] 07Universal-Omega 036f137d6 - Add medik configs for Lakehub wiki (T6306) (#3289) [06:34:55] [02mw-config] 07Reception123 commented on pull request 03#3285: Rename Flow to StructuredDiscussions in ManageWiki, and resort it alp… - 13https://git.io/JTmKl [06:35:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTmKR [06:35:12] [02miraheze/services] 07MirahezeSSLBot 0300f3cc7 - BOT: Updating services config for wikis [06:35:15] [02mw-config] 07Reception123 commented on pull request 03#3285: Rename Flow to StructuredDiscussions in ManageWiki, and resort it alp… - 13https://git.io/JTmKE [12:55:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTYZW [12:55:12] [02miraheze/services] 07MirahezeSSLBot 03c9c2374 - BOT: Updating services config for wikis [14:23:28] PROBLEM - wiki.hrznstudio.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.hrznstudio.com' expires in 15 day(s) (Fri 30 Oct 2020 14:20:01 GMT +0000). [14:28:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTYEV [14:29:00] [02miraheze/ssl] 07MirahezeSSLBot 03332213a - Bot: Update SSL cert for wiki.hrznstudio.com [14:36:48] RECOVERY - wiki.hrznstudio.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hrznstudio.com' will expire on Tue 12 Jan 2021 13:28:51 GMT +0000. [15:12:54] My wiki (cfd.miraheze.org) stopped working for two days. [15:13:06] This is the error message: [15:13:09] MediaWiki internal error. [15:14:04] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.25, 19.45, 18.01 [15:15:59] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.78, 18.48, 17.84 [15:17:10] cfd005: Did you make any modification in ManageWiki before the error? What was it? [15:18:10] No. I did nothing to the ManageWiki. [15:18:46] ok [15:20:03] @Site Reliability Engineers on cfdwiki bug: Original exception: [a26049bb3595ab41e4902691] 2020-10-14 15:18:52: Fatal exception of type "MediaWiki\Storage\NameTableAccessException". ^ [15:23:06] ^ hopes that's a punctuation error or typo as then @paladox will probably have the PR done before he even replies here heh [15:28:51] @Hispano76 just noting that you can ping @ System administrators and not only SRE when it comes to mediawiki errors 🙂 [15:32:53] !log fix content_models on cfdwiki [15:32:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:33:50] Thanks a lot. [15:46:56] :) [15:55:10] dmehus still doing it [15:55:12] what is "RequestWiki queue management" [16:06:08] Hello Osprey1990! If you have any questions, feel free to ask and someone should answer soon. [16:07:11] @Naleksuh yeah, well, I realize wiki requestors are notified of new comments now, but the RequestWikiQueue can fill up in a hurry. I generally wait at least a few hours before procedurally declining, though. Also, I'm not sure the content of the notification e-mails. Requestors may be spurred into editing their request when they see it's been declined versus commented on. As wiki creators, though, we only generate activity based on our approvals [16:07:11] and creations, not on our comments. So it's actually to the wiki creators' benefit to decline a request, noting what information is missing or why it was declined in your decline reason. It's not mandatory to do it either way, but we're pretty flexible in allowing each wiki creator to use their own style [16:07:48] Sure, but am I allowed to 'use my own style' when you decline requests others are already handling? [16:07:56] I'm having problems with my template and I don't know how to post to Phabricator. You can see the problem here: https://pokemonfakemon.miraheze.org/wiki/User:Osprey1990/Sandbox [16:07:57] [ User:Osprey1990/Sandbox - PokémonFakémon ] - pokemonfakemon.miraheze.org [16:08:13] I want to use my template like a citation [16:09:23] @Naleksuh We're both using our own style; I'm waiting a few hours, generally, before managing the RequestWikiQueue. Plus, another reason which may encourage you to adapt this approach, by declining the request, the requestor will generate an RC entry when they edit the request to add a new comment, which improves the trackability of the request [16:09:57] Yeah, but I mean that you comment on requests I already commented on and declining them [16:09:57] Which is somewhat preventing me from doing it my way [16:13:47] @Naleksuh Oh, well I reiterate your comment in the decline reason as I'm not sure what the content of the decline e-mail is, like if it's using the most recently added comment or not. Could probably test this by requesting a wiki then declining it [16:14:25] why do you like your method, though? You did say you would like to be able to monitor wiki request changes and comments in RC; this would allow for this [16:14:33] dmehus: I don't think the email uses any of the content [16:14:49] that would be used in the Echo notification, but at least as far as I'm aware the email wouldn't contain any specific comments [16:14:56] https://www.irccloud.com/pastebin/IDc3ZkbQ/ [16:14:57] [ Snippet | IRCCloud ] - www.irccloud.com [16:15:20] I think the best solution would be simply editing the createwiki extension to remove these issues [16:15:34] Like I want to ensure notifications for comments [16:15:39] Does anyone here have an alt [16:15:58] Osprey1990: You can check this out or ask for guidance on the MetaWiki community board. Phabricator is not for template but for MediaWiki configurations https://en.wikipedia.org/wiki/Template:Cite_web [16:15:59] [WIKIPEDIA] Template:Cite web | "..." [16:16:09] there are on-wiki notifications for comments (via Echo) [16:16:16] they've been added by JohnLewis recently [16:16:33] Thanks hispano76, I'll see if I can figure it out [16:16:47] dmehus: "As wiki creators, though, we only generate activity based on our approvals and creations, not on our comments." hm? [16:17:13] Yeah there is notifications for comments [16:17:13] What I want though is notifications in recent changes for comments [16:17:28] I think he's referring to those wiki stats under your userpage (User:John/WikiStats) when he means activity [16:17:36] as those count wiki creations per user [16:17:59] Naleksuh: oh, well wouldn't a log entry for every single comment be quite messy and overwhelming? [16:18:31] and what use would that be? Users can see the comments via Echo, and a wiki creator can see that someone has commented before when they access the "Request comments" tab [16:18:32] There are generally only 1-2 comments [16:18:51] I've seen more quite a few times, and what would the benefit of an extra log for comments me? [16:18:53] *be [16:22:10] @JohnLewis, referring to **Special:ActiveUsers** [16:22:30] and what @Reception123 said for the approvals, yes [16:22:35] dmehus: oh, okay :) [16:24:22] Sure, I just wish others would stop actioning on requests that already were [16:24:22] Because its just unnecessarily complicated and confusing for the person requesting [16:24:47] "dmehus: I don't think the email uses any of the content [16:24:47] 09:14 that would be used in the Echo notification, but at least as far as I'm aware the email wouldn't contain any specific comments" [16:24:47] Okay, fair enough. Don't need to add any additional comment in declining then. [16:24:47] "I think the best solution would be simply editing the createwiki extension to remove these issues" [16:24:47] Perhaps, yeah, though we'd probably want to add another classification besides "in review" such as an "awaiting response" classification, to better manage the wiki request queue [16:25:59] Yeah my ideas for categories would be New, Awaiting Creator, Awaiting Requestor, Approved, Denied [16:26:32] what's the difference between New and awaiting creator? [16:26:38] by definition all new requests are awaiting a creator [16:27:13] and to me having that many categories would be what would be unnecessarily complicated [16:27:26] yeah I agree with @Reception123. I would just have a "new" (which would be "inreview" renamed), "awaiting follow up", "approved", and "declined" [16:27:38] and while JohnLewis knows more about it than me, there is an AI project that is in the works and will be used eventually [16:27:48] see https://phabricator.miraheze.org/T5105 [16:27:49] [ ⚓ T5105 Investigate and Implement basic Machine Learning concepts for automatic wiki creation ] - phabricator.miraheze.org [16:28:36] so I'm also not sure the work of creating all those new categories is necessary, especially if the system changes following the machine learning concept, though as I said I'm not sure what the status with that is [16:28:57] @Naleksuh, well, I don't mind when other wiki creators action requests I'm following. It's helpful. Sometimes I do disagree with another wiki creator creating a wiki that shouldn't have been approved, but, at the same time, it wasn't me approving it, either, and it's not all that common [16:29:15] It’s still being worked on but is kind of blocked on a few separate things [16:30:29] oh, will those things be difficult to overcome? [16:32:12] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.08, 22.63, 19.52 [16:34:08] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.88, 21.57, 19.54 [16:34:47] no i'm being replaced with machine ;(( [16:35:43] Reception123: no, just the standard lack of time [16:35:51] ah [16:35:59] Naleksuh: lol, it won't quite be like that :P [16:37:13] PROBLEM - test2 Puppet on test2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 2 failures [16:37:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JTYy0 [16:37:23] [02miraheze/puppet] 07paladox 03fcc1ad7 - test2: Upgrade to mediawiki 1.35 [16:37:25] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [16:37:26] [02puppet] 07paladox opened pull request 03#1507: test2: Upgrade to mediawiki 1.35 - 13https://git.io/JTYyg [16:37:34] [02puppet] 07paladox closed pull request 03#1507: test2: Upgrade to mediawiki 1.35 - 13https://git.io/JTYyg [16:37:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTYya [16:37:37] [02miraheze/puppet] 07paladox 03c28d239 - test2: Upgrade to mediawiki 1.35 (#1507) [16:37:38] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [16:37:40] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [16:37:59] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.68, 19.14, 18.98 [16:38:02] !log upgrade test2 to 1.35 [16:38:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:38:43] paladox: thanks! :D [16:40:25] PROBLEM - test2 MediaWiki Rendering on test2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 261 bytes in 0.028 second response time [16:43:48] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.65, 20.63, 19.39 [16:44:24] PROBLEM - test2 MediaWiki Rendering on test2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.343 second response time [16:45:32] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTYS2 [16:45:34] [02miraheze/ssl] 07Reception123 035598594 - attempt to renew savage-wiki.com [16:45:43] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.22, 19.09, 18.98 [16:51:33] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.01, 22.50, 20.58 [16:52:24] RECOVERY - test2 MediaWiki Rendering on test2 is OK: HTTP OK: HTTP/1.1 200 OK - 1260 bytes in 0.055 second response time [16:53:28] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.60, 22.23, 20.69 [16:54:37] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:54:52] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:54:56] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:55:20] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:55:21] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:55:23] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_pull_MediaWiki core] [16:55:32] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 30.44, 24.59, 21.70 [16:55:32] PROBLEM - cp9 Puppet on cp9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:55:58] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:56:11] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:56:21] Reception123 [16:56:25] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:56:30] Notice: /Stage[main]/Mediawiki::Nginx/Exec[nginx-syntax]/returns: nginx: [emerg] SSL_CTX_use_PrivateKey_file("/etc/ssl/private/savage-wiki.com.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch) [16:57:04] paladox: hm, I don't get why [16:57:10] See the latest email to tech@ [16:57:25] I gave them the CSR we used in 2018 [16:57:25] private key doesn't match. [16:57:27] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.42, 22.83, 21.40 [16:57:33] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:57:34] you must have generated a CSR with a new key? [16:57:42] Nope, I used the same one [16:57:46] From the Phab task [16:58:24] well then the cert does not match [16:58:24] PROBLEM - test2 MediaWiki Rendering on test2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2500 bytes in 0.024 second response time [16:59:41] paladox: that's odd [17:00:50] PROBLEM - ping6 on misc1 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 451.43 ms [17:02:32] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTYHV [17:02:33] [02miraheze/ssl] 07Reception123 0376547d3 - Revert [17:02:36] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [17:02:50] RECOVERY - ping6 on misc1 is OK: PING OK - Packet loss = 0%, RTA = 39.52 ms [17:02:51] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:02:52] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:03:03] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTYHw [17:03:05] [02miraheze/ssl] 07paladox 0332acf4f - Fix savage-wiki.com cert [17:03:17] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.85, 23.43, 21.86 [17:03:23] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [17:03:25] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:03:27] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:03:30] RECOVERY - cp9 Puppet on cp9 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:03:33] paladox: oh, what did you do? [17:03:49] copy the three certs that they provided [17:03:52] *copied [17:04:11] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:04:28] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:05:40] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:07:09] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 14.86, 21.18, 21.53 [17:09:47] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+1/-0/±1] 13https://git.io/JTYQR [17:09:49] [02miraheze/mediawiki] 07paladox 03ed1761c - Add EventStreamConfig as required by EventLogging [17:10:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JTYQz [17:10:13] [02miraheze/mw-config] 07paladox 03f5c51ee - Install EventStreamConfig [17:10:15] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3 [17:10:16] [02mw-config] 07paladox opened pull request 03#3295: Install EventStreamConfig - 13https://git.io/JTYQ2 [17:10:43] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JTYQr [17:10:44] [02miraheze/mw-config] 07paladox 03e42e9eb - Update extension-list [17:10:46] [02mw-config] 07paladox synchronize pull request 03#3295: Install EventStreamConfig - 13https://git.io/JTYQ2 [17:10:53] [02mw-config] 07paladox closed pull request 03#3295: Install EventStreamConfig - 13https://git.io/JTYQ2 [17:10:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JTYQK [17:10:56] [02miraheze/mw-config] 07paladox 03f17c363 - Install EventStreamConfig (#3295) * Install EventStreamConfig * Update extension-list [17:10:57] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [17:10:58] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3 [17:10:59] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.68, 18.50, 20.33 [17:12:26] RECOVERY - test2 MediaWiki Rendering on test2 is OK: HTTP OK: HTTP/1.1 200 OK - 18035 bytes in 2.169 second response time [17:13:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JTYQ5 [17:13:37] [02miraheze/mw-config] 07paladox 031a141fe - Update Purge to use extension registration [17:13:38] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3 [17:13:40] [02mw-config] 07paladox opened pull request 03#3296: Update Purge to use extension registration - 13https://git.io/JTYQd [17:13:57] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:14:18] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JTYQx [17:14:20] [02miraheze/mediawiki] 07paladox 0379e5481 - Update Purge [17:14:40] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JTYQp [17:14:41] [02miraheze/mw-config] 07paladox 03ef28046 - Update extension-list [17:14:43] [02mw-config] 07paladox synchronize pull request 03#3296: Update Purge to use extension registration - 13https://git.io/JTYQd [17:14:49] hi [17:15:08] PROBLEM - mw5 MediaWiki Rendering on mw5 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 284 bytes in 0.005 second response time [17:15:11] PROBLEM - wiki.hibernusmc.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.hibernusmc.net could not be found [17:15:11] hi [17:15:14] [02mw-config] 07paladox closed pull request 03#3296: Update Purge to use extension registration - 13https://git.io/JTYQd [17:15:16] PROBLEM - jobrunner2 MediaWiki Rendering on jobrunner2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 284 bytes in 0.014 second response time [17:15:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JTY7f [17:15:16] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [17:15:17] [02miraheze/mw-config] 07paladox 03763820d - Update Purge to use extension registration (#3296) * Update Purge to use extension registration * Update extension-list [17:15:22] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [17:15:26] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [17:15:36] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [17:15:37] Miraheze me esta dando un 503, he pensado que os interesaría saberlo [17:16:00] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [17:16:09] PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 284 bytes in 0.025 second response time [17:16:14] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [17:16:17] PROBLEM - mw7 MediaWiki Rendering on mw7 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 284 bytes in 0.006 second response time [17:16:22] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 284 bytes in 0.008 second response time [17:16:35] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [17:16:37] PROBLEM - mw6 MediaWiki Rendering on mw6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 284 bytes in 0.007 second response time [17:16:49] huh [17:16:51] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is WARNING: WARNING - NGINX Error Rate is 53% [17:17:36] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is WARNING: WARNING - NGINX Error Rate is 54% [17:18:47] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JTY72 [17:18:49] [02miraheze/mw-config] 07paladox 035f18a3f - Revert "Install EventStreamConfig (#3295)" This reverts commit f17c363b6f67203ab2b41d68a47d343cc358c947. [17:18:51] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 43% [17:19:08] RECOVERY - mw5 MediaWiki Rendering on mw5 is OK: HTTP OK: HTTP/1.1 200 OK - 17073 bytes in 0.084 second response time [17:19:14] RECOVERY - jobrunner2 MediaWiki Rendering on jobrunner2 is OK: HTTP OK: HTTP/1.1 200 OK - 17121 bytes in 0.163 second response time [17:19:36] RECOVERY - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is OK: OK - NGINX Error Rate is 17% [17:19:43] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JTY76 [17:19:44] [02miraheze/mediawiki] 07paladox 0322bda85 - Change EventStreamConfig to REL1_35 [17:19:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:20:12] RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 17075 bytes in 1.575 second response time [17:20:13] RECOVERY - mw7 MediaWiki Rendering on mw7 is OK: HTTP OK: HTTP/1.1 200 OK - 17074 bytes in 0.621 second response time [17:20:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:20:22] RECOVERY - jobrunner1 MediaWiki Rendering on jobrunner1 is OK: HTTP OK: HTTP/1.1 200 OK - 17121 bytes in 0.124 second response time [17:20:35] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [17:20:37] RECOVERY - mw6 MediaWiki Rendering on mw6 is OK: HTTP OK: HTTP/1.1 200 OK - 17074 bytes in 0.166 second response time [17:20:51] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 3% [17:20:51] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 1% [17:21:13] !log rebuild lc on mw* and jobrunner* [17:21:14] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [17:21:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:21:23] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [17:21:24] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [17:22:04] RECOVERY - wiki.hibernusmc.net - reverse DNS on sslhost is OK: rDNS OK - wiki.hibernusmc.net reverse DNS resolves to cp7.miraheze.org [17:24:24] PROBLEM - test2 MediaWiki Rendering on test2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.033 second response time [17:25:11] miraheze/mw-config/paladox-patch-3/ef28046 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/735796143 [17:25:27] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.92, 19.32, 18.07 [17:25:33] Thanks, Miraheze runs again [17:31:17] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.68, 22.35, 19.72 [17:33:14] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.69, 18.39, 13.89 [17:34:11] miraheze/mw-config/master/5f18a3f - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/735797160 [17:35:13] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.38, 17.61, 14.17 [17:40:57] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.52, 22.81, 21.86 [17:42:43] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JTYFY [17:42:45] [02miraheze/mw-config] 07paladox 03ee2a82e - Only install EventStreamConfig on 1.35 [17:42:46] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbvb3 [17:42:48] [02mw-config] 07paladox opened pull request 03#3297: Only install EventStreamConfig on 1.35 - 13https://git.io/JTYFO [17:42:52] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.49, 25.05, 22.82 [17:43:18] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JTYFW [17:43:20] [02miraheze/mw-config] 07paladox 03b1ace95 - Update extension-list [17:43:21] [02mw-config] 07paladox synchronize pull request 03#3297: Only install EventStreamConfig on 1.35 - 13https://git.io/JTYFO [17:43:46] [02mw-config] 07paladox closed pull request 03#3297: Only install EventStreamConfig on 1.35 - 13https://git.io/JTYFO [17:43:47] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JTYFR [17:43:49] [02miraheze/mw-config] 07paladox 033fd71a1 - Only install EventStreamConfig on 1.35 (#3297) * Only install EventStreamConfig on 1.35 * Update extension-list [17:43:51] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbvb3 [17:43:52] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [17:44:50] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.94, 22.43, 22.08 [17:46:24] RECOVERY - test2 MediaWiki Rendering on test2 is OK: HTTP OK: HTTP/1.1 200 OK - 18761 bytes in 0.252 second response time [17:52:51] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.35, 21.17, 21.35 [17:54:49] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.44, 18.47, 20.37 [18:28:05] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JTYxN [18:28:06] [02miraheze/mediawiki] 07paladox 0346e383c - Update ArticleRatings to 722fc89a1632b3e371f17264d2a49b19f45cb64f [18:34:36] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 99s [18:40:49] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.66, 21.90, 18.45 [19:00:59] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 10.29, 11.24, 6.75 [19:03:16] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.94, 8.00, 6.18 [19:05:49] PROBLEM - wiki.exnihilolinux.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.exnihilolinux.org could not be found [19:05:52] PROBLEM - www.wikiescola.com.br - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.wikiescola.com.br could not be found [19:05:54] PROBLEM - wiki.rebirthofthenight.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.rebirthofthenight.com could not be found [19:06:00] PROBLEM - history.sdtef.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for history.sdtef.org could not be found [19:06:03] PROBLEM - dc.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dc.miraheze.org could not be found [19:06:03] PROBLEM - wiki.make717.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.make717.org could not be found [19:07:07] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 3.39, 6.26, 5.95 [19:12:32] RECOVERY - wiki.exnihilolinux.org - reverse DNS on sslhost is OK: rDNS OK - wiki.exnihilolinux.org reverse DNS resolves to cp6.miraheze.org [19:12:38] RECOVERY - wiki.rebirthofthenight.com - reverse DNS on sslhost is OK: rDNS OK - wiki.rebirthofthenight.com reverse DNS resolves to cp7.miraheze.org [19:12:38] RECOVERY - www.wikiescola.com.br - reverse DNS on sslhost is OK: rDNS OK - www.wikiescola.com.br reverse DNS resolves to cp6.miraheze.org [19:12:53] RECOVERY - history.sdtef.org - reverse DNS on sslhost is OK: rDNS OK - history.sdtef.org reverse DNS resolves to cp6.miraheze.org [19:12:57] RECOVERY - dc.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - dc.miraheze.org reverse DNS resolves to cp6.miraheze.org [19:12:58] RECOVERY - wiki.make717.org - reverse DNS on sslhost is OK: rDNS OK - wiki.make717.org reverse DNS resolves to cp7.miraheze.org [19:52:36] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [20:01:21] no access in Miraheze (MetaWiki) [20:04:49] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 10.19, 16.76, 23.14 [20:14:51] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.10, 17.63, 20.38 [20:26:36] Why? [20:48:34] It was strange, I couldn't access Miraheze but I could access Wikipedia- Now it's normal [20:50:45] Ok [21:03:56] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.82, 20.93, 17.82 [21:05:51] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.28, 17.40, 16.90 [21:06:12] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JTOYq [21:06:13] [02miraheze/mediawiki] 07paladox 03fb94201 - Update ProtectSite to fc8cf0527d0493a79a3c3e83e9398a83f25ec607 [21:10:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTOYE [21:10:15] [02miraheze/mw-config] 07paladox 037ade98c - Fix loading Truglass in i18n [21:11:42] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JTOY6 [21:11:43] [02miraheze/mw-config] 07paladox 03548762b - Migrate CirrusSearch to extension registration [21:11:45] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbvb3 [21:11:46] [02mw-config] 07paladox opened pull request 03#3298: Migrate CirrusSearch to extension registration - 13https://git.io/JTOYi [21:12:15] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTOYX [21:12:16] [02miraheze/mw-config] 07paladox 03666cd7a - Update extension-list [21:12:44] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JTOYD [21:12:45] [02miraheze/mw-config] 07paladox 0390f49c5 - Update LocalExtensions.php [21:12:47] [02mw-config] 07paladox synchronize pull request 03#3298: Migrate CirrusSearch to extension registration - 13https://git.io/JTOYi [21:12:55] [02mw-config] 07paladox closed pull request 03#3298: Migrate CirrusSearch to extension registration - 13https://git.io/JTOYi [21:12:56] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JTOYy [21:12:58] [02miraheze/mw-config] 07paladox 031603a15 - Migrate CirrusSearch to extension registration (#3298) * Migrate CirrusSearch to extension registration * Update LocalExtensions.php [21:12:59] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbvb3 [21:13:01] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [21:14:38] miraheze/mw-config/paladox-patch-4/90f49c5 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/735856297 [21:29:34] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JTO3I [21:29:36] [02miraheze/mediawiki] 07paladox 0334c71f2 - Update NumberedHeadings to 39b189ca60f6736ab3cacc8aad8ccc77a3dc7a3e [21:34:03] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JTO3V [21:34:05] [02miraheze/mediawiki] 07paladox 0360379af - Update NumberedHeadings to ada69e7fd80bc01bcfd518588559a4ff23f51834 [22:01:03] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JTOGd [22:01:04] [02miraheze/mw-config] 07paladox 03bbdbd62 - Change wgResourceLoaderMaxage for test2 [22:01:06] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbvb3 [22:01:11] [02mw-config] 07paladox opened pull request 03#3299: Change wgResourceLoaderMaxage for test2 - 13https://git.io/JTOGF [22:02:46] miraheze/mw-config/paladox-patch-4/bbdbd62 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/735866373 [22:06:27] [02mw-config] 07paladox closed pull request 03#3299: Change wgResourceLoaderMaxage for test2 - 13https://git.io/JTOGF [22:06:28] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTOZs [22:06:30] [02miraheze/mw-config] 07paladox 03365aae2 - Change wgResourceLoaderMaxage for test2 (#3299) [22:06:31] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbvb3 [22:06:33] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [22:11:56] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JTOZK [22:11:58] [02miraheze/mediawiki] 07paladox 0347f28f9 - Add CookieWarning to merge-plugin [22:12:19] [02mediawiki] 07dependabot-preview[bot] edited pull request 03#353: Bump extensions/WikiForum from `5d65ed7` to `9c69eaa` - 13https://git.io/JTk1I [22:13:11] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JTOZD [22:13:13] [02miraheze/mediawiki] 07paladox 03b1fbcd4 - Add CookieWarning to merge-plugin [22:13:45] [02mediawiki] 07dependabot-preview[bot] edited pull request 03#353: Bump extensions/WikiForum from `5d65ed7` to `9c69eaa` - 13https://git.io/JTk1I [22:13:53] [02miraheze/mediawiki] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/WikiForum-8b9fadf [+0/-0/±1] 13https://git.io/JTOZH [22:13:54] [02miraheze/mediawiki] 07dependabot-preview[bot] 03925de8b - Bump extensions/WikiForum from `ee0c1b2` to `8b9fadf` Bumps [extensions/WikiForum](https://github.com/wikimedia/mediawiki-extensions-WikiForum) from `ee0c1b2` to `8b9fadf`. - [Release notes](https://github.com/wikimedia/mediawiki-extensions-WikiForum/releases) - [Commits](https://github.com/wikimedia/mediawiki-extensions- [22:13:54] WikiForum/compare/ee0c1b2839df552ca9edea213ed42018cd7d453b...8b9fadf98e89cb1ab4505255a80b39fc5c66692e) Signed-off-by: dependabot-preview[bot] [22:13:55] [ GitHub - wikimedia/mediawiki-extensions-WikiForum: Github mirror of MediaWiki extension WikiForum - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access fo ] - github.com [22:13:55] [ Releases · wikimedia/mediawiki-extensions-WikiForum · GitHub ] - github.com [22:13:56] [ Page not found · GitHub · GitHub ] - github.com [22:13:56] [02mediawiki] 07dependabot-preview[bot] opened pull request 03#358: Bump extensions/WikiForum from `ee0c1b2` to `8b9fadf` - 13https://git.io/JTOZQ [22:13:57] [02mediawiki] 07dependabot-preview[bot] created branch 03dependabot/submodules/REL1_35/extensions/WikiForum-8b9fadf - 13https://git.io/vbL5b [22:13:59] [02mediawiki] 07dependabot-preview[bot] labeled pull request 03#358: Bump extensions/WikiForum from `ee0c1b2` to `8b9fadf` - 13https://git.io/JTOZQ [22:14:00] [02mediawiki] 07dependabot-preview[bot] commented on pull request 03#353: Bump extensions/WikiForum from `5d65ed7` to `9c69eaa` - 13https://git.io/JTOZ5 [22:14:02] [02mediawiki] 07dependabot-preview[bot] closed pull request 03#353: Bump extensions/WikiForum from `5d65ed7` to `9c69eaa` - 13https://git.io/JTk1I [22:14:03] [02miraheze/mediawiki] 07dependabot-preview[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/WikiForum-9c69eaa [22:14:05] [02mediawiki] 07dependabot-preview[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/WikiForum-9c69eaa - 13https://git.io/vbL5b [22:19:59] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:20:24] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:20:37] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:20:52] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:21:06] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:21:14] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:22:38] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [22:23:44] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.58, 20.62, 16.84 [22:25:26] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JTOno [22:25:28] [02miraheze/mediawiki] 07paladox 0372876ba - Revert "Add CookieWarning to merge-plugin" This reverts commit b1fbcd464f8969bfc64beb9a29f481dc53e4737c. [22:25:39] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.19, 18.08, 16.32 [22:32:29] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [22:33:12] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:33:57] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:36:52] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:37:10] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:56:07] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JTOCc [22:56:09] [02miraheze/mediawiki] 07paladox 031996790 - Update CookieWarning to bac2e0e9f9dcaa7149b746f4c8bb8450f3bafd92