[00:41:23] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 344.37 ms
[00:43:27] <icinga-miraheze>	 RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 249.26 ms
[01:08:21] <icinga-miraheze>	 PROBLEM - cp12 Varnish Backends on cp12 is WARNING: No backends detected. If this is an error, see readme.txt
[01:08:47] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb
[01:09:39] <icinga-miraheze>	 PROBLEM - cp12 HTTPS on cp12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.366 second response time
[01:09:55] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb
[01:10:08] <icinga-miraheze>	 PROBLEM - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is CRITICAL: CRITICAL - NGINX Error Rate is 99%
[01:16:58] <Universal_Omega>	 Oh, Zppix I forgot to clear your staffwiki notifications. Sorry about that. I will for sure do it first thing tomorrow. 
[01:17:21] <Zppix>	 Universal_Omega:  Yeah i even tried to do it via API and still didnt do it
[01:21:38] <icinga-miraheze>	 RECOVERY - cp12 HTTPS on cp12 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2196 bytes in 0.398 second response time
[01:21:42] <paladox>	 !log restart varnish on cp12 as it appears to have crashed
[01:21:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:21:55] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[01:22:08] <icinga-miraheze>	 RECOVERY - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is OK: OK - NGINX Error Rate is 4%
[01:22:22] <icinga-miraheze>	 RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy
[01:22:47] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[06:34:12] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 349.28 ms
[06:48:40] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 350.08 ms
[06:52:49] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 347.04 ms
[07:07:18] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 352.20 ms
[07:09:21] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 348.67 ms
[07:36:11] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 350.53 ms
[07:38:15] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 346.23 ms
[07:39:43] <Zppix>	 i dont see any user facing issues
[07:46:30] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 350.53 ms
[07:48:33] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 347.90 ms
[08:07:08] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 350.56 ms
[08:09:12] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 349.98 ms
[08:27:49] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 351.58 ms
[08:29:53] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 345.12 ms
[08:37:52] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.33, 3.79, 2.53
[08:38:08] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 350.66 ms
[08:39:50] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.57, 2.70, 2.28
[08:40:12] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 346.75 ms
[08:58:35] <Zppix>	 Reception123:  what is up with cp*
[09:00:11] <Reception123>	 I have no idea
[09:01:13] <Zppix>	 i dont see anything user-side, could it be under heavy load or something Reception123 
[09:06:13] <Reception123>	 Unfortunately, I don't currently have access to look :(
[09:07:13] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 352.65 ms
[09:09:16] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 346.00 ms
[09:21:41] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 352.10 ms
[09:23:45] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 349.64 ms
[09:25:49] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 355.18 ms
[09:27:52] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 346.38 ms
[09:36:08] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 353.13 ms
[09:38:11] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 346.74 ms
[09:42:19] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 351.36 ms
[09:44:22] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 346.38 ms
[09:53:27] <icinga-miraheze>	 PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 9.44, 4.92, 2.74
[09:55:21] <icinga-miraheze>	 RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 1.51, 3.40, 2.43
[10:57:00] <JohnLewis>	 for cp3, it's kind of expected
[10:57:36] <JohnLewis>	 latency between the UK and Singapore isn't going to be good
[11:47:47] <Reception123>	 That does, but why the sudden Icinga warnings? 
[12:21:18] <icinga-miraheze>	 RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 254.10 ms
[12:43:28] <Not-71a9>	 [02miraheze/CreateWiki] 07JohnFLewis pushed 033 commits to 03master [+0/-0/±8] 13https://git.io/Jt5xB
[12:43:29] <Not-71a9>	 [02miraheze/CreateWiki] 07JohnFLewis 03a749c40 - Change catch type to MWException, TypeError no longer thrown
[12:43:31] <Not-71a9>	 [02miraheze/CreateWiki] 07JohnFLewis 03c9dd807 - T6788: Changes to both creator and requester workflows
[12:43:32] <Not-71a9>	 [02miraheze/CreateWiki] 07JohnFLewis 03f15f9eb - Merge remote-tracking branch 'origin/master'
[12:44:28] <miraheze-github>	 miraheze/CreateWiki - JohnFLewis the build passed.
[13:14:52] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 2.73, 3.85, 2.15
[13:16:51] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.67, 2.73, 1.95
[13:17:27] <Not-71a9>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/Jt5jm
[13:17:29] <Not-71a9>	 [02miraheze/mediawiki] 07paladox 03082f4a3 - Update WikiSEO
[13:20:31] <Not-71a9>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jt5jC
[13:20:33] <Not-71a9>	 [02miraheze/mw-config] 07paladox 03dd2d3d1 - Add wgNaverSiteVerificationKey
[13:20:34] <Not-71a9>	 [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3
[13:20:36] <Not-71a9>	 [02mw-config] 07paladox opened pull request 03#3738: Add wgNaverSiteVerificationKey - 13https://git.io/Jt5jW
[13:21:24] <Not-71a9>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jt5j0
[13:21:26] <Not-71a9>	 [02miraheze/mw-config] 07paladox 035e85fc4 - Update ManageWikiSettings.php
[13:21:27] <Not-71a9>	 [02mw-config] 07paladox synchronize pull request 03#3738: Add wgNaverSiteVerificationKey - 13https://git.io/Jt5jW
[13:21:42] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[13:22:01] <Not-71a9>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jt5ju
[13:22:03] <Not-71a9>	 [02miraheze/mw-config] 07paladox 03f93bb29 - Update ManageWikiSettings.php
[13:22:04] <Not-71a9>	 [02mw-config] 07paladox synchronize pull request 03#3738: Add wgNaverSiteVerificationKey - 13https://git.io/Jt5jW
[13:22:32] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[13:23:17] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[13:23:36] <Not-71a9>	 [02mw-config] 07paladox closed pull request 03#3738: Add wgNaverSiteVerificationKey - 13https://git.io/Jt5jW
[13:23:38] <Not-71a9>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jt5jK
[13:23:39] <Not-71a9>	 [02miraheze/mw-config] 07paladox 034f1be35 - Add wgNaverSiteVerificationKey (#3738)
[13:23:41] <Not-71a9>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2
[13:23:42] <Not-71a9>	 [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3
[13:24:42] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[13:37:07] <Not-71a9>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/Jtdek
[13:37:09] <Not-71a9>	 [02miraheze/mediawiki] 07paladox 03ac77593 - Update WikiSEO
[13:38:13] <Not-71a9>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtdeL
[13:38:15] <Not-71a9>	 [02miraheze/mw-config] 07paladox 039b94479 - Fix default value for site verification keys
[13:39:16] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[13:54:19] <MH-Discord>	 <R4356th> paladox: there should be a config in LocalWiki.php for a wiki which requested that feature (Naver site verification key). You might want to remove that and configure it on-wiki.
[13:54:38] <paladox>	 Its up to stewards to do that
[13:55:01] <MH-Discord>	 <R4356th> Huh? I have noticed Sysadmins doing this.
[14:06:29] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 280.39 ms
[14:08:32] <icinga-miraheze>	 RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 281.08 ms
[14:19:01] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.13, 2.69
[14:23:00] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.13, 3.19, 2.83
[14:28:05] <JohnLewis>	 Sysadmins should only be doing it if they’re moving it from LS to MW, unless I have misunderstood the problem here
[14:33:02] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.53, 3.15
[14:37:01] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.71, 3.39, 3.21
[14:41:02] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.51, 3.29
[14:45:00] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.21, 3.86, 3.47
[14:51:00] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.67, 3.54
[14:59:00] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.01, 3.21, 3.37
[15:03:01] <paladox>	 JohnLewis it was a new config, so the request is to set it which is for Stewards
[15:03:05] <paladox>	 to do
[15:26:10] <dmehus>	 > <R4356th> paladox: there should be a config in LocalWiki.php for a wiki which requested that feature (Naver site verification key). You might want to remove that and configure it on-wiki.
[15:26:10] <dmehus>	 Are we adding this to ManageWiki, and will it use the `managewiki` or `managewiki-restricted` user right? Either way, I do support this moving to on-wiki if that's what we're doing
[15:26:48] <RhinosF1>	 If I read the logs right then no reason to restrict
[15:26:48] <MirahezeBot>	 RhinosF1: 2021-02-24 - 00:28:53GMT <dmehus> tell RhinosF1 reminder to test yesterday—whatever that means
[15:26:48] <MirahezeBot>	 RhinosF1: 2021-02-24 - 00:45:35GMT <dmehus> tell RhinosF1 ack
[15:26:56] <RhinosF1>	 And hi dmehus
[15:27:15] <dmehus>	 RhinosF1, yeah, I agree, I don't think the Yandex site verification was restricted
[15:27:32] <RhinosF1>	 dmehus: I can't see why you'd need to
[15:27:46] <dmehus>	 yeah, same
[15:27:54] <dmehus>	 so stewards would only do when there's no active bureaucrats and the local community has requested it
[15:28:35] <dmehus>	 and hi RhinosF1 :)
[15:29:43] <Not-71a9>	 [02mw-config] 07Universal-Omega reviewed pull request 03#3738 commit - 13https://git.io/Jtdt6
[15:29:44] <Not-71a9>	 [02mw-config] 07Universal-Omega reviewed pull request 03#3738 commit - 13https://git.io/Jtdti
[15:36:41] <MH-Discord>	 <R4356th> The original request is in https://phabricator.miraheze.org/T6764.
[15:36:42] <MirahezeBot>	 [ ⚓ T6764 I want to add meta to the head of html ] - phabricator.miraheze.org
[15:38:24] <dmehus>	 R4356th, ack
[15:39:53] <RhinosF1>	 That's an option in WikiSEO I think
[15:40:24] <MH-Discord>	 <R4356th> Yes, I used a hook instead of any extension.
[15:45:59] <Not-71a9>	 [02mw-config] 07paladox reviewed pull request 03#3738 commit - 13https://git.io/Jtdqb
[15:48:14] <Not-71a9>	 [02mw-config] 07Universal-Omega reviewed pull request 03#3738 commit - 13https://git.io/JtdmU
[15:53:52] <Universal_Omega>	 `8:03 AM 25B100<+ paladox> JohnLewis it was a new config, so the request is to set it which is for Stewards` no? It is an old config in LocalWiki.php via a hook.
[15:54:03] <Universal_Omega>	 Whoops. Bad formatting. 
[15:54:16] <paladox>	 i was told it was a config request
[15:55:17] <Universal_Omega>	 paladox: No, it's already there. They were saying it's an old config request and was already added to LW via a hook.
[15:55:47] <paladox>	 oh, i don't see wgNaverSiteVerificationKey set elsewhere
[15:55:55] <paladox>	 as in it only exists where i set it
[15:56:23] <Universal_Omega>	 paladox: it isn't set as that, it's set as a hook to add a meta tag in LW.
[15:56:37] <paladox>	 oh
[15:56:49] <paladox>	 could you move it to the config please? :)
[16:03:57] <Universal_Omega>	 paladox: https://github.com/miraheze/mw-config/blob/9b94479780e5f87fca80496b806b6e097c908bd0/LocalWiki.php#L267-L273 but yes, I can in about an hour. 
[16:03:57] <MirahezeBot>	 [ mw-config/LocalWiki.php at 9b94479780e5f87fca80496b806b6e097c908bd0 · miraheze/mw-config · GitHub ] - github.com
[16:04:17] <paladox>	 thanks!
[16:06:41] <Not-71a9>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JtdY7
[16:06:42] <Not-71a9>	 [02miraheze/ssl] 07Reception123 0380947cf - add archive.a2b2.org cert
[16:07:21] <Universal_Omega>	 No problem
[17:22:23] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.71, 3.35
[17:33:49] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.82, 3.22, 3.30
[18:20:52] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 344.13 ms
[18:22:52] <icinga-miraheze>	 RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 250.06 ms
[18:23:40] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 1.37, 4.36, 3.25
[18:25:36] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.91, 3.49, 3.06
[18:26:13] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.54, 3.22
[18:27:33] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.75, 2.56, 2.77
[18:28:07] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.45, 3.23
[18:30:01] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.31, 3.35, 3.22
[18:39:36] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.52, 3.35
[18:41:31] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.51, 3.18, 3.25
[18:47:14] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 2.38, 5.08, 3.78
[18:49:14] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.53, 3.49, 3.35
[18:51:13] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 21.62, 10.58, 5.91
[18:52:17] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.69, 3.45
[18:53:27] <Not-71a9>	 [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jtd06
[18:53:28] <Not-71a9>	 [02miraheze/CreateWiki] 07Universal-Omega 03f1afb31 - Fix grammar
[18:53:30] <Not-71a9>	 [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL
[18:53:31] <Not-71a9>	 [02CreateWiki] 07Universal-Omega opened pull request 03#189: Fix grammar - 13https://git.io/Jtd0i
[18:54:12] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.68, 3.38, 3.37
[18:54:30] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[18:59:11] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.00, 2.71, 3.79
[19:03:10] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.07, 2.97, 3.58
[19:05:11] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.95, 2.20, 3.22
[19:06:15] <Universal_Omega>	 dmehus: ^ look good to you?
[19:17:45] <Reception123>	 !log added Sario to bots@
[19:17:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:22:23] <dmehus>	 Universal_Omega, which?
[19:23:01] <dmehus>	 Reception123, what's that log entry for? Don't we want to log that in MirahezeBotsWiki:Server admin log?
[19:23:31] <Reception123>	 dmehus: for email
[19:23:47] <Universal_Omega>	 dmehus: https://git.io/Jtd0i 
[19:23:48] <MirahezeBot>	 [ Fix grammar by Universal-Omega · Pull Request #189 · miraheze/CreateWiki · GitHub ] - git.io
[19:23:49] <dmehus>	 Reception123, ah, right, makes sense. Thanks! :)
[19:24:30] <dmehus>	 Universal_Omega, yes, your grammatical changes are an improvement
[19:24:51] <Not-71a9>	 [02CreateWiki] 07Universal-Omega closed pull request 03#189: Fix grammar - 13https://git.io/Jtd0i
[19:24:52] <Not-71a9>	 [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtdEA
[19:24:54] <Not-71a9>	 [02miraheze/CreateWiki] 07Universal-Omega 030ba79a8 - Fix grammar (#189)
[19:24:55] <Not-71a9>	 [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL
[19:24:57] <Not-71a9>	 [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1
[19:25:35] <Universal_Omega>	 dmehus: merged. Will update CW soon.
[19:25:42] <dmehus>	 Universal_Omega, can you change `subject` to `subject(s)` actually?
[19:25:53] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[19:25:55] <dmehus>	 That would be ideal, in case the request has more than one subject
[19:26:18] <dmehus>	 and also change `of` to `about`?
[19:26:28] <Universal_Omega>	 Oh. Did I change that? Sorry. Fixing 
[19:26:36] <dmehus>	 No problem.
[19:26:45] <dmehus>	 That's fine. Just one more small commit to push :)
[19:27:11] <Universal_Omega>	 Yeah.
[19:28:29] <Universal_Omega>	 dmehus: `Will this wiki's main subjects be about living people?` look right now? Or should the word `main` be dropped?
[19:31:52] <Not-71a9>	 [02CreateWiki] 07dmehus opened pull request 03#190: Further grammatical and clarity fixes - 13https://git.io/Jtdus
[19:31:56] <icinga-miraheze>	 PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (4 critical updates).
[19:32:52] <miraheze-github>	 miraheze/CreateWiki - dmehus the build passed.
[19:33:39] <dmehus>	 Universal_Omega, either way. We might conflict on our changes, so merge yours first then mine.
[19:34:09] <dmehus>	 RhinosF1, ^
[19:34:38] <icinga-miraheze>	 PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (6 critical updates).
[19:34:42] <RhinosF1>	 Yes that's MacFan
[19:35:12] <icinga-miraheze>	 PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (6 critical updates).
[19:35:35] <icinga-miraheze>	 PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (6 critical updates).
[19:35:51] <icinga-miraheze>	 PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 36 packages available for upgrade (6 critical updates).
[19:36:03] <dmehus>	 RhinosF1, ack
[19:36:07] <icinga-miraheze>	 PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (6 critical updates).
[19:36:11] <icinga-miraheze>	 PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 38 packages available for upgrade (6 critical updates).
[19:36:52] <icinga-miraheze>	 PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (6 critical updates).
[19:37:35] <Universal_Omega>	 dmehus, what about: `Will this wiki contain information about living people?`?
[19:40:53] <icinga-miraheze>	 PROBLEM - services4 APT on services4 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (4 critical updates).
[19:43:14] <icinga-miraheze>	 PROBLEM - services3 APT on services3 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (4 critical updates).
[19:46:54] <Zppix>	 Universal_Omega:  dont forget to purge my staffwiki notifications
[19:47:55] <Universal_Omega>	 Zppix: yep, I'm looking into a permanent solution to prevent staff notifications globally, but once that's done I'll purge yours and it shouldn't happen again.
[19:48:31] <Zppix>	 it should happen for me period after that
[19:48:39] <Zppix>	 shouldnt*
[19:53:31] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.39, 3.08
[19:55:30] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.03, 3.26, 3.07
[19:55:47] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out
[19:57:45] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Databases db11 on bacula2 is WARNING: WARNING: Full, 541500 files, 40.45GB, 2021-02-07 01:25:00 (2.5 weeks ago)
[20:03:07] <dmehus>	 > what about: `Will this wiki contain information about living people?`?
[20:03:07] <dmehus>	 Universal_Omega, that seems fine
[20:04:31] <JohnLewis>	 My intention behind the wording was to make it about primary usage
[20:04:36] <dmehus>	 Universal_Omega, perhaps we could just disable the Echo notifications on `staffwiki` generally? I can't see as there is a massive need for them, as sysadmins should be reviewing it, presumably, in relation to updates discussed in private IRC channels or via e-mails
[20:04:54] <dmehus>	 JohnLewis, ah, yeah
[20:05:07] <JohnLewis>	 Arguably a wiki about gaming, would have information about people, that doesn't mean it's a wiki about living people that would require manual review and manual creation
[20:05:32] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.50, 3.27
[20:05:43] <dmehus>	 JohnLewis, true, in most cases, unless it's a "Gaming" wiki about gamers, but in most cases, yes
[20:06:06] <JohnLewis>	 Well if it's a wiki about gamers, it's about people ;)
[20:06:20] <dmehus>	 yeah, I just mean the "Gaming" category
[20:06:22] <dmehus>	 but yeah
[20:07:02] <Not-71a9>	 [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jtdzu
[20:07:04] <Not-71a9>	 [02miraheze/CreateWiki] 07Universal-Omega 03a85e7be - Fix wording
[20:07:05] <Not-71a9>	 [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL
[20:07:07] <Not-71a9>	 [02CreateWiki] 07Universal-Omega opened pull request 03#191: Fix wording - 13https://git.io/Jtdzz
[20:07:16] <dmehus>	 A wiki about some Roblox game mod containing cheat codes, installation instructions, and other details about plot or gaming elements likely wouldn't
[20:07:32] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.46, 3.17, 3.18
[20:08:13] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[20:10:55] <Not-71a9>	 [02CreateWiki] 07Universal-Omega closed pull request 03#191: Fix wording - 13https://git.io/Jtdzz
[20:10:57] <Not-71a9>	 [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jtdzr
[20:10:58] <Not-71a9>	 [02miraheze/CreateWiki] 07Universal-Omega 03c279be4 - Fix wording (#191)
[20:11:00] <Not-71a9>	 [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1
[20:11:02] <Not-71a9>	 [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL
[20:11:33] <Universal_Omega>	 JohnLewis: fixed wording while maintaining the original meaning of it. 
[20:11:57] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[20:19:03] <Not-71a9>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#190: Further grammatical and clarity fixes - 13https://git.io/Jtdus
[20:20:11] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[20:22:48] <Universal_Omega>	 Reception123: if you have time, do you mind helping Zppix clear his staffwiki notifications? Just asking you since you did it last time for MacFan4000 (I think). I'm not entirely sure how to use the query that you used, from SAL/2020. If you don't have time, I can figure it out.
[20:23:20] <Reception123>	 I can do it yeah
[20:23:33] <Zppix>	 its no rush :) just hate red little bell
[20:24:27] <MH-Discord>	 <Cheems> Hi
[20:24:55] <Universal_Omega>	 Reception123: Thanks! Zppix: yeah, I understand that.
[20:25:36] <Zppix>	 Reception123:  if you need anything from my end to clear it lmk
[20:25:56] <Zppix>	 im around im trying to help WMF sre out with an automatic gerrit commit system thingy
[20:26:07] <Zppix>	 by first trying to decode their conftool
[20:26:56] <Reception123>	 !log MariaDB [metawiki]> DELETE FROM echo_unread_wikis WHERE euw_user = '9736' AND euw_id = '2768838';
[20:26:59] <Reception123>	 Zppix: check now
[20:27:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:27:20] * Zppix looks
[20:27:41] <Zppix>	 Reception123:  no more staffwiki notifications ty
[20:27:53] <Reception123>	 great, no problem
[20:28:01] <Universal_Omega>	 Thanks for doing that, Reception123.
[20:28:08] <dmehus>	 Reception123, is 9736 Zppix' local wiki userID, and what's the `2768838` ID?
[20:28:29] <Reception123>	 dmehus: it's the notification ID
[20:28:31] <Zppix>	 dmehus:  no those ids are specific to echo IIRC
[20:28:39] <dmehus>	 oh
[20:28:59] <Zppix>	 it would be a bad thing to disclose a userid in public IIRC
[20:29:14] <dmehus>	 yeah
[20:29:36] <Zppix>	 i mean i still have 2fa and stuff so to access any of mh accounts you would need my phone
[20:29:38] <dmehus>	 so it's a specific Echo notifications user ID, not the actual wiki user ID
[20:29:45] <Zppix>	 i believe
[20:29:47] <Zppix>	 I could be wrong
[20:29:48] <dmehus>	 ah
[20:30:01] <Zppix>	 my echo knowledge isnt what it used to be
[20:30:09] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.32, 3.14
[20:30:28] <Reception123>	 It's the echo notification ID yes
[20:30:32] * Zppix is also currently banging my hand for volunteering to try to figure out a way for wm to do automatic commits
[20:30:38] <Zppix>	 head*
[20:30:39] <Reception123>	 but I'm not really sure why disclosing a user ID would be bad?
[20:31:16] <Zppix>	 Reception123:  yeah i am thinking token 
[20:31:22] * Zppix has been awake for 24h
[20:32:23] <Universal_Omega>	 @JohnLewis/paladox: thoughts/objections on implementing https://phabricator.miraheze.org/T6881 ?
[20:32:24] <MirahezeBot>	 [ ⚓ T6881 Let each wiki control their robots.txt via MediaWiki:Robots.txt ] - phabricator.miraheze.org
[20:32:52] <dmehus>	 > but I'm not really sure why disclosing a user ID would be bad?
[20:32:52] <dmehus>	 Reception123, yeah I wondered that as well, tbh. It's just a userID. You can't view the page, so it doesn't really mean anything
[20:33:18] <Universal_Omega>	 He meant user token dmehus
[20:33:57] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 3.01, 3.05
[20:34:20] <Reception123>	 oh yeah, token is a different story 
[20:34:32] <paladox>	 Universal_Omega no objections from me.
[20:34:33] <dmehus>	 Universal_Omega, oh, I mean his User:Zppix page ID, so must've been a misunderstanding
[20:34:36] <Reception123>	 ID would be harmless, especially since one could even guess an ID by simply looking at the user account creation log
[20:34:48] <Reception123>	 for example John or SPF would have ID 1 on Meta
[20:35:07] <Universal_Omega>	 Also the API can get that, I think.
[20:35:16] <dmehus>	 Reception123, and the Special:Log/newusers isn't even visible to non-members, so yeah, pretty harmless
[20:35:19] <Reception123>	 yeah, tokens aren't involved in the Echo deletion SQL anyway
[20:35:37] <dmehus>	 Universal_Omega, API results can be returned on private wikis? That shouldn't be possible
[20:36:06] <Universal_Omega>	 Yes they can, but only from the private wikis and if you have access, I think. dmehus
[20:36:41] <dmehus>	 Universal_Omega, ah okay, right, makes sense. But not search engine robots, since `*` wouldn't have access
[20:37:22] <SPF|Cloud>	 hi
[20:37:35] <Universal_Omega>	 Hi SPF|Cloud!
[20:37:35] <dmehus>	 hi, SPF|Cloud
[20:38:37] <SPF|Cloud>	 hey all
[20:39:46] <paladox>	 SPF|Cloud hi
[20:44:52] <Reception123>	 Universal_Omega: for future occasions - https://meta.miraheze.org/wiki/Tech:Deleting_Echo_notifications :)
[20:44:53] <MirahezeBot>	 [ Tech:Deleting Echo notifications - Miraheze Meta ] - meta.miraheze.org
[20:45:49] <RhinosF1>	 Hi SPF|Cloud
[20:46:02] <Universal_Omega>	 Reception123: thanks 
[20:47:29] <Reception123>	 no problem
[20:57:10] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.71, 3.37, 2.23
[20:59:09] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.06, 2.52, 2.06
[21:07:34] <paladox>	 !log restart graylog-server as appeared stuck on logging in?
[21:07:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:21:42] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.19, 2.91
[21:25:31] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.19, 3.30, 3.04
[21:27:08] <paladox>	 SPF|Cloud moving our conversation here regarding logging.
[21:27:44] <paladox>	 SPF has brought up that there is some license changes in elasticsearch/mongodb.
[21:28:01] <SPF|Cloud>	 and graylog
[21:28:40] <paladox>	 He's brought up some suggestions like dropping graylog and only remotely logging to files, or moving to ELK (ES/Logstash/Kibana) or even staying with graylog.
[21:29:14] <SPF|Cloud>	 not moving to ELK, since ELK is subject to the same change
[21:29:25] <paladox>	 and that the SRE should come up with a vision for  about future logging here
[21:29:56] <paladox>	 (Also he brought up grafanas loki)
[21:30:16] <paladox>	 JohnLewis Reception123 you may be interested in this convo ^
[21:30:26] <SPF|Cloud>	 the choice is either staying with graylog (regardless of license), migrating to a different tool (only grafana looki seems somewhat viable here), migrating to pure log files on remote servers or dropping central logging as a whole
[21:33:31] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.53, 3.24
[21:35:31] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.00, 3.19, 3.15
[21:39:55] <dmehus>	 Reception123, ty for creating that instructional page :)
[21:44:10] <MH-Discord>	 <DarkMatterMan4500> Isn't Reception123 asleep right now?
[21:48:35] <dmehus>	 He will be soon if he isn't already, and quite possibly may be.
[21:48:48] <dmehus>	 It's nearly 11 pm there.
[22:03:09] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 6.85, 5.24, 2.75
[22:04:26] <SPF|Cloud>	 paladox: https://phabricator.miraheze.org/P387
[22:04:27] <MirahezeBot>	 [ ✎ P387 syslog-ng log to local file ] - phabricator.miraheze.org
[22:04:40] <SPF|Cloud>	 err, wait a minute :)
[22:05:10] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.45, 3.70, 2.48
[22:05:29] <paladox>	 i cannot log into graylog
[22:05:34] <paladox>	 its just stuck loading
[22:06:12] <SPF|Cloud>	 paste fixed now
[22:06:39] <SPF|Cloud>	 cookies?
[22:07:08] <paladox>	 i don't seem to have any cookies at least for graylog.miraheze.org
[22:07:10] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.64, 2.68, 2.26
[22:07:42] <paladox>	 ok, fixed
[22:15:31] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.43, 3.17
[22:19:31] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.00, 3.19, 3.13
[22:31:01] <paladox>	 SPF|Cloud also we'll need to fix mediawiki log so the timestamp is appended to the log timestamp
[22:31:14] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.59, 3.35
[22:31:21] <paladox>	 otherwise it just uses the time the log is generated at and not when it was generated on the machine.
[22:31:33] <icinga-miraheze>	 PROBLEM - jobrunner4 Current Load on jobrunner4 is WARNING: WARNING - load average: 5.75, 4.55, 3.14
[22:31:47] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out
[22:31:48] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Databases db13 on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out
[22:33:02] <icinga-miraheze>	 PROBLEM - jobrunner3 Current Load on jobrunner3 is CRITICAL: CRITICAL - load average: 6.08, 4.70, 3.26
[22:33:46] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Databases db11 on bacula2 is WARNING: WARNING: Full, 541500 files, 40.45GB, 2021-02-07 01:25:00 (2.6 weeks ago)
[22:33:48] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Databases db13 on bacula2 is WARNING: WARNING: Full, 207255 files, 80.26GB, 2021-02-07 04:21:00 (2.5 weeks ago)
[22:35:04] <icinga-miraheze>	 RECOVERY - jobrunner3 Current Load on jobrunner3 is OK: OK - load average: 4.75, 4.73, 3.45
[22:35:30] <icinga-miraheze>	 RECOVERY - jobrunner4 Current Load on jobrunner4 is OK: OK - load average: 4.02, 4.73, 3.58
[22:37:00] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.20, 3.13, 3.26
[22:45:04] <icinga-miraheze>	 PROBLEM - jobrunner3 Current Load on jobrunner3 is CRITICAL: CRITICAL - load average: 6.05, 5.43, 4.32
[22:47:04] <icinga-miraheze>	 PROBLEM - jobrunner3 Current Load on jobrunner3 is WARNING: WARNING - load average: 5.83, 5.59, 4.52
[22:57:04] <icinga-miraheze>	 RECOVERY - jobrunner3 Current Load on jobrunner3 is OK: OK - load average: 4.64, 5.06, 4.78
[23:04:59] <icinga-miraheze>	 PROBLEM - jobrunner3 Current Load on jobrunner3 is CRITICAL: CRITICAL - load average: 6.33, 5.33, 4.91
[23:06:59] <icinga-miraheze>	 RECOVERY - jobrunner3 Current Load on jobrunner3 is OK: OK - load average: 3.08, 4.56, 4.69