[04:45:04] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjNlf [04:45:05] [02miraheze/MirahezeMagic] 07translatewiki 03203ca59 - Localisation updates from https://translatewiki.net. [04:45:07] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjNlJ [04:45:08] [02miraheze/ManageWiki] 07translatewiki 039fa906f - Localisation updates from https://translatewiki.net. [06:25:54] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3101 MB (12% inode=94%); [09:40:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjNRo [09:40:12] [02miraheze/services] 07MirahezeSSLBot 03415465b - BOT: Updating services config for wikis [11:43:04] [02mw-config] 07RhinosF1 opened pull request 03#2737: make NS_MAIN index on horizonwiki - 13https://git.io/fjNEp [11:43:35] [02mw-config] 07RhinosF1 commented on pull request 03#2737: make NS_MAIN index on horizonwiki - 13https://git.io/fjNEh [11:45:49] * RhinosF1 waits for someone to merge and do the mwrestriced bit [11:48:25] I am running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 2.8.0.0 [libirc v. 1.0.3] my source code is licensed under GPL and located at https://github.com/benapetr/wikimedia-bot I will be very happy if you fix my bugs or implement new features [11:48:25] @help [11:48:27] [ wm-bot - Meta ] - meta.wikimedia.org [11:48:29] [ GitHub - benapetr/wikimedia-bot: IRC bot that is being used on number of wikimedia channels ] - github.com [11:48:46] @seen JohnLewis [11:48:46] RhinosF1: Last time I saw JohnLewis they were quitting the network with reason: Quit: Connection closed for inactivity N/A at 8/23/2019 4:58:05 AM (6h50m40s ago) [11:49:26] I will let you know when I see JohnLewis and I will deliver that message to them [11:49:26] @notify JohnLewis pls merge https://git.io/fjNEh and do the restricted bit [11:49:27] [ make NS_MAIN index on horizonwiki by RhinosF1 · Pull Request #2737 · miraheze/mw-config · GitHub ] - git.io [13:45:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjNgs [13:45:11] [02miraheze/services] 07MirahezeSSLBot 03d6b0c0b - BOT: Updating services config for wikis [15:21:23] PROBLEM - wiki.s23.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.s23.org' expires in 7 day(s) (Sat 31 Aug 2019 03:19:24 PM GMT +0000). [16:29:54] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [18:35:35] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:35:37] PROBLEM - misc2 Redis Process on misc2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:35:38] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:35:46] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:36:15] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:51] PROBLEM - misc2 Disk Space on misc2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:36:52] PROBLEM - misc2 SSH on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:57] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [18:37:00] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:37:07] PROBLEM - Host misc2 is DOWN: PING CRITICAL - Packet loss = 100% [18:37:22] PROBLEM - db4 Current Load on db4 is CRITICAL: CRITICAL - load average: 13.05, 7.68, 4.18 [18:37:35] JohnLewis: ^ [18:38:07] Error 503 Backend fetch failed, forwarded for 2a00:23c4:9e1e:e900:e5ab:8d47:3b4e:da75, 127.0.0.1 [18:38:07] (Varnish XID 566483334) via cp4 at Fri, 23 Aug 2019 18:37:51 GMT. [18:38:14] We're down [18:38:57] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 31% [18:39:03] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [18:40:50] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [18:42:24] Reception123: any chance you're stable enough to bring us back up at least [18:42:28] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [18:42:57] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 41% [18:44:59] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [18:45:00] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 47% [18:46:58] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 54% [18:48:51] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 25% [18:50:29] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [18:50:58] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 24% [18:51:44] Nope, not for another 40 mins at least [18:52:00] Though if JohnLewis joined he should be able to [18:52:57] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 35% [18:53:58] * RhinosF1 pokes JohnLewis with a stick [18:54:28] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [18:55:45] Not db4 this time [18:55:46] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=db4.miraheze.org&var-port=9100 [18:55:47] [ Grafana ] - grafana.miraheze.org [18:56:27] paladox: what the hell is that load? [18:56:35] 300 odd % [18:56:58] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 51% [18:57:27] Oh [18:57:28] Damn, didn’t read that [18:57:29] JohnLewis: ^ [18:57:51] paladox: disk space is fine for now but that load is through the roof and more [18:58:23] Well then I wouldn't be sure even when I have access but now can't do anything more than you guys [18:58:35] ~30GB on db4 so guessing it's the load https://icinga.miraheze.org/monitoring/service/show?host=db4&service=db4%20Disk%20Space [18:58:36] [ Icinga Web 2 Login ] - icinga.miraheze.org [18:58:58] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 28% [18:58:59] JohnLewis is our only hope [18:59:00] Reception123: my knowledge goes as far as the load shouldn't be that high :) [18:59:37] help, miraheze is 503ing [18:59:57] po209: hi. We're aware, sorry for the inconvenience [18:59:57] po209: we know [19:00:23] Looks like the load on a dB is through the roof but that's as much as we can say at the moment [19:01:24] also icinga reports misc2 as down [19:01:53] Voidwalker: known [19:02:04] Icinga says a lot at the moment [19:02:23] Sometimes it's everything moaning that other stuff are down [19:03:28] Ohh [19:03:29] That’ll be it [19:03:30] If redis goes down [19:03:31] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=misc2.miraheze.org&var-port=9100 [19:03:32] [ Grafana ] - grafana.miraheze.org [19:03:53] Ah [19:03:56] paladox: meta is back up but plain text [19:04:22] paladox: could RN have shut it down? (No access there either atm) [19:04:50] Bugger I used the wrong email to email John... [19:04:51] That’s why there’s likley high load [19:04:52] Reception123: possible [19:05:37] Will look in ~20 mins [19:08:57] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [19:12:57] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 55% [19:12:58] login miraheze no work paladox JohnLewis [19:13:15] Yup, aware [19:13:17] Hispano76: we are aware. Sorry for the inconvenience [19:13:20] It should be fixed soon [19:20:57] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 37% [19:21:00] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 55% [19:22:56] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 34% [19:26:39] RECOVERY - Host misc2 is UP: PING WARNING - Packet loss = 86%, RTA = 0.38 ms [19:26:52] RECOVERY - misc2 SSH on misc2 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u6 (protocol 2.0) [19:27:00] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 0.37, 0.10, 0.03 [19:27:01] RhinosF1: paladox we're back! [19:27:34] RECOVERY - misc2 Redis Process on misc2 is OK: PROCS OK: 1 process with args 'redis-server' [19:27:46] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:27:47] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 54 minutes ago with 0 failures [19:27:48] Back!! Thanks everyone for the response. Already filed https://phabricator.miraheze.org/T4670 as incident follow up [19:27:49] [ ⚓ T4670 Bandwidth should be monitored to prevent downtime ] - phabricator.miraheze.org [19:28:09] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 41810 bytes in 0.211 second response time [19:28:14] RECOVERY - misc2 Disk Space on misc2 is OK: DISK OK - free space: / 44552 MB (72% inode=99%); [19:28:28] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [19:28:47] Hispano76, po209: thanks for the reports - restored [19:28:50] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [19:29:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:29:03] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [19:29:04] Now for incinga to spam us with recovering messages [19:29:38] The one time I'll be happy for Icinga spam [19:33:22] PROBLEM - db4 Current Load on db4 is WARNING: WARNING - load average: 0.74, 3.82, 7.20 [19:33:56] Still calming down it looks like [19:35:20] It's going the right way on grafana [19:35:22] RECOVERY - db4 Current Load on db4 is OK: OK - load average: 1.35, 3.06, 6.51 [19:36:27] Yey, everything 100% recovered [19:37:24] Yup [20:58:53] paladox: https://mobile.twitter.com/RamNode/status/1164984303530496001 [20:58:53] [ Twitter ] - mobile.twitter.com [21:04:21] paladox: also an offer on when you add £5 to a new instance they give an extra 50% credit [21:11:52] Yup and it’s dollers not pounds :) [21:14:23] paladox: never checked that - I read it too quickly [21:59:23] JohnLewis: https://github.com/miraheze/mw-config/pull/2737 [21:59:24] [ make NS_MAIN index on horizonwiki by RhinosF1 · Pull Request #2737 · miraheze/mw-config · GitHub ] - github.com [22:00:38] RhinosF1: NSMAIN doesn't exist [22:01:57] [02mw-config] 07RhinosF1 synchronize pull request 03#2737: make NS_MAIN index on horizonwiki - 13https://git.io/fjNEp [22:02:27] JohnLewis: try that ^ [22:02:32] Should be NS_MAIN [22:02:40] paladox: already done [22:03:01] Ok [22:35:06] Question regarding my question here: https://meta.miraheze.org/wiki/Community_noticeboard#Why_is_member_in_User_Rights%3A_Groups_you_cannot_change.3F [22:35:10] [ Community noticeboard - Miraheze Meta ] - meta.miraheze.org [22:35:46] MarkDilley: hi, which right are you looking to enable? [22:36:22] I am trying to give people basic access to the wiki, so member? [22:36:43] Right now, I have to give them administrative and I don't trust that [22:36:50] If you go to managewiki/permissions for the user you want to be able to assign the right then you'll see a section with a grid where you can enable it [22:36:53] MarkDilley: ^ [22:37:54] I see the Groups Assignment tab and these headings: Can add to others Can remove from others Can add to self Can remove from self [22:38:01] not sure what to do after that [22:38:57] MarkDilley: tick 'can add to others' for 'member' when looking at the group and then whichever group you're in will be able to add that right [22:39:08] Can remove from others - lets them revoke it [22:39:29] and add to self / remove from self - lets the add/remove it from their account [22:42:24] Hey guys finally finished version 1.0.0 of the global checkuser extension [22:42:38] https://github.com/examknow/GlobalCheckUser/tree/master [22:42:39] [ GitHub - examknow/GlobalCheckUser ] - github.com [22:42:46] [02mw-config] 07JohnFLewis closed pull request 03#2737: make NS_MAIN index on horizonwiki - 13https://git.io/fjNEp [22:42:47] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjNXT [22:42:49] [02miraheze/mw-config] 07RhinosF1 03734253a - make NS_MAIN index on horizonwiki (#2737) * make NS_MAIN index on horizonwiki per T4669 * + missing _ [22:43:10] JohnLewis: did you do the ManageWiki restricted bit [22:43:24] doing no [22:43:26] w [22:43:52] Examknow: you should probably add a request to phab so it can go through review [22:44:11] Thanks for trying RhinosF1 - I am not seeing that clicking on that grid and saving it, moved the member into groups you can change, it is still in groups you cannot change [22:44:36] ok [22:45:15] MarkDilley: which wiki? [22:45:27] crusbd.miraheze.org [22:45:52] could somebody change the email on my phab account to examknow@outlook.com [22:47:19] JohnLewis: ^ that's your next job [22:47:40] can do it via Phab yourself :) [22:48:22] it will not let me as it is showing the email as an email that i do not have access to [22:48:36] hm? [22:48:37] JohnLewis: can you look at the wiki for MarkDilley and work out what he's done and help him set 'member' up right. It's private so I can't see what he's tried etc. [22:48:53] RhinosF1/ JohnLewis: give me a sec [22:49:33] RhinosF1: they've done it to the member group [22:49:45] JohnLewis: it wants me to verify an email that I do not have access to [22:50:24] Examknow: oh, so it's the only email on your account which was previously unverified? [22:50:27] MarkDilley: please move the setting you just changed to the 'administrator' or crat groups so they can add it to member [22:50:40] JohnLewis, ok, I tried something again and it looks like it hasn't done what I expected. Please help now [22:50:47] ok, let me look [22:50:51] JohnLewis: Yes [22:51:04] Examknow: okay, give me a few mins [22:51:57] MarkDilley: can you give me some access to see the wiki [22:53:15] exciting Phabricator is fatalling via command line [22:54:33] JohnLewis: PM? [22:56:13] RhinosF1: sure [22:56:26] in the meantime is there anything I can do to help around here [22:57:02] any system stuff for me to do? [22:57:46] Examknow: not atm I don't think, keep an eye on phab. Most stuff are restricted that need doing [23:01:46] MarkDilley: Can you please give me access to your wiki or turn off private wiki so I can check it out? [23:02:29] Examknow: i'm on it [23:08:05] JohnLewis: Any luck with my phabricator account [23:08:18] Examknow: nope, still looking [23:09:12] Examknow: can you tell me what domain the email you're being ask to verify is with? [23:09:31] examknow@outlook.com [23:10:40] JohnLewis: what’s the error? [23:10:41] if that's the one you're being asked to verify, then surely there's nothing for me to do :) [23:10:51] paladox: Call to undefined method PhabricatorUserEditor::updateUser() [23:11:00] Oh [23:11:44] it is not [23:11:47] * RhinosF1 sneaks off so he can sleep [23:11:50] JohnLewis: I’m pretty sure examknow is having the same issue as spF|Cloud when using the outlook email [23:12:05] paladox: no [23:12:13] the email he has with his account is one he can't access anymore [23:12:26] Oh [23:13:08] is there a way to delete my phab account and I can make a new one? [23:13:28] There is [23:13:49] ok you can just delete my account and I will create a new one [23:14:23] Examknow: account gone [23:14:35] !log deleted Examknow account on Phabricator [23:14:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:16:29] JohnLewis, paladox: It worked thanks :) ! [23:18:18] JohnLewis: To request a security review should I just put it under tasks? [23:19:14] yeah [23:25:47] Security Review is now up on phabricator [23:32:03] If anybody wants to help just give me your GitHub username and I can add you to the project