[00:00:32] Also, we are down [00:00:42] no, because the idea of pre minimal release is you don't release any information until it is 100% safe [00:01:13] You’re not releasing any information by simply stating that there is a security issue/legal issue/whatever [00:01:38] I deal with this frequently, and I’m obligated to state what the issue is, but not to share details about the issue [00:02:30] for example, I frequently get calls about suspected malware/viruses on corporate platforms. If I have to remotely access their servers and lock their databases, I am required to say “active security issue” or something like that [00:03:33] 99% of the time its just a bug in the software that’s easy for me or someone else to fix, but occasionally I’ve seen worse [00:03:55] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:04] okay, but you're commenting on a circumstance you know nothing about right now [00:04:29] Understood, but just in general stating what the issue is usually isn’t compromising anything [00:04:41] If you’re dealing with a security issue, saying such isn’t a problem (usually) [00:05:15] plus in the act of OSing something, stating it is an OS is sufficient [00:05:50] But the reason for the OS (in a general sense) should also be given ideally [00:06:08] (i.e. security threat, copyright or legal issue, ToU violation, etc) [00:06:11] Nothing more than that [00:06:33] well the removal implies [00:06:43] Meh [00:06:46] Anyway, we are down [00:06:46] honestly, when responding quickly - adding a rationale is the least of my issues. [00:06:51] I know. [00:07:17] I hope there wasn’t a server breach :/ [00:07:36] things should be back as they are [00:07:47] (well they are for me, if not give me links and say why) [00:07:51] Hmm, styleing for templates aren’t loading for me on meta [00:07:55] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:08:08] I can’t access Meta at all [00:08:14] I can [00:08:20] I can access my wiki, but it’s deformed, I can’t access anything else [00:08:27] MacFan4000: styling works for me [00:08:37] Miraheze is down for me except on weatherwiki [00:08:38] That sounds bad! A sysadmin should be here shortly to investigate. If you haven't already, please file a Phabricator ticket to facilitate the process! [00:08:39] cp2 and cp4 have the new CSP [00:08:47] not checked cp5 [00:08:53] What is ZppixBot responding to? [00:08:56] you [00:09:00] csp? [00:09:02] I didn’t ping it [00:09:06] content security policy [00:09:11] AmandaCatherine: miraheze is down triggers that [00:09:16] oh [00:09:22] Miraheze is down [00:09:22] That sounds bad! A sysadmin should be here shortly to investigate. If you haven't already, please file a Phabricator ticket to facilitate the process! [00:09:31] Well, it is except on my wiki right now [00:09:47] I’m not going to file a Phab yet given that it may be security-related [00:10:28] well a phab task will do nothing [00:10:31] https://usercontent.irccloud-cdn.com/file/P6ERNH59/IMG_0083.PNG [00:10:33] because I'm not going to check phab [00:10:39] That’s how things look for me [00:10:47] MacFan4000: is that... good? [00:11:01] ^ that looks like the userboxes are broken [00:11:08] ^^ [00:11:15] But when I try to access any wiki except my own, I just get a blank white screen [00:11:18] They should be floating to the right [00:11:24] ooh [00:11:29] it looked normal to me :P [00:11:51] Okay, confirmed that I can access the other wiki where I have rights (knowledgeboxwiki) [00:11:55] AmandaCatherine: clear browser cache [00:11:57] But other than that, MH is down for me [00:12:09] or check if there's any console errors [00:12:22] Already cleared cache - I’ll check my VPN error log [00:12:30] MacFan4000: looks good to me in my own view [00:12:36] again, browser cache probably [00:15:43] Wtf happened [00:16:00] CnocBride: you’re guess is as good as mine [00:16:16] MH is down for me except on the two wikis where I have elevated rights, others say that it’s up but malformed.. [00:16:25] John seems to hint at a possible security issue [00:18:17] JohnLewis: nothing in my error logs [00:18:22] MH is fine for me [00:18:23] But now I can’t even login to Miraheze [00:18:48] Did we change the Meta logo? The black/white logo is meh [00:19:11] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAYLL [00:19:13] [02miraheze/puppet] 07JohnFLewis 03239412a - update CSP again [00:19:50] Zppix: that's a result of an issue [00:19:54] https://i.imgur.com/ZbhACJ3.jpg this is how Meta looks for me right now [00:20:01] Miraheze is up, but I can’t log in [00:20:17] Its back to color now [00:20:25] JohnLewis: CSP? [00:20:52] Zppix: Content Security Policy [00:22:52] two [00:22:55] oops [00:23:01] that's the last change now [00:23:19] things should even out now. things may just be cached either browser wise or us wise [00:25:37] JohnLewis: I could run a huge purge api request on meta to make sure the CSP doesnt screw it up [00:25:52] Zppix: I mean in varnish [00:25:59] ah [00:26:07] !log deleting test1deletewiki and test11deletewiki (mine) [00:26:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:26:16] completely forgot about varnish [00:36:00] Okay, interface is back to normal, but I still can’t login [00:36:28] Oh nvm [00:36:30] Yes I can [00:40:29] good good [00:41:12] JohnLewis: everything okay tech/security wise? [00:41:27] yeah, just working on disclosure [00:42:30] Can you explain what this “Content Security Policy” is? [00:42:36] Is this a new global policy of some sort? [00:43:21] no, it's a HTTP standard for informing a browser of what it can and can't load [00:44:06] So whatever just happened involved someone mistakenly accessing something that they shouldn’t have? [00:44:51] no [00:45:00] intention and deliberate [00:45:16] Oh, so we were compromised, in a way [00:45:17] and wasn't a user, was a user getting other users to access [00:45:33] so meat puppetry in a way? [00:45:44] Recruiting someone to do something that isn’t allowed [00:46:19] no [00:46:36] deceiving users into loading an external script [00:46:44] That did what? [00:47:00] What did the External script do? Give them a virus? [00:47:52] So THAT’s what you OSed (the link to the malicious script) [00:48:39] yes, that's what I OS'd [00:49:16] Can you say what exactly made the script malicious (in a general sense)? [00:52:15] And, I guess most importantly, can you certify without a doubt that there is NOT currently any active threat to any Miraheze wiki or user? [00:52:26] it is not currently a threat [00:52:42] Thank you [00:52:58] I don't want to give information in fragments, I'm just finishing up and getting approved the disclosure notifications and metawiki page [00:53:20] What about the Phab task? [00:55:38] will be public as part of disclosure [00:55:49] Ok [01:07:39] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAYt3 [01:07:40] [02miraheze/puppet] 07JohnFLewis 03781ba9a - add wmflabs.org to CSP [01:18:14] [02miraheze/MirahezeMagic] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/fAYtd [01:18:15] [02miraheze/MirahezeMagic] 07JohnFLewis 0385e2b8e - Create sendBulkEmails.php [01:18:48] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fAYtb [01:18:50] [02miraheze/mediawiki] 07JohnFLewis 03873e2b1 - update MM [01:31:03] [02mw-config] 07Amanda-Catherine opened pull request 03#2402: rem siteadmin from weatherwiki stewards - 13https://git.io/fAYqI [01:32:06] PuppyKun: you got your way :) ^ [01:33:54] ;D [01:35:46] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 6052 [01:38:44] https://weather.miraheze.org/m/uK [01:38:45] Title: [ Login required - Weather Wiki ] - weather.miraheze.org [01:38:51] PuppyKun ^ [01:40:19] Mobile [01:41:02] Oh, I just linked to the diff where I removed the siteadmin stuff from the local steward policy, and included “and NDKilla wasn’t thrilled with it being there in the first place” [01:41:06] :P [01:51:57] Meh... does anyone know if ZppixBot has a global CA account? [01:52:21] Getting kind of tired of the bot not being able to read my wiki and therefore rendering every link to my wiki that I post here as “login required” [01:53:09] AmandaCatherine: I dont think it does? Maybe [01:53:18] there is an account, but the URL reader does not login to the sites it checks [01:53:49] So even if we put the global CA account on my local wiki user list, it still wouldn’t be able to read the wiki? [01:53:59] Despite the fact that the wiki is only private to anonymous users [01:54:10] nope [01:54:18] Bah [01:54:29] the script in use doesn't connect to any on wiki functionality [01:54:31] https://meta.miraheze.org/wiki/2018-08-26_Security_Disclosure [01:54:32] Title: [ 2018-08-26 Security Disclosure - Miraheze Meta ] - meta.miraheze.org [01:54:49] Thank you [01:55:14] It uses the account only to get phab task data [01:55:19] !log emailing disclosure emails to all users on jawp2chwiki [01:55:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:55:26] (will log result) [01:56:10] [02mw-config] 07MacFan4000 closed pull request 03#2402: rem siteadmin from weatherwiki stewards - 13https://git.io/fAYqI [01:56:11] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAYqF [01:56:13] [02miraheze/mw-config] 07Amanda-Catherine 03c537a22 - rem siteadmin from weatherwiki stewards (#2402) No longer needed thanks to ProtectSite’s ability to lock the entire wiki indefinitely [01:56:36] AmandaCatherine: see the link John posted for why we were busy earlier [01:56:54] I knew there was a security issue.. that was pretty obvious [01:57:08] Only security and legal issues would need to be kept private [01:58:05] Well that's a more in depth explanation [02:02:35] So, if I understand it correctly, that community was using a third-party application to conduct a poll, and said application was collecting more than just what the user’s submitted in the poll? [02:03:18] it seems to be a tool the community made [02:03:48] which stored IP/UA information and passed along the users CSRF [02:04:11] But what was the purpose of the tool? [02:04:21] Based on the name it looks like a polling gadget to me [02:04:21] poll [02:04:45] So... was the poll asking for the user’s filling it out to supply said information? [02:04:58] Or was there a malfunction somewhere that was collecting information that wasn’t submitted? [02:05:13] whether it was or not, it was collecting CSRF tokens [02:05:59] ...because? [02:06:00] and we do not allow users to collect IP/UA info and it wasn't in their privacy policy [02:06:15] Why was it collecting that? [02:06:20] because, it was. As said, intentions are not known. [02:07:29] I mean, if it was being collected and sent to whoever was keeping tabs of the poll results, that’s one thing. But if it was being sent to a third party like the Phab task suggests... yeah... that would be scary [02:07:58] if its being sent off Miraheze, it's third party. [02:08:10] The user collecting the pole results is third party [02:08:38] We can't let anyone collect information used to act on user accounts without explicit permission [02:08:41] no matter for what purpose, IP/UA/CSRF info should not be leaving Miraheze or being collected by users. Esp the CSRF token [02:08:47] (Technically oath apps do this but with permission) [02:08:51] So the user collecting the poll results wasn’t a member of Miraheze [02:09:03] they were of the community [02:09:04] That doesnt matter honestly [02:09:34] Csrf tokens are out of scope for collection regardless of who was doing it [02:09:36] So it wasn’t “leaving Miraheze” as long as they didn’t post it to some external website/blog/forum/whatever [02:09:49] I’d be willing to bet that it wasn’t intentional [02:10:06] Bs [02:10:25] Theres no way you accidentally write a script that queries the mediawiki api for private information. [02:10:47] but it was leaving Miraheze [02:11:22] it was going to a third party site not controlled by Miraheze, was being collected not for Miraheze purposes and against our privacy policy (and the communities privacy policy) [02:15:02] Ah [02:15:31] PuppyKun: what I meant is that they probably wrote the script for something else, and that it had a security bug in it that they didn’t catch [02:15:42] extremely careless [02:15:50] no, it wasn't a bug [02:16:04] the user was actually storing the IP and UAs [02:16:12] Huh [02:16:27] And that information wasn’t actually part of the survey? [02:16:39] well, yes [02:16:53] (i.e. stuff like what location do you access from, what device/browser do you access from, etc.) [02:17:01] Those types of questions were or were not in the survey? [02:17:20] you can ask those question but you can not retrieve the users IP and UA [02:17:51] it was retrieved without permission and against our privacy policy. [02:18:30] A UA is simply just what device you’re using, what browser you’re using, and the model and version of each [02:18:44] But the IP... yeah, that’s stretching it [02:18:53] and is sensitive information under EU regs [02:19:56] And there’s the unreasonably strict EU regs for everyone [02:20:39] We consider IP data to be sensitive, but not necessarily UA data, since the joe-average citizen isn’t going to know someone’s identity by just knowing what device and web browser they are accessing a website from [02:21:47] unfortunately we have no say in what the EU think [02:22:07] “We” meaning “Canada” in my last message [02:22:24] I understand, but it’s just another example of where the EU is probably being too cautious [02:22:44] yeah... [02:23:03] Fortunately or unfortunately we provide a service to people based in the eu [02:24:04] I mean, seriously, you’re not going to be able to figure out someone’s RL identity simply by looking at what web browser and device they were using at a given time [02:24:27] either can you by knowing their gender really [02:24:35] but it's still sensitive information [02:24:37] The only way to take a UA and convert it into something that can determine an RL identity requires access to applications and software that even in Canada are highly restricted [02:25:02] Gender says nothing about an actual RL ID [02:30:05] BTW does the CSRF have anything to do with the “there seems to be a problem with your login session and it has been canceled as a precaution against session hijacking” nonsense that pops up every now and then? [02:30:34] sort of [02:36:03] Anyway [02:36:05] !night [02:36:05] Good night, sweet dreams and dream of the miraheze servers without errors! [03:57:44] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [05:57:57] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAYZo [05:57:59] [02miraheze/puppet] 07Reception123 033a23e76 - remove DROP grant from wikiadmin MediaWiki admins should not be using DROP. (Hope this doesn't actually affect something) [06:24:26] !log running wikibackups for all wikis [06:24:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [09:26:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAYgt [09:26:40] [02miraheze/puppet] 07paladox 03090c917 - Add youtube.com to CSP fixes T3521 [09:26:45] Reception123: ^^ [09:26:59] ah, it was blacklisting the actual site [09:27:00] I see [09:27:33] Yep [09:27:40] I'll run puppet to see if that fixes [09:27:47] Ok thanks [09:27:54] Needs puppet1 to be done first [09:28:00] yes, I know. already done [09:28:46] Oh [09:28:48] www.youtube-nocookie.com [09:28:52] I need to add ^^ [09:28:53] Too [09:29:28] paladox: it still says "Requests to the server have been blocked by an extension." [09:29:47] paladox: oh, it's www.youtube and you only added youtube [09:29:49] I'll make it *.youtube.com [09:30:42] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAYgW [09:30:43] [02miraheze/puppet] 07Reception123 03af1898a - keep youtube *. in CSP to support www [09:33:17] Reception123: try now [09:34:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAYgN [09:34:18] [02miraheze/puppet] 07paladox 03d405df6 - Add www.youtube-nocookie.com to CSP [09:35:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAYgh [09:35:04] [02miraheze/puppet] 07paladox 034672436 - Update default.vcl [09:35:31] ok [09:42:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAY2W [09:42:07] [02miraheze/puppet] 07paladox 03d435925 - Rm drop from mediawiki grants [09:42:09] Reception123: ^^ [09:42:13] paladox: works now, great, thanks :) [09:42:37] paladox: how do we know it's not necessary for ON `%wiki%`.* TO 'mediawiki'@'%'; though? [09:42:59] as I asked SPF earlier, doesn't mediawiki need DROP to execute some functions? [09:43:11] Have no idea [09:43:18] We will find out soon :D [09:43:31] paladox: heh that is not the best approach :P [09:43:38] I guess so [09:43:39] In that case, I won't run the grants until John is here [09:43:45] But we prioritise security [09:43:47] Though [09:44:01] So if it breaks we can recert [09:44:03] Revert [09:44:13] see -staff [09:44:17] Ok [09:51:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAY2b [09:51:23] [02miraheze/mw-config] 07paladox 032f953ae - Update MissingWiki.php [09:52:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAYav [09:52:48] [02miraheze/puppet] 07paladox 032436efa - Add maxcdn.bootstrapcdn.com to CSP [09:53:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAYaU [09:53:50] [02miraheze/puppet] 07paladox 03fdc0fc4 - Add twitter.com to CSP [10:58:16] I'd just like to commend the Operations members, the Pioneer and anyone else who was involved with resolving this security issue. [10:58:21] You're a credit to the community 😃 [11:00:00] @CnocBride: Thanks :) [11:00:15] you should specifically thank Jianhui67 [11:00:15] *JohnLewis [11:04:10] Thank you John :DD For everything you've done for Miraheze [11:22:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAY6Z [11:22:56] [02miraheze/puppet] 07paladox 033456c0a - Add *.creativecommons.org to CSP [11:25:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAY6a [11:25:42] [02miraheze/puppet] 07paladox 03ebee840 - Add images.uncyc.org to CSP [12:27:44] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2932 [13:05:02] MacFan4000: you about? [13:08:22] Cyberpower678: yes [13:09:40] MacFan4000: hi. I appreciate you bending head over to for me regarding the OAuth consumers, but you inadvertantly disabled the wrong one. ;p [13:10:44] MacFan4000: Can you re-enable https://meta.miraheze.org/w/index.php?title=Special:OAuthListConsumers/view/91409d445b28606bb2d839d45bfaa70b [13:10:45] Title: [ List OAuth applications - Miraheze Meta ] - meta.miraheze.org [13:11:09] MacFan4000: and then disable https://meta.miraheze.org/w/index.php?title=Special:OAuthListConsumers/view/beb6f3c7bba43f126831188ae83140a5 [13:11:10] Title: [ List OAuth applications - Miraheze Meta ] - meta.miraheze.org [13:11:34] And then I'll recreate that consumer this time taking note of the consumer secret. :p [13:13:12] Done and done [13:16:52] MacFan4000: And I proposed the new one, which I noted the keys down this time. :p [13:17:53] Cyberpower678: done [13:18:05] Sweet. No we should be good. :D [13:36:03] !log REVOKE SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE VIEW ON `%wik%`.* FROM 'mediawiki'@'%'; [13:36:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:36:09] !log REVOKE SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE VIEW ON `%wik%`.* FROM 'wikiadmin'@'%'; [13:36:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:36:53] just realized the new CSP stuff broke my global.js lol [13:36:57] but fine :P [13:37:10] !log REVOKED DROP from wikiadmin [13:37:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:39:30] revi: heh, do you need a whitelist for a site? :) [13:39:46] well.. I load my global.css and js from css/js-wiki-cdn.reviservices.com [13:40:16] (I had a plan to merge them to one domain but was too lazy) [13:42:47] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAY7S [13:42:49] [02miraheze/puppet] 07Reception123 03ed0ae90 - add www.mikrodev.com to CSP [13:45:04] *.reviservices.com [13:45:07] Should do it [13:45:19] on it [13:45:31] indeed :P [13:45:56] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAY7p [13:45:58] [02miraheze/puppet] 07Reception123 030732ffa - add *.reviservices.com to CSP [13:46:01] revi: ^ here you go :) [13:46:04] awesome [13:46:25] should be in effect in a few seconds [13:47:02] Reception123: your fast :D [13:47:20] I want to move this whitelist to a yam file [13:47:20] paladox: Puppet is fast :P [13:47:42] Heh [13:47:43] paladox: well, in theory, we should try to not make a huge list, and encourage people to use internal not external stuff [13:47:49] Yeh [13:47:58] But the list is already huge :P [13:59:50] Perhaps a whitelist would be good. But where to put it? [14:00:04] (Yam) [14:00:11] yaml* [14:02:44] but we want to minimise the sites whitelisted anyway [14:02:46] soo [14:03:14] paladox: ^ [14:03:37] Yep but still it looks horrible the way it is [14:06:04] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:06:44] it does, but having a YAML encourages additions [14:06:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [14:07:14] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [14:07:16] uh [14:07:16] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 87% [14:07:36] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:07:37] Did they do reboots [14:07:40] Of the nodes? [14:07:58] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:08:04] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [14:08:18] they shouldn't be rebooting several nodes at once [14:08:24] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:08:36] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:09:10] PROBLEM - misc4 Current Load on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:09:16] Someone rebooted cp4 [14:09:20] Uh [14:09:22] PROBLEM - bacula1 Bacula Misc4 Lizardfs Master on bacula1 is CRITICAL: CRITICAL: Timeout or unknown client: misc4-fd [14:09:40] PROBLEM - hellointernet.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:09:50] PROBLEM - misc4 Disk Space on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:10:02] And were down :D [14:10:08] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:10:25] PROBLEM - misc4 phd on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:10:28] They rebooted everything [14:10:35] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:10:53] Someone is going to have to ssh into mw* [14:10:56] And start ssh [14:10:59] And cron [14:11:13] JohnLewis: ^^ [14:11:39] RECOVERY - bacula1 Bacula Misc4 Lizardfs Master on bacula1 is OK: OK: Full, 16 files, 120.1MB, 2018-08-26 00:05:00 (1.6 days ago) [14:11:41] uh [14:11:41] PROBLEM - hellointernet.miraheze.org - GlobalSign on sslhost is WARNING: WARNING - Certificate '*.miraheze.org' expires in 26 day(s) (Sat 22 Sep 2018 08:12:13 PM GMT +0000). [14:11:41] no [14:11:43] mw is fine [14:11:53] PROBLEM - cp4 Disk Space on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:11:57] cp4 is fine [14:12:08] Ok [14:12:11] PROBLEM - test1 HTTP on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:14] everything seems fine? [14:12:20] How comes mw* shows as down for cp4? [14:12:22] And cp2? [14:12:27] "Everything is fine" [14:12:30] cp4 and test1 are super slow [14:12:37] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures [14:12:43] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.25, 0.12, 0.11 [14:13:05] mw* are responsive and no issues [14:13:55] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:37] PROBLEM - cp4 HTTP on cp4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:39] RECOVERY - misc4 phd on misc4 is OK: PROCS OK: 1 process with args 'phd' [14:14:55] PROBLEM - test1 php7.2-fpm on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:15:05] PROBLEM - test1 Disk Space on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:15:15] RECOVERY - misc4 Current Load on misc4 is OK: OK - load average: 0.00, 0.10, 0.23 [14:15:24] curl -L miraheze.org works for me on my linux box but can't load using desktop browser xc [14:15:35] Uh, you say no issues, but mw* is depooled on cp5+2 causing 593s [14:15:41] 503* [14:15:50] I know [14:15:53] but there is no issue [14:16:01] RECOVERY - misc4 Disk Space on misc4 is OK: DISK OK - free space: / 53510 MB (87% inode=99%); [14:16:05] RECOVERY - cp4 Disk Space on cp4 is OK: DISK OK - free space: / 23165 MB (56% inode=99%); [14:16:11] mw3 just served a request to me. Blame slow proxies [14:16:15] RECOVERY - test1 HTTP on test1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 459 bytes in 0.356 second response time [14:16:26] yeah [14:16:30] cp4 was super slow for me [14:16:39] but mw* was lightning fast [14:16:41] RECOVERY - cp4 HTTP on cp4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1010 bytes in 7.589 second response time [14:16:51] RECOVERY - test1 php7.2-fpm on test1 is OK: PROCS OK: 5 processes with command name 'php-fpm7.2' [14:16:55] I'd say so. Dom loaded in 140 ms [14:16:57]