[00:03:04] JohnLewis, sup [00:06:07] hey [12:22:47] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 461 [12:24:47] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [12:35:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHbt [12:35:13] [02miraheze/services] 07MirahezeSSLBot 0340014ee - BOT: Updating services config for wikis [13:16:20] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'allthetropes.orain.org' expires in 15 day(s) (Tue 13 Nov 2018 01:13:24 PM GMT +0000). [13:26:25] PROBLEM - athenapedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'athenapedia.org' expires in 15 day(s) (Tue 13 Nov 2018 01:23:40 PM GMT +0000). [13:26:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHAH [13:26:39] [02miraheze/ssl] 07MirahezeSSLBot 0349a3dd5 - Bot: Update SSL cert for athenapedia.org [13:32:25] RECOVERY - athenapedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'athenapedia.org' will expire on Sat 26 Jan 2019 12:26:32 PM GMT +0000. [13:40:10] PROBLEM - garrettcountyguide.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'garrettcountyguide.com' expires in 15 day(s) (Tue 13 Nov 2018 01:37:18 PM GMT +0000). [13:40:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHxC [13:40:23] [02miraheze/ssl] 07MirahezeSSLBot 0319b293b - Bot: Update SSL cert for garrettcountyguide.com [13:42:08] RECOVERY - garrettcountyguide.com - LetsEncrypt on sslhost is OK: OK - Certificate 'garrettcountyguide.com' will expire on Sat 26 Jan 2019 12:40:16 PM GMT +0000. [13:50:21] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [13:50:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHxS [13:50:56] [02miraheze/ssl] 07MirahezeSSLBot 03228a0c4 - Bot: Update SSL cert for garrettcountyguide.com [13:51:28] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHx5 [13:51:30] [02miraheze/ssl] 07MirahezeSSLBot 03884dec4 - Bot: Update SSL cert for garrettcountyguide.com [13:51:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHxA [13:51:43] [02miraheze/ssl] 07MirahezeSSLBot 0305ad238 - Bot: Update SSL cert for allthetropes.orain.org [13:51:47] woo it works [13:51:49] * paladox pushes fix [13:55:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHpI [13:55:27] [02miraheze/ssl] 07MirahezeSSLBot 037d32b22 - Bot: Update SSL cert for garrettcountyguide.com [13:55:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHpL [13:55:35] [02miraheze/ssl] 07MirahezeSSLBot 0346c73b0 - Bot: Update SSL cert for allthetropes.orain.org [13:57:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHpq [13:57:31] [02miraheze/puppet] 07paladox 038c62642 - Fix file locking in mirahezerenewssl [14:02:21] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.orain.org' will expire on Sat 26 Jan 2019 12:55:29 PM GMT +0000. [14:06:22] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [14:22:08] presented without comment https://usercontent.irccloud-cdn.com/file/eRofRNg5/IMG_5041.PNG [14:23:40] ah thanks [14:24:10] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHhR [14:24:11] [02miraheze/landing] 07paladox 03d38ff59 - Fix spelling [14:24:14] revi ^^ [14:24:25] :) [14:24:46] and Reception123 should check their inbox [14:25:26] revi: looking :) [14:25:35] miraheze/landing/master/d38ff59 - paladox The build is still failing. https://travis-ci.org/miraheze/landing/builds/447408491 [14:34:55] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHhQ [14:34:57] [02miraheze/landing] 07paladox 0313228d7 - Remove "is-active" from "Wiki" navbar item [14:36:22] miraheze/landing/master/13228d7 - paladox The build is still failing. https://travis-ci.org/miraheze/landing/builds/447411087 [14:38:26] Sorry, could a system administrator import this file into DestinosWiki? https://phabricator.miraheze.org/T3725 [14:38:27] Title: [ ⚓ T3725 Request to import XML file to DestinosWiki ] - phabricator.miraheze.org [14:41:17] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHje [14:41:18] [02miraheze/landing] 07paladox 031e00697 - Update meta tag to be less restrictive [14:41:26] Wiki1776_ yep [14:42:36] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=destinoswiki --report=1 Taller_Wiki-20181024160736.xml on mw1 [14:42:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:42:47] miraheze/landing/master/1e00697 - paladox The build is still failing. https://travis-ci.org/miraheze/landing/builds/447412860 [14:44:32] thanks thanks thanks paladox [14:44:37] your welcome :) [14:44:37] :) [15:34:45] PROBLEM - meta.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'meta.orain.org' expires in 15 day(s) (Tue 13 Nov 2018 03:31:25 PM GMT +0000). [15:36:46] paladox: ^ where’s the bot? [15:37:46] im not sure, im looking [15:40:45] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQv6 [15:40:46] [02miraheze/ssl] 07paladox 03f833814 - Set "disable_event" to false for more certs [15:40:50] Fixed i think [15:44:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQvF [15:44:31] [02miraheze/ssl] 07MirahezeSSLBot 03b71240e - Bot: Update SSL cert for meta.orain.org [15:52:46] RECOVERY - meta.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.orain.org' will expire on Sat 26 Jan 2019 02:44:23 PM GMT +0000. [16:00:13] PROBLEM - miraheze.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.com' expires in 15 day(s) (Tue 13 Nov 2018 03:57:16 PM GMT +0000). [16:00:21] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 650 [16:00:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQfi [16:00:28] [02miraheze/ssl] 07MirahezeSSLBot 03b99a050 - Bot: Update SSL cert for miraheze.com [16:02:09] RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Sat 26 Jan 2019 03:00:20 PM GMT +0000. [16:02:21] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [16:06:21] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 382 [16:12:22] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [16:14:06] https://meta.miraheze.org/w/index.php?title=Special:CentralNotice&subaction=noticeDetail¬ice=Landing+Page+Launch PLEASE set an impression diet [16:14:08] Title: [ Central notice admin - Miraheze Meta ] - meta.miraheze.org [16:14:09] unless it is a downtime [16:14:43] downtime or an super duper important things like read-only notices [16:15:04] even Wikipedia donation (ads) sets an impression diet AFAIK [16:18:31] done [16:19:30] thanks [16:20:09] your welcome :) [19:13:53] !log upgrading phabricator on misc4 [19:13:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:10:36] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQsv [20:10:38] [02miraheze/landing] 07paladox 037c3fa8c - Fix spacing the log [20:12:04] miraheze/landing/master/7c3fa8c - paladox The build is still failing. https://travis-ci.org/miraheze/landing/builds/447507859 [20:12:48] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQsY [20:12:50] [02miraheze/landing] 07paladox 0305ef5f6 - Add some more meta tags [20:13:49] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQsn [20:13:51] [02miraheze/landing] 07paladox 038677279 - Fix spelling mistake [20:14:22] miraheze/landing/master/05ef5f6 - paladox The build is still failing. https://travis-ci.org/miraheze/landing/builds/447508403 [20:15:20] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQsC [20:15:21] [02miraheze/landing] 07paladox 03bbf2959 - Update index.html [20:16:53] miraheze/landing/master/bbf2959 - paladox The build is still failing. https://travis-ci.org/miraheze/landing/builds/447509187 [20:20:09] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 309 [20:22:09] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [20:33:54] PROBLEM - public.revi.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'public.revi.wiki' expires in 15 day(s) (Tue 13 Nov 2018 08:30:27 PM GMT +0000). [20:34:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQGm [20:34:11] [02miraheze/ssl] 07MirahezeSSLBot 03f28c069 - Bot: Update SSL cert for public.revi.wiki [20:34:14] PROBLEM - madgenderscience.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'madgenderscience.wiki' expires in 15 day(s) (Tue 13 Nov 2018 08:30:56 PM GMT +0000). [20:34:28] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQGY [20:34:29] [02miraheze/ssl] 07MirahezeSSLBot 038c4c894 - Bot: Update SSL cert for madgenderscience.wiki [20:36:08] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 387 [20:38:06] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [20:41:55] RECOVERY - public.revi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'public.revi.wiki' will expire on Sat 26 Jan 2019 07:34:01 PM GMT +0000. [20:42:07] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 786 [20:42:15] RECOVERY - madgenderscience.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'madgenderscience.wiki' will expire on Sat 26 Jan 2019 07:34:22 PM GMT +0000. [20:50:41] PROBLEM - mw1 Disk Space on mw1 is WARNING: DISK WARNING - free space: / 14550 MB (18% inode=98%); [20:51:34] thats me [20:52:34] !log rm /home/reception/nonsensopediawiki31072018.xml on mw1 [20:52:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:52:41] RECOVERY - mw1 Disk Space on mw1 is OK: DISK OK - free space: / 21787 MB (28% inode=98%); [20:52:47] Reception123: heh [20:52:49] how much was that? [20:53:43] Reception123 7.1g [20:54:13] !log rm -rf /home/reception/bpwiki (saves another 11g) [20:54:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:00:24] PROBLEM - podpedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'podpedia.org' expires in 15 day(s) (Tue 13 Nov 2018 08:56:52 PM GMT +0000). [21:00:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQZm [21:00:36] [02miraheze/ssl] 07MirahezeSSLBot 035b2a34d - Bot: Update SSL cert for podpedia.org [21:02:04] Ah ok [21:02:06] That's a lot [21:02:22] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Sat 26 Jan 2019 08:00:29 PM GMT +0000. [21:14:13] PROBLEM - lodge.jsnydr.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lodge.jsnydr.com' expires in 15 day(s) (Tue 13 Nov 2018 09:11:35 PM GMT +0000). [21:14:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQZ5 [21:14:28] [02miraheze/ssl] 07MirahezeSSLBot 031fd2367 - Bot: Update SSL cert for lodge.jsnydr.com [21:15:09] PROBLEM - poserdazfreebies.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'poserdazfreebies.orain.org' expires in 15 day(s) (Tue 13 Nov 2018 09:12:43 PM GMT +0000). [21:15:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQZF [21:15:23] [02miraheze/ssl] 07MirahezeSSLBot 031477019 - Bot: Update SSL cert for poserdazfreebies.orain.org [21:16:49] PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.autocountsoft.com' expires in 15 day(s) (Tue 13 Nov 2018 09:13:52 PM GMT +0000). [21:17:05] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQnf [21:17:07] [02miraheze/ssl] 07MirahezeSSLBot 03b43330f - Bot: Update SSL cert for wiki.autocountsoft.com [21:21:09] RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'poserdazfreebies.orain.org' will expire on Sat 26 Jan 2019 08:15:15 PM GMT +0000. [21:22:13] RECOVERY - lodge.jsnydr.com - LetsEncrypt on sslhost is OK: OK - Certificate 'lodge.jsnydr.com' will expire on Sat 26 Jan 2019 08:14:21 PM GMT +0000. [21:22:49] RECOVERY - wiki.autocountsoft.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.autocountsoft.com' will expire on Sat 26 Jan 2019 08:16:59 PM GMT +0000. [21:30:37] [02miraheze/landing] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQn7 [21:30:38] [02miraheze/landing] 07MacFan4000 03100c402 - Attempt to fix travis [21:32:12] miraheze/landing/master/100c402 - MacFan4000 The build is still failing. https://travis-ci.org/miraheze/landing/builds/447530342 [21:32:57] MacFan4000 you broke the button now [21:33:49] I didn’t see any releated CSS [21:35:44] paladox: https://test1.miraheze.org/landing/ [21:35:45] Title: [ Miraheze ] - test1.miraheze.org [21:35:52] Doesn’t look broken to me [21:36:01] MacFan4000 it does [21:36:04] click the donate button [21:36:08] no dialog shows [21:36:08] (I made sure it had the latest version) [21:36:14] Works for me [21:36:37] it dosen't for me [21:36:45] Strange [21:36:46] i clicked donate next to the join now [21:37:01] Oh [21:37:07] i know exactly why it's broken. [21:37:10] I clicked it in the menu bar [21:37:17] because im using JS to find the id [21:48:18] [02miraheze/landing] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQc9 [21:48:19] [02miraheze/landing] 07MacFan4000 03fb56413 - Fix [21:49:05] Fixed [21:49:42] miraheze/landing/master/fb56413 - MacFan4000 The build was fixed. https://travis-ci.org/miraheze/landing/builds/447535872 [21:50:21] And build fixed [21:57:45] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxQCT [21:57:47] [02miraheze/landing] 07paladox 03b97fd63 - Update index.html [22:22:55] Hey, this might sound dumb but.. what did I do wrong here? I was trying to install an extension https://maiasongcontest.miraheze.org/wiki/LocalSettings.php [22:22:56] Title: [ LocalSettings.php - Maia Song Contest ] - maiasongcontest.miraheze.org [22:23:56] You enable extensions using Special:ManageWikiExtensions [22:24:50] I tried but it keeps unticking "YouTube" [22:24:56] i just want to embed a video >.< [22:25:08] joshhhhhhhh hi, you want Special:ManageWikiExtensions [22:25:26] per Voidwalker which i just saw commented now [22:25:27] :P [22:25:46] oh mess I think i was forgetting to submit [22:25:50] so it wasnt saving the tick LOL [22:27:13] okay it worked [22:27:21] is there a way to make the video smaller using the Youtube extension? [22:54:05] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [22:58:03] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 358 [23:06:06] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [23:37:36] HTTP CRITICAL - Unable to open TCP socket [23:40:36] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 11.22, 4.90, 2.00 [23:40:56] uh [23:42:36] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [23:42:52] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 2 backends are down. mw1 mw3 [23:42:56] HTTP CRITICAL - Unable to open TCP socket [23:43:19] oh [23:43:27] hang on [23:43:30] rottenlinks [23:44:14] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:45:00] !log reboot mw1 to stop RottenLinks maint script [23:45:08] HTTP CRITICAL - Unable to open TCP socket [23:45:48] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 93% [23:45:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [23:45:55] !log reboot mw2 to stop RottenLinks maint script [23:45:56] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [23:47:10] PROBLEM - cp5 HTTPS on cp5 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3814 bytes in 1.030 second response time [23:47:18] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [23:47:26] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3801 bytes in 0.020 second response time [23:47:34] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:47:54] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3801 bytes in 0.405 second response time [23:48:10] PROBLEM - mw1 php7.2-fpm on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:48:14] PROBLEM - mw1 SSH on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:48:17] PROBLEM - mw2 Disk Space on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:48:53] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 14.39, 7.80, 3.85 [23:49:05] PROBLEM - Host mw1 is DOWN: PING CRITICAL - Packet loss = 100% [23:49:21] PROBLEM - mw2 php7.2-fpm on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:49:25] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:49:35] PROBLEM - Host mw2 is DOWN: PING CRITICAL - Packet loss = 100% [23:51:07] RECOVERY - Host mw1 is UP: PING OK - Packet loss = 0%, RTA = 2.96 ms [23:51:11] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/1.1 200 OK - 23556 bytes in 2.104 second response time [23:51:15] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [23:51:19] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 37% [23:51:27] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 23556 bytes in 0.048 second response time [23:51:31] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 1.14, 0.35, 0.12 [23:51:37] RECOVERY - Host mw2 is UP: PING OK - Packet loss = 0%, RTA = 0.88 ms [23:51:41] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:45] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 0.22, 0.05, 0.02 [23:51:49] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 27% [23:51:55] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 23565 bytes in 0.510 second response time [23:51:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:52:01] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 22759 bytes in 0.042 second response time [23:52:09] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:52:17] RECOVERY - mw2 Disk Space on mw2 is OK: DISK OK - free space: / 27296 MB (35% inode=98%); [23:52:51] RECOVERY - mw2 SSH on mw2 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0) [23:52:51] !log reboot mw1 to stop RottenLinks maint script at 23:45:00 utc [23:52:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:53:01] !log reboot mw2 to stop RottenLinks maint script at 23:45:55 UTC [23:53:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:53:11] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 22759 bytes in 0.042 second response time [23:53:13] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:16] !log reboot mw3 [23:53:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:55:19] PROBLEM - mw3 php7.2-fpm on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:55:27] PROBLEM - mw3 JobRunner Service on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:55:33] PROBLEM - mw3 SSH on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:53] PROBLEM - mw3 Disk Space on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:56:31] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:56:35] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:56:43] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:56:45] PROBLEM - Host mw3 is DOWN: PING CRITICAL - Packet loss = 100% [23:58:31] PROBLEM - test1 Puppet on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:58:45] RECOVERY - Host mw3 is UP: PING OK - Packet loss = 0%, RTA = 0.54 ms [23:58:57] PROBLEM - test1 SSH on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:59:07] RECOVERY - mw3 php7.2-fpm on mw3 is OK: PROCS OK: 5 processes with command name 'php-fpm7.2' [23:59:09] PROBLEM - test1 Disk Space on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:59:13] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 0.44, 0.11, 0.03 [23:59:17] RECOVERY - mw3 JobRunner Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [23:59:27] RECOVERY - mw3 SSH on mw3 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0) [23:59:43] RECOVERY - mw3 Disk Space on mw3 is OK: DISK OK - free space: / 60536 MB (78% inode=99%);