[00:00:26] paladox: SPF|Cloud i think Not-def6 should be banned for spamming xD jk [00:00:41] /kickban Zppix [00:00:42] :D [00:00:47] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:01:36] paladox: umm Zppix != Not-def6 [00:01:41] :P [00:01:57] maybe you disguised your self as Zppix but are in fact Not-def6 [00:02:07] You could be JohnLewis for all we know [00:02:22] paladox: no no we all know the only other person that is me is ZppixBot [00:03:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8YC [00:03:14] [02miraheze/puppet] 07paladox 037fb3b0d - Update php_fpm.pp [00:03:33] lol [00:03:48] paladox: you ought to just make it commit message, rewriting the entirety of puppet xD [00:04:42] LOL [00:05:17] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:06:00] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [00:07:59] !log restarted php-fpm across all mw hosts [00:08:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8Y8 [00:08:06] [02miraheze/puppet] 07paladox 03dac8260 - Update mediawiki-includes.conf.erb [00:08:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:08:34] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [00:11:07] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:23:50] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8Yz [00:23:52] [02miraheze/mw-config] 07Southparkfan 030764c08 - Add profiling code [00:24:31] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8Yg [00:24:33] [02miraheze/mw-config] 07Southparkfan 033868b36 - Use strict if checking [00:26:14] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8Y2 [00:26:14] [02miraheze/mw-config] 07Southparkfan 03ac6f544 - Remove strict if [00:36:41] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [00:38:48] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [00:43:43] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [00:43:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb [00:44:20] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [00:44:34] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2 [00:48:35] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8YH [00:48:37] [02miraheze/puppet] 07Southparkfan 03019e788 - disable tideways [00:49:26] SpF|Cloud: you need to absent that [00:49:26] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8Y7 [00:49:27] [02miraheze/mw-config] 07Southparkfan 0318b4b0f - Disable profiling [00:50:46] quickly removed all traces anyways [00:51:07] !log killed tideways module on mediawiki servers because it seemed to cause downtime [00:52:27] Ah ok [00:57:51] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%); [00:58:30] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [01:06:56] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [01:07:04] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [01:07:08] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:07:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:26:11] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:26:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:26:54] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [01:27:29] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:27:59] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [01:28:00] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [01:29:25] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:29:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:29:57] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [01:30:26] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.004 second response time [01:30:49] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [01:30:50] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [01:45:43] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw2 mw3 [01:47:55] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [01:48:13] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [01:50:16] !log rolling restart of php-fpm on mw[123] [01:50:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:50:48] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [01:51:04] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [01:51:44] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [02:56:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [02:56:18] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [02:59:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:59:19] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [04:09:51] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 10.46, 8.33, 5.13 [04:15:23] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.40, 7.79, 5.98 [04:17:57] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 3.31, 5.93, 5.55 [04:45:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8sI [04:45:15] [02miraheze/services] 07MirahezeSSLBot 0366bfa7d - BOT: Updating services config for wikis [06:26:43] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3201 MB (13% inode=94%); [06:45:27] PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [06:53:19] RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:47] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:41:12] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [07:43:46] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24592 bytes in 0.004 second response time [07:44:10] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [07:55:42] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [07:58:15] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [10:05:58] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8cN [10:06:00] [02miraheze/puppet] 07Southparkfan 035775be4 - Install tideways conditionally [10:06:38] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8cx [10:06:39] [02miraheze/puppet] 07Southparkfan 0329f9a83 - Install tideways on test1 [12:05:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8lk [12:05:11] [02miraheze/services] 07MirahezeSSLBot 03ecf874d - BOT: Updating services config for wikis [12:42:26] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [12:42:27] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [12:43:17] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [12:44:04] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [12:44:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [12:48:54] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [12:49:02] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [12:49:46] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [12:50:29] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [12:50:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:50:21] !log rhinos@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php /home/rhinos/Wiesepedia-Respaldo.xml --wiki wiesepediawiki (x2 erroring with Warning: in_array() expects parameter 2 to be array, string given in /srv/mediawiki/config/LocalWiki.php on line 298 but proceeding) [14:50:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:54:04] paladox: https://github.com/miraheze/mw-config/blob/18b4b0f0de93f08d75fa48e1458042ad4bbc14f2/LocalWiki.php#L298 <-- that's you - see above [14:54:05] [ mw-config/LocalWiki.php at 18b4b0f0de93f08d75fa48e1458042ad4bbc14f2 · miraheze/mw-config · GitHub ] - github.com [14:56:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8BM [14:56:18] [02miraheze/mw-config] 07paladox 03f64c09f - Fix in_array [14:56:52] RhinosF1 fixed [14:57:00] paladox: good [14:57:32] paladox: do you want me to test it? [14:57:51] miraheze/mw-config/master/f64c09f - paladox The build was broken. https://travis-ci.org/miraheze/mw-config/builds/597678661 [14:57:51] You can if you want, but it'll definitly work [14:58:26] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8By [14:58:27] [02miraheze/mw-config] 07paladox 031380214 - Update LocalWiki.php [14:59:26] paladox: after travis errored? [14:59:36] miraheze/mw-config/master/1380214 - paladox The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/597679743 [14:59:45] RhinosF1 hmm? [15:00:05] paladox: you said that as travis moaned [15:00:55] Yup, was a syntax error [15:07:19] !log restarting that import to test paladox fix [15:07:25] paladox: worked [15:07:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:07:34] great! [15:07:47] though note that it was only a warning so you didn't have to stop the dump [15:08:22] paladox: Oh I know, I didn't, it was done anyway but the file was slightly broke at the end [15:08:32] oh, ok [15:10:02] !log rhinos@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki wiesepediawiki (to fix recent changes after import) [15:10:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:12:07] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.39, 7.03, 5.58 [15:14:43] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.40, 6.40, 5.55 [15:29:05] !log rhinos@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --update --wiki wiesepediawiki (post import clean up) [15:29:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:30:21] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [15:30:49] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [15:31:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb [15:33:55] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [15:35:11] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/Je80U [15:35:12] [02miraheze/MirahezeMagic] 07translatewiki 03a039cc0 - Localisation updates from https://translatewiki.net. [15:35:16] [ Main page - translatewiki.net ] - translatewiki.net. [15:36:55] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [15:36:58] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [15:37:10] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [15:37:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:45:21] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3 [15:46:11] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:48:14] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 3.19, 3.42, 1.98 [15:48:26] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [15:48:59] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:49:57] [02landing] 07Giorgio68 opened pull request 03#15: italian translation - 13https://git.io/Je80B [15:51:01] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.77, 2.71, 1.94 [15:58:06] [02landing] 07Giorgio68 opened pull request 03#16: italian option value added - 13https://git.io/Je80P [15:58:38] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [15:58:50] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw3 [15:58:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:59:29] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:00:00] [02landing] 07RhinosF1 closed pull request 03#15: italian translation - 13https://git.io/Je80B [16:00:01] [02miraheze/landing] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je80D [16:00:03] [02miraheze/landing] 07Giorgio68 03955e29f - italian translation (#15) [16:00:29] [02landing] 07RhinosF1 closed pull request 03#16: italian option value added - 13https://git.io/Je80P [16:00:31] [02miraheze/landing] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je80H [16:00:32] [02miraheze/landing] 07Giorgio68 0362077b5 - italian option value added (#16) [16:02:13] paladox: remind me to check that when puppet runs [16:02:34] * RhinosF1 wonders why he can't set a timer [16:02:42] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:02:43] .in 10mins it [16:02:45] RhinosF1: Okay, I will set the reminder for: 2019-10-14 - 17:12:44BST [16:04:21] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [16:05:01] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [16:05:24] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [16:05:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:05:44] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:05:45] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.641 second response time [16:07:05] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [16:11:36] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.73, 7.20, 5.62 [16:12:44] RhinosF1: it [16:14:14] see you've set a reminder. [16:14:33] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.97, 7.91, 6.15 [16:19:08] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.75, 7.10, 5.69 [16:19:48] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 3.30, 4.44, 3.16 [16:22:39] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 3.99, 5.88, 5.50 [16:24:42] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.06, 7.26, 6.73 [16:27:44] paladox: I did [16:27:52] And it works [16:27:54] yup [16:31:10] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.94, 5.64, 6.22 [16:36:29] PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:40:01] RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:49:44] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 0.45, 2.19, 3.60 [16:52:32] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 0.71, 1.58, 3.13 [17:04:00] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:07:02] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:08:55] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.76, 6.97, 5.92 [17:14:21] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.94, 6.85, 6.20 [17:17:11] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 2.95, 5.37, 5.75 [18:07:56] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8ub [18:07:57] [02miraheze/mw-config] 07paladox 03db7e277 - make loginwiki use /mnt/mediawiki-static again [18:08:10] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.10, 6.35, 4.83 [18:11:17] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.69, 6.68, 5.31 [18:14:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:16:35] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [18:16:38] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:16:39] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw3 [18:17:27] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [18:18:29] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:22:11] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.438 second response time [18:25:03] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:25:36] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:28:27] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.390 second response time [18:31:24] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:32:57] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.674 second response time [18:34:08] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [18:34:13] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [18:38:30] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 2.643 second response time [18:42:26] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw2 mw3 [18:42:30] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3 [18:48:51] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [18:49:09] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [18:49:14] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [18:49:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:50:52] Hello GethN7! If you have any questions, feel free to ask and someone should answer soon. [18:51:09] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:02:28] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [19:02:30] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [19:03:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:05:46] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [19:05:46] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [19:06:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:09:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je8z9 [19:09:41] [02miraheze/puppet] 07paladox 0379f46bc - varnish: Rate limit amazon bot in nginx [19:09:43] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [19:09:46] [02puppet] 07paladox opened pull request 03#1100: varnish: Rate limit amazon bot in nginx - 13https://git.io/Je8zH [19:10:21] [02puppet] 07paladox synchronize pull request 03#1100: varnish: Rate limit amazon bot in nginx - 13https://git.io/Je8zH [19:10:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je8zQ [19:10:24] [02miraheze/puppet] 07paladox 030ff7a24 - Update mediawiki.conf [19:20:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8gW [19:20:12] [02miraheze/services] 07MirahezeSSLBot 0397bb294 - BOT: Updating services config for wikis [19:23:03] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'm.miraheze.org' expires in 15 day(s) (Wed 30 Oct 2019 07:16:33 PM GMT +0000). [19:27:19] [02puppet] 07paladox closed pull request 03#1100: varnish: Rate limit amazon bot in nginx - 13https://git.io/Je8zH [19:27:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8gE [19:27:22] [02miraheze/puppet] 07paladox 0359ba010 - varnish: Rate limit amazon bot in nginx (#1100) * varnish: Rate limit amazon bot in nginx * Update mediawiki.conf [19:27:23] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [19:27:25] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [19:31:38] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:37:25] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:40:06] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [19:40:49] PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 1 failures [19:43:04] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [19:43:12] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [19:44:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:44:59] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:45:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8g5 [19:45:37] [02miraheze/puppet] 07paladox 03ab8ebbd - varnish: Fix rate limit in nginx config [19:46:23] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [19:46:35] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [19:46:35] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [19:46:46] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [19:47:00] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [19:47:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:47:55] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:49:09] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Je8gb [19:49:11] [02miraheze/puppet] 07paladox 034ca5916 - Revert "varnish: Fix rate limit in nginx config" This reverts commit ab8ebbdcd326f89bd0ef52bd5c4d82c8ed09cf83. [19:49:12] [02miraheze/puppet] 07paladox 032f67283 - Revert "varnish: Rate limit amazon bot in nginx (#1100)" This reverts commit 59ba010b818e2ab799c72c72c0d8c8b65c1c96a8. [19:49:24] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:24:52] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:25:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [20:28:49] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [20:29:08] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:29:30] [02mw-config] 07Pix1234 opened pull request 03#2773: Add AutoCreatePage ext (T4792) - 13https://git.io/Je82w [20:29:45] ^ looking [20:31:31] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [20:31:55] !log rhinos@test1:/srv/mediawiki/config$ sudo -u www-data git fetch --all [20:32:28] !log rhinos@test1:/srv/mediawiki/config$ sudo -u www-data git reset --hard origin/master [20:33:06] !log rhinos@test1:/srv/mediawiki/config$ sudo -u www-data git pull [20:35:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:36:18] PROBLEM - wiki.exnihilolinux.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.exnihilolinux.org' expires in 15 day(s) (Wed 30 Oct 2019 08:28:25 PM GMT +0000). [20:36:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je82D [20:36:34] [02miraheze/ssl] 07MirahezeSSLBot 03dae8dc7 - Bot: Update SSL cert for wiki.exnihilolinux.org [20:36:36] [02mw-config] 07Pix1234 synchronize pull request 03#2773: Add AutoCreatePage ext (T4792) - 13https://git.io/Je82w [20:36:52] PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 1 failures [20:37:22] paladox: i'm about to deploy onto test1!!! [20:37:29] oh [20:37:34] RhinosF1 you still can [20:37:38] just puppet is disabled [20:37:46] I need to do a emergency rate limit test [20:38:14] paladox: ideally with puppet would be easier but I guess I'll manually do it [20:38:15] PROBLEM - www.lab612.at - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.lab612.at' expires in 15 day(s) (Wed 30 Oct 2019 08:31:20 PM GMT +0000). [20:38:29] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:38:30] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je82S [20:38:31] [02miraheze/ssl] 07MirahezeSSLBot 03b95d8b8 - Bot: Update SSL cert for www.lab612.at [20:38:32] You should always try it manually [20:38:41] otherwise it's spamming the repo :P [20:38:54] paladox: but next time tell me as I was in the middle of forcing puppet to run to clear icinga [20:39:41] oh [20:39:50] ok, sorry. [20:40:22] [02mw-config] 07RhinosF1 synchronize pull request 03#2773: Add AutoCreatePage ext (T4792) - 13https://git.io/Je82w [20:40:39] PROBLEM - ndg.nenawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nenawiki.org' expires in 15 day(s) (Wed 30 Oct 2019 08:34:51 PM GMT +0000). [20:40:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je82H [20:40:54] [02miraheze/ssl] 07MirahezeSSLBot 034c372d3 - Bot: Update SSL cert for ndg.nenawiki.org [20:41:04] PROBLEM - nenawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nenawiki.org' expires in 15 day(s) (Wed 30 Oct 2019 08:34:51 PM GMT +0000). [20:41:19] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je82Q [20:41:21] [02miraheze/ssl] 07MirahezeSSLBot 0376880e4 - Bot: Update SSL cert for nenawiki.org [20:41:31] [02mw-config] 07RhinosF1 synchronize pull request 03#2773: Add AutoCreatePage ext (T4792) - 13https://git.io/Je82w [20:43:13] [02mw-config] 07RhinosF1 closed pull request 03#2773: Add AutoCreatePage ext (T4792) - 13https://git.io/Je82w [20:43:13] RECOVERY - ndg.nenawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nenawiki.org' will expire on Sun 12 Jan 2020 07:41:13 PM GMT +0000. [20:43:14] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Je82F [20:43:16] [02miraheze/mw-config] 07Pix1234 038ed5403 - Add AutoCreatePage ext (T4792) (#2773) [20:43:30] RECOVERY - nenawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nenawiki.org' will expire on Sun 12 Jan 2020 07:41:13 PM GMT +0000. [20:43:36] RECOVERY - www.lab612.at - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lab612.at' will expire on Sun 12 Jan 2020 07:38:24 PM GMT +0000. [20:43:40] PROBLEM - www.sdiy.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sdiy.info' expires in 15 day(s) (Wed 30 Oct 2019 08:39:51 PM GMT +0000). [20:44:11] RECOVERY - wiki.exnihilolinux.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.exnihilolinux.org' will expire on Sun 12 Jan 2020 07:36:27 PM GMT +0000. [20:44:22] PROBLEM - sdiy.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sdiy.info' expires in 15 day(s) (Wed 30 Oct 2019 08:39:51 PM GMT +0000). [20:44:30] !log rhinos@test1:/srv/mediawiki/config$ sudo -u www-data git pull [20:44:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:44:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je82A [20:44:39] [02miraheze/ssl] 07MirahezeSSLBot 03f834876 - Bot: Update SSL cert for sdiy.info [20:45:26] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki (test1) [20:45:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:46:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je82p [20:46:11] [02miraheze/puppet] 07paladox 0333cb7e2 - varnish: Rate limit user agents that match the list [20:46:33] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je82j [20:46:35] [02miraheze/mw-config] 07RhinosF1 03ba71990 - fix [20:47:02] !log rerun last two due to error [20:47:03] ZppixBot: [20:47:05] Zppix: [20:48:02] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on test1 [20:48:09] PROBLEM - www.programming.red - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'programming.red' expires in 15 day(s) (Wed 30 Oct 2019 08:43:04 PM GMT +0000). [20:48:13] Zppix: you should be able to test now [20:48:50] PROBLEM - programming.red - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'programming.red' expires in 15 day(s) (Wed 30 Oct 2019 08:43:04 PM GMT +0000). [20:49:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8ae [20:49:07] [02miraheze/ssl] 07MirahezeSSLBot 0323b38b5 - Bot: Update SSL cert for programming.red [20:50:26] !log last two on all mw* servers and git pulling config on mw3 due to puppet killing itself [20:50:54] paladox: [20:50:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:50:59] yup? [20:51:04] please tell me that's you breaking test1 [20:51:14] paladox: http error 500 [20:51:37] RhinosF1: bit hard to test on test1 atm [20:51:41] nope [20:51:46] all i did was a config [20:51:49] on nginx [20:52:26] * Zppix slowly backs away [20:52:26] paladox: test1 is down so it's either your config or AutoCreatePages? [20:52:37] It'll have to be the later [20:53:10] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [20:53:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:53:19] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw3 [20:53:20] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [20:53:51] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8aL [20:53:53] [02miraheze/mw-config] 07RhinosF1 035c3f634 - http 500 [20:54:58] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:55:16] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:55:45] paladox: we're back so why did that break us? [20:56:10] I'm not sure [20:56:11] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:56:16] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [20:56:51] paladox: i'm getting file can't be opened trying to view nginx logs so what do they say [20:57:04] I am busy at the moment [20:57:07] so cannot look [20:57:13] sorry :( [20:57:51] JohnLewis: ^ [20:58:57] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8a3 [20:58:58] [02miraheze/mw-config] 07RhinosF1 03606e0e8 - fix indentation [20:59:22] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [20:59:38] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 5.529 second response time [20:59:51] Zppix: that might of being you - blame CI [20:59:59] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [21:00:14] hey Examknow [21:00:27] there's nothing really ever useful in nginx logs from a mw-admins perspective [21:00:28] hey [21:01:06] JohnLewis: hmm, we're no longer 500 on test1 the extension still isn't in Special:Version [21:02:00] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:02:09] * RhinosF1 makes sure he's fixed and unreverted everything [21:02:37] RhinosF1: /var/log/mediawiki/debuglogs/ are your friends [21:02:39] esp the php one [21:02:49] JohnLewis: ok [21:02:56] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8al [21:02:58] [02miraheze/mw-config] 07RhinosF1 03e9ca11e - Update LocalSettings.php [21:03:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8a8 [21:03:01] [02miraheze/puppet] 07paladox 03cdb6da3 - Fix [21:04:24] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [21:04:29] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [21:04:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:04:35] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [21:05:32] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.633 second response time [21:05:44] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:05:57] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.009 second response time [21:06:13] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:06:14] JohnLewis: look at exception.log in /var/log/mediawiki/debuglogs [21:06:29] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:09:04] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8a2 [21:09:05] [02miraheze/mw-config] 07RhinosF1 03e4f061a - Update LocalSettings.php [21:09:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8aa [21:09:20] [02miraheze/puppet] 07paladox 03b50a24f - fix [21:10:23] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sun 12 Jan 2020 07:44:31 PM GMT +0000. [21:11:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8a6 [21:11:25] [02miraheze/puppet] 07paladox 03d78a7e5 - mediawiki: Disable readahead cache [21:11:36] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:11:47] RECOVERY - www.programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Sun 12 Jan 2020 07:48:59 PM GMT +0000. [21:12:11] RECOVERY - programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Sun 12 Jan 2020 07:48:59 PM GMT +0000. [21:12:34] RECOVERY - www.sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sun 12 Jan 2020 07:44:31 PM GMT +0000. [21:14:32] !log reboot misc4 [21:15:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:16:16]