[02:00:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8Kg [02:00:12] [02miraheze/services] 07MirahezeSSLBot 0389c4d60 - BOT: Updating services config for wikis [02:42:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:42:39] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [02:43:13] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [02:43:42] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:45:06] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:45:16] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [02:46:02] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [02:46:33] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:51:02] /cs flags #miraheze [02:51:05] ohp [03:45:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je86Z [03:45:10] [02miraheze/services] 07MirahezeSSLBot 0396d8f41 - BOT: Updating services config for wikis [04:37:20] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 4 minutes ago with 8 failures. Failed resources (up to 3 shown) [04:39:35] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 6 minutes ago with 8 failures. Failed resources (up to 3 shown) [04:40:14] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 6 minutes ago with 8 failures. Failed resources (up to 3 shown) [04:40:16] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 98 failures. Last run 6 minutes ago with 98 failures. Failed resources (up to 3 shown) [04:40:26] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 175 failures. Last run 6 minutes ago with 175 failures. Failed resources (up to 3 shown): File[stablestate.org],File[stablestate.org_private],File[wiki.joust.ro],File[wiki.joust.ro_private] [04:45:24] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [04:45:51] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:45:51] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [04:45:52] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:45:53] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [11:38:06] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_phabricator-extensions] [11:43:07] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [11:50:22] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:55:21] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:17:06] PROBLEM - en.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:17:06] PROBLEM - or.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:17:27] PROBLEM - commons.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:17:29] PROBLEM - la.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:17:33] PROBLEM - mg.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:17:33] PROBLEM - mr.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:17:36] PROBLEM - ml.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:17:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je859 [15:17:41] PROBLEM - hif.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:17:42] [02miraheze/ssl] 07MirahezeSSLBot 034cd831a - Bot: Update SSL cert for en.gyaanipedia.co.in [15:17:58] PROBLEM - sa.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:18:05] PROBLEM - meta.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:18:07] PROBLEM - pa.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:18:08] PROBLEM - mai.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:18:32] PROBLEM - ta.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:18:37] PROBLEM - te.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:18:54] PROBLEM - bn.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:19:03] PROBLEM - hi.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:19:16] PROBLEM - kn.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:19:17] PROBLEM - gu.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:19:23] PROBLEM - fa.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Thu 31 Oct 2019 03:14:12 PM GMT +0000). [15:22:59] RECOVERY - meta.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:23:07] RECOVERY - mai.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:23:07] RECOVERY - pa.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:23:36] RECOVERY - ta.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:23:43] RECOVERY - te.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:23:52] RECOVERY - bn.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:24:03] RECOVERY - hi.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:24:14] RECOVERY - kn.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:24:15] RECOVERY - gu.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:24:16] RECOVERY - fa.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:24:23] RECOVERY - en.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:24:24] RECOVERY - or.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:24:48] RECOVERY - la.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:24:48] RECOVERY - commons.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:24:58] RECOVERY - mg.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:24:58] RECOVERY - mr.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:24:59] RECOVERY - ml.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:25:05] RECOVERY - hif.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [15:25:06] RECOVERY - sa.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000. [16:25:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8F0 [16:25:28] [02miraheze/puppet] 07paladox 03f9188b7 - prometheus: Fix directory to save for gdnsd [17:02:16] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8bL [17:02:17] [02miraheze/puppet] 07Southparkfan 03a1840e8 - Add zppix to icinga [17:03:30] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8bO [17:03:31] [02miraheze/puppet] 07Southparkfan 038c3d2b2 - Mail alias for zppix [17:04:06] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8bG [17:04:07] [02miraheze/puppet] 07Southparkfan 030c5f93f - add zppix go icingaweb2 group [17:26:30] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8bX [17:26:32] [02miraheze/puppet] 07Southparkfan 033dd86b0 - Add zppix to shell accounts [17:30:08] !log run MatomoAnalyticsHooks::wikiCreation( 'mlgyaanipediawiki' ); in eval.php on mw1 [17:30:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:32:23] PROBLEM - www.modesofdiscourse.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.modesofdiscourse.com' expires in 15 day(s) (Thu 31 Oct 2019 06:27:49 PM GMT +0000). [18:32:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8AU [18:32:38] [02miraheze/ssl] 07MirahezeSSLBot 034436c0d - Bot: Update SSL cert for www.modesofdiscourse.com [18:34:14] !log upgrade sudo on mw[123] [18:34:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:35:03] !log upgrade sudo on db[45] [18:35:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:36:05] !log upgrade sudo on misc[1234] [18:36:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:38:57] !log upgrade sudo on cp[234] [18:39:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:41:07] !log upgrade sudo on lizardfs[45] [18:41:11] !log upgrade sudo on bacula1 [18:41:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:41:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:41:47] !log upgrade sudo on test1 [18:41:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:42:48] RECOVERY - www.modesofdiscourse.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.modesofdiscourse.com' will expire on Mon 13 Jan 2020 05:32:31 PM GMT +0000. [18:43:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8AR [18:43:55] [02miraheze/puppet] 07paladox 03566ee63 - Update storage.pp [18:44:27] !log zppix@test1:/$ sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --update --wiki test1wiki [18:44:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:06:19] [02puppet] 07Pix1234 opened pull request 03#1101: modify my email for cvt - 13https://git.io/Je8AQ [19:07:19] [02puppet] 07paladox closed pull request 03#1101: modify my email for cvt - 13https://git.io/Je8AQ [19:07:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8A5 [19:07:22] [02miraheze/puppet] 07Pix1234 039b54497 - modify my email for cvt (#1101) [19:12:47] PROBLEM - www.jawright.cc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.jawright.cc' expires in 15 day(s) (Thu 31 Oct 2019 07:08:54 PM GMT +0000). [19:13:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8Aj [19:13:05] [02miraheze/ssl] 07MirahezeSSLBot 03360de83 - Bot: Update SSL cert for www.jawright.cc [19:13:52] PROBLEM - jawright.cc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.jawright.cc' expires in 15 day(s) (Thu 31 Oct 2019 07:08:54 PM GMT +0000). [19:24:00] RECOVERY - jawright.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'www.jawright.cc' will expire on Mon 13 Jan 2020 06:12:57 PM GMT +0000. [19:24:58] RECOVERY - www.jawright.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'www.jawright.cc' will expire on Mon 13 Jan 2020 06:12:57 PM GMT +0000. [19:52:56] [Announcement]: An appeal process under the Username Policy will shortly be set up and the No Open Proxies policy now indicates that the community advises a 6 month long block duration with discretion advised. A discussion on the definition of Harrasment and related terms will be opened shortly. [20:00:07] no [20:00:23] RhinosF1: [20:01:24] RhinosF1: can I not read or? [20:01:29] Zppix: [20:02:40] idk [20:02:59] Why you say no then? [20:03:05] idk [20:03:36] I don’t either [20:12:52] PROBLEM - commonswiki.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expires in 7 day(s) (Wed 23 Oct 2019 08:12:13 PM GMT +0000). [20:12:52] PROBLEM - hellointernet.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expires in 7 day(s) (Wed 23 Oct 2019 08:12:13 PM GMT +0000). [20:13:12] PROBLEM - cnt.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expires in 7 day(s) (Wed 23 Oct 2019 08:12:13 PM GMT +0000). [20:13:13] PROBLEM - intp.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expires in 7 day(s) (Wed 23 Oct 2019 08:12:13 PM GMT +0000). [20:13:16] PROBLEM - sims.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expires in 7 day(s) (Wed 23 Oct 2019 08:12:13 PM GMT +0000). [20:13:17] PROBLEM - marthaspeaks.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expires in 7 day(s) (Wed 23 Oct 2019 08:12:13 PM GMT +0000). [20:13:43] PROBLEM - unmade.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expires in 7 day(s) (Wed 23 Oct 2019 08:12:13 PM GMT +0000). [20:14:26] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expires in 7 day(s) (Wed 23 Oct 2019 08:12:13 PM GMT +0000). [20:14:28] PROBLEM - tallguysfree.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expires in 7 day(s) (Wed 23 Oct 2019 08:12:13 PM GMT +0000). [20:14:34] PROBLEM - studynotekr.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expires in 7 day(s) (Wed 23 Oct 2019 08:12:13 PM GMT +0000). [20:14:39] PROBLEM - piwik.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expires in 7 day(s) (Wed 23 Oct 2019 08:12:13 PM GMT +0000). [20:14:52] that don't look good [20:15:35] Voidwalker: it doesn't :( [20:49:06] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [20:49:07] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je8pr [20:49:09] [02miraheze/mw-config] 07paladox 0368befae - Match using === rather then in_array [20:49:10] [02mw-config] 07paladox opened pull request 03#2775: Match using === rather then in_array - 13https://git.io/Je8po [20:50:29] [02mw-config] 07paladox closed pull request 03#2775: Match using === rather then in_array - 13https://git.io/Je8po [20:50:30] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8pM [20:50:31] [02miraheze/mw-config] 07paladox 03d8b46b6 - Match using === rather then in_array (#2775) [20:55:57] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8pp [20:55:58] [02miraheze/mw-config] 07Southparkfan 036a242cd - Fix in_array order [20:59:39] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8hk [20:59:41] [02miraheze/mw-config] 07paladox 039f49d27 - animeshoes -> animeshoeswiki [21:00:26] hey eir [21:00:28] Examknow: [21:02:44] RhinosF1: eir is a freenode bot that does not talk [21:03:18] Examknow: I mis-pinged you [21:04:07] ok [21:04:59] Examknow: I'm about to launch a new RfC. Following implementing the last one. [21:05:08] which I just did [21:05:14] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8h3 [21:05:16] [02miraheze/mw-config] 07Southparkfan 03f0cb921 - Enable profiling on test1 [21:08:43] [NEW RfC] https://meta.miraheze.org/wiki/Requests_for_Comment/Harassment [21:08:44] [ Requests for Comment/Harassment - Miraheze Meta ] - meta.miraheze.org [21:19:28] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8hV [21:19:29] [02miraheze/mw-config] 07Southparkfan 0330d7e10 - Add extra wikis for InstantCommons caching + increase expiration [21:24:37] Examknow: your channel doesn't let me rejoin or self invite [21:24:44] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:24:46] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:24:46] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [21:24:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:24:59] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3 [21:25:00] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [21:28:10] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:28:23] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 5.776 second response time [21:31:05] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.006 second response time [21:31:34] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [21:31:36] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [21:33:19] [02miraheze/mediawiki] 07Pix1234 pushed 031 commit to 03REL1_33 [+1/-0/±1] 13https://git.io/Je8jv [21:33:21] [02miraheze/mediawiki] 07Pix1234 034fd9ebc - add Last Modified ext [21:35:53] Zppix: ^ where's the PR for config? [21:36:25] and the capital letters are missing [21:36:45] fixing the capital [21:36:51] and see your pms RhinosF1 [21:37:01] (submodules are a new concept for me) [21:37:41] Zppix: yes but I can review a PR that isn't there [21:37:45] can't [21:37:56] ok [21:38:28] i can send you the diff RhinosF1 ? [21:38:50] Zppix: shouldn't it be on GitHub though? [21:39:05] RhinosF1: i cloned the repo and did my edits locally [21:39:29] Zppix: how are you going to merge though? Open a PR? [21:39:38] i was just going to push [21:39:50] ah [21:41:16] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:41:36] Zppix: send me the diff then [21:43:53] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.408 second response time [21:43:59] [02miraheze/mediawiki] 07Pix1234 pushed 032 commits to 03REL1_33 [+1/-1/±2] 13https://git.io/Je8j3 [21:44:01] [02miraheze/mediawiki] 07Pix1234 03ab2874b - removed ext to fix capitalization [21:44:02] [02miraheze/mediawiki] 07Pix1234 03650ed5b - readd Last Modified [21:44:25] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [21:44:28] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:44:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:45:12] Zppix: looks good [21:46:42] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Je8jZ [21:46:43] [02miraheze/mw-config] 07Pix1234 030d00de9 - add last modified to test1 [21:47:37] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8jn [21:47:38] [02miraheze/mw-config] 07RhinosF1 03c9c585f - fix sp [21:48:32] !log sudo -u www-data git pull in /srv/mediawiki/config and /w/ on test1 [21:48:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:50:45] * RhinosF1 knows about test1 [21:53:09] for some reason metawiki is served to me by cp4 and cp2 at the same time? [21:53:32] SPF|Cloud: double the caching the better :P ? [21:54:40] yeah, cp2 is serving me [21:54:57] [02miraheze/mw-config] 07Pix1234 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Je8j8 [21:54:59] [02miraheze/mw-config] 07Pix1234 03bd8a246 - replace wfloadextension with require_once [21:55:00] [02miraheze/mw-config] 07Pix1234 03cd5c7cd - Merge branch 'master' of https://github.com/miraheze/mw-config [21:55:01] [ GitHub - miraheze/mw-config: Production MediaWiki configuration ] - github.com [21:57:35] !log git pull on test1 for /config [21:58:00] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:59:24] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [21:59:46] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [22:01:06] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [22:01:08] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:01:12] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 3.476 second response time [22:01:28] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [22:03:05] !log depool mw1 [22:03:15] !log enable debug log level on php-fpm (mw1) [22:04:47] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [22:05:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:06:01] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [22:06:23] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [22:06:32] !log rhinos@test1:/srv/mediawiki/config$ sudo -u www-data rm -rf /srv/mediawiki/w/extensions/lastmodified [22:06:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:07:24] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:08:16] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [22:10:15] !log rhinos@test1:/srv/mediawiki/config$ sudo -u www-data git reset --hard [22:10:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:11:43] !log rhinos@test1:/srv/mediawiki/config$ sudo -u www-data rm -rf /srv/mediawiki/w/extensions/LastModified [22:11:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:13:51] !log rhinos@test1:/srv/mediawiki/w/extensions$ sudo -u www-data git reset --hard [22:13:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:16:45] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8jX [22:16:47] [02miraheze/mw-config] 07RhinosF1 03fa79729 - Update LocalSettings.php [22:20:04] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [22:20:09] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [22:21:03] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/Je8jS [22:21:05] [02miraheze/mediawiki] 07RhinosF1 03fb1c6b4 - attempt to fix [22:21:53] !log rhinos@test1:/srv/mediawiki/w$ sudo -u www-data git pull [22:21:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:22:39] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [22:22:42] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:23:59] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je8j5 [22:24:01] [02miraheze/mw-config] 07RhinosF1 0347f5b5d - re enable on test1 after fix [22:24:41] !log rhinos@test1:/srv/mediawiki/config$ sudo -u www-data git pull [22:24:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:27:54] !log rebuildlc on test1 [22:27:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:37:11] !log rebuildlc on test1 [22:37:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:39:55] !log with --wiki test1wiki [22:40:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:43:28] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:44:16] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:44:16] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [22:44:24] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw3 [22:45:00] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:45:04] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [22:45:29] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:45:43] .status mhtest offline [22:45:54] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [22:47:11] .status mhtest offline [22:47:16] RhinosF1 updating User:RhinosF1/Status! [22:47:33] RhinosF1: Done! [22:47:49] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.005 second response time [22:47:56] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [22:48:21] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.390 second response time [22:48:38] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.687 second response time [22:48:51] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [22:49:15] [02miraheze/dns] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4eO [22:49:17] [02miraheze/dns] 07Southparkfan 03feb2a1d - Add lizardfs6 [22:49:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:50:26] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [22:56:55] central notice should go live in about an hour [22:57:02] (for the RfC) [22:57:02] PROBLEM - lizardfs6 Disk Space on lizardfs6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:57:19] \o/ [22:57:21] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:58:00] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:59:37] PROBLEM - Host lizardfs6 is DOWN: PING CRITICAL - Packet loss = 100% [23:01:37] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:21:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/Je4eg [23:21:25] [02miraheze/puppet] 07paladox 03fbd0d18 - Pool in lizardfs6 as it's own chunkserver and master [23:21:27] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [23:21:28] [02puppet] 07paladox opened pull request 03#1102: Pool in lizardfs6 as it's own chunkserver and master - 13https://git.io/Je4e2 [23:22:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je4ew [23:22:36] [02miraheze/puppet] 07paladox 03c0fbbfa - Update site.pp [23:22:37] [02puppet] 07paladox synchronize pull request 03#1102: Pool in lizardfs6 as it's own chunkserver and master - 13https://git.io/Je4e2 [23:24:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je4eo [23:24:09] [02miraheze/puppet] 07paladox 03918ad9e - Update storage_firewall.yaml [23:24:10] [02puppet] 07paladox synchronize pull request 03#1102: Pool in lizardfs6 as it's own chunkserver and master - 13https://git.io/Je4e2 [23:24:21] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Je4eK [23:24:23] [02miraheze/mw-config] 07Pix1234 0347bf278 - prepare to move lastmodified to prod [23:24:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je4e6 [23:24:51] [02miraheze/puppet] 07paladox 03f80345f - Update lizardfs6.yaml [23:24:52] [02puppet] 07paladox synchronize pull request 03#1102: Pool in lizardfs6 as it's own chunkserver and master - 13https://git.io/Je4e2 [23:25:26] [02puppet] 07paladox closed pull request 03#1102: Pool in lizardfs6 as it's own chunkserver and master - 13https://git.io/Je4e2 [23:25:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/Je4eP [23:25:29] [02miraheze/puppet] 07paladox 0373b822d - Pool in lizardfs6 as it's own chunkserver and master (#1102) * Pool in lizardfs6 as it's own chunkserver and master * Update site.pp * Update storage_firewall.yaml * Update lizardfs6.yaml [23:25:30] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [23:25:32] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [23:26:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4eX [23:26:59] [02miraheze/puppet] 07paladox 031aa34af - Update storage.pp [23:28:38] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki and sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw1 [23:28:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:31:06] !log same on mw[23] [23:31:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:31:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4eD [23:31:23] [02miraheze/puppet] 07paladox 0336b8f19 - Update mediawiki.pp [23:35:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4eQ [23:35:09] [02miraheze/services] 07MirahezeSSLBot 03e88bae9 - BOT: Updating services config for wikis [23:36:49] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4ed [23:36:51] [02miraheze/dns] 07paladox 033ad724a - Add static-temp [23:37:56] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [23:37:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je4eb [23:37:59] [02miraheze/puppet] 07paladox 0395b5495 - Add static-temp [23:38:08] [02puppet] 07paladox opened pull request 03#1103: Add static-temp - 13https://git.io/Je4eN [23:39:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je4ex [23:39:13] [02miraheze/puppet] 07paladox 037e72b11 - Update default.vcl [23:39:15] [02puppet] 07paladox synchronize pull request 03#1103: Add static-temp - 13https://git.io/Je4eN [23:39:21] [02puppet] 07paladox closed pull request 03#1103: Add static-temp - 13https://git.io/Je4eN [23:39:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Je4ep [23:39:24] [02miraheze/puppet] 07paladox 03df07b5b - Add static-temp (#1103) * Add static-temp * Update default.vcl [23:41:02] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [23:41:03] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [23:41:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4ej [23:41:51] [02miraheze/puppet] 07paladox 032b455e1 - Update storage.pp [23:44:36] !log install rsync on lizardfs4 [23:46:20] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw2 mw3 [23:46:43] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [23:47:55] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:48:15] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:48:17] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [23:48:34] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3 [23:49:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:53:33] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.683 second response time [23:53:42] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.389 second response time [23:55:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4vq [23:55:37] [02miraheze/mw-config] 07paladox 0326b04df - Migrate metawiki to /mnt/mediawiki-static-new