[00:02:27] PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:02:41] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:02:47] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:19] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:35] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:01] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:03] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:07] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:09] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_31 [+0/-0/±2] 13https://git.io/fpg9j [00:04:10] [02miraheze/mediawiki] 07JohnFLewis 03f51ed22 - update CW and MW [00:04:11] ^^ risky deployment :P [00:04:13] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:15] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:17] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:24] JohnLewis puppet disable on all mw* [00:04:28] then enable it on mw1 :P [00:04:45] nah [00:04:48] it should be fine [00:05:12] then I can rm the old permissions files [00:05:25] cause now we use $wiki-$module style instead of $module-$wiki :P [00:05:45] heh [00:10:09] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:11:20] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [00:12:04] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:06] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:08] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:12] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:14] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [00:12:16] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [00:12:18] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:26] RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:42] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:48] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:13:38] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:14:02] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [00:14:47] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_31 [+0/-0/±2] 13https://git.io/fpgHa [00:14:49] [02miraheze/mediawiki] 07JohnFLewis 036da6600 - Revert "update CW and MW" This reverts commit f51ed22ce41507ab3c6076d8a5423ee346b3f970. [00:17:04] hello? [00:17:09] hi [00:17:46] i cant get on the website [00:17:59] but i can go on the wikis [00:18:00] what do you mean by you "i cant get on the website"? [00:18:08] meta [00:18:10] you cannot access https://miraheze.org? [00:18:11] Title: [ Miraheze ] - miraheze.org [00:18:19] or https://meta.miraheze.org/ [00:18:20] Title: [ Meta ] - meta.miraheze.org [00:18:20] ? [00:18:48] meta.miraheze [00:19:00] what error do you get? [00:19:39] nvm [00:19:42] its working now [00:20:38] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpgHN [00:20:39] [02miraheze/ManageWiki] 07JohnFLewis 038f1b925 - add in missing foreach [00:21:14] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_31 [+0/-0/±2] 13https://git.io/fpgHj [00:21:16] [02miraheze/mediawiki] 07JohnFLewis 03eac9679 - update MW/CW [02:15:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpgNY [02:15:14] [02miraheze/services] 07MirahezeSSLBot 0338ab71b - BOT: Updating services config for wikis [02:35:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpgNh [02:35:13] [02miraheze/services] 07MirahezeSSLBot 03c97547b - BOT: Updating services config for wikis [04:42:57] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [05:45:35] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [06:20:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp2JL [06:20:13] [02miraheze/services] 07MirahezeSSLBot 0366d7f63 - BOT: Updating services config for wikis [06:41:40] Hello Reguyla__! If you have any questions feel free to ask and someone should answer soon. [09:37:15] HTTP CRITICAL - Unable to open TCP socket [09:52:19] Hello wm-bot564! If you have any questions feel free to ask and someone should answer soon. [11:20:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp2lJ [11:20:16] [02miraheze/services] 07MirahezeSSLBot 03e53dbdd - BOT: Updating services config for wikis [13:28:57] PROBLEM - astrapedia.ru - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'astrapedia.ru' expires in 15 day(s) (Fri 14 Dec 2018 01:26:46 PM GMT +0000). [15:35:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp2QE [15:35:16] [02miraheze/services] 07MirahezeSSLBot 03379afa6 - BOT: Updating services config for wikis [17:54:10] !log delete from page WHERE page_title LIKE 'NonDizionario:%' and page_namespace=0; on db4 [17:54:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:11:58] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [18:12:00] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fpaI4 [18:12:01] [02miraheze/ssl] 07paladox 03aa2cdc7 - Remove taotac.info domain [18:12:03] [02ssl] 07paladox opened pull request 03#98: Remove taotac.info domain - 13https://git.io/fpaI0 [18:12:16] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/fpaIg [18:12:17] [02miraheze/ssl] 07paladox 03721af7b - Delete taotac.info.crt [18:12:19] [02ssl] 07paladox synchronize pull request 03#98: Remove taotac.info domain - 13https://git.io/fpaI0 [18:12:47] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fpaI6 [18:12:49] [02miraheze/ssl] 07paladox 034f7a741 - Update redirects.yaml [18:12:51] [02ssl] 07paladox synchronize pull request 03#98: Remove taotac.info domain - 13https://git.io/fpaI0 [18:12:57] [02ssl] 07paladox closed pull request 03#98: Remove taotac.info domain - 13https://git.io/fpaI0 [18:12:59] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±2] 13https://git.io/fpaIi [18:13:00] [02miraheze/ssl] 07paladox 03abcb9e4 - Remove taotac.info domain (#98) * Remove taotac.info domain * Delete taotac.info.crt * Update redirects.yaml [18:13:02] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [18:13:03] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [18:15:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpaIp [18:15:16] [02miraheze/services] 07MirahezeSSLBot 03dddbd67 - BOT: Updating services config for wikis [18:22:24] HTTP CRITICAL - Unable to open TCP socket [18:22:29] uh metas down [18:24:09] paladox: what happened? [18:25:52] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [18:26:55] PROBLEM - misc1 Current Load on misc1 is CRITICAL: CRITICAL - load average: 1.33, 2.05, 1.06 [18:27:06] HTTP CRITICAL - Unable to open TCP socket [18:27:17] RECOVERY - wiki.tensorflow.community - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tensorflow.community' will expire on Sun 13 Jan 2019 02:14:29 PM GMT +0000. [18:27:19] RECOVERY - decrypted.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'decrypted.wiki' will expire on Thu 27 Dec 2018 07:16:36 PM GMT +0000. [18:27:20] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18134 bytes in 0.023 second response time [18:27:24] RECOVERY - studentwiki.ddns.net - LetsEncrypt on sslhost is OK: OK - Certificate 'studentwiki.ddns.net' will expire on Sun 17 Feb 2019 12:32:04 PM GMT +0000. [18:27:32] RECOVERY - wiki.mikrodev.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mikrodev.com' will expire on Wed 16 Jan 2019 05:01:31 PM GMT +0000. [18:27:33] RECOVERY - wiki.lostminecraftminers.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lostminecraftminers.org' will expire on Thu 03 Jan 2019 10:51:19 AM GMT +0000. [18:27:35] RECOVERY - www.sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Fri 22 Feb 2019 11:44:22 PM GMT +0000. [18:27:36] RECOVERY - wiki.besuccess.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.besuccess.com' will expire on Mon 25 Feb 2019 10:08:45 PM GMT +0000. [18:27:38] RECOVERY - bebaskanpengetahuan.org - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.org' will expire on Thu 03 Jan 2019 09:47:53 AM GMT +0000. [18:27:40] +0000. [18:27:43] RECOVERY - marthaspeaks.miraheze.org - GlobalSign on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [18:27:44] RECOVERY - orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 22 Feb 2019 06:35:03 PM GMT +0000. [18:27:49] RECOVERY - wikiescola.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiescola.com.br' will expire on Fri 22 Feb 2019 11:16:00 PM GMT +0000. [18:27:51] RECOVERY - wiki.taotac.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.taotac.org' will expire on Sat 15 Dec 2018 07:37:11 AM GMT +0000. [18:27:53] RECOVERY - athenapedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'athenapedia.org' will expire on Sat 26 Jan 2019 12:26:32 PM GMT +0000. [18:27:55] RECOVERY - wiki.bonessdiving.club - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.bonessdiving.club' will expire on Wed 20 Feb 2019 03:06:21 AM GMT +0000. [18:27:56] RECOVERY - wiki.campaign-labour.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.campaign-labour.org' will expire on Mon 28 Jan 2019 07:38:45 PM GMT +0000. [18:27:59] RECOVERY - wikipuk.cl - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipuk.cl' will expire on Mon 25 Feb 2019 09:46:36 PM GMT +0000. [18:28:01] RECOVERY - www.mywiki.icu - LetsEncrypt on sslhost is OK: OK - Certificate 'www.mywiki.icu' will expire on Mon 24 Dec 2018 04:42:58 PM GMT +0000. [18:28:03] RECOVERY - wiki.grottocenter.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.grottocenter.org' will expire on Sat 23 Feb 2019 11:08:27 PM GMT +0000. [18:28:05] RECOVERY - monarchists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'monarchists.wiki' will expire on Wed 23 Jan 2019 08:31:31 AM GMT +0000. [18:28:07] RECOVERY - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.marinebiodiversitymatrix.org' will expire on Sun 20 Jan 2019 05:59:29 AM GMT +0000. [18:28:09] on Tue 29 Jan 2019 03:30:57 PM GMT +0000. [18:28:11] RECOVERY - holonet.pw - LetsEncrypt on sslhost is OK: OK - Certificate 'holonet.pw' will expire on Wed 20 Feb 2019 10:28:15 AM GMT +0000. [18:28:13] RECOVERY - saveta.org - LetsEncrypt on sslhost is OK: OK - Certificate 'saveta.org' will expire on Fri 22 Feb 2019 11:58:29 PM GMT +0000. [18:28:15] RECOVERY - exocortex.sillyendpoints.com - LetsEncrypt on sslhost is OK: OK - Certificate 'exocortex.sillyendpoints.com' will expire on Sun 24 Feb 2019 11:42:42 PM GMT +0000. [18:28:17] RECOVERY - spiral.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spiral.wiki' will expire on Sat 23 Feb 2019 10:43:09 PM GMT +0000. [18:28:19] RECOVERY - wikiverte.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiverte.pl' will expire on Tue 08 Jan 2019 08:06:03 PM GMT +0000. [18:28:21] 00. [18:28:23] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [18:28:27] RECOVERY - wiki.kourouklides.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kourouklides.com' will expire on Sat 16 Feb 2019 01:54:13 PM GMT +0000. [18:28:29] RECOVERY - savage-wiki.com - RapidSSL on sslhost is OK: OK - Certificate 'savage-wiki.com' will expire on Fri 04 Dec 2020 12:00:00 PM GMT +0000. [18:28:31] RECOVERY - disabled.life - LetsEncrypt on sslhost is OK: OK - Certificate 'disabled.life' will expire on Mon 25 Feb 2019 10:14:43 PM GMT +0000. [18:28:33] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18135 bytes in 0.097 second response time [18:28:35] RECOVERY - reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000. [18:28:37] RECOVERY - wiki.ldmsys.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ldmsys.net' will expire on Sat 16 Feb 2019 01:55:40 PM GMT +0000. [18:28:39] RECOVERY - www.eerstelijnszones.be - LetsEncrypt on sslhost is OK: OK - Certificate 'eerstelijnszones.be' will expire on Sat 23 Feb 2019 12:11:01 AM GMT +0000. [18:28:41] RECOVERY - wiki.macc.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Tue 01 Jan 2019 08:31:01 PM GMT +0000. [18:28:43] RECOVERY - en.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 08 Feb 2019 09:02:37 AM GMT +0000. [18:28:45] RECOVERY - te.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'te.gyaanipedia.co.in' will expire on Thu 14 Feb 2019 03:39:50 PM GMT +0000. [18:28:47] RECOVERY - www.evanswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.evanswiki.org' will expire on Tue 18 Dec 2018 05:04:24 PM GMT +0000. [18:28:49] RECOVERY - programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Fri 22 Feb 2019 11:47:07 PM GMT +0000. [18:28:51] RECOVERY - docs.websmart.media - LetsEncrypt on sslhost is OK: OK - Certificate 'docs.websmart.media' will expire on Sat 15 Dec 2018 08:04:36 AM GMT +0000. [18:28:53] RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Thu 03 Jan 2019 10:55:50 AM GMT +0000. [18:28:55] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Fri 22 Feb 2019 11:44:22 PM GMT +0000. [18:28:57] RECOVERY - bn.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'bn.gyaanipedia.co.in' will expire on Sun 10 Feb 2019 05:16:03 PM GMT +0000. [18:29:01] RECOVERY - es.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Sat 23 Feb 2019 10:45:35 PM GMT +0000. [18:29:03] RECOVERY - fikcyjnatv.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'fikcyjnatv.pl' will expire on Mon 31 Dec 2018 12:40:06 AM GMT +0000. [18:29:07] RECOVERY - wiki.ameciclo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ameciclo.org' will expire on Wed 19 Dec 2018 03:35:40 PM GMT +0000. [18:29:11] RECOVERY - wiki.dobots.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dobots.nl' will expire on Mon 24 Dec 2018 05:59:34 PM GMT +0000. [18:29:23] RECOVERY - adadevelopersacademy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'adadevelopersacademy.wiki' will expire on Mon 31 Dec 2018 12:39:05 AM GMT +0000. [18:31:47] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [18:32:07] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [18:32:11] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 22922 bytes in 0.017 second response time [18:32:15] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 22922 bytes in 0.006 second response time [18:32:31] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [18:32:43] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 23728 bytes in 0.515 second response time [18:33:25] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/1.1 200 OK - 23696 bytes in 2.037 second response time [18:33:29] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 3% [18:33:45] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 0% [18:34:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:34:29] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:34:37] RECOVERY - misc1 Current Load on misc1 is OK: OK - load average: 0.25, 1.63, 1.44 [18:34:55] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 22922 bytes in 0.012 second response time [18:44:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpaqM [18:44:24] [02miraheze/puppet] 07paladox 03859e32b - Add missing notify param in nginx::site [18:46:02] paladox: what if I don't want to give one? [18:46:14] JohnLewis it should undef? [18:46:15] you've probably broken a few stanzas [18:46:33] yes, but then no changes will ever get deployed? [18:47:21] okk [18:48:42] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:51:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpam4 [18:51:39] [02miraheze/puppet] 07paladox 0368527a5 - Fix $notify param [18:51:49] JohnLewis ^^ [18:52:20] JohnLewis does that work for you? Puppet works though (no failures) [18:52:36] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:52:40] you couldn't define the service in the config? [18:53:00] JohnLewis you mean nginx-syntax? [18:53:02] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:53:05] no [18:53:12] I mean service [18:53:22] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:53:25] JohnLewis service 'nginx' ? [18:53:36] yes..? [18:53:37] well why would it be in nginx::site? [18:53:54] because it is... [18:54:06] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:54:10] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:54:12] anyways brb - dinner. [18:54:21] https://github.com/miraheze/puppet/blob/master/modules/nginx/manifests/site.pp#L7 [18:54:21] Title: [ puppet/site.pp at master · miraheze/puppet · GitHub ] - github.com [18:54:30] you couldn't have put it there? [18:54:42] JohnLewis would that work? [18:54:49] Putting Service['nginx'] [18:54:50] there [18:54:56] I'm asking you [19:02:05] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [19:02:55] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:02:57] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:03:57] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:56:56] JohnLewis no im pretty sure that will fail from my knowledge of doing something like that before (i think) not 100% sure [19:57:13] ok [19:57:28] but im sure that's invalid as it won't be able to find the service seeing as it would be loaded before ::nginx [19:58:17] ok [20:28:02] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [20:30:02] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [20:55:21] PROBLEM - db4 Current Load on db4 is CRITICAL: CRITICAL - load average: 8.99, 6.55, 4.28 [21:05:20] PROBLEM - db4 Current Load on db4 is WARNING: WARNING - load average: 7.09, 7.78, 6.15 [21:06:10] huh [21:09:20] PROBLEM - db4 Current Load on db4 is CRITICAL: CRITICAL - load average: 8.12, 7.92, 6.59 [21:12:49] !log installing prometheus-mysqld-exporter on db4 [21:12:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:13:20] PROBLEM - db4 Current Load on db4 is WARNING: WARNING - load average: 6.11, 7.34, 6.70 [21:15:21] RECOVERY - db4 Current Load on db4 is OK: OK - load average: 3.66, 6.17, 6.36 [21:19:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fpaBD [21:19:37] [02miraheze/puppet] 07paladox 03e2eded0 - Add grants for "exporter" user [21:19:39] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [21:19:40] [02puppet] 07paladox opened pull request 03#910: Add grants for "exporter" user - 13https://git.io/fpaBy [21:23:58] [02puppet] 07paladox synchronize pull request 03#910: Add grants for "exporter" user - 13https://git.io/fpaBy [21:24:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fpaRs [21:24:01] [02miraheze/puppet] 07paladox 03418e7ba - Update default-grants.sql.erb [21:25:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fpaRn [21:25:15] [02miraheze/puppet] 07paladox 03548228c - Update db.pp [21:25:17] [02puppet] 07paladox synchronize pull request 03#910: Add grants for "exporter" user - 13https://git.io/fpaBy [21:27:43] [02puppet] 07paladox closed pull request 03#910: Add grants for "exporter" user - 13https://git.io/fpaBy [21:27:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fpaRr [21:27:46] [02miraheze/puppet] 07paladox 03e55fbd9 - Add grants for "exporter" user (#910) * Add grants for "exporter" user * Update default-grants.sql.erb * Update db.pp [21:27:47] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [21:27:49] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [21:33:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpa04 [21:33:05] [02miraheze/puppet] 07paladox 039d31a97 - Update config.pp [21:34:11] !log applying exporter grants on db4 [21:34:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:41:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpaEY [21:41:23] [02miraheze/puppet] 07paladox 0345cf4d9 - Update grants for exporter to use localhost [22:50:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpaV5 [22:50:13] [02miraheze/services] 07MirahezeSSLBot 030287229 - BOT: Updating services config for wikis [23:11:21] PROBLEM - db4 Current Load on db4 is CRITICAL: CRITICAL - load average: 13.47, 7.45, 5.13 [23:30:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [23:31:00] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [23:32:15] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2203 bytes in 0.057 second response time [23:32:30] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3801 bytes in 0.403 second response time [23:32:34] aware ^^ [23:33:13] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:33:21] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:33:27] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 54% [23:33:31] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [23:33:35] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4127 bytes in 0.063 second response time [23:33:47] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (111 "Connection refused") [23:34:19] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[idomysql-import-schema] [23:34:30] (the load went so high on db4 that mysql restarted) [23:34:54] ok, but this is downtime [23:35:02] yes indeed [23:35:23] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [23:35:39] PROBLEM - cp5 HTTPS on cp5 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3814 bytes in 0.960 second response time [23:35:45] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3801 bytes in 0.009 second response time [23:36:21] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 84% [23:39:47] RECOVERY - db4 MySQL on db4 is OK: Uptime: 92 Threads: 54 Questions: 2001 Slow queries: 0 Opens: 163 Flush tables: 1 Open tables: 157 Queries per second avg: 21.750 [23:40:09] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 41038 bytes in 0.187 second response time [23:40:27] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 23719 bytes in 0.604 second response time [23:40:59] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [23:41:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:41:11] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 4% [23:41:21] PROBLEM - db4 Current Load on db4 is WARNING: WARNING - load average: 1.57, 3.48, 7.75 [23:41:23] online [23:41:31] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 2% [23:41:33] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 18290 bytes in 0.118 second response time [23:41:39] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/1.1 200 OK - 23696 bytes in 2.032 second response time [23:41:43] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 23696 bytes in 0.012 second response time [23:42:17] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:42:21] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 52% [23:44:21] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 14% [23:45:21] RECOVERY - db4 Current Load on db4 is OK: OK - load average: 2.32, 2.74, 6.45 [23:55:22] !log running rebuildall.php on nonciclopediawiki on mw1 [23:55:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master