[00:03:02] PROBLEM - Puppet on misc4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:02] PROBLEM - Puppet on swift2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:08] xd [00:04:12] PROBLEM - MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:59] uh [00:05:12] PROBLEM - MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:32] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [00:06:02] RECOVERY - MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 31031 bytes in 0.105 second response time [00:06:12] PROBLEM - MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:22] PROBLEM - Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [00:06:32] PROBLEM - Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [00:06:52] RECOVERY - Bacula - Private Git on bacula1 is OK: OK: Full, 1372 files, 1.467MB, 2018-07-08 00:05:00 (1.8 minutes ago) [00:07:22] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [00:07:28] investigating [00:08:00] RECOVERY - MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 31031 bytes in 0.099 second response time [00:08:30] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 34% [00:08:32] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [00:09:00] RECOVERY - MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 31031 bytes in 0.089 second response time [00:09:20] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 17% [00:09:30] RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:09:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvKa [00:09:35] [02miraheze/puppet] 07paladox 039d03be6 - Update proxy-server.conf.erb [00:10:00] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:11:02] RECOVERY - Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [00:11:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvKr [00:11:50] [02miraheze/puppet] 07paladox 03efb9be7 - Update proxy-server.conf.erb [00:12:00] RECOVERY - Puppet on swift2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:13:01] RECOVERY - Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:13:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvKo [00:13:10] [02miraheze/puppet] 07paladox 031724795 - Update object-server.conf.erb [05:05:01] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2642 [05:59:02] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [06:22:41] PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.62, 3.26, 2.87 [06:24:41] RECOVERY - Current Load on swift1 is OK: OK - load average: 3.16, 3.15, 2.87 [06:43:41] PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.47, 3.37, 3.07 [06:44:31] PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.2-gettext] [06:52:31] RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:20:32] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw3 [07:22:30] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [07:28:30] PROBLEM - Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [07:28:32] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [07:29:10] PROBLEM - MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:29:20] PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [07:33:01] RECOVERY - MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 31031 bytes in 0.108 second response time [07:33:21] RECOVERY - Varnish Backends on cp4 is OK: All 3 backends are healthy [07:34:31] RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [07:35:01] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:59:00] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5467 [09:19:35] [02miraheze/services] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNv9n [09:19:37] [02miraheze/services] 07Reception123 03dba8bf6 - add marthaspeaks and thesciencearchives to services [10:14:31] PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [10:22:31] RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:53:01] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [11:54:31] PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [12:02:31] RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:43:02] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2644 [15:17:00] PROBLEM - Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:39:01] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [16:32:33] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNfvv [16:32:34] [02miraheze/mw-config] 07paladox 03b9dea60 - Set wgScoreFileBackend to miraheze-swift [17:18:31] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 45% [17:20:32] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 18% [17:33:00] PROBLEM - wiki.ameciclo.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate wiki.ameciclo.org expires in 15 day(s) (Tue 24 Jul 2018 05:29:56 PM GMT +0000). [17:33:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNfUs [17:33:16] [02miraheze/ssl] 07MirahezeSSLBot 03c24f259 - Bot: Update SSL cert for wiki.ameciclo.org [17:36:24] PROBLEM - wiki.jacksonheights.nyc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate wiki.jacksonheights.nyc expires in 15 day(s) (Tue 24 Jul 2018 05:33:56 PM GMT +0000). [17:36:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNfU4 [17:36:36] [02miraheze/ssl] 07MirahezeSSLBot 0393cead3 - Bot: Update SSL cert for wiki.jacksonheights.nyc [17:40:25] PROBLEM - publictestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate publictestwiki.com expires in 15 day(s) (Tue 24 Jul 2018 05:36:42 PM GMT +0000). [17:40:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNfU2 [17:40:36] [02miraheze/ssl] 07MirahezeSSLBot 0310b9abc - Bot: Update SSL cert for publictestwiki.com [17:42:25] RECOVERY - wiki.jacksonheights.nyc - LetsEncrypt on sslhost is OK: OK - Certificate wiki.jacksonheights.nyc will expire on Sat 06 Oct 2018 04:36:29 PM GMT +0000. [17:42:55] RECOVERY - wiki.ameciclo.org - LetsEncrypt on sslhost is OK: OK - Certificate wiki.ameciclo.org will expire on Sat 06 Oct 2018 04:33:08 PM GMT +0000. [17:44:25] PROBLEM - nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate nonbinary.wiki expires in 15 day(s) (Tue 24 Jul 2018 05:41:38 PM GMT +0000). [17:44:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNfUo [17:44:36] [02miraheze/ssl] 07MirahezeSSLBot 034211d06 - Bot: Update SSL cert for nonbinary.wiki [17:48:25] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate miraheze.wiki expires in 15 day(s) (Tue 24 Jul 2018 05:45:34 PM GMT +0000). [17:48:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNfUD [17:48:37] [02miraheze/ssl] 07MirahezeSSLBot 036ec4a0a - Bot: Update SSL cert for miraheze.wiki [17:52:15] RECOVERY - publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate publictestwiki.com will expire on Sat 06 Oct 2018 04:40:29 PM GMT +0000. [17:52:25] RECOVERY - nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate nonbinary.wiki will expire on Sat 06 Oct 2018 04:44:30 PM GMT +0000. [18:07:49] Hello hl`! If you have any questions feel free to ask and someone should answer soon. [18:34:26] PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [18:42:24] RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:54:32] Hello [19:03:54] Hey. When you request a custom domain, how does that work? [19:06:23] Have you read? https://meta.miraheze.org/wiki/Custom_domains [19:06:24] Title: [ Custom domains - Miraheze Meta ] - meta.miraheze.org [19:06:56] auroraborealisss: [19:10:19] auroraborealisss: ? [19:11:04] I've read through now & set the nameservers [19:11:14] im not sure how much else needs to be done, im hoping u can help? [19:27:22] auroraborealisss: https://phabricator.miraheze.org/maniphest/task/edit/form/14/ [19:27:23] Title: [ Login ] - phabricator.miraheze.org [19:30:28] I requested it when I requested my wiki [19:30:51] auroraborealisss: https://meta.miraheze.org/wiki/Special:RequestWikiQueue/5342? [19:30:52] Title: [ Wiki requests queue - Miraheze Meta ] - meta.miraheze.org [19:31:02] And you also need to have a separate Phabricator request for the custom domain [19:32:35] ah ok [21:27:42] [02ssl] 07MacFan4000 opened pull request 03#40: add wiki.rmbrk.sk cert - 13https://git.io/fNfYw [21:31:38] [02ssl] 07paladox closed pull request 03#40: add wiki.rmbrk.sk cert - 13https://git.io/fNfYw [21:31:39] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fNfY1 [21:31:40] [02miraheze/ssl] 07MacFan4000 03e8b8384 - add wiki.rmbrk.sk cert (#40) * Create wiki.rmbrk.sk.crt * Update certs.yaml [21:38:31] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNfYd [21:38:32] [02miraheze/mw-config] 07MacFan4000 035c1b8bd - Add rmbrkwiki custom domain [21:44:29] PROBLEM - Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.rmbrk.sk_private] [21:44:59] PROBLEM - Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.rmbrk.sk_private] [21:45:01] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.rmbrk.sk_private] [21:45:09] PROBLEM - Puppet on cp5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.rmbrk.sk_private] [21:47:18] Uh paladox ^^ [21:47:25] huh [21:48:00] oh i didn't add the private cert [21:48:04] It’s the private key it seems [21:48:11] Ah, ok [21:48:52] tsk tsk [21:48:57] MacFan4000 did it print the private key location? [21:49:19] itll be in the ssl directory on mw1 [21:49:28] Uh, I already disconnected [21:49:33] found it [21:50:26] done [21:52:30] RECOVERY - Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [21:53:00] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [21:53:10] RECOVERY - Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:56:04] PROCS CRITICAL: 2 processes with args 'redis-server' for misc2 [21:58:02] MacFan4000 ? [21:58:05] it's working for me [21:59:51] In icinga it says that [22:00:10] Nvm [22:10:47] paladox: https://phabricator.miraheze.org/T3329#62946 [22:10:49] Title: [ ⚓ T3329 RMBRK ] - phabricator.miraheze.org [22:11:17] JohnLewis that's caching i presume, as it does the same thing for me, if i add /wiki/m it works [22:11:30] indeed [22:11:39] so remove it from the cache [22:12:49] JohnLewis i did "ban req.http.Host == wiki.rmbrk.sk" [22:12:53] dosen't seem to have fix it [22:13:59] on all servers? [22:14:13] wfm now [22:14:15] works now [22:14:24] JohnLewis i did it on cp4 first [22:14:28] as i am in the eu :) [22:15:01] still, should do it on all anyway [22:15:20] JohnLewis i've done it on all now. [22:18:46] JohnLewis works for you now? [22:19:00] yeah [22:19:42] ok great [22:26:40] PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.63, 3.34, 2.84 [22:30:40] PROBLEM - Current Load on swift1 is CRITICAL: CRITICAL - load average: 5.32, 3.89, 3.14 [22:32:40] PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.23, 3.77, 3.19 [22:38:01] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [22:38:31] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [22:39:21] PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [22:43:59] PROBLEM - Current Load on swift2 is WARNING: WARNING - load average: 3.70, 3.53, 2.93 [22:44:01] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:44:29] RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:44:31] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [22:45:19] RECOVERY - Varnish Backends on cp4 is OK: All 3 backends are healthy [22:45:59] RECOVERY - Current Load on swift2 is OK: OK - load average: 3.20, 3.38, 2.94 [22:49:59] PROBLEM - Current Load on swift2 is WARNING: WARNING - load average: 3.71, 3.56, 3.10 [22:52:00] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fNfsg [22:52:02] [02miraheze/mediawiki] 07paladox 035225be1 - Update DynamicPageList3 to 2e1a468af049c238133b86e8cbd44763bf2cac8c [22:55:12] [02WikiDiscover] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vhUAp [22:55:13] [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNfsw [22:55:15] [02miraheze/WikiDiscover] 07paladox 03a53c9d5 - Add missing $wiki variable [22:55:16] [02WikiDiscover] 07paladox opened pull request 03#7: Add missing $wiki variable - 13https://git.io/fNfsr [22:56:42] [02WikiDiscover] 07paladox closed pull request 03#7: Add missing $wiki variable - 13https://git.io/fNfsr [22:57:26] [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNfsK [22:57:28] [02miraheze/WikiDiscover] 07paladox 03ab1e720 - Remove $wiki var from WikiDiscoverWikisPager [22:58:02] reat, now remove it from the function [22:58:06] [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNfsP [22:58:07] [02miraheze/WikiDiscover] 07paladox 037545a8f - Update WikiDiscoverWikisPager.php [22:58:26] JohnLewis yeh, i accidentially pressed the submit button instead of pull [22:58:32] was ment to do that in one change [22:59:09] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fNfsX [22:59:11] [02miraheze/mediawiki] 07paladox 03acf2889 - Update WikiDiscover [22:59:36] Hi [22:59:45] hi [23:00:10] PROBLEM - MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:00:20] PROBLEM - Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [23:00:30] PROBLEM - Varnish Backends on cp5 is CRITICAL: 2 backends are down. mw2 mw3 [23:00:57] 5 0 3 [23:02:00] RECOVERY - MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 31035 bytes in 0.106 second response time [23:02:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNfsH [23:02:46] [02miraheze/puppet] 07paladox 034422e91 - Update proxy-server.conf.erb [23:03:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNfsQ [23:03:04] [02miraheze/puppet] 07paladox 0397106ed - Increase container runner to 2 [23:03:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNfs7 [23:03:15] [02miraheze/puppet] 07paladox 038724c92 - Increase account runner to 2 [23:03:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNfs5 [23:03:26] [02miraheze/puppet] 07paladox 0394aab63 - Increase object runner to 2