[00:03:40] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:54] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:11:40] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:11:52] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:15:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdq9 [00:15:15] [02miraheze/services] 07MirahezeSSLBot 0334894ed - BOT: Updating services config for wikis [00:35:16] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [03:00:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAd3x [03:00:16] [02miraheze/services] 07MirahezeSSLBot 035eb9670 - BOT: Updating services config for wikis [04:57:16] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 6480 [06:10:16] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdZM [06:10:17] [02miraheze/services] 07MirahezeSSLBot 03fc4b351 - BOT: Updating services config for wikis [07:29:14] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [07:57:34] PROBLEM - wiki.ngscott.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ngscott.net' expires in 15 day(s) (Mon 08 Oct 2018 07:55:01 AM GMT +0000). [07:57:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdca [07:57:46] [02miraheze/ssl] 07MirahezeSSLBot 03c9a6216 - Bot: Update SSL cert for wiki.ngscott.net [08:01:34] RECOVERY - wiki.ngscott.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ngscott.net' will expire on Fri 21 Dec 2018 06:57:39 AM GMT +0000. [10:10:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdlo [10:10:15] [02miraheze/services] 07MirahezeSSLBot 031879119 - BOT: Updating services config for wikis [13:18:06] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdEs [13:18:08] [02miraheze/puppet] 07Southparkfan 035846345 - Cache piwik.js file in varnish [13:18:51] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=starmetalwiki --report=1 user%27s+Wiki%21-20180919040511.xml on mw1 [13:18:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:27:27] [02miraheze/MatomoAnalytics] 07Southparkfan pushed 0317 commits to 03Southparkfan-patch-1 [+8/-0/±20] 13https://git.io/fAdEw [13:27:29] [02miraheze/MatomoAnalytics] 07Southparkfan 038e90f2f - Merge branch 'master' into Southparkfan-patch-1 [13:27:31] [02MatomoAnalytics] 07Southparkfan synchronize pull request 03#5: Tidy up MatomoAnalytics class - 13https://git.io/fAZqX [13:27:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdEo [13:27:51] [02miraheze/mw-config] 07paladox 032e84d8d - Fix nonciclopediawiki namespaces per T3638 [13:29:56] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdED [13:29:58] [02miraheze/mw-config] 07paladox 030caefd9 - Fix nonciclopediawiki namespaces part 2 [13:30:49] [02miraheze/MatomoAnalytics] 07Southparkfan pushed 031 commit to 03Southparkfan-patch-1 [+0/-0/±1] 13https://git.io/fAdES [13:30:50] [02miraheze/MatomoAnalytics] 07Southparkfan 03b988019 - Update MatomoAnalytics.php [13:30:52] [02MatomoAnalytics] 07Southparkfan synchronize pull request 03#5: Tidy up MatomoAnalytics class - 13https://git.io/fAZqX [13:32:40] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/fAdEH [13:32:41] [02miraheze/puppet] 07Southparkfan 03628ef25 - Delete allow_multi.pp Does not work. [13:38:57] miraheze/mw-config/master/2e84d8d - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/431864532 [13:48:52] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [13:48:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fAduc [13:48:55] [02miraheze/puppet] 07paladox 033349769 - Change 'any' to 'Anywhere' in ufw [13:48:56] [02puppet] 07paladox opened pull request 03#846: Change 'any' to 'Anywhere' in ufw - 13https://git.io/fAduW [13:56:15] [02puppet] 07paladox closed pull request 03#846: Change 'any' to 'Anywhere' in ufw - 13https://git.io/fAduW [13:56:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [13:56:18] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [14:03:15] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdze [14:03:16] [02miraheze/puppet] 07Southparkfan 03dd3bb52 - Fix unless regex inside ufw::allow cat -A revealed some hidden whitespace [14:12:04] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdzC [14:12:06] [02miraheze/puppet] 07Southparkfan 03d4e9be2 - Get a fix for all unless situation inside ufw::allow gotta catch 'em all.. :-) [14:29:24] !log recreating phabstatic backup on bacula1 [14:29:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:31:11] !log recreating LIZARDFS1-CHUNKSERVER* backup on bacula1 [14:31:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:37:25] !log MariaDB [mhglobal]> delete from matomo where matomo_id = 3246; (dummy entry) [14:37:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:41:06] !log above entry also deleted in matomo SitesManager [14:41:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:41:24] RECOVERY - bacula1 Bacula Phabricator Static on bacula1 is OK: OK: Full, 69568 files, 4.111GB, 2018-09-22 14:40:00 (1.4 minutes ago) [14:41:50] !log PURGE BINARY LOGS BEFORE '2018-09-22 15:41:00'; on db4 [14:41:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:45:54] RECOVERY - bacula1 Bacula Private Git on bacula1 is OK: OK: Full, 1557 files, 1.736MB, 2018-09-22 14:44:00 (1.9 minutes ago) [14:52:14] [02miraheze/MatomoAnalytics] 07Southparkfan pushed 031 commit to 03Southparkfan-patch-1 [+0/-0/±1] 13https://git.io/fAd2L [14:52:15] [02miraheze/MatomoAnalytics] 07Southparkfan 03180bfe3 - Final fixes - Ignore cache (can't use $this inside static function) - Don't close db connections, causes issues - Fix variable names [14:52:17] [02MatomoAnalytics] 07Southparkfan synchronize pull request 03#5: Tidy up MatomoAnalytics class - 13https://git.io/fAZqX [14:52:35] [02MatomoAnalytics] 07Southparkfan edited pull request 03#5: Tidy up MatomoAnalytics class - 13https://git.io/fAZqX [14:53:05] !log deleted all dummy wikis from matomo table and matomo SitesManager [14:53:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:54:09] [02MatomoAnalytics] 07Southparkfan closed pull request 03#5: Tidy up MatomoAnalytics class - 13https://git.io/fAZqX [14:54:09] !log recreating db4 backup on bacula1 [14:54:10] [02miraheze/MatomoAnalytics] 07Southparkfan pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fAd2s [14:54:11] [02miraheze/MatomoAnalytics] 07Southparkfan 036791a06 - Tidy up MatomoAnalytics class (#5) - Use native Http/FormatJson wrappers with wfAppendQuery - PHP short array syntax [14:54:13] [02MatomoAnalytics] 07Southparkfan deleted branch 03Southparkfan-patch-1 - 13https://git.io/fN4LT [14:54:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:54:15] [02miraheze/MatomoAnalytics] 07Southparkfan deleted branch 03Southparkfan-patch-1 [14:54:36] paladox: if you would like, you can deploy this [14:54:41] SPF|Cloud ok [14:54:43] deploying [14:55:29] SPF|Cloud done [14:55:29] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fAd2l [14:55:31] [02miraheze/mediawiki] 07paladox 033c5442c - Update MA [14:58:36] SPF|Cloud deployed [15:00:23] RECOVERY - bacula1 Disk Space on bacula1 is OK: DISK OK - free space: / 312458 MB (65% inode=99%); [15:48:46] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [15:48:47] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fAdVP [15:48:49] [02miraheze/ssl] 07paladox 0354f724c - Add www.b1tcash.org ssl cert [15:48:50] [02ssl] 07paladox opened pull request 03#59: Add www.b1tcash.org ssl cert - 13https://git.io/fAdVX [15:49:09] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/fAdVM [15:49:10] [02miraheze/ssl] 07paladox 03b4c0f5a - Create www.b1tcash.org.crt [15:49:12] [02ssl] 07paladox synchronize pull request 03#59: Add www.b1tcash.org ssl cert - 13https://git.io/fAdVX [15:49:51] [02ssl] 07paladox closed pull request 03#59: Add www.b1tcash.org ssl cert - 13https://git.io/fAdVX [15:49:53] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fAdVD [15:49:54] [02miraheze/ssl] 07paladox 03bc8f950 - Add www.b1tcash.org ssl cert (#59) * Add www.b1tcash.org ssl cert * Create www.b1tcash.org.crt [15:49:56] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [15:49:57] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [15:53:03] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [15:53:04] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fAdV5 [15:53:06] [02miraheze/ssl] 07paladox 03e297cc3 - Redirect www.b1tcash.org to b1tcash.org [15:53:07] [02ssl] 07paladox opened pull request 03#60: Redirect www.b1tcash.org to b1tcash.org - 13https://git.io/fAdVd [15:53:18] [02ssl] 07paladox synchronize pull request 03#60: Redirect www.b1tcash.org to b1tcash.org - 13https://git.io/fAdVd [15:53:20] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fAdVb [15:53:21] [02miraheze/ssl] 07paladox 03eef5884 - Update redirects.yaml [15:53:36] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fAdVA [15:53:38] [02miraheze/ssl] 07paladox 03da54582 - Update certs.yaml [15:53:40] [02ssl] 07paladox synchronize pull request 03#60: Redirect www.b1tcash.org to b1tcash.org - 13https://git.io/fAdVd [15:53:49] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/fAdVx [15:53:50] [02miraheze/ssl] 07paladox 035aac420 - Delete www.b1tcash.org.crt [15:53:52] [02ssl] 07paladox synchronize pull request 03#60: Redirect www.b1tcash.org to b1tcash.org - 13https://git.io/fAdVd [15:54:06] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fAdVp [15:54:08] [02miraheze/ssl] 07paladox 031b5d95d - Update b1tcash.org.crt [15:54:09] [02ssl] 07paladox synchronize pull request 03#60: Redirect www.b1tcash.org to b1tcash.org - 13https://git.io/fAdVd [15:55:28] [02ssl] 07paladox closed pull request 03#60: Redirect www.b1tcash.org to b1tcash.org - 13https://git.io/fAdVd [15:55:29] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±3] 13https://git.io/fAdwe [15:55:31] [02miraheze/ssl] 07paladox 03884a1c6 - Redirect www.b1tcash.org to b1tcash.org (#60) * Redirect www.b1tcash.org to b1tcash.org * Update redirects.yaml * Update certs.yaml * Delete www.b1tcash.org.crt * Update b1tcash.org.crt [15:55:32] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [15:55:34] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [17:02:39] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdoK [17:02:40] [02miraheze/mw-config] 07paladox 033cdf5d1 - Set wgPFEnableStringFunctions for ktswiki fixes T3643 [18:54:45] PROBLEM - wiki.ombre.io - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ombre.io' expires in 15 day(s) (Mon 08 Oct 2018 06:51:11 PM GMT +0000). [18:54:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdXf [18:54:58] [02miraheze/ssl] 07MirahezeSSLBot 03139011f - Bot: Update SSL cert for wiki.ombre.io [19:02:45] RECOVERY - wiki.ombre.io - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ombre.io' will expire on Fri 21 Dec 2018 05:54:51 PM GMT +0000. [19:23:02] PROBLEM - wikiescola.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikiescola.com.br' expires in 15 day(s) (Mon 08 Oct 2018 07:19:31 PM GMT +0000). [19:23:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAd1L [19:23:18] [02miraheze/ssl] 07MirahezeSSLBot 035f6fd85 - Bot: Update SSL cert for wikiescola.com.br [19:33:03] RECOVERY - wikiescola.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiescola.com.br' will expire on Fri 21 Dec 2018 06:23:10 PM GMT +0000. [20:15:23] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2185 bytes in 0.148 second response time [20:15:48] um [20:16:03] oh [20:44:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdyN [20:44:45] [02miraheze/puppet] 07paladox 0370ffa35 - Fix if check Hope this won't break like the last time [20:46:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdyh [20:46:21] [02miraheze/puppet] 07paladox 03d606a0b - Include ssl::wildcard in matomo puppet class [20:47:37] !log restarting db4 (it's a emergency to get the new cert) [20:49:58] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [20:50:02] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [20:50:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [20:50:30] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 89% [20:50:32] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - check plugin (check_jobqueue) or PHP errors [20:50:37] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 92% [20:51:22] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4127 bytes in 0.049 second response time [20:51:26] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (111 "Connection refused") [20:52:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [20:52:12] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:52:30] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 44% [20:52:32] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [20:52:34] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:52:38] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 57% [20:52:50] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:52:54] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:53:04] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:53:18] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:53:22] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 18283 bytes in 0.239 second response time [20:53:26] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 40153 bytes in 2.859 second response time [20:53:50] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[idomysql-import-schema] [20:53:58] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [20:54:02] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [20:54:04] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:54:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:54:30] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 1% [20:54:36] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 2% [20:57:30] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [20:58:08] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 18% [21:02:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdSV [21:02:38] [02miraheze/puppet] 07paladox 03b0bdb88 - Remove ssl::wildcard from mediawiki class wildcard cert is added with ssl::hiera so no need to duplicate things [21:03:40] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [21:04:24] [02puppet] 07JohnFLewis commented on commit 03b0bdb884ae7ac7689d5c58aa88dca98653712ceb - 13https://git.io/fAdSK [21:04:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdS6 [21:04:34] [02miraheze/puppet] 07paladox 034aa3ed0 - Remove ssl::wildcard from varnish puppet class wildcard cert is added with ssl::hiera so no need to duplicate things [21:11:01] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [21:11:09] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [21:11:11] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [21:11:13] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [21:11:37] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [21:14:03] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:17:06] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [21:17:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fAd9T [21:17:09] [02miraheze/puppet] 07paladox 030a3d2d1 - Fix if defined check in ssl::wildcard Also include ssl::wildcard back in varnish and mediawiki puppet class [21:17:10] [02puppet] 07paladox opened pull request 03#847: Fix if defined check in ssl::wildcard - 13https://git.io/fAd9k [21:18:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fAd9Y [21:18:07] [02miraheze/puppet] 07paladox 03179b605 - Update nginx.pp [21:18:08] [02puppet] 07paladox synchronize pull request 03#847: Fix if defined check in ssl::wildcard - 13https://git.io/fAd9k [21:18:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fAd9O [21:18:30] [02miraheze/puppet] 07paladox 03f646d89 - Update init.pp [21:18:31] [02puppet] 07paladox synchronize pull request 03#847: Fix if defined check in ssl::wildcard - 13https://git.io/fAd9k [21:21:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fAd9Z [21:21:17] [02miraheze/puppet] 07paladox 03f6713ec - Update wildcard.pp [21:21:19] [02puppet] 07paladox synchronize pull request 03#847: Fix if defined check in ssl::wildcard - 13https://git.io/fAd9k [21:21:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/fAd9n [21:21:45] [02miraheze/puppet] 07paladox 0348f4829 - Fix if defined check in ssl::wildcard (#847) * Fix if defined check in ssl::wildcard Also include ssl::wildcard back in varnish and mediawiki puppet class * Update nginx.pp * Update init.pp * Update wildcard.pp [21:21:46] [02puppet] 07paladox closed pull request 03#847: Fix if defined check in ssl::wildcard - 13https://git.io/fAd9k [21:21:48] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [21:21:49] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [21:23:12] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:23:18] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:23:55] lol and a new problem pops up :P [21:25:07] it's puppetdb [21:29:12] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:32:52] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:33:16] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [21:34:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:43:41] !log recreating STATIC* backup on bacula1 [21:43:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:59:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdHR [21:59:37] [02miraheze/puppet] 07paladox 03f4b60eb - Update LizardFS-* backup expirary date to 13 days [22:04:06] !log delete and recreated lizardfs-chunkserver* and lizardfs-master from bacula1 [22:04:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:06:04] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 41% [22:08:52] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdHS [22:08:53] [02miraheze/puppet] 07JohnFLewis 0345cc1eb - update pool size comment [22:17:13] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 32% [22:20:21] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 6366 [22:21:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fAdQI [22:21:55] [02miraheze/puppet] 07paladox 03237274e - Fix blacklisted UA [22:21:57] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [22:21:58] [02puppet] 07paladox opened pull request 03#848: Fix blacklisted UA - 13https://git.io/fAdQL [22:22:26] [02puppet] 07paladox synchronize pull request 03#848: Fix blacklisted UA - 13https://git.io/fAdQL [22:22:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fAdQt [22:22:29] [02miraheze/puppet] 07paladox 037f52d05 - Update mediawiki.conf [22:23:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAdQm [22:23:17] [02miraheze/puppet] 07paladox 0353f845c - Fix blacklisted UA (#848) * Fix blacklisted UA * Update mediawiki.conf [22:23:19] [02puppet] 07paladox closed pull request 03#848: Fix blacklisted UA - 13https://git.io/fAdQL [22:23:20] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [22:23:21] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [22:43:19] [02miraheze/puppet] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fAd7T [22:43:21] [02miraheze/puppet] 07JohnFLewis 03d0b44ed - track TLS protocol and ciphers in access.log [22:43:22] [02miraheze/puppet] 07JohnFLewis 038647e8a - Merge branch 'master' of github.com:miraheze/puppet [22:47:22] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [22:47:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fAd73 [22:47:25] [02miraheze/puppet] 07paladox 0384fa24f - Reload nginx service when nginx.conf or fastcgi_params is changed [22:47:27] [02puppet] 07paladox opened pull request 03#849: Reload nginx service when nginx.conf or fastcgi_params is changed - 13https://git.io/fAd7Z [22:47:52] [02puppet] 07JohnFLewis closed pull request 03#849: Reload nginx service when nginx.conf or fastcgi_params is changed - 13https://git.io/fAd7Z [22:47:53] [02miraheze/puppet] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fAd7n [22:47:55] [02miraheze/puppet] 07JohnFLewis 03c4ee21b - Merge pull request #849 from miraheze/paladox-patch-1 Reload nginx service when nginx.conf or fastcgi_params is changed [22:48:02] JohnLewis thanks :) [22:54:29] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [22:54:31] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1