[00:20:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuGW [00:20:15] [02miraheze/services] 07MirahezeSSLBot 0307a8722 - BOT: Updating services config for wikis [00:26:02] PROBLEM - bacula1 Disk Space on bacula1 is WARNING: DISK WARNING - free space: / 97859 MB (20% inode=99%); [00:58:02] RECOVERY - bacula1 Disk Space on bacula1 is OK: DISK OK - free space: / 129611 MB (27% inode=99%); [01:09:10] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhucK [01:09:12] [02miraheze/ssl] 07paladox 0379877eb - Redirect intp.miraheze.org to taswin.miraheze.org per T4027 [01:17:47] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhucN [01:17:48] [02miraheze/dns] 07paladox 0378c29de - Depool cp5 [01:18:53] !log reboot cp5 [01:19:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:27:12] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuC4 [01:27:14] [02miraheze/dns] 07paladox 03c5a9039 - Revert "Depool cp5" This reverts commit 78c29deadbcda4ab708a55157895187aad608450. [01:32:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuCX [01:32:31] [02miraheze/puppet] 07paladox 0373cf7df - Update grafana's apt repo url [01:33:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuCM [01:33:53] [02miraheze/puppet] 07paladox 030b59655 - Fix grafana apt url [01:35:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuCD [01:35:08] [02miraheze/puppet] 07paladox 030bac0a7 - Update init.pp [02:05:07] PROBLEM - bacula1 Disk Space on bacula1 is WARNING: DISK WARNING - free space: / 97771 MB (20% inode=99%); [02:41:09] RECOVERY - bacula1 Disk Space on bacula1 is OK: DISK OK - free space: / 127792 MB (26% inode=99%); [03:00:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhu8u [03:00:15] [02miraheze/services] 07MirahezeSSLBot 03e0b4d45 - BOT: Updating services config for wikis [05:00:14] PROBLEM - mw3 JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [05:02:14] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 6395 [05:05:07] PROBLEM - bacula1 Disk Space on bacula1 is WARNING: DISK WARNING - free space: / 99846 MB (20% inode=99%); [05:19:55] RECOVERY - bacula1 Bacula Static on bacula1 is OK: OK: Diff, 409214 files, 30.47GB, 2019-01-20 05:17:00 (2.9 minutes ago) [05:23:46] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.35, 1.60, 0.77 [05:27:46] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.63, 1.76, 1.04 [05:29:47] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.51, 1.64, 1.08 [05:33:47] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.95, 2.32, 1.49 [05:40:13] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:42:14] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [05:45:46] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 0.31, 1.56, 1.66 [05:46:59] RECOVERY - bacula1 Bacula Phabricator Static on bacula1 is OK: OK: Diff, 44177 files, 517.1MB, 2019-01-20 05:46:00 (59.0 seconds ago) [05:53:47] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.26, 1.86, 1.70 [05:55:46] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.65, 1.74, 1.67 [05:59:46] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.47, 1.68, 1.66 [06:03:46] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.93, 1.88, 1.75 [06:05:47] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.37, 1.68, 1.69 [06:11:47] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 3.49, 2.42, 1.97 [06:17:46] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 0.77, 1.99, 2.00 [06:23:45] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 0.17, 0.93, 1.52 [09:20:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuuc [09:20:14] [02miraheze/services] 07MirahezeSSLBot 03e3776ef - BOT: Updating services config for wikis [09:32:14] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [15:15:51] !log set global local_infile=OFF; on db4 [15:15:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:15:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuXs [15:15:57] [02miraheze/puppet] 07paladox 034ca02fc - MariaDB: Set local_infile to 0 [15:17:47] !log upgrade phabricator to fix a bug on misc4 [15:17:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:31:58] [02puppet] 07JohnFLewis commented on commit 034ca02fc4192523122c06eddd54940b1918efacd2 - 13https://git.io/fhuXH [15:34:50] [02puppet] 07paladox commented on commit 034ca02fc4192523122c06eddd54940b1918efacd2 - 13https://git.io/fhuXA [15:35:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuXp [15:35:33] [02miraheze/puppet] 07paladox 0380638c4 - mariadb: Fix Code Style inconsistency [16:47:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuMb [16:47:08] [02miraheze/puppet] 07paladox 03a7c21ac - Reduce redis heap to 530MB [16:48:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuMj [16:48:47] [02miraheze/puppet] 07paladox 038ad9d7d - Update matomo to 3.8.0-rc3 [17:00:13] PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [17:02:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuDa [17:02:53] [02miraheze/puppet] 07paladox 036bb6294 - Remove prometheus blackbox_exporter from misc2 [17:12:32] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [17:12:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fhuD5 [17:12:35] [02miraheze/puppet] 07paladox 0374d4fd6 - Update redis systemd service [17:12:36] [02puppet] 07paladox opened pull request 03#980: Update redis systemd service - 13https://git.io/fhuDd [17:16:14] PROBLEM - mw3 JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [17:17:33] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [17:17:38] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [17:18:13] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [17:18:25] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [17:18:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [17:18:53] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 52% [17:19:07] hmm [17:20:14] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [17:20:53] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 3% [17:21:26] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [17:21:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuyn [17:21:31] [02miraheze/puppet] 07paladox 03a49799f - Revert "Reduce redis heap to 530MB" This reverts commit a7c21ac30d3ba63c20e51ef347b8628a426c0875. [17:21:32] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [17:22:08] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [17:22:21] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:22:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:39:45] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [17:41:05] [02mw-config] 07The-Voidwalker opened pull request 03#2613: fix pdfbook - 13https://git.io/fhuyx [17:41:58] [02mw-config] 07paladox commented on pull request 03#2613: fix pdfbook - 13https://git.io/fhuyp [17:43:56] [02mw-config] 07The-Voidwalker closed pull request 03#2613: fix pdfbook - 13https://git.io/fhuyx [17:43:58] [02mw-config] 07The-Voidwalker commented on pull request 03#2613: fix pdfbook - 13https://git.io/fhuSf [17:44:44] [02mw-config] 07The-Voidwalker opened pull request 03#2614: fix pdfbook - 13https://git.io/fhuST [17:45:24] [02mw-config] 07paladox closed pull request 03#2614: fix pdfbook - 13https://git.io/fhuST [17:45:25] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuSL [17:45:27] [02miraheze/mw-config] 07The-Voidwalker 0344b4d50 - fix pdfbook (#2614) take two [17:50:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuS8 [17:50:41] [02miraheze/puppet] 07paladox 034399bc2 - Update prometheus.yml.erb [17:56:41] [02puppet] 07paladox synchronize pull request 03#980: Update redis systemd service - 13https://git.io/fhuDd [17:56:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fhuSX [17:56:44] [02miraheze/puppet] 07paladox 03fa20d55 - Update redis-server.systemd [17:56:47] [02puppet] 07paladox closed pull request 03#980: Update redis systemd service - 13https://git.io/fhuDd [17:56:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuS1 [17:56:50] [02miraheze/puppet] 07paladox 0352e7d1c - Update redis systemd service (#980) * Update redis systemd service * Update redis-server.systemd [17:56:52] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [17:56:53] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [17:56:55] hi [17:57:16] hi [18:55:18] huh, PdfBook is not compatible with MediaWiki 1.32 [19:28:08] Voidwalker what error is thrown? [19:28:32] none, it just uses a hook removed in 1.32 [19:31:27] What's new? [19:32:27] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhu7K [19:32:28] [02miraheze/mediawiki] 07paladox 035657ea9 - Update AbuseFilter [19:48:31] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [19:48:32] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fhu54 [19:48:34] [02miraheze/mw-config] 07paladox 0346c4fe5 - Set wgGlobalBlockingDatabase in GlobalExtension [19:48:35] [02mw-config] 07paladox opened pull request 03#2615: Set wgGlobalBlockingDatabase in GlobalExtension - 13https://git.io/fhu5B [19:49:00] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fhu5R [19:49:01] [02miraheze/mw-config] 07paladox 0328bf4f6 - Update GlobalExtensions.php [19:49:03] [02mw-config] 07paladox synchronize pull request 03#2615: Set wgGlobalBlockingDatabase in GlobalExtension - 13https://git.io/fhu5B [19:49:20] [02mw-config] 07paladox closed pull request 03#2615: Set wgGlobalBlockingDatabase in GlobalExtension - 13https://git.io/fhu5B [19:49:21] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fhu50 [19:49:23] [02miraheze/mw-config] 07paladox 038a93d7b - Set wgGlobalBlockingDatabase in GlobalExtension (#2615) * Set wgGlobalBlockingDatabase in GlobalExtension * Update GlobalExtensions.php [19:49:24] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [19:49:26] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [19:52:21] miraheze/mw-config/paladox-patch-1/28bf4f6 - paladox The build is still failing. https://travis-ci.org/miraheze/mw-config/builds/482129584 [19:58:51] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhu5K [19:58:52] [02miraheze/mediawiki] 07paladox 03f48f6c2 - Update GlobalPreferences [20:03:33] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhu59 [20:03:35] [02miraheze/mediawiki] 07paladox 039b5ea63 - Update News [21:42:08] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [21:42:12] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [21:42:14] PROBLEM - mw3 JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [21:42:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [21:42:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [21:42:53] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [21:45:02] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [21:46:49] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 55% [21:46:53] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 54% [21:50:14] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [21:54:14] PROBLEM - mw3 JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [21:58:53] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 39% [22:00:14] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [22:02:53] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 50% [22:04:14] PROBLEM - mw3 JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [22:04:53] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 36% [22:12:49] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 37% [22:13:00] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3817 bytes in 0.009 second response time [22:14:22] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 45% [22:17:00] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 23711 bytes in 0.013 second response time [22:18:20] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 24% [22:21:41] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [22:21:52] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:22:08] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [22:22:14] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [22:22:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:23:32] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [22:28:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuxJ [22:28:01] [02miraheze/puppet] 07paladox 0345f6d69 - Update prometheus.yml.erb [22:29:34] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [22:29:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fhuxI [22:29:37] [02miraheze/puppet] 07paladox 03a46d7bb - Switch prometheus master to misc4 [22:29:38] [02puppet] 07paladox opened pull request 03#981: Switch prometheus master to misc4 - 13https://git.io/fhuxL [22:30:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fhuxq [22:30:28] [02miraheze/puppet] 07paladox 033dfc997 - Update site.pp [22:30:30] [02puppet] 07paladox synchronize pull request 03#981: Switch prometheus master to misc4 - 13https://git.io/fhuxL [22:30:44] !log shutting prometheus master down on misc2 [22:30:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:31:41] [02puppet] 07paladox closed pull request 03#981: Switch prometheus master to misc4 - 13https://git.io/fhuxL [22:31:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fhuxZ [22:31:43] [02miraheze/puppet] 07paladox 03e5e4797 - Switch prometheus master to misc4 (#981) * Switch prometheus master to misc4 * Update site.pp [22:31:45] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [22:31:46] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [22:32:15] !log setting prometheus master up on misc4 [22:32:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:33:49] PROBLEM - misc2 Prometheus on misc2 is CRITICAL: connect to address 81.4.127.174 and port 9090: Connection refused [22:34:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhux0 [22:34:57] [02miraheze/puppet] 07paladox 03aa07175 - Update varnish_prometheus_exporter.pp [22:35:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuxE [22:35:17] [02miraheze/puppet] 07paladox 0306bdff6 - Update mysqld_exporter.pp [22:36:28] !log reboot misc2 to reduce ram [22:36:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:41:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhuxX [22:41:33] [02miraheze/puppet] 07paladox 03e2e1c98 - Increase redis heap to 650M