[03:30:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjIxl [03:30:13] [02miraheze/services] 07MirahezeSSLBot 037d45764 - BOT: Updating services config for wikis [05:02:06] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Wed 05 Jun 2019 12:45:23 PM GMT +0000. [05:10:34] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:04:12] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Wed 05 Jun 2019 12:45:23 PM GMT +0000. [07:08:37] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:23:16] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Wed 05 Jun 2019 12:45:23 PM GMT +0000. [07:31:45] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:37] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Wed 05 Jun 2019 12:45:23 PM GMT +0000. [08:39:22] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:58:47] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 596 [10:08:47] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [10:58:47] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 1074 [11:22:47] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [12:58:14] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Wed 05 Jun 2019 12:45:23 PM GMT +0000. [13:02:39] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:28:51] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Wed 05 Jun 2019 12:45:23 PM GMT +0000. [16:33:16] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:33:22] Hello ElDrago! If you have any questions feel free to ask and someone should answer soon. [18:13:02] [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLOE [18:13:03] [02miraheze/IncidentReporting] 07translatewiki 030fffb03 - Localisation updates from https://translatewiki.net. [18:13:05] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLOu [18:13:06] [02miraheze/MirahezeMagic] 07translatewiki 0388fb8b6 - Localisation updates from https://translatewiki.net. [18:34:37] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLOy [18:34:38] [02miraheze/mw-config] 07Reception123 03681551a - allow ext images on piratapentewiki T4230 [18:50:47] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 340 [18:52:47] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [19:30:47] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 310 [19:32:47] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [19:57:38] !log depool mw1 [19:57:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:58:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLs4 [19:58:33] [02miraheze/puppet] 07paladox 03c27001e - Downgrade mw1 back to php 7.2 due to a segfault [20:01:59] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [20:02:07] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [20:02:30] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [20:02:42] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [20:02:50] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [20:03:11] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [20:05:07] !log repool mw1 [20:05:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:05:59] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [20:06:29] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [20:06:50] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [20:07:11] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [20:16:42] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [20:19:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fjLs5 [20:19:55] [02miraheze/puppet] 07paladox 0329a3e47 - php: Update apt key [20:20:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLsd [20:20:59] [02miraheze/puppet] 07paladox 03f81616b - php: Fix syntax error [20:23:05] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 46757 MB (12% inode=96%); [20:23:49] !log depool mw3 [20:23:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:24:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLsx [20:24:22] [02miraheze/puppet] 07paladox 03feaeb4c - mw3: Downgrade to php 7.2 [20:26:47] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 327 [20:26:50] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [20:27:11] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [20:27:59] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw3 [20:28:47] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [20:28:50] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [20:29:11] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [20:29:59] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [20:30:19] !log repool mw3 [20:30:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:31:18] !log depool mw2 [20:31:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:33:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLGq [20:33:41] [02miraheze/puppet] 07paladox 03d118be5 - mw2: Downgrade php to 7.2 [20:34:03] PROBLEM - mw2 php-fpm on mw2 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.3' [20:34:09] PROBLEM - mw2 Puppet on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [20:34:30] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [20:34:50] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [20:35:11] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [20:35:59] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw2 [20:41:11] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [20:41:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjLGs [20:41:14] [02miraheze/puppet] 07paladox 034bcdacc - php: Move apt repo [20:41:16] [02puppet] 07paladox opened pull request 03#991: php: Move apt repo - 13https://git.io/fjLGG [20:42:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjLGZ [20:42:17] [02miraheze/puppet] 07paladox 038e7ac4a - Update php_fpm.pp [20:42:18] [02puppet] 07paladox synchronize pull request 03#991: php: Move apt repo - 13https://git.io/fjLGG [20:50:04] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [20:50:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fjLGR [20:50:07] [02miraheze/puppet] 07paladox 03041ce46 - php: Move apt repo back [20:50:08] [02puppet] 07paladox opened pull request 03#992: php: Move apt repo back - 13https://git.io/fjLG0 [20:50:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fjLGE [20:50:35] [02miraheze/puppet] 07paladox 033cb4999 - Update php_fpm.pp [20:50:37] [02puppet] 07paladox synchronize pull request 03#992: php: Move apt repo back - 13https://git.io/fjLG0 [20:51:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLGu [20:51:55] [02miraheze/puppet] 07paladox 03fb0b3b1 - Update init.pp [20:52:11] [02puppet] 07paladox closed pull request 03#991: php: Move apt repo - 13https://git.io/fjLGG [20:52:19] [02puppet] 07paladox closed pull request 03#992: php: Move apt repo back - 13https://git.io/fjLG0 [20:52:21] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [20:52:23] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [20:52:28] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [20:52:29] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [20:56:03] RECOVERY - mw2 php-fpm on mw2 is OK: PROCS OK: 3 processes with command name 'php-fpm7.2' [20:58:07] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:58:18] !log repool mw2 [20:58:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:58:30] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [20:58:50] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [20:59:11] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [20:59:59] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [21:42:45] * Hispano76 saluda [21:47:33] !log starting import for pseudocienciawiki [21:47:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:50:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLZs [21:50:14] [02miraheze/services] 07MirahezeSSLBot 03e7adfb4 - BOT: Updating services config for wikis [21:54:23] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±2] 13https://git.io/fjLZC [21:54:25] [02miraheze/mediawiki] 07paladox 0379a08c6 - Update PageForms and Cargo [22:00:46] [02miraheze/DataDump] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLZ0 [22:00:47] [02miraheze/DataDump] 07paladox 03a13c4ba - Allow users to delete dumps even if they failed Change-Id: I96df8f08f74cd8d0f4358102635d7cb56ae70c61 [22:01:16] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjLZz [22:01:17] [02miraheze/mediawiki] 07paladox 03a2359e8 - Update DataDump [22:03:31] [02miraheze/DataDump] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLZg [22:03:33] [02miraheze/DataDump] 07paladox 03fed2fcc - Allow users to delete dumps even if they failed part 2 Change-Id: I2f73b7b2e0fbf579dfad8f83e961232bfacb3f0e [22:03:55] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjLZ2 [22:03:57] [02miraheze/mediawiki] 07paladox 03bf9e27c - Update DD [22:39:30] [02miraheze/DataDump] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLnv [22:39:32] [02miraheze/DataDump] 07paladox 03d323663 - Fix syntax error in DataDumpGenerateJob.php Change-Id: Ie9cffa653b38fe0333e2aea6778df1a496e895af [22:39:57] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjLnf [22:39:59] [02miraheze/mediawiki] 07paladox 03dec3032 - Update DD [22:58:28] [02miraheze/DataDump] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLnn [22:58:30] [02miraheze/DataDump] 07paladox 03c72457f - Fix syntax part 2 Change-Id: Ie3ff7b389cc2354c47e78c737900c4d8be6d864c [22:58:56] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjLnC [22:58:58] [02miraheze/mediawiki] 07paladox 03cadd60a - Update DD [23:05:06] [02miraheze/DataDump] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLnl [23:05:08] [02miraheze/DataDump] 07paladox 037722064 - Fix command Change-Id: I15495002bfc61e12659f85c7d291a1c061025804 [23:05:34] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjLn8 [23:05:36] [02miraheze/mediawiki] 07paladox 03446c417 - Update DD [23:09:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLn0 [23:09:33] [02miraheze/mw-config] 07paladox 0347fb192 - Use wgDBname for $wgDataDumpDirectory [23:12:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLnu [23:12:38] [02miraheze/mw-config] 07paladox 03f6131c8 - Fix path in $wgDataDumpDirectory [23:13:05] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 39604 MB (10% inode=96%); [23:21:05] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 45114 MB (12% inode=96%); [23:32:48] [02miraheze/DataDump] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLnw [23:32:49] [02miraheze/DataDump] 07paladox 03c4facec - Try to create directory if it does not exist Change-Id: Ie9f1428129657fd9c34054dc7fe462c68fc7a519 [23:33:15] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjLnr [23:33:16] [02miraheze/mediawiki] 07paladox 03baf3b46 - Update DD [23:38:42] [02miraheze/DataDump] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLnP [23:38:44] [02miraheze/DataDump] 07paladox 03985cc8f - Switch backend->create to backend->prepare Change-Id: I16b3675199b439052fdc25b31bd9585729400507 [23:39:05] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjLnX [23:39:06] [02miraheze/mediawiki] 07paladox 03f4203dc - Update DD [23:49:05] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 38428 MB (10% inode=96%); [23:50:40] [02miraheze/DataDump] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLn5 [23:50:41] [02miraheze/DataDump] 07paladox 034121506 - Fix "datadump-generated-error"" i18n Change-Id: I584cce484f71dc310eb05f059f18a2471219af3f [23:51:10] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjLnd [23:51:12] [02miraheze/mediawiki] 07paladox 03951f008 - Update DD [23:52:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjLnF [23:52:38] [02miraheze/mw-config] 07paladox 0314d6551 - Update LocalExtensions.php [23:54:38] !log running lc on mw* [23:54:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:55:05] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 44932 MB (12% inode=96%);