[00:02:30] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:02:48] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:02:54] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:00] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:06] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:08] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:20] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:22] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:28] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:48] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:50] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:58] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:06] PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:16] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:11:06] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [00:11:08] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [00:11:20] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [00:11:28] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [00:11:48] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [00:11:50] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [00:11:56] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [00:12:07] RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:47] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:53] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:13:01] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:14:17] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:14:31] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:20:01] [02RottenLinks] 07JohnFLewis created branch 03master - 13https://git.io/fxBrr [02:20:02] [02miraheze/RottenLinks] 07JohnFLewis pushed 031 commit to 03master [+10/-0/±0] 13https://git.io/fxBro [02:20:04] [02miraheze/RottenLinks] 07JohnFLewis 0371ac80a - initial commit of code [02:23:11] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_31 [+1/-0/±1] 13https://git.io/fxBrS [02:23:13] [02miraheze/mediawiki] 07JohnFLewis 03a16f12c - add RL [02:25:33] [02miraheze/RottenLinks] 07JohnFLewis pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/fxBrQ [02:25:35] [02miraheze/RottenLinks] 07JohnFLewis 03b3ab298 - move into dir [02:26:01] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fxBr5 [02:26:03] [02miraheze/mediawiki] 07JohnFLewis 033379820 - update RL [02:43:01] [02miraheze/RottenLinks] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±4] 13https://git.io/fxBoL [02:43:02] [02miraheze/RottenLinks] 07JohnFLewis 03c01c793 - remove PRIMARY KEY [02:43:04] [02miraheze/RottenLinks] 07JohnFLewis 03ae9122f - general code fixes [02:43:27] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fxBot [02:43:28] [02miraheze/mediawiki] 07JohnFLewis 03ec5d5d1 - update RL [02:45:58] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/fxBoO [02:45:59] [02miraheze/mw-config] 07JohnFLewis 03428b764 - add RL config [02:46:38] !log creating rottenlinks table on all wikis [02:46:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:57:37] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.85, 1.52, 0.85 [02:58:30] JohnLewis: are you running that on test1? :P [02:58:42] yes [02:58:59] Ok [02:59:35] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_pull_MediaWiki core] [02:59:37] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.09, 1.34, 0.86 [03:24:27] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.80, 1.70, 1.47 [03:28:25] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.54, 1.70, 1.53 [03:34:22] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.87, 1.68, 1.54 [03:38:20] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.82, 1.51, 1.52 [05:38:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [05:38:10] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:38:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [05:40:02] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [05:40:06] RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Thu 03 Jan 2019 10:55:37 AM GMT +0000. [05:40:36] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:18:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [06:18:35] PROBLEM - oecumene.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:19:23] PROBLEM - wiki.pupilliam.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:19:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [06:20:29] RECOVERY - oecumene.org - LetsEncrypt on sslhost is OK: OK - Certificate 'oecumene.org' will expire on Mon 24 Dec 2018 04:23:18 PM GMT +0000. [06:20:45] PROBLEM - meregos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:21:21] RECOVERY - wiki.pupilliam.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pupilliam.com' will expire on Sat 29 Dec 2018 07:42:40 PM GMT +0000. [06:22:45] RECOVERY - meregos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'meregos.com' will expire on Wed 31 Oct 2018 05:26:54 PM GMT +0000. [06:23:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:24:17] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [09:26:59] PROBLEM - bacula1 Bacula Static Lizardfs1 on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:27:23] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:28:59] PROBLEM - bacula1 Bacula Static Lizardfs1 on bacula1 is WARNING: WARNING: Full, 754423 files, 79.37GB, 2018-09-23 18:21:00 (2.8 weeks ago) [09:29:23] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 1568 files, 1.760MB, 2018-09-30 00:08:00 (1.9 weeks ago) [13:39:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxBNW [13:39:24] [02miraheze/puppet] 07paladox 030ee2a2b - motd: make priorities explicitly integers Follows https://github.com/wikimedia/puppet/commit/82ebd50da644fa7a4046a039ecea1390def70041 [13:39:25] Title: [ puppet: make all priorities explicitly integers · wikimedia/puppet@82ebd50 · GitHub ] - github.com [13:43:18] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [13:43:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxBNr [13:43:21] [02miraheze/puppet] 07paladox 03d476b54 - mediawiki: drop support for php7.0 [13:43:22] [02puppet] 07paladox opened pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:46:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxBNM [13:46:11] [02miraheze/puppet] 07paladox 03c3b2d41 - php: Add request_terminate_timeout to www-7.2.conf [13:46:12] [02puppet] 07paladox synchronize pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:49:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxBN7 [13:49:26] [02miraheze/puppet] 07paladox 0347f1e63 - php: Add more php package from the mediawiki class This will allow us to include the php class in mediawiki. [13:49:28] [02puppet] 07paladox synchronize pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:50:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxBNd [13:50:22] [02miraheze/puppet] 07paladox 039ea6dc4 - Update php7_2.pp [13:50:24] [02puppet] 07paladox synchronize pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:50:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-1/±0] 13https://git.io/fxBNb [13:50:34] [02miraheze/puppet] 07paladox 0356aec63 - Rename php7_2.pp to php.pp [13:50:36] [02puppet] 07paladox synchronize pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:50:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-1/±0] 13https://git.io/fxBNN [13:50:43] [02miraheze/puppet] 07paladox 03fd33a6e - Delete php7.pp [13:50:44] [02puppet] 07paladox synchronize pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:51:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxBAJ [13:51:50] [02miraheze/puppet] 07paladox 03eecfa5b - Update nginx.pp [13:51:51] [02puppet] 07paladox synchronize pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:52:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxBAT [13:52:19] [02miraheze/puppet] 07paladox 03aff91ac - Update monitoring.pp [13:52:21] [02puppet] 07paladox synchronize pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:52:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-1/±0] 13https://git.io/fxBAk [13:52:35] [02miraheze/puppet] 07paladox 03a32213a - Delete www-7.2.conf [13:52:37] [02puppet] 07paladox synchronize pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:52:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-1/±0] 13https://git.io/fxBAt [13:52:41] [02miraheze/puppet] 07paladox 037720119 - Delete www-7.0.conf [13:52:43] [02puppet] 07paladox synchronize pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:52:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-1/±0] 13https://git.io/fxBAq [13:52:48] [02miraheze/puppet] 07paladox 031e629d2 - Delete php-fpm-7.0.conf [13:52:50] [02puppet] 07paladox synchronize pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:52:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-1/±0] 13https://git.io/fxBAm [13:52:55] [02miraheze/puppet] 07paladox 03ac4b36b - Delete php-7.0.ini [13:52:56] [02puppet] 07paladox synchronize pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:53:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxBAs [13:53:33] [02miraheze/puppet] 07paladox 03281b0b2 - Update php-7.2.ini [13:53:35] [02puppet] 07paladox synchronize pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:54:53] PROBLEM - phab.miraheze.wiki on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxBAc [13:55:28] [02miraheze/puppet] 07paladox 03f07e2e7 - Update test1.yaml [13:55:29] [02puppet] 07paladox synchronize pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:55:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxBA8 [13:55:43] [02miraheze/puppet] 07paladox 03e7bcddc - Update common.yaml [13:55:44] [02puppet] 07paladox synchronize pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:58:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-6/±8] 13https://git.io/fxBAo [13:58:20] [02miraheze/puppet] 07paladox 036fb277d - mediawiki: drop support for php7.0 (#880) * mediawiki: drop support for php7.0 * php: Add request_terminate_timeout to www-7.2.conf * php: Add more php package from the mediawiki class This will allow us to include the php class in mediawiki. * Update php7_2.pp * Rename php7_2.pp to php.pp * Delete php7.pp * Update nginx.pp * Update monitoring.pp * Delete www-7.2.conf [13:58:20] * Delete www-7.0.conf * Delete php-fpm-7.0.conf * Delete php-7.0.ini * Update php-7.2.ini * Update test1.yaml * Update common.yaml [13:58:22] [02puppet] 07paladox closed pull request 03#880: mediawiki: drop support for php7.0 - 13https://git.io/fxBNK [13:58:23] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [13:58:25] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [13:59:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxBAM [13:59:55] [02miraheze/puppet] 07paladox 034d94a64 - Fix duplicate service [14:01:35] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 20 seconds ago with 2 failures. Failed resources (up to 3 shown): Package[php7.2-dev],Package[php7.2-gettext] [14:03:36] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:03:53] [02puppet] 07paladox closed pull request 03#864: Stunnel: Enforce tlsv1.2+ - 13https://git.io/fxUcQ [14:03:55] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [14:03:57] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [14:09:24] RECOVERY - phab.miraheze.wiki on sslhost is OK: OK - Certificate 'phab.miraheze.wiki' will expire on Mon 07 Jan 2019 06:29:53 PM GMT +0000. [14:34:16] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5720 [16:48:14] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [17:35:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxRkj [17:35:13] [02miraheze/services] 07MirahezeSSLBot 03ddde8f7 - BOT: Updating services config for wikis [17:37:33] [02miraheze/RottenLinks] 07MacFan4000 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/fxRIJ [17:37:34] [02miraheze/RottenLinks] 07MacFan4000 03f70aa9a - Create .travis.yml [17:38:18] [02miraheze/RottenLinks] 07MacFan4000 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/fxRII [17:38:19] [02miraheze/RottenLinks] 07MacFan4000 037ce702f - Create .trais.phplint.sh [17:39:12] miraheze/RottenLinks/master/f70aa9a - MacFan4000 The build failed. https://travis-ci.org/miraheze/RottenLinks/builds/441059165 [17:39:37] miraheze/RottenLinks/master/7ce702f - MacFan4000 The build failed. https://travis-ci.org/miraheze/RottenLinks/builds/441059300 [17:40:07] [02miraheze/RottenLinks] 07MacFan4000 pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/fxRIZ [17:40:08] [02miraheze/RottenLinks] 07MacFan4000 03c8479b1 - Rename .trais.phplint.sh to .travis.phplint.sh [17:41:22] miraheze/RottenLinks/master/c8479b1 - MacFan4000 The build was fixed. https://travis-ci.org/miraheze/RottenLinks/builds/441059696 [17:46:07] RobertSterbal: it has been security reviewed by us [17:46:11] https://phabricator.miraheze.org/T2385#45063 [17:46:12] Title: [ ⚓ T2385 Enable extensions on unionnorteamericana.miraheze.org ] - phabricator.miraheze.org [17:46:35] We can’t install it until the patch linked in the comment is merged [19:19:49] [02puppet] 07JohnFLewis commented on commit 030ee2a2b538b8c9a4fdf8af513a1db2ee17f08b2e - 13https://git.io/fxRm6 [21:30:22] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5744 [21:40:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxRZl [21:40:14] [02miraheze/services] 07MirahezeSSLBot 0319ed597 - BOT: Updating services config for wikis [22:30:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxRcf [22:30:17] [02miraheze/services] 07MirahezeSSLBot 03bdb0e1d - BOT: Updating services config for wikis