[00:02:48] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:02:50] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:14] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:16] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:22] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [00:03:24] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [00:03:34] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [00:03:39] uhhhh [00:03:56] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:34] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [00:04:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [00:04:38] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [00:04:52] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:05:14] uh [00:05:18] okay... [00:05:22] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [00:06:33] any idea what's up? [00:06:38] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [00:07:30] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:07:34] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 57% [00:07:40] could it have been php-fpm? [00:08:28] looking in the logs it looks like php-fpm maybe [00:08:34] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [00:08:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:08:44] [pool www] child 15011 exited with code 0 after 973.333654 seconds from start [00:08:53] im glad we have some fallback so it restarts the child [00:09:23] otherwise we may have had a longer downtime [00:09:34] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:11:14] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [00:11:16] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [00:11:34] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 1% [00:11:50] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [00:12:46] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:50] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:14:52] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:48:46] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 470 [11:06:46] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [12:51:27] hey paladox, u there? [12:52:08] sup btw :) [13:00:23] sysadmin_wannabe: hi [13:43:43] Hi [13:44:27] hey [13:45:00] hi sysadmin_wannabe [13:56:54] sup Wiki1776, do you have a wiki on miraheze? if so, please share :) [14:13:41] am I the only one with Visual Editor problems? [14:14:08] i get unknown error/parser error [14:26:47] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 6025 [14:38:01] sysadmin_wannabe: hi [14:57:17] hey [15:42:02] !log php dumpBackup.php --full --include-files --uploads --output=gzip:/home/paladox/wiki2.xml.gz --wiki=checkinwiki on mw2 [15:42:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:55:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHmh [15:55:13] [02miraheze/services] 07MirahezeSSLBot 03eb7a9cd - BOT: Updating services config for wikis [16:02:30] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=psiconautawiki --report=1 /home/paladox/wiki2.xml on mw2 [16:02:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:18:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHOC [16:18:05] [02miraheze/puppet] 07paladox 0360df238 - Install php7.2-zip [16:43:08] PROBLEM - misc3 Puppet on misc3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [16:50:45] PROBLEM - misc3 Parsoid on misc3 is CRITICAL: connect to address 185.52.1.144 and port 8142: Connection refused [16:57:47] known [17:27:29] RECOVERY - misc3 Parsoid on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.144 port 8142 [17:38:27] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [17:38:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fxHZ0 [17:38:30] [02miraheze/puppet] 07paladox 038e1cece - Update parsoid to use the git repo Instead of using the package lets use the git repo. [17:38:31] [02puppet] 07paladox opened pull request 03#883: Update parsoid to use the git repo - 13https://git.io/fxHZE [17:42:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/fxHZK [17:42:22] [02miraheze/puppet] 07paladox 03849a095 - Create parsoid.systemd.erb [17:42:23] [02puppet] 07paladox synchronize pull request 03#883: Update parsoid to use the git repo - 13https://git.io/fxHZE [17:43:10] [02puppet] 07paladox synchronize pull request 03#883: Update parsoid to use the git repo - 13https://git.io/fxHZE [17:43:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fxHZP [17:43:13] [02miraheze/puppet] 07paladox 0329868a1 - Update init.pp [17:44:05] [02puppet] 07paladox closed pull request 03#883: Update parsoid to use the git repo - 13https://git.io/fxHZE [17:44:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fxHZ1 [17:44:08] [02miraheze/puppet] 07paladox 0336ff006 - Update parsoid to use the git repo (#883) * Update parsoid to use the git repo Instead of using the package lets use the git repo. * Create parsoid.systemd.erb * Update init.pp [17:44:10] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [17:44:11] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [17:45:08] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:45:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHZy [17:45:52] [02miraheze/puppet] 07paladox 0324c36b6 - Update config.yaml [17:51:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHZA [17:51:21] [02miraheze/puppet] 07paladox 03ab53d4f - Update init.pp [17:52:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHZx [17:52:11] [02miraheze/puppet] 07paladox 03ad27a18 - Update init.pp [17:57:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fxHn3 [17:57:23] [02miraheze/puppet] 07paladox 0330435c1 - Fix parsoid logs [17:57:24] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [17:57:26] [02puppet] 07paladox opened pull request 03#884: Fix parsoid logs - 13https://git.io/fxHns [17:58:59] [02puppet] 07paladox synchronize pull request 03#884: Fix parsoid logs - 13https://git.io/fxHns [17:59:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fxHnW [17:59:02] [02miraheze/puppet] 07paladox 03d9ef5b4 - Update init.pp [17:59:12] [02puppet] 07paladox synchronize pull request 03#884: Fix parsoid logs - 13https://git.io/fxHns [17:59:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fxHn8 [17:59:15] [02miraheze/puppet] 07paladox 0303718c2 - Update init.pp [18:02:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/fxHnV [18:02:18] [02miraheze/puppet] 07paladox 033ac327a - Create parsoid.rsyslog.conf [18:02:20] [02puppet] 07paladox synchronize pull request 03#884: Fix parsoid logs - 13https://git.io/fxHns [18:02:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/fxHnw [18:02:57] [02miraheze/puppet] 07paladox 03f16a729 - Create logrotate.conf [18:02:58] [02puppet] 07paladox synchronize pull request 03#884: Fix parsoid logs - 13https://git.io/fxHns [18:03:08] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:03:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fxHn6 [18:03:45] [02miraheze/puppet] 07paladox 0318d3352 - Update parsoid.systemd.erb [18:03:47] [02puppet] 07paladox synchronize pull request 03#884: Fix parsoid logs - 13https://git.io/fxHns [18:03:53] [02puppet] 07paladox closed pull request 03#884: Fix parsoid logs - 13https://git.io/fxHns [18:03:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±2] 13https://git.io/fxHnX [18:03:56] [02miraheze/puppet] 07paladox 030b6290d - Fix parsoid logs (#884) * Fix parsoid logs * Update init.pp * Update init.pp * Create parsoid.rsyslog.conf * Create logrotate.conf * Update parsoid.systemd.erb [18:03:57] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:03:59] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [18:06:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHn7 [18:06:08] [02miraheze/puppet] 07paladox 032037dfa - fix duplicate error [18:08:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHnA [18:08:25] [02miraheze/puppet] 07paladox 032c2bb22 - fix syntax error [18:10:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHcJ [18:10:02] [02miraheze/puppet] 07paladox 033b9cbc7 - Update init.pp [18:11:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHck [18:11:19] [02miraheze/puppet] 07paladox 035f7afe6 - fix [18:12:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHcZ [18:13:01] [02miraheze/puppet] 07paladox 032c25ef4 - Update parsoid.systemd.erb [18:13:08] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:13:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHcC [18:13:45] [02miraheze/puppet] 07paladox 037442dd6 - Update parsoid.rsyslog.conf [18:20:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHcr [18:20:16] [02miraheze/services] 07MirahezeSSLBot 03ea1d579 - BOT: Updating services config for wikis [18:25:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHcH [18:25:13] [02miraheze/services] 07MirahezeSSLBot 03ee0de62 - BOT: Updating services config for wikis [18:46:46] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [18:50:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHWe [18:50:15] [02miraheze/services] 07MirahezeSSLBot 0315b382c - BOT: Updating services config for wikis [19:15:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHlc [19:15:14] [02miraheze/services] 07MirahezeSSLBot 031efde83 - BOT: Updating services config for wikis [19:20:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHlz [19:20:16] [02miraheze/services] 07MirahezeSSLBot 03881779d - BOT: Updating services config for wikis [19:30:17] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxH8v [19:30:18] [02miraheze/services] 07MirahezeSSLBot 03aab37d0 - BOT: Updating services config for wikis [19:40:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxH8z [19:40:14] [02miraheze/services] 07MirahezeSSLBot 03fa81ed0 - BOT: Updating services config for wikis [19:45:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxH8Q [19:45:13] [02miraheze/services] 07MirahezeSSLBot 03fa6bdd4 - BOT: Updating services config for wikis [19:48:47] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 6371 [20:00:44] !log php importImages.php /home/paladox/mnt --search-recursively --wiki=psiconautawiki on mw2 [20:00:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:05:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxH4E [20:05:13] [02miraheze/services] 07MirahezeSSLBot 03cdf6a4a - BOT: Updating services config for wikis [20:09:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxH4w [20:09:07] [02miraheze/mw-config] 07paladox 03d1ec4ea - Remove calexitwiki from LS.php The wiki was deleted so it no longer needs to be in here [20:25:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHBg [20:25:14] [02miraheze/services] 07MirahezeSSLBot 03edbf15d - BOT: Updating services config for wikis [20:30:16] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHB7 [20:30:17] [02miraheze/services] 07MirahezeSSLBot 038992e37 - BOT: Updating services config for wikis [21:20:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHEv [21:20:15] [02miraheze/services] 07MirahezeSSLBot 03a20c292 - BOT: Updating services config for wikis [21:59:12] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHuD [21:59:13] [02miraheze/ManageWiki] 07paladox 03b8e52a7 - Fix deleting cdb file when wiki is deleted [21:59:55] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fxHu9 [21:59:56] [02miraheze/mediawiki] 07paladox 0322b327d - Update MW [22:02:32] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_32 [+0/-0/±6] 13https://git.io/fxHuN [22:02:34] [02miraheze/mediawiki] 07Krinkle 03c16a778 - resoureloader: Restore support for plain callbacks in mediawiki.base's RLQ Follows-up dec800968e, which moved the processing of callbacks that require 'modules' from startup.js to mediawiki.base.js. In doing so, it made an incorrect assumption. It assumed that the simple signature of RLQ.push(Function) is not needed after 'mediawiki.base' loads. It is true that [22:02:34] RLQ.push() is mostly an internal interface, and we only use it from within the HTML output, and that once the async pipeline has finished and startup.js has processed the simple callbacks, only calls with secondary signatures remain in the queue. But, while it is true that we don't use RLQ.push() outside the HTML, it is not true that the HTML will fully load and execute inline scripts before any of the async scripts [22:02:34] execute. As such, the call to RLQ.push() in the HTML footer was sometimes being ignored because 'mediawiki.base' had already loaded by now. Bug: T208093 Change-Id: I25012a2c6f41968b1b4f85614a3bc0416512d530 (cherry picked from commit cb7e0ea507c55b49fd22f734fbfe048fe87d5557) [22:02:35] [02miraheze/mediawiki] 07paladox 034336880 - Merge branch 'REL1_32' of https://github.com/wikimedia/mediawiki into REL1_32 [22:02:39] Title: [ GitHub - wikimedia/mediawiki: 🌻 The collaborative editing software that runs Wikipedia. This is a mirror from gerrit.wikimedia.org. See https://www.mediawiki.org/wiki/Developer_access for contributing ] - github.com [22:03:13] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fxHuA [22:03:14] [02miraheze/mediawiki] 07paladox 03e7bf250 - Update MW [22:04:20] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHzJ [22:04:22] [02miraheze/ManageWiki] 07paladox 0376e9a61 - Use unlink instead of exec [22:04:55] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fxHzI [22:04:57] [02miraheze/mediawiki] 07paladox 030d4ec6b - Update MW [22:06:46] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [22:07:22] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fxHzm [22:07:24] [02miraheze/mediawiki] 07paladox 0391e1b49 - Update MW [22:20:16] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHz5 [22:20:18] [02miraheze/services] 07MirahezeSSLBot 037836ffb - BOT: Updating services config for wikis [22:28:48] !log php updateExternalLinks.php --wiki=sdiywiki on mw2 [22:28:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:29:16] realistically should be ran on every wiki [22:34:25] Yep [22:34:33] I'll get to the cron [22:35:02] JohnLewis though it removes existing entrys [22:35:03] Dropping all existing recorded entries [22:35:11] yes [22:35:56] JohnLewis so if we run it every 24 hours, there's a chance it may run it twice. [22:36:13] huh? [22:36:30] JohnLewis well we have 2,700+ wiki's [22:36:37] 24 is too frequent for the amount of HTTP requests [22:36:40] the larger ones att may take a hour or 2 to complete [22:36:45] oh [22:36:54] JohnLewis what's your suggested time window then? [22:36:57] every week? [22:37:14] for a total drop, yes or every few days [22:37:21] ok [22:42:22] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [22:42:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fxHgg [22:42:25] [02miraheze/puppet] 07paladox 035673d9e - Add RotenLinks updateExternalLinks.php cron [22:42:27] [02puppet] 07paladox opened pull request 03#885: Add RotenLinks updateExternalLinks.php cron - 13https://git.io/fxHg2 [22:42:30] JohnLewis review ^^ please? [22:43:15] theres a 1 day difference [22:44:00] hmm [22:44:03] ah [22:44:32] JohnLewis in the weekday field i would do '*/2' right? [22:45:00] I mean you could or you can just get rid of 0 [22:45:02] or 6 [22:45:23] ok [22:45:39] [02puppet] 07paladox synchronize pull request 03#885: Add RotenLinks updateExternalLinks.php cron - 13https://git.io/fxHg2 [22:45:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fxHgK [22:45:42] [02miraheze/puppet] 07paladox 0398a1461 - Update cron.pp [22:45:48] based on your feedback, i've done 0 and 4 [22:46:03] okay [22:46:18] [02puppet] 07paladox closed pull request 03#885: Add RotenLinks updateExternalLinks.php cron - 13https://git.io/fxHg2 [22:46:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxHgi [22:46:21] [02miraheze/puppet] 07paladox 03ae5168f - Add RotenLinks updateExternalLinks.php cron (#885) * Add RotenLinks updateExternalLinks.php cron * Update cron.pp [22:46:23] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [22:46:24] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [22:53:19] work? [22:59:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxH2O [22:59:59] [02miraheze/puppet] 07paladox 0362fd0c3 - Only run "RotenLinks updateExternalLinks.php on all wikis" on dump hosts [23:00:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxH23 [23:00:12] [02miraheze/puppet] 07paladox 03bc369c4 - Update cron.pp [23:00:38] uh [23:00:46] that cron isn't a dump [23:01:01] it doesn't meet that files purpose [23:01:18] JohnLewis i only want it to run on one of the mw* [23:01:21] not all of them :) [23:01:24] hmm [23:01:27] * paladox reverts [23:01:34] yes... but it's not a dump [23:05:02] [02miraheze/puppet] 07paladox pushed 033 commits to 03master [+0/-0/±3] 13https://git.io/fxH2c [23:05:04] [02miraheze/puppet] 07paladox 03e3f5258 - Revert "Update cron.pp" This reverts commit bc369c48cc7798f8d5a3c400e5797c8855002556. [23:05:05] [02miraheze/puppet] 07paladox 03970bebc - Revert "Only run "RotenLinks updateExternalLinks.php on all wikis" on dump hosts" This reverts commit 62fd0c3fac34b6ef0513aa50acf90a039241be3d. [23:05:07] [02miraheze/puppet] 07paladox 03dd5ab0d - Only run "RotenLinks updateExternalLinks.php on all wikis" cron on mw1 [23:05:09] JohnLewis ^^ [23:05:23] uh [23:05:42] we never hard code FQDNs [23:05:47] hmm? [23:06:11] if you're using a servers name to hardcode, you're doing something wrong [23:06:52] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:06:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fxH2R [23:06:55] [02miraheze/puppet] 07paladox 03fe06942 - Use hiera to decide where "RotenLinks updateExternalLinks.php on all wikis" cron runs [23:06:57] [02puppet] 07paladox opened pull request 03#886: Use hiera to decide where "RotenLinks updateExternalLinks.php on all … - 13https://git.io/fxH20 [23:07:01] I'll make your life easier, there's a whole manifest dedicated to a job runner [23:07:06] [02puppet] 07paladox edited pull request 03#886: Use hiera to decide where "RotenLinks updateExternalLinks.php on all wikis" cron runs - 13https://git.io/fxH20 [23:07:11] [02puppet] 07paladox edited pull request 03#886: Use hiera to decide where "RotenLinks updateExternalLinks.php on all wikis" cron runs - 13https://git.io/fxH20 [23:07:22] [02puppet] 07JohnFLewis closed pull request 03#886: Use hiera to decide where "RotenLinks updateExternalLinks.php on all wikis" cron runs - 13https://git.io/fxH20 [23:07:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fxH2u [23:07:33] [02miraheze/puppet] 07paladox 03bd52abf - Update mw1.yaml [23:08:11] ok [23:08:40] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [23:08:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxH2o [23:08:43] [02miraheze/puppet] 07paladox 03fae71f8 - Move RottenLinks cron to jobrunner class [23:08:44] [02puppet] 07paladox opened pull request 03#887: Move RottenLinks cron to jobrunner class - 13https://git.io/fxH2K [23:09:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxH2i [23:09:20] [02miraheze/puppet] 07paladox 038260b6d - Update jobrunner.pp [23:09:22] [02puppet] 07paladox synchronize pull request 03#887: Move RottenLinks cron to jobrunner class - 13https://git.io/fxH2K [23:09:26] [02puppet] 07paladox closed pull request 03#887: Move RottenLinks cron to jobrunner class - 13https://git.io/fxH2K [23:09:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fxH2P [23:09:29] [02miraheze/puppet] 07paladox 0393e1951 - Move RottenLinks cron to jobrunner class (#887) * Move RottenLinks cron to jobrunner class * Update jobrunner.pp [23:09:31] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [23:09:32] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [23:09:33] JohnLewis done [23:10:08] good [23:12:23] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:13:07] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:13:17] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:14:15] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:14:21] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:15:07] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:16:15] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:17:17] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures