[00:01:42] <Not-2d3>	 [02miraheze/mediawiki] 07paladox pushed 0322 commits to 03REL1_31 [+0/-0/±39] 13https://git.io/fpWuZ
[00:01:44] <Not-2d3>	 [02miraheze/mediawiki] 07arlolra 03b1baa56 - <ins>/<del> elements can be phrasing or flow The changes to the parserTests.txt highlight the differing opinions that doBlockLevels and Remex had on whether these should be paragraph wrapped. Since the only time they wouldn't have been was when found on a line with other flow tags, this likely isn't a behaviour that was depended on in practice. And, indeed, the task
[00:01:44] <Not-2d3>	 describes this as a bug. A sampling of pages from an insource:/\<(ins|del)\>/ search on wiki bears this out. Bug: T17491 Change-Id: I311da777a63aa3c45013f2cfc090be35a022497e (cherry picked from commit 5a7f860b7859146d006d09c29f542be835165870)
[00:01:45] <Not-2d3>	 [02miraheze/mediawiki] 07tstarling 0325b0bce - RemexCompatMunger: Don't call endTag() in case B/b This was naïve, the linked bug documents a case where endTag() was called despite children of the p-wrap still being in TreeBuilder's stack. Instead, wait for the parent of the p-wrap to have endTag() called on it, I've submitted a patch which will clean up the node in that case. Bug: T200827 Change-Id:
[00:01:45] <Not-2d3>	 I34694813eace9cadabf2db8f9ccca83d1368cfad (cherry picked from commit 10c8cfea305ec1d450b16ad54ebddb5f910016f4)
[00:01:47] <Not-2d3>	 [02miraheze/mediawiki] 07legoktm 03c26c3ba - Upgrade wikimedia/remex-html to 2.0.0 Depends-On: Ie13945649314853cbd5707363f3a10da55752743 Change-Id: Ib6c8aaa797c128c273cde8095eb0bb1527fc0e21 (cherry picked from commit 9cac6c5645cbde9c48a4fac43c8dfdd977bb200f)
[00:01:48] <Not-2d3>	 [02miraheze/mediawiki] ... and 19 more commits.
[00:02:45] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:02:53] <icinga-miraheze>	 PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:03:06] <icinga-miraheze>	 PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:03:08] <icinga-miraheze>	 PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:03:16] <icinga-miraheze>	 PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:03:40] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:04:22] <icinga-miraheze>	 PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:04:30] <icinga-miraheze>	 PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:05:22] <Not-2d3>	 [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_32 [+0/-0/±2] 13https://git.io/fpWur
[00:05:24] <Not-2d3>	 [02miraheze/mediawiki] 07paladox 0356a1c58 - Use $revQuery['joins'] in query in populateSearchIndex Bug: T209885 Change-Id: Iaf53179535030064788eb107c4ebdd398ed306e4
[00:05:26] <Not-2d3>	 [02miraheze/mediawiki] 07paladox 03e72a786 - Merge branch 'REL1_32' of https://github.com/wikimedia/mediawiki into REL1_32
[00:05:26] <ZppixBot>	 Title: [ GitHub - wikimedia/mediawiki: 🌻 The collaborative editing software that runs Wikipedia. This is a mirror from gerrit.wikimedia.org. See https://www.mediawiki.org/wiki/Developer_access for contributing ] - github.com
[00:11:08] <icinga-miraheze>	 RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures  
[00:11:16] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures  
[00:11:38] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures  
[00:12:22] <icinga-miraheze>	 RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[00:12:32] <icinga-miraheze>	 RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures  
[00:12:44] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[00:12:52] <icinga-miraheze>	 RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[00:13:04] <icinga-miraheze>	 RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures  
[02:07:40] <paladox>	 !log sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki loginwiki on mw*
[02:07:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[04:50:12] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpWPm
[04:50:13] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot 03a1b0b5e - BOT: Updating services config for wikis
[06:09:51] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is CRITICAL: CRITICAL: Diff, 7587 files, 48.46MB, 2018-10-21 06:08:00 (4.3 weeks ago)  
[07:34:58] <icinga-miraheze>	 PROBLEM - www.radviser.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[07:36:54] <icinga-miraheze>	 RECOVERY - www.radviser.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.radviser.org' will expire on Sat 09 Feb 2019 06:56:36 PM GMT +0000.  
[11:35:11] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fplTB
[11:35:13] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot 035a74527 - BOT: Updating services config for wikis
[16:08:55] <JohnLewis>	 !log MariaDB [atlasinternationalwiki]> UPDATE actor SET actor_name = "Ohallora~old" WHERE actor_user = "1";
[16:08:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:09:33] <JohnLewis>	 paladox: ^
[16:09:48] <JohnLewis>	 please do not run the removeUnusedAccounts.php script ever again, thanks
[16:10:00] <JohnLewis>	 and especially without logging it
[16:10:00] <paladox>	 Ok
[16:17:34] <JohnLewis>	 !log MariaDB [atlasinternationalwiki]> UPDATE actor SET actor_name = "Void~old" WHERE actor_id = "4";
[16:17:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:18:43] <Not-2d3>	 [02mw-config] 07Amanda-Catherine opened pull request 03#2560: Enable moderation on socioatlaswiki T3806 - 13https://git.io/fplVv
[16:19:43] <Not-2d3>	 [02mw-config] 07JohnFLewis reviewed pull request 03#2560 commit - 13https://git.io/fplVt
[16:20:43] <Not-2d3>	 [02mw-config] 07Amanda-Catherine reviewed pull request 03#2560 commit - 13https://git.io/fplVq
[16:20:52] <travis-ci>	 Amanda-Catherine/mw-config/patch-6/98c7ee4 - Amanda The build passed. https://travis-ci.com/Amanda-Catherine/mw-config/builds/92037321
[16:33:10] <icinga-miraheze>	 PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.95, 1.75, 0.82  
[16:35:08] <icinga-miraheze>	 PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.91, 1.86, 0.98  
[16:35:20] <Not-2d3>	 [02mw-config] 07MacFan4000 reviewed pull request 03#2560 commit - 13https://git.io/fplrI
[16:37:08] <icinga-miraheze>	 RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 0.27, 1.25, 0.86  
[16:41:09] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is UNKNOWN: NRPE: Unable to read output  
[16:41:25] <Not-2d3>	 [02mw-config] 07Amanda-Catherine synchronize pull request 03#2560: Enable moderation on socioatlaswiki T3806 - 13https://git.io/fplVv
[16:43:11] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 1637 files, 1.913MB, 2018-11-11 00:05:00 (1.4 weeks ago)  
[16:46:17] <Not-2d3>	 [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fplKo
[16:46:18] <Not-2d3>	 [02miraheze/mw-config] 07MacFan4000 03e63001c - Enable subpages for a namespace on nonciclopediawiki T3768
[16:46:44] <Not-2d3>	 [02mw-config] 07MacFan4000 closed pull request 03#2560: Enable moderation on socioatlaswiki T3806 - 13https://git.io/fplVv
[16:46:45] <Not-2d3>	 [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fplK9
[16:46:47] <Not-2d3>	 [02miraheze/mw-config] 07Amanda-Catherine 03c001a08 - Enable moderation on socioatlaswiki T3806 (#2560) * Enable moderation on socioatlaswiki T3806 * Update LocalSettings.php
[16:48:17] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is UNKNOWN: NRPE: Unable to read output  
[16:48:25] <Not-2d3>	 [02mw-config] 07The-Voidwalker opened pull request 03#2561: fix - 13https://git.io/fplKx
[16:48:36] <travis-ci>	 miraheze/mw-config/master/c001a08 - Amanda The build was broken. https://travis-ci.org/miraheze/mw-config/builds/457541250
[16:49:15] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Daemon on bacula1 is CRITICAL: PROCS CRITICAL: 1 process with UID = 110 (bacula)   
[16:49:25] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Static on bacula1 is UNKNOWN: NRPE: Unable to read output  
[16:49:29] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is UNKNOWN: NRPE: Unable to read output  
[16:49:41] <Voidwalker>	 MacFan4000, https://git.io/fplKx
[16:49:42] <ZppixBot>	 Title: [ fix by The-Voidwalker · Pull Request #2561 · miraheze/mw-config · GitHub ] - git.io
[16:50:33] <paladox>	 ignore bacula1 that's me (i was trying to look inside the static backup to see if it worked backing up)
[16:50:47] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Databases db4 on bacula1 is UNKNOWN: NRPE: Unable to read output  
[16:51:01] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output  
[16:51:15] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Daemon on bacula1 is OK: PROCS OK: 2 processes with UID = 110 (bacula)   
[16:51:21] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Static on bacula1 is WARNING: WARNING: Diff, 148627 files, 15.59GB, 2018-10-21 05:22:00 (4.4 weeks ago)  
[16:51:24] <Voidwalker>	 paladox, can you merge the pr?
[16:51:28] <paladox>	 yup
[16:51:29] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 1637 files, 1.913MB, 2018-11-11 00:05:00 (1.4 weeks ago)  
[16:51:40] <Not-2d3>	 [02mw-config] 07paladox closed pull request 03#2561: fix - 13https://git.io/fplKx
[16:51:41] <Not-2d3>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpl6l
[16:51:43] <Not-2d3>	 [02miraheze/mw-config] 07The-Voidwalker 03487691c - fix (#2561)
[16:52:27] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is CRITICAL: CRITICAL: Diff, 7587 files, 48.46MB, 2018-10-21 06:08:00 (4.3 weeks ago)  
[16:52:47] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Databases db4 on bacula1 is OK: OK: Full, 895583 files, 63.73GB, 2018-11-19 23:18:00 (17.6 hours ago)  
[16:53:01] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 651  
[16:53:05] <icinga-miraheze>	 PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down.  mw3  
[16:54:43] <travis-ci>	 miraheze/mw-config/master/487691c - The-Voidwalker The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/457543453
[16:55:03] <icinga-miraheze>	 RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy  
[16:57:30] <Not-2d3>	 [02puppet] 07revi commented on pull request 03#895: Update conduct@ and cocc@ - 13https://git.io/fplit
[16:59:34] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Static on bacula1 is UNKNOWN: NRPE: Unable to read output  
[16:59:52] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is UNKNOWN: NRPE: Unable to read output  
[17:00:46] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is UNKNOWN: NRPE: Unable to read output  
[17:01:28] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Static on bacula1 is WARNING: WARNING: Diff, 148627 files, 15.59GB, 2018-10-21 05:22:00 (4.4 weeks ago)  
[17:01:50] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 1637 files, 1.913MB, 2018-11-11 00:05:00 (1.4 weeks ago)  
[17:02:42] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is CRITICAL: CRITICAL: Diff, 7587 files, 48.46MB, 2018-10-21 06:08:00 (4.4 weeks ago)  
[17:22:02] <Not-2d3>	 [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fplXL
[17:22:03] <Not-2d3>	 [02miraheze/mw-config] 07MacFan4000 03175dfd6 - subpages for all NS on nonciclopediawiki per T3768
[17:23:33] <Not-2d3>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fplXn
[17:23:34] <Not-2d3>	 [02miraheze/mw-config] 07paladox 0323fb10b - Set actor to MIGRATION_NEW for nonciclopediawiki
[17:25:57] <paladox>	 !log ran migrateActor on atlasinternationalwiki to fix flow issue
[17:26:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:31:53] <paladox>	 !log re doing backups for private git and phabstatic
[17:31:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:35:16] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Phabricator Static on bacula1 is OK: OK: Full, 70458 files, 4.192GB, 2018-11-20 17:34:00 (1.2 minutes ago)  
[17:35:42] <icinga-miraheze>	 PROBLEM - db4 Current Load on db4 is WARNING: WARNING - load average: 7.45, 6.76, 4.87  
[17:35:47] <paladox>	 uh
[17:37:38] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Private Git on bacula1 is OK: OK: Full, 1652 files, 1.949MB, 2018-11-20 17:37:00 (37.0 seconds ago)  
[17:41:58] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fplMt
[17:42:00] <Not-2d3>	 [02miraheze/puppet] 07paladox 03f4a998d - Fix check_bacula_static to use the correct backup
[17:43:28] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Static on bacula1 is OK: OK: Full, 2503190 files, 186.2GB, 2018-11-20 16:26:00 (1.3 hours ago)  
[17:45:17] <icinga-miraheze>	 PROBLEM - db4 Current Load on db4 is CRITICAL: CRITICAL - load average: 8.03, 7.52, 6.15  
[17:49:05] <paladox>	 !log restarting mysql on db4
[17:50:24] <paladox>	 that's a emergency restart since the load would have eventually impacted prod.
[17:51:05] <icinga-miraheze>	 PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down.  mw1 mw2 mw3  
[17:51:09] <icinga-miraheze>	 RECOVERY - db4 Current Load on db4 is OK: OK - load average: 2.45, 6.51, 6.37  
[17:51:19] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down.  mw1 mw2 mw3  
[17:51:23] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down.  mw1 mw2 mw3  
[17:51:51] <icinga-miraheze>	 PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 69%  
[17:51:59] <icinga-miraheze>	 PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 89%  
[17:52:45] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb  
[17:53:03] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb  
[17:53:05] <icinga-miraheze>	 RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy  
[17:53:15] <icinga-miraheze>	 PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 74%  
[17:53:19] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy  
[17:53:33] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online  
[17:53:35] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online  
[17:53:49] <icinga-miraheze>	 RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 2%  
[17:53:59] <icinga-miraheze>	 RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 4%  
[17:55:15] <icinga-miraheze>	 PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 54%  
[17:57:16] <icinga-miraheze>	 RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 19%  
[17:57:57] <MacFan4000>	 .weather
[17:57:59] <ZppixBot>	 Yahoo! Weather - Portland, ME, US: Rain And Snow, 1°C (33°F), Humidity: 92%, Fresh breeze 10.7m/s (↙)
[17:58:16] <MacFan4000>	 another day of nonstop snow :|
[18:02:20] <Voidwalker>	 same, except it's 11 degrees colder here
[18:02:48] <Voidwalker>	 or it was earlier, looks like it's warmed up a bit
[18:02:59] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[18:03:10] <icinga-miraheze>	 PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 3.51, 1.85, 0.91  
[18:05:08] <icinga-miraheze>	 PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.41, 1.76, 1.00  
[18:07:08] <icinga-miraheze>	 RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 0.20, 1.19, 0.88  
[18:17:13] <Not-2d3>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fplSd
[18:17:15] <Not-2d3>	 [02miraheze/mediawiki] 07JohnFLewis 031722682 - update CW
[18:20:43] <Not-2d3>	 [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpl9J
[18:20:45] <Not-2d3>	 [02miraheze/CreateWiki] 07JohnFLewis 03aaf01ff - add an if check clause
[18:21:14] <Not-2d3>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fpl9I
[18:21:16] <Not-2d3>	 [02miraheze/mediawiki] 07JohnFLewis 035b8e6e9 - update CW
[18:22:51] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down.  mw2  
[18:23:05] <icinga-miraheze>	 PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 2 backends are down.  mw2 mw3  
[18:23:19] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down.  mw2 mw3  
[18:25:20] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy  
[18:26:46] <JohnLewis>	 !log depool mw2
[18:26:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:26:51] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy  
[18:27:05] <icinga-miraheze>	 RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy  
[19:06:54] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb  
[19:08:03] <icinga-miraheze>	 PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down.  mw1  
[19:10:05] <icinga-miraheze>	 RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy  
[19:12:58] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online  
[19:17:20] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down.  mw2  
[19:17:38] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down.  mw2  
[19:18:04] <icinga-miraheze>	 PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down.  mw2  
[19:24:04] <icinga-miraheze>	 RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy  
[19:25:18] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy  
[19:57:01] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fplNb
[19:57:01] <Not-2d3>	 [02miraheze/puppet] 07paladox 0351c2d3e - Increase upload size to 250M fixes T3797
[19:57:03] <Not-2d3>	 [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS
[19:57:04] <Not-2d3>	 [02puppet] 07paladox opened pull request 03#896: Increase upload size to 250M fixes T3797 - 13https://git.io/fplNA
[20:00:17] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fplAm
[20:00:19] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot 03b93d682 - BOT: Updating services config for wikis
[20:03:31] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fplAP
[20:03:32] <Not-2d3>	 [02miraheze/puppet] 07paladox 0312cc49a - Update nginx.conf.erb
[20:03:34] <Not-2d3>	 [02puppet] 07paladox synchronize pull request 03#896: Increase upload size to 250M fixes T3797 - 13https://git.io/fplNA
[20:04:04] <paladox>	 will deploy in a bit, brb
[20:08:52] <Wiki-1776>	 paladox, https://prnt.sc/lkum11 
[20:08:52] <ZppixBot>	 Title: [ Screenshot by Lightshot ] - prnt.sc
[20:09:48] <Wiki-1776>	 (in destinoswiki, mediatecawiki...)
[20:10:45] <paladox>	 Wiki-1776 works for me https://destinos.miraheze.org/wiki/Especial:PáginasEspeciales
[20:10:46] <ZppixBot>	 Title: [ Páginas especiales - Destinos Wiki ] - destinos.miraheze.org
[20:23:05] <Wiki-1776>	 hum
[21:00:55] <Not-2d3>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp8eb
[21:00:56] <Not-2d3>	 [02miraheze/mw-config] 07paladox 0352683d3 - Increase max file upload size to 250M T3797
[21:01:17] <Not-2d3>	 [02puppet] 07paladox closed pull request 03#896: Increase upload size to 250M fixes T3797 - 13https://git.io/fplNA
[21:01:19] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fp8eA
[21:01:20] <Not-2d3>	 [02miraheze/puppet] 07paladox 031a04773 - Increase upload size to 250M fixes T3797 (#896) * Increase upload size to 250M fixes T3797 * Update nginx.conf.erb
[21:01:22] <Not-2d3>	 [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS
[21:01:23] <Not-2d3>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2
[21:20:02] <Not-2d3>	 [02CreateWiki] 07Southparkfan commented on commit 03aaf01ff159b65c1a8b7d13e79e7cff4ecc99365d - 13https://git.io/fp8f0
[21:20:34] <SPF|Cloud>	 I love CDB btw :)
[21:35:41] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp8J1
[21:35:43] <Not-2d3>	 [02miraheze/puppet] 07paladox 0316afd24 - Remove dumps files before generating them There was a issue where it would not clean or replace the file before it inserted the fresh dump
[21:49:15] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Daemon on bacula1 is CRITICAL: PROCS CRITICAL: 1 process with UID = 110 (bacula)   
[21:49:29] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Databases db4 on bacula1 is UNKNOWN: NRPE: Unable to read output  
[21:49:47] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Static on bacula1 is UNKNOWN: NRPE: Unable to read output  
[21:50:33] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is UNKNOWN: NRPE: Unable to read output  
[21:51:15] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Daemon on bacula1 is OK: PROCS OK: 2 processes with UID = 110 (bacula)   
[21:51:27] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Databases db4 on bacula1 is OK: OK: Full, 895583 files, 63.73GB, 2018-11-19 23:18:00 (22.6 hours ago)  
[21:51:41] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Static on bacula1 is OK: OK: Full, 2503190 files, 186.2GB, 2018-11-20 16:26:00 (5.4 hours ago)  
[21:52:29] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Phabricator Static on bacula1 is OK: OK: Full, 70458 files, 4.192GB, 2018-11-20 17:34:00 (4.3 hours ago)  
[22:01:09] <icinga-miraheze>	 PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 4.83, 2.80, 1.33  
[22:07:02] <MacFan4000>	 !log running jobs on all wikis
[22:07:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:07:10] <icinga-miraheze>	 PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.10, 1.90, 1.46  
[22:08:40] <paladox>	 !log rebooted bacula1
[22:08:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:08:58] <paladox>	 !log restarting mysql on db4 (from [17:49:05] UTC)
[22:09:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:09:08] <icinga-miraheze>	 RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 0.68, 0.17, 0.06  
[22:09:56] <icinga-miraheze>	 PROBLEM - dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Thu 06 Dec 2018 10:07:54 PM GMT +0000).  
[22:10:12] <Not-2d3>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp8kp
[22:10:14] <Not-2d3>	 [02miraheze/ssl] 07MirahezeSSLBot 0331b83b8 - Bot: Update SSL cert for dariawiki.org
[22:21:57] <icinga-miraheze>	 RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Mon 18 Feb 2019 09:10:04 PM GMT +0000.  
[22:23:57] <icinga-miraheze>	 PROBLEM - kunwok.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kunwok.org' expires in 15 day(s) (Thu 06 Dec 2018 10:21:13 PM GMT +0000).  
[22:24:11] <Not-2d3>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp8Ls
[22:24:13] <Not-2d3>	 [02miraheze/ssl] 07MirahezeSSLBot 03318f94e - Bot: Update SSL cert for kunwok.org
[22:31:57] <icinga-miraheze>	 RECOVERY - kunwok.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kunwok.org' will expire on Mon 18 Feb 2019 09:24:04 PM GMT +0000.  
[22:35:28] <icinga-miraheze>	 PROBLEM - pl.nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pl.nonbinary.wiki' expires in 15 day(s) (Thu 06 Dec 2018 10:31:43 PM GMT +0000).  
[22:35:41] <Not-2d3>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp8tl
[22:35:42] <Not-2d3>	 [02miraheze/ssl] 07MirahezeSSLBot 0359fca0f - Bot: Update SSL cert for pl.nonbinary.wiki
[22:39:14] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fp8ty
[22:39:15] <Not-2d3>	 [02miraheze/puppet] 07paladox 03d637117 - Split up Volumes in bacula This changes DB4 to 40gb x 5 = 200gb. This also changes static backup too.
[22:39:17] <Not-2d3>	 [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS
[22:39:18] <Not-2d3>	 [02puppet] 07paladox opened pull request 03#897: Split up Volumes in bacula - 13https://git.io/fp8t9
[22:41:28] <icinga-miraheze>	 RECOVERY - pl.nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pl.nonbinary.wiki' will expire on Mon 18 Feb 2019 09:35:35 PM GMT +0000.  
[22:42:44] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[22:42:54] <icinga-miraheze>	 PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[22:43:06] <icinga-miraheze>	 PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[22:43:38] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[22:51:39] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures  
[22:52:44] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[22:52:52] <icinga-miraheze>	 RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[22:53:05] <icinga-miraheze>	 RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures