[00:02:20] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNeja
[00:02:22] <Not-683e>	 [02miraheze/puppet] 07paladox 032bce324 - Update object-server.conf.erb
[00:02:30] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNejV
[00:02:31] <Not-683e>	 [02miraheze/puppet] 07paladox 034cc66f2 - Update container-server.conf.erb
[00:02:40] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNejr
[00:02:41] <Not-683e>	 [02miraheze/puppet] 07paladox 0372f81be - Update account-server.conf.erb
[00:03:09] <icinga-miraheze>	 PROBLEM - Puppet on ns1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:08:57] <icinga-miraheze>	 PROBLEM - Current Load on nfs1 is WARNING: WARNING - load average: 0.81, 1.62, 3.77  
[00:08:59] <icinga-miraheze>	 PROBLEM - Current Load on swift2 is WARNING: WARNING - load average: 0.75, 1.59, 3.75  
[00:11:07] <icinga-miraheze>	 RECOVERY - Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures  
[00:12:57] <icinga-miraheze>	 RECOVERY - Current Load on nfs1 is OK: OK - load average: 0.76, 1.27, 3.14  
[00:12:59] <icinga-miraheze>	 RECOVERY - Current Load on swift2 is OK: OK - load average: 0.86, 1.28, 3.13  
[00:13:57] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is CRITICAL: CRITICAL - load average: 9.70, 7.95, 7.07  
[00:16:58] <icinga-miraheze>	 PROBLEM - Check correctness of the icinga configuration on misc1 is CRITICAL: Icinga configuration contains errors  
[00:18:30] <icinga-miraheze>	 RECOVERY - Check correctness of the icinga configuration on misc1 is OK: Icinga configuration is correct  
[00:30:36] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNve1
[00:30:37] <Not-683e>	 [02miraheze/puppet] 07paladox 03ddabd3e - Update container-server.conf.erb
[00:30:57] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNveM
[00:30:59] <Not-683e>	 [02miraheze/puppet] 07paladox 03d1957bc - Update object-server.conf.erb
[00:46:19] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvv3
[00:46:20] <Not-683e>	 [02miraheze/puppet] 07paladox 03dab548d - Update account-server.conf.erb
[00:50:18] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvvc
[00:50:19] <Not-683e>	 [02miraheze/puppet] 07paladox 03a466abf - Update data.yaml
[00:54:10] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvvu
[00:54:11] <Not-683e>	 [02miraheze/puppet] 07paladox 03baa111d - Update object-server.conf.erb
[00:54:39] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvvg
[00:54:41] <Not-683e>	 [02miraheze/puppet] 07paladox 0393a0646 - Update container-server.conf.erb
[01:23:16] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvf5
[01:23:17] <Not-683e>	 [02miraheze/puppet] 07paladox 03a235148 - Update data.yaml
[01:25:05] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvfF
[01:25:06] <Not-683e>	 [02miraheze/puppet] 07paladox 03f5161ef - Update object-server.conf.erb
[01:27:55] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvJv
[01:27:57] <Not-683e>	 [02miraheze/puppet] 07paladox 03b8da98a - Update data.yaml
[01:32:24] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvJG
[01:32:25] <Not-683e>	 [02miraheze/puppet] 07paladox 032866417 - Update data.yaml
[01:34:58] <icinga-miraheze>	 PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[01:36:58] <icinga-miraheze>	 RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[01:49:29] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 1.38, 1.21, 3.95  
[01:55:29] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 2.17, 1.81, 3.34  
[02:16:59] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5470  
[02:52:41] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvUd
[02:52:42] <Not-683e>	 [02miraheze/mw-config] 07paladox 03bff4ec6 - Set wgTimelineFileBackend to miraheze-swift
[03:17:40] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvT4
[03:17:42] <Not-683e>	 [02miraheze/mw-config] 07paladox 03b1ed9e1 - Update FileBackend.php
[03:41:47] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvTD
[03:41:48] <Not-683e>	 [02miraheze/mw-config] 07paladox 03f0dfe4b - Update GlobalExtensions.php
[03:45:28] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvT5
[03:45:30] <Not-683e>	 [02miraheze/mw-config] 07paladox 03eca3436 - Update FileBackend.php
[04:16:58] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[04:42:30] <icinga-miraheze>	 PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 82%  
[04:44:30] <icinga-miraheze>	 PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 42%  
[04:46:28] <icinga-miraheze>	 RECOVERY - HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 2%  
[06:26:28] <icinga-miraheze>	 PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 49%  
[06:28:28] <icinga-miraheze>	 RECOVERY - HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 31%  
[08:40:29] <icinga-miraheze>	 PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 47%  
[08:42:29] <icinga-miraheze>	 RECOVERY - HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 31%  
[09:05:39] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.42, 3.20, 2.71  
[09:07:39] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 2.80, 3.08, 2.72  
[09:14:59] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[09:21:00] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures  
[09:52:30] <icinga-miraheze>	 PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 79%  
[09:56:30] <icinga-miraheze>	 RECOVERY - HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 37%  
[11:04:58] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2623  
[11:32:00] <Not-683e>	 [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_31 [+1/-0/±1] 13https://git.io/fNv3x
[11:32:02] <Not-683e>	 [02miraheze/mediawiki] 07Reception123 03c35ea9e - Install LinkTarget
[11:32:09] <Not-683e>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/fNv3p
[11:32:10] <Not-683e>	 [02miraheze/mw-config] 07Reception123 03650b98e - add LinkTarget extension
[11:32:55] <travis-ci>	 miraheze/mw-config/master/650b98e - Reception123 The build was broken. https://travis-ci.org/miraheze/mw-config/builds/401195436
[11:41:00] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output  
[11:42:58] <icinga-miraheze>	 PROBLEM - MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 222 bytes in 0.019 second response time  
[11:43:00] <icinga-miraheze>	 PROBLEM - MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 222 bytes in 0.023 second response time  
[11:43:58] <icinga-miraheze>	 PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb  
[11:44:00] <icinga-miraheze>	 PROBLEM - MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 222 bytes in 0.016 second response time  
[11:44:18] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp4 is CRITICAL: 3 backends are down.  mw1 mw2 mw3  
[11:44:28] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp2 is CRITICAL: 3 backends are down.  mw1 mw2 mw3  
[11:44:30] <icinga-miraheze>	 PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb  
[11:45:02] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvsY
[11:45:03] <Not-683e>	 [02miraheze/puppet] 07paladox 03efc1d44 - Update object-server.conf.erb
[11:46:19] <paladox>	 oh
[11:46:23] <paladox>	 Reception123 
[11:46:27] <paladox>	 i am going to fix your error
[11:46:29] <icinga-miraheze>	 PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 82%  
[11:47:17] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvsG
[11:47:19] <Not-683e>	 [02miraheze/mw-config] 07paladox 0376d6409 - Update ManageWiki.php
[11:47:59] <icinga-miraheze>	 RECOVERY - MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 31044 bytes in 0.153 second response time  
[11:48:43] <travis-ci>	 miraheze/mw-config/master/76d6409 - paladox The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/401197812
[11:48:59] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 3851  
[11:49:59] <icinga-miraheze>	 RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online  
[11:50:19] <icinga-miraheze>	 RECOVERY - HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 15%  
[11:50:21] <icinga-miraheze>	 RECOVERY - Varnish Backends on cp4 is OK: All 3 backends are healthy  
[11:50:29] <icinga-miraheze>	 PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 46%  
[11:50:31] <icinga-miraheze>	 RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online  
[11:50:59] <icinga-miraheze>	 RECOVERY - MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 31050 bytes in 0.992 second response time  
[11:52:39] <Reception123>	 ah whoops
[11:52:39] <Reception123>	 sorry
[11:52:45] <Reception123>	 Have to get the hang of PHP again
[11:53:12] <paladox>	 heh
[11:56:37] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and and rebuild LC
[11:56:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[12:07:00] <icinga-miraheze>	 PROBLEM - Current Load on test1 is CRITICAL: CRITICAL - load average: 2.31, 1.72, 1.00  
[12:08:58] <icinga-miraheze>	 RECOVERY - Current Load on test1 is OK: OK - load average: 1.27, 1.54, 1.02  
[12:09:08] <icinga-miraheze>	 PROBLEM - Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]  
[12:15:58] <icinga-miraheze>	 PROBLEM - Current Load on test1 is WARNING: WARNING - load average: 1.78, 1.74, 1.29  
[12:24:59] <icinga-miraheze>	 RECOVERY - Puppet on test1 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures  
[13:05:00] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[13:12:57] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvZo
[13:12:59] <Not-683e>	 [02miraheze/puppet] 07paladox 0353c82a8 - Update object-server.conf.erb
[13:13:22] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvZ6
[13:13:24] <Not-683e>	 [02miraheze/puppet] 07paladox 03cc42894 - Update container-server.conf.erb
[13:13:43] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvZP
[13:13:45] <Not-683e>	 [02miraheze/puppet] 07paladox 037a0462c - Update account-server.conf.erb
[13:25:23] <MacFan4000>	 !m paladox
[13:25:23] <[d__d]>	 You're doing good work, paladox!
[13:25:34] <paladox>	 thanks :)
[13:43:12] <Not-683e>	 [02miraheze/mediawiki] 07paladox pushed 0310 commits to 03REL1_31 [+0/-0/±31] 13https://git.io/fNvnr
[13:43:13] <Not-683e>	 [02miraheze/mediawiki] 07zaori 03b1fe638 - Update git submodules * Update skins/Timeless from branch 'REL1_31' to 2126f76540981047f072d672be5c15475a2556cf - Remove nonfunctional jquery mobile stuff Does not appear to work, and is possibly breaking things elsewise as well. In particular: chrome text selection in general, possibly iphone support? May or may not resolve the following; please test (does seem to
[13:43:13] <Not-683e>	 effectively resolve T183215, for now, at least, but dunno at all about the other two.): Bug: T161470 Bug: T183215 Bug: T188387 Change-Id: I2d4aae98fe47bce5a873bc44b6cea58f111f5086
[13:43:15] <Not-683e>	 [02miraheze/mediawiki] 07paladox 030e7eb69 - Update git submodules * Update skins/Timeless from branch 'REL1_31' to 6860d4d96db3bfcc92172f08ddfe4a2344229937 - Fix sidebar closing on iOS this follows this fix [1] [1] https://stackoverflow.com/questions/11397028/document-click-function-for-touch-device Bug: T161470 Change-Id: I20e6a0e18a239131b94e4ab949b7784f241e4d40
[13:43:16] <Not-683e>	 [02miraheze/mediawiki] 07edwardspec 0363a4e21 - Update git submodules * Update skins/MonoBook from branch 'REL1_31' to 53a78553e4b6cb091df318bf70291330f1425429 - Missing 'class' in "newtalk" notification MonoBookTemplate incorrectly calls getIfExists() for newtalk/undelete: attributes of wrapper <div> must be under the 'parameters' key. Change-Id: I5392477d493174b0b5598927bfbcec2029b0987e (cherry picked from
[13:43:16] <ZppixBot>	 Title: [ jquery - document .click function for touch device - Stack Overflow ] - stackoverflow.com
[13:43:16] <Not-683e>	 commit 73ad6a7835b587035e127d32d4667ba16e43ad03)
[13:43:18] <Not-683e>	 [02miraheze/mediawiki] ... and 7 more commits.
[13:45:45] <paladox>	 Reception123 there was a security update so i merged the branch ^^
[13:45:52] <paladox>	 even though 1.31.1 has not been released yet
[14:56:13] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvWD
[14:56:14] <Not-683e>	 [02miraheze/puppet] 07paladox 03b6675b8 - Move swiftExport to /etc/
[14:56:49] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvWH
[14:56:50] <Not-683e>	 [02miraheze/puppet] 07paladox 034c49260 - Update data.yaml
[15:03:50] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvlv
[15:03:51] <Not-683e>	 [02miraheze/puppet] 07paladox 03e2c2f0e - Update init.pp
[15:05:26] <Not-683e>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[15:05:27] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNvlk
[15:05:29] <Not-683e>	 [02miraheze/puppet] 07paladox 03f6a7c2d - phabricator: remove support for php5
[15:05:30] <Not-683e>	 [02puppet] 07paladox opened pull request 03#770: phabricator: remove support for php5 - 13https://git.io/fNvlI
[15:05:56] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/fNvlL
[15:05:58] <Not-683e>	 [02miraheze/puppet] 07paladox 030046c3c - Delete php.ini
[15:05:59] <Not-683e>	 [02puppet] 07paladox synchronize pull request 03#770: phabricator: remove support for php5 - 13https://git.io/fNvlI
[15:06:41] <Not-683e>	 [02puppet] 07paladox closed pull request 03#770: phabricator: remove support for php5 - 13https://git.io/fNvlI
[15:06:43] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/fNvlm
[15:06:44] <Not-683e>	 [02miraheze/puppet] 07paladox 03f793250 - phabricator: remove support for php5 (#770) * phabricator: remove support for php5 * Delete php.ini
[15:06:46] <Not-683e>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[15:06:47] <Not-683e>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[15:15:00] <icinga-miraheze>	 PROBLEM - Puppet on swift1 is WARNING: WARNING: Puppet is currently disabled, message: temp worker increase for swift, last run 4 minutes ago with 0 failures  
[15:21:58] <icinga-miraheze>	 PROBLEM - Puppet on swift2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 11 minutes ago with 0 failures  
[15:29:18] <icinga-miraheze>	 PROBLEM - HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 65%  
[15:31:19] <icinga-miraheze>	 RECOVERY - HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 11%  
[15:42:29] <icinga-miraheze>	 PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 49%  
[15:44:29] <icinga-miraheze>	 RECOVERY - HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 13%  
[16:04:39] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.85, 3.62, 3.18  
[16:14:39] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 2.43, 3.06, 3.13  
[16:17:49] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNv46
[16:17:51] <Not-683e>	 [02miraheze/puppet] 07paladox 03872b1a0 - Add swift1 and swift2 to bacula
[16:17:52] <Not-683e>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[16:17:54] <Not-683e>	 [02puppet] 07paladox opened pull request 03#771: Add swift1 and swift2 to bacula - 13https://git.io/fNv4P
[16:22:11] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNv4d
[16:22:12] <Not-683e>	 [02miraheze/puppet] 07paladox 037663663 - Update site.pp
[16:22:14] <Not-683e>	 [02puppet] 07paladox synchronize pull request 03#771: Add swift1 and swift2 to bacula - 13https://git.io/fNv4P
[17:26:52] <Not-683e>	 [02puppet] 07paladox synchronize pull request 03#771: Add swift1 and swift2 to bacula - 13https://git.io/fNv4P
[17:26:54] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNv0m
[17:26:55] <Not-683e>	 [02miraheze/puppet] 07paladox 030c06688 - Update bacula-dir.conf
[17:30:48] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNv0n
[17:30:49] <Not-683e>	 [02miraheze/puppet] 07paladox 03b6a75c8 - Update nrpe.cfg.erb
[17:30:51] <Not-683e>	 [02puppet] 07paladox synchronize pull request 03#771: Add swift1 and swift2 to bacula - 13https://git.io/fNv4P
[17:32:12] <MH-Discord>	 <NeoMahler> when is it planned to have the images back?
[17:32:25] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNv0C
[17:32:26] <Not-683e>	 [02miraheze/puppet] 07paladox 032f8d53e - Update director.pp
[17:32:28] <Not-683e>	 [02puppet] 07paladox synchronize pull request 03#771: Add swift1 and swift2 to bacula - 13https://git.io/fNv4P
[17:33:22] <Not-683e>	 [02puppet] 07paladox closed pull request 03#771: Add swift1 and swift2 to bacula - 13https://git.io/fNv4P
[17:33:24] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/fNv04
[17:33:25] <Not-683e>	 [02miraheze/puppet] 07paladox 0363128f8 - Add swift1 and swift2 to bacula (#771) * Add swift1 and swift2 to bacula * Update site.pp * Update bacula-dir.conf * Update nrpe.cfg.erb * Update director.pp
[17:33:41] <paladox>	 NeoMahler hmm, which wiki is having problems with images?
[17:34:04] <MH-Discord>	 <NeoMahler> nonbinary.wiki, they don't show
[17:34:11] <paladox>	 i will have a look now
[17:34:23] <MH-Discord>	 <NeoMahler> there's a sitenotice explaining the situation, I just wanted to know when is it planned to work again
[17:35:13] <paladox>	 NeoMahler fixed now.
[17:35:29] <MH-Discord>	 <NeoMahler> oh, well, that's magic 😮
[17:35:33] <MH-Discord>	 <NeoMahler> thanks!
[17:35:39] <paladox>	 NeoMahler it should mostly work now. Though some wiki's may be affected like yours. Just needed to fix permissions on the container.
[17:36:14] <MH-Discord>	 <NeoMahler> I don't know what this means, but I'm glad it works now!
[17:42:41] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.54, 3.38, 3.22  
[17:43:01] <icinga-miraheze>	 PROBLEM - Bacula - Static Swift1 on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.  
[17:43:11] <icinga-miraheze>	 PROBLEM - Bacula - Static Swift2 on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.  
[17:44:41] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 3.01, 3.31, 3.21  
[18:05:54] <MH-Discord>	 <NeoMahler> is there an option to download a backup of a wiki?
[18:16:31] <icinga-miraheze>	 PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 44%  
[18:17:57] <paladox>	 NeoMahler hi, yes. We can add wiki's to our dumps puppet class which then gets uploaded to static.miraheze.org/dumps/
[18:18:05] <paladox>	 NeoMahler also you have a js error here https://nonbinary.wiki/wiki/MediaWiki:Common.js
[18:18:06] <ZppixBot>	 Title: [ MediaWiki:Common.js - Nonbinary ] - nonbinary.wiki
[18:18:09] <paladox>	 needs to be wrapped in
[18:18:23] <paladox>	 mw.loader.using( ['mediawiki.util'] ).done( function () {
[18:18:25] <paladox>	 .....
[18:18:31] <icinga-miraheze>	 RECOVERY - HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 28%  
[18:18:33] <paladox>	 } );
[18:31:30] <MH-Discord>	 <NeoMahler> fixed, I think
[18:31:40] <MH-Discord>	 <NeoMahler> can nonbinary.wiki be added to the dumps then?
[18:34:15] <paladox>	 NeoMahler ok, i can do it weekly or monthly?
[18:35:07] <MH-Discord>	 <NeoMahler> monthly will be enough
[18:35:11] <paladox>	 ok
[18:35:38] <Not-683e>	 [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS
[18:35:40] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fNvu3
[18:35:41] <Not-683e>	 [02miraheze/puppet] 07paladox 03f61c83b - Add nonbinarywiki to dumps
[18:35:43] <Not-683e>	 [02puppet] 07paladox opened pull request 03#772: Add nonbinarywiki to dumps - 13https://git.io/fNvus
[18:35:49] <paladox>	 NeoMahler could you +1 ^^ please?
[18:36:11] <Not-683e>	 [02puppet] 07paladox synchronize pull request 03#772: Add nonbinarywiki to dumps - 13https://git.io/fNvus
[18:36:13] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fNvun
[18:36:14] <Not-683e>	 [02miraheze/puppet] 07paladox 030c74ea5 - Update dumps.pp
[18:36:31] <Not-683e>	 [02puppet] 07paladox synchronize pull request 03#772: Add nonbinarywiki to dumps - 13https://git.io/fNvus
[18:36:32] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fNvuC
[18:36:34] <Not-683e>	 [02miraheze/puppet] 07paladox 037039040 - Update dumps.pp
[18:36:40] <MH-Discord>	 <NeoMahler> uhm... how do I do that?
[18:36:47] <Not-683e>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[18:36:48] <Not-683e>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[18:37:05] <paladox>	 NeoMahler you just need to write a comment with +1 :)
[18:37:24] <Not-683e>	 [02puppet] 07NeoMahler commented on pull request 03#772: Add nonbinarywiki to dumps - 13https://git.io/fNvul
[18:37:36] <MH-Discord>	 <NeoMahler> mmh like that?
[18:37:41] <paladox>	 NeoMahler thanks! Yep. Merging.
[18:37:46] <Not-683e>	 [02puppet] 07paladox closed pull request 03#772: Add nonbinarywiki to dumps - 13https://git.io/fNvus
[18:37:48] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvu8
[18:37:49] <Not-683e>	 [02miraheze/puppet] 07paladox 0351109d0 - Add nonbinarywiki to dumps (#772) * Add nonbinarywiki to dumps * Update dumps.pp * Update dumps.pp
[18:37:51] <Not-683e>	 [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS
[18:37:52] <Not-683e>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2
[18:39:25] <MH-Discord>	 <NeoMahler> so when the dump is generated, it will appear as nonbinarywiki.xml, right?
[18:39:29] <MH-Discord>	 <NeoMahler> in https://static.miraheze.org/dumps/
[18:39:29] <ZppixBot>	 Title: [ Listing of /v1/AUTH_admin/dumps/ ] - static.miraheze.org
[18:39:34] <paladox>	 yep
[18:39:51] <MH-Discord>	 <NeoMahler> and it will contain all pages in all namespaces?
[18:40:12] <MH-Discord>	 <NeoMahler> I assume files are excluded
[18:40:51] <paladox>	 I think so yeh
[18:41:14] <MH-Discord>	 <NeoMahler> perfect, thanks!
[18:41:48] <paladox>	 your welcome :)
[18:57:01] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2635  
[19:02:42] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.69, 3.46, 3.17  
[19:07:02] <icinga-miraheze>	 RECOVERY - Puppet on swift1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[19:08:02] <icinga-miraheze>	 RECOVERY - Puppet on swift2 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures  
[19:10:42] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 2.37, 2.98, 3.10  
[19:39:44] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvzd
[19:39:46] <Not-683e>	 [02miraheze/puppet] 07paladox 03d85906c - Update dumps.pp
[19:45:47] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvge
[19:45:49] <Not-683e>	 [02miraheze/puppet] 07paladox 03669578c - Update dumps.pp
[19:49:17] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvgJ
[19:49:19] <Not-683e>	 [02miraheze/puppet] 07paladox 03cbd8a54 - Update dumps.pp
[19:53:01] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[20:02:53] <Not-683e>	 [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvgR
[20:02:54] <Not-683e>	 [02miraheze/ManageWiki] 07paladox 03d5eb096 - Add isset check for mwoptions
[20:03:29] <JohnLewis>	 h
[20:03:37] <JohnLewis>	 it's set or it's not
[20:03:50] <Not-683e>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fNvg0
[20:03:51] <Not-683e>	 [02miraheze/mediawiki] 07paladox 03e36c0c7 - Update ManageWiki
[20:04:45] <Not-683e>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvgu
[20:04:46] <Not-683e>	 [02miraheze/ManageWiki] 07JohnFLewis 032bc97d2 - rm uneccessary double check
[20:04:56] <JohnLewis>	 paladox: ^
[20:05:05] <paladox>	 ah ok
[20:05:16] <paladox>	 JohnLewis though if someone passes a '' it will return true
[20:05:52] <JohnLewis>	 and if someone passes false, it should return true
[20:08:11] <paladox>	 JohnLewis that's why i had isset & var
[20:08:12] <paladox>	 :)
[20:08:31] <JohnLewis>	 but then if its false, it returns false
[20:08:57] <paladox>	 well if $mwoptions = $det['options']; that returns false it will still be true in isset.
[20:09:09] <JohnLewis>	 and it should be
[20:09:52] <paladox>	 ok
[20:14:51] <JohnLewis>	 can you pull my changes please
[20:16:45] <paladox>	 JohnLewis pull?
[20:16:53] <paladox>	 JohnLewis it's already deployed!
[20:17:11] <paladox>	 oh
[20:17:16] <paladox>	 i know what you mean now
[20:17:55] <Not-683e>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fNvgD
[20:17:57] <Not-683e>	 [02miraheze/mediawiki] 07paladox 03201af7d - Update ManageWiki
[20:18:05] <paladox>	 JohnLewis done!
[20:36:18] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNv23
[20:36:19] <Not-683e>	 [02miraheze/mw-config] 07paladox 039d5d4a3 - Update Sitenotice.php
[20:36:29] <paladox>	 JohnLewis ^^? Looks good to you?
[20:36:34] <paladox>	 also https://poserdazfreebies.miraheze.org/wiki/User:Robkelk/sandbox/image_creation_party
[20:36:35] <ZppixBot>	 Title: [ User:Robkelk/sandbox/image creation party - Poser and Daz Studio Free Resources Wiki ] - poserdazfreebies.miraheze.org
[20:37:06] <JohnLewis>	 yeah
[21:25:00] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[21:31:01] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures  
[21:43:23] <Wiki-1776>	 Hello
[21:43:30] <paladox>	 hi
[21:55:41] <paladox>	 !log delete wiki ironsagawiki per request https://phabricator.miraheze.org/T3325#62858
[21:55:42] <ZppixBot>	 Title: [ ⚓ T3325 Extension enabling rquest ] - phabricator.miraheze.org
[21:55:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:41:41] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.56, 3.63, 2.93  
[22:47:41] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 3.04, 3.26, 2.99  
[22:53:51] <Not-683e>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[22:53:52] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNvr4
[22:53:54] <Not-683e>	 [02miraheze/puppet] 07paladox 03513a5d3 - Switch grafana to use misc1
[22:53:55] <Not-683e>	 [02puppet] 07paladox opened pull request 03#773: Switch grafana to use misc1 - 13https://git.io/fNvrB
[22:54:55] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNvr0
[22:54:56] <Not-683e>	 [02miraheze/puppet] 07paladox 037c7b7bb - Update site.pp
[22:54:58] <Not-683e>	 [02puppet] 07paladox synchronize pull request 03#773: Switch grafana to use misc1 - 13https://git.io/fNvrB
[22:56:19] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNvrE
[22:56:21] <Not-683e>	 [02miraheze/puppet] 07paladox 039c12189 - Update misc1.yaml
[22:56:22] <Not-683e>	 [02puppet] 07paladox synchronize pull request 03#773: Switch grafana to use misc1 - 13https://git.io/fNvrB
[22:56:34] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNvru
[22:56:35] <Not-683e>	 [02miraheze/puppet] 07paladox 030b4179b - Update misc2.yaml
[22:56:37] <Not-683e>	 [02puppet] 07paladox synchronize pull request 03#773: Switch grafana to use misc1 - 13https://git.io/fNvrB
[22:58:07] <paladox>	 JohnLewis lets get the party started! Merging.....
[22:58:12] <Not-683e>	 [02puppet] 07paladox closed pull request 03#773: Switch grafana to use misc1 - 13https://git.io/fNvrB
[22:58:14] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/fNvrg
[22:58:15] <Not-683e>	 [02miraheze/puppet] 07paladox 03d97dfe8 - Switch grafana to use misc1 (#773) * Switch grafana to use misc1 * Update site.pp * Update misc1.yaml * Update misc2.yaml
[22:58:17] <Not-683e>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[22:58:19] <Not-683e>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[22:59:26] <Not-683e>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvra
[22:59:28] <Not-683e>	 [02miraheze/dns] 07paladox 037e5f522 - Update miraheze.org
[22:59:38] <Not-683e>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvrw
[22:59:40] <Not-683e>	 [02miraheze/dns] 07paladox 03d9480f6 - Remove ganglia
[23:03:38] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvr6
[23:03:40] <Not-683e>	 [02miraheze/puppet] 07paladox 0372538ef - Update grafana.pp
[23:04:30] <icinga-miraheze>	 PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[23:04:34] <JohnLewis>	 uh
[23:04:54] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvrX
[23:04:55] <Not-683e>	 [02miraheze/puppet] 07paladox 03b3d40ad - Update grafana.pp
[23:04:59] <JohnLewis>	 why are you defining something as icinga not in icinga?
[23:05:35] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvrM
[23:05:37] <Not-683e>	 [02miraheze/puppet] 07paladox 03d336ba1 - Update grafana.pp
[23:06:02] <JohnLewis>	 paladox
[23:06:15] <paladox>	 JohnLewis it's icinga accessing port 80
[23:06:44] <JohnLewis>	 that's 3 commits for a non issue
[23:06:57] <JohnLewis>	 80 and 443 are standard ports
[23:07:06] <JohnLewis>	 its not just icinga
[23:07:18] <paladox>	 oh i see
[23:07:21] <paladox>	 i will rename it then
[23:07:36] <paladox>	 JohnLewis not sure what to do
[23:07:38] <paladox>	 remove it?
[23:08:05] <JohnLewis>	 do what every other module does
[23:08:19] <paladox>	 ok and remove it.
[23:08:33] <JohnLewis>	 remove what?
[23:09:04] <paladox>	 JohnLewis https://github.com/miraheze/puppet/blob/master/modules/role/manifests/mediawiki.pp#L5 and https://github.com/miraheze/puppet/blob/master/modules/role/manifests/icinga.pp#L5
[23:09:05] <ZppixBot>	 Title: [ puppet/mediawiki.pp at master · miraheze/puppet · GitHub ] - github.com
[23:09:06] <ZppixBot>	 Title: [ puppet/icinga.pp at master · miraheze/puppet · GitHub ] - github.com
[23:09:10] <paladox>	 as it is in the icinga module
[23:09:15] <paladox>	 it will conflict with grafana's one
[23:09:23] <paladox>	 due to the port
[23:11:15] <paladox>	 Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Duplicate declaration: Exec[ufw-allow-tcp-from-any-to-any-port-80]
[23:11:18] <paladox>	 like JohnLewis ^^
[23:11:52] <JohnLewis>	 yes remove it ofc
[23:11:57] <paladox>	 ok
[23:12:09] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvr5
[23:12:10] <Not-683e>	 [02miraheze/puppet] 07paladox 036ed9262 - Update grafana.pp
[23:12:16] <JohnLewis>	 but that's 3 commits that shouldn't have happened
[23:14:31] <icinga-miraheze>	 RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures  
[23:14:55] <paladox>	 JohnLewis yeh and grafana is back now!
[23:14:59] <paladox>	 https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=mw2.miraheze.org&var-port=9100
[23:14:59] <ZppixBot>	 Title: [ Grafana ] - grafana.miraheze.org
[23:31:25] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNvoc
[23:31:27] <Not-683e>	 [02miraheze/puppet] 07paladox 0399606ad - Update prometheus_exporter.pp
[23:33:22] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp4 is CRITICAL: 2 backends are down.  mw1 mw2  
[23:33:31] <paladox>	 uh
[23:35:20] <icinga-miraheze>	 RECOVERY - Varnish Backends on cp4 is OK: All 3 backends are healthy