[00:02:46] PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:26] PROBLEM - Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:28] PROBLEM - Puppet on test1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:11:26] RECOVERY - Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [00:12:47] RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:13:27] RECOVERY - Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:15:27] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:19:27] PROBLEM - Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 3 minutes ago with 1 failures [06:12:46] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5536 [09:05:27] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [09:13:25] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [12:11:45] paladox: hi :) [12:30:06] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNmaZ [12:30:08] [02miraheze/puppet] 07Reception123 0323655e9 - add temporary key for myself [13:13:59] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNmV7 [13:14:00] [02miraheze/mw-config] 07Reception123 03ac85574 - disable anon edits on arkorder per T3349 resolves T3349 [13:14:51] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/fNmVb [13:14:53] [02miraheze/dns] 07Reception123 034e8f99a - add russopedia.info zone [13:39:08] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fNmwE [13:39:09] [02miraheze/ssl] 07Reception123 037783f46 - add russopedia.info cert [14:01:06] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNmrm [14:01:08] [02miraheze/mw-config] 07Reception123 03e583a42 - add russopedia.info custom domain per T3350 [14:02:47] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [14:13:32] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNmrR [14:13:33] [02miraheze/mw-config] 07Reception123 037b38778 - wgCommentsInRecentChanges true for newusopediawiki per T3351 [14:14:48] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2670 [14:33:01] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNmrF [14:33:03] [02miraheze/mw-config] 07Reception123 03e425db6 - namespace corrections per T3340 [14:40:49] PROBLEM - JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [14:43:07] PROBLEM - MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 222 bytes in 0.036 second response time [14:43:27] PROBLEM - MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 222 bytes in 0.037 second response time [14:43:29] PROBLEM - MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 222 bytes in 0.013 second response time [14:43:39] Reception123: ^^^ [14:43:47] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [14:44:06] urgh [14:44:07] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [14:44:19] I know that is not to blame, but Travis CI not doing its job again [14:44:48] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 89% [14:44:50] PROBLEM - Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:44:55] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNmKf [14:44:57] [02miraheze/mw-config] 07Reception123 033132dea - rm arbitrary text [14:45:42] Reception123: https://github.com/miraheze/ManageWiki/pull/13 [14:45:43] Title: [ separate extensions in ManageWiki by Reception123 · Pull Request #13 · miraheze/ManageWiki · GitHub ] - github.com [14:46:05] JohnLewis: ok, will look [14:46:31] miraheze/mw-config/master/3132dea - Reception123 The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/403916331 [14:46:48] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [14:46:50] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [14:47:08] RECOVERY - MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 30098 bytes in 0.105 second response time [14:47:28] RECOVERY - MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 30099 bytes in 0.212 second response time [14:47:38] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 57% [14:47:48] RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:48:08] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:48:30] Reception123: hi :) [14:48:48] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 3% [14:49:38] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 20% [14:52:44] [02ManageWiki] 07Reception123 synchronize pull request 03#13: separate extensions in ManageWiki - 13https://git.io/fNttj [14:52:46] [02miraheze/ManageWiki] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/fNmKg [14:52:47] [02miraheze/ManageWiki] 07Reception123 0389bd07c - fixes [16:16:49] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [17:52:33] revi: if your around, can i pm you, cvt related [17:52:33] Zppix: 2018-07-02 - 20:36:08CDT tell Zppix thanks to .mangle, I have yet another personality to add to my repoitre of personalities [18:06:18] I've created https://meta.miraheze.org/wiki/User:Void/ManageWiki with the hopes of making a sort of help guide or user documentation [18:06:19] Title: [ User:Void/ManageWiki - Miraheze Meta ] - meta.miraheze.org [18:34:47] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNmSI [18:34:48] [02miraheze/mw-config] 07paladox 03d8b0bb3 - Switch DBO_SSL on [18:36:07] Voidwalker: in f12 dev console to get something using mw.config whats the syntax [18:36:33] mw.config.get( 'value' ) [18:36:49] Ok [18:37:06] yu [18:37:07] ty* [18:37:17] np [18:39:06] Voidwalker: im screwing around on my local dev mw instance making random useless mw scripts and i couldnt remember the syntax :P [18:39:45] :P [18:50:46] !log depool mw1 [18:50:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:50:58] !log upgrade mw1 to stretch 9.5 (point update) [18:51:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:52:49] !log reboot mw1 for point update. [18:52:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:53:06] might as well make one big log statement [18:53:26] !log repool mw1 [18:53:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:56:45] oh that reminds me keep my stuff up to date... let's see... only 26 package updates... not bad [18:57:10] heh [18:57:56] beats 1400 [18:57:59] !log depool mw2, update mw2 to stretch 9.5 (point release), restart mw2 and repool m2 [18:58:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:58:17] Voidwalker 1400 packages? [18:58:19] what os? [18:59:01] that was a few months ago when I upgraded my raspberry pi to stretch [18:59:11] ah [19:04:27] !log depool mw3, update mw3 to stretch 9.5 (point release), restart mw3 and repool mw3 [19:04:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:07:07] !log upgrading misc3 to stretch 9.5 point release (also reboot) [19:07:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:09:47] !log upgrading misc2 to stretch 9.5 point release (also reboot) [19:09:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:10:07] not going to touch misc1 or db4 until i speak with someone from ops first. [19:15:19] !log upgrading bacula1 to stretch 9.5 point release (also reboot) [19:15:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:15:28] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 4 minutes ago with 4 failures. Failed resources (up to 3 shown): Package[dirmngr],Package[apache2],Package[salt-minion],Package[php7.2-gettext] [19:15:30] these are all security updates fyi [19:17:21] !log upgrading puppet1 to stretch 9.5 point release (also reboot) [19:17:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:17:28] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:22:48] PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:22:50] PROBLEM - Puppet on cp2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:23:08] PROBLEM - Puppet on swift2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:23:28] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:23:30] PROBLEM - Puppet on cp5 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:23:38] PROBLEM - Puppet on bacula1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:27:39] RECOVERY - Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:28:49] RECOVERY - Puppet on db4 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:30:59] !log upgrade to stretch 9.5 (point release) on test1 and restart [19:31:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:31:09] RECOVERY - Puppet on swift2 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:31:29] RECOVERY - Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:32:47] RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:32:49] RECOVERY - Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:33:27] RECOVERY - Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:35:09] PROBLEM - Current Load on test1 is CRITICAL: connect to address 185.52.2.243 port 5666: Connection refused [19:35:27] PROBLEM - Disk Space on test1 is CRITICAL: connect to address 185.52.2.243 port 5666: Connection refused [19:35:29] PROBLEM - Puppet on test1 is CRITICAL: connect to address 185.52.2.243 port 5666: Connection refused [19:35:47] PROBLEM - php7.2-fpm on test1 is CRITICAL: connect to address 185.52.2.243 port 5666: Connection refused [19:35:57] PROBLEM - HTTP on test1 is CRITICAL: connect to address 185.52.2.243 and port 80: Connection refused [19:35:59] PROBLEM - HTTPS on test1 is CRITICAL: connect to address 185.52.2.243 and port 443: Connection refused [19:36:57] RECOVERY - HTTP on test1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 409 bytes in 0.003 second response time [19:36:59] RECOVERY - php7.2-fpm on test1 is OK: PROCS OK: 5 processes with command name php-fpm7.2 [19:37:01] RECOVERY - MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 30102 bytes in 4.641 second response time [19:38:57] PROBLEM - Puppet on test1 is CRITICAL: connect to address 185.52.2.243 port 5666: Connection refused [19:42:48] PROBLEM - papelor.io - LetsEncrypt on sslhost is WARNING: WARNING - Certificate papelor.io expires in 15 day(s) (Mon 30 Jul 2018 07:40:21 PM GMT +0000). [19:43:00] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNmHQ [19:43:02] [02miraheze/ssl] 07MirahezeSSLBot 0344e4679 - Bot: Update SSL cert for papelor.io [19:52:50] RECOVERY - papelor.io - LetsEncrypt on sslhost is OK: OK - Certificate papelor.io will expire on Fri 12 Oct 2018 06:42:54 PM GMT +0000. [20:17:31] Voidwalker: replied. Awesome work on the help page btw:D [20:18:12] no prob [20:18:58] Voidwalker: JohnLewis would a general help button somewhere on the special page work? [20:19:12] PROBLEM - Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 20 hours ago with 1 failures [20:19:16] Not sure if we can implement that with HTMLForm but it would be nice [20:20:04] [02ManageWiki] 07paladox synchronize pull request 03#13: separate extensions in ManageWiki - 13https://git.io/fNttj [20:20:05] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/fNm73 [20:20:07] [02miraheze/ManageWiki] 07paladox 039433ca2 - Update SpecialManageWikiExtensions.php [20:20:25] [02ManageWiki] 07paladox synchronize pull request 03#13: separate extensions in ManageWiki - 13https://git.io/fNttj [20:20:26] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/fNm7Z [20:20:28] [02miraheze/ManageWiki] 07paladox 03e074529 - Update SpecialManageWikiExtensions.php [20:29:55] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fNm7o [20:29:57] [02miraheze/mediawiki] 07paladox 03564e16d - Update SocialProfile [20:33:47] !log sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw* [20:33:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master