[00:03:10] PROBLEM - Puppet on cp5 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:45] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:33] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-81.4.121.113-to-any-port-81] [00:05:00] PROBLEM - Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-81.4.121.113-to-any-port-81] [00:07:12] meh [00:07:26] SPF|Cloud ^^ (happened again). Will depool later and reboot. [00:08:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNgPS [00:08:04] [02miraheze/puppet] 07paladox 034d53074 - Fix http icinga2 checks for nginx [00:09:51] HTTP CRITICAL - Unable to open TCP socket [00:11:00] RECOVERY - Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [00:11:10] RECOVERY - Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [00:11:17] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [00:11:37] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [00:13:09] HTTP CRITICAL - Unable to open TCP socket [00:24:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNgPp [00:24:08] [02miraheze/puppet] 07paladox 03024d22d - Update conf.pp [00:27:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNgXf [00:27:01] [02miraheze/puppet] 07paladox 03c275a23 - Update site.pp [00:36:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNgXO [00:36:10] [02miraheze/puppet] 07paladox 037c8bad0 - Update site.pp [00:37:19] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 557 bytes in 1.029 second response time [00:37:21] RECOVERY - test1 HTTP on test1 is OK: HTTP OK: HTTP/1.1 302 Moved Temporarily - 405 bytes in 0.018 second response time [00:37:33] RECOVERY - cp2 HTTP on cp2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 549 bytes in 0.420 second response time [00:37:49] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 549 bytes in 0.411 second response time [00:38:03] RECOVERY - cp5 HTTP on cp5 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 557 bytes in 1.062 second response time [00:38:07] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 302 Moved Temporarily - 405 bytes in 0.083 second response time [00:38:11] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 556 bytes in 0.044 second response time [00:38:50] Hey [00:38:53] Just checking in [00:40:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNgX4 [00:40:34] [02miraheze/puppet] 07paladox 03417d156 - Update conf.pp [00:40:42] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 557 bytes in 1.001 second response time [00:40:43] RECOVERY - test1 HTTP on test1 is OK: HTTP OK: HTTP/1.1 302 Moved Temporarily - 405 bytes in 0.090 second response time [00:40:47] RECOVERY - mw3 HTTP on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 459 bytes in 0.037 second response time [00:40:53] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 459 bytes in 0.081 second response time [00:40:57] RECOVERY - cp2 HTTP on cp2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 551 bytes in 0.422 second response time [00:41:07] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 459 bytes in 0.031 second response time [00:41:13] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 459 bytes in 0.049 second response time [00:41:17] RECOVERY - mw2 HTTP on mw2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 459 bytes in 0.038 second response time [00:41:27] RECOVERY - cp4 HTTP on cp4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 556 bytes in 0.026 second response time [00:41:31] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 302 Moved Temporarily - 405 bytes in 0.041 second response time [00:41:35] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 556 bytes in 0.022 second response time [00:41:41] RECOVERY - cp5 HTTP on cp5 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 557 bytes in 1.057 second response time [00:42:35] Whoa, here we go again [00:43:29] hello [00:43:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNgXE [00:43:39] [02miraheze/puppet] 07paladox 032c2e794 - Update roles.ini.erb [00:47:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNgXa [00:47:15] [02miraheze/puppet] 07paladox 0304c335b - Update roles.ini.erb [00:50:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNgX6 [00:50:45] [02miraheze/puppet] 07paladox 035d26516 - Add macfan to users.conf for icinga2 [01:00:01] .help [01:00:01] Hang on, I'm creating a list. [01:00:03] I've posted a list of my commands at https://ptpb.pw/WTkw - You can see more info about any of these commands by doing .help (e.g. .help time) [01:02:21] .help ping [01:02:21] AmandaCatherine: Reply to ping command. [01:04:11] I could’ve sworn there was a way to ping another user who may not be around, and then it will notify them when they are active [01:05:12] Oh, figured it out, I think [01:05:16] .help tell [01:05:16] AmandaCatherine: Give someone a message the next time they're seen [01:05:17] AmandaCatherine: e.g. ZppixBot, tell Embolalia he broke something again. [01:05:49] .tell Reception123 that the globalblock-exempt for founder group was discussed back in the days of WikiCanada when I kept running into LP’s rangeblocks [01:05:50] AmandaCatherine: I'll pass that on when Reception123 is around. [01:17:24] !night [01:17:24] Good night, sweet dreams and dream of the miraheze servers without errors! [07:28:41] .tell AmandaCatherine I had never discussed it, and I would oppose this, since requesting ipblock-exempt exists for a reason. Please make a request there, for locally or globally being exempted [07:28:41] Reception123: I'll pass that on when AmandaCatherine is around. [07:28:45] Reception123: 2018-07-29 - 01:05:50 tell Reception123 that the globalblock-exempt for founder group was discussed back in the days of WikiCanada when I kept running into LP’s rangeblocks [09:10:09] [02miraheze/services] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNg7x [09:10:11] [02miraheze/services] 07Reception123 03edfef60 - add "acqint" to parsoid [10:40:59] paladox: around? [12:11:34] Hello [12:44:55] wgExemptFromUserRobotsControl is for a Wiki not to appear on websites? [12:45:15] err [12:45:37] in the search engines Google, Bing....* [13:16:51] RECOVERY - cp5 HTTP on cp5 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 557 bytes in 1.057 second response time [13:28:18] Wiki-1776: yes, for a site not to be indexed by search engines [13:29:11] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2849 [13:29:11] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2852 [13:34:39] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-9100],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-9131] [13:34:59] PROBLEM - Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-9100],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-9131] [13:36:23] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNgx1 [13:36:24] [02miraheze/dns] 07paladox 03ddd8208 - depool cp2 to reboot it [13:38:58] !log reboot cp2 [13:39:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:40:01] Ah. Thanks Reception123. In a script import, you can select to add templates/pages/modules in subpages under User:Example/Imports/... (Which is one of the options of Special:Import) [13:41:09] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNgxd [13:41:11] [02miraheze/dns] 07paladox 0367fd043 - Revert "depool cp2 to reboot it" This reverts commit ddd82080135ec5ba4aea15d27b99bc9e8ebf70e9. [13:51:18] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:53:00] RECOVERY - Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:02:41] PROBLEM - www.guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:41] PROBLEM - guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:25] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw3 [14:03:31] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:8902::f03c:91ff:fe07:444e/cpweb [14:03:33] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:8902::f03c:91ff:fe07:444e/cpweb [14:03:46] uh [14:04:31] PROBLEM - Puppet on cp5 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:04:40] mw3 i cannot ssh into. [14:05:01] PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [14:05:13] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:05:20] hmm, why so many puppet errors? [14:05:29] oh [14:06:03] RECOVERY - www.guiasdobrasil.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'sni61771.cloudflaressl.com' will expire on Sat 02 Feb 2019 11:59:59 PM GMT +0000. [14:06:23] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [14:06:33] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 3 backends are healthy [14:06:33] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [14:06:37] RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:06:39] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:06:45] RECOVERY - guiasdobrasil.com.br - LetsEncrypt on sslhost is OK: OK - Certificate sni61771.cloudflaressl.com will expire on Sat 02 Feb 2019 11:59:59 PM GMT +0000. [14:07:15] PROBLEM - Puppet on lizardfs1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:07:17] PROBLEM - Puppet on cp4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:07:37] PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/home/johnflewis] [14:07:41] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [14:08:17] PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:08:19] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:08:21] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [14:08:45] PROBLEM - Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [14:09:05] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [14:09:21] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [14:10:55] PROBLEM - guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:11:05] RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [14:12:13] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [14:12:15] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [14:12:17] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:13:16] RECOVERY - Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:13:18] RECOVERY - Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:13:23] RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:13:55] PROBLEM - guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:56] PROBLEM - Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [14:15:26] PROBLEM - www.guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:57] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [14:16:17] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [14:16:48] RECOVERY - Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:17:03] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:37:57] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [14:38:26] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [15:00:45] i've disabled puppet on misc2 temporarily when i test something [15:02:15] PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 11 minutes ago with 0 failures [15:03:13] PROBLEM - Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 12 minutes ago with 0 failures [15:09:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2eO [15:09:58] [02miraheze/puppet] 07paladox 0315831c8 - Enable db ssl for piwik I've already enabled this in piwik just adding to puppet. [15:11:13] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [15:11:52] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:45] is it necessary to have both of those running at the same time? [15:13:07] Voidwalker yeh, one is running icinga 1.x and one is icinga 2.x [15:13:17] we are trying to migrate to icinga 2.x (big upgrade) [15:13:21] but needs testing [15:27:54] tempted to say quiet one of them and make paladox op [15:27:55] >:( [15:28:13] PuppyKun op? [15:28:27] +z so you would get the messages [15:28:53] PuppyKun what would be the command to only show messages from icinga2-miraheze to opped users? [15:29:52] oh thanks [15:30:06] do you want an expiry time on those? [15:30:15] (eir can auto -z and remove the quiet) [15:30:31] PuppyKun yeh, maybe a day or 2 [15:30:32] please [16:59:41] hello [16:59:52] hi [17:19:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2TM [17:19:41] [02miraheze/puppet] 07paladox 0347e53ee - Set icingadmin email to noreply@ [17:20:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2TS [17:20:28] [02miraheze/puppet] 07paladox 0319f888f - set irc user to noreply@ in icinga2 [17:40:31] paladox: you forgot to deop [17:51:45] Reception123 i didn't forget too [17:51:58] i need to be opped to see icinga2 bot. [17:52:00] paladox: what is the reason to stay opped for so long? [17:52:08] paladox: did you set a mode on it? [17:52:18] Reception123 PuppyKun did it for me :) [17:52:24] to prevent spamming the message twice [17:52:26] you could probably test it in another channel tbh, but I guess it's fine, sorry for deoping [17:54:20] ok [17:56:56] i plan to do the icinga upgrade on misc1 today Reception123 [17:56:58] so this is temp [17:58:49] ok [18:06:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2It [18:06:10] [02miraheze/puppet] 07paladox 033338db7 - Update director.pp [18:11:10] PROBLEM - Bacula - Static Lizardfs1 on bacula1 is CRITICAL: CRITICAL: no terminated jobs [18:31:31] PROBLEM - wiki.nvda-nl.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate wiki.nvda-nl.org expires in 15 day(s) (Tue 14 Aug 2018 06:29:16 PM GMT +0000). [18:31:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2Ln [18:31:44] [02miraheze/ssl] 07MirahezeSSLBot 0318b64d5 - Bot: Update SSL cert for wiki.nvda-nl.org [18:33:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2Ll [18:33:14] [02miraheze/puppet] 07paladox 0384e6bae - Update ssl-renew.sh [18:48:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fN2LN [18:48:18] [02miraheze/puppet] 07paladox 03d465530 - Cleanup /var/spool/icinga2/perfdata files older then 3 days [18:48:20] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [18:48:21] [02puppet] 07paladox opened pull request 03#786: Cleanup /var/spool/icinga2/perfdata files older then 3 days - 13https://git.io/fN2LA [18:59:31] PROBLEM - wiki.nvda-nl.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.nvda-nl.org and port 443: Connection refused [19:01:15] PROBLEM - wiki.nvda-nl.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate wiki.nvda-nl.org expires in 15 day(s) (Tue 14 Aug 2018 06:29:16 PM GMT +0000). [19:01:53] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:03:13] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:05:04] RECOVERY - Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:05:16] RECOVERY - HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 557 bytes in 0.041 second response time [19:05:18] RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:05:35] that was caused by a private cert which i updated. [19:05:54] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:13:16] by updated i mean i replaced the private cert as it broke. So it all works now. [19:19:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2qB [19:19:53] [02miraheze/mw-config] 07paladox 03cb486a1 - Increase wgMaxImageArea for nonbinarywiki to 2.5e7 [19:42:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2mq [19:42:05] [02miraheze/puppet] 07paladox 035cd321f - Update config.yaml [19:46:46] [02miraheze/services] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2mz [19:46:48] [02miraheze/services] 07paladox 03d4a1aa8 - Add pdacwiki to services [19:52:27] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [19:52:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/fN2mb [19:52:30] [02miraheze/puppet] 07paladox 030f992c1 - Add ServicesSettings [19:52:31] [02puppet] 07paladox opened pull request 03#787: Add ServicesSettings - 13https://git.io/fN2mN [19:52:48] [02puppet] 07paladox synchronize pull request 03#787: Add ServicesSettings - 13https://git.io/fN2mN [19:52:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/fN2mA [19:52:51] [02miraheze/puppet] 07paladox 03136d650 - Update init.pp [19:56:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+1/-0/±0] 13https://git.io/fN2YL [19:56:31] [02miraheze/puppet] 07paladox 038671aa9 - Create ServicesSettings.php [19:56:32] [02puppet] 07paladox synchronize pull request 03#787: Add ServicesSettings - 13https://git.io/fN2mN [19:56:42] [02puppet] 07paladox closed pull request 03#787: Add ServicesSettings - 13https://git.io/fN2mN [19:56:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fN2Ym [19:56:45] [02miraheze/puppet] 07paladox 0342a9e38 - Add ServicesSettings (#787) * Add ServicesSettings * Update init.pp * Create ServicesSettings.php [19:56:46] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [19:56:48] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [19:57:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2Ys [19:57:20] [02miraheze/puppet] 07paladox 039d701c7 - Update ServicesSettings.php [19:58:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2YB [19:58:55] [02miraheze/puppet] 07paladox 03dd9fb16 - Update ServicesSettings.php [19:59:43] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:59:59] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2YR [20:00:01] [02miraheze/mw-config] 07paladox 039ae4ea3 - Load ServicesSettings.php at the top of LS.php [20:01:43] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:06:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2YD [20:06:38] [02miraheze/mw-config] 07paladox 03436fc7a - Update LocalWiki.php [20:10:06] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2Yp [20:10:07] [02miraheze/mw-config] 07paladox 0395beead - Update LocalWiki.php [20:30:23] PROBLEM - Restbase on misc3 is CRITICAL: connect to address 185.52.1.144 and port 7231: Connection refused [20:30:33] known [20:34:23] RECOVERY - Restbase on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.144 port 7231 [20:38:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN23t [20:38:56] [02miraheze/puppet] 07paladox 033647e1f - Update config.yaml.erb [20:39:23] PROBLEM - Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:39:43] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:40:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN233 [20:40:03] [02miraheze/puppet] 07paladox 03881628c - Update config.yaml.erb [20:44:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN23E [20:44:49] [02miraheze/puppet] 07paladox 03fd0f4d3 - Update mathoid.systemd [20:45:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN23u [20:45:33] [02miraheze/puppet] 07paladox 03879b9fc - Update mathoid.systemd [20:47:44] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:49:46] [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fN236 [20:49:47] [02miraheze/mw-config] 07paladox 038bb93c0 - Revert "Update LocalWiki.php" This reverts commit 95beead0207bd17827b6dd9b777ce3bee0cb69cb. [20:49:49] [02miraheze/mw-config] 07paladox 03ee6daa4 - Revert "Update LocalWiki.php" This reverts commit 436fc7a0930ccc11661a1a1732a1639e7db3257e. [20:59:43] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:35:27] RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [21:46:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2G5 [21:46:34] [02miraheze/puppet] 07paladox 03d11f1f8 - Replace concurrent_checks with MaxConcurrentChecks concurrent_checks is deprecated and replaced with MaxConcurrentChecks [21:47:09] [02puppet] 07paladox closed pull request 03#786: Cleanup /var/spool/icinga2/perfdata files older then 3 days - 13https://git.io/fN2LA [21:47:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2GF [21:47:12] [02miraheze/puppet] 07paladox 035ad701a - Cleanup /var/spool/icinga2/perfdata files older then 3 days (#786) [21:47:14] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [21:47:15] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [21:51:47] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [21:51:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fN2Zf [21:51:50] [02miraheze/puppet] 07paladox 03bf837d6 - Update checker icinga2 [21:51:51] [02puppet] 07paladox opened pull request 03#788: Update checker icinga2 - 13https://git.io/fN2ZJ [21:54:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2Zk [21:54:21] [02miraheze/puppet] 07paladox 035548251 - Update params.pp [21:54:29] [02puppet] 07paladox closed pull request 03#788: Update checker icinga2 - 13https://git.io/fN2ZJ [21:54:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2ZI [21:54:32] [02miraheze/puppet] 07paladox 034f842c2 - Update checker icinga2 (#788) [21:54:34] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [21:54:36] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [21:55:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2ZL [21:55:04] [02miraheze/puppet] 07paladox 0347a5071 - Update params.pp [22:09:27] PROBLEM - Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:09:47] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:12:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+3/-1/±1] 13https://git.io/fN2ZK [22:12:03] [02miraheze/puppet] 07paladox 03e242ba6 - icinga2: Add mail scripts [22:13:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/fN2Zi [22:13:40] [02miraheze/puppet] 07paladox 03f793735 - icinga2: fix location of mail scripts [22:24:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2nn [22:24:04] [02miraheze/puppet] 07paladox 0362f6b23 - Update params.pp [22:33:26] RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [22:33:46] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [22:44:00] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [22:44:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fN2nF [22:44:03] [02miraheze/puppet] 07paladox 03654d00b - icinga2.miraheze.org -> icinga.miraheze.org [22:44:04] [02puppet] 07paladox opened pull request 03#789: icinga2.miraheze.org -> icinga.miraheze.org - 13https://git.io/fN2nb [22:45:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fN2nx [22:45:25] [02miraheze/puppet] 07paladox 036038000 - Update apache.conf [22:45:26] [02puppet] 07paladox synchronize pull request 03#789: icinga2.miraheze.org -> icinga.miraheze.org - 13https://git.io/fN2nb [22:46:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fN2nj [22:46:26] [02miraheze/puppet] 07paladox 0365bb49c - Update web.pp [22:46:27] [02puppet] 07paladox synchronize pull request 03#789: icinga2.miraheze.org -> icinga.miraheze.org - 13https://git.io/fN2nb [22:46:43] [02puppet] 07paladox closed pull request 03#789: icinga2.miraheze.org -> icinga.miraheze.org - 13https://git.io/fN2nb [22:46:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/fN2cf [22:46:46] [02miraheze/puppet] 07paladox 03c3b621d - icinga2.miraheze.org -> icinga.miraheze.org (#789) * icinga2.miraheze.org -> icinga.miraheze.org * Update apache.conf * Update web.pp [22:46:48] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [22:46:49] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [22:47:17] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [22:47:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fN2cU [22:47:20] [02miraheze/puppet] 07paladox 03879627f - Switch misc1 to icinga2 [22:47:22] [02puppet] 07paladox opened pull request 03#790: Switch misc1 to icinga2 - 13https://git.io/fN2cT [22:47:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fN2cL [22:47:51] [02miraheze/puppet] 07paladox 0314af577 - Update site.pp [22:47:53] [02puppet] 07paladox synchronize pull request 03#790: Switch misc1 to icinga2 - 13https://git.io/fN2cT [22:48:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fN2ct [22:48:55] [02miraheze/puppet] 07paladox 034fc345f - Update misc1.yaml [22:48:57] [02puppet] 07paladox synchronize pull request 03#790: Switch misc1 to icinga2 - 13https://git.io/fN2cT [22:49:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fN2cq [22:49:16] [02miraheze/puppet] 07paladox 03d885394 - Update mw1.yaml [22:49:18] [02puppet] 07paladox synchronize pull request 03#790: Switch misc1 to icinga2 - 13https://git.io/fN2cT [22:49:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fN2cm [22:49:28] [02miraheze/puppet] 07paladox 03db5a7dc - Update mw2.yaml [22:49:29] [02puppet] 07paladox synchronize pull request 03#790: Switch misc1 to icinga2 - 13https://git.io/fN2cT [22:49:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fN2cY [22:49:40] [02miraheze/puppet] 07paladox 030ea6194 - Update mw3.yaml [22:49:42] [02puppet] 07paladox synchronize pull request 03#790: Switch misc1 to icinga2 - 13https://git.io/fN2cT [22:51:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fN2cs [22:51:20] [02miraheze/puppet] 07paladox 03454c92b - Update misc1.yaml [22:51:22] [02puppet] 07paladox synchronize pull request 03#790: Switch misc1 to icinga2 - 13https://git.io/fN2cT [22:52:29] !log upgrade icinga to icinga2 on misc1 [22:52:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:52:42] [02puppet] 07paladox closed pull request 03#790: Switch misc1 to icinga2 - 13https://git.io/fN2cT [22:52:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/fN2cZ [22:52:45] [02miraheze/puppet] 07paladox 033359e33 - Switch misc1 to icinga2 (#790) * Switch misc1 to icinga2 * Update site.pp * Update misc1.yaml * Update mw1.yaml * Update mw2.yaml * Update mw3.yaml * Update misc1.yaml [22:52:46] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [22:52:48] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [23:04:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2cr [23:04:21] [02miraheze/puppet] 07paladox 03b3365a5 - Update irc.py [23:05:24] RECOVERY - misc1 Current Load on misc1 is OK: OK - load average: 0.97, 1.47, 1.26 [23:05:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [23:05:48] PROBLEM - bacula1 Bacula Databases db4 on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:06:56] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:07:06] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:07:10] PROBLEM - bacula1 Bacula Lizardfs1 Lizardfs Chunkserver1 on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:16:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2ch [23:16:54] [02miraheze/puppet] 07paladox 03e396b30 - Set nrpe_timeout to '60s' [23:17:27] PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:34:22] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [23:39:38] RECOVERY - bacula1 Bacula Lizardfs1 Lizardfs Chunkserver1 on bacula1 is OK: OK: Full, 7 files, 116.4KB, 2018-07-29 00:06:00 (23.6 hours ago) [23:45:36] PROBLEM - bacula1 Bacula Lizardfs1 Lizardfs Chunkserver1 on bacula1 is CRITICAL: CRITICAL: Timeout or unknown client: lizardfs1-fd [23:46:16] PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:51:35] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:51:57] RECOVERY - bacula1 Bacula Databases db4 on bacula1 is OK: OK: Diff, 293946 files, 49.86GB, 2018-07-15 02:57:00 (2.1 weeks ago) [23:52:17] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:52:37] RECOVERY - bacula1 Bacula Lizardfs1 Lizardfs Chunkserver1 on bacula1 is OK: OK: Full, 7 files, 116.4KB, 2018-07-29 00:06:00 (23.8 hours ago) [23:52:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:53:29] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2Cd [23:53:39] [02miraheze/puppet] 07paladox 034870171 - Move test1 back to mediawiki [23:54:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fN2Cb [23:54:11] [02miraheze/puppet] 07paladox 036a6f26f - Update test1.yaml [23:56:41] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:58:05] PROBLEM - bacula1 Bacula Lizardfs1 Lizardfs Chunkserver1 on bacula1 is CRITICAL: CRITICAL: Timeout or unknown client: lizardfs1-fd [23:58:55] PROBLEM - test1 Check correctness of the icinga configuration on test1 is CRITICAL: Icinga configuration contains errors [23:59:36] rip [23:59:49] paladox: hi [23:59:52] PuppyKun hi!