[00:40:32] https://phabricator.miraheze.org/T4352 Any news? [00:40:33] [ ⚓ T4352 Create the Miraheze-feed-es channel ] - phabricator.miraheze.org [01:19:46] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.88, 9.73, 6.11 [01:21:42] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 3.71, 7.36, 5.67 [01:23:38] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.05, 6.04, 5.39 [03:23:15] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.07, 6.66, 6.03 [03:27:07] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.06, 6.57, 6.17 [03:30:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeqM2 [03:30:11] [02miraheze/services] 07MirahezeSSLBot 034935d12 - BOT: Updating services config for wikis [03:32:58] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.84, 7.26, 6.52 [03:36:52] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.95, 7.73, 6.84 [04:06:09] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:08:07] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 27 minutes ago with 0 failures [04:15:44] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:38:44] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [05:05:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeqDo [05:05:10] [02miraheze/services] 07MirahezeSSLBot 031d8fdf7 - BOT: Updating services config for wikis [05:22:51] PROBLEM - glusterfs1 Puppet on glusterfs1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:41:20] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 4.90, 7.08, 7.86 [05:51:19] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.81, 5.48, 6.72 [05:59:20] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.13, 6.47, 6.56 [06:01:19] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.38, 6.18, 6.45 [06:02:51] RECOVERY - glusterfs1 Puppet on glusterfs1 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:25:29] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3146 MB (13% inode=94%); [06:55:22] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.72, 6.84, 5.97 [06:59:19] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.60, 6.57, 6.06 [07:03:19] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.30, 7.18, 6.39 [07:05:19] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.45, 7.39, 6.57 [07:07:19] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.83, 6.68, 6.40 [07:22:47] PROBLEM - glusterfs2 Puppet on glusterfs2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:22:47] RECOVERY - glusterfs2 Puppet on glusterfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:23:21] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.15, 6.64, 5.65 [08:25:19] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.73, 7.09, 5.91 [08:27:19] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.26, 6.78, 5.94 [12:10:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeqF0 [12:10:13] [02miraheze/services] 07MirahezeSSLBot 037ee64d0 - BOT: Updating services config for wikis [12:52:42] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.21, 1.95, 1.39 [12:54:41] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.25, 1.69, 1.36 [13:41:59] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.71, 1.64, 1.51 [13:43:57] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.37, 1.59, 1.51 [14:42:51] PROBLEM - glusterfs1 Puppet on glusterfs1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:02:51] RECOVERY - glusterfs1 Puppet on glusterfs1 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [15:42:47] PROBLEM - glusterfs2 Puppet on glusterfs2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:52:51] PROBLEM - glusterfs1 Puppet on glusterfs1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:20:41] !log purge binary logs before '2019-09-10 02:00:00'; [16:20:42] ^ paladox [16:20:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:20:49] thanks! [16:20:50] urgh forgot to mention on db4 again [16:21:20] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JemTJ [16:21:21] [02miraheze/puppet] 07Reception123 039bc7b81 - add rhinosf1 to mw-admins [16:22:51] RECOVERY - glusterfs1 Puppet on glusterfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:28:57] !log created mail account for 'rhinosf1' [16:29:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:29:08] RhinosF1: please check if it works (you can do that easily on mail.miraheze.org) [16:35:57] !log added RhinosF1 to @miraheze/mw-admins team on github [16:36:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:36:37] Reception123: which ID did you use :P [16:37:00] JohnLewis: 507, right after Void's 506 [16:37:11] void has an email..? [16:37:26] Reception123: no email from github [16:37:30] JohnLewis: yes, SPF created it [16:37:41] mkay [16:37:49] JohnLewis: see -staff [16:38:53] RhinosF1: try again now? [16:39:36] Reception123: https://github.com/orgs/miraheze/people [16:39:37] [ People · Miraheze · GitHub ] - github.com [16:39:58] RhinosF1: :) [16:42:47] RECOVERY - glusterfs2 Puppet on glusterfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:48:06] !log sudo -u www-data php CreateLocalAccount.php --wiki=macrochasmwiki ' GambinoMacro' [16:48:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:48:39] !log on test1 [16:48:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:49:35] RhinosF1: hm? [16:49:40] !log created Matomo account for RhinosF1 [16:49:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:49:48] Reception123: yes? [16:49:58] RhinosF1: you logged "on test1" [16:50:17] Reception123: because I forgot to note it on the bit above [16:50:38] RhinosF1: ohh ok [16:50:46] tbh It doesn't matter for that really what it was ran on [16:50:49] anyway test1 shouldn't really be used for actual maintenance scripts unless it's imports [16:51:01] it was what I was logged intp [16:59:36] !log MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 9205; [16:59:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:33:52] PROBLEM - mw5 Puppet on mw5 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [17:34:02] PROBLEM - mw6 Puppet on mw6 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [17:35:30] paladox: we're already getting mw5 and mw6? [17:43:42] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [17:44:02] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:45:31] RhinosF1: this will be of interest to you https://twitter.com/Wikimedia/status/1171476639064055808?s=20 [17:45:31] [ Wikimedia op Twitter: "Today we announced $2.5 million in support from @craignewmark philanthropies to advance security efforts for Wikipedia and the other Wikimedia projects. https://t.co/rS4LaXjoRr" ] - twitter.com [17:50:18] Zppix: strange timing [17:50:35] RhinosF1: I think it was a very generous donation [17:51:35] :O [17:51:41] Zppix: it is [17:51:53] Just wondering what prompted it [17:53:54] [02mw-config] 07Reception123 closed pull request 03#2752: add .naf per T4702 - 13https://git.io/fjjjx [17:53:55] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JemI4 [17:53:57] [02miraheze/mw-config] 07RhinosF1 031886f78 - add .naf per T4702 (#2752) [17:55:09] RhinosF1: I mean an ~18 hour ddos could do thaty [17:57:27] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JemIR [17:57:28] [02miraheze/puppet] 07JohnFLewis 03c430041 - add mw5,6 [17:58:30] Zppix: yeah [18:03:28] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JemIr [18:03:30] [02miraheze/mw-config] 07RhinosF1 03b0e7a5f - Add Travis alerts for RhinosF1 [18:03:44] Reception123: first commit [18:03:55] PROBLEM - mw5 php-fpm on mw5 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.2' [18:03:58] PROBLEM - mw6 php-fpm on mw6 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.2' [18:04:15] PROBLEM - mw6 HTTPS on mw6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:31] PROBLEM - mw5 HTTPS on mw5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:06:54] :) [18:06:56] whats with mw5 and 6? new mw* server? [18:07:34] * Zppix is debating if i want to continue un-f###ing irc relay [18:07:36] !log created an icinga account for rhinosf1 [18:07:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:10:19] Reception123: (see my last question about mw) [18:10:48] Zppix: If I could answer I would [18:10:50] JohnLewis: ^ [18:10:59] oh :P [18:11:13] capacity increase, cost decrease msotly [18:14:50] so yes? [18:25:05] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JemIj [18:25:07] [02miraheze/puppet] 07Reception123 033025478 - add rhinosf1 to icinga [18:26:00] Reception123: read what you just committed [18:26:19] forgot to click save :( [18:26:20] display_name [18:26:38] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JemLe [18:26:40] [02miraheze/puppet] 07Reception123 030c6328d - correction [18:27:02] paladox: looking at the PRs what are you doing with https://github.com/miraheze/puppet/pull/1009 ? [18:27:03] [ vpncloud: Automate firewall by paladox · Pull Request #1009 · miraheze/puppet · GitHub ] - github.com [18:27:17] we are not going to use vpncloud [18:27:20] so i should abandon it [18:27:27] [02puppet] 07paladox closed pull request 03#1009: vpncloud: Automate firewall - 13https://git.io/fjRod [18:27:28] ok [18:27:30] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [18:27:32] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [18:28:03] paladox: and https://github.com/miraheze/puppet/pull/1060? [18:28:04] [ stunnel: Set TCP_NODELAY=1 for both the server and client by paladox · Pull Request #1060 · miraheze/puppet · GitHub ] - github.com [18:28:16] [02puppet] 07paladox closed pull request 03#1060: stunnel: Set TCP_NODELAY=1 for both the server and client - 13https://git.io/fjH6l [18:28:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [18:28:19] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:28:20] Reception123 abandoned. [18:28:25] RECOVERY - mw6 HTTPS on mw6 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.011 second response time [18:29:02] RECOVERY - mw5 HTTPS on mw5 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.012 second response time [18:34:43] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 2 minutes ago with 29 failures. Failed resources (up to 3 shown): Package[php7.2-cli],File[/etc/php/7.2/cli/conf.d],File[/etc/php/7.2/cli/php.ini],Package[php7.2-fpm] [18:36:00] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 3 minutes ago with 29 failures. Failed resources (up to 3 shown): Package[php7.2-cli],File[/etc/php/7.2/cli/conf.d],File[/etc/php/7.2/cli/php.ini],Package[php7.2-fpm] [18:44:55] RECOVERY - mw6 php-fpm on mw6 is OK: PROCS OK: 3 processes with command name 'php-fpm7.2' [18:45:10] RECOVERY - mw5 php-fpm on mw5 is OK: PROCS OK: 3 processes with command name 'php-fpm7.2' [18:48:47] I'm such an idiot [18:48:51] I know why its not working [18:48:52] ... [18:49:05] You know if i checked my email every now and then [18:50:52] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:52:52] PROBLEM - mw4 Puppet on mw4 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 1 day ago with 0 failures [18:57:00] what? you mean you don't check at least twice/three times a day? [19:00:03] :P [19:04:40] [02miraheze/puppet] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±8] 13https://git.io/JemLF [19:04:42] [02miraheze/puppet] 07JohnFLewis 03665c785 - add into configs [19:04:43] [02miraheze/puppet] 07JohnFLewis 030b035f3 - Merge branch 'master' of github.com:miraheze/puppet [19:08:28] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 6.61, 4.19, 2.74 [19:08:58] :OOOO [19:08:59] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 4 backends are down. mw1 mw3 mw5 mw6 [19:09:01] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:09:06] PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:09:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:09:19] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=lizardfs4.miraheze.org&var-port=9100 [19:09:20] [ Grafana ] - grafana.miraheze.org [19:10:02] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 4 backends are down. mw2 mw4 mw5 mw6 [19:10:07] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw1 mw4 mw5 mw6 [19:10:35] Someone caused io to spike [19:12:08] SWEET SWEET SWEET VICTORY [19:12:12] MH DISCORD IS BACK [19:12:13] !!!! [19:12:36] RhinosF1: i fixed it! [19:12:59] uhhhh, lizardfs4 is too slow to do anything... [19:15:21] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:16:44] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [19:18:42] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 9% [19:24:21] PROBLEM - lizardfs4 SSH on lizardfs4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:24:46] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:24:48] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:25:15] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 52% [19:25:30] !log reboot lizardfs4 [19:25:35] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [19:25:37] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24516 bytes in 0.696 second response time [19:25:41] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 54% [19:27:14] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 32% [19:28:51] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24522 bytes in 0.391 second response time [19:28:53] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24516 bytes in 0.389 second response time [19:29:21] PROBLEM - mw4 HTTPS on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:28] !log [20:25:30] <+paladox> !log reboot lizardfs4 [19:29:31] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 26% [19:29:32] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [19:29:37] !log umounted mediawiki-static temporarily on mw1 [19:30:00] RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 20 minutes ago with 0 failures [19:30:14] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [19:30:29] RECOVERY - lizardfs4 SSH on lizardfs4 is OK: SSH OK - OpenSSH_7.9p1 Debian-10 (protocol 2.0) [19:31:09] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [19:31:27] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 54% [19:31:36] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.60, 0.79, 0.31 [19:32:14] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:33:08] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 32% [19:33:32] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:35:18] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [19:35:23] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [19:35:27] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24500 bytes in 0.678 second response time [19:37:13] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 8% [19:37:17] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:37:21] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 5% [19:37:28] RECOVERY - mw4 HTTPS on mw4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.008 second response time [19:38:19] !log restarted php-fpm on mw2 [19:38:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:39:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:40:03] !log [19:40:06] !log [20:25:30] <+paladox> !log reboot lizardfs4 [19:40:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:40:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:01:37] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 3.55, 3.52, 2.96 [20:03:33] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 2.78, 3.18, 2.90 [20:14:16] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 5.32, 4.21, 3.49 [20:18:14] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 3.26, 3.97, 3.59 [20:22:14] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 4.25, 3.48, 3.43 [20:28:14] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 2.57, 3.67, 3.61 [20:30:15] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 4.13, 3.97, 3.73 [20:32:14] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 3.50, 3.75, 3.68 [20:35:44] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [20:35:45] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/Jemmk [20:35:47] [02miraheze/mw-config] 07RhinosF1 032c67a9a - testing SP UPD config for T4708 [20:36:06] [02mw-config] 07RhinosF1 opened pull request 03#2753: testing SP UPD config for T4708 - 13https://git.io/JemmI [20:36:14] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 4.83, 4.03, 3.79 [20:36:16] RhinosF1 that config is already defined. [20:37:09] miraheze/mw-config/RhinosF1-patch-1/2c67a9a - RhinosF1 The build passed. https://travis-ci.org/miraheze/mw-config/builds/583349634 [20:38:00] [02mw-config] 07RhinosF1 closed pull request 03#2753: testing SP UPD config for T4708 - 13https://git.io/JemmI [20:38:13] paladox: then that's not doing as intentded [20:38:35] you'll need to fix it if it's not working as intended :) [20:39:08] paladox: Well in theory (as that's set to false) shouldn't Friend/Foe links not show on profiles [20:39:24] I wouldn't know :) [20:39:38] paladox: that's what I'd expect [20:39:40] you'll have to look in SP source code [20:40:28] Reception123: you thought that as well didn't you? [20:42:02] * RhinosF1 might have an idea - I'm going to enable it on test1 and see if I'm right [20:44:38] Yeah... [20:44:53] Go ahead, maybe you'll get it working :) [20:46:09] hi [20:46:20] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-2 - 13https://git.io/vbvb3 [20:46:21] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/Jemmc [20:46:23] [02miraheze/mw-config] 07RhinosF1 032197ab2 - test UserProfileDisplay [20:46:32] [02mw-config] 07RhinosF1 opened pull request 03#2754: test UserProfileDisplay - 13https://git.io/JemmC [20:46:44] hispano76_: thx [20:47:47] miraheze/mw-config/RhinosF1-patch-2/2197ab2 - RhinosF1 The build passed. https://travis-ci.org/miraheze/mw-config/builds/583353566 [20:48:27] [02mw-config] 07RhinosF1 closed pull request 03#2754: test UserProfileDisplay - 13https://git.io/JemmC [20:48:29] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jemm4 [20:48:30] [02miraheze/mw-config] 07RhinosF1 03e57f9c2 - test UserProfileDisplay (#2754) [20:58:46] !log /srv/mediawiki/w/composer.phar install - mw4 [20:59:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:02:05] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:03:17] hispano76_: I'm doing https://phabricator.miraheze.org/T4684 for you [21:03:18] [ ⚓ T4684 Request the deletion of mediatecawiki and ucroniasantiguowiki ] - phabricator.miraheze.org [21:05:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JemmS [21:05:10] [02miraheze/services] 07MirahezeSSLBot 03531a922 - BOT: Updating services config for wikis [21:05:26] !log depool mw4 [21:05:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:06:14] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 1.32, 3.53, 3.97 [21:10:14] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 0.18, 1.62, 3.07 [21:26:21] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:26:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:26:38] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:27:37] 503 [21:27:42] paladox: RhinosF1 [21:27:51] Zppix: known [21:27:53] yup, should be resolving. [21:28:19] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24516 bytes in 2.814 second response time [21:30:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:30:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:31:13] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03revert-2754-RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JemY3 [21:31:15] [02miraheze/mw-config] 07RhinosF1 0378ad2b4 - Revert "test UserProfileDisplay (#2754)" This reverts commit e57f9c2479c2c5066bde5e58868de7ec63606134. [21:31:16] [02mw-config] 07RhinosF1 created branch 03revert-2754-RhinosF1-patch-2 - 13https://git.io/vbvb3 [21:31:21] paladox: im not getting favicon on meta again on cp2 [21:31:42] [02mw-config] 07RhinosF1 opened pull request 03#2755: Revert "test UserProfileDisplay" - 13https://git.io/JemYs [21:31:53] [02mw-config] 07RhinosF1 closed pull request 03#2755: Revert "test UserProfileDisplay" - 13https://git.io/JemYs [21:31:55] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JemYZ [21:31:56] [02miraheze/mw-config] 07RhinosF1 03cc5c31d - Revert "test UserProfileDisplay (#2754)" (#2755) This reverts commit e57f9c2479c2c5066bde5e58868de7ec63606134. [21:31:58] [02miraheze/mw-config] 07RhinosF1 deleted branch 03revert-2754-RhinosF1-patch-2 [21:31:59] [02mw-config] 07RhinosF1 deleted branch 03revert-2754-RhinosF1-patch-2 - 13https://git.io/vbvb3 [21:32:09] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 - 13https://git.io/vbvb3 [21:32:10] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 [21:32:12] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [21:32:13] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [21:32:37] miraheze/mw-config/revert-2754-RhinosF1-patch-2/78ad2b4 - RhinosF1 The build has errored. https://travis-ci.org/miraheze/mw-config/builds/583373603 [21:33:13] ^ just git + travis in a mood [21:34:08] miraheze/mw-config/revert-2754-RhinosF1-patch-2/78ad2b4 - RhinosF1 The build has errored. https://travis-ci.org/miraheze/mw-config/builds/583373603 [21:34:24] whats wrong with the favicon on meta? [21:34:37] Zppix: what is wrong with it? [21:34:49] RhinosF1: its not showing for me im on CP2 [21:35:06] and its not my cache [21:35:10] Zppix: i'm normally cp2 and fine [21:35:51] OK RhinosF1 [21:36:03] thanks RhinosF1 [21:36:12] hispano76_: yep, both done [21:36:40] !log ban req.http.Host == meta.miraheze.org && req.url ~ "^/favicon.ico" [21:36:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:36:45] Zppix try now [21:36:50] !log that is on cp2 [21:36:54] paladox: fine now, why does it keep doing that [21:36:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:37:03] i'm not sure :( [21:37:49] * RhinosF1 made git and travis happy again (it was reading a deleted branch) [21:44:56] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%); [21:48:19] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:50:17] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:56:27] [02mw-config] 07RhinosF1 commented on pull request 03#2741: Add wgDisqusShortname to ManageWiki - 13https://git.io/JemYo [21:56:52] [02mw-config] 07RhinosF1 closed pull request 03#2741: Add wgDisqusShortname to ManageWiki - 13https://git.io/fjAtG [21:56:53] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JemYK [21:56:55] [02miraheze/mw-config] 07paladox 03a3c90c5 - Add wgDisqusShortname to ManageWiki (#2741) [22:24:24] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.62, 6.52, 5.30 [22:30:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JemOk [22:30:11] [02miraheze/services] 07MirahezeSSLBot 031c86159 - BOT: Updating services config for wikis [22:32:24] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.25, 6.76, 5.94