[00:01:29] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3725 MB (19% inode=93%); [00:04:20] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.41, 22.15, 17.38 [00:08:16] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.65, 18.99, 17.23 [01:29:22] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.99, 2.22, 1.52 [01:31:26] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.94, 1.72, 1.42 [01:33:28] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.00, 1.53, 1.40 [01:55:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7Li [01:55:16] [02miraheze/services] 07MirahezeSSLBot 03f9d7ad4 - BOT: Updating services config for wikis [02:43:45] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.48, 2.10, 1.41 [02:45:43] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.76, 1.64, 1.32 [05:50:00] !log renamed feralworthwikiwiki to feraltradinghandbookwiki [05:50:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [05:50:23] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7Oj [05:50:24] [02miraheze/services] 07MirahezeSSLBot 035190d54 - BOT: Updating services config for wikis [06:01:42] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jf733 [06:01:44] [02miraheze/ssl] 07Reception123 035ac09d2 - add guia.esporo.net cert [06:15:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf736 [06:15:15] [02miraheze/services] 07MirahezeSSLBot 03bbaef9e - BOT: Updating services config for wikis [06:34:20] https://meta.miraheze.org/wiki/Special:Contributions/SANICANIC [06:34:22] who is this person? [06:34:25] [ User contributions for SANICANIC - Miraheze Meta ] - meta.miraheze.org [07:21:15] Naleksuh: hi. what do you mean? [09:16:02] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 4.43, 3.28, 2.17 [09:17:57] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.17, 2.77, 2.11 [10:35:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7RM [10:35:14] [02miraheze/services] 07MirahezeSSLBot 03f38eb6e - BOT: Updating services config for wikis [10:56:37] Good morning [11:11:34] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 3.88, 3.96, 2.72 [11:13:32] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.40, 3.00, 2.51 [11:17:18] RhinosF1: good morning [11:17:34] hi [12:07:22] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 4.32, 4.76, 2.95 [12:09:17] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.00, 3.78, 2.80 [12:11:15] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.28, 2.93, 2.60 [12:23:26] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2031 MB (10% inode=93%); [12:31:25] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 2804 MB (14% inode=93%); [13:18:40] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 5.89, 6.36, 3.87 [13:24:43] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.02, 3.15, 3.24 [13:27:21] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 1719 MB (8% inode=93%); [13:37:25] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 2732 MB (14% inode=93%); [13:43:35] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 1702 MB (8% inode=93%); [13:44:56] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 5.61, 4.64, 3.54 [13:50:54] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.69, 3.66, 3.48 [13:52:56] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.98, 3.06, 3.28 [13:55:21] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 2713 MB (14% inode=93%); [14:22:08] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.19, 3.41, 3.13 [14:22:36] thats a lot of problems [14:24:05] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 4.76, 3.65, 3.23 [14:24:45] Unexpected error (list index out of range) from catbeard at 2020-06-15 14:24:45.054941. Message was: nope [14:26:03] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 3.11, 3.73, 3.34 [14:28:04] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 4.53, 3.92, 3.44 [14:32:03] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.68, 3.42, 3.35 [14:36:00] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.62, 3.40, 3.40 [15:02:43] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 13.27, 5.22, 3.35 [15:07:56] !log varnish> ban req.http.Host == static.miraheze.org && req.url ~ "^/allthetropeswiki/4/4e/Pain5.jpg" - cp8 [15:07:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:08:35] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 1.55, 3.52, 3.21 [15:10:32] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.20, 3.05, 3.06 [15:12:34] hi [15:13:16] ny-killer[m]: Didn't we ban you? [15:13:48] nope [15:15:38] PROBLEM - bacula2 Bacula Private Git on bacula2 is WARNING: WARNING: Full, 4274 files, 12.40MB, 2020-06-07 15:12:00 (1.1 weeks ago) [15:17:11] paladox: didnt work [15:17:23] Hm? [15:17:31] *gban ny-killer*!*@* [15:17:31] 04ny-killer*!*@* has been globally banned and will be automatically kickbanned from all channels on sight.04 [15:26:32] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 3.29, 3.64, 3.17 [15:28:32] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 4.15, 3.51, 3.16 [15:31:27] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.81, 19.43, 14.59 [15:32:29] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.67, 3.76, 3.40 [15:33:24] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 1968 MB (10% inode=93%); [15:33:30] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.68, 17.41, 14.47 [15:34:28] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 5.36, 4.41, 3.68 [15:35:16] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7iN [15:35:17] [02miraheze/services] 07MirahezeSSLBot 039f96698 - BOT: Updating services config for wikis [15:35:57] paladox: cp8/mw7 and he's still seeing the old image [15:36:26] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.89, 3.33, 3.36 [15:37:20] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 2585 MB (13% inode=93%); [15:41:26] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 1567 MB (8% inode=93%); [15:51:22] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 2566 MB (13% inode=93%); [16:00:11] !log varnish> ban req.http.Host == allthetropes.org && req.url ~ "^/wiki/File:Pain5.jpg" - cp8 [16:00:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:13:47] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.47, 1.80, 1.33 [16:15:48] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.80, 1.38, 1.23 [16:43:57] Well done SigmaBot [16:45:47] paladox: can you ban the ?id [17:03:22] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.06, 18.21, 14.16 [17:04:00] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 6.29, 4.65, 3.01 [17:05:25] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.26, 16.65, 14.19 [17:05:55] what do you mean what do i mean [17:05:57] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.44, 3.85, 2.92 [17:05:59] it was a very simple question [17:07:34] Naleksuh: what was the question, sorry i just joined [17:07:45] https://meta.miraheze.org/wiki/Special:Contributions/SANICANIC [17:07:45] who is this person? [17:07:46] [ User contributions for SANICANIC - Miraheze Meta ] - meta.miraheze.org [17:07:46] Naleksuh: hi. what do you mean? [17:07:47] [ User contributions for SANICANIC - Miraheze Meta ] - meta.miraheze.org [17:09:24] its a miraheze user i dont know what else you want to know? [17:09:52] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.58, 2.94, 2.78 [17:21:47] RhinosF1 huh? [17:21:52] oh [17:22:32] *gban Cocopuff*!*@* [17:22:32] 04Cocopuff*!*@* has been globally banned and will be automatically kickbanned from all channels on sight.04 [17:22:36] that is better [17:25:23] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 1630 MB (8% inode=93%); [17:26:05] paladox: /ban *!?id445178@* [17:27:50] Ty [17:30:53] *gban *!?id445178@* [17:30:53] 04*!?id445178@* has been globally banned and will be automatically kickbanned from all channels on sight.04 [17:32:11] prob shouldnt do *gbans in this channel [17:32:40] oh yeah forgot about the publog [17:33:20] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 2422 MB (12% inode=93%); [17:34:13] Texas: did I not do that? [17:34:21] oh idk [17:39:28] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 1641 MB (8% inode=93%); [17:44:32] PROBLEM - phab1 Current Load on phab1 is CRITICAL: CRITICAL - load average: 4.84, 2.89, 1.28 [17:47:36] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 2405 MB (12% inode=93%); [17:48:37] RECOVERY - phab1 Current Load on phab1 is OK: OK - load average: 1.24, 3.08, 1.83 [18:02:25] PROBLEM - tradwiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address tradwiki.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [18:04:27] RECOVERY - tradwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.tradwiki.org' will expire on Fri 14 Aug 2020 15:24:46 GMT +0000. [18:04:57] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7SA [18:04:58] [02miraheze/mw-config] 07paladox 03c4893fd - Update LocalExtensions.php [18:05:39] paladox: why order change? [18:05:51] RhinosF1: probably to fix popups not working [18:06:03] RhinosF1: it will probably force dependencies to load before popups doeso [18:06:08] Ah that task [18:06:42] Makes sense [18:06:57] Zppix: Is there any tech tasks that you guys need my help with (like config or extension installs)? Please ping me if there is. :) [18:07:07] Texas: check phab [18:07:17] * Texas looks [18:07:41] because that is how it shows on https://www.mediawiki.org/wiki/Extension:Popups ? [18:07:42] [ Extension:Popups - MediaWiki ] - www.mediawiki.org [18:09:18] Zppix: On T5750 is the Chameleon skin request denied? [18:09:25] paladox, Reception123: ^ [18:09:35] .task T5750 [18:09:37] .task T5750 [18:09:37] https://phabricator.miraheze.org/T5750 - File extensions + skin install [Open] authored by Lakelimbo, assigned to None [18:09:39] https://phabricator.miraheze.org/T5750 - File extensions + skin install [Open] authored by Lakelimbo, assigned to None [18:10:16] https://phabricator.miraheze.org/T3388#65891 [18:10:17] [ ⚓ T3388 Add an Skin:Chameleon to my wiki ] - phabricator.miraheze.org [18:10:17] Texas: it will need to go under security review if it hasnt already [18:10:36] Zppix: Should I mark it as blocked or just leave a comment? [18:11:02] thats up to paladox as hes sre duty [18:11:07] k [18:11:11] s/paladox/reception [18:11:11] Zppix meant to say: thats up to reception as hes sre duty [18:11:21] k [18:12:31] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [18:13:05] .t Reception123 [18:13:11] 2020-06-15 - 20:13:11CEST [18:13:19] hmm pretty late there [18:19:12] PROBLEM - www.tradwiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.tradwiki.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [18:19:38] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 1.78, 3.85, 2.97 [18:20:21] Zppix: tradwiki is gone [18:21:13] RECOVERY - www.tradwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.tradwiki.org' will expire on Fri 14 Aug 2020 15:24:46 GMT +0000. [18:21:17] RhinosF1: handling now [18:21:36] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 3.23, 4.15, 3.21 [18:21:53] [02miraheze/ssl] 07Pix1234 created branch 03remove-tradwiki 13https://git.io/Jf79K [18:21:55] [02ssl] 07Pix1234 created branch 03remove-tradwiki - 13https://git.io/vxP9L [18:22:15] [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03remove-tradwiki [+0/-1/±0] 13https://git.io/Jf796 [18:22:17] [02miraheze/ssl] 07Pix1234 0395ea638 - Delete www.tradwiki.org.crt [18:22:54] [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03remove-tradwiki [+0/-0/±1] 13https://git.io/Jf79P [18:22:55] [02miraheze/ssl] 07Pix1234 03970d0f4 - rm tradwiki.org [18:23:07] [02ssl] 07Pix1234 opened pull request 03#322: Remove tradwiki - 13https://git.io/Jf79X [18:23:19] [02ssl] 07Pix1234 closed pull request 03#322: Remove tradwiki - 13https://git.io/Jf79X [18:23:20] [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/Jf791 [18:23:22] [02miraheze/ssl] 07Pix1234 03d18e7eb - Remove tradwiki (#322) * Delete www.tradwiki.org.crt * rm tradwiki.org [18:23:23] [02miraheze/ssl] 07Pix1234 deleted branch 03remove-tradwiki [18:23:25] [02ssl] 07Pix1234 deleted branch 03remove-tradwiki - 13https://git.io/vxP9L [18:23:36] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 1.90, 3.47, 3.08 [18:24:01] [02dns] 07Pix1234 opened pull request 03#149: tradwiki.org no longer points to us - 13https://git.io/Jf79D [18:24:12] paladox: can you merge ^ [18:24:36] [02dns] 07paladox closed pull request 03#149: tradwiki.org no longer points to us - 13https://git.io/Jf79D [18:24:38] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/Jf799 [18:24:39] [02miraheze/dns] 07Pix1234 0319e0109 - tradwiki.org no longer points to us (#149) [18:25:33] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.41, 3.08, 2.98 [18:30:20] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf79h [18:30:21] [02miraheze/services] 07MirahezeSSLBot 0302bd989 - BOT: Updating services config for wikis [18:34:10] ty Zppix [18:34:19] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[www.tradwiki.org],File[www.tradwiki.org_private] [18:34:37] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[www.tradwiki.org],File[www.tradwiki.org_private] [18:34:41] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[www.tradwiki.org],File[www.tradwiki.org_private] [18:34:41] Zppix: ^ [18:34:44] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[www.tradwiki.org],File[www.tradwiki.org_private] [18:34:49] !log rebuildRC and initsitestats on wizpedia101wiki after completion of T5760 [18:34:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:35:09] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[www.tradwiki.org],File[www.tradwiki.org_private] [18:35:19] paladox: why is it alerting it doesnt exist at all in ssl [18:35:43] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[www.tradwiki.org],File[www.tradwiki.org_private] [18:35:46] Zppix: private repo? [18:35:55] it still shouldnt alert [18:36:01] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[www.tradwiki.org],File[www.tradwiki.org_private] [18:36:04] Jun 15 18:31:53 test2 puppet-agent[25430]: (/Stage[main]/Ssl::Hiera/Ssl::Hiera::Certs[tradwikiorg]/File[www.tradwiki.org]) Could not evaluate: Could not retrieve information from environment production source(s) puppet:///ssl/certificates/www.tradwiki.org.crt [18:36:08] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[www.tradwiki.org],File[www.tradwiki.org_private] [18:36:16] paladox: its been removed from ssl repo though [18:36:34] https://github.com/miraheze/ssl/commit/d18e7eb633f750e99666d7c6fb05bed3282f9ac2 [18:36:35] [ Remove tradwiki (#322) · miraheze/ssl@d18e7eb · GitHub ] - github.com [18:37:04] maybe the ssl repo didn't update in time? [18:37:14] paladox: maybe let me force run puppet on jobrunner [18:37:25] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[www.tradwiki.org],File[www.tradwiki.org_private] [18:37:27] Zppix https://github.com/miraheze/ssl/search?q=tradwiki.org&unscoped_q=tradwiki.org [18:37:28] [ Search · tradwiki.org · GitHub ] - github.com [18:38:02] [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7H8 [18:38:04] [02miraheze/ssl] 07Pix1234 0314339ba - rm tradwiki [18:38:59] paladox: fixed, i did a search for it and it didnt find it in redirect.yaml thats weird darn ctrl +f [18:39:53] heh [18:41:07] paladox: oh well not like it broke anything tbh just puppet complaining really [18:42:38] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:42:43] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:42:53] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [18:43:01] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [18:43:03] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:43:06] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:43:27] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:43:54] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:45:43] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:49:26] uh was I pinged? [18:49:48] umm yes [18:50:09] 13:09:18 Zp*ix: On T5750 is the Chameleon skin request denied? [18:50:20] Reception123: ^ apparently you needed to decide that [18:50:27] .task T5750 [18:50:29] https://phabricator.miraheze.org/T5750 - File extensions + skin install [Open] authored by Lakelimbo, assigned to None [19:01:52] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 7.61, 5.27, 3.50 [19:10:04] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.89, 3.29, 3.26 [20:01:10] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.30, 18.75, 15.03 [20:01:35] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 2.42, 4.12, 3.25 [20:03:10] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.05, 18.76, 15.50 [20:03:36] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 3.57, 3.87, 3.26 [20:05:36] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.34, 2.98, 3.01 [20:12:19] paladox: what's with cp7 + load? [20:12:39] HDD? [20:13:34] paladox: change parameters and see discord [20:13:37] !s [20:13:39] Please wait while I check the status of Miraheze Services. [20:13:40] RhinosF1: Status report finished. There are currently 0 dead services and 6 alive services. To view the full report, say !status. [20:13:46] one wiki then [20:15:49] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7dn [20:15:50] [02miraheze/mw-config] 07paladox 037a68f9b - veforall requires visualeditor [20:17:02] paladox: do we need to do toggelextension for all wikis with veforall on [20:28:31] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 6.96, 5.12, 3.60 [20:34:34] i don't think so, at least i haven't herd anymore reports. [20:34:55] paladox: that doesn't mean they aren't any [20:35:08] Fix it because you can [20:35:35] true, but if people really had a problem we would have herd by now. People wouldn't just let their wiki sit there and break. [20:35:36] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.24, 20.55, 17.54 [20:37:04] also i had to go through some hops for this wiki. [20:37:39] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.93, 17.17, 16.67 [20:37:53] 28 wikis have it enabled [20:38:10] paladox: how many of them don't have ve on? [20:38:48] all 28 have ve enabled... [20:39:02] paladox: then nothing to do :) [20:40:49] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 1.17, 3.56, 3.95 [20:44:52] paladox: can we change parameters for ^ [20:45:57] no [20:46:27] i mean it's possible but i'm saying no to that. [20:48:56] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.62, 2.32, 3.23 [20:49:05] RhinosF1 https://github.com/miraheze/puppet/blob/master/modules/base/templates/icinga/nrpe.cfg.erb#L18 [20:49:06] [ puppet/nrpe.cfg.erb at master · miraheze/puppet · GitHub ] - github.com [20:50:17] paladox: if it's flapping between critical and warning constantly and not being handled, the check is broke. [20:50:28] not really broken? [20:50:34] Something needs changing or downtiming/acking [20:50:39] it's flapping because it's on a HDD. [20:50:44] acking/downtiming wouldn't work [20:50:48] downtime has an expiry [20:51:03] acking would be removed as soon as it changes to OK [20:51:35] paladox: It shouldn't be alerting critical constantly if nothing is critical [20:51:50] That's not a feasible situation and leads to complanct [20:52:05] > hdd [20:52:15] If the check isn't suitable for HDDs then it shouldnt be running on a service with a HDD [20:52:32] if you want to have this changed, please file a task. [20:53:56] * RhinosF1 goes ti [20:53:57] To [21:01:01] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2113 MB (10% inode=93%); [21:08:48] paladox: filed, of the other unhandled issues. 4 are LE certs - I assume will auto renew soon or need deleting, 2 are gluster, 1 is test2 puppet (fix or disable), 1 is bacula. [21:10:38] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7bs [21:10:40] [02miraheze/ssl] 07paladox 03a9f73e3 - Update wiki.macc.nyc.crt [21:11:42] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7bn [21:11:43] [02miraheze/dns] 07paladox 03b5cc2bc - Update miraheze.wiki [21:13:50] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7bW [21:13:52] [02miraheze/ssl] 07paladox 032ecb40e - Update miraheze.wiki.crt [21:13:59] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7b8 [21:14:01] [02miraheze/dns] 07paladox 03cc61603 - Update miraheze.wiki [21:15:00] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7bR [21:15:01] [02miraheze/dns] 07paladox 03825e824 - Update miraheze.org [21:15:43] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7bu [21:15:45] [02miraheze/dns] 07paladox 03248b241 - Update miraheze.org [21:16:28] paladox: puppet's just failed for syntax on ns2 [21:16:37] i know [21:17:40] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7bz [21:17:41] [02miraheze/dns] 07paladox 03660e4b0 - Update miraheze.org [21:18:59] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7b2 [21:19:01] [02miraheze/ssl] 07paladox 030ec9a8a - Update m.miraheze.org.crt [21:20:00] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±0] 13https://git.io/Jf7br [21:20:01] [02miraheze/ssl] 07paladox 03e29b8c5 - Update wiki.x1c7.com.crt [21:23:11] RECOVERY - miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Sun 13 Sep 2020 20:13:23 GMT +0000. [21:23:11] RECOVERY - wiki.macc.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Sun 13 Sep 2020 20:10:11 GMT +0000. [21:24:19] paladox: can you please ack/downtime/fix the other alerts [21:26:34] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7bx [21:26:35] [02miraheze/dns] 07paladox 03de1805d - Update orain.org [21:27:01] * RhinosF1 will be filing tasks for any he can't fix himself that aren't ack'd/downtimed and don't have tasks open to handle that are 24 hours old in the morning [21:27:18] We need to start handling alerts not letting them sit there [21:27:34] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7bh [21:27:35] [02miraheze/ssl] 07paladox 0305c9dbd - Update orain.org.crt [21:28:25] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7bj [21:28:27] [02miraheze/dns] 07paladox 035a31624 - Update miraheze.com [21:29:23] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7Nv [21:29:25] [02miraheze/ssl] 07paladox 03f6c795b - Update miraheze.com.crt [21:33:05] RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Sun 13 Sep 2020 20:27:17 GMT +0000. [21:33:08] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Sun 13 Sep 2020 20:27:17 GMT +0000. [21:33:25] RECOVERY - orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Sun 13 Sep 2020 20:27:17 GMT +0000. [21:33:31] RECOVERY - meta.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Sun 13 Sep 2020 20:27:17 GMT +0000. [21:43:23] RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Sun 13 Sep 2020 20:28:55 GMT +0000. [21:44:58] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jf7Nz [21:45:00] [02miraheze/ssl] 07paladox 03f07f217 - Remove wiki.x1c7.com [21:45:01] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [21:45:06] [02ssl] 07paladox opened pull request 03#323: Remove wiki.x1c7.com - 13https://git.io/Jf7Ng [21:45:24] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/Jf7Na [21:45:26] [02miraheze/ssl] 07paladox 03145d912 - Delete wiki.x1c7.com.crt [21:45:27] [02ssl] 07paladox synchronize pull request 03#323: Remove wiki.x1c7.com - 13https://git.io/Jf7Ng [21:45:34] [02ssl] 07paladox closed pull request 03#323: Remove wiki.x1c7.com - 13https://git.io/Jf7Ng [21:45:35] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/Jf7NV [21:45:37] [02miraheze/ssl] 07paladox 0371cea82 - Remove wiki.x1c7.com (#323) * Remove wiki.x1c7.com * Delete wiki.x1c7.com.crt [21:45:38] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [21:45:40] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [21:47:41] paladox: still m.miraheze.org [21:47:50] But then icinga will clean [21:47:56] Yey!!! [21:48:39] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7N6 [21:48:41] [02miraheze/ssl] 07paladox 0388b770b - Update m.miraheze.org.crt [21:48:41] * RhinosF1 now just wishes you'd used a better reason than 'ACK' - 'Service not in use' or 'Alert expected due to Txxx' would be more helpful [21:50:44] And yes I plan to email SRE tommorow morning to share my thoughts on handling alerts better [21:56:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf7Nj [21:56:56] [02miraheze/puppet] 07paladox 030e91814 - Pull in m.miraheze.org cert [21:56:57] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [21:56:59] [02puppet] 07paladox opened pull request 03#1406: Pull in m.miraheze.org cert - 13https://git.io/Jf7Ae [21:57:06] [02puppet] 07paladox closed pull request 03#1406: Pull in m.miraheze.org cert - 13https://git.io/Jf7Ae [21:57:08] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Jf7Av [21:57:09] [02miraheze/puppet] 07paladox 03104c08f - Merge pull request #1406 from miraheze/paladox-patch-8 Pull in m.miraheze.org cert [21:58:15] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS [21:58:16] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [21:59:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7At [21:59:38] [02miraheze/puppet] 07paladox 0352fbe86 - Update init.pp [21:59:57] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Sun 13 Sep 2020 20:48:06 GMT +0000. [22:00:05] Yey [22:00:21] icinga-miraheze must think it's in heaven [22:00:32] No unhandeld alerts [22:02:31] lol [22:05:14] Texas: i dont think I've ever seen that [22:06:41] been a long time [22:07:27] paladox: Thanks for handling what you can. Useful ack messages that tell people something about the issue would be nice though like "Service not in use" or "due to Txxxx" [22:07:40] ACK doesn't tell anyone anything [22:07:46] Voidwalker: certainly [22:09:03] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 4.20, 4.31, 2.95 [22:09:29] icinga-miraheze: you didn't last. At least we have a task for that [22:10:23] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7A6 [22:10:25] [02miraheze/services] 07MirahezeSSLBot 0399eb540 - BOT: Updating services config for wikis [22:11:03] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.40, 3.30, 2.76 [22:30:23] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7x0 [22:30:25] [02miraheze/services] 07MirahezeSSLBot 0311a0564 - BOT: Updating services config for wikis [22:54:41] PROBLEM - phab1 Current Load on phab1 is CRITICAL: CRITICAL - load average: 6.27, 4.82, 2.23 [22:56:38] RECOVERY - phab1 Current Load on phab1 is OK: OK - load average: 1.29, 3.39, 2.02 [23:05:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf7ph [23:05:16] [02miraheze/services] 07MirahezeSSLBot 03bd499b4 - BOT: Updating services config for wikis [23:11:05] Hello yorko-miller[m]! If you have any questions, feel free to ask and someone should answer soon. [23:11:55] ^ trolling on WMF chans [23:15:13] how long are the bans for? [23:17:08] Naleksuh: As long as they need to be. We try to not keep very wide bans. They are logged on a private wiki. [23:18:17] I mean how long are the sigmabot bans for [23:18:20] ten minutes? [23:19:42] Naleksuh: I think eir removes them after a while. [23:47:01] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 4.37, 4.24, 2.37 [23:49:02] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 0.88, 2.99, 2.14