[00:00:19] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJwEj [00:00:21] [02miraheze/services] 07MirahezeSSLBot 033d03bdc - BOT: Updating services config for wikis [00:02:53] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3815 MB (15% inode=93%); [00:03:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JJwuk [00:03:45] [02miraheze/puppet] 07paladox 038ce8acb - Change paradox ssh key [00:03:47] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [00:03:48] [02puppet] 07paladox opened pull request 03#1465: Change paradox ssh key - 13https://git.io/JJwuI [00:03:59] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.67, 22.90, 17.71 [00:04:56] PROBLEM - tla.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tla.awiki.org could not be found [00:05:44] [02puppet] 07paladox closed pull request 03#1465: Change paradox ssh key - 13https://git.io/JJwuI [00:05:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJwuY [00:05:47] [02miraheze/puppet] 07paladox 03de71c7a - Change paradox ssh key (#1465) [00:05:49] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [00:05:50] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [00:08:48] PROBLEM - bacula2 Bacula Private Git on bacula2 is WARNING: WARNING: Full, 4480 files, 13.03MB, 2020-07-26 00:06:00 (1.1 weeks ago) [00:09:57] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.46, 22.28, 19.03 [00:13:58] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 38.74, 27.47, 21.46 [00:14:52] [ANNOUNCEMENT] MirahezeBot v8.0 is now released! This is our biggest release yet. Thank you all for your continued support. -The MirahezeBot team [00:15:57] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.17, 22.95, 20.56 [00:15:59] <-CloudGuy38-> What's the notable changes with Mirahezebot v8.0? [00:17:56] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.56, 18.71, 19.28 [00:42:08] > What's the notable changes with Mirahezebot v8.0? @-CloudGuy38- RhinosF1 will probably be too happy to explain the changes in v8.0, unless @Zppix wants to. Not going to ping him, though, since he's sleeping. 🙂 [00:42:34] <-CloudGuy38-> It's near my sleeping time [00:42:57] > It's near my sleeping time @-CloudGuy38- Oh, okay, sounds good. Tomorrow then? [00:43:19] <-CloudGuy38-> yes, but I will still be here until I go offline [00:46:56] okay, that's cool [01:14:01] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.96, 23.52, 19.72 [01:17:59] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.54, 19.89, 19.04 [01:20:19] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJwgt [01:20:20] [02miraheze/services] 07MirahezeSSLBot 034e77770 - BOT: Updating services config for wikis [01:41:00] RECOVERY - tla.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'tla.awiki.org' will expire on Sat 31 Oct 2020 11:20:36 GMT +0000. [02:05:07] Hello Wagglie! If you have any questions, feel free to ask and someone should answer soon. [02:07:13] do not ban me from this server rhinosf1 [02:15:55] ^ @Zppix [02:16:33] Hola Doug [02:16:54] Please, request global ban to joaquinito01 [02:18:26] Wagglie, your name is awfully similar to Waggle from earlier today. [02:18:47] So no one told you life was gonna be this way [02:19:27] The behavioural characteristics between your messages and Waggle's are uncanny. [02:19:53] 🦆 [02:39:09] Hello RhinoF1! If you have any questions, feel free to ask and someone should answer soon. [02:41:31] @CVT that makes two 🦆 s of an LTA, and I suspect I know who it is [02:55:43] @doug only stewards and sysadmins have op in irc [02:57:35] @Zppix Yeah, I know, but all sysadmins are GS at present, so that's why I used that [02:57:51] Not true [02:57:55] Pioneer isnt [02:58:04] no that's not what I meant [02:58:06] sorry [02:58:14] I meant all GS are sysadmins [02:58:29] Not true [02:58:36] Pioneer isnt a sysadmin [02:59:00] Me and reception are the only CVT members that are sysadmins [03:35:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJwwC [03:35:13] [02miraheze/services] 07MirahezeSSLBot 0320524cb - BOT: Updating services config for wikis [03:42:13] > Me and reception are the only CVT members that are sysadmins @Zppix Yeah, I know, but I think of CVT as an umbrella term that encompasses both stewards and GS. Yeah, The Pioneer is absolutely a steward and thus part of CVT, but only you and Reception123 are GS and sysadmins. [03:43:08] Plus, I didn't want to ping the system administrators group because I didn't want to wake RhinosF1 as I know he's sleeping. So thought I'd just ping the CVT group and someone who was also a sysadmin would answer that. [03:43:19] So that's my thought process. Hope that clarifies. [03:44:22] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.79, 18.44, 15.71 [03:46:22] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.69, 15.77, 15.06 [03:54:50] PROBLEM - wiki.apap04.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.apap04.com' expires in 15 day(s) (Wed 19 Aug 2020 03:46:05 GMT +0000). [03:57:20] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 100% [03:59:22] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 265.39 ms [03:59:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJwwH [03:59:42] [02miraheze/ssl] 07MirahezeSSLBot 033aacd68 - Bot: Update SSL cert for wiki.apap04.com [04:15:12] RECOVERY - wiki.apap04.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.apap04.com' will expire on Sun 01 Nov 2020 02:59:34 GMT +0000. [04:24:28] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 8.20, 5.92, 4.64 [04:26:28] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 4.42, 5.19, 4.53 [04:32:42] !log reception@jobrunner1:/srv/mediawiki/w/extensions/ManageWiki/maintenance$ sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki=xhoriawiki --overwrite [04:32:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:37:29] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JJwr6 [04:37:30] [02miraheze/ssl] 07Reception123 03d4ac398 - add wiki.lct.app cert [04:39:09] !log reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=vilaryawiki /home/reception/Wikipedia-20200802155856.xml [04:39:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:40:20] PROBLEM - oecumene.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'oecumene.org' expires in 15 day(s) (Wed 19 Aug 2020 04:35:14 GMT +0000). [04:45:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJwrQ [04:45:13] [02miraheze/services] 07MirahezeSSLBot 035ff69d4 - BOT: Updating services config for wikis [04:45:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJwrd [04:45:57] [02miraheze/ssl] 07MirahezeSSLBot 03bd5d509 - Bot: Update SSL cert for oecumene.org [04:47:35] Wow, that was fast at getting a custom domain set up, @Reception123. I literally just approved lctwiki a couple hours ago, and didn't even know they wanted a custom domain. They must be familiar with getting custom domains set up. 😛 [04:47:55] @Doug well the process isn't that difficult (at least for users :P) [04:48:15] they just need to read the domain page and point their domain to us and they're done (if they choose Lets Encrypt) [04:48:16] yeah, that's true...it's well explained 🙂 [04:48:25] yep [04:48:40] there's a few more steps for us sysadmins but you get used to it [04:48:45] and it's quick [04:49:24] Looked at what you do on your end, is that literally all you have to do is just generate the LetsEncrypt certificate, modify that certs config file, and update the server settings in ManageWiki? [04:49:27] yeah [04:49:49] there's also the private key that has to be copied to puppet2 [04:49:56] ah, yes [04:49:59] (which is why only puppet-users can fully do certs) [04:50:01] that's the puppet part then [04:50:28] but before when I was mw-admin I did certs but just had to have an ops member (what SRE was called at the time) do the private cert [04:51:05] Oh and if the custom domain is pointed to us via nameservers we also need to add it to the DNS repo [04:51:08] ah, yeah, so now it's simplified since mw-admins can request puppet access [04:51:28] > Oh and if the custom domain is pointed to us via nameservers we also need to add it to the DNS repo ah, I see [04:52:03] > ah, yeah, so now it's simplified since mw-admins can request puppet access @Doug yeah when I was mw-admin I worked on getting that done [04:52:17] before the ssl GitHub repository was part of puppet but we split it so that puppet-users can handle certs too [04:53:06] ah, cool [04:53:26] there's also a group called "cache-admins" that can get access to cache proxy servers but it's not used atm as it's not very necessary [04:53:39] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 9.82, 9.02, 6.20 [04:54:12] yeah... I could see that, as there's always usually one SRE online and you've got multiple cache proxy servers [04:54:28] if one fails, I assume traffic is just rerouted through the others? [04:54:29] and in theory if we had a lot of sysadmins there could be more groups for individual access to different areas (like phab-admins, mail-admins, redis-admins, etc.) [04:54:48] > if one fails, I assume traffic is just rerouted through the others? @Doug yeah normally it should depool itself and everything goes through the others [04:55:03] yeah, mail-admins wouldn't be necessary since only staff and board have e-mail, and I doubt Miraheze wants to get into e-mail hosting [04:55:09] but it might need a reboot (which cache-admins couldn't do anyway) [04:55:36] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 4.95, 7.67, 6.06 [04:55:46] > yeah, mail-admins wouldn't be necessary since only staff and board have e-mail, and I doubt Miraheze wants to get into e-mail hosting @Doug yeah the WMF have all sorts of different groups but for now we definitely don't need more [04:57:35] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 6.39, 8.04, 6.43 [04:59:32] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 3.73, 6.48, 6.04 [05:00:34] RECOVERY - oecumene.org - LetsEncrypt on sslhost is OK: OK - Certificate 'oecumene.org' will expire on Sun 01 Nov 2020 03:45:49 GMT +0000. [05:05:21] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 6.24, 8.70, 7.26 [05:07:19] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 3.33, 6.90, 6.77 [05:12:45] yeah...well I'm going to head to bed now, a bit earlier than usual but am tired 😛 [05:13:12] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 10.21, 8.51, 7.50 [05:21:03] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 5.92, 7.55, 7.61 [05:22:34] PROBLEM - meta.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.orain.org could not be found [05:23:00] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 8.55, 7.95, 7.74 [05:24:58] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 6.38, 7.92, 7.79 [05:26:56] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 6.32, 8.37, 8.03 [05:29:29] RECOVERY - meta.orain.org - reverse DNS on sslhost is OK: rDNS OK - meta.orain.org reverse DNS resolves to cp7.miraheze.org [05:34:46] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 5.30, 7.66, 8.00 [05:36:45] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 6.40, 8.35, 8.29 [05:50:45] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 4.57, 6.33, 7.86 [05:52:44] Reception123: what's on jobrunner1 causing load? [05:52:45] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 9.86, 7.69, 8.16 [05:54:43] RhinosF1: well I'm running an import but it shouldn't cause such a high load [05:55:04] Reception123: check htop/top [05:56:13] RhinosF1: seems to be the import yeah [05:57:33] !log nice -5 for the import logged above [05:57:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:59:48] !log re-run GDPR scripts [05:59:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:00:46] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 5.25, 7.08, 7.99 [06:06:56] @Zppix Thanks for your response on my talk, I was asleep. [06:10:33] Also miraheze.org doesn't work for me. [06:17:22] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [06:18:23] @MrJaroslavik: just seen [06:18:30] !sre Miraheze.org is down [06:18:49] RhinosF1: urgh, probably a syntax error [06:19:08] Reception123: can you look? Not even out of bed yet [06:19:18] yeah I'll take a look in a few [06:20:45] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 5.62, 6.13, 6.65 [06:25:23] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:38] RhinosF1: I doubt the period is supposed to be on the right side of the ''... https://github.com/miraheze/landing/commit/67570adb286ca186720070672c32a0335db7c169#diff-8fc97db3f912c2edd5adacb64fc8b3acR172 [06:48:39] [ Add Czech Translation (2) (#40) · miraheze/landing@67570ad · GitHub ] - github.com [06:48:52] but it seems to be done like that above too [06:50:39] Reception123: hmm [06:55:54] RhinosF1: oh it seems to be the lack of period that's problematic [06:56:00] I didn't know those worked with php [06:56:23] Ok [06:56:30] [02miraheze/landing] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJwik [06:56:32] [02miraheze/landing] 07Reception123 034712eb6 - attempt to fix syntax error [06:57:19] miraheze/landing/master/4712eb6 - Reception123 The build was fixed. https://travis-ci.org/miraheze/landing/builds/714372829 [06:57:40] ^ RhinosF1 yay! [07:00:26] RhinosF1: I don't see why travis wasn't checking in the first place though [07:02:40] It doesn't seem back yet [07:05:11] Reception123: hmm [07:05:19] RhinosF1: back now [07:05:35] But I still don't see what happened with Travis before the merge [07:05:40] Ok [07:05:54] We need to make sure it works [07:05:58] Travis seems to be playing up on the Miraheze org [07:10:19] * Reception123 wonders why [07:10:41] But remember I saw it happen on mw-config too [07:15:01] Reception123: it's only the Miraheze org [07:15:11] I guess we need to refresh it [07:15:49] It was only slow on MirahezeBots because it's has 4 test builds to run and then it has to wait for them before doing the deploy build [07:17:39] Ok then [07:17:45] Do you know how to refresh it? [07:20:00] Let me ask [07:22:01] This happens with the sopel devs often [07:40:14] ok [07:42:45] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.97, 5.82, 4.95 [07:44:46] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 9.10, 7.13, 5.55 [07:46:46] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 3.43, 5.74, 5.24 [07:54:30] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_34 [+1/-0/±1] 13https://git.io/JJwXW [07:54:31] [02miraheze/mediawiki] 07Reception123 03f7bc202 - Install MobileTagsPlugin ext. [07:54:33] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 10.43, 7.18, 5.57 [07:54:45] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 8.63, 7.26, 6.00 [07:56:33] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.44, 6.50, 5.53 [07:56:45] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 5.25, 6.61, 5.93 [08:02:31] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.78, 6.49, 5.72 [08:02:44] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 8.30, 6.83, 6.10 [08:03:05] Reception123: what's up with load? [08:03:13] I'm not doing anything [08:03:28] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [08:03:46] Reception123: https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?viewPanel=287&orgId=1&var-job=node&var-node=mw5.miraheze.org&var-port=9100 [08:03:46] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.71, 19.37, 16.96 [08:03:47] [ Grafana ] - grafana.miraheze.org [08:03:52] RhinosF1: yeah it must be puppet being annoying when adding a new submodule [08:03:52] What the **** [08:03:54] you know the drill [08:04:02] Reception123: no that disk usage [08:04:13] That's a huge spike [08:04:31] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 8.32, 7.46, 6.19 [08:04:35] hm that's very odd [08:04:42] Reception123: very [08:04:44] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 6.67, 6.98, 6.26 [08:04:53] Something doesn't add up with this load [08:05:09] it makes no sense [08:05:19] why is it that I can never do one thing without being bothered by icinga-miraheze [08:05:28] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [08:05:41] Reception123: can you scan top + access logs to make sure everything looks good [08:05:46] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.89, 18.23, 16.86 [08:05:52] RhinosF1: yeah was doing that [08:06:28] Good [08:06:29] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [08:06:31] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.17, 6.58, 6.02 [08:06:52] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.59, 6.05, 6.01 [08:07:05] ^ RhinosF1 that seems to be fine, was almost definitely puppet [08:07:09] but the space I don't get [08:07:23] Reception123: it's usage [08:07:35] Something was trying to use the disk a lot [08:11:58] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [08:12:33] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [08:12:42] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [08:13:41] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.74, 20.18, 16.42 [08:15:33] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [08:15:38] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:15:41] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.12, 19.09, 16.48 [08:16:28] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:19:42] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 19.74, 24.10, 19.57 [08:21:43] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 14.31, 20.41, 18.78 [08:23:44] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.56, 17.95, 18.11 [08:23:47] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.71, 5.74, 5.47 [08:24:31] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [08:25:47] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.49, 5.39, 5.39 [08:26:01] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:28:42] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:28:44] [02puppet] 07RhinosF1 opened pull request 03#1466: 403 clients with no UserAgent - 13https://git.io/JJwM2 [08:28:51] Reception123: ^ [08:30:40] [02puppet] 07Reception123 commented on pull request 03#1466: 403 clients with no UserAgent - 13https://git.io/JJwM6 [08:30:56] [02puppet] 07Reception123 closed pull request 03#1466: 403 clients with no UserAgent - 13https://git.io/JJwM2 [08:30:58] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJwMP [08:30:59] [02miraheze/puppet] 07RhinosF1 03076f218 - 429 clients with no UserAgent (#1466) [08:33:45] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.69, 5.97, 5.44 [08:33:59] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±4] 13https://git.io/JJwM5 [08:34:01] [02miraheze/mw-config] 07Reception123 03fce5110 - Add MobileTabsPlugin ext. T5999 [08:34:02] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [08:34:22] [02mw-config] 07Reception123 opened pull request 03#3202: Add MobileTabsPlugin ext. T5999 - 13https://git.io/JJwMd [08:35:15] [02puppet] 07RhinosF1 commented on pull request 03#1466: 403 clients with no UserAgent - 13https://git.io/JJwMp [08:35:44] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 3.58, 5.20, 5.24 [08:44:41] New accounts flood? [08:44:42] https://cdn.discordapp.com/attachments/435711390544560128/739765673365995580/IMG_20200803_104419.jpg [08:45:07] !log sudo service nginx reload on cp3 [08:45:43] @MrJaroslavik hmm, if we see another probably need to block for them to calm down [08:46:04] PROBLEM - mw4 MediaWiki Rendering on mw4 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 338 bytes in 0.011 second response time [08:46:05] PROBLEM - mw5 MediaWiki Rendering on mw5 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 338 bytes in 0.007 second response time [08:46:17] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [08:46:27] PROBLEM - test2 MediaWiki Rendering on test2 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 338 bytes in 0.005 second response time [08:46:43] Reception123: revert ^ [08:46:46] PROBLEM - mw6 MediaWiki Rendering on mw6 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 338 bytes in 0.105 second response time [08:46:53] PROBLEM - cp7 HTTPS on cp7 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 338 bytes in 0.007 second response time [08:47:08] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03revert-1466-patch-15 [+0/-0/±1] 13https://git.io/JJwyL [08:47:10] [02miraheze/puppet] 07Reception123 030ea76b2 - Revert "429 clients with no UserAgent (#1466)" This reverts commit 076f218106bed668f827724be4be0de766d3b96c. [08:47:11] [02puppet] 07Reception123 created branch 03revert-1466-patch-15 - 13https://git.io/vbiAS [08:47:13] [02puppet] 07Reception123 opened pull request 03#1467: Revert "403 clients with no UserAgent" - 13https://git.io/JJwyq [08:47:18] [02puppet] 07Reception123 closed pull request 03#1467: Revert "403 clients with no UserAgent" - 13https://git.io/JJwyq [08:47:19] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJwym [08:47:21] [02miraheze/puppet] 07Reception123 035cdca2b - Revert "429 clients with no UserAgent (#1466)" (#1467) This reverts commit 076f218106bed668f827724be4be0de766d3b96c. [08:47:24] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 338 bytes in 0.008 second response time [08:47:29] PROBLEM - mw7 MediaWiki Rendering on mw7 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 338 bytes in 0.004 second response time [08:47:40] Reception123: force puppet everywhere please [08:47:44] yes [08:47:47] PROBLEM - cp6 HTTPS on cp6 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 338 bytes in 0.006 second response time [08:47:56] PROBLEM - cp9 HTTPS on cp9 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 338 bytes in 0.332 second response time [08:48:47] PROBLEM - cp3 HTTPS on cp3 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 338 bytes in 1.006 second response time [08:51:38] Reception123: has puppet ran and nginx been reloaded? If so, can we force icinga to recheck? [08:51:54] yes will try to force [08:52:58] still running on cp3 since it's slow [08:54:04] RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 18546 bytes in 0.019 second response time [08:54:05] RECOVERY - mw5 MediaWiki Rendering on mw5 is OK: HTTP OK: HTTP/1.1 200 OK - 18546 bytes in 0.007 second response time [08:54:17] RECOVERY - jobrunner1 MediaWiki Rendering on jobrunner1 is OK: HTTP OK: HTTP/1.1 200 OK - 18545 bytes in 0.009 second response time [08:54:19] there it goes [08:54:21] RECOVERY - cp6 HTTPS on cp6 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1936 bytes in 0.004 second response time [08:54:22] RECOVERY - mw7 MediaWiki Rendering on mw7 is OK: HTTP OK: HTTP/1.1 200 OK - 18546 bytes in 0.006 second response time [08:54:23] RECOVERY - mw6 MediaWiki Rendering on mw6 is OK: HTTP OK: HTTP/1.1 200 OK - 18546 bytes in 0.006 second response time [08:54:24] RECOVERY - cp7 HTTPS on cp7 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1936 bytes in 0.021 second response time [08:54:27] RECOVERY - test2 MediaWiki Rendering on test2 is OK: HTTP OK: HTTP/1.1 200 OK - 18546 bytes in 0.007 second response time [09:01:25] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 4526 files, 13.14MB, 2020-08-03 09:00:00 (1.4 minutes ago) [09:05:24] RECOVERY - cp9 HTTPS on cp9 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1936 bytes in 0.332 second response time [09:09:23] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1935 bytes in 1.024 second response time [09:21:46] [02puppet] 07doraemonfan2020 opened pull request 03#1468: Add new headers - 13https://git.io/JJw9n [10:26:26] PROBLEM - wiki.ciptamedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ciptamedia.org' expires in 15 day(s) (Wed 19 Aug 2020 10:21:38 GMT +0000). [10:33:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJwdp [10:33:43] [02miraheze/ssl] 07MirahezeSSLBot 03674673b - Bot: Update SSL cert for wiki.ciptamedia.org [10:34:37] PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'adadevelopersacademy.wiki' expires in 15 day(s) (Wed 19 Aug 2020 10:32:02 GMT +0000). [10:41:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJwFn [10:41:23] [02miraheze/ssl] 07MirahezeSSLBot 033034ddd - Bot: Update SSL cert for adadevelopersacademy.wiki [10:43:36] PROBLEM - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.gesamtschule-nordkirchen.de' expires in 15 day(s) (Wed 19 Aug 2020 10:34:54 GMT +0000). [10:43:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJwFB [10:43:55] [02miraheze/ssl] 07MirahezeSSLBot 032e89153 - Bot: Update SSL cert for wiki.gesamtschule-nordkirchen.de [10:46:37] RECOVERY - wiki.ciptamedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ciptamedia.org' will expire on Sun 01 Nov 2020 09:33:35 GMT +0000. [10:55:12] RECOVERY - adadevelopersacademy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'adadevelopersacademy.wiki' will expire on Sun 01 Nov 2020 09:41:15 GMT +0000. [10:57:06] RECOVERY - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Sun 01 Nov 2020 09:43:47 GMT +0000. [10:58:46] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/JJwb3 [10:58:48] [02miraheze/mw-config] 07Reception123 0344770ab - add wikiapiary footer for circleyversewiki per req requested via Discord [10:58:49] [02mw-config] 07Reception123 created branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [10:58:51] [02mw-config] 07Reception123 opened pull request 03#3203: add wikiapiary footer for circleyversewiki per req - 13https://git.io/JJwbs [10:59:07] ^ RhinosF1 yup annoying travis is gone [10:59:13] now that it's gone I miss it [10:59:21] Heh [11:02:07] [02mw-config] 07Reception123 synchronize pull request 03#3203: add wikiapiary footer for circleyversewiki per req - 13https://git.io/JJwbs [11:02:08] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/JJwbC [11:02:10] [02miraheze/mw-config] 07Reception123 03b9065ce - fix link [11:02:28] [02mw-config] 07Reception123 closed pull request 03#3203: add wikiapiary footer for circleyversewiki per req - 13https://git.io/JJwbs [11:02:29] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJwbW [11:02:31] [02miraheze/mw-config] 07Reception123 037f4ad08 - add wikiapiary footer for circleyversewiki per req (#3203) * add wikiapiary footer for circleyversewiki per req requested via Discord * fix link [11:02:32] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [11:02:34] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-2 [11:02:44] RhinosF1: a repo without travis is not very good, did you get a chance to ask the sopel folks? [11:03:05] * Reception123 remembers the dark times before we had travis [11:03:49] miraheze/mw-config/Reception123-patch-2/b9065ce - Reception123 The build has errored. https://travis-ci.org/miraheze/mw-config/builds/714429902 [11:04:02] uh that's not what my linter said... [11:04:35] RhinosF1: now travis is lying to me [11:27:17] Reception123: ha! [11:28:01] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JJwNF [11:28:02] [02miraheze/puppet] 07Reception123 03d1b7fe7 - whitelist wikiapiary.com ran by trusted MediaWiki contributor Kghbln, needs one more SRE approval [11:28:04] [02puppet] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbiAS [11:28:08] [02puppet] 07Reception123 opened pull request 03#1469: whitelist wikiapiary.com - 13https://git.io/JJwNb [12:14:40] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.83, 5.75, 4.53 [12:16:40] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 3.83, 4.85, 4.34 [13:53:55] [02puppet] 07paladox closed pull request 03#1468: Add new headers - 13https://git.io/JJw9n [13:55:23] paladox: what would that have even done? [13:57:43] I would guess break it, this is the same user that keeps creating pulls for varnish and then we close them. So I think it’s JoeA. [13:59:48] paladox: oh [14:00:06] paladox: did you see the tech@ email [14:00:55] Reception123: how did we not spot who that is [14:01:40] What email? [14:04:03] paladox: high load + disk earlier and a lot of traffic from what looks to be bad bot/crawler with UA of - attempt to block it failed and had to be reverted [14:04:13] Any way to handle it? [14:06:05] You would just block it’s UA [14:06:11] that's the thing [14:06:13] see -staff [14:09:54] paladox: see my commit to puppet, we tried, it blocked icinga and many users [14:20:05] var ~ "-" looks to be a regex match, which matches any string including the "-" character. To match only that character using ~, the expression must be "^-$" [14:22:44] ^ [14:22:45] RhinosF1: Reception123 try ^ [14:22:47] You can test using nginx [14:22:48] On test2 [14:23:10] Reception123: ^ [14:23:18] paladox: how would I test exactly? [14:57:11] Reception123 you would just edit the mediawiki file under nginx on test2 [14:57:13] and restart it [14:58:03] paladox: OK, and then? [14:58:44] wait a bit, as test2 is in varnish [14:58:51] so if it fails you will get the same issue [15:01:23]