[00:01:32] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3962 MB (16% inode=93%); [00:12:38] @Void Regarding PR #3189, since you're modifying the same area as my #3188, should I do a new PR #3190 to fix my proposed sandbox[0-15] to follow your syntax or can I amend my existing PR? [00:16:23] Why don't we merge it together [00:17:27] I'll suggest changes to your PR, and then close mine, so that they can all be done at once [00:20:11] [02mw-config] 07The-Voidwalker reviewed pull request 03#3188 commit - 13https://git.io/JJ4SV [00:21:10] [02mw-config] 07The-Voidwalker commented on pull request 03#3189: Fix invalid regex - 13https://git.io/JJ4Sw [00:21:12] [02mw-config] 07The-Voidwalker closed pull request 03#3189: Fix invalid regex - 13https://git.io/JJ4yb [00:21:32] @Doug ^ [00:22:01] @Void That sounds good, let me look. [00:22:25] Yup, I'm thinking you might just be able to click a button and merge in my suggested changes [00:22:45] Though, I've not yet played around with suggested changes very much [00:23:24] Yep, looks good. Let me look. I don't see a merge...just a "close pull request," but that might be because I left the option to allow other reviewers to modify my PR? [00:23:31] so it might already be merged? [00:24:11] Check inside my review in the Suggested Changes area [00:24:30] [02mw-config] 07dmehus reviewed pull request 03#3188 commit - 13https://git.io/JJ4SK [00:25:15] Since it (still doesn't) seem merged into the actual PR on my end (I can't edit your pull request) [00:28:12] Yeah, I can't see where the merge is. When I go into my commit, it doesn't seem to have your changes, so I'm not sure if maybe paladox or RhinosF1 or someone has to merge the suggested changes? There's probably a merge button, but am not seeing it. [00:29:40] It's not in "commit suggestion" and I don't want to "close pull request" as I'm concerned that will cancel the PR. [00:30:05] @paladox, what do I need to do to merge in @Void's suggested changes from #3189 into #3188? [00:30:54] You just have to edit the branch it created under your fork [00:31:02] Oh. [00:31:10] Gotcha. Thanks. [00:31:21] Or, see https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/incorporating-feedback-in-your-pull-request#applying-suggested-changes [00:31:22] [ Incorporating feedback in your pull request - GitHub Docs ] - docs.github.com [00:31:55] @Doug you should be able to just click commit suggestion and add it in [00:32:17] I don't see it as a separate branch under https://github.com/dmehus/mw-config/branches. [00:32:17] [ Branches · dmehus/mw-config · GitHub ] - github.com [00:32:36] that's because it's not, it exists only on the PR discussion tab [00:32:58] see the link I included [00:33:03] [02mw-config] 07dmehus synchronize pull request 03#3188: Update LocalSettings.php - 13https://git.io/JJ4yM [00:33:14] Oh, do, I think that worked. [00:33:16] lol [00:33:46] I just had to click "commit suggested changes." Should be good now, I'm hoping? [00:33:51] :D [00:33:53] looks good [00:36:13] thanks, yeah that should be better. So basically, with your change to the number syntax, it'll check for those subdomains with the numbers 1-15 anywhere following the subdomain, versus just blocking subdomains 1-15? [00:36:28] it's like an OR, more or less [00:41:33] No, the change matches any two digits following the subdomain, the previous version only matches the subdomain followed by 0, 1, or 5 [00:53:26] > No, my change matches any two digits following the subdomain, the previous version only matches the subdomain followed by 0, 1, or 5 @Void Ah, okay, so the 1 and 2 in that context don't strictly mean the numerals 1 and 2, it's more less saying that any digit in the first two positions will be prohibited, i.e., mw01, mw10, m15, m99, etc.? [01:08:13] yeah, my change matches mw1, mw15, mw99, mw2, while the old one actually only matches the first of those [01:25:49] PROBLEM - cp9 APT on cp9 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (1 critical updates). [02:19:56] Hello juboxi! If you have any questions, feel free to ask and someone should answer soon. [02:32:41] Hi e [05:12:02] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+1/-0/±0] 13https://git.io/JJ4pG [05:12:03] [02miraheze/ssl] 07Reception123 03a620c81 - add wiki.climatechange.ai cert [05:12:05] [02ssl] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vxP9L [05:12:06] [02ssl] 07Reception123 opened pull request 03#344: add wiki.climatechange.ai cert - 13https://git.io/JJ4pZ [05:12:46] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JJ4pW [05:12:48] [02miraheze/ssl] 07Reception123 039504eb4 - Update certs.yaml [05:12:49] [02ssl] 07Reception123 synchronize pull request 03#344: add wiki.climatechange.ai cert - 13https://git.io/JJ4pZ [05:13:26] [02ssl] 07Reception123 closed pull request 03#344: add wiki.climatechange.ai cert - 13https://git.io/JJ4pZ [05:13:28] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JJ4p8 [05:13:29] [02miraheze/ssl] 07Reception123 038ba8638 - add wiki.climatechange.ai cert (#344) * add wiki.climatechange.ai cert * Update certs.yaml [05:13:30] [02ssl] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vxP9L [05:13:32] [02miraheze/ssl] 07Reception123 deleted branch 03Reception123-patch-1 [06:10:19] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ4hx [06:10:20] [02miraheze/services] 07MirahezeSSLBot 03fc3aee8 - BOT: Updating services config for wikis [06:12:26] RECOVERY - cp9 APT on cp9 is OK: APT OK: 36 packages available for upgrade (0 critical updates). [06:25:34] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [06:30:28] !log sudo apt-get upgrade on cp9 [06:30:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:32:30] !log sudo apt-get upgrade on rdb1 [06:32:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:33:33] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:33:39] RECOVERY - rdb1 APT on rdb1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:11:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.67, 21.21, 14.44 [07:11:34] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 5.63, 14.14, 7.49 [07:13:20] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.40, 18.59, 14.31 [07:15:31] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.49, 7.86, 6.37 [07:17:31] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 0.74, 5.40, 5.64 [09:20:34] PROBLEM - pubwiki.lab.co.pl - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,51.77.107.210,51.89.160.142' [09:20:38] PROBLEM - wiki.velaan.org - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,51.77.107.210,51.89.160.142' [09:20:38] PROBLEM - es.publictestwiki.com - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,51.77.107.210,51.89.160.142' [09:20:41] PROBLEM - te.gyaanipedia.co.in - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2607:5300:205:200::2ac4,51.222.27.129' [09:20:54] PROBLEM - index.pdcommunity.ir - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2607:5300:205:200::2ac4,51.222.27.129' [09:20:55] PROBLEM - sims.miraheze.org - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,51.77.107.210,51.89.160.142' [09:21:00] PROBLEM - www.tradwiki.org - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.222.27.129' [09:21:01] paladox, Reception123: we got a cp down again [09:21:03] PROBLEM - miraheze.ga - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,51.77.107.210,51.89.160.142' [09:21:10] PROBLEM - wiki.animalrebellion.cz - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,51.222.27.129' [09:21:18] PROBLEM - wiki.bonziworldrevived.tk - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.222.27.129' [09:21:20] PROBLEM - wiki.coderdojosaintpaul.org - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.222.27.129' [09:21:21] PROBLEM - madzebrascience.wiki - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,51.222.27.129' [09:21:21] PROBLEM - wiki.ciptamedia.org - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.222.27.129' [09:21:34] PROBLEM - mai.gyaanipedia.co.in - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2607:5300:205:200::2ac4,51.77.107.210,51.89.160.142' [09:21:36] PROBLEM - miraheze.gq - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.222.27.129' [09:21:37] PROBLEM - wiki.8b8t.com - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.222.27.129' [09:21:45] PROBLEM - bh.gyaanipedia.co.in - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2607:5300:205:200::2ac4,51.222.27.129' [09:21:46] PROBLEM - rivworld.wiki - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.222.27.129' [09:21:48] PROBLEM - bebaskanpengetahuan.id - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.222.27.129' [09:21:50] PROBLEM - wiki.teamrelectric.ca - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.222.27.129' [09:21:50] PROBLEM - wiki.bullshit.systems - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.222.27.129' [09:21:53] PROBLEM - wiki.xysspon.com - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2607:5300:205:200::2ac4,51.77.107.210,51.89.160.142' [09:21:54] PROBLEM - baharna.org - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.222.27.129' [09:21:58] PROBLEM - permanentfuturelab.wiki - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.222.27.129' [09:23:43] RhinosF1: oh yikes [09:24:17] Reception123: If that's cp7 can we just depool in dns? [09:24:27] This is going off like every 12 hours [09:24:34] https://phabricator.miraheze.org/T5967 [09:24:35] [ ⚓ T5967 Cache Proxy (cp7?) health check flapping constantly ] - phabricator.miraheze.org [09:25:22] RhinosF1: well I only see it complaining about the certs, cp7 says it's OK [09:25:52] Reception123: that alert is a cryptic new way of saying a cache proxy depooled [09:26:20] well if it's already depooled I see no use in depooling it via DNS [09:26:28] though when paladox is around he should look into why it's acting so weird [09:26:51] Reception123: it's going every ~12 hours [09:26:59] There's something very broke [09:27:13] well since you mention paladox did you tell him yesterday? [09:27:17] RECOVERY - pubwiki.lab.co.pl - DNS on sslhost is OK: DNS OK: 0.037 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:27:24] RECOVERY - wiki.velaan.org - DNS on sslhost is OK: DNS OK: 0.038 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:27:32] RECOVERY - te.gyaanipedia.co.in - DNS on sslhost is OK: DNS OK: 0.040 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:27:35] RECOVERY - es.publictestwiki.com - DNS on sslhost is OK: DNS OK: 0.050 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:27:37] Reception123: yes see -staff [09:27:44] RECOVERY - sims.miraheze.org - DNS on sslhost is OK: DNS OK: 0.036 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:27:51] RECOVERY - wiki.animalrebellion.cz - DNS on sslhost is OK: DNS OK: 0.052 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:27:52] RECOVERY - index.pdcommunity.ir - DNS on sslhost is OK: DNS OK: 0.139 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:27:54] RECOVERY - www.tradwiki.org - DNS on sslhost is OK: DNS OK: 0.055 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:27:59] RECOVERY - miraheze.ga - DNS on sslhost is OK: DNS OK: 0.037 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:28:09] RECOVERY - wiki.bonziworldrevived.tk - DNS on sslhost is OK: DNS OK: 0.039 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:28:09] RECOVERY - wiki.coderdojosaintpaul.org - DNS on sslhost is OK: DNS OK: 0.172 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:28:11] RECOVERY - wiki.ciptamedia.org - DNS on sslhost is OK: DNS OK: 0.032 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:28:17] RECOVERY - miraheze.gq - DNS on sslhost is OK: DNS OK: 0.037 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:28:18] RECOVERY - madzebrascience.wiki - DNS on sslhost is OK: DNS OK: 0.119 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:28:19] RECOVERY - mai.gyaanipedia.co.in - DNS on sslhost is OK: DNS OK: 0.061 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:28:20] RECOVERY - wiki.8b8t.com - DNS on sslhost is OK: DNS OK: 0.044 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:28:27] RECOVERY - bebaskanpengetahuan.id - DNS on sslhost is OK: DNS OK: 0.039 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:28:27] There's something bloody wrong with that server [09:28:30] RECOVERY - wiki.bullshit.systems - DNS on sslhost is OK: DNS OK: 0.034 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:28:34] RECOVERY - wiki.xysspon.com - DNS on sslhost is OK: DNS OK: 0.032 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:28:35] RECOVERY - bh.gyaanipedia.co.in - DNS on sslhost is OK: DNS OK: 0.054 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:28:35] RECOVERY - baharna.org - DNS on sslhost is OK: DNS OK: 0.098 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:28:40] RECOVERY - rivworld.wiki - DNS on sslhost is OK: DNS OK: 0.043 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:28:47] RECOVERY - wiki.teamrelectric.ca - DNS on sslhost is OK: DNS OK: 0.031 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [09:28:57] RECOVERY - permanentfuturelab.wiki - DNS on sslhost is OK: DNS OK: 0.051 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [10:33:36] !log reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php initSiteStats.php --update --wiki erislywiki [10:33:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:35:41] !log added --active to the above command [10:35:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:19] Hello Diamochang! If you have any questions, feel free to ask and someone should answer soon. [13:15:52] Hello twojukeboksi! If you have any questions, feel free to ask and someone should answer soon. [13:30:19] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJBYz [13:30:21] [02miraheze/services] 07MirahezeSSLBot 031a0bda8 - BOT: Updating services config for wikis [13:44:57] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03Drop-php-72 [+0/-0/±1] 13https://git.io/JJBOG [13:44:59] [02miraheze/mw-config] 07RhinosF1 030732d67 - Drop PHP 7.2 Check See https://phabricator.wikimedia.org/T257879 If that gets confirmed, I'll go and do the same on all other repos. [13:45:00] [ ⚓ T257879 Drop PHP 7.2 support in MediaWiki 1.35 ] - phabricator.wikimedia.org [13:45:00] [02mw-config] 07RhinosF1 created branch 03Drop-php-72 - 13https://git.io/vbvb3 [13:45:19] [02mw-config] 07RhinosF1 opened pull request 03#3190: Drop PHP 7.2 Check - 13https://git.io/JJBOl [13:45:44] [02mw-config] 07RhinosF1 assigned pull request 03#3190: Drop PHP 7.2 Check - 13https://git.io/JJBOl [13:46:39] miraheze/mw-config/Drop-php-72/0732d67 - RhinosF1 The build passed. https://travis-ci.org/miraheze/mw-config/builds/711722277 [13:47:08] paladox: ^ for whenever we switch to 1.35, will do the same on every other repo then. [13:47:09] PROBLEM - fc.songcontests.eu - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.222.27.129' [13:47:18] PROBLEM - test1.miraheze.org - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.222.27.129' [13:47:32] PROBLEM - index.pdcommunity.ir - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2607:5300:205:200::2ac4,51.222.27.129' [13:47:33] paladox: now might also be the perfect time to live test and check logs :) [13:48:01] it's being caused by one of the mw* [13:48:31] paladox: mw5, https://grafana.miraheze.org/d/iWQm-pOZz/nginx-appservers?viewPanel=12&orgId=1&refresh=5s&from=now-5m&to=now&var-instance=mw5.miraheze.org:9113 [13:48:32] [ Grafana ] - grafana.miraheze.org [13:48:47] Can you check it's logs and test everything is sane while it's down [13:48:51] https://grafana.miraheze.org/d/iWQm-pOZz/nginx-appservers?viewPanel=15&orgId=1&refresh=5s&from=now-5m&to=now&var-instance=mw5.miraheze.org:9113 [13:48:52] [ Grafana ] - grafana.miraheze.org [13:48:56] is what you would look like [13:49:10] *at [13:49:52] paladox: yeah, I can see on the now-2m that it's dropped [13:49:56] the access logs are irrelevent right now as we know why. DNS hit the timeout so depooled [13:50:10] Maybe nginx logs for errors? [13:50:23] already checked and nothing of interest that would explain [13:51:15] Hmm [13:51:17] Jul 25 07:05:39 mw5 nginx-prometheus-exporter[3343]: 2020/07/25 07:05:39 Error getting stats: failed to get http://127.0.0.1:8090/server-status: Get "http://127.0.0.1:8090/server-status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [13:52:14] Okay [13:53:01] https://github.com/prometheus/prometheus/issues/1438 [13:53:02] [ context deadline exceeded · Issue #1438 · prometheus/prometheus · GitHub ] - github.com [13:54:06] RECOVERY - fc.songcontests.eu - DNS on sslhost is OK: DNS OK: 0.049 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [13:54:09] RECOVERY - test1.miraheze.org - DNS on sslhost is OK: DNS OK: 0.049 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [13:54:29] RECOVERY - index.pdcommunity.ir - DNS on sslhost is OK: DNS OK: 0.036 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [13:56:07] RhinosF1 it's not a prometheus issue... [13:58:00] paladox: searching the error you gave only came up with it [13:58:18] (Client.Timeout exceeded while awaiting headers) [13:58:31] it timedout since nginx took to long to respond [14:02:25] !log depool mw4 [14:02:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:02:55] !log restart nginx & php7.3-fpm on mw4 [14:02:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:02:59] !log repool mw4 [14:03:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:03:04] !log depool and repool mw5 [14:03:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:54:02] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJBsN [15:05:14] [02miraheze/services] 07MirahezeSSLBot 032ff7d27 - BOT: Updating services config for wikis [16:39:46] !log reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php runJobs.php --wiki yablestudiowiki [16:39:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:49:01] For what technical reason isn't used Meta for GUPs? [16:50:19] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJBC1 [16:50:21] [02miraheze/services] 07MirahezeSSLBot 0357e67a8 - BOT: Updating services config for wikis [16:52:44] > For what technical reason isn't used Meta for GUPs? @MrJaroslavik I'm not sure. I wondered that originally, but I do kind of like it on Loginwiki. I'd like to see the global Interwiki table moved to Loginwiki, so that Meta can have a local interwiki table editable by any Meta sysop and, after a successful community discussion, an opting back in of Meta to interwiki-admin global group. [16:58:09] Hey! I noticed the report on the stewards noticeboard for the user Suicide Is Very Funny 1488. I would like to request that this name be oversighted because to me it is very offensive, because of personal matters. If at all possible, please let me know. Thanks! [17:00:56] It is not just for me, some people will likely be offended as well. [17:09:49] will do once CU'd [17:11:06] @TFFfan Agree; that username is worth oversighting. 🙂 [17:33:25] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [17:50:03] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2646 MB (10% inode=93%); [19:55:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJRtL [19:55:13] [02miraheze/services] 07MirahezeSSLBot 0324d91fa - BOT: Updating services config for wikis [20:20:22] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJR3Q [20:20:24] [02miraheze/services] 07MirahezeSSLBot 033b358e8 - BOT: Updating services config for wikis [20:40:34] [02mw-config] 07paladox edited pull request 03#3188: Fix regex for wgCreateWikiBlacklistedSubdomains - 13https://git.io/JJ4yM [20:41:38] [02mw-config] 07paladox closed pull request 03#3188: Fix regex for wgCreateWikiBlacklistedSubdomains - 13https://git.io/JJ4yM [20:41:40] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJRs7 [20:41:41] [02miraheze/mw-config] 07dmehus 03398a599 - Update LocalSettings.php (#3188) * Update LocalSettings.php Adding "sandbox" to restricted subdomains list because we've had a couple wiki request declines in the past month (https://meta.miraheze.org/wiki/Special:RequestWikiQueue/12598 and https://meta.miraheze.org/wiki/Special:RequestWikiQueue/13380) and I don't want to see it inadvertently get created. Also, [20:41:42] adding "noc" as a restricted subdomain as I didn't see it in this list, and it's highly conceivable Miraheze may want to use it for technical coordination purposes, either as a wiki or some other website. Finally, though I don't want to create it now, I was talking with Reception123 on Discord about ways we can (a) increase Meta participation and (b) community volunteerism, and one of the ways in which I think this [20:41:42] would be helpful would be through an Outreach wiki, so am requesting we restrict this subdomain for now. * Update LocalSettings.php Co-authored-by: The-Voidwalker Co-authored-by: The-Voidwalker [20:41:43] [ Wiki requests queue - Miraheze Meta ] - meta.miraheze.org [20:41:44] [ Wiki requests queue - Miraheze Meta ] - meta.miraheze.org [20:41:51] [02mw-config] 07paladox closed pull request 03#3190: Drop PHP 7.2 Check - 13https://git.io/JJBOl [20:41:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJRsF [20:41:54] [02miraheze/mw-config] 07RhinosF1 033555f12 - Drop PHP 7.2 Check (#3190) See https://phabricator.wikimedia.org/T257879 If that gets confirmed, I'll go and do the same on all other repos. [20:41:55] [ ⚓ T257879 Drop PHP 7.2 support in MediaWiki 1.35 ] - phabricator.wikimedia.org [20:42:33] [02mw-config] 07paladox closed pull request 03#3147: Add author protection - 13https://git.io/JJtO8 [20:42:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJRsx [20:42:36] [02miraheze/mw-config] 07RhinosF1 038b1510a - Update LocalExtensions.php (#3147) [20:42:42] [02miraheze/mw-config] 07paladox deleted branch 03T5878 [20:42:43] [02mw-config] 07paladox deleted branch 03T5878 - 13https://git.io/vbvb3 [20:44:07] miraheze/mw-config/master/8b1510a - RhinosF1 The build passed. https://travis-ci.org/miraheze/mw-config/builds/711805081 [21:09:37] [02MirahezeMagic] 07paladox commented on pull request 03#135: restore previous changes and fixes - 13https://git.io/JJRG7 [21:11:01] [02MirahezeMagic] 07paladox deleted a comment on pull request 03#135: restore previous changes and fixes - 13https://git.io/JJRG7 [21:13:11] [02MirahezeMagic] 07paladox commented on pull request 03#135: restore previous changes and fixes - 13https://git.io/JJRGp [21:21:18] [02MirahezeMagic] 07paladox synchronize pull request 03#135: restore previous changes and fixes - 13https://git.io/JJ3lk [21:21:49] [02MirahezeMagic] 07paladox commented on pull request 03#135: restore previous changes and fixes - 13https://git.io/JJRZL [21:22:11] [02MirahezeMagic] 07paladox closed pull request 03#135: restore previous changes and fixes - 13https://git.io/JJ3lk [21:22:13] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJRZq [21:22:14] [02miraheze/MirahezeMagic] 07The-Voidwalker 038b80d4e - restore previous changes and fixes (#135) * restore previous changes, and attempt a few fixes * this is needed * this should now work up until saving the file * Update generateSitemapIndex.py Co-authored-by: paladox [21:22:46] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJRZm [21:22:47] [02miraheze/mediawiki] 07paladox 0375eb886 - Update MirahezeMagic [21:26:02] [02puppet] 07paladox closed pull request 03#1444: jobrunner: Don't nice scripts - 13https://git.io/JJstt [21:26:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJRZW [21:26:05] [02miraheze/puppet] 07paladox 03ee963f3 - jobrunner: Don't nice scripts (#1444) Not needed anymore. [21:26:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [21:26:08] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://git.io/vbiAS [21:27:04] [02puppet] 07paladox closed pull request 03#1436: Add NicoVideo to CSP per T5535 - 13https://git.io/JJqPG [21:27:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJRZE [21:27:07] [02miraheze/puppet] 07Elaeagnifolia 03dcbaf15 - Add NicoVideo to CSP per T5535 (#1436) [21:27:53] [02CreateWiki] 07paladox closed pull request 03#156: Check if deleted.json exists otherwise create an array - 13https://git.io/JJ4rH [21:27:55] [02CreateWiki] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vpJTL [21:27:57] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-3 [21:36:13] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.65, 22.12, 16.03 [21:36:25] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 9.34, 6.39, 3.94 [21:38:13] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.93, 22.06, 16.73 [21:39:33] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:40:01] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:40:21] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 38.39, 25.38, 18.50 [21:40:55] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:40:57] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:40:59] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:41:59] !log drop a bunch of never indexed sitemaps from search console, add the new master index. Will drop all in use ones excluding master once google decide to index the master sitemap. [21:42:16] paladox: where's MirahezeLogbot? [21:42:16] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.06, 23.93, 18.78 [21:42:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2607:5300:205:200::2ac4/cpweb [21:42:28] mon1? [21:42:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:42:57] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [21:43:11] paladox: I meant it wasn't talking [21:43:30] That took a bit to log [21:43:33] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:44:13] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.97, 24.88, 19.69 [21:44:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:46:13] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.72, 23.89, 19.98 [21:48:15] PROBLEM - wiki.minkyu.kim - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:20] PROBLEM - wiki.xysspon.com - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:22] PROBLEM - kkutu.wiki - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:22] PROBLEM - wiki.ostans.com - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:24] PROBLEM - minetestserver-cc.ddns.net - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:24] PROBLEM - indiancannabis.wiki - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:25] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 4.60, 7.10, 6.17 [21:48:25] PROBLEM - wiki.erisly.com - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:26] PROBLEM - mh142.com - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:27] PROBLEM - programming.red - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:31] PROBLEM - franchise.franchising.org.ua - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:31] PROBLEM - www.programming.red - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:31] PROBLEM - www.netazar.org - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:31] PROBLEM - piwik.miraheze.org - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:31] PROBLEM - infectowiki.com - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:35] PROBLEM - la.gyaanipedia.co.in - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:38] PROBLEM - runzeppelin.ru - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:40] PROBLEM - wiki.galactee.org - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:42] PROBLEM - wiki.make717.org - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:48:46] PROBLEM - www.openonderwijs.org - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:49:24] paladox: what's dns on now? [21:49:34] hmm? [21:49:40] paladox: the alerts [21:50:00] your not making sense with what you say, please rephase what you want me to answer. [21:50:25] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 4.03, 6.00, 5.88 [21:50:27] the dns depooled a cp* [21:50:34] '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' [21:52:00] it depooled 51.77.107.210 [21:52:06] cp6 [21:52:13] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.55, 18.91, 19.11 [21:52:27] but that'll be due to it hitting a timeout on the healthcheck (which is done mw* side) [21:52:59] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [21:53:48] i wonder what's causing nginx to take a while to respond [21:54:06] paladox: these alerts are going off way too often? something is seriously wrong if cps are depooling this often. [21:54:14] yes [21:54:20] it's already established that it's bad [21:54:42] paladox: can we please ensure you find someone to work on it as soon as possible [21:55:02] RECOVERY - kkutu.wiki - DNS on sslhost is OK: DNS OK: 0.213 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:03] RECOVERY - wiki.ostans.com - DNS on sslhost is OK: DNS OK: 0.034 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:03] well John/SPF seem to be out [21:55:06] RECOVERY - indiancannabis.wiki - DNS on sslhost is OK: DNS OK: 0.011 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:10] RECOVERY - wiki.erisly.com - DNS on sslhost is OK: DNS OK: 0.010 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:11] RECOVERY - mh142.com - DNS on sslhost is OK: DNS OK: 0.012 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:11] RECOVERY - wiki.minkyu.kim - DNS on sslhost is OK: DNS OK: 0.045 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:12] RECOVERY - programming.red - DNS on sslhost is OK: DNS OK: 0.013 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:18] RECOVERY - wiki.xysspon.com - DNS on sslhost is OK: DNS OK: 0.019 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:19] RECOVERY - franchise.franchising.org.ua - DNS on sslhost is OK: DNS OK: 0.013 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:19] RECOVERY - www.programming.red - DNS on sslhost is OK: DNS OK: 0.080 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:19] RECOVERY - www.netazar.org - DNS on sslhost is OK: DNS OK: 0.076 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:20] RECOVERY - piwik.miraheze.org - DNS on sslhost is OK: DNS OK: 0.070 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:20] RECOVERY - infectowiki.com - DNS on sslhost is OK: DNS OK: 0.058 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:21] RECOVERY - wiki.make717.org - DNS on sslhost is OK: DNS OK: 0.010 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:22] RECOVERY - minetestserver-cc.ddns.net - DNS on sslhost is OK: DNS OK: 0.010 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:28] RECOVERY - www.openonderwijs.org - DNS on sslhost is OK: DNS OK: 0.010 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:33] RECOVERY - la.gyaanipedia.co.in - DNS on sslhost is OK: DNS OK: 0.013 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:35] RECOVERY - runzeppelin.ru - DNS on sslhost is OK: DNS OK: 0.009 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:41] RECOVERY - wiki.galactee.org - DNS on sslhost is OK: DNS OK: 0.010 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [21:55:52] paladox: can you comment on the task I linked you and then email them both [21:56:01] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:56:54] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:58:20] done [22:00:58] paladox: hopefully someone can look soon [22:06:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJRn6 [22:06:47] [02miraheze/puppet] 07paladox 031153600 - extensions setup: Do not install dev packages in composer [22:07:23] yup [22:14:13] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJRnH [22:14:14] [02miraheze/MirahezeMagic] 07paladox 03afcbf07 - Use w+ not r+ [22:14:40] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJRn5 [22:14:42] [02miraheze/mediawiki] 07paladox 03bf69a9a - Update MirahezeMagic [22:17:33] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[antispoof_composer] [22:19:33] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:29:53] !log depool and repool mw4 [22:29:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:30:31] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:32:30] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [22:34:37] !log depool and repool mw5 [22:34:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:37:46] !log depool and repool mw6 [22:37:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:39:43] !log depool and repool mw7 [22:39:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:20:18] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJRCj [23:20:20] [02miraheze/services] 07MirahezeSSLBot 0336af86f - BOT: Updating services config for wikis