[00:57:13] <icinga-miraheze>	 RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 97.51 ms
[00:57:28] <icinga-miraheze>	 RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 103.32 ms
[00:57:55] <icinga-miraheze>	 RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 97.91 ms
[01:01:23] <icinga-miraheze>	 PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 149.05 ms
[01:01:42] <icinga-miraheze>	 PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 156.47 ms
[01:02:04] <icinga-miraheze>	 PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 147.56 ms
[01:04:06] <icinga-miraheze>	 RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 98.48 ms
[01:05:30] <icinga-miraheze>	 RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 97.87 ms
[01:05:49] <icinga-miraheze>	 RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 101.64 ms
[01:11:44] <icinga-miraheze>	 PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 151.44 ms
[01:12:03] <icinga-miraheze>	 PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 155.53 ms
[01:12:16] <icinga-miraheze>	 PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.27 ms
[01:14:07] <icinga-miraheze>	 RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 102.03 ms
[01:14:18] <icinga-miraheze>	 RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 96.64 ms
[01:15:52] <icinga-miraheze>	 RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 98.08 ms
[01:38:32] <icinga-miraheze>	 PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 151.40 ms
[01:38:53] <icinga-miraheze>	 PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.85 ms
[01:38:55] <icinga-miraheze>	 PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 153.48 ms
[01:42:39] <icinga-miraheze>	 RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 97.42 ms
[01:42:57] <icinga-miraheze>	 RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 97.73 ms
[01:43:04] <icinga-miraheze>	 RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 101.90 ms
[01:48:53] <icinga-miraheze>	 PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 150.88 ms
[01:49:07] <icinga-miraheze>	 PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 150.38 ms
[01:49:19] <icinga-miraheze>	 PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 152.73 ms
[01:59:13] <icinga-miraheze>	 RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 99.12 ms
[01:59:19] <icinga-miraheze>	 RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 98.42 ms
[01:59:39] <icinga-miraheze>	 RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 103.49 ms
[02:07:30] <icinga-miraheze>	 PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.70 ms
[02:07:33] <icinga-miraheze>	 PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 150.26 ms
[02:07:58] <icinga-miraheze>	 PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 155.94 ms
[02:11:35] <icinga-miraheze>	 RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 98.06 ms
[02:11:38] <icinga-miraheze>	 RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 101.18 ms
[02:12:05] <icinga-miraheze>	 RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 102.35 ms
[06:25:48] <Not-7024>	 [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/Jt1fj
[06:25:50] <Not-7024>	 [02miraheze/dns] 07Reception123 031d9f06c - add datacrondatabase.com zone
[06:54:03] <Not-7024>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jt1JQ
[06:54:05] <Not-7024>	 [02miraheze/ssl] 07Reception123 039d326c3 - add datacrondatabase.com cert
[07:43:38] <icinga-miraheze>	 PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.005 second response time
[07:44:09] <icinga-miraheze>	 PROBLEM - cp11 Stunnel Http for mw10 on cp11 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.002 second response time
[07:44:11] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw9 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.769 second response time
[07:44:15] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.195.236.219/cpweb, 51.195.236.250/cpweb, 2001:41d0:800:178a::5/cpweb, 2001:41d0:800:1bbd::4/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb
[07:44:15] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.195.236.219/cpweb, 51.195.236.250/cpweb, 2001:41d0:800:178a::5/cpweb, 2001:41d0:800:1bbd::4/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb
[07:44:35] <icinga-miraheze>	 PROBLEM - cp11 Stunnel Http for mw11 on cp11 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.002 second response time
[07:45:36] <icinga-miraheze>	 RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20673 bytes in 0.228 second response time
[07:45:45] <icinga-miraheze>	 PROBLEM - mail2 IMAP on mail2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:46:06] <icinga-miraheze>	 RECOVERY - cp11 Stunnel Http for mw10 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 15131 bytes in 0.004 second response time
[07:46:12] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw9 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15130 bytes in 1.038 second response time
[07:46:13] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[07:46:13] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[07:46:34] <icinga-miraheze>	 RECOVERY - cp11 Stunnel Http for mw11 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 15131 bytes in 0.005 second response time
[08:42:23] <Reception123>	 paladox: how would I be able to unzip a rar file on jobrunner?
[08:45:29] <icinga-miraheze>	 RECOVERY - mail2 IMAP on mail2 is OK: IMAP OK - 0.006 second response time on 51.195.236.253 port 143 [* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS LOGINDISABLED] Dovecot (Debian) ready.]
[08:46:11] <RhinosF1>	 Morning
[08:46:11] <MirahezeBot>	 RhinosF1: 2021-02-07 - 23:06:03GMT <MH-Discord> tell RhinosF1 okay, night. sleep well. Sent by @Doug
[08:47:24] <RhinosF1>	 Reception123: https://www.google.com/amp/s/www.tecmint.com/how-to-open-extract-and-create-rar-files-in-linux/amp/
[08:47:24] <MirahezeBot>	 [ How to Open, Extract and Create RAR Files in Linux ] - www.google.com
[08:47:35] <RhinosF1>	 We might want to add that to jobrunner.pp
[08:50:05] <Not-7024>	 [02puppet] 07RhinosF1 opened pull request 03#1645: jobrunner: install unrar - 13https://git.io/Jt1tf
[08:50:23] <RhinosF1>	 Reception123: ^
[08:53:49] <icinga-miraheze>	 PROBLEM - mail2 IMAP on mail2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:54:59] <RhinosF1>	 That's bad
[08:55:47] <RhinosF1>	 Seems like it goes off a bit though
[09:02:24] <Reception123>	 RhinosF1: thanks, yeah I wanted to make sure if we had to install a package or if there was another way
[09:06:08] <Not-7024>	 [02puppet] 07Reception123 closed pull request 03#1645: jobrunner: install unrar - 13https://git.io/Jt1tf
[09:06:10] <Not-7024>	 [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt1ty
[09:06:11] <Not-7024>	 [02miraheze/puppet] 07RhinosF1 032c2cd82 - jobrunner: install unrar (#1645)
[09:35:37] <icinga-miraheze>	 PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[unrar]
[09:35:43] <icinga-miraheze>	 PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[unrar]
[09:35:51] <RhinosF1>	 Reception123: ^
[09:43:10] <Reception123>	 RhinosF1: guess it didn't like your PR. Any ideas or should I revert?
[09:44:09] <RhinosF1>	 Reception123: what's the full error?
[09:46:57] <Reception123>	 it's not happy
[09:47:00] <Reception123>	 https://www.irccloud.com/pastebin/v8tK3gtC/
[09:47:00] <MirahezeBot>	 [ Snippet | IRCCloud ] - www.irccloud.com
[09:47:28] <RhinosF1>	 Reception123: ye revert
[09:47:37] <Not-7024>	 [02miraheze/puppet] 07Reception123 pushed 031 commit to 03revert-1645-patch-28 [+0/-0/±1] 13https://git.io/Jt1m5
[09:47:39] <Not-7024>	 [02miraheze/puppet] 07Reception123 0378a22d8 - Revert "jobrunner: install unrar (#1645)"
[09:47:40] <Not-7024>	 [02puppet] 07Reception123 created branch 03revert-1645-patch-28 - 13https://git.io/vbiAS
[09:47:41] <Not-7024>	 [02puppet] 07Reception123 opened pull request 03#1646: Revert "jobrunner: install unrar" - 13https://git.io/Jt1md
[09:47:45] <Reception123>	 guess we'll see what was wrong when paladox is around
[09:47:48] <Not-7024>	 [02puppet] 07Reception123 closed pull request 03#1646: Revert "jobrunner: install unrar" - 13https://git.io/Jt1md
[09:47:49] <Not-7024>	 [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt1mF
[09:47:51] <Not-7024>	 [02miraheze/puppet] 07Reception123 036aee6b9 - Revert "jobrunner: install unrar (#1645)" (#1646)
[09:49:26] <icinga-miraheze>	 RECOVERY - mail2 IMAP on mail2 is OK: IMAP OK - 0.010 second response time on 51.195.236.253 port 143 [* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS LOGINDISABLED] Dovecot (Debian) ready.]
[10:03:37] <icinga-miraheze>	 RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[10:03:42] <icinga-miraheze>	 RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[10:09:37] <icinga-miraheze>	 PROBLEM - mail2 IMAP on mail2 is CRITICAL: No data received from host
[11:00:48] <JohnLewis>	 !log restart dovecot on mail2
[11:00:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:01:05] <icinga-miraheze>	 RECOVERY - mail2 IMAP on mail2 is OK: IMAP OK - 0.005 second response time on 51.195.236.253 port 143 [* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS LOGINDISABLED] Dovecot (Debian) ready.]
[11:01:19] <RhinosF1>	 ty JohnLewis, i was gonna ask why IMAP keeps flapping but I guess you know
[11:02:26] <JohnLewis>	 Not looked at it yet, I just wanted to access my email :P
[11:49:09] <Reception123>	 !log bash ./wikibackups.sh /home/reception/publicwikis.txt /srv/mediawiki/w/maintenance/dumpBackup.php (wikibackups)
[11:49:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:33:08] <dmehus>	 heh that's always nice when you're able to fix a server problem because you wanted to access your e-mail, eh, JohnLewis?
[12:34:10] <JohnLewis>	 always :P
[12:34:15] <dmehus>	 heh
[12:34:30] <dmehus>	 Reception123, did you miss adding the server you ran that command ^ on?
[12:34:43] <RhinosF1>	 I guess you're awake again dmehus
[12:35:12] <dmehus>	 RhinosF1, yeah, heh. I think I'll stay up now. It's a bit early, but meh.
[12:35:12] <RhinosF1>	 It'll be one of the jobrunners but iirc it's always the same one
[12:35:29] <Reception123>	 dmehus: whoops yeah, was jobrunner3
[12:35:33] <dmehus>	 RhinosF1, well not always, I think he alternates between jobrunners?
[12:35:43] <Reception123>	 dmehus: not for my general backups :)
[12:35:59] <dmehus>	 oh ok, true, yeah, you always use jobrunner3 for that don't you?
[12:36:04] <RhinosF1>	 dmehus: the backup script is only on one of them
[12:36:19] <dmehus>	 RhinosF1, heh yeah just realized that now. My bad
[12:36:39] <dmehus>	 that's just like running jobs where you alternate, I think?
[12:36:55] <RhinosF1>	 Most things can run on either
[12:37:12] <RhinosF1>	 Jobrunners are just MediaWiki servers that don't get traffic
[12:37:17] <RhinosF1>	 Mostly
[12:37:35] <dmehus>	 well, yeah, but I think Reception123 likes to alternate between jobrunner3 and jobrunner4 when he uses `runJobs.php`
[12:37:47] <dmehus>	 but yeah, you could always either server
[12:37:59] <dmehus>	 and you may not alternate as it's likely not a rule
[12:38:08] <Reception123>	 I do for most things yeah, but not the backups as the scripts are on one server so it's easier 
[12:38:16] <dmehus>	 yeah
[12:38:45] * dmehus wonders if Reception123 has the most files in his jobrunner3 home directory of all sysadmins
[12:39:14] * RhinosF1 normally just sees what mood he's in when picking a runner
[12:39:17] <Reception123>	 Heh
[12:39:44] <dmehus>	 lol, you're mood weighs heavily in your decisions eh, RhinosF1? :P
[12:40:08] <JohnLewis>	 Running jobs I do 3, investigating jobs I use 4 :P
[12:40:17] <RhinosF1>	 dmehus: easiest way, the two servers are virtually the same
[12:40:30] <dmehus>	 JohnLewis, oh, that's an interesting methodology as well, heh
[12:40:35] <RhinosF1>	 Unless it's the sitemap stuff
[12:40:40] <dmehus>	 RhinosF1, ack
[12:40:50] <RhinosF1>	 Because only one of them does scheduled jobs
[12:41:00] <dmehus>	 are sitemap jobs only run on one jobrunner as well?
[12:41:03] <dmehus>	 ah
[12:41:47] <RhinosF1>	 Anything that runs as a cron is only on one server
[12:43:58] <dmehus>	 ah, yeah, that makes sense to consolidate the cron jobs on one server
[12:58:06] <Reception123>	 JohnLewis: a bit of an emabarassing question, but I can't figure out why https://github.com/miraheze/puppet/commit/6aee6b9294c909258191d34144bbf4e61d74e1a1 didn't work
[12:58:06] <MirahezeBot>	 [ Revert "jobrunner: install unrar (#1645)" (#1646) · miraheze/puppet@6aee6b9 · GitHub ] - github.com
[12:58:14] <Reception123>	 trying to get rar installed so I can unrar an image file on jobrunner3 
[13:12:43] <paladox>	 https://packages.debian.org/buster/rar
[13:12:44] <MirahezeBot>	 [ Debian -- Details of package rar in buster ] - packages.debian.org
[13:12:44] <paladox>	 You need to add the non-free source
[13:13:22] <Reception123>	 oh
[13:19:17] <dmehus>	 JohnLewis, any idea why this log entry is attributed to "Global rename script" instead of to "Dmehus" when the user was globally renamed (https://dospedia.miraheze.org/w/index.php?title=Special:Log&logid=193)? I can't figure it out because on other wikis for the same rename and user, I'm correctly attributed
[13:19:18] <MirahezeBot>	 [ All public logs - Dospedia ] - dospedia.miraheze.org
[13:19:39] <dmehus>	 I suspect it's a bug in whatever that global rename script is, but no clue what it is
[13:20:08] <dmehus>	 I asked Reception123 and he said he's as stumped as I am
[13:45:35] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 51.195.236.250/cpweb
[13:45:54] <icinga-miraheze>	 PROBLEM - cp10 Stunnel Http for mw9 on cp10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[13:47:51] <icinga-miraheze>	 RECOVERY - cp10 Stunnel Http for mw9 on cp10 is OK: HTTP OK: HTTP/1.1 200 OK - 15130 bytes in 0.263 second response time
[13:49:38] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[14:16:38] <Not-7024>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt1Bf
[14:16:40] <Not-7024>	 [02miraheze/puppet] 07paladox 0335be0ff - dovecot: Only allow tls v1.2+
[14:22:05] <RhinosF1>	 dmehus: your account wasn't attached at point of rename so it couldn't attribute to you as you didn't exist
[14:22:16] <RhinosF1>	 I think we can reassign
[14:25:38] <dmehus>	 RhinosF1, I don't think that's it at all actually because other stewards attach their accounts at the point of renaming, whether just before or shortly after.
[14:25:57] <dmehus>	 I suspect it's a bug of some sort, but I'm not sure with what.
[14:26:07] <RhinosF1>	 dmehus: it's the only thing I can think of though
[14:26:26] <dmehus>	 yeah, that's too easy, though. I suspect it's something very mysterious
[14:28:58] <RhinosF1>	 dmehus: it's a catch
[14:29:10] <RhinosF1>	 For some reason creating your account failed
[14:30:27] <RhinosF1>	 dmehus: I can reassign the edit if it bothers you on that wiki
[14:30:45] <dmehus>	 I'm not too concerned about the attribution
[14:31:03] <dmehus>	 Mainly concerned about what caused it.
[14:31:18] <dmehus>	 If that's the case, there should be something in graylog then for that timestamp then that correlates to the rename
[14:32:13] <RhinosF1>	 I can try and poke graylog but I wouldn't worry as it's doing its job by catching it and not exploding
[14:33:44] <icinga-miraheze>	 PROBLEM - wiki.kourouklides.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kourouklides.com could not be found
[14:33:45] <icinga-miraheze>	 PROBLEM - www.mcpk.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mcpk.wiki could not be found
[14:33:45] <icinga-miraheze>	 PROBLEM - ipv6bolivia.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ipv6bolivia.tk could not be found
[14:33:46] <icinga-miraheze>	 PROBLEM - tensegritywiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tensegritywiki.com could not be found
[14:33:47] <icinga-miraheze>	 PROBLEM - pt.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pt.graalmilitary.com could not be found
[14:33:53] <icinga-miraheze>	 PROBLEM - wiki.macc.nyc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.macc.nyc could not be found
[14:33:53] <icinga-miraheze>	 PROBLEM - indiancannabis.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for indiancannabis.wiki could not be found
[14:33:56] <icinga-miraheze>	 PROBLEM - wiki.fraterniterebellion.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fraterniterebellion.com could not be found
[14:33:58] <icinga-miraheze>	 PROBLEM - wiki.campaign-labour.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.campaign-labour.org could not be found
[14:33:58] <icinga-miraheze>	 PROBLEM - spiral.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for spiral.wiki could not be found
[14:34:17] <dmehus>	 Well, the only thing it could possibly be, if your theory is correct, that it was the jobrunner that failed to attach my account before the rename on that wiki was complicated, which be cause of a lack of server resources or some other configuration issue we should probably fix
[14:35:31] <dmehus>	 If I'm being honest, not every report in #miraheze-sre needs an answer if you're not sure what it is and don't have time to thoroughly troubleshoot. You can just ignore it and leave it for someone else when they're available :)
[14:40:34] <icinga-miraheze>	 RECOVERY - pt.graalmilitary.com - reverse DNS on sslhost is OK: rDNS OK - pt.graalmilitary.com reverse DNS resolves to cp11.miraheze.org
[14:40:45] <icinga-miraheze>	 RECOVERY - wiki.kourouklides.com - reverse DNS on sslhost is OK: rDNS OK - wiki.kourouklides.com reverse DNS resolves to cp10.miraheze.org
[14:40:45] <icinga-miraheze>	 RECOVERY - www.mcpk.wiki - reverse DNS on sslhost is OK: rDNS OK - www.mcpk.wiki reverse DNS resolves to cp10.miraheze.org
[14:40:48] <icinga-miraheze>	 RECOVERY - ipv6bolivia.tk - reverse DNS on sslhost is OK: rDNS OK - ipv6bolivia.tk reverse DNS resolves to cp11.miraheze.org
[14:40:49] <icinga-miraheze>	 RECOVERY - tensegritywiki.com - reverse DNS on sslhost is OK: rDNS OK - tensegritywiki.com reverse DNS resolves to cp11.miraheze.org
[14:40:49] <icinga-miraheze>	 RECOVERY - wiki.fraterniterebellion.com - reverse DNS on sslhost is OK: rDNS OK - wiki.fraterniterebellion.com reverse DNS resolves to cp10.miraheze.org
[14:40:50] <icinga-miraheze>	 RECOVERY - wiki.macc.nyc - reverse DNS on sslhost is OK: rDNS OK - wiki.macc.nyc reverse DNS resolves to cp11.miraheze.org
[14:40:51] <icinga-miraheze>	 RECOVERY - indiancannabis.wiki - reverse DNS on sslhost is OK: rDNS OK - indiancannabis.wiki reverse DNS resolves to cp11.miraheze.org
[14:40:54] <icinga-miraheze>	 RECOVERY - wiki.campaign-labour.org - reverse DNS on sslhost is OK: rDNS OK - wiki.campaign-labour.org reverse DNS resolves to cp11.miraheze.org
[14:40:54] <icinga-miraheze>	 RECOVERY - spiral.wiki - reverse DNS on sslhost is OK: rDNS OK - spiral.wiki reverse DNS resolves to cp10.miraheze.org
[14:48:38] <Not-7024>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jt10k
[14:48:39] <Not-7024>	 [02miraheze/mw-config] 07Universal-Omega 033edbbb8 - Fix DynamicPageList3
[14:48:40] <Not-7024>	 [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3
[14:49:05] <Not-7024>	 [02mw-config] 07Universal-Omega opened pull request 03#3723: Fix DynamicPageList3 - 13https://git.io/Jt10t
[14:50:06] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[14:56:32] <Not-7024>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jt10w
[14:56:33] <Not-7024>	 [02miraheze/mw-config] 07Universal-Omega 03c657e71 - Update LocalWiki.php
[14:56:35] <Not-7024>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3723: Fix DynamicPageList3 - 13https://git.io/Jt10t
[14:57:36] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[14:59:43] <Not-7024>	 [02mw-config] 07Universal-Omega edited pull request 03#3723: Fix DynamicPageList3 (T6856) - 13https://git.io/Jt10t
[15:00:34] <Not-7024>	 [02mw-config] 07Universal-Omega closed pull request 03#3723: Fix DynamicPageList3 (T6856) - 13https://git.io/Jt10t
[15:00:36] <Not-7024>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jt10F
[15:00:37] <Not-7024>	 [02miraheze/mw-config] 07Universal-Omega 03e5e1911 - Fix DynamicPageList3 (T6856) (#3723)
[15:00:39] <Not-7024>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2
[15:00:40] <Not-7024>	 [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3
[15:01:41] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[15:06:39] <RhinosF1>	 dmehus: I don't even see that wiki in graylog
[15:09:49] <RhinosF1>	 dmehus: I think the log got purged
[15:10:45] <dmehus>	 oh
[15:11:05] <dmehus>	 We purge graylog after only 2 weeks?
[15:11:34] <dmehus>	 either that, or we're not logging that sort of data / errors to graylog?
[15:12:38] <RhinosF1>	 dmehus: I see data for that wiki going back to 2021-01-22 05:51:46.000 +00:00 but just not the rename
[15:13:24] <dmehus>	 Oh. Weird.
[15:13:49] <dmehus>	 what day did we migrate the servers?
[15:13:59] <RhinosF1>	 recently
[15:14:04] <RhinosF1>	 there's old server logs
[15:14:16] <RhinosF1>	 just not the ones i want
[15:14:34] <dmehus>	 hrm
[15:22:23] <RhinosF1>	 Universal_Omega: https://phabricator.miraheze.org/T6857 if you want something to look at
[15:22:24] <MirahezeBot>	 [ ⚓ T6857 RequestWikiQueue should handle Unknown Request ] - phabricator.miraheze.org
[15:24:05] <RhinosF1>	 JohnLewis: could we use https://docs.graylog.org/en/4.0/pages/alerts.html#alerts to set off high rate of exception alerts
[15:24:06] <MirahezeBot>	 [ Alerts — Graylog 4.0.0 documentation ] - docs.graylog.org
[15:52:25] <paladox>	 Universal_Omega https://github.com/Universal-Omega/DynamicPageList3/pull/12
[15:52:26] <MirahezeBot>	 [ Fix "Argument 1 passed to DPL\\Lister\\Lister::setSectionSeparators() must be of the type array, null given" by paladox · Pull Request #12 · Universal-Omega/DynamicPageList3 · GitHub ] - github.com
[15:53:34] <Universal_Omega>	 paladox: thanks, looking.
[15:57:54] <paladox>	 Cargo seems quite spammy with entries such as https://graylog.miraheze.org/messages/graylog_92/5923bd42-706e-11eb-b57d-0200001a24a4
[15:58:09] <paladox>	 which leads to line https://github.com/wikimedia/mediawiki-extensions-Cargo/blob/master/includes/CargoSQLQuery.php#L1555
[15:58:09] <MirahezeBot>	 [ mediawiki-extensions-Cargo/CargoSQLQuery.php at master · wikimedia/mediawiki-extensions-Cargo · GitHub ] - github.com
[15:59:01] <paladox>	 https://github.com/wikimedia/mediawiki-extensions-Cargo/commit/7e0e31a6641f0c07849ce45fec9b2ed5beb552ff#diff-2370c33202589a293eb6235a0ccc198062dc690735fd380ae5d1195d39c66d7c
[15:59:02] <MirahezeBot>	 [ Avoid assignment in conditions · wikimedia/mediawiki-extensions-Cargo@7e0e31a · GitHub ] - github.com
[15:59:09] <paladox>	 changed how it is done from [] to ->
[16:00:42] <RhinosF1>	 paladox: according to https://graylog.miraheze.org/search?q=mediawiki_exception_message%3A%22PHP+Notice%3A+Undefined+property%3A+stdClass%3A%3A%241%22&rangetype=relative&streams=5f8c6fd446640840f104b0ba&relative=900 there's about 450 a minute and that's happened twice
[16:01:02] <Universal_Omega>	 paladox: yeah, that commit in cargo already created another exception I fixed as well. It made a quite large change, which had unintended side effects. 
[16:01:16] <RhinosF1>	 they're highly concentrated bursts
[16:01:20] <Universal_Omega>	 And also I merged your PR to DPL3 paladox.
[16:01:52] <paladox>	 thanks!
[16:01:59] <paladox>	 Universal_Omega you fixed the cargo issue?
[16:02:36] <Universal_Omega>	 paladox: No problem. I fixed one of the cargo issues, but I don't think this one.
[16:02:51] <paladox>	 oh ok
[16:03:15] <Universal_Omega>	 That commit just had some unintended side effects to it.
[16:03:28] <Not-7024>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/Jt1gB
[16:03:29] <Not-7024>	 [02miraheze/mediawiki] 07paladox 03306c410 - Update DynamicPageList3
[16:04:24] <RhinosF1>	 Universal_Omega: looking at the stats that trace is causing most of our recent errors. Could we maybe given the devs a nudge to fix it?
[16:04:32] <RhinosF1>	 or add tests
[16:04:54] <paladox>	 oh wgCargoFileDataColumns
[16:05:12] <paladox>	 well i don't think its that
[16:05:40] <Universal_Omega>	 paladox: Oh. Did I mess up on another config change yesterday?
[16:05:45] <paladox>	 No
[16:05:59] <paladox>	 i don't think this is related to your config change for cargo
[16:06:04] <paladox>	 you are using the defaults
[16:07:33] <Universal_Omega>	 Oh good. Thanks!
[16:09:17] <icinga-miraheze>	 RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[16:12:29] <icinga-miraheze>	 PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[16:30:34] <paladox>	 Universal_Omega this https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Cargo/+/664261 should fix it
[16:31:12] <paladox>	 i dumped the contents of $alias
[16:31:16] <paladox>	 and i saw stuff like COUNT(SkillName)
[16:31:30] <Universal_Omega>	 Oh good. Thanks!
[16:32:27] <dmehus>	 yay, your hostmask finally joined with you at the same time, R4356th
[16:33:48] <R4356th>	 I forgot to rejoin after verifying via NickServ last time in #miraheze. :P
[16:34:53] <icinga-miraheze>	 RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:35:19] <icinga-miraheze>	 PROBLEM - files.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for files.petrawiki.org could not be found
[16:42:07] <icinga-miraheze>	 RECOVERY - files.petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - files.petrawiki.org reverse DNS resolves to cp11.miraheze.org
[16:42:11] <dmehus>	 R4356th, oh, heh
[16:45:01] <icinga-miraheze>	 PROBLEM - wikiru.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikiru.wildterra2.com could not be found
[16:45:01] <icinga-miraheze>	 PROBLEM - privacy-wiki.0x.no - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for privacy-wiki.0x.no could not be found
[16:45:02] <icinga-miraheze>	 PROBLEM - wiki.responsibly.ai - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.responsibly.ai could not be found
[16:45:02] <icinga-miraheze>	 PROBLEM - wiki.mikrodev.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mikrodev.com could not be found
[16:45:02] <icinga-miraheze>	 PROBLEM - www.erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.erikapedia.com could not be found
[16:45:04] <icinga-miraheze>	 PROBLEM - dc.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dc.miraheze.org could not be found
[16:45:04] <icinga-miraheze>	 PROBLEM - wiki.teamrelectric.ca - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.teamrelectric.ca could not be found
[16:45:04] <icinga-miraheze>	 PROBLEM - wiki.vinesh.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.vinesh.eu.org could not be found
[16:45:06] <icinga-miraheze>	 PROBLEM - wikiescola.com.br - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikiescola.com.br could not be found
[16:45:07] <icinga-miraheze>	 PROBLEM - en.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.petrawiki.org could not be found
[16:45:12] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found
[16:45:12] <icinga-miraheze>	 PROBLEM - en.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.gyaanipedia.com could not be found
[16:45:12] <icinga-miraheze>	 PROBLEM - wiki.anglish.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.anglish.info could not be found
[16:45:13] <icinga-miraheze>	 PROBLEM - wiki.helioss.co - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.helioss.co could not be found
[16:51:42] <icinga-miraheze>	 RECOVERY - privacy-wiki.0x.no - reverse DNS on sslhost is OK: rDNS OK - privacy-wiki.0x.no reverse DNS resolves to cp10.miraheze.org
[16:51:45] <icinga-miraheze>	 RECOVERY - wikiru.wildterra2.com - reverse DNS on sslhost is OK: rDNS OK - wikiru.wildterra2.com reverse DNS resolves to cp11.miraheze.org
[16:51:45] <icinga-miraheze>	 RECOVERY - wiki.responsibly.ai - reverse DNS on sslhost is OK: rDNS OK - wiki.responsibly.ai reverse DNS resolves to cp11.miraheze.org
[16:51:46] <icinga-miraheze>	 RECOVERY - wiki.mikrodev.com - reverse DNS on sslhost is OK: rDNS OK - wiki.mikrodev.com reverse DNS resolves to cp10.miraheze.org
[16:51:46] <icinga-miraheze>	 RECOVERY - www.erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - www.erikapedia.com reverse DNS resolves to cp11.miraheze.org
[16:51:50] <icinga-miraheze>	 RECOVERY - wiki.teamrelectric.ca - reverse DNS on sslhost is OK: rDNS OK - wiki.teamrelectric.ca reverse DNS resolves to cp10.miraheze.org
[16:51:51] <icinga-miraheze>	 RECOVERY - dc.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - dc.miraheze.org reverse DNS resolves to cp11.miraheze.org
[16:51:51] <icinga-miraheze>	 RECOVERY - wiki.vinesh.eu.org - reverse DNS on sslhost is OK: rDNS OK - wiki.vinesh.eu.org reverse DNS resolves to cp11.miraheze.org
[16:51:53] <icinga-miraheze>	 RECOVERY - wikiescola.com.br - reverse DNS on sslhost is OK: rDNS OK - wikiescola.com.br reverse DNS resolves to cp10.miraheze.org
[16:51:59] <icinga-miraheze>	 RECOVERY - en.petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - en.petrawiki.org reverse DNS resolves to cp10.miraheze.org
[16:52:05] <icinga-miraheze>	 RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: rDNS OK - franchise.franchising.org.ua reverse DNS resolves to cp11.miraheze.org
[16:52:06] <icinga-miraheze>	 RECOVERY - en.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - en.gyaanipedia.com reverse DNS resolves to cp10.miraheze.org
[16:52:07] <icinga-miraheze>	 RECOVERY - wiki.anglish.info - reverse DNS on sslhost is OK: rDNS OK - wiki.anglish.info reverse DNS resolves to cp11.miraheze.org
[16:52:08] <icinga-miraheze>	 RECOVERY - wiki.helioss.co - reverse DNS on sslhost is OK: rDNS OK - wiki.helioss.co reverse DNS resolves to cp10.miraheze.org
[16:58:46] <Not-7024>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/Jt1wc
[16:58:48] <Not-7024>	 [02miraheze/mediawiki] 07paladox 03b1465a8 - Update Cargo
[17:32:27] <Reception123>	 RhinosF1: r.e. https://phabricator.miraheze.org/T6857 if you're tagging things, might as well set the priority, it won't hurt
[17:32:28] <MirahezeBot>	 [ ⚓ T6857 RequestWikiQueue should handle Unknown Request ] - phabricator.miraheze.org
[17:35:10] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.03, 6.35, 5.05
[17:37:09] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.78, 5.38, 4.86
[17:47:35] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 267.25 ms
[17:49:36] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 267.30 ms
[17:57:38] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 268.22 ms
[18:03:44] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 268.86 ms
[18:07:45] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 267.65 ms
[18:09:45] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 269.70 ms
[18:11:47] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 270.33 ms
[18:12:19] <RhinosF1>	 Reception123: we should probably set normal by default
[18:13:07] <RhinosF1>	 On https://phabricator.miraheze.org/transactions/editengine/maniphest.task/view/1/
[18:13:08] <MirahezeBot>	 [ Form 1 ] - phabricator.miraheze.org
[18:13:48] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 267.55 ms
[18:15:51] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 267.46 ms
[18:16:13] <Reception123>	 yeah, or we could reorganise the forms a bit and have one for bugs and keep https://phabricator.miraheze.org/maniphest/task/edit/form/7/ for requests
[18:16:14] <MirahezeBot>	 [ Login ] - phabricator.miraheze.org
[18:16:39] <RhinosF1>	 Reception123: we could look at templates too
[18:17:05] <RhinosF1>	 Or try and have a task subtypes like wikimedia do
[18:17:23] <Reception123>	 Yeah, I think reorganizing a bit could be effective 
[18:17:50] <icinga-miraheze>	 RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 249.21 ms
[18:19:53] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 267.50 ms
[18:20:01] <RhinosF1>	 Reception123: why don't you talk to wikimedia and see how they do the prod error form
[18:20:59] <Reception123>	 let me have a look at it first and see what it's like (I don't think I've ever used it)
[18:21:54] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 269.77 ms
[18:22:07] <Reception123>	 RhinosF1: is https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/view/46/ what you mean?
[18:22:08] <MirahezeBot>	 [ Form 46 ] - phabricator.wikimedia.org
[18:22:34] <RhinosF1>	 Yes Reception123
[18:22:40] <Reception123>	 ok
[18:26:06] <RhinosF1>	 Reception123: we'd just have graylog instead of phatility
[18:26:45] <Reception123>	 yeah
[18:26:45] <RhinosF1>	 That's just a bridge between logstash and phab
[18:27:57] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 267.57 ms
[18:31:59] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 268.97 ms
[18:35:12] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 258.62 ms
[18:40:06] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 267.81 ms
[18:44:10] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 268.20 ms
[18:46:10] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 269.86 ms
[18:52:04] <dmehus>	 RhinosF1, oh, SPF|Cloud would like that, I think. He's all about centralized logging. It does make sense actually, as long as the centralized logs log everything
[19:17:58] <icinga-miraheze>	 RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 249.58 ms
[21:09:35] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::1c30/cpweb
[21:11:34] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[21:20:47] <RhinosF1>	 .op
[21:20:47] <MirahezeBot>	 Attempting to OP...
[21:21:02] * RhinosF1 is standing by for incident
[21:29:22] <RhinosF1>	 .deop
[21:29:29] <RhinosF1>	 Things recovering
[21:36:24] <dmehus>	 paladox or JohnLewis, https://static.miraheze.org/braveninewiki/2/24/Icon-MP-1.png is still displaying for me (using cp9) despite reverting...can we flush or purge that image from the Varnish cache system?
[21:36:41] <RhinosF1>	 I can see it too
[21:37:08] <paladox>	 root@mw9:/mnt/mediawiki-static# ls braveninewiki/2/24/Icon-MP-1.png
[21:37:08] <paladox>	 braveninewiki/2/24/Icon-MP-1.png
[21:37:49] <RhinosF1>	 dmehus: it ain't deleted ^
[21:38:04] <RhinosF1>	 Or if it is the it's not moved it properly in static
[21:38:39] <dmehus>	 Oh, John told me he saw the original version, though
[21:39:05] <RhinosF1>	 dmehus: link me to the file on wiki
[21:39:17] <dmehus>	 I clicked "revert" on the oldest, good version of the image but I'm still seeing the Chinese flag vandalism
[21:39:20] <dmehus>	 sure
[21:39:52] <RhinosF1>	 paladox: do you see a painting or a Chinese flag?
[21:39:56] <dmehus>	 https://bravenine.miraheze.org/wiki/File:Icon-MP-1.png
[21:39:57] <MirahezeBot>	 [ File:Icon-MP-1.png - Brave Nine Wiki ] - bravenine.miraheze.org
[21:40:13] <paladox>	 its anime for me
[21:40:26] <RhinosF1>	 Yeah
[21:40:38] <RhinosF1>	 paladox: check cp9 then for Doug
[21:40:46] <RhinosF1>	 dmehus: try ctrl+F5 too
[21:41:11] <paladox>	 ^
[21:42:11] <dmehus>	 yeah I did
[21:42:13] <dmehus>	 https://bravenine.miraheze.org/wiki/File:Icon-MP-1.png
[21:42:13] <MirahezeBot>	 [ File:Icon-MP-1.png - Brave Nine Wiki ] - bravenine.miraheze.org
[21:42:22] <dmehus>	 I think it's cp9 / Varnish
[21:42:27] <RhinosF1>	 paladox: varnish ban?
[21:42:59] <dmehus>	 Nevermind, Ctrl+F5 worked for me.
[21:43:11] <dmehus>	 but not a bad idea to kill it from Varnish
[21:43:13] <paladox>	 we don't have cp9
[21:43:13] <RhinosF1>	 Ah ok good
[21:43:18] <dmehus>	 oops
[21:43:22] <dmehus>	 whatever new cp9 is then
[21:43:28] <RhinosF1>	 Canada
[21:43:29] <dmehus>	 cp12
[21:43:35] <dmehus>	 sorry wrong server
[21:44:07] <paladox>	 oh
[21:44:14] <paladox>	 !log varnish> ban req.http.Host == static.miraheze.org && req.url ~ "^/braveninewiki/2/24/Icon-MP-1.png" - cp12
[21:44:17] <paladox>	 already banned it
[21:44:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:41:18] <dmehus>	 paladox, ty