[00:24:49] <Not-f817>	 [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjjDP
[00:24:51] <Not-f817>	 [02miraheze/IncidentReporting] 07paladox 03d002878 - round $outageTotal
[00:25:09] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjjDX
[00:25:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 03bc831b3 - BOT: Updating services config for wikis
[00:26:12] <JohnLewis>	 paladox: why jsut outage total?
[00:26:16] <JohnLewis>	 why not visible too?
[00:26:23] <JohnLewis>	 also where is the Changelog entry?
[00:26:23] <paladox>	 oh
[00:26:26] <Not-f817>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjjD1
[00:26:27] <Not-f817>	 [02miraheze/mediawiki] 07paladox 03e3d422c - Update IncidentReporting
[00:26:28] <paladox>	 right
[00:26:34] <paladox>	 err, let me add that
[00:26:42] <JohnLewis>	 and then the version change
[00:26:56] <Not-f817>	 [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjjDM
[00:26:58] <Not-f817>	 [02miraheze/IncidentReporting] 07paladox 03314ee33 - Round $outageVisible
[00:27:45] <Not-f817>	 [02IncidentReporting] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fh5YJ
[00:27:46] <Not-f817>	 [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fjjDD
[00:27:48] <Not-f817>	 [02miraheze/IncidentReporting] 07paladox 03494221f - Bump version to 1.0.3
[00:27:49] <Not-f817>	 [02IncidentReporting] 07paladox opened pull request 03#4: Bump version to 1.0.3 - 13https://git.io/fjjDy
[00:28:06] <Not-f817>	 [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fjjDS
[00:28:07] <Not-f817>	 [02miraheze/IncidentReporting] 07paladox 0333af133 - Update extension.json
[00:28:09] <Not-f817>	 [02IncidentReporting] 07paladox synchronize pull request 03#4: Bump version to 1.0.3 - 13https://git.io/fjjDy
[00:28:23] <Not-f817>	 [02IncidentReporting] 07paladox closed pull request 03#4: Bump version to 1.0.3 - 13https://git.io/fjjDy
[00:28:25] <Not-f817>	 [02miraheze/IncidentReporting] 07paladox pushed 033 commits to 03master [+0/-0/±4] 13https://git.io/fjjD9
[00:28:26] <Not-f817>	 [02miraheze/IncidentReporting] 07paladox 03418cdb1 - Merge pull request #4 from miraheze/paladox-patch-2 Bump version to 1.0.3
[00:28:56] <paladox>	 JohnLewis done ^
[00:28:57] <Not-f817>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjjDH
[00:28:59] <Not-f817>	 [02miraheze/mediawiki] 07paladox 03757e577 - Update IncidentReporting
[00:29:04] <JohnLewis>	 k
[00:29:50] <Not-f817>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjjDQ
[00:29:52] <Not-f817>	 [02miraheze/mediawiki] 07paladox 032939fba - Update ArticleFeedbackv5
[00:30:19] <Not-f817>	 [02miraheze/IncidentReporting] 07paladox deleted branch 03paladox-patch-2
[00:30:20] <Not-f817>	 [02IncidentReporting] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fh5YJ
[00:32:16] <travis-ci>	 miraheze/IncidentReporting/paladox-patch-2/33af133 - paladox The build has errored. https://travis-ci.com/miraheze/IncidentReporting/builds/126304734
[00:34:03] <paladox>	 !log UPDATE  `incidents` SET i_outage_total = '51',i_outage_visible = '53.15' WHERE i_id = '21'; - db4
[00:34:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[00:34:25] <JohnLewis>	 paladox: why 53.13?
[00:34:39] <paladox>	 I was only looking at trying to fix the query :)
[00:34:42] <paladox>	 but good idea
[00:34:44] <paladox>	 need to do both
[00:39:39] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[00:41:07] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[00:41:19] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[00:41:32] <paladox>	 that's due to the ddos attack against wikimedia
[00:41:38] <paladox>	 it carn't access phabricator
[01:13:41] <icinga-miraheze>	 PROBLEM - wiki.lostminecraftminers.org - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[01:14:11] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjjyZ
[01:14:13] <Not-f817>	 [02miraheze/puppet] 07paladox 03abef6e4 - lizardfs:client: Set atboot to false
[01:26:43] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[01:26:51] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[01:28:43] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[01:28:48] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[01:39:50] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjjy2
[01:39:51] <Not-f817>	 [02miraheze/puppet] 07paladox 03c0630c2 - Revert "lizardfs:client: Set atboot to false" This reverts commit abef6e4369a07ae2bac64bcd763405efe2d189a7.
[01:53:59] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[01:56:33] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[02:05:37] <ZppixBot>	 [ANNOUNCEMENT] Attention: For those unaware, Wikimedia is under a DDoS attack, so it may be unreachable, for updates please watch @wikipedia on twitter or #wikimedia-tech
[02:36:23] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[06:22:17] <icinga-miraheze>	 PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:26:11] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2989 MB (12% inode=94%);
[07:02:17] <icinga-miraheze>	 RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[07:33:26] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[09:03:26] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[09:13:10] <RhinosF1>	 Reception123: ping
[09:29:53] <Reception123>	 hi
[12:15:18] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjj5d
[12:15:19] <Not-f817>	 [02miraheze/puppet] 07paladox 033c9ec0e - disable enable_proton by default
[12:45:22] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures
[12:49:17] <icinga-miraheze>	 PROBLEM - misc3 restbase on misc3 is CRITICAL: connect to address 185.52.1.71 and port 7231: Connection refused
[12:51:15] <icinga-miraheze>	 RECOVERY - misc3 restbase on misc3 is OK: TCP OK - 0.005 second response time on 185.52.1.71 port 7231
[12:59:49] <Zppix>	 Jesus i feel bad for wmf operations  rn
[13:00:28] <icinga-miraheze>	 PROBLEM - misc3 electron on misc3 is CRITICAL: connect to address 185.52.1.71 and port 3000: Connection refused
[13:17:52] <RhinosF1>	 Zppix: tools still down
[13:18:54] <Zppix>	 Ik
[13:19:06] <Zppix>	 ZppixBot: ping
[13:19:12] <Zppix>	 .ping
[13:19:13] <ZppixBot>	 Pong!
[13:19:31] <Zppix>	 You use to be able to mention its name and a cmd :/
[13:19:54] <RhinosF1>	 Zppix: let's hope the promised restart doesn't come
[13:20:07] <Zppix>	 It probably will
[13:24:26] <RhinosF1>	 Zppix: I should be around all day if it does
[13:24:46] <Zppix>	 Ditto
[13:28:08] <RhinosF1>	 Zppix: any way to clear a notification from  a private wiki?
[13:28:15] <Zppix>	 Idk
[13:28:27] <Zppix>	 RhinosF1: update from operations its quiet for now
[13:28:47] <RhinosF1>	 Zppix: due same time as last night so 3 hours
[13:29:22] <Zppix>	 18 hour long ddos RhinosF1 thats got to be a record
[13:29:48] <RhinosF1>	 Zppix: actual DDoS was 9 hours but for time to recover probably
[13:29:50] <Reception123>	 I'm surprised it could happen to Wikipedia...
[13:30:08] <icinga-miraheze>	 RECOVERY - misc3 electron on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.71 port 3000
[13:30:25] <Zppix>	 Reception123: everything that has a internet connection it can happen to, no one is invulnerable
[13:31:56] <RhinosF1>	 Reception123: it was at least 2.5x normal traffic
[13:32:08] <RhinosF1>	 Up to 10x based on estimates
[13:32:08] <Reception123>	 Zppix: yeah, though it had to be a large attack still
[13:32:28] <Zppix>	 It was
[13:32:30] <RhinosF1>	 Reception123: I'll pm you cause not sure how public the rate was
[13:32:41] <Zppix>	 Pm me too?
[13:33:26] <Reception123>	 ok
[13:33:39] <RhinosF1>	 Zppix, Reception123: done
[13:38:31] <Reception123>	 saw
[13:41:15] <Zppix>	 Seen
[13:59:46] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2640 MB (10% inode=94%);
[14:11:02] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[14:13:37] <RhinosF1>	 Hey Voidwalker
[14:13:57] <Voidwalker>	 hi
[14:14:47] <RhinosF1>	 Voidwalker: you forgot something you said you'd do the other day
[14:15:14] <Voidwalker>	 yeah
[14:15:14] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2661 MB (11% inode=94%);
[14:15:25] <RhinosF1>	 Voidwalker: any chance of a close?
[14:35:14] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2617 MB (10% inode=94%);
[15:33:34] <icinga-miraheze>	 PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[15:53:34] <icinga-miraheze>	 RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:01:38] <RhinosF1>	 paladox: I noticed a bug in Incident Reporting, shouldn't the log drop /edit off the page name?
[18:05:35] <paladox>	 Yes i would think so.
[18:08:59] <RhinosF1>	 paladox: then we have a bug? Phab or a chance of a quick fix
[18:09:40] <paladox>	 i could have sworn i remember discovering it. But forgot if it's intentional. So for now put it in phab.
[18:10:38] <RhinosF1>	 Will do in a minute
[18:12:18] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:13:26] <RhinosF1>	 paladox: https://phabricator.miraheze.org/T4704
[18:13:27] <ZppixBot>	 [ ⚓ T4704 Remove /edit from Incident Reporting logs ] - phabricator.miraheze.org
[18:13:34] <paladox>	 thanks
[18:16:12] <Zppix>	 I really should make ZppixBot  not treat phab links as a link but as if it was a .task
[18:16:18] <Zppix>	 but i cant be arsed
[18:16:37] <RhinosF1>	 Np, can you look at the issue with Saved filters on RC as well paladox
[18:16:48] <Zppix>	 I wondered what happened to my filter 
[18:17:34] <RhinosF1>	 https://phabricator.miraheze.org/T4703
[18:17:35] <ZppixBot>	 [ ⚓ T4703 The website won't remember options I set, such as syntax highlighting or RC saved filters ] - phabricator.miraheze.org
[18:22:18] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[19:08:26] <paladox>	 !log editing /etc/network/interfaces on misc3 
[19:09:21] <paladox>	 uh
[19:09:24] <paladox>	 umounting again
[19:10:17] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[19:10:31] <icinga-miraheze>	 PROBLEM - misc3 Current Load on misc3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:10:33] <icinga-miraheze>	 PROBLEM - misc3 SSH on misc3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:10:35] <icinga-miraheze>	 PROBLEM - misc3 electron on misc3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:10:38] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[19:10:39] <icinga-miraheze>	 PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:10:40] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for misc3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:10:43] <icinga-miraheze>	 PROBLEM - mw1 HTTPS on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:10:45] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[19:10:46] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:10:53] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:11:01] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:11:01] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:11:04] <icinga-miraheze>	 PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:11:05] <icinga-miraheze>	 PROBLEM - misc3 Lizardfs Master Port 3 on misc3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:11:10] <Voidwalker>	 what did you do :P
[19:11:12] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:11:12] <icinga-miraheze>	 PROBLEM - misc3 zotero on misc3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:11:17] <icinga-miraheze>	 PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 44%
[19:11:18] <icinga-miraheze>	 PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:11:23] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:11:23] <icinga-miraheze>	 PROBLEM - misc3 lizard.miraheze.org HTTPS on misc3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:11:29] <icinga-miraheze>	 PROBLEM - Host misc3 is DOWN: PING CRITICAL - Packet loss = 100%
[19:11:34] <icinga-miraheze>	 PROBLEM - cp2 Stunnel Http for misc3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:11:39] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for misc3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:11:55] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[19:11:56] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:12:03] <icinga-miraheze>	 PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:12:11] <paladox>	 back
[19:12:22] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[19:12:28] <icinga-miraheze>	 PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 45%
[19:12:30] <icinga-miraheze>	 PROBLEM - mw1 MirahezeRenewSsl on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:12:31] <icinga-miraheze>	 PROBLEM - mw1 SSH on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:12:38] <icinga-miraheze>	 RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24516 bytes in 0.392 second response time
[19:12:38] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for misc3 on cp3 is OK: HTTP OK: Status line output matched "401" - 381 bytes in 0.501 second response time
[19:12:53] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24522 bytes in 0.508 second response time
[19:12:56] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24500 bytes in 0.633 second response time
[19:12:59] <icinga-miraheze>	 RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.006 second response time
[19:13:06] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24500 bytes in 0.005 second response time
[19:13:17] <icinga-miraheze>	 RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 2%
[19:13:18] <icinga-miraheze>	 PROBLEM - mw1 php-fpm on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:13:19] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24500 bytes in 0.675 second response time
[19:13:29] <icinga-miraheze>	 RECOVERY - Host misc3 is UP: PING OK - Packet loss = 0%, RTA = 0.67 ms
[19:13:29] <icinga-miraheze>	 PROBLEM - misc3 Lizardfs Master Port 1 on misc3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:13:32] <icinga-miraheze>	 RECOVERY - cp2 Stunnel Http for misc3 on cp2 is OK: HTTP OK: Status line output matched "401" - 381 bytes in 0.302 second response time
[19:13:35] <icinga-miraheze>	 RECOVERY - misc3 Lizardfs Master Port 1 on misc3 is OK: TCP OK - 0.003 second response time on 185.52.1.71 port 9419
[19:13:36] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for misc3 on cp4 is OK: HTTP OK: Status line output matched "401" - 381 bytes in 0.007 second response time
[19:13:50] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:14:01] <icinga-miraheze>	 RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24516 bytes in 0.392 second response time
[19:14:16] <icinga-miraheze>	 PROBLEM - mw1 Disk Space on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:14:17] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[19:14:24] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:14:28] <icinga-miraheze>	 RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 6%
[19:14:45] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[19:16:39] <icinga-miraheze>	 RECOVERY - mw1 MirahezeRenewSsl on mw1 is OK: TCP OK - 7.000 second response time on 185.52.1.75 port 5000
[19:16:42] <icinga-miraheze>	 RECOVERY - mw1 SSH on mw1 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u6 (protocol 2.0)
[19:16:51] <icinga-miraheze>	 RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.010 second response time
[19:17:11] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24500 bytes in 0.662 second response time
[19:17:21] <icinga-miraheze>	 RECOVERY - mw1 php-fpm on mw1 is OK: PROCS OK: 4 processes with command name 'php-fpm7.2'
[19:17:54] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 15 minutes ago with 0 failures
[19:17:55] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy
[19:18:01] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24516 bytes in 0.027 second response time
[19:18:22] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy
[19:18:23] <icinga-miraheze>	 RECOVERY - mw1 Disk Space on mw1 is OK: DISK OK - free space: / 32319 MB (42% inode=98%);
[19:18:32] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 2.61, 0.84, 0.29
[19:18:38] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy
[19:19:40] <icinga-miraheze>	 RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24516 bytes in 0.394 second response time
[19:22:43] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[19:32:58] <Zppix>	 paladox:  i think miraheze is  a little woozy
[19:33:07] <paladox>	 heh, it was my fault
[19:33:13] <paladox>	 quickly got it up
[19:33:22] <paladox>	 *back
[19:37:44] <RhinosF1>	 Zppix: https://mobile.twitter.com/UKDrillas/status/1170417459582705664
[19:37:45] <ZppixBot>	 [ Twitter ] - mobile.twitter.com
[19:39:39] <Zppix>	 RhinosF1:  meh
[19:39:42] <Zppix>	 RhinosF1:  hes lying
[19:40:05] <Reception123>	 paladox: they're laughing at you :P 
[19:40:17] <paladox>	 hahaha
[19:40:19] <Reception123>	 but yeah, I wouldn't trust anything they say
[19:40:33] <RhinosF1>	 Zppix: exactly what a doxxed person would say
[19:47:59] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjjh0
[19:48:01] <Not-f817>	 [02miraheze/puppet] 07paladox 03df784bc - mediawiki: Make php fpm childs configurable
[21:08:11] <Zppix>	 paladox:  does miraheze not do ipv6?
[21:08:34] <paladox>	 We have a ipv6 for some of our servers.
[21:09:16] <Zppix>	 oh
[21:09:41] <Zppix>	 paladox:  hows the RN issue ?
[21:10:10] <paladox>	 Zppix resolved (if you mean from yesturday)
[21:10:40] <paladox>	 i have a new one opened today, different issue.
[21:10:42] <Zppix>	 paladox:  How come i dont get an favicon on meta
[21:11:19] <paladox>	 i get one Zppix 
[21:11:41] <Zppix>	 i dont think its my cache let me see
[21:11:57] <Zppix>	 paladox:  i cleared my cache and dont get any
[21:12:08] <paladox>	 hmm
[21:13:01] <paladox>	 Zppix https://meta.miraheze.org/favicon.ico
[21:13:04] <paladox>	 does that work?
[21:13:59] <Zppix>	 wfm
[21:14:16] <Zppix>	 do you want my connection info paladox ?
[21:14:30] <Zppix>	 could be a bad cache on cp
[21:14:31] <paladox>	 ok, what browser do you use?
[21:14:40] <Zppix>	 paladox:  firefox latest on x32 win
[21:15:33] <paladox>	 !log ban req.http.Host == meta.miraheze.org && req.url ~ "^/favicon.ico" - cp2
[21:15:37] <paladox>	 Zppix try now please
[21:15:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:16:04] <Zppix>	 just for reference im on cp2
[21:16:17] <Zppix>	 paladox:  have one now
[21:16:22] <paladox>	 great :)
[21:16:39] <Voidwalker>	 mind if I do https://phabricator.miraheze.org/T4702 ?
[21:16:40] <ZppixBot>	 [ ⚓ T4702 Add some feature and extensions ] - phabricator.miraheze.org
[21:16:45] <Zppix>	 was i right then paladox ?
[21:20:29] <Zppix>	 Voidwalker:  is it possible with your script to make it force any action to be marked as bot even w/o flooder right? if so i can look at modifying it for you so you dont have to do the work
[21:20:38] <Zppix>	 massgblock script*
[21:21:02] <Voidwalker>	 not that I'm aware of
[21:21:35] <Voidwalker>	 best you could do is make the script force you into a group before executing
[21:22:01] <Zppix>	 Voidwalker:  I ask because i almost blocked an entire asn w/o flooder luckily i close the tab before it got to far
[21:22:14] <Zppix>	 Voidwalker:  hmm ill look into doing that and ill let you know
[21:22:31] <Voidwalker>	 sure
[21:22:39] <Zppix>	 should it force users to be in flooder? or does steward grant use of bot flag?
[21:23:41] <Voidwalker>	 since the script is a CVT tool, I'd say you should go for flooder, or allow the user to specify what group (but default to flooder or something similar)
[21:24:20] <Zppix>	 Voidwalker:  I think for now ill just require flooder unless the user is in group steward?
[21:25:54] <Voidwalker>	 how does that make sense for the steward?
[21:26:15] <Zppix>	 Voidwalker:  oh i was under the impression steward was by default able to use bot flag 
[21:26:16] <Zppix>	 nm
[21:26:40] <Voidwalker>	 but I'd say, go for what change you want to make, and I'll see about making my change to it later
[21:26:52] <Zppix>	 ok
[21:28:55] <paladox>	 Zppix: I guess
[21:29:23] <Voidwalker>	 oh, paladox, should I do https://phabricator.miraheze.org/T4702 ?
[21:29:24] <ZppixBot>	 [ ⚓ T4702 Add some feature and extensions ] - phabricator.miraheze.org
[21:29:47] <Voidwalker>	 I have the page ready in a tab, so it can be done immediately
[21:33:47] <Not-f817>	 [02ssl] 07RhinosF1 opened pull request 03#216: Remove cert ( wiki.lostminecraftminers.org ) - 13https://git.io/fjjjd
[21:34:02] <Not-f817>	 [02ssl] 07RhinosF1 edited pull request 03#216: Remove cert ( wiki.lostminecraftminers.org ) - 13https://git.io/fjjjd
[21:34:44] <Not-f817>	 [02ssl] 07RhinosF1 synchronize pull request 03#216: Remove cert ( wiki.lostminecraftminers.org ) - 13https://git.io/fjjjd
[21:35:10] <RhinosF1>	 paladox: ^
[21:35:21] <Voidwalker>	 whelp, done it anyway, don't yell at me plox :)
[21:36:50] <RhinosF1>	 Voidwalker: meh, done for his other wiki
[21:37:09] <RhinosF1>	 (he literally copied the task description)
[21:38:46] <Voidwalker>	 I'll be back later, hope nothing catches fire :P
[21:39:21] <RhinosF1>	 Voidwalker: give me a sec adding naf now if paladox will merge
[21:39:46] <Not-f817>	 [02mw-config] 07RhinosF1 opened pull request 03#2752: add .naf per T4702 - 13https://git.io/fjjjx
[22:08:05] <Zppix>	 Voidwalker:  it works https://publictestwiki.com/wiki/User:Zppix/testgblock.js
[22:08:06] <ZppixBot>	 [ User:Zppix/testgblock.js - TestWiki ] - publictestwiki.com
[22:08:35] <RhinosF1>	 well done Zppix - can you poke about wm-bot again?
[22:08:53] <Zppix>	 yea
[22:09:41] <RhinosF1>	 thx Zppix 
[22:11:56] <Zppix>	 RhinosF1:  {{done}}
[22:12:16] <RhinosF1>	 :)
[22:17:36] <Zppix>	 RhinosF1:  twitter suspended ukdrillas
[22:18:40] <RhinosF1>	 Zppix: that might be for the best
[22:19:01] <RhinosF1>	 hopefully it means he's about to arrested and it was co-ordinated
[22:19:40] <Zppix>	 *shrugs*
[22:20:16] <paladox>	 Voidwalker yes please :)
[22:20:38] <RhinosF1>	 paladox: can you merge my patches /mw-config and /ssl
[22:20:43] <Not-f817>	 [02ssl] 07paladox closed pull request 03#216: Remove cert ( wiki.lostminecraftminers.org ) - 13https://git.io/fjjjd
[22:20:44] <paladox>	 yup, done
[22:20:45] <Not-f817>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JeeeV
[22:20:46] <Not-f817>	 [02miraheze/ssl] 07RhinosF1 0364aac82 - Remove cert ( wiki.lostminecraftminers.org ) (#216) * Remove cert ( wiki.lostminecraftminers.org ) domain inactive * rmv wiki.lostminecraftminers.org
[22:22:49] <Not-f817>	 [02ssl] 07RhinosF1 opened pull request 03#217: rmv wiki.articmetrosystem.tk - 13https://git.io/Jeeer
[22:23:02] <RhinosF1>	 paladox: and mw-config + another ssl incoming
[22:23:31] <paladox>	 i'm leaving the mw config change for now (asking anothe oper about it first)
[22:23:32] <Voidwalker>	 Zppix, that's one way to do it, yeah
[22:23:55] <RhinosF1>	 paladox: k
[22:24:00] <Not-f817>	 [02ssl] 07RhinosF1 synchronize pull request 03#217: rmv wiki.articmetrosystem.tk - 13https://git.io/Jeeer
[22:24:22] <RhinosF1>	 paladox: looks like it's used by win rar to me
[22:24:33] <RhinosF1>	 and ^ that's the other ssl patch
[22:24:48] <Not-f817>	 [02ssl] 07paladox closed pull request 03#217: rmv wiki.articmetrosystem.tk - 13https://git.io/Jeeer
[22:24:49] <Not-f817>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/Jeeei
[22:24:51] <RhinosF1>	 paladox: puppet just crashed
[22:24:51] <Not-f817>	 [02miraheze/ssl] 07RhinosF1 0321e0817 - rmv wiki.articmetrosystem.tk (#217) * rmv wiki.articmetrosystem.tk domain not pointed and inactive on who.is * Update certs.yaml
[22:24:54] <paladox>	 done
[22:25:00] <Zppix>	 Voidwalker:  well its ready to go live if you want to update your script
[22:25:12] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.lostminecraftminers.org]
[22:25:17] <RhinosF1>	 paladox: puppet is moaning about that change saying lost file
[22:25:30] <icinga-miraheze>	 PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.lostminecraftminers.org]
[22:25:43] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.lostminecraftminers.org]
[22:25:45] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.lostminecraftminers.org]
[22:25:57] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.lostminecraftminers.org]
[22:26:13] <RhinosF1>	 icinga-miraheze: I just removed it!
[22:26:23] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.lostminecraftminers.org]
[22:26:29] <Voidwalker>	 thanks icinga-miraheze, very helpful
[22:26:39] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.lostminecraftminers.org]
[22:27:08] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[22:27:29] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.lostminecraftminers.org]
[22:27:32] <paladox>	 should recover now
[22:27:33] <Zppix>	 lol
[22:27:39] <paladox>	 (/Stage[main]/Ssl::Hiera/Ssl::Hiera::Certs[lostminecraftminers]/File[wiki.lostminecraftminers.org]) Could not evaluate: Could not retrieve information from environment production source(s) puppet:///ssl/certificates/wiki.lostminecraftminers.org.crt
[22:27:47] <paladox>	 was the error i was seeing
[22:28:13] <RhinosF1>	 paladox: yes that
[22:28:34] <paladox>	 anyways brb
[22:30:42] <Zppix>	 Voidwalker:  i guess thats one way
[22:31:23] <Voidwalker>	 yeah, I thought why enclose the whole thing when a return statement is easier :P
[22:32:40] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[22:32:47] <RhinosF1>	 here we go!
[22:33:28] <icinga-miraheze>	 RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[22:33:45] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:33:50] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[22:33:54] <Voidwalker>	 Zppix, added to the script
[22:34:35] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:34:52] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:35:29] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:35:44] <RhinosF1>	 all back
[22:37:56] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.31, 6.39, 4.27
[22:39:51] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.28, 6.38, 4.52
[22:58:37] <Zppix>	 Voidwalker:  probably because im not the best in javascript so i thought if and else never failed me
[22:59:43] <Voidwalker>	 I'm going to make it better in the future, but I got a lot on my plate to work on
[23:00:40] <Zppix>	 Voidwalker:  i cant possibly see how you can make the script better tbh
[23:01:37] <Voidwalker>	 I was thinking of adding an additional field to the interface that allows you to promote yourself if you're not currently in the group
[23:02:33] <Voidwalker>	 also considering adding a way to bypass the flooder check, but adding in another check to make sure you can actually block and stuff
[23:02:36] <Zppix>	 meh thats just quality of life, the change me/you made is more important as you don't want to flood rc with blocks
[23:02:50] <Zppix>	 why would you want to bypass flooder check
[23:03:32] <Voidwalker>	 honestly, no idea (which puts it very low on my list of priorities)
[23:09:59] <paladox>	 I’m kind of around
[23:10:03] <paladox>	 (Mobile now)
[23:13:23] <Zppix>	 paladox:  are you sure your not a square
[23:18:15] <hispano76_>	 hello
[23:18:53] <hispano76_>	 any bouncer irc services you recommend?
[23:25:12] <paladox>	 Zppix: Lol
[23:25:13] <paladox>	 Hispano76: I use xshellz