[01:41:12] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±2] 13https://git.io/fjhjY
[01:41:14] <Not-f817>	 [02miraheze/puppet] 07paladox 0376ec52a - prometheus: Configure prometheus-node-exporter config This adjust the config so that it exposes more metrics that are disabled by default. Also disables the diskstats metric if not on kvm. Config is copied from [1] and [2]. [1] https://github.com/wikimedia/puppet/blob/2daf8fbd7151244e5f81608c077c07cf4ee71df5/modules/prometheus/manifests/node_exporter.pp [2]
[01:41:14] <Not-f817>	 https://github.com/wikimedia/puppet/blob/9f292046f05cb32fce8579ec5fc7d71b9e82b642/modules/prometheus/templates/etc/default/prometheus-node-exporter-0.17.erb
[01:41:14] <ZppixBot>	 [ puppet/node_exporter.pp at 2daf8fbd7151244e5f81608c077c07cf4ee71df5 · wikimedia/puppet · GitHub ] - github.com
[01:41:16] <ZppixBot>	 [ puppet/prometheus-node-exporter-0.17.erb at 9f292046f05cb32fce8579ec5fc7d71b9e82b642 · wikimedia/puppet · GitHub ] - github.com
[01:44:39] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±2] 13https://git.io/fjhjn
[01:44:41] <Not-f817>	 [02miraheze/puppet] 07paladox 0341fc9af - prometheus: Configure prometheus-node-exporter config This adjust the config so that it exposes more metrics that are disabled by default. Also disables the diskstats metric if not on kvm. Config is copied from [1] and [2]. [1] https://github.com/wikimedia/puppet/blob/2daf8fbd7151244e5f81608c077c07cf4ee71df5/modules/prometheus/manifests/node_exporter.pp [2]
[01:44:41] <Not-f817>	 https://github.com/wikimedia/puppet/blob/9f292046f05cb32fce8579ec5fc7d71b9e82b642/modules/prometheus/templates/etc/default/prometheus-node-exporter-0.17.erb
[01:44:41] <ZppixBot>	 [ puppet/node_exporter.pp at 2daf8fbd7151244e5f81608c077c07cf4ee71df5 · wikimedia/puppet · GitHub ] - github.com
[01:44:43] <ZppixBot>	 [ puppet/prometheus-node-exporter-0.17.erb at 9f292046f05cb32fce8579ec5fc7d71b9e82b642 · wikimedia/puppet · GitHub ] - github.com
[01:59:25] <icinga-miraheze>	 PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:02:19] <icinga-miraheze>	 PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:02:28] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:02:28] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:02:36] <paladox>	 that's me
[02:02:42] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:02:42] <icinga-miraheze>	 PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:02:46] <icinga-miraheze>	 PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:02:46] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:02:49] <icinga-miraheze>	 PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:02:51] <icinga-miraheze>	 PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:02:55] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:02:57] <icinga-miraheze>	 PROBLEM - lizardfs3 Puppet on lizardfs3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:02:59] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:03:07] <icinga-miraheze>	 PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:03:25] <icinga-miraheze>	 RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
[02:03:34] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:03:40] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:03:42] <icinga-miraheze>	 PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:03:45] <icinga-miraheze>	 PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:04:06] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:04:23] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±2] 13https://git.io/fjhjQ
[02:04:24] <Not-f817>	 [02miraheze/puppet] 07paladox 030d3cc77 - prometheus: Configure prometheus-node-exporter config This adjust the config so that it exposes more metrics that are disabled by default. Also disables the diskstats metric if not on kvm. Config is copied from [1] and [2]. [1] https://github.com/wikimedia/puppet/blob/2daf8fbd7151244e5f81608c077c07cf4ee71df5/modules/prometheus/manifests/node_exporter.pp [2]
[02:04:24] <Not-f817>	 https://github.com/wikimedia/puppet/blob/9f292046f05cb32fce8579ec5fc7d71b9e82b642/modules/prometheus/templates/etc/default/prometheus-node-exporter-0.17.erb
[02:04:24] <ZppixBot>	 [ puppet/node_exporter.pp at 2daf8fbd7151244e5f81608c077c07cf4ee71df5 · wikimedia/puppet · GitHub ] - github.com
[02:04:26] <ZppixBot>	 [ puppet/prometheus-node-exporter-0.17.erb at 9f292046f05cb32fce8579ec5fc7d71b9e82b642 · wikimedia/puppet · GitHub ] - github.com
[02:06:28] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures
[02:07:42] <icinga-miraheze>	 RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[02:07:45] <icinga-miraheze>	 RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[02:08:06] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[02:08:42] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[02:08:42] <icinga-miraheze>	 RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:08:45] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[02:08:46] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:08:49] <icinga-miraheze>	 RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:08:51] <icinga-miraheze>	 RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:08:58] <icinga-miraheze>	 RECOVERY - lizardfs3 Puppet on lizardfs3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:09:07] <icinga-miraheze>	 RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:09:34] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:09:40] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:10:28] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[02:10:42] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:12:19] <icinga-miraheze>	 RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[02:12:55] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:13:04] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:18:50] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±2] 13https://git.io/fjjeJ
[02:18:52] <Not-f817>	 [02miraheze/puppet] 07paladox 031fca467 - prometheus: Configure prometheus-node-exporter config This adjust the config so that it exposes more metrics that are disabled by default. Also disables the diskstats metric if not on kvm. Config is copied from [1] and [2]. [1] https://github.com/wikimedia/puppet/blob/2daf8fbd7151244e5f81608c077c07cf4ee71df5/modules/prometheus/manifests/node_exporter.pp [2]
[02:18:52] <Not-f817>	 https://github.com/wikimedia/puppet/blob/9f292046f05cb32fce8579ec5fc7d71b9e82b642/modules/prometheus/templates/etc/default/prometheus-node-exporter-0.17.erb
[02:18:53] <ZppixBot>	 [ puppet/node_exporter.pp at 2daf8fbd7151244e5f81608c077c07cf4ee71df5 · wikimedia/puppet · GitHub ] - github.com
[02:18:55] <ZppixBot>	 [ puppet/prometheus-node-exporter-0.17.erb at 9f292046f05cb32fce8579ec5fc7d71b9e82b642 · wikimedia/puppet · GitHub ] - github.com
[02:22:54] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±2] 13https://git.io/fjjeT
[02:22:56] <Not-f817>	 [02miraheze/puppet] 07paladox 0326a79d3 - prometheus: Configure prometheus-node-exporter config This adjust the config so that it exposes more metrics that are disabled by default. Also disables the diskstats metric if not on kvm. Config is copied from [1] and [2]. [1] https://github.com/wikimedia/puppet/blob/2daf8fbd7151244e5f81608c077c07cf4ee71df5/modules/prometheus/manifests/node_exporter.pp [2]
[02:22:56] <Not-f817>	 https://github.com/wikimedia/puppet/blob/9f292046f05cb32fce8579ec5fc7d71b9e82b642/modules/prometheus/templates/etc/default/prometheus-node-exporter-0.17.erb
[02:22:56] <ZppixBot>	 [ puppet/node_exporter.pp at 2daf8fbd7151244e5f81608c077c07cf4ee71df5 · wikimedia/puppet · GitHub ] - github.com
[02:22:58] <ZppixBot>	 [ puppet/prometheus-node-exporter-0.17.erb at 9f292046f05cb32fce8579ec5fc7d71b9e82b642 · wikimedia/puppet · GitHub ] - github.com
[02:24:12] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±2] 13https://git.io/fjjek
[02:24:14] <Not-f817>	 [02miraheze/puppet] 07paladox 03e2fa91f - prometheus: Configure prometheus-node-exporter config This adjust the config so that it exposes more metrics that are disabled by default. Also disables the diskstats metric if not on kvm. Config is copied from [1] and [2]. [1] https://github.com/wikimedia/puppet/blob/2daf8fbd7151244e5f81608c077c07cf4ee71df5/modules/prometheus/manifests/node_exporter.pp [2]
[02:24:14] <Not-f817>	 https://github.com/wikimedia/puppet/blob/9f292046f05cb32fce8579ec5fc7d71b9e82b642/modules/prometheus/templates/etc/default/prometheus-node-exporter-0.17.erb
[02:24:14] <ZppixBot>	 [ puppet/node_exporter.pp at 2daf8fbd7151244e5f81608c077c07cf4ee71df5 · wikimedia/puppet · GitHub ] - github.com
[02:24:16] <ZppixBot>	 [ puppet/prometheus-node-exporter-0.17.erb at 9f292046f05cb32fce8579ec5fc7d71b9e82b642 · wikimedia/puppet · GitHub ] - github.com
[02:26:35] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±2] 13https://git.io/fjjeq
[02:26:36] <Not-f817>	 [02miraheze/puppet] 07paladox 03baac95e - prometheus: Configure prometheus-node-exporter config This adjust the config so that it exposes more metrics that are disabled by default. Also disables the diskstats metric if not on kvm. Config is copied from [1] and [2]. [1] https://github.com/wikimedia/puppet/blob/2daf8fbd7151244e5f81608c077c07cf4ee71df5/modules/prometheus/manifests/node_exporter.pp [2]
[02:26:36] <Not-f817>	 https://github.com/wikimedia/puppet/blob/9f292046f05cb32fce8579ec5fc7d71b9e82b642/modules/prometheus/templates/etc/default/prometheus-node-exporter-0.17.erb
[02:26:37] <ZppixBot>	 [ puppet/node_exporter.pp at 2daf8fbd7151244e5f81608c077c07cf4ee71df5 · wikimedia/puppet · GitHub ] - github.com
[02:26:39] <ZppixBot>	 [ puppet/prometheus-node-exporter-0.17.erb at 9f292046f05cb32fce8579ec5fc7d71b9e82b642 · wikimedia/puppet · GitHub ] - github.com
[06:26:29] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2956 MB (12% inode=94%);
[12:36:28] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members]
[12:42:28] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[13:03:57] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 12.84, 7.94, 5.09
[13:15:56] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 4.31, 6.93, 6.69
[13:17:56] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.16, 6.22, 6.45
[14:05:25] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%);
[14:34:47] <RhinosF1>	 Voidwalker: see my userpage - is that css any useful?
[14:36:14] <MH-Discord>	 <Zppix> @RhinosF1 i love how you got threatened to be blocked by an IP
[14:36:19] <Voidwalker>	 I don't really think so
[14:37:06] <RhinosF1>	 Voidwalker: I'll mark it as not done then
[14:37:30] <RhinosF1>	 @Zppix hmm, they did say on another wiki so unless they were logged out
[14:38:21] <Voidwalker>	 it is a proxy, so I would be fairly willing to discard it as trolling
[14:38:39] <RhinosF1>	 Voidwalker: ah cool
[14:40:16] <RhinosF1>	 Voidwalker: no, I knew I recognised them. It's System Operator!!
[14:40:21] <MH-Discord>	 <Zppix> I still left a passive-aggressive reply
[14:40:26] <RhinosF1>	 JohnLewis: ^
[14:40:37] <MH-Discord>	 <Zppix> Is system op LTA or something?
[14:40:48] <MH-Discord>	 <Zppix> (Needs to really catch up)
[14:41:03] <icinga-miraheze>	 PROBLEM - reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: connect to address reviwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[14:41:16] <Voidwalker>	 I'll have to let someone else handle that for now
[14:41:21] <RhinosF1>	 @Zppix Globally locked user (not quite LTA yet though)
[14:41:38] <MH-Discord>	 <Zppix> Voidwalker im around what needs handled?
[14:41:48] <RhinosF1>	 They were blocked by SPF|Cloud and then I believe created MetaBot (which was ToU blocked)
[14:41:54] <MH-Discord>	 <Zppix> Ok
[14:42:12] <RhinosF1>	 @Zppix I'll get John to check for suspected ToU issue
[14:42:29] <JohnLewis>	 check on what and whom?
[14:42:32] <MH-Discord>	 <Zppix> Ill check the ASN of that ip and see if i can get any rangeblocks
[14:42:50] <Voidwalker>	 the IP itself has already been rangeblocked as a proxy FYI
[14:42:57] <icinga-miraheze>	 RECOVERY - reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.
[14:43:23] <RhinosF1>	 JohnLewis: My talk page (the IP as evading a ToU block)
[14:43:32] <JohnLewis>	 {{nocomment}}
[14:44:15] <MH-Discord>	 <Zppix> Voidwalker, is it safe to rangeblock all ips ran by that asn?
[14:45:04] <JohnLewis>	 who owns the ASN?
[14:45:12] <MH-Discord>	 <Zppix> FranTec solutions
[14:45:14] <MH-Discord>	 <Zppix> AS53667 FranTech Solutions
[14:45:56] <JohnLewis>	 it's a web host so presumably yes
[14:46:30] <MH-Discord>	 <Zppix> Ok
[14:46:33] <MH-Discord>	 <Zppix> Doing it now
[14:55:04] <MH-Discord>	 <Zppix> (I did majority of it ill continue it when i can switch to a pc
[16:02:47] <icinga-miraheze>	 PROBLEM - reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: connect to address reviwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[16:04:43] <icinga-miraheze>	 RECOVERY - reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.
[16:19:16] <icinga-miraheze>	 PROBLEM - reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: connect to address reviwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[16:25:24] <RhinosF1>	 revi: ^
[16:27:51] <RhinosF1>	 PLEASE SPEAK TO revi BEFORE REMOVING
[16:28:28] <JohnLewis>	 hm?
[16:29:16] <revi>	 well I just forgot to renew the domain
[16:29:19] <revi>	 nice reminder
[16:29:42] <revi>	 I'll do it tomorrow morning, too sleepy now. (== you can suppress it for the time being)
[16:29:45] <RhinosF1>	 JohnLewis: revis domaine Has expired, I'm saying that because I left them a message so no one removes it until they responded but they have now so it's moot
[16:30:47] <icinga-miraheze>	 RECOVERY - reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.
[16:31:06] <JohnLewis>	 RhinosF1: either way, it's standard practise both for security and housekeeping to do so. The discretion lies with anyone within Operations and they have full freedom to do as they wish within reason regardless of what a user says with the status of things
[16:31:37] <RhinosF1>	 JohnLewis: well true
[16:34:44] <icinga-miraheze>	 PROBLEM - reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: connect to address reviwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[16:40:43] <icinga-miraheze>	 RECOVERY - reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.
[16:47:44] <icinga-miraheze>	 PROBLEM - reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: connect to address reviwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[16:49:38] <icinga-miraheze>	 RECOVERY - reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.
[16:55:09] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjj3P
[16:55:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 03e467885 - BOT: Updating services config for wikis
[17:05:02] <icinga-miraheze>	 PROBLEM - reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: connect to address reviwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[17:08:52] <icinga-miraheze>	 RECOVERY - reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.
[17:29:35] <icinga-miraheze>	 PROBLEM - reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: connect to address reviwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[17:31:16] <icinga-miraheze>	 PROBLEM - www.reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: connect to address www.reviwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[17:33:11] <icinga-miraheze>	 RECOVERY - www.reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.
[17:41:49] <icinga-miraheze>	 PROBLEM - www.reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: connect to address www.reviwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[17:43:06] <MH-Discord>	 <Zppix> @Void I respect the amount of work it took to lock all 300+ accts the other day
[17:43:37] <MH-Discord>	 <Zppix> Why they dont have a ip equivalent of lock multiple accounts is beyond me
[17:43:43] <icinga-miraheze>	 RECOVERY - www.reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.
[17:45:13] <Voidwalker>	 hah, now that you mention it, a mass blocking feature could be pretty handy
[17:45:41] <Voidwalker>	 but, for now, that's what my script is for
[17:45:56] <MH-Discord>	 <Zppix> Voidwalker, is the script linked on cvtwiki?
[17:46:20] <Voidwalker>	 I should actually see about turning that into an actual interface though, because that would look so much better
[17:46:29] <MH-Discord>	 <Zppix> Pls
[17:46:56] <MH-Discord>	 <Zppix> Voidwalker, if you do that ill force labster to pay you 😛
[17:47:34] <Voidwalker>	 https://cvt.miraheze.org/wiki/User:Void/massGBlock
[17:47:34] <ZppixBot>	 [ Permission error - Miraheze CVT Wiki ] - cvt.miraheze.org
[17:47:34] <icinga-miraheze>	 PROBLEM - www.reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: connect to address www.reviwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[17:47:42] <MH-Discord>	 <Zppix> Thanks
[17:52:31] <Voidwalker>	 y'know what, I'll turn that into an interface soon
[17:53:22] <MH-Discord>	 <Zppix> I love you (no homo)
[17:53:55] <Voidwalker>	 :)
[17:55:59] <MH-Discord>	 <Zppix> Voidwalker although i wish it would let me set the reason without changing the script
[17:56:40] <Voidwalker>	 yeah, that's one of the things I intend to fix
[17:57:05] <Reception123>	 someone needs to make a "Script Master" barnstar and give it to Voidwalker :)
[18:01:00] <icinga-miraheze>	 RECOVERY - www.reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.
[18:01:02] <Voidwalker>	 I've also noticed that some ASN lists give you consecutive or overlapping IP ranges, and I built a python script to consolidate a list of IP ranges into the smallest list that still covers all the addresses
[18:01:27] <Voidwalker>	 now I'm going to convert that script to JS and put it in my massGBlock script
[18:02:40] <Zppix>	 Voidwalker:  You know I technically should be able to determine the ip ranges by hand, since thats covered by my CCENT but yeah umm lets just say unless i have google dont ask me to find an ip range 
[18:03:26] <Voidwalker>	 yeah, why do it by hand, when you can plug it into a script and be done with it? :P
[18:03:52] <Zppix>	 Voidwalker: well because thats what im supposed to be able to do
[18:04:51] <icinga-miraheze>	 PROBLEM - www.reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: connect to address www.reviwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[18:08:41] <icinga-miraheze>	 RECOVERY - www.reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.
[18:21:08] <icinga-miraheze>	 PROBLEM - www.reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: connect to address www.reviwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[18:27:34] <Voidwalker>	 JS in mediawiki doesn't allow the use of classes
[18:27:54] <Voidwalker>	 properly, at least
[18:30:11] <Zppix>	 lol
[18:30:42] <icinga-miraheze>	 RECOVERY - www.reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.
[18:33:52] <Voidwalker>	 it's weird though, because there are things that act like classes elsewhere
[18:34:34] <icinga-miraheze>	 PROBLEM - www.reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: connect to address www.reviwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[18:35:09] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjjGP
[18:35:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 03e702746 - BOT: Updating services config for wikis
[18:37:06] <Voidwalker>	 oh, wait a moment
[18:38:31] <icinga-miraheze>	 RECOVERY - www.reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.
[18:39:06] <Voidwalker>	 it just uses a whacky version that doesn't look very pretty
[18:42:31] <icinga-miraheze>	 PROBLEM - www.reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: connect to address www.reviwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[19:36:14] <RhinosF1>	 Reception123: technical barnstar would suit Voidwalker best - or maybe defender of the wiki
[19:39:57] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.08, 5.76, 4.23
[19:41:18] <Reception123>	 RhinosF1: yeah funny enough one of the two non-sysadmin stewards gets that one :D
[19:41:56] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.87, 6.52, 4.71
[19:42:37] <Zppix>	 lol
[19:43:50] <RhinosF1>	 Reception123: yeah true
[19:43:56] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.29, 5.59, 4.58
[19:57:56] <RhinosF1>	 Reception123, paladox, Voidwalker: feel like doing https://phabricator.miraheze.org/T4696
[19:57:57] <ZppixBot>	 [ ⚓ T4696 I want to be able to use the Restricted features on Special:ManageWiki ] - phabricator.miraheze.org
[20:02:26] <paladox>	 i can do it later
[20:04:41] <RhinosF1>	 paladox: fantastic
[20:11:33] <paladox>	 !log apt-get upgrade on misc1 and puppet1
[20:11:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:23:34] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:23:41] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:24:00] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:24:06] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:24:12] <Zppix>	 paladox:  ^
[20:24:20] <paladox>	 that's me
[20:24:28] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:24:29] <paladox>	 and only a puppet failure, will recover in the next run
[20:24:32] <Zppix>	 paladox:  Of course your breaking everything lol
[20:24:36] <paladox>	 :P
[20:24:42] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:24:46] <icinga-miraheze>	 PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:24:54] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 2 minutes ago with 5 failures. Failed resources (up to 3 shown)
[20:26:29] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 169 failures. Last run 3 minutes ago with 169 failures. Failed resources (up to 3 shown): File[wiki.openhatch.org],File[wiki.openhatch.org_private],File[wiki.doverdistrictscouts.co.uk],File[wiki.doverdistrictscouts.co.uk_private]
[20:33:35] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[20:34:00] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[20:34:06] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures
[20:34:28] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:34:42] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[20:34:46] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:35:09] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:35:40] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:36:28] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:46:36] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjjnH
[20:46:37] <Not-f817>	 [02miraheze/puppet] 07paladox 03c9ba779 - lizardfs: Remove lizardfs[123]
[20:46:39] <Not-f817>	 [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS
[20:46:40] <Not-f817>	 [02puppet] 07paladox opened pull request 03#1078: lizardfs: Remove lizardfs[123] - 13https://git.io/fjjnQ
[20:47:06] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjjn7
[20:47:07] <Not-f817>	 [02miraheze/puppet] 07paladox 031095d83 - Update storage_firewall.yaml
[20:47:09] <Not-f817>	 [02puppet] 07paladox synchronize pull request 03#1078: lizardfs: Remove lizardfs[123] - 13https://git.io/fjjnQ
[20:47:57] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjjnF
[20:47:59] <Not-f817>	 [02miraheze/puppet] 07paladox 030237a98 - Update site.pp
[20:48:00] <Not-f817>	 [02puppet] 07paladox synchronize pull request 03#1078: lizardfs: Remove lizardfs[123] - 13https://git.io/fjjnQ
[20:48:11] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/fjjnb
[20:48:13] <Not-f817>	 [02miraheze/puppet] 07paladox 036365bea - lizardfs: Remove lizardfs[123] (#1078) * lizardfs: Remove lizardfs[123] * Update storage_firewall.yaml * Update site.pp
[20:48:14] <Not-f817>	 [02puppet] 07paladox closed pull request 03#1078: lizardfs: Remove lizardfs[123] - 13https://git.io/fjjnQ
[20:49:04] <Not-f817>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4
[20:49:06] <Not-f817>	 [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS
[20:49:23] <paladox>	 !log stop lizardfs-chunkserver on lizardfs[123]
[20:49:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:51:28] <icinga-miraheze>	 PROBLEM - lizardfs2 Lizardfs Chunkserver Port on lizardfs2 is CRITICAL: connect to address 81.4.124.61 and port 9422: Connection refused
[20:51:50] <icinga-miraheze>	 PROBLEM - lizardfs1 Lizardfs Chunkserver Port on lizardfs1 is CRITICAL: connect to address 81.4.101.157 and port 9422: Connection refused
[20:53:56] <paladox>	 !log restarting lizardfs-chunkserver on lizardfs4
[20:54:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:57:03] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3
[20:57:13] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[20:57:16] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[20:58:24] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1
[20:58:44] <paladox>	 !log restarting lizardfs-chunkserver on lizardfs5
[20:59:03] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[21:00:28] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[21:01:53] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[21:02:59] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[21:03:22] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[21:03:50] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24500 bytes in 0.009 second response time
[21:04:13] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[21:04:25] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[21:24:21] <paladox>	 !log revoke puppet cert for lizardfs[123]
[21:24:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:29:58] <paladox>	 !log revoke firewall for lizardfs[123] on misc3
[21:30:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:31:44] <Not-f817>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjjco
[21:31:45] <Not-f817>	 [02miraheze/dns] 07paladox 037dd9c22 - Remove lizardfs[123] from dns
[21:33:08] <icinga-miraheze>	 PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:33:53] <icinga-miraheze>	 PROBLEM - lizardfs3 Puppet on lizardfs3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:34:07] <icinga-miraheze>	 PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:37:59] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Static on bacula1 is CRITICAL: CRITICAL: Timeout or unknown client: lizardfs4-fd
[21:39:02] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjjcy
[21:39:03] <Not-f817>	 [02miraheze/puppet] 07paladox 036e8a505 - Fix lizardfs4 address
[22:09:42] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:52:18] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[22:52:34] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[22:52:38] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[22:53:36] <icinga-miraheze>	 PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[22:53:38] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2
[22:53:40] <RhinosF1>	 paladox: ^ slow loading as well
[22:53:51] <paladox>	 checking.
[22:53:58] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3
[22:54:33] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3
[22:54:36] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24516 bytes in 1.609 second response time
[22:55:35] <icinga-miraheze>	 RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24500 bytes in 0.392 second response time
[22:55:58] <icinga-miraheze>	 PROBLEM - unrecnations.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'unrecnations.wiki' expires in 15 day(s) (Sat 21 Sep 2019 10:53:21 PM GMT +0000).
[22:56:11] <Not-f817>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjjCb
[22:56:13] <Not-f817>	 [02miraheze/ssl] 07MirahezeSSLBot 03b188029 - Bot: Update SSL cert for unrecnations.wiki
[22:56:58] <RhinosF1>	 hope u get it sorted paladox, off to sleep
[22:57:02] <paladox>	 !log sudo service lizardfs-master restart on misc3
[22:57:38] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[22:57:58] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[22:58:19] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[22:58:21] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[22:58:33] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[23:03:49] <icinga-miraheze>	 RECOVERY - unrecnations.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'unrecnations.wiki' will expire on Wed 04 Dec 2019 09:56:04 PM GMT +0000.