[00:01:20] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqUSR
[00:01:22] <Not-71a9>	 [02miraheze/puppet] 07paladox 0314dcbe2 - Update init.pp
[00:01:23] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1677: varnish: Convert to systemd::service and also fix init script - 13https://git.io/JqUV2
[00:01:30] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JqUS0
[00:01:31] <Not-71a9>	 [02miraheze/puppet] 07paladox 030adccb8 - Delete varnish-override.conf.erb
[00:01:33] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1677: varnish: Convert to systemd::service and also fix init script - 13https://git.io/JqUV2
[00:05:36] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.42, 6.63, 4.49
[00:07:35] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.67, 5.80, 4.45
[01:26:49] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqU7U
[01:26:50] <Not-71a9>	 [02miraheze/puppet] 07paladox 03b18a419 - Update init.pp
[01:26:52] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1677: varnish: Convert to systemd::service and also fix init script - 13https://git.io/JqUV2
[01:28:00] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/JqU7k
[01:28:01] <Not-71a9>	 [02miraheze/puppet] 07paladox 037c3a1eb - Create reload-vcl.py
[01:28:03] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1677: varnish: Convert to systemd::service and also fix init script - 13https://git.io/JqUV2
[01:28:14] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqU7I
[01:28:15] <Not-71a9>	 [02miraheze/puppet] 07paladox 033db4778 - Update init.pp
[01:28:17] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1677: varnish: Convert to systemd::service and also fix init script - 13https://git.io/JqUV2
[01:31:06] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqU7n
[01:31:07] <Not-71a9>	 [02miraheze/puppet] 07paladox 036140813 - Update varnish.systemd.erb
[01:31:09] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1677: varnish: Convert to systemd::service and also fix init script - 13https://git.io/JqUV2
[01:36:00] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqU72
[01:36:01] <Not-71a9>	 [02miraheze/puppet] 07paladox 03f76a8dd - Update varnish.systemd.erb
[01:36:03] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1677: varnish: Convert to systemd::service and also fix init script - 13https://git.io/JqUV2
[01:36:28] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.236.219/cpweb, 2001:41d0:800:178a::5/cpweb
[01:36:32] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.236.219/cpweb, 2001:41d0:800:178a::5/cpweb
[01:36:52] <icinga-miraheze>	 PROBLEM - cp10 Varnish Backends on cp10 is WARNING: No backends detected. If this is an error, see readme.txt
[01:37:10] <icinga-miraheze>	 PROBLEM - cp10 HTTPS on cp10 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.007 second response time
[01:38:30] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[01:38:32] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[01:38:59] <icinga-miraheze>	 PROBLEM - cp10 Puppet on cp10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[varnish]
[01:39:11] <icinga-miraheze>	 RECOVERY - cp10 HTTPS on cp10 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2196 bytes in 0.008 second response time
[01:40:58] <icinga-miraheze>	 RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures
[01:49:18] <paladox>	 !log depool cp10 && 11
[01:49:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:49:40] <Not-71a9>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqU5U
[01:49:42] <Not-71a9>	 [02miraheze/dns] 07paladox 030175d38 - Depool cp10/11
[01:53:26] <icinga-miraheze>	 RECOVERY - cp11 Puppet on cp11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[01:55:19] <icinga-miraheze>	 PROBLEM - cp11 Varnish Backends on cp11 is WARNING: No backends detected. If this is an error, see readme.txt
[01:55:52] <Not-71a9>	 [02puppet] 07paladox edited pull request 03#1677: varnish: Convert to systemd::service and also fix init script - 13https://git.io/JqUV2
[01:58:52] <Not-71a9>	 [02puppet] 07paladox closed pull request 03#1677: varnish: Convert to systemd::service and also fix init script - 13https://git.io/JqUV2
[01:58:54] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-2/±1] 13https://git.io/JqU5M
[01:58:55] <Not-71a9>	 [02miraheze/puppet] 07paladox 03da00b37 - varnish: Convert to systemd::service and also fix init script (#1677)
[01:58:57] <Not-71a9>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[01:58:58] <Not-71a9>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[01:59:46] <Not-71a9>	 [02miraheze/dns] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JqU5D
[01:59:48] <Not-71a9>	 [02miraheze/dns] 07paladox 036589c64 - Revert "Depool cp10/11"
[01:59:49] <Not-71a9>	 [02miraheze/dns] 07paladox 03dda8946 - Depool cp12
[02:01:23] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: CRITICAL: Timeout or unknown client: db11-fd
[02:03:09] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Databases db13 on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[02:04:17] <icinga-miraheze>	 PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:04:23] <icinga-miraheze>	 PROBLEM - dbbackup2 MariaDB on dbbackup2 is UNKNOWN: check_mysql: Invalid hostname/address - dbbackup2.miraheze.orgUsage: check_mysql [-d database] [-H host] [-P port] [-s socket] [-u user] [-p password] [-S] [-l] [-a cert] [-k key] [-C ca-cert] [-D ca-dir] [-L ciphers] [-f optfile] [-g group]
[02:04:43] <Universal_Omega>	 paladox: 502 on all wikis.
[02:04:55] <Not-71a9>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqUdU
[02:04:56] <Not-71a9>	 [02miraheze/dns] 07paladox 031e8c1ed - Revert "Depool cp12"
[02:05:34] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Databases db11 on bacula2 is WARNING: WARNING: Full, 541500 files, 40.45GB, 2021-02-07 01:25:00 (3.9 weeks ago)
[02:05:44] <paladox>	 should be back
[02:05:50] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:800:178a::5/cpweb, 2001:41d0:800:1bbd::4/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb
[02:05:54] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb
[02:06:00] <Universal_Omega>	 Not for me. Paladox
[02:06:08] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Databases db13 on bacula2 is WARNING: WARNING: Full, 207255 files, 80.26GB, 2021-02-07 04:21:00 (3.8 weeks ago)
[02:06:12] <paladox>	 Works for me.
[02:06:26] <Universal_Omega>	 hmm. Still down when trying to visit metax for me.
[02:06:27] <icinga-miraheze>	 PROBLEM - cp12 HTTPS on cp12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.306 second response time
[02:06:34] <icinga-miraheze>	 PROBLEM - dbbackup2 MariaDB on dbbackup2 is CRITICAL: Host 'mon2.miraheze.org' is not allowed to connect to this MariaDB server
[02:06:40] <icinga-miraheze>	 PROBLEM - cp12 Varnish Backends on cp12 is WARNING: No backends detected. If this is an error, see readme.txt
[02:06:43] <icinga-miraheze>	 PROBLEM - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is CRITICAL: CRITICAL - NGINX Error Rate is 99%
[02:06:46] <icinga-miraheze>	 PROBLEM - phab2 Puppet on phab2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:06:49] <icinga-miraheze>	 PROBLEM - bebaskanpengetahuan.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bebaskanpengetahuan.id could not be found
[02:06:52] <Universal_Omega>	 paladox
[02:07:02] <icinga-miraheze>	 PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:07:09] <icinga-miraheze>	 PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:07:24] <paladox>	 I can load https://meta.miraheze.org/wiki/Miraheze
[02:07:25] <MirahezeBot>	 [ Meta ] - meta.miraheze.org
[02:07:31] <icinga-miraheze>	 PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:07:32] <icinga-miraheze>	 PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:07:33] <dmehus>	 !sre Users are reporting 502 bad gateways. Could this be related to the very rapid mass editing by DeciduousWater534?
[02:07:33] <paladox>	 oh i see why
[02:07:34] <icinga-miraheze>	 PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:07:37] <icinga-miraheze>	 PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:07:46] <icinga-miraheze>	 PROBLEM - wiki.sdbx.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sdbx.dev could not be found
[02:07:51] <paladox>	 grr
[02:07:52] <icinga-miraheze>	 PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:08:20] <dmehus>	 Hrm, that's an interesting error. "Failed to apply catalog, zero resources tracked by Puppet."
[02:08:28] <dmehus>	 What's that about, Universal_Omega?
[02:09:26] <paladox>	 does it load now?
[02:09:30] <Universal_Omega>	 dmehus: I don't know. I am mobile right now, just saw the 502s and wandered if paladox's changes caused it.
[02:09:47] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[02:09:49] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[02:09:52] <Universal_Omega>	 paladox: yes. Thanks!
[02:09:58] <paladox>	 thank you for reporting!
[02:10:22] <icinga-miraheze>	 RECOVERY - cp12 HTTPS on cp12 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2196 bytes in 0.431 second response time
[02:10:26] <icinga-miraheze>	 RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:10:43] <icinga-miraheze>	 RECOVERY - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is OK: OK - NGINX Error Rate is 4%
[02:10:50] <Not-71a9>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqUdB
[02:10:51] <Not-71a9>	 [02miraheze/dns] 07paladox 03dc8fa0b - Depool sg
[02:11:03] <dmehus>	 Universal_Omega, ah okay. Seems like it was related to paladox' changes then, and unrelated to the mass editing
[02:12:09] <Universal_Omega>	 dmehus: yeah, it was all wikis though also, so couldn't have been something done on a specific wiki.
[02:12:14] <Universal_Omega>	 paladox: no problem
[02:12:25] <Universal_Omega>	 Thanks for fixing. 
[02:13:36] <dmehus>	 Universal_Omega, well, the mass editing was occurring on multiple wikis, but yeah I guess
[02:13:39] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[02:13:42] <icinga-miraheze>	 RECOVERY - bebaskanpengetahuan.id - reverse DNS on sslhost is OK: rDNS OK - bebaskanpengetahuan.id reverse DNS resolves to cp11.miraheze.org
[02:13:46] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb
[02:13:47] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb
[02:13:50] <icinga-miraheze>	 PROBLEM - cp3 HTTPS on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 1.017 second response time
[02:14:03] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is WARNING: No backends detected. If this is an error, see readme.txt
[02:14:05] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 100%
[02:14:34] <icinga-miraheze>	 RECOVERY - wiki.sdbx.dev - reverse DNS on sslhost is OK: rDNS OK - wiki.sdbx.dev reverse DNS resolves to cp11.miraheze.org
[02:15:04] <Not-71a9>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqUdo
[02:15:06] <Not-71a9>	 [02miraheze/dns] 07paladox 0385ec499 - Revert "Depool sg"
[02:15:45] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[02:15:46] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[02:15:47] <icinga-miraheze>	 RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2194 bytes in 1.266 second response time
[02:16:00] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 5%
[02:16:11] <Zppix>	 dmehus:  a high edit volume would affect the DBS more likely than the actual mw host
[02:29:00] <icinga-miraheze>	 RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[02:29:38] <icinga-miraheze>	 RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[02:29:52] <icinga-miraheze>	 RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:30:38] <icinga-miraheze>	 RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:31:48] <icinga-miraheze>	 RECOVERY - phab2 Puppet on phab2 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[02:32:19] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqUFt
[02:32:21] <Not-71a9>	 [02miraheze/puppet] 07paladox 0338b4ce3 - matomo: Update to 4.2.1
[02:34:51] <icinga-miraheze>	 RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[02:35:00] <icinga-miraheze>	 RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:35:14] <icinga-miraheze>	 RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:56:47] <dmehus>	 Zppix, ah, true, good point yeah
[04:05:28] <icinga-miraheze>	 PROBLEM - bacula2 APT on bacula2 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected.
[04:07:28] <icinga-miraheze>	 PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates).
[05:26:14] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.83, 3.17, 2.51
[05:28:11] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.59, 3.63, 2.75
[05:32:08] <icinga-miraheze>	 RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.22, 3.27, 2.85
[06:03:31] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.82, 3.22, 2.81
[06:09:29] <icinga-miraheze>	 RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.23, 3.01, 2.91
[06:21:28] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.39, 3.58, 3.07
[06:23:28] <icinga-miraheze>	 RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.13, 3.14, 2.97
[06:45:29] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.29, 3.53, 2.90
[06:47:28] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.97, 3.75, 3.07
[06:51:28] <icinga-miraheze>	 RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 1.99, 3.00, 2.92
[07:23:28] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.85, 3.20, 2.64
[07:25:28] <icinga-miraheze>	 RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 3.17, 3.22, 2.72
[07:45:28] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.92, 2.98, 2.50
[07:47:28] <icinga-miraheze>	 RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 3.39, 3.15, 2.63
[08:29:28] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.13, 3.02, 2.50
[08:31:28] <icinga-miraheze>	 RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.66, 2.98, 2.55
[08:45:29] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.92, 3.22, 2.67
[08:51:29] <icinga-miraheze>	 RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.17, 3.16, 2.89
[09:30:02] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.17, 3.24, 2.81
[09:31:59] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.90, 3.49, 2.96
[09:33:57] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.07, 3.69, 3.10
[09:35:54] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.58, 3.68, 3.17
[09:37:52] <icinga-miraheze>	 RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 1.71, 2.92, 2.95
[09:51:33] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.28, 3.32, 2.88
[09:53:32] <icinga-miraheze>	 PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.98, 3.50, 3.00
[09:55:31] <icinga-miraheze>	 RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 3.14, 3.32, 3.00
[15:22:39] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqTBF
[15:22:41] <Not-71a9>	 [02miraheze/puppet] 07paladox 03c5c0e34 - varnish: Upgrade bootstrap for error page
[15:23:30] <Not-71a9>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqTBp
[15:23:31] <Not-71a9>	 [02miraheze/mw-config] 07paladox 03a987d5b - Upgrade bootstrap to 4.6.0
[15:24:35] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[15:27:03] <Not-71a9>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqTRf
[15:27:04] <Not-71a9>	 [02miraheze/mw-config] 07paladox 03327287f - Fix bootstrap url
[15:27:54] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqTRJ
[15:27:56] <Not-71a9>	 [02miraheze/puppet] 07paladox 03f524e6d - Fix bootstrap url
[15:28:05] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[17:01:24] <Reception123>	 paladox: JohnLewis wanted to ask for a final opinion regarding https://phabricator.miraheze.org/T6771 . The general opinion in the MW team is that it's too risky to allow, but Infra team thoughts would be appreciated :)
[17:01:25] <MirahezeBot>	 [ ⚓ T6771 Request to allow archive.org embedding for tuscriaturas wiki ] - phabricator.miraheze.org
[17:05:38] <paladox>	 I think we're at the stage where we need to be more conservative with the CSP whitelist.
[17:06:29] <paladox>	 the header is large and i remember JohnLewis saying after he introduced it about the size.
[17:12:49] <Zppix>	 Is it possible to allow it on just that wiki?
[17:15:22] <RhinosF1>	 Universal_Omega was gonna look at that
[17:18:37] <dmehus>	 Why not create an a global abuse filter to limit embedding of archive.org files only to certain sound recordings, video recordings, and PDF files; all other file types would be disallowed?
[17:24:56] <JohnLewis>	 It is possible, if you create another stanza but we've move the CSP to varnish, which doesn't do per-server name now
[17:28:52] <dmehus>	 JohnLewis, ah, thanks. What about the suggestion above to whitelist the domain, but not do so until a global abuse filter is created? We could change the task's status to "low" or "stalled" until the abuse filter is created
[17:33:19] <JohnLewis>	 If we can't whitelist the whole domain without issue, it won't be whitelisted at all - as to do so would both require dependency on the filter to work and the filter to be effective at attempts to circumvent it
[17:36:07] <dmehus>	 JohnLewis, okay, the concern I have, though, is there's been no quantitative or qualitative evidence presented that make the case against whitelisting. That is to say, there's been no evidence presented that the Internet Archive would not be safe to whitelist; it's all been gut feelings and conjecture
[17:40:24] <dmehus>	 I would also add that `archive.org` is whitelisted, in whatever whitelist they use, on some or all of the Wikimedia wikis, as there is a "reflinks" reference/citation checking user script/gadget that works on the Wikimedia wikis but does not work on Miraheze as the domain has not been whitelisted
[17:48:43] <JohnLewis>	 I don't think Wikimedia has a CSP on main wikis, I'm aware MediaWiki only recently gained support for a CSP
[17:49:15] <MH-Discord>	 <DarkMatterMan4500> CSP?
[17:49:33] <JohnLewis>	 Content Security Policy
[17:49:55] <MH-Discord>	 <DarkMatterMan4500> Oh.
[17:50:29] <JohnLewis>	 But, without an actual policy in place, if majority if SRE seem to not like the idea of adding archive.org, it makes it harder to justify adding it
[18:06:45] <Reception123>	 well the main issue is: is there really a substantial risk if we allow it?
[18:07:08] <Reception123>	 I do think we should at least in the future have a clearer criteria for approving CSP whitelists, as the criteria is definitely not clear really
[18:08:10] <dmehus>	 So far, I haven't seen any evidence of that, and I'm wondering we couldn't maybe approve it as a trial basis, with an abuse filter to publicly log all embeds of archive.org files. If there's any sign of problems, we can revoke its whitelist immediately maybe?
[18:12:29] <JohnLewis>	 I trust Reception123 to proceed however he deems appropriate here
[18:13:22] <Reception123>	 JohnLewis: I feel like it's a difficult question because while there doesn't seem to be support for it, it's also not clear what criteria we should apply when determining what should get added and what shouldn't
[18:13:29] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqTwI
[18:13:30] <Reception123>	 so I'm not very sure how to proceed
[18:13:30] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega 03f11e11e - ManageWikiTypes: Create new contentmodel type; cleanup class
[18:13:32] <Not-71a9>	 [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vpSns
[18:13:35] <Not-71a9>	 [02ManageWiki] 07Universal-Omega opened pull request 03#255: ManageWikiTypes: Create new contentmodel type; cleanup class - 13https://git.io/JqTwL
[18:14:37] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[18:17:53] <JohnLewis>	 Reception123: there is no clear criteria, but you're now in a decision making position so, there's a decision to be made :)
[18:21:29] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqTwV
[18:21:30] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega 03435f7da - Update ManageWikiTypes parameters
[18:21:32] <Not-71a9>	 [02ManageWiki] 07Universal-Omega synchronize pull request 03#255: ManageWikiTypes: Create new contentmodel type; cleanup class - 13https://git.io/JqTwL
[18:21:59] <RhinosF1>	 JohnLewis: I think it's just fine 2 SRE happy with it now
[18:22:06] <RhinosF1>	 Which I'm not sure we do have
[18:22:11] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqTwo
[18:22:13] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega 0344eae81 - Update ManageWikiTypes.php
[18:22:14] <Not-71a9>	 [02ManageWiki] 07Universal-Omega synchronize pull request 03#255: ManageWikiTypes: Create new contentmodel type; cleanup class - 13https://git.io/JqTwL
[18:22:32] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[18:22:49] <JohnLewis>	 It is true our currently policy is sign-off from two SREs
[18:24:12] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[18:24:33] <Reception123>	 yeah, the current de-facto policy is sign-off by two SREs, so unless a second agrees I wouldn't be comfortable approving
[18:25:01] <Reception123>	 unless of course we modify the policy and change the conditions/process
[18:25:10] <Reception123>	 (which would then also include a criteria to be followed)
[18:43:46] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqTrN
[18:43:48] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega 0350a0b74 - Update ManageWikiTypes.php
[18:43:49] <Not-71a9>	 [02ManageWiki] 07Universal-Omega synchronize pull request 03#255: ManageWikiTypes: Create new contentmodel type; cleanup class - 13https://git.io/JqTwL
[18:44:54] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[18:45:56] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqToJ
[18:45:57] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega 03d49a2ec - Update ManageWikiFormFactoryBuilder.php
[18:45:59] <Not-71a9>	 [02ManageWiki] 07Universal-Omega synchronize pull request 03#255: ManageWikiTypes: Create new contentmodel type; cleanup class - 13https://git.io/JqTwL
[18:46:59] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[19:06:57] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqTKq
[19:06:58] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega 03964d2ba - More ManageWikiNamespaces improvements
[19:07:00] <Not-71a9>	 [02ManageWiki] 07Universal-Omega synchronize pull request 03#255: ManageWikiTypes: Create new contentmodel type; cleanup class - 13https://git.io/JqTwL
[19:07:59] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[19:08:24] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqTKG
[19:08:25] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega 036591ace - sort contentmodel type
[19:08:27] <Not-71a9>	 [02ManageWiki] 07Universal-Omega synchronize pull request 03#255: ManageWikiTypes: Create new contentmodel type; cleanup class - 13https://git.io/JqTwL
[19:08:55] <Not-71a9>	 [02ManageWiki] 07Universal-Omega edited pull request 03#255: Multiple improvements to ManageWikiNamespaces - 13https://git.io/JqTwL
[19:09:24] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[19:13:17] <Not-71a9>	 [02ManageWiki] 07Universal-Omega edited pull request 03#255: Multiple improvements to ManageWikiNamespaces - 13https://git.io/JqTwL
[19:16:58] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqTKK
[19:16:59] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega 0341b89ec - Remove ManageWikiNamespacesExtraContentModels config
[19:17:01] <Not-71a9>	 [02ManageWiki] 07Universal-Omega synchronize pull request 03#255: Multiple improvements to ManageWikiNamespaces - 13https://git.io/JqTwL
[19:17:52] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[19:21:52] <dmehus>	 Universal_Omega, Reception123, or RhinosF1, there's approximately 50 jobs outstanding on `metawiki` and has been for quite awhile. Mind running jobs?
[19:23:59] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqTKQ
[19:24:00] <Not-71a9>	 [02miraheze/puppet] 07paladox 03bee891e - graylog: Set max_result_window to 10000000 on graylog template
[19:24:02] <Not-71a9>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[19:24:03] <Not-71a9>	 [02puppet] 07paladox opened pull request 03#1678: graylog: Set max_result_window to 10000000 on graylog template - 13https://git.io/JqTK7
[19:25:30] <Universal_Omega>	 !log universalomega@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=metawiki (Failed with Fatal error: Allowed memory size of 157286400 bytes exhausted )
[19:25:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:26:03] <Universal_Omega>	 paladox, or JohnLewis: `Fatal error: Allowed memory size of 157286400 bytes exhausted (tried to allocate 221572064 bytes) in /srv/mediawiki/w/extensions/CreateWiki/vendor/php-ai/php-ml/src/ModelManager.php on line 35`
[19:26:03] <JohnLewis>	 What jobs keep piling up?
[19:26:21] <JohnLewis>	 Yeah, it require about 900MB of memory 
[19:28:41] <dmehus>	 Interesting
[19:29:02] <dmehus>	 do we need to consider increasing the available ram allocated to the jbr servers?
[19:29:20] <JohnLewis>	 There’s no lack of memory
[19:29:28] <dmehus>	 oh
[19:29:33] <JohnLewis>	 its to do with the memory allocation UO have the job when he ran it
[19:29:44] <dmehus>	 ah
[19:31:18] <RhinosF1>	 The actual job queue has it upped for CW AI
[19:32:23] <RhinosF1>	 https://github.com/miraheze/puppet/blob/master/modules/mediawiki/templates/jobrunner.json.erb#L44
[19:32:24] <MirahezeBot>	 [ puppet/jobrunner.json.erb at master · miraheze/puppet · GitHub ] - github.com
[19:32:27] <RhinosF1>	 Universal_Omega: ^
[19:51:03] <Not-71a9>	 [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqTi6
[19:51:04] <Not-71a9>	 [02miraheze/puppet] 07JohnFLewis 0375c319b - detect language on landing page
[19:54:13] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqTiH
[19:54:15] <Not-71a9>	 [02miraheze/puppet] 07paladox 0368c4e83 - Update graylog.pp
[19:54:16] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1678: graylog: Set max_result_window to 10000000 on graylog template - 13https://git.io/JqTK7
[19:54:29] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1678: graylog: Set max_result_window to 10000000 on graylog template - 13https://git.io/JqTK7
[19:54:31] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqTi7
[19:54:32] <Not-71a9>	 [02miraheze/puppet] 07paladox 030c9d1eb - Update graylog.pp
[19:54:51] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/JqTi5
[19:54:52] <Not-71a9>	 [02miraheze/puppet] 07paladox 035f1df32 - Create index_template.json
[19:54:54] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1678: graylog: Set max_result_window to 10000000 on graylog template - 13https://git.io/JqTK7
[19:55:35] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqTiF
[19:55:36] <Not-71a9>	 [02miraheze/puppet] 07paladox 03336336b - Update index_template.json
[19:55:38] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1678: graylog: Set max_result_window to 10000000 on graylog template - 13https://git.io/JqTK7
[19:57:37] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqTip
[19:57:39] <Not-71a9>	 [02miraheze/puppet] 07paladox 03781fe23 - Update index_template.json
[19:57:41] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1678: graylog: Set max_result_window to 10000000 on graylog template - 13https://git.io/JqTK7
[21:12:16] <Not-71a9>	 [02puppet] 07paladox edited pull request 03#1678: graylog: Set max_result_window to 10000000 on graylog template - 13https://git.io/JqTK7
[21:12:33] <Not-71a9>	 [02puppet] 07paladox closed pull request 03#1678: graylog: Set max_result_window to 10000000 on graylog template - 13https://git.io/JqTK7
[21:12:34] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JqTD8
[21:12:36] <Not-71a9>	 [02miraheze/puppet] 07paladox 03c21e253 - graylog: Set max_result_window to 10000000 on graylog template (#1678)
[21:13:23] <Not-71a9>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[21:13:24] <Not-71a9>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[22:03:40] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqT9k
[22:03:42] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega 0302118ec - Add support for constant namespace related configs to MWN
[22:03:43] <Not-71a9>	 [02ManageWiki] 07Universal-Omega synchronize pull request 03#255: Multiple improvements to ManageWikiNamespaces - 13https://git.io/JqTwL
[22:04:41] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[22:07:06] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03patch1 [+4/-2/±4] 13https://git.io/JqT9n
[22:07:08] <Not-71a9>	 [02miraheze/puppet] 07paladox 037d9dbac - Adds support for mysql instances in mysqld_exporter
[22:07:09] <Not-71a9>	 [02puppet] 07paladox created branch 03patch1 - 13https://git.io/vbiAS
[22:07:19] <Not-71a9>	 [02puppet] 07paladox opened pull request 03#1679: Adds support for mysql instances in mysqld_exporter - 13https://git.io/JqT9c
[22:07:37] <Not-71a9>	 [02ManageWiki] 07Universal-Omega edited pull request 03#255: Multiple improvements to ManageWikiNamespaces - 13https://git.io/JqTwL
[22:09:07] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03patch1 [+0/-0/±1] 13https://git.io/JqT9l
[22:09:09] <Not-71a9>	 [02miraheze/puppet] 07paladox 03edebaef - Update db.pp
[22:09:10] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1679: Adds support for mysql instances in mysqld_exporter - 13https://git.io/JqT9c
[22:10:54] <Not-71a9>	 [02ManageWiki] 07Universal-Omega edited pull request 03#255: Multiple improvements to ManageWikiNamespaces - 13https://git.io/JqTwL
[22:11:19] <Not-71a9>	 [02ManageWiki] 07Universal-Omega edited pull request 03#255: Multiple improvements to ManageWikiNamespaces - 13https://git.io/JqTwL
[22:12:54] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03patch1 [+0/-0/±1] 13https://git.io/JqT91
[22:12:55] <Not-71a9>	 [02miraheze/puppet] 07paladox 03d439a8a - Update instance.pp
[22:12:57] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1679: Adds support for mysql instances in mysqld_exporter - 13https://git.io/JqT9c
[22:15:13] <RhinosF1>	 !log rhinos@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/migrateAccount.php --username=Hatelynnloudjrok --wiki=loathsomecharacterswiki --auto (per request of Stewards so account can be locked)
[22:15:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:15:16] <RhinosF1>	 dmehus: ^
[22:18:36] <dmehus>	 RhinosF1, ack and ty. Ag, `migrateAccount.php`. Cool.
[22:18:44] <dmehus>	 Have we ever run that before?
[22:19:01] <RhinosF1>	 dmehus: when similar things to today happened
[22:19:04] <dmehus>	 s/Ag/Ah
[22:19:04] <MirahezeBot>	 dmehus meant to say: RhinosF1, ack and ty. Ah, `migrateAccount.php`. Cool.
[22:19:08] <dmehus>	 ah, neat
[22:19:22] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqT9h
[22:19:23] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega 033728202 - Add config variables to labels for ManageWikiNamespaces
[22:19:25] <Not-71a9>	 [02ManageWiki] 07Universal-Omega synchronize pull request 03#255: Multiple improvements to ManageWikiNamespaces - 13https://git.io/JqTwL
[22:19:34] <Not-71a9>	 [02ManageWiki] 07Universal-Omega closed pull request 03#254: Add config variables to ManageWikiNamespaces labels - 13https://git.io/JqUKZ
[22:19:36] <Not-71a9>	 [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns
[22:19:38] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1
[22:19:39] <RhinosF1>	 dmehus: it's rare but it happens
[22:20:21] <Not-71a9>	 [02ManageWiki] 07Universal-Omega edited pull request 03#255: Multiple improvements to ManageWikiNamespaces - 13https://git.io/JqTwL
[22:20:22] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[22:23:35] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03patch1 [+0/-0/±1] 13https://git.io/JqTHG
[22:23:37] <Not-71a9>	 [02miraheze/puppet] 07paladox 032dcbfb7 - Update prometheus.yml.erb
[22:23:39] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1679: Adds support for mysql instances in mysqld_exporter - 13https://git.io/JqT9c
[22:29:34] <icinga-miraheze>	 PROBLEM - dbbackup2 Puppet on dbbackup2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 16 minutes ago with 0 failures
[22:30:32] <icinga-miraheze>	 PROBLEM - db13 Puppet on db13 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures
[22:30:47] <icinga-miraheze>	 PROBLEM - db11 Puppet on db11 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures
[22:31:17] <icinga-miraheze>	 PROBLEM - db12 Puppet on db12 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures
[22:33:13] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03patch1 [+0/-1/±0] 13https://git.io/JqTQe
[22:33:15] <Not-71a9>	 [02miraheze/puppet] 07paladox 03add52d0 - Delete mysqld_exporter.pp
[22:33:16] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1679: Adds support for mysql instances in mysqld_exporter - 13https://git.io/JqT9c
[22:33:24] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03patch1 [+0/-0/±1] 13https://git.io/JqTQv
[22:33:25] <Not-71a9>	 [02miraheze/puppet] 07paladox 036f72e90 - Update site.pp
[22:33:27] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1679: Adds support for mysql instances in mysqld_exporter - 13https://git.io/JqT9c
[22:33:32] <icinga-miraheze>	 RECOVERY - dbbackup2 Puppet on dbbackup2 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[22:33:51] <Not-71a9>	 [02ManageWiki] 07Universal-Omega edited pull request 03#255: Multiple improvements to ManageWikiNamespaces - 13https://git.io/JqTwL
[22:34:44] <Not-71a9>	 [02ManageWiki] 07Universal-Omega edited pull request 03#255: Multiple improvements to ManageWikiNamespaces - 13https://git.io/JqTwL
[22:34:48] <icinga-miraheze>	 RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[22:37:17] <icinga-miraheze>	 RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[22:38:32] <icinga-miraheze>	 RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:44:01] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqTQ2
[22:44:02] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega 03b8a4fd4 - Add support for localised help and labels for ManageWikiNamespacesAdditional
[22:44:04] <Not-71a9>	 [02ManageWiki] 07Universal-Omega synchronize pull request 03#255: Multiple improvements to ManageWikiNamespaces - 13https://git.io/JqTwL
[22:44:34] <Not-71a9>	 [02ManageWiki] 07Universal-Omega edited pull request 03#255: Multiple improvements to ManageWikiNamespaces - 13https://git.io/JqTwL
[22:45:08] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[22:55:47] <icinga-miraheze>	 PROBLEM - wiki.mcpirevival.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mcpirevival.tk All nameservers failed to answer the query.
[22:56:00] <icinga-miraheze>	 PROBLEM - ipv6bolivia.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - ipv6bolivia.tk All nameservers failed to answer the query.
[22:59:35] <Not-71a9>	 [02ManageWiki] 07Universal-Omega edited pull request 03#255: Multiple improvements to ManageWikiNamespaces - 13https://git.io/JqTwL
[23:01:02] <Not-71a9>	 [02puppet] 07paladox closed pull request 03#1679: Adds support for mysql instances in mysqld_exporter - 13https://git.io/JqT9c
[23:01:04] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+4/-3/±6] 13https://git.io/JqT7Z
[23:01:05] <Not-71a9>	 [02miraheze/puppet] 07paladox 03e2bc175 - Adds support for mysql instances in mysqld_exporter (#1679)
[23:01:07] <Not-71a9>	 [02puppet] 07paladox deleted branch 03patch1 - 13https://git.io/vbiAS
[23:01:08] <Not-71a9>	 [02miraheze/puppet] 07paladox deleted branch 03patch1
[23:02:37] <icinga-miraheze>	 RECOVERY - wiki.mcpirevival.tk - reverse DNS on sslhost is OK: rDNS OK - wiki.mcpirevival.tk reverse DNS resolves to cp10.miraheze.org
[23:02:59] <icinga-miraheze>	 RECOVERY - ipv6bolivia.tk - reverse DNS on sslhost is OK: rDNS OK - ipv6bolivia.tk reverse DNS resolves to cp10.miraheze.org
[23:10:43] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqT7y
[23:10:45] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega 03d83651b - set Wikibase formatterUrlProperty to 'P39' for nbdbwiki (T6867)
[23:11:49] <Not-71a9>	 [02mw-config] 07Universal-Omega closed pull request 03#3752: Add opk to $wgFileExtensions options (T6918) - 13https://git.io/Jthgt
[23:11:50] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqT77
[23:11:52] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega 03f1570ae - Add opk to $wgFileExtensions options (T6918) (#3752)
[23:11:53] <Not-71a9>	 [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3
[23:11:55] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5
[23:12:16] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[23:12:52] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[23:35:29] <Universal_Omega>	 !log universalomega@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/extensions/Wikibase/repo/maintenance/rebuildPropertyInfo.php --rebuild-all --force --wiki=nbdbwiki
[23:35:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:41:45] <Not-71a9>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+1/-0/±1] 13https://git.io/JqTda
[23:41:47] <Not-71a9>	 [02miraheze/mediawiki] 07Universal-Omega 03c0b3f8a - Install HasSomeColours (T6863)
[23:47:18] <Not-71a9>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-1/±1] 13https://git.io/JqTd9
[23:47:20] <Not-71a9>	 [02miraheze/mediawiki] 07Universal-Omega 03be4419e - Revert "Install HasSomeColours (T6863)"
[23:50:38] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JqTdp
[23:50:40] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega 0385224d5 - add HasSomeColours skin
[23:50:41] <Not-71a9>	 [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3
[23:51:26] <Not-71a9>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+1/-0/±1] 13https://git.io/JqTdh
[23:51:28] <Not-71a9>	 [02miraheze/mediawiki] 07Universal-Omega 0332c62f6 - Install HasSomeColours (T6863)
[23:51:54] <Not-71a9>	 [02mw-config] 07Universal-Omega opened pull request 03#3755: add HasSomeColours skin (T6863) - 13https://git.io/JqTdj
[23:52:46] <Not-71a9>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JqTFf
[23:52:47] <Not-71a9>	 [02miraheze/mediawiki] 07Universal-Omega 03fa25057 - add ignore = dirty
[23:53:27] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[23:54:04] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JqTFI
[23:54:05] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega 03134890b - Update LocalSettings.php
[23:54:07] <Not-71a9>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3755: add HasSomeColours skin (T6863) - 13https://git.io/JqTdj
[23:55:02] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JqTFt
[23:55:04] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega 0315d9082 - Update LocalExtensions.php
[23:55:05] <Not-71a9>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3755: add HasSomeColours skin (T6863) - 13https://git.io/JqTdj
[23:55:07] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[23:56:07] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[23:57:02] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JqTFG
[23:57:04] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega 037ce47c3 - Update ManageWikiExtensions.php
[23:57:05] <Not-71a9>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3755: add HasSomeColours skin (T6863) - 13https://git.io/JqTdj
[23:58:37] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.