[00:09:25] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[00:11:21] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is WARNING: WARNING: Full, 80402 files, 2.603GB, 2019-08-10 12:53:00 (2.9 weeks ago)
[01:35:40] <Hispano76_>	 @staff
[01:37:14] <Hispano76_>	 JohnLewis paladox Reception123 https://es.publictestwiki.com/wiki/Especial:ManageWiki  uncheck "inactive" please. I do not have rights to consul
[01:37:15] <ZppixBot>	 [ Gestionar un wiki - PruebaWiki ] - es.publictestwiki.com
[02:00:08] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjxpM
[02:00:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 030d93bfe - BOT: Updating services config for wikis
[06:26:43] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2975 MB (12% inode=94%);
[06:35:09] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjxj9
[06:35:11] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 03ee23c90 - BOT: Updating services config for wikis
[07:11:11] <Reception123>	 !log apt upgrade lilypond lilypond-data on mw*
[07:11:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[07:15:57] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.3-apcu],Package[php7.3-redis]
[07:24:21] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:35:11] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjpeo
[08:35:12] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 037513576 - BOT: Updating services config for wikis
[09:17:03] <Not-f817>	 [02landing] 077yultukuri7 opened pull request 03#12: japanese translation - 13https://git.io/fjpeh
[14:00:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjpUB
[14:00:12] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 039a609cb - BOT: Updating services config for wikis
[14:02:51] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2633 MB (10% inode=94%);
[14:31:29] <Reception123>	 !log reception@mw1:/srv/mediawiki/w/extensions/ManageWiki/maintenance$ sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki dfcharacterwiki --overwrite
[14:31:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:32:58] <Not-f817>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjpTk
[14:33:00] <Not-f817>	 [02miraheze/mw-config] 07paladox 032c95a97 - Fix setting REMOTE_ADDR Also correctly add misc3 ipv6 address
[14:34:16] <Not-f817>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fjpTL
[14:34:18] <Not-f817>	 [02miraheze/ssl] 07Reception123 03b5acc88 - add wiki.valkyrienskies.org cert
[14:45:11] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjpTc
[14:45:12] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 030ccfc0d - BOT: Updating services config for wikis
[14:45:59] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config]
[14:52:47] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjpT0
[14:52:49] <Not-f817>	 [02miraheze/puppet] 07paladox 03b4b708d - Update init.pp
[14:54:32] <paladox>	 !log root@misc2:/home/paladox# apt-get install libgd3 libxml2 php-common php-igbinary php-imagick php-mailparse php-mbstring php-mysql php-xml
[14:54:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:57:30] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjpTu
[14:57:32] <Not-f817>	 [02miraheze/puppet] 07paladox 034510694 - Update redis.pp
[15:00:37] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb
[15:02:35] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[15:49:59] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjpkC
[15:50:01] <Not-f817>	 [02miraheze/puppet] 07paladox 0387ba3a0 - Update redis exporter to 1.1.0
[20:18:05] <Hispano76_>	 hello
[20:19:00] <RhinosF1>	 Hi Hispano76_
[20:20:13] <Hispano76_>	 you can uncheck "inactive" from the wiki es.publictestwiki.miraheze.org. I don't have permission to modify ManageWiki 
[20:20:34] <Hispano76_>	 https://es.publictestwiki.com/wiki/Especial:CambiosRecientes?hidebots=1&limit=1000&days=180&urlversion=2 *
[20:20:35] <ZppixBot>	 [ Cambios recientes - PruebaWiki ] - es.publictestwiki.com
[20:21:56] <RhinosF1>	 Hispano76_: neither do I (atm)
[20:22:10] <RhinosF1>	 paladox: ^
[20:22:34] <Hispano76_>	 :/
[20:23:59] <paladox>	 done
[20:24:22] <Hispano76_>	 thanks paladox 
[20:24:25] <Hispano76_>	 :)
[20:24:27] <paladox>	 !log upgrade grafana on misc1
[20:24:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:25:34] <paladox>	 !log apt-get install icingacli icingaweb2 icingaweb2-common icingaweb2-module-doc icingaweb2-module-monitoring - misc1
[20:25:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:26:04] <paladox>	 !log apt-get install php-intl php-ldap php-mailparse php-mbstring php-mysql php-xml  php php-common php-curl php-igbinary php-imagick - misc1
[20:26:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:26:31] <paladox>	 !log upgrade nodejs on misc1
[20:26:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:14:32] <Not-f817>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjpqZ
[21:14:33] <Not-f817>	 [02miraheze/mediawiki] 07paladox 03f4e8919 - Update AbuseFilter
[21:14:44] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown)
[21:15:13] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown)
[21:15:20] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 25 failures. Last run 3 minutes ago with 25 failures. Failed resources (up to 3 shown)
[21:15:25] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 333 failures. Last run 2 minutes ago with 333 failures. Failed resources (up to 3 shown)
[21:15:54] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 335 failures. Last run 3 minutes ago with 335 failures. Failed resources (up to 3 shown)
[21:16:00] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 335 failures. Last run 3 minutes ago with 335 failures. Failed resources (up to 3 shown)
[21:16:12] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 328 failures. Last run 3 minutes ago with 328 failures. Failed resources (up to 3 shown)
[21:16:20] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 236 failures. Last run 3 minutes ago with 236 failures. Failed resources (up to 3 shown): File[allthetropes.org],File[allthetropes.org_private],File[aman.info.tm],File[aman.info.tm_private]
[21:17:11] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 159 failures. Last run 3 minutes ago with 159 failures. Failed resources (up to 3 shown): File[wiki.make717.org],File[wiki.make717.org_private],File[thelonsdalebattalion.co.uk],File[thelonsdalebattalion.co.uk_private]
[21:18:53] <RhinosF1>	 paladox: GC? ^
[21:23:12] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[21:23:19] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[21:23:53] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[21:24:19] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[21:24:44] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:25:15] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[21:27:29] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[21:28:17] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:28:25] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:30:51] <paladox>	 RhinosF1 yeh
[21:30:57] <RhinosF1>	 :)
[21:39:25] <paladox>	 !log upgrade phabricator on misc4
[21:39:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:44:30] <icinga-miraheze>	 PROBLEM - misc4 phd on misc4 is CRITICAL: PROCS CRITICAL: 0 processes with args 'phd'
[21:45:16] <RhinosF1>	 paladox: ^ and icinga guest is down
[21:45:25] <RhinosF1>	 incorrect pw
[21:45:25] <paladox>	 hiya hmm
[21:46:05] <RhinosF1>	 paladox: can u check icinga? can u log in? If not, someone's reset the guest account details
[21:46:18] <RhinosF1>	 if so, even
[21:47:04] <paladox>	 !log delete from daemon_log where pid = 10833; on db4
[21:47:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:48:11] <paladox>	 !log delete from daemon_log where pid = 15083; on db4
[21:48:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:48:19] <paladox>	 !log delete from daemon_log where pid = 15565; on db4
[21:48:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:48:30] <icinga-miraheze>	 RECOVERY - misc4 phd on misc4 is OK: PROCS OK: 1 process with args 'phd'
[21:48:59] <RhinosF1>	 paladox: can u check icinga?
[21:49:12] <paladox>	 yup
[21:49:35] <paladox>	 fixed
[21:49:36] <RhinosF1>	 paladox: if you're login works, then we need to reset the guest pw back as it's not working
[21:49:47] <RhinosF1>	 paladox: thx, seen and back in
[22:06:28] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 13.09, 7.95, 5.33
[22:12:27] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.53, 7.34, 6.05
[22:14:27] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.01, 6.72, 5.99
[23:05:36] <Not-f817>	 [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS
[23:05:38] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjpmn
[23:05:39] <Not-f817>	 [02miraheze/puppet] 07paladox 038770f53 - bacula: Replace lizardfs1 with lizardfs4
[23:05:41] <Not-f817>	 [02puppet] 07paladox opened pull request 03#1075: bacula: Replace lizardfs1 with lizardfs4 - 13https://git.io/fjpmc
[23:11:03] <Not-f817>	 [02puppet] 07paladox closed pull request 03#1075: bacula: Replace lizardfs1 with lizardfs4 - 13https://git.io/fjpmc
[23:11:04] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjpmB
[23:11:06] <Not-f817>	 [02miraheze/puppet] 07paladox 03c49451f - bacula: Replace lizardfs1 with lizardfs4 (#1075)
[23:11:07] <Not-f817>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4
[23:11:09] <Not-f817>	 [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS
[23:13:10] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.24, 6.37, 3.86
[23:15:09] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.77, 6.43, 4.21
[23:19:15] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.26, 6.93, 4.92
[23:21:15] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.84, 7.57, 5.39
[23:21:46] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is UNKNOWN: NRPE: Unable to read output
[23:22:24] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Static on bacula1 is UNKNOWN: NRPE: Unable to read output
[23:22:59] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Databases db4 on bacula1 is UNKNOWN: NRPE: Unable to read output
[23:23:34] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Databases db5 on bacula1 is UNKNOWN: NRPE: Unable to read output
[23:23:36] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is UNKNOWN: NRPE: Unable to read output
[23:25:02] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Daemon on bacula1 is CRITICAL: PROCS CRITICAL: 1 process with UID = 116 (bacula)
[23:25:24] <paladox>	 huh
[23:31:59] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Databases db5 on bacula1 is WARNING: WARNING: Full, 422 files, 19.13GB, 2019-08-10 13:18:00 (3.1 weeks ago)
[23:32:20] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CRITICAL: Full, 4137 files, 8.488MB, 2019-08-11 00:05:00 (3.0 weeks ago)
[23:32:21] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is WARNING: WARNING: Full, 80402 files, 2.603GB, 2019-08-10 12:53:00 (3.1 weeks ago)
[23:33:02] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Daemon on bacula1 is OK: PROCS OK: 2 processes with UID = 116 (bacula)
[23:33:37] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Static on bacula1 is CRITICAL: CRITICAL: Timeout or unknown client: lizardfs1-fd
[23:33:40] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Databases db4 on bacula1 is WARNING: WARNING: Full, 764611 files, 70.58GB, 2019-08-04 02:46:00 (4.0 weeks ago)
[23:39:20] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.85, 7.86, 7.33
[23:46:20] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Databases db4 on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:46:28] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Databases db5 on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:46:46] <icinga-miraheze>	 PROBLEM - bacula1 SSH on bacula1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:47:24] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:48:00] <icinga-miraheze>	 PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:48:45] <icinga-miraheze>	 PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:49:19] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.70, 5.82, 6.69
[23:50:02] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Daemon on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:50:29] <icinga-miraheze>	 PROBLEM - Host bacula1 is DOWN: PING CRITICAL - Packet loss = 100%
[23:53:18] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.41, 6.86, 6.94
[23:55:18] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 10.77, 8.38, 7.48