[00:02:40] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.15, 19.30, 12.34
[00:03:55] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2490 MB (10% inode=93%);
[00:04:40] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 10.76, 16.13, 12.06
[00:20:22] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJZj6
[00:20:23] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot 03e939431 - BOT: Updating services config for wikis
[01:23:13] <icinga-miraheze>	 PROBLEM - mw7 APT on mw7 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates).
[01:39:13] <icinga-miraheze>	 PROBLEM - services1 APT on services1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates).
[02:31:33] <icinga-miraheze>	 PROBLEM - cloud2 APT on cloud2 is CRITICAL: APT CRITICAL: 54 packages available for upgrade (1 critical updates).
[02:35:49] <icinga-miraheze>	 PROBLEM - test2 APT on test2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates).
[02:38:52] <icinga-miraheze>	 PROBLEM - gluster1 APT on gluster1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates).
[03:20:21] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJnJP
[03:20:23] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot 031a9cf8d - BOT: Updating services config for wikis
[05:10:03] <icinga-miraheze>	 PROBLEM - cloud1 APT on cloud1 is CRITICAL: APT CRITICAL: 54 packages available for upgrade (1 critical updates).
[05:16:15] <icinga-miraheze>	 PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 13 packages available for upgrade (1 critical updates).
[05:19:13] <Not-3033>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JJnIP
[05:19:15] <Not-3033>	 [02miraheze/ssl] 07Reception123 03a311e30 - add wiki.azimuth.navy cert
[06:07:32] <icinga-miraheze>	 RECOVERY - cloud1 APT on cloud1 is OK: APT OK: 53 packages available for upgrade (0 critical updates).
[06:08:54] <icinga-miraheze>	 PROBLEM - mw5 APT on mw5 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates).
[06:09:13] <icinga-miraheze>	 RECOVERY - services1 APT on services1 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[06:12:54] <icinga-miraheze>	 RECOVERY - mw5 APT on mw5 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[06:14:20] <icinga-miraheze>	 RECOVERY - cloud2 APT on cloud2 is OK: APT OK: 53 packages available for upgrade (0 critical updates).
[06:25:14] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJnqR
[06:25:16] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot 03be7d83b - BOT: Updating services config for wikis
[06:25:23] <icinga-miraheze>	 RECOVERY - gluster1 APT on gluster1 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[06:35:15] <icinga-miraheze>	 RECOVERY - mw7 APT on mw7 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[06:41:21] <icinga-miraheze>	 PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates).
[06:46:06] <icinga-miraheze>	 RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 12 packages available for upgrade (0 critical updates).
[06:57:21] <icinga-miraheze>	 RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[06:58:39] <icinga-miraheze>	 RECOVERY - test2 APT on test2 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[08:21:32] <icinga-miraheze>	 PROBLEM - mw4 APT on mw4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates).
[11:33:03] <RhinosF1>	 .in 28mins ToU
[11:33:03] <MirahezeBot>	 RhinosF1: Okay, will remind at 2020-07-18 - 13:01:03BST
[11:47:13] <icinga-miraheze>	 PROBLEM - services2 APT on services2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates).
[11:49:03] <RhinosF1>	 .at 12:50 do it
[11:49:03] <MirahezeBot>	 RhinosF1: Okay, will remind in 56 secs
[11:50:01] <MirahezeBot>	 RhinosF1: do it
[12:01:05] <MirahezeBot>	 RhinosF1: ToU
[12:03:12] <icinga-miraheze>	 PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zymonic.com' expires in 15 day(s) (Mon 03 Aug 2020 11:58:31 GMT +0000).
[12:06:45] <MH-Discord>	 <MrJaroslavik> Why isn't enabled translation documentation? (on WMF > https://i.nuuls.com/9tlN4.jpeg)
[12:07:13] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJnC3
[12:07:15] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot 033a4a443 - Bot: Update SSL cert for wiki.zymonic.com
[12:08:00] <Not-3033>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JJnCZ
[12:08:01] <Not-3033>	 [02miraheze/puppet] 07paladox 03649bcbc - Ircrcbot: change lineRate to 0.5
[12:08:03] <Not-3033>	 [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://git.io/vbiAS
[12:08:04] <Not-3033>	 [02puppet] 07paladox opened pull request 03#1455: Ircrcbot: change lineRate to 0.5 - 13https://git.io/JJnCn
[12:14:58] <Reception123>	 !log sudo service ircrcbot stop
[12:14:59] <Reception123>	 ^ RhinosF1 
[12:15:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:15:14] <RhinosF1>	 Reception123: let's give it 60 seconds to fully stop
[12:15:54] <RhinosF1>	 .in 60s Reception123: start Relay
[12:15:55] <MirahezeBot>	 RhinosF1: Okay, will remind at 2020-07-18 - 13:16:54BST
[12:16:34] <icinga-miraheze>	 RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Fri 16 Oct 2020 11:07:07 GMT +0000.
[12:16:56] <MirahezeBot>	 RhinosF1: Reception123: start Relay
[12:17:12] <Reception123>	 !log sudo service ircrcbot start
[12:17:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:26:14] <RhinosF1>	 Reception123: yey
[12:36:19] <MH-Discord>	 <Avengium (Ángel)> I like when things get solved
[12:42:14] <RhinosF1>	 paladox: lowering lineRate doesn't help. That'll just make things worse. It needs intervention when the lag is high.
[12:53:56] <icinga-miraheze>	 PROBLEM - mw6 APT on mw6 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates).
[12:56:43] <Not-3033>	 [02puppet] 07RhinosF1 commented on pull request 03#1455: Ircrcbot: change lineRate to 0.5 - 13https://git.io/JJnWX
[13:03:49] <Zppix>	 0..
[13:03:50] <Zppix>	 3
[13:04:07] <Zppix>	 .
[13:04:44] <Zppix>	 oops
[13:07:57] <RhinosF1>	 Hey Zppix
[13:09:01] <Zppix>	 hi
[13:24:46] <hispano76>	 When you create a wiki under the language es-419, you end up creating it in English, that's a known mistake? 
[13:25:09] <RhinosF1>	 no
[13:25:19] <RhinosF1>	 Are you sure it's in English?
[13:25:27] <RhinosF1>	 hispano76: which wiki?
[13:25:38] <hispano76>	 for example siasawiki ideawikiwiki
[13:26:36] <RhinosF1>	 .mh ideawiki .
[13:26:36] <MirahezeBot>	 https://ideawiki.miraheze.org/wiki/.
[13:28:40] <RhinosF1>	 hispano76: it looks deliberate
[13:28:44] <RhinosF1>	 https://github.com/wikimedia/mediawiki/commit/a19320bf90b58437020a4bd50803e0d29a8a9269
[13:28:45] <MirahezeBot>	 [ Add the es-419 language code to support South American Spanish · wikimedia/mediawiki@a19320b · GitHub ] - github.com
[13:28:56] <RhinosF1>	 There's nothing actually translated into es-419
[13:29:50] <RhinosF1>	 I'll raise it when F1 finishes
[13:30:11] <hispano76>	 um, but it's misleading and confusing to the user :|
[13:30:29] <hispano76>	 ah, ok
[13:31:01] <hispano76>	 thanks RhinosF1 
[13:34:12] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1439 MB (5% inode=93%);
[13:36:12] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 1459 MB (6% inode=93%);
[13:48:15] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1444 MB (5% inode=93%);
[13:49:48] <RhinosF1>	 hispano76: 100% agree
[13:51:16] <hispano76>	 :)
[14:47:14] <icinga-miraheze>	 PROBLEM - puppet2 APT on puppet2 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (1 critical updates).
[14:47:50] <RhinosF1>	 paladox: can you push any apt's that need doing?
[14:49:11] <paladox>	 !log upgrade libnss3 on mw*
[14:49:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:49:32] <icinga-miraheze>	 RECOVERY - mw4 APT on mw4 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[14:51:10] <paladox>	 !log upgrade libnss3 on puppet2
[14:51:14] <icinga-miraheze>	 RECOVERY - puppet2 APT on puppet2 is OK: APT OK: 4 packages available for upgrade (0 critical updates).
[14:51:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:51:40] <paladox>	 !log upgrade libnss3 on services*
[14:51:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:51:56] <icinga-miraheze>	 RECOVERY - mw6 APT on mw6 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[14:53:12] <icinga-miraheze>	 RECOVERY - services2 APT on services2 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[14:54:57] <paladox>	 !log upgrade puppet-agent puppetdb puppetdb-termini puppetserver - puppet2
[14:55:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:55:23] <RhinosF1>	 paladox: ty, can you ack/downtime/handle the other alerts?
[14:56:42] <RhinosF1>	 With useful messages
[14:57:23] <paladox>	 done
[14:57:34] <RhinosF1>	 paladox: acknowledging an alert with the alert reason isn't helpful. Why is disk space high? Why is puppet failing?
[15:01:13] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1328 MB (5% inode=93%);
[15:01:28] <icinga-miraheze>	 PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[15:01:43] <icinga-miraheze>	 PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[15:02:28] <paladox>	 done
[15:03:25] <RhinosF1>	 Thanks paladox
[15:03:50] <RhinosF1>	 paladox: puppet on gluster1,mail1 and services2 has just failed
[15:04:10] <RhinosF1>	 Puppet's crashing
[15:04:53] <RhinosF1>	 Yeah we got a widespread puppet failure
[15:04:57] <icinga-miraheze>	 PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown)
[15:05:00] <icinga-miraheze>	 PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown)
[15:05:01] <icinga-miraheze>	 PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 2 minutes ago with 13 failures. Failed resources (up to 3 shown)
[15:05:15] <RhinosF1>	 paladox: you just updated stuff on puppet2, could it be linked?
[15:05:27] <icinga-miraheze>	 PROBLEM - db12 Puppet on db12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown)
[15:05:27] <icinga-miraheze>	 PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown)
[15:05:35] <icinga-miraheze>	 PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Puppet has 23 failures. Last run 2 minutes ago with 23 failures. Failed resources (up to 3 shown)
[15:05:46] <icinga-miraheze>	 PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 494 failures. Last run 2 minutes ago with 494 failures. Failed resources (up to 3 shown)
[15:05:51] <icinga-miraheze>	 PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown)
[15:05:59] <icinga-miraheze>	 PROBLEM - db11 Puppet on db11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown)
[15:06:12] <icinga-miraheze>	 PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 509 failures. Last run 2 minutes ago with 509 failures. Failed resources (up to 3 shown)
[15:06:17] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown)
[15:06:29] <icinga-miraheze>	 PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown)
[15:06:40] <icinga-miraheze>	 PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown)
[15:06:48] <icinga-miraheze>	 PROBLEM - cp9 Puppet on cp9 is CRITICAL: CRITICAL: Puppet has 209 failures. Last run 2 minutes ago with 209 failures. Failed resources (up to 3 shown): File[antiguabarbudacalypso.com_private],File[spiral.wiki],File[spiral.wiki_private],File[permanentfuturelab.wiki]
[15:06:54] <icinga-miraheze>	 PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 494 failures. Last run 3 minutes ago with 494 failures. Failed resources (up to 3 shown)
[15:06:58] <icinga-miraheze>	 PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/home/southparkfan]
[15:07:00] <icinga-miraheze>	 PROBLEM - gluster2 Puppet on gluster2 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 4 minutes ago with 7 failures. Failed resources (up to 3 shown)
[15:07:01] <paladox>	 puppet works for me
[15:07:03] <icinga-miraheze>	 PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 512 failures. Last run 3 minutes ago with 512 failures. Failed resources (up to 3 shown)
[15:07:12] <RhinosF1>	 paladox: icinga does not agree with you
[15:07:14] <icinga-miraheze>	 PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 513 failures. Last run 3 minutes ago with 513 failures. Failed resources (up to 3 shown)
[15:07:20] <paladox>	 i just ran puppet so...
[15:07:26] <RhinosF1>	 They are 22 servers with puppet errors
[15:07:40] <RhinosF1>	 21 now
[15:07:43] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 508 failures. Last run 4 minutes ago with 508 failures. Failed resources (up to 3 shown)
[15:07:44] <paladox>	 root@mw4:/home/paladox# puppet agent -tv
[15:07:44] <paladox>	 xxxx
[15:07:44] <paladox>	 Notice: Applied catalog in 24.60 seconds
[15:07:45] <RhinosF1>	 Please check why
[15:07:54] <paladox>	 ><@paladox> i just ran puppet so...
[15:08:03] <RhinosF1>	 paladox: check why it's failed in logs
[15:08:05] <paladox>	 ><@paladox> puppet works for me
[15:08:10] <RhinosF1>	 Something must have happened
[15:08:14] <RhinosF1>	 Even if it works now
[15:08:22] <paladox>	 I don't need to check as it works.
[15:08:28] <paladox>	 i know why it failed
[15:08:52] <RhinosF1>	 Then say why it failed
[15:09:06] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 121 failures. Last run 2 minutes ago with 121 failures. Failed resources (up to 3 shown): File[antiguabarbudacalypso.com],File[antiguabarbudacalypso.com_private],File[spiral.wiki],File[spiral.wiki_private]
[15:09:10] <RhinosF1>	 If you know, say it failed because x, it will recover in y
[15:09:45] <paladox>	 I logged it earlier? upgrading restart the puppetserver & puppetdb
[15:09:56] <paladox>	 so puppet must have ran when the service was restarting?
[15:10:07] <RhinosF1>	 15:54:57 <paladox> !log upgrade puppet-agent puppetdb puppetdb-termini puppetserver - puppet2
[15:10:12] <RhinosF1>	 No mention of a restart
[15:10:13] <paladox>	 yup
[15:10:18] <paladox>	 the package does it
[15:10:19] <paladox>	 ...
[15:10:56] <paladox>	 anyways it works now :)
[15:11:08] <RhinosF1>	 paladox: next time, say when I ask the first time that you restarted it and yes it should recover as it next runs
[15:12:06] <paladox>	 ok
[15:12:27] <icinga-miraheze>	 RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[15:12:40] <icinga-miraheze>	 RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[15:12:54] <icinga-miraheze>	 RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[15:12:55] <icinga-miraheze>	 RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[15:12:56] <icinga-miraheze>	 RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[15:12:59] <icinga-miraheze>	 RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures
[15:12:59] <icinga-miraheze>	 RECOVERY - gluster2 Puppet on gluster2 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[15:13:01] <icinga-miraheze>	 RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[15:13:03] <icinga-miraheze>	 RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[15:13:13] <icinga-miraheze>	 RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[15:13:24] <RhinosF1>	 Yey
[15:13:27] <icinga-miraheze>	 RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:13:27] <icinga-miraheze>	 RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:13:35] <icinga-miraheze>	 RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:13:43] <icinga-miraheze>	 RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures
[15:13:46] <icinga-miraheze>	 RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[15:13:49] <icinga-miraheze>	 RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:14:00] <icinga-miraheze>	 RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:14:10] <icinga-miraheze>	 RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:14:16] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:14:47] <icinga-miraheze>	 RECOVERY - cp9 Puppet on cp9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:15:03] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[15:15:17] <paladox>	 why is varnish 14gb on cp8?
[15:15:18] <MH-Discord>	 <-CloudGuy38-> Ok when will BSW be deleted under complex way
[15:15:21] <paladox>	 err
[15:15:21] <paladox>	 cp3
[15:15:42] <RhinosF1>	 paladox: you just acked the alert. You're supposed to look.
[15:16:00] <RhinosF1>	 @-CloudGuy38-: when someone has free time
[15:16:50] <paladox>	 DAEMON_OPTS="-a :81 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s file,/var/lib/varnish/mediawiki/varnish_storage.bin,13G -p thread_pool_min=250 -p thread_pool_max=250 -p thread_pool_timeout=120 -p workspace_backend=128k -p timeout_idle=120 -p http_max_hdr=128"
[15:16:54] <MH-Discord>	 <-CloudGuy38-> ok
[15:19:33] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: CRITICAL: no terminated jobs
[15:19:48] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Databases db12 on bacula2 is CRITICAL: CRITICAL: no terminated jobs
[15:23:21] <icinga-miraheze>	 PROBLEM - lama.madooa.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lama.madooa.org' expires in 15 day(s) (Mon 03 Aug 2020 15:19:55 GMT +0000).
[15:26:54] <Not-3033>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJn0l
[15:26:55] <Not-3033>	 [02miraheze/mw-config] 07paladox 033447098 - Disable wgEnableSidebarCache It's causing unexpected issues. Bug: T5931
[15:26:57] <Not-3033>	 [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3
[15:26:58] <Not-3033>	 [02mw-config] 07paladox opened pull request 03#3180: Disable wgEnableSidebarCache - 13https://git.io/JJn08
[15:27:25] <Not-3033>	 [02mw-config] 07paladox closed pull request 03#3180: Disable wgEnableSidebarCache - 13https://git.io/JJn08
[15:27:27] <Not-3033>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJn04
[15:27:28] <Not-3033>	 [02miraheze/mw-config] 07paladox 03e3c7d76 - Disable wgEnableSidebarCache (#3180) It's causing unexpected issues. Bug: T5931
[15:27:30] <Not-3033>	 [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3
[15:27:31] <Not-3033>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1
[15:27:54] <RhinosF1>	 paladox: do we need to purge anything.
[15:28:10] <paladox>	 no, i wouldn't think so
[15:28:40] <travis-ci>	 miraheze/mw-config/paladox-patch-1/3447098 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/709509291
[15:29:13] <RhinosF1>	 paladox: ack
[15:31:14] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJn02
[15:31:15] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot 03b088d1e - Bot: Update SSL cert for lama.madooa.org
[15:43:18] <icinga-miraheze>	 RECOVERY - lama.madooa.org - LetsEncrypt on sslhost is OK: OK - Certificate 'lama.madooa.org' will expire on Fri 16 Oct 2020 14:31:07 GMT +0000.
[15:47:28] <icinga-miraheze>	 PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 6.05, 8.72, 4.84
[15:49:28] <icinga-miraheze>	 RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.88, 6.47, 4.50
[15:50:19] <Not-3033>	 [02puppet] 07paladox closed pull request 03#1455: Ircrcbot: change lineRate to 0.5 - 13https://git.io/JJnCn
[15:50:21] <Not-3033>	 [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://git.io/vbiAS
[15:50:23] <Not-3033>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11
[15:50:33] <RhinosF1>	 paladox: I litteraly said that won't help
[15:50:47] <RhinosF1>	 It's set to 1 to prevent it disconnecting
[15:50:53] <RhinosF1>	 During a flood
[15:50:54] <paladox>	 RhinosF1 i closed the pull?
[15:51:04] <RhinosF1>	 Oh
[15:51:06] <RhinosF1>	 Good
[15:51:23] <RhinosF1>	 I will try and fix that script if I can to be better
[16:11:36] <icinga-miraheze>	 PROBLEM - mon1 Puppet on mon1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 9 minutes ago with 0 failures
[16:23:44] <Jakeukalane>	 hello
[16:24:30] <Jakeukalane>	 is there a way to put two images centered in the same box, one to the side of the other?
[16:25:12] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJnze
[16:25:14] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot 034f9d736 - BOT: Updating services config for wikis
[16:48:22] <icinga-miraheze>	 PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 129s
[16:53:02] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.41, 5.94, 3.22
[16:54:56] <RhinosF1>	 paladox: why does that slave lag daily
[16:57:02] <icinga-miraheze>	 RECOVERY - db11 Current Load on db11 is OK: OK - load average: 4.41, 6.66, 4.27
[17:21:59] <paladox>	 SPF|Cloud would know why as replicas are his thing :)
[17:45:02] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.21, 19.68, 13.24
[17:47:56] <Not-3033>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JJnV4
[17:47:58] <Not-3033>	 [02miraheze/mw-config] 07Reception123 0318a0c30 - wmgTranslateDocumentationLanguageCode to qqq on metawiki requested
[17:47:59] <Not-3033>	 [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3
[17:48:03] <Not-3033>	 [02mw-config] 07Reception123 opened pull request 03#3181: wmgTranslateDocumentationLanguageCode to qqq on metawiki - 13https://git.io/JJnVR
[17:48:22] <icinga-miraheze>	 RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s
[17:49:03] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.94, 18.08, 14.21
[17:50:17] <MH-Discord>	 <Avengium (Ángel)> Hi
[17:55:59] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.81, 21.64, 17.27
[17:57:55] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.60, 18.36, 16.66
[18:01:17] <icinga-miraheze>	 PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates).
[18:03:04] <hispano76>	 hi
[18:03:53] <MH-Discord>	 <Avengium (Ángel)> miraheze was going slow, but it seems he's back
[18:04:04] <MH-Discord>	 <Avengium (Ángel)> * miraheze is ack
[18:04:09] <MH-Discord>	 <Avengium (Ángel)> tyop
[18:04:20] <hispano76>	 ok
[18:13:08] <Not-3033>	 [02mw-config] 07Reception123 closed pull request 03#3181: wmgTranslateDocumentationLanguageCode to qqq on metawiki - 13https://git.io/JJnVR
[18:13:09] <Not-3033>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJnw6
[18:13:11] <Not-3033>	 [02miraheze/mw-config] 07Reception123 0392615c6 - wmgTranslateDocumentationLanguageCode to qqq on metawiki (#3181) requested
[18:13:12] <Not-3033>	 [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3
[18:13:14] <Not-3033>	 [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1
[18:16:43] <RhinosF1>	 @MrJaroslavik: ^
[18:19:59] <MH-Discord>	 <MrJaroslavik> Thanks for the ping
[18:21:41] <MH-Discord>	 <MrJaroslavik> I still don't see documentation option.
[18:24:07] <RhinosF1>	 @MrJaroslavik: give it 10 mins
[18:39:06] <Not-3033>	 [02mw-config] 07Reception123 commented on commit 03e3c7d76b8e3ad724a7155582242461977968cce8 - 13https://git.io/JJnrr
[19:10:12] <RhinosF1>	 !log blacklist https://moviepedia.miraheze.org/wiki/Once_Upon_a in search console
[19:10:13] <MirahezeBot>	 [ Once Upon a - Moviepedia ] - moviepedia.miraheze.org
[19:10:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:10:29] <MH-Discord>	 <RhinosF1> @Cocopuff2018 ^
[19:11:45] <MH-Discord>	 <Cocopuff2018> Ok.
[19:11:58] <MH-Discord>	 <Cocopuff2018> Oh it was redicrected
[19:21:23] <RhinosF1>	 Ye, then deleted
[19:42:56] <MH-Discord>	 <MrJaroslavik> I don't see docs option. After cache clear and relog :(
[19:45:12] <MH-Discord>	 <MrJaroslavik> I am not familiar with this but shouldn't be there true instead of language?
[19:45:32] <MH-Discord>	 <MrJaroslavik> Reception123 RhinosF1 ^
[19:47:41] <Reception123>	 The WMF also has it as "qqq"
[19:47:57] <MH-Discord>	 <MrJaroslavik> Oh
[19:47:58] <RhinosF1>	 Hmm
[19:48:52] <MH-Discord>	 <MrJaroslavik> Hm, so we will check it tomorrow.
[19:53:28] <MH-Discord>	 <MrJaroslavik> Another. What do you think about "translate" button in sidebar? Yeh, i know about trend "Removing than adding".
[20:21:36] <MH-Discord>	 <Avengium (Ángel)> Thanks to Paladox for solving activate Extension:LinkTitles. Thanks @paladox  🎉 🥳 👍
[20:22:47] <MH-Discord>	 <Avengium (Ángel)> next one of my tickets is: T5890 Template Plantilla:Multimedia for tuscriaturas wiki , but i'm going to dinner. Until tomorrow
[22:01:34] <RhinosF1>	 !log run makeTestEdits.php to test RC Feed
[22:01:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:48:49] <RhinosF1>	 ^ all know
[23:12:19] <RhinosF1>	 hi
[23:12:23] <MH-Discord>	 <RhinosF1> hi
[23:12:34] <MH-Discord>	 <RhinosF1> relay restored