[01:22:57] <icinga-miraheze>	 PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.80, 1.46, 1.20
[01:24:57] <icinga-miraheze>	 RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.91, 1.28, 1.16
[03:35:54] <icinga-miraheze>	 RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp10.miraheze.org
[05:39:59] <Reception123>	 !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki creepypastowiki --username-prefix="wikia:creepypasto" /home/reception/escreepypasto_pages_full.xml
[05:40:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:39:58] <MH-Discord>	 <GR (he/him) [SocDemWiki]> Sorry @Doug , didnt see the message. Its literally me opening a page, editing or previewing. No import.
[09:41:53] <icinga-miraheze>	 PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com
[11:23:44] <icinga-miraheze>	 PROBLEM - you.r-fit.cc - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:30:49] <icinga-miraheze>	 RECOVERY - you.r-fit.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'you.r-fit.cc' will expire on Fri 02 Jul 2021 05:49:20 GMT +0000.
[13:03:24] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 5.28, 3.80, 1.91
[13:05:24] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.70, 3.02, 1.86
[14:08:32] <dmehus>	 @GR (he/him) [SocDemWiki], hrm, strange... :(
[14:24:18] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 388.85 ms
[14:26:22] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 303.11 ms
[14:28:26] <icinga-miraheze>	 RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 257.26 ms
[14:35:24] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.90, 4.93, 3.92
[14:37:24] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.41, 4.37, 3.84
[15:21:32] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.41, 6.18, 3.28
[15:25:32] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.31, 2.97, 2.62
[15:43:54] <icinga-miraheze>	 RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp11.miraheze.org
[15:47:40] <Not-818c>	 [02miraheze/puppet] 07dependabot[bot] pushed 031 commit to 03dependabot/bundler/modules/graylog/puppet-6.13.0 [+0/-0/±1] 13https://git.io/JOYh5
[15:47:42] <Not-818c>	 [02miraheze/puppet] 07dependabot[bot] 03e2604be - build(deps): bump puppet from 4.4.2 to 6.13.0 in /modules/graylog
[15:47:43] <Not-818c>	 [02puppet] 07dependabot[bot] opened pull request 03#1739: build(deps): bump puppet from 4.4.2 to 6.13.0 in /modules/graylog - 13https://git.io/JOYhd
[15:47:45] <Not-818c>	 [02puppet] 07dependabot[bot] created branch 03dependabot/bundler/modules/graylog/puppet-6.13.0 - 13https://git.io/vbiAS
[15:47:46] <Not-818c>	 [02puppet] 07dependabot[bot] labeled pull request 03#1739: build(deps): bump puppet from 4.4.2 to 6.13.0 in /modules/graylog - 13https://git.io/JOYhd
[16:03:07] <paladox>	 why is dependabot updating puppet huh
[16:10:57] <RhinosF1>	 Someone must have it on
[17:04:51] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.70, 6.68, 5.08
[17:06:51] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.10, 5.99, 5.01
[17:30:07] <Not-818c>	 [02ManageWiki] 07Reception123 closed pull request 03#267: A little maintenance to make things easier to read - 13https://git.io/JOLvc
[17:30:09] <Not-818c>	 [02miraheze/ManageWiki] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JOOnZ
[17:30:10] <Not-818c>	 [02miraheze/ManageWiki] 07Universal-Omega 039dd2e64 - A little maintenance to make things easier to read (#267)
[17:31:09] <miraheze-github>	 miraheze/ManageWiki - Reception123 the build passed.
[17:36:00] <Not-818c>	 [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JOOc3
[17:36:02] <Not-818c>	 [02miraheze/mediawiki] 07Reception123 03497b265 - Update ManageWiki
[17:59:31] <Not-818c>	 [02CreateWiki] 07Reception123 closed pull request 03#208: [BUG FIX] fix logging when private is false (0) - 13https://git.io/JOLCw
[17:59:36] <Not-818c>	 [02miraheze/CreateWiki] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JOOlo
[17:59:37] <Not-818c>	 [02miraheze/CreateWiki] 07Universal-Omega 03d0515da - [BUG FIX] fix logging when private is false (0) (#208)
[18:00:47] <miraheze-github>	 miraheze/CreateWiki - Reception123 the build passed.
[18:04:48] <Not-818c>	 [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JOO8C
[18:04:50] <Not-818c>	 [02miraheze/mediawiki] 07Reception123 036dc2b5d - Update CreateWiki
[18:11:16] <icinga-miraheze>	 PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[18:11:59] <icinga-miraheze>	 PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[18:12:24] <icinga-miraheze>	 PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[18:12:52] <icinga-miraheze>	 PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[18:13:06] <icinga-miraheze>	 PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[18:13:09] <icinga-miraheze>	 PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[18:23:14] <Not-818c>	 [02mw-config] 07Reception123 closed pull request 03#3830: Reorder extension sections - 13https://git.io/JOLtF
[18:23:15] <Not-818c>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JOOR0
[18:23:17] <Not-818c>	 [02miraheze/mw-config] 07Universal-Omega 03d4c919c - Reorder extension sections (#3830)
[18:25:21] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[18:36:25] <icinga-miraheze>	 RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[18:36:51] <icinga-miraheze>	 RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[18:37:05] <icinga-miraheze>	 RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[18:37:12] <icinga-miraheze>	 RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[18:37:15] <icinga-miraheze>	 RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:38:01] <icinga-miraheze>	 RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:47:56] <icinga-miraheze>	 PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com
[21:24:46] <MH-Discord>	 <MrJaroslavik> It's possible to see source code of  MirahezeBot's module that's used in cvt-feed?
[21:26:00] <dmehus>	 MrJaroslavik, yes, most of the code is public
[21:26:52] <MH-Discord>	 <MrJaroslavik> Yes, but i didn't found this part, maybe i am blind.
[21:27:07] <Universal_Omega>	 I don't think that module is. Could be wrong though.
[21:27:10] <dmehus>	 @MrJaroslavik, it should be at https://github.com/MirahezeBots/sopel-CVTFeed (or most of it anyway)
[21:27:10] <MirahezeBot>	 [ GitHub - MirahezeBots/sopel-CVTFeed: miraheze cvt feed plugin for sopel ] - github.com
[21:27:34] <dmehus>	 Universal_Omega, it wasn't when it was EK's SigmaBotMH code
[21:27:43] <dmehus>	 but MacFan4000 rewrote it
[21:27:45] <Universal_Omega>	 Oh interesting I didn't know that.
[21:27:48] <Universal_Omega>	 Thanks 
[21:27:49] <dmehus>	 and simplified it quite a bit
[21:27:52] <dmehus>	 np
[21:28:15] <Universal_Omega>	 Oh so it's only been public for 12 days. That's why I didn't know that.
[21:28:21] <dmehus>	 yeah
[21:28:25] <dmehus>	 exactly :)
[21:34:41] <MH-Discord>	 <MrJaroslavik> Ooh, i searched for "cvtpattern" on discord and know how it works. Thanks
[21:46:49] <dmehus>	 ah, okay, cool :)
[21:47:54] <icinga-miraheze>	 RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp11.miraheze.org
[21:56:54] <icinga-miraheze>	 PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com
[22:19:34] <MH-Discord>	 <MrJaroslavik> I originally thought there are some "actions" like block/lock/abuse log defined in the code. And looks like it's keyword-based.
[22:55:07] <icinga-miraheze>	 PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.12, 2.55, 1.66
[22:59:06] <icinga-miraheze>	 RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.69, 1.57, 1.45
[23:06:16] <icinga-miraheze>	 PROBLEM - runzeppelin.ru - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for runzeppelin.ru could not be found
[23:06:16] <icinga-miraheze>	 PROBLEM - wiki.tulpa.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.tulpa.info could not be found
[23:06:17] <icinga-miraheze>	 PROBLEM - wikien.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikien.wildterra2.com could not be found
[23:06:17] <icinga-miraheze>	 PROBLEM - wiki.mcpirevival.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mcpirevival.tk could not be found
[23:06:18] <icinga-miraheze>	 PROBLEM - www.dariawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.dariawiki.org could not be found
[23:07:06] <icinga-miraheze>	 PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static]
[23:07:29] <icinga-miraheze>	 PROBLEM - mw9 Check Gluster Clients on mw9 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs'
[23:13:05] <icinga-miraheze>	 RECOVERY - wiki.tulpa.info - reverse DNS on sslhost is OK: rDNS OK - wiki.tulpa.info reverse DNS resolves to cp10.miraheze.org
[23:13:06] <icinga-miraheze>	 RECOVERY - wikien.wildterra2.com - reverse DNS on sslhost is OK: rDNS OK - wikien.wildterra2.com reverse DNS resolves to cp11.miraheze.org
[23:13:10] <icinga-miraheze>	 RECOVERY - www.dariawiki.org - reverse DNS on sslhost is OK: rDNS OK - www.dariawiki.org reverse DNS resolves to cp11.miraheze.org
[23:13:11] <dmehus>	 MrJaroslavik, yeah, the actions used to be defined in the code, afaik, and the formatting was a little different, but was a lot more opaque as the code wasn't public
[23:13:14] <icinga-miraheze>	 RECOVERY - runzeppelin.ru - reverse DNS on sslhost is OK: rDNS OK - runzeppelin.ru reverse DNS resolves to cp11.miraheze.org
[23:13:16] <icinga-miraheze>	 RECOVERY - wiki.mcpirevival.tk - reverse DNS on sslhost is OK: rDNS OK - wiki.mcpirevival.tk reverse DNS resolves to cp10.miraheze.org
[23:13:20] <SPF|Cloud>	 mw9 mount broken?
[23:13:41] <dmehus>	 oh
[23:13:56] <dmehus>	 hrm
[23:14:09] <dmehus>	 did we do any changes to mediawiki-static today?
[23:14:27] <SPF|Cloud>	 not that I can see
[23:14:34] <SPF|Cloud>	 the daemon OOM'd
[23:14:39] <dmehus>	 ah
[23:14:44] <SPF|Cloud>	 but puppet cannot remount due to another error
[23:15:06] <darkmatterman450>	 Are you referring to those GitHub puppets?
[23:16:26] <SPF|Cloud>	 huh...
[23:17:00] <SPF|Cloud>	 !log glusterfs process crashed on mw9 due to OOM, puppet could not reconnect because 'mkdir: /mnt/mediawiki-static' already exists
[23:17:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:17:03] <darkmatterman450>	 SPF|Cloud Sorry, I just didn't know what puppet you were talking about.
[23:17:21] <SPF|Cloud>	 !log fixed glusterfs on mw9 by running umount -l /mnt/mediawiki-static
[23:17:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:17:29] <icinga-miraheze>	 RECOVERY - mw9 Check Gluster Clients on mw9 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs'
[23:17:31] <SPF|Cloud>	 darkmatterman450: there's no such thing as a second puppet
[23:17:48] <darkmatterman450>	 Second puppet? :/
[23:18:00] <SPF|Cloud>	 there's only one type of 'puppet', that is our puppet setup
[23:18:21] <SPF|Cloud>	 we don't rely on a second type of configuration or so
[23:18:32] <SPF|Cloud>	 and I don't know what you meant with 'GitHub puppets'
[23:18:58] <darkmatterman450>	 I thought the 'puppet' was from GitHub. That's why I asked.
[23:33:05] <dmehus>	 SPF|Cloud, okay if I triage https://phabricator.miraheze.org/T7134 as `high`, or should I use `normal`?
[23:33:05] <MirahezeBot>	 [ ⚓ T7134 Puppet cannot remount GlusterFS mount if directory exists ] - phabricator.miraheze.org
[23:33:27] <SPF|Cloud>	 I'll let John triage it
[23:33:33] <dmehus>	 okay, sounds good
[23:34:23] <SPF|Cloud>	 Infrastructure is his department, since this seems to be a very rare situation, I don't see a need to invoke an emergency action here (ie 'drop all work')
[23:34:44] <SPF|Cloud>	 but it's good practice to track these errors in phabricator
[23:35:09] <icinga-miraheze>	 RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:36:00] <dmehus>	 SPF|Cloud, yeah
[23:40:51] <Not-818c>	 [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JOOpJ
[23:40:53] <Not-818c>	 [02miraheze/puppet] 07Southparkfan 038e399fc - Readd PHP-FPM slow log
[23:44:30] <SPF|Cloud>	 why is the git branch on puppet3 set to 'patch-1' instead of 'master'?
[23:45:16] <SPF|Cloud>	 !log puppet3: switch from patch-1 to master branch for puppet git repo
[23:45:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:48:07] <dmehus>	 no clue heh
[23:48:25] <dmehus>	 maybe related to migration to new infrastructure a month or two ago?
[23:48:34] <dmehus>	 never got switched back?
[23:49:00] <SPF|Cloud>	 seems to be p.aladox' ATS work
[23:49:08] <dmehus>	 ah
[23:49:37] <SPF|Cloud>	 I like your way of thinking, you have an idea for every error :P