[00:05:27] <icinga-miraheze>	 PROBLEM - cornetto.online - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[00:07:26] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 3490 files, 6.356MB, 2019-03-10 00:05:00 (1.1 weeks ago)
[01:47:41] <Hispano76>	 hi
[03:03:25] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 508
[03:11:25] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[03:33:43] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown)
[03:34:02] <icinga-miraheze>	 PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown)
[03:34:08] <icinga-miraheze>	 PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown)
[03:34:09] <icinga-miraheze>	 PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown)
[03:34:22] <icinga-miraheze>	 PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown)
[03:34:23] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown)
[03:34:54] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 355 failures. Last run 3 minutes ago with 355 failures. Failed resources (up to 3 shown)
[03:35:11] <icinga-miraheze>	 PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown)
[03:35:15] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 3 minutes ago with 30 failures. Failed resources (up to 3 shown)
[03:35:18] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown)
[03:35:33] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 196 failures. Last run 2 minutes ago with 196 failures. Failed resources (up to 3 shown): File[antiguabarbudacalypso.com],File[antiguabarbudacalypso.com_private],File[spiral.wiki],File[spiral.wiki_private]
[03:35:42] <icinga-miraheze>	 PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Puppet has 124 failures. Last run 2 minutes ago with 124 failures. Failed resources (up to 3 shown): File[/etc/nginx/fastcgi_params],File[antiguabarbudacalypso.com],File[antiguabarbudacalypso.com_private],File[spiral.wiki]
[03:35:57] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 316 failures. Last run 3 minutes ago with 316 failures. Failed resources (up to 3 shown): File[spiral.wiki_private],File[permanentfuturelab.wiki],File[permanentfuturelab.wiki_private],File[private.revi.wiki]
[03:36:30] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 358 failures. Last run 2 minutes ago with 358 failures. Failed resources (up to 3 shown): File[antiguabarbudacalypso.com_private],File[spiral.wiki],File[spiral.wiki_private],File[permanentfuturelab.wiki]
[03:37:01] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 362 failures. Last run 2 minutes ago with 362 failures. Failed resources (up to 3 shown): File[/home/nagiosre],File[/etc/nginx/mime.types],File[/etc/nginx/fastcgi_params],File[antiguabarbudacalypso.com]
[03:37:48] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 357 failures. Last run 3 minutes ago with 357 failures. Failed resources (up to 3 shown): File[spiral.wiki],File[spiral.wiki_private],File[permanentfuturelab.wiki],File[permanentfuturelab.wiki_private]
[03:38:44] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 355 failures. Last run 4 minutes ago with 355 failures. Failed resources (up to 3 shown): File[permanentfuturelab.wiki],File[permanentfuturelab.wiki_private],File[private.revi.wiki],File[private.revi.wiki_private]
[03:42:07] <icinga-miraheze>	 RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[03:43:11] <icinga-miraheze>	 RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[03:43:14] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[03:43:17] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[03:43:43] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:44:00] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[03:44:01] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:44:09] <icinga-miraheze>	 RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:44:22] <icinga-miraheze>	 RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:44:23] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:44:54] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:45:32] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:45:41] <icinga-miraheze>	 RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:45:59] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[03:46:36] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[03:46:57] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[03:47:09] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[05:25:18] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs]
[05:33:17] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:43:26] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 853
[06:55:10] <Not-5be1>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjvVN
[06:55:11] <Not-5be1>	 [02miraheze/services] 07MirahezeSSLBot 032149d51 - BOT: Updating services config for wikis
[07:01:25] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[07:43:26] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 561
[07:51:25] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[07:55:30] <Not-5be1>	 [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/fjvwu
[07:55:31] <Not-5be1>	 [02miraheze/MirahezeMagic] 07translatewiki 034fa3988 - Localisation updates from https://translatewiki.net.
[07:55:33] <Not-5be1>	 [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjvwz
[07:55:34] <Not-5be1>	 [02miraheze/IncidentReporting] 07translatewiki 03fab7277 - Localisation updates from https://translatewiki.net.
[12:14:18] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 40262 MB (10% inode=96%);
[15:40:13] <Not-5be1>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjvD5
[15:40:14] <Not-5be1>	 [02miraheze/services] 07MirahezeSSLBot 03d2a351a - BOT: Updating services config for wikis
[15:53:04] * Hispano76 saluda
[15:53:17] <Voidwalker>	 hi
[15:53:33] <Hispano76>	 :] 
[16:59:53] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[17:00:35] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[17:00:51] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[17:00:57] <paladox>	 known ^^
[17:01:09] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[17:07:57] <JohnLewis>	 hm?
[17:09:10] <paladox>	 JohnLewis upstream maint
[17:09:13] <paladox>	 gerrit's down
[17:09:21] <JohnLewis>	 we shouldn't be using gerrit
[17:10:20] <paladox>	 ok
[17:11:01] <JohnLewis>	 and if it turns out we are... that needs to be changed asap
[17:12:25] <paladox>	 JohnLewis we are for one submodule only
[17:12:35] <paladox>	 https://github.com/miraheze/mediawiki/blob/REL1_32/.gitmodules#L390
[17:12:35] <ZppixBot>	 Title: [ mediawiki/.gitmodules at REL1_32 · miraheze/mediawiki · GitHub ] - github.com
[17:13:44] <Not-5be1>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjv9m
[17:13:46] <Not-5be1>	 [02miraheze/mediawiki] 07paladox 03a5b48b0 - Switch Editcount to GitHub mirror
[17:20:18] <JohnLewis>	 Awesome
[17:48:02] <paladox>	 !log restarting puppetserver && puppetdb on puppet1
[17:48:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:56:58] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:05:12] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[18:07:51] <Not-5be1>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjvH0
[18:07:53] <Not-5be1>	 [02miraheze/mediawiki] 07paladox 035745a7a - Fix link
[18:09:13] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[18:17:13] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[18:18:01] <paladox>	 JohnLewis we may have to revert the change
[20:44:44] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[20:45:36] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[20:46:06] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:46:32] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:35:13] <icinga-miraheze>	 PROBLEM - lizardfs1 Disk Space on lizardfs1 is CRITICAL: DISK CRITICAL - free space: / 16895 MB (10% inode=98%);
[21:36:20] <icinga-miraheze>	 PROBLEM - lizardfs2 Disk Space on lizardfs2 is CRITICAL: DISK CRITICAL - free space: / 16895 MB (10% inode=98%);
[21:57:25] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 1037
[22:05:18] <ZppixBot>	 Hello flyingcat! If you have any questions feel free to ask and someone should answer soon.
[22:05:47] <flyingcat>	 Hi, anyone here? I wasn't able to get into the Discord, so I thought I'd try the IRC channel.
[22:06:35] <Voidwalker>	 hi
[22:06:42] <Voidwalker>	 what's up?
[22:07:25] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[22:10:45] <flyingcat>	 I asked some questions here: https://meta.miraheze.org/wiki/Talk:Miraheze#Some_questions Was wondering if someone could give me answers on the kinds of dumps that can be imported.
[22:10:45] <ZppixBot>	 Title: [ Talk:Miraheze - Miraheze Meta ] - meta.miraheze.org
[22:12:12] <Voidwalker>	 we can use SQL dumps, though I'm not yet certain how you can import abuse filter configuration
[22:13:09] <Voidwalker>	 I've been working on setting up my test environment, so I can hopefully get back to you on how it can be done (without importing individual filters)
[22:13:13] <flyingcat>	 Abuse filter info can be dumped in XML and SQL dumps. Wikia doesn't provide it, but I can use scripts to get it into an intermediate wiki and dump that. You just have to handle it.
[22:14:30] <Voidwalker>	 if it's something that's normally included by the software, then it is something we can import
[22:15:54] <flyingcat>	 Also what's with Discord? It says error 522.
[22:16:10] <flyingcat>	 Thanks, good to know.
[22:18:24] <Voidwalker>	 trying to join the server?
[22:19:56] <flyingcat>	 I was going to, but I can't get there. It loads for a while and then comes up with a Cloudflare error.
[22:20:17] <Voidwalker>	 hmm, try using this invite https://discord.gg/GwmmCY
[22:20:18] <ZppixBot>	 Title: [ Miraheze ] - discord.gg
[22:21:23] <Voidwalker>	 if that doesn't work, then it's something on discord's side
[22:23:32] <JohnLewis>	 !log set ns_content = 1 in defaults for ns_namespace_id = 0
[22:23:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:26:07] <flyingcat>	 Thanks Voidwalker, that worked.
[23:05:55] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs]
[23:13:57] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:27:08] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[23:28:10] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[23:28:36] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[23:28:48] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[23:35:19] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[23:36:26] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:36:44] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:37:07] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures