[00:04:58] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown) [00:05:10] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown) [00:05:38] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 332 failures. Last run 2 minutes ago with 332 failures. Failed resources (up to 3 shown) [00:05:44] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 289 failures. Last run 3 minutes ago with 289 failures. Failed resources (up to 3 shown) [00:06:49] !log restarted puppetserver on puppet1 [00:06:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:07:05] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 322 failures. Last run 3 minutes ago with 322 failures. Failed resources (up to 3 shown) [00:07:12] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Puppet has 300 failures. Last run 2 minutes ago with 300 failures. Failed resources (up to 3 shown): File[aman.info.tm],File[aman.info.tm_private],File[www.zenbuddhism.info],File[www.zenbuddhism.info_private] [00:07:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjecx [00:07:56] [02miraheze/puppet] 07paladox 03e323177 - Update to roundcubemail 1.4-rc1 [00:08:22] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 289 failures. Last run 1 minute ago with 289 failures. Failed resources (up to 3 shown): File[wiki.inebriation-confederation.com],File[wiki.inebriation-confederation.com_private],File[wiki.autocountsoft.com],File[wiki.autocountsoft.com_private] [00:08:32] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 291 failures. Last run 2 minutes ago with 291 failures. Failed resources (up to 3 shown): File[froggy.info],File[froggy.info_private],File[wiki.inebriation-confederation.com],File[wiki.inebriation-confederation.com_private] [00:08:38] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 295 failures. Last run 2 minutes ago with 295 failures. Failed resources (up to 3 shown): File[wiki.worlduniversityandschool.org],File[wiki.worlduniversityandschool.org_private],File[nonbinary.wiki],File[nonbinary.wiki_private] [00:09:29] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 291 failures. Last run 3 minutes ago with 291 failures. Failed resources (up to 3 shown): File[froggy.info],File[froggy.info_private],File[wiki.inebriation-confederation.com],File[wiki.inebriation-confederation.com_private] [00:12:50] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [00:12:59] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [00:13:35] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [00:14:57] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [00:15:00] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:15:36] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:16:44] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [00:16:46] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [00:17:28] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:18:12] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:25:47] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjeCL [00:25:49] [02miraheze/mw-config] 07paladox 03863a02f - Remove cp5 from wgSquidServers [01:27:19] PROBLEM - bacula1 Bacula Static on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:27:19] 5 03 [01:27:26] PROBLEM - bacula1 Disk Space on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:29:20] PROBLEM - bacula1 Bacula Static on bacula1 is WARNING: WARNING: Full, 3066142 files, 232.8GB, 2019-02-24 03:08:00 (2.6 weeks ago) [01:29:24] PROBLEM - bacula1 Disk Space on bacula1 is WARNING: DISK WARNING - free space: / 85188 MB (17% inode=99%); [03:45:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjelL [03:45:12] [02miraheze/services] 07MirahezeSSLBot 03bf34f7a - BOT: Updating services config for wikis [06:08:57] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 40271 MB (10% inode=96%); [08:59:35] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [09:07:32] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:36:36] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443],Exec[ufw-allow-tcp-from-185.52.1.75-to-any-port-81],Exec[ufw-allow-tcp-from-185.52.2.113-to-any-port-81] [12:32:37] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:46:36] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 3 minutes ago with 15 failures. Failed resources (up to 3 shown): Service[puppet],Service[rsyslog],Exec[ufw-default-deny],Service[ufw] [12:47:04] oh well we dont need any of that ^ [12:49:07] Reception123: are you able to nominate people for permissions at request for perms? [12:49:30] if that's a general question, then yeah [12:50:04] JohnLewis: yes [12:50:15] JohnLewis: I wasnt sure if you could nominate or if the user had to request [12:51:04] JohnLewis: ty [12:51:13] yw [12:54:36] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:16:36] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443],Exec[ufw-allow-tcp-from-185.52.1.75-to-any-port-81],Exec[ufw-allow-tcp-from-185.52.2.113-to-any-port-81] [13:18:42] !log adding swap to cp3 (256MB) [13:18:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:22:07] !log increasing swap to 512M on cp3 [13:26:35] !log [13:22:07] <+paladox> !log increasing swap to 512M on cp3 [13:29:08] paladox: always breaking crpa [13:29:10] crap* [13:29:14] heh [13:29:22] Zppix this time it was JohnLewis :D [13:29:56] paladox: sure... [13:30:16] heh [13:32:48] .uptime [13:32:48] I've been sitting here for 15 days, 17:41:20 and I keep going! [13:34:37] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:37:17] !log [13:22:07] <+paladox> !log increasing swap to 512M on cp3 [13:37:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:05:24] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 645 [15:15:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjeaQ [15:15:13] [02miraheze/services] 07MirahezeSSLBot 03ab4dd59 - BOT: Updating services config for wikis [15:24:07] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs] [15:24:20] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs] [15:24:25] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs] [15:24:59] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs] [15:25:55] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs] [15:26:37] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs] [15:27:48] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs] [15:28:14] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs] [15:28:38] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs] [15:29:30] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs] [15:32:07] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [15:32:20] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [15:32:25] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [15:32:58] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [15:33:55] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [15:34:37] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:36:00] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:36:11] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:36:46] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:37:26] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:45:24] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [16:13:24] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 370 [16:15:24] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [16:39:30] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs] [16:47:27] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:49:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjerI [16:49:24] [02miraheze/puppet] 07paladox 03a48451c - Disable nginx monitoring for elasticsearch1 https [16:51:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjer3 [16:51:44] [02miraheze/puppet] 07paladox 03edf331d - Monitor elasticsearch-lb.miraheze.org [16:52:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjers [16:52:35] [02miraheze/puppet] 07paladox 035cb7043 - Update elasticsearch.pp [17:07:21] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjeri [17:07:23] [02miraheze/mw-config] 07paladox 0348c11df - Disable flow on allthetropeswiki if using cli [17:54:00] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjeoR [17:54:02] [02miraheze/mw-config] 07MacFan4000 0357d8128 - Update Sitenotice.php [17:56:12] [02mw-config] 07The-Voidwalker opened pull request 03#2650: spelling - 13https://git.io/fjeoE [17:57:50] [02mw-config] 07paladox closed pull request 03#2650: spelling - 13https://git.io/fjeoE [17:57:51] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjeoz [17:57:53] [02miraheze/mw-config] 07The-Voidwalker 0389bf0fa - spelling (#2650) [18:13:55] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjeoi [18:13:57] [02miraheze/mw-config] 07paladox 0326044e3 - Update Sitenotice.php [19:33:20] * Hispano76 saluda [19:36:19] "!Miraheze is currently low on storage for uploads. Because of this most upload attempts will fail. We apologize for this." - why? [19:43:29] Its this image uploads only or even pages are effected? [20:00:33] BurningPrincess, image uploads only [20:04:56] Thanks. [20:20:33] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:26:02] * BurningPrincess offers Voidwalker a hug [20:28:46] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:37:05] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:37:33] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:51:43] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:58:01] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:34:27] Hello julientm! If you have any questions feel free to ask and someone should answer soon. [23:11:07] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjeXe [23:11:09] [02miraheze/mw-config] 07paladox 03c69876b - Update Sitenotice.php [23:12:18] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 52449 MB (14% inode=96%);