[01:06:04] I think... something went wrong out there.. https://prnt.sc/n6ofhr [01:06:05] Title: [ Screenshot by Lightshot ] - prnt.sc [01:06:54] comparative https://static.miraheze.org/mediatecawiki/6/62/Logo_Mediateca_Wiki.svg :| [02:40:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjIVU [02:40:13] [02miraheze/services] 07MirahezeSSLBot 03c0a84b1 - BOT: Updating services config for wikis [04:50:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjIwT [04:50:14] [02miraheze/services] 07MirahezeSSLBot 033e6a4e7 - BOT: Updating services config for wikis [06:34:36] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.2-apcu],Package[php7.2-redis] [06:41:41] PROBLEM - bacula1 Bacula Databases db4 on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:42:35] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:43:41] PROBLEM - bacula1 Bacula Databases db4 on bacula1 is WARNING: WARNING: Diff, 53489 files, 55.68GB, 2019-03-17 01:51:00 (2.5 weeks ago) [08:45:43] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 384 [08:51:39] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [08:55:36] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 717 [09:02:54] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:03:31] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [09:40:38] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Wed 05 Jun 2019 12:45:23 PM GMT +0000. [11:20:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjIXc [11:20:15] [02miraheze/services] 07MirahezeSSLBot 03df51cae - BOT: Updating services config for wikis [11:54:31] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 41% [11:56:31] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 27% [12:04:04] Nginx stop trying to be an attention-seeker [14:19:45] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:47] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown) [15:14:51] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 372 failures. Last run 2 minutes ago with 372 failures. Failed resources (up to 3 shown) [15:14:55] PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown) [15:14:59] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown) [15:15:05] PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown) [15:15:05] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 23 failures. Last run 2 minutes ago with 23 failures. Failed resources (up to 3 shown) [15:15:31] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 21 failures. Last run 2 minutes ago with 21 failures. Failed resources (up to 3 shown) [15:15:55] PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown) [15:16:11] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 377 failures. Last run 3 minutes ago with 377 failures. Failed resources (up to 3 shown) [15:16:15] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 377 failures. Last run 3 minutes ago with 377 failures. Failed resources (up to 3 shown) [15:16:20] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 3 minutes ago with 20 failures. Failed resources (up to 3 shown) [15:16:20] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown) [15:16:27] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 371 failures. Last run 3 minutes ago with 371 failures. Failed resources (up to 3 shown) [15:16:35] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 383 failures. Last run 3 minutes ago with 383 failures. Failed resources (up to 3 shown) [15:16:36] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 51 failures. Last run 4 minutes ago with 51 failures. Failed resources (up to 3 shown) [15:16:56] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 380 failures. Last run 3 minutes ago with 380 failures. Failed resources (up to 3 shown) [15:17:36] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Puppet has 233 failures. Last run 3 minutes ago with 233 failures. Failed resources (up to 3 shown): File[/etc/systemd/system/varnish.service],File[wildcard.miraheze.org],File[wildcard.miraheze.org_private],File[/etc/default/stunnel4] [15:17:38] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 371 failures. Last run 3 minutes ago with 371 failures. Failed resources (up to 3 shown): File[/etc/default/varnish],File[/etc/systemd/system/varnish.service],File[wildcard.miraheze.org],File[wildcard.miraheze.org_private] [15:21:04] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [15:23:31] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [15:23:55] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [15:24:04] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [15:24:08] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [15:24:19] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [15:24:20] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:35] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:36] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [15:24:47] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:51] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:55] RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:59] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:25:05] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:25:09] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [15:25:39] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [15:26:26] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:27:35] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:56:21] [02mw-config] 07joshwarecom opened pull request 03#2659: Update LocalSettings.php - 13https://git.io/fjI7p [18:58:32] [02mw-config] 07paladox commented on pull request 03#2659: Update LocalSettings.php - 13https://git.io/fjI5v [18:58:33] [02mw-config] 07paladox closed pull request 03#2659: Update LocalSettings.php - 13https://git.io/fjI7p [19:00:51] [02mw-config] 07joshwarecom commented on pull request 03#2659: Update LocalSettings.php - 13https://git.io/fjI5J [20:30:01] PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 7 minutes ago with 0 failures [22:40:53] [02miraheze/DataDump] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjIbR [22:40:54] [02miraheze/DataDump] 07paladox 03b34b80f - Revert "Rm mw.org link" This reverts commit d5e0c6cc269ba7b4bfe2b31e9778b4081ead44e5. [22:44:07] [02miraheze/DataDump] 07paladox pushed 031 commit to 03master [+0/-0/±0] 13https://git.io/fjIbu [22:44:09] [02miraheze/DataDump] 07paladox 03984813c - Merge remote-tracking branch 'upstream/master' * upstream/master: Revert "Rm mw.org link" Rm mw.org link