[00:03:38] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:05:05] [02puppet] 07paladox closed pull request 03#905: Redirect *.miraheze.com -> *.miraheze.org - 13https://git.io/fpEtv [00:05:07] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [00:05:09] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [00:10:45] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/fpopW [00:10:46] [02miraheze/MirahezeMagic] 07paladox 039f97f5e - Add basic GenerateMirahezeSitemap php script to execute generateSitemap.php [00:11:39] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [00:11:54] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fpop8 [00:11:55] [02miraheze/mediawiki] 07paladox 032db05b2 - Update MirahezeMagic [00:16:11] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpopu [00:16:13] [02miraheze/MirahezeMagic] 07paladox 03f943b0a - Remove double quotes [00:16:34] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fpopg [00:16:35] [02miraheze/mediawiki] 07paladox 037e22bf0 - Update MirahezeMagic [00:17:36] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fpopa [00:17:38] [02miraheze/mediawiki] 07paladox 0372bc869 - Update MirahezeMagic [00:18:19] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/fpopV [00:18:20] [02miraheze/MirahezeMagic] 07paladox 032b04cf2 - Rename generateMirahezeSitemap.hpp to generateMirahezeSitemap.php [00:18:47] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fpopo [00:18:48] [02miraheze/mediawiki] 07paladox 03f274a45 - Update MirahezeMagic [00:19:26] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpopK [00:19:28] [02miraheze/MirahezeMagic] 07paladox 036fdd51d - Update generateMirahezeSitemap.php [00:19:59] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fpopi [00:20:01] [02miraheze/mediawiki] 07paladox 034c285b0 - Update MirahezeMagic [00:35:56] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpopd [00:35:57] [02miraheze/MirahezeMagic] 07paladox 038a63776 - Fix script [00:37:24] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fpopN [00:37:26] [02miraheze/mediawiki] 07paladox 038b78d81 - Update MirahezeMagic [00:38:36] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpohv [00:38:37] [02miraheze/MirahezeMagic] 07paladox 03011d40f - use single quotes instead of double quotes. [00:40:05] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fpohT [00:40:07] [02miraheze/mediawiki] 07paladox 030e35f4b - Update MirahezeMagic [01:11:20] [02puppet] 07MacFan4000 opened pull request 03#914: Add me to puppet-users - 13https://git.io/fpoj1 [01:12:39] [02puppet] 07JohnFLewis closed pull request 03#914: Add me to puppet-users - 13https://git.io/fpoj1 [01:12:40] [02miraheze/puppet] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fpoj9 [01:12:42] [02miraheze/puppet] 07MacFan4000 03b4578a3 - Add me to puppet-users [01:12:43] [02miraheze/puppet] 07JohnFLewis 03e14c01d - Merge pull request #914 from MacFan4000/patch-26 Add me to puppet-users [01:35:15] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [01:35:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fpKeH [01:35:18] [02miraheze/puppet] 07paladox 0317657e7 - Redirect /robots.txt to /robots.php [01:35:19] [02puppet] 07paladox opened pull request 03#915: Redirect /robots.txt to /robots.php - 13https://git.io/fpKeQ [01:35:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fpKed [01:35:50] [02miraheze/puppet] 07paladox 03f55b2e2 - Update init.pp [01:35:52] [02puppet] 07paladox synchronize pull request 03#915: Redirect /robots.txt to /robots.php - 13https://git.io/fpKeQ [01:39:12] [02puppet] 07MacFan4000 opened pull request 03#916: Add sudo rule for puppet-users - 13https://git.io/fpKeh [01:39:42] [02puppet] 07MacFan4000 edited pull request 03#916: Add sudo rule for puppet-users - 13https://git.io/fpKeh [01:42:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-1/±0] 13https://git.io/fpKvf [01:42:23] [02miraheze/puppet] 07paladox 032f170fe - Update and rename robots.txt to robots.php [01:42:25] [02puppet] 07paladox synchronize pull request 03#915: Redirect /robots.txt to /robots.php - 13https://git.io/fpKeQ [01:46:44] [02puppet] 07paladox synchronize pull request 03#915: Redirect /robots.txt to /robots.php - 13https://git.io/fpKeQ [01:46:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fpKvI [01:46:47] [02miraheze/puppet] 07paladox 03217f139 - Update mediawiki.conf [01:47:02] [02puppet] 07paladox closed pull request 03#915: Redirect /robots.txt to /robots.php - 13https://git.io/fpKeQ [01:47:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±2] 13https://git.io/fpKvL [01:47:05] [02miraheze/puppet] 07paladox 0363d4ca6 - Redirect /robots.txt to /robots.php (#915) * Redirect /robots.txt to /robots.php * Update init.pp * Update and rename robots.txt to robots.php * Update mediawiki.conf [01:47:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [01:47:08] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [01:48:25] paladox: you've killed the comments [01:48:34] hmm? [01:48:35] oh [01:48:35] comments are an integral part of robots.txts [01:49:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKvt [01:49:36] [02miraheze/puppet] 07paladox 03b664f5c - Update robots.php [01:49:50] JohnLewis done [01:53:31] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKvG [01:53:33] [02miraheze/MirahezeMagic] 07paladox 030b378d7 - Fix script [01:54:14] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fpKvc [01:54:15] [02miraheze/mediawiki] 07paladox 032ac4551 - Update MirahezeMagic [01:55:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKv8 [01:55:57] [02miraheze/puppet] 07paladox 033af746c - Add newline [01:57:53] [02puppet] 07MacFan4000 closed pull request 03#916: Add sudo rule for puppet-users - 13https://git.io/fpKeh [02:01:51] [02puppet] 07MacFan4000 synchronize pull request 03#916: Add sudo rule for puppet-users - 13https://git.io/fpKeh [02:01:53] [02puppet] 07MacFan4000 reopened pull request 03#916: Add sudo rule for puppet-users - 13https://git.io/fpKeh [02:03:16] !log /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/generateMirahezeSitemap.php on mw1 in a screen [02:03:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:20:48] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [02:25:34] [02puppet] 07MacFan4000 synchronize pull request 03#916: Add sudo rule for puppet-users - 13https://git.io/fpKeh [02:27:33] [02puppet] 07MacFan4000 synchronize pull request 03#916: Add sudo rule for puppet-users - 13https://git.io/fpKeh [02:34:51] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [02:34:53] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fpKfG [02:34:55] [02miraheze/ssl] 07paladox 03dace6da - Add www.netazar.org.key ssl certificate [02:34:56] [02ssl] 07paladox opened pull request 03#104: Add www.netazar.org.key ssl certificate - 13https://git.io/fpKfZ [02:35:22] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/fpKfC [02:35:24] [02miraheze/ssl] 07paladox 034056460 - Create www.netazar.org.crt [02:35:25] [02ssl] 07paladox synchronize pull request 03#104: Add www.netazar.org.key ssl certificate - 13https://git.io/fpKfZ [02:35:47] [02ssl] 07paladox synchronize pull request 03#104: Add www.netazar.org.key ssl certificate - 13https://git.io/fpKfZ [02:35:48] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fpKfR [02:35:50] [02miraheze/ssl] 07paladox 03cf89dd0 - Update certs.yaml [02:37:53] [02ssl] 07paladox closed pull request 03#104: Add www.netazar.org.key ssl certificate - 13https://git.io/fpKfZ [02:37:54] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fpKfV [02:37:56] [02miraheze/ssl] 07paladox 03545173c - Add www.netazar.org.key ssl certificate (#104) * Add www.netazar.org.key ssl certificate * Create www.netazar.org.crt * Update certs.yaml [02:37:57] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [02:37:59] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [02:41:32] [02puppet] 07MacFan4000 closed pull request 03#916: Add sudo rule for puppet-users - 13https://git.io/fpKeh [02:44:05] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [02:44:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fpKfH [02:44:08] [02miraheze/puppet] 07paladox 03c168a96 - Add a "generate sitemaps for all wikis" cron And run it every saturday [02:44:09] [02puppet] 07paladox opened pull request 03#917: Add a "generate sitemaps for all wikis" cron - 13https://git.io/fpKfQ [02:44:30] [02puppet] 07paladox edited pull request 03#917: Add a "generate sitemaps for all wikis" cron fixes T1765 - 13https://git.io/fpKfQ [02:44:34] [02puppet] 07paladox closed pull request 03#917: Add a "generate sitemaps for all wikis" cron fixes T1765 - 13https://git.io/fpKfQ [02:44:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKfd [02:44:37] [02miraheze/puppet] 07paladox 0355a86b7 - Add a "generate sitemaps for all wikis" cron (#917) Add a "generate sitemaps for all wikis" cron fixes T1765 [02:44:38] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [02:44:39] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [04:20:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKUu [04:20:15] [02miraheze/services] 07MirahezeSSLBot 03a37e6f6 - BOT: Updating services config for wikis [06:24:59] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 6535 [06:25:15] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.2-gettext] [06:33:13] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:00:16] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKLz [08:00:17] [02miraheze/services] 07MirahezeSSLBot 03dffdd26 - BOT: Updating services config for wikis [08:05:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKLK [08:05:13] [02miraheze/services] 07MirahezeSSLBot 036e03a9b - BOT: Updating services config for wikis [08:46:39] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [10:05:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKqQ [10:05:12] [02miraheze/services] 07MirahezeSSLBot 030c5ba83 - BOT: Updating services config for wikis [12:24:42] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 3211 [13:32:40] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [15:12:40] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 360 [15:32:39] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [15:52:24] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 403 [15:54:20] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [16:09:44] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://git.io/fpKla [16:09:46] [02miraheze/ssl] 07MacFan4000 031646d5d - Create ta.gyaanipedia.co.in.crt [16:09:47] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://git.io/vxP9L [16:11:03] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://git.io/fpKlr [16:11:05] [02miraheze/ssl] 07MacFan4000 0337a6531 - Update certs.yaml [16:11:34] [02ssl] 07MacFan4000 opened pull request 03#105: Mac fan4000 patch 1 - 13https://git.io/fpKli [16:11:44] [02ssl] 07MacFan4000 closed pull request 03#105: Mac fan4000 patch 1 - 13https://git.io/fpKli [16:11:46] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fpKlP [16:11:47] [02miraheze/ssl] 07MacFan4000 03bb7e5ff - Mac fan4000 patch 1 (#105) * Create ta.gyaanipedia.co.in.crt * Update certs.yaml [16:27:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpK8R [16:27:43] [02miraheze/puppet] 07paladox 03dd6711c - Update prometheus-gdnsd-stats [16:31:29] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 654 [17:11:47] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://git.io/vxP9L [17:11:49] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [17:12:41] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://git.io/vxP9L [17:12:43] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://git.io/fpKBo [17:12:44] [02miraheze/ssl] 07MacFan4000 0308caae7 - add a cert [17:13:30] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://git.io/fpKBi [17:13:32] [02miraheze/ssl] 07MacFan4000 033355e3b - Update certs.yaml [17:14:11] [02ssl] 07MacFan4000 opened pull request 03#106: add a cert - 13https://git.io/fpKBP [17:14:17] [02ssl] 07MacFan4000 closed pull request 03#106: add a cert - 13https://git.io/fpKBP [17:14:18] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fpKBX [17:14:20] [02miraheze/ssl] 07MacFan4000 037d78e80 - add a cert (#106) * add a cert * Update certs.yaml [17:14:24] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [17:14:25] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://git.io/vxP9L [17:24:02] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[pa.gyaanipedia.co.in] [17:24:04] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[pa.gyaanipedia.co.in] [17:24:08] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[pa.gyaanipedia.co.in] [17:24:14] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[pa.gyaanipedia.co.in] [17:24:42] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[pa.gyaanipedia.co.in] [17:25:01] uh [17:25:15] I pushed priv and pub [17:25:18] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [17:26:12] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[pa.gyaanipedia.co.in] [17:28:49] MacFan4000: is the key correct? [17:28:55] *crt [17:29:42] The name is incorrect MacFan4000 [17:29:46] http/ [17:29:51] Is not the file path [17:30:24] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/fpKRE [17:30:25] [02miraheze/ssl] 07paladox 033da5aa0 - Fix file path [17:30:49] yep just found that [17:31:44] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [17:32:30] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:33:52] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:39:57] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:40:13] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [17:41:55] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [17:42:13] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:02:28] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:02:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fpK0E [18:02:31] [02miraheze/puppet] 07paladox 031eb5ef5 - Convert prometheus-gdnsd-stats to python3 [18:02:32] [02puppet] 07paladox opened pull request 03#918: Convert prometheus-gdnsd-stats to python3 - 13https://git.io/fpK0u [18:02:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fpK0g [18:02:53] [02miraheze/puppet] 07paladox 03a06f959 - Update node_gdnsd.pp [18:02:55] [02puppet] 07paladox synchronize pull request 03#918: Convert prometheus-gdnsd-stats to python3 - 13https://git.io/fpK0u [18:03:02] [02puppet] 07paladox closed pull request 03#918: Convert prometheus-gdnsd-stats to python3 - 13https://git.io/fpK0u [18:03:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fpK02 [18:03:05] [02miraheze/puppet] 07paladox 0386e650d - Convert prometheus-gdnsd-stats to python3 (#918) * Convert prometheus-gdnsd-stats to python3 * Update node_gdnsd.pp [18:03:07] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:03:08] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [18:14:19] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:19:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpK0F [18:19:30] [02miraheze/puppet] 07paladox 0339d38ec - Update init.pp [18:22:15] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:55:59] PROBLEM - ns1 GDNSD Datacenters on ns1 is UNKNOWN: NRPE: Unable to read output [18:56:25] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [18:59:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:00:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fpKul [19:00:30] [02miraheze/puppet] 07paladox 034a8ae7f - Convert check_gdnsd_datacenters.py to python3 ref T3647 [19:00:32] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [19:00:33] [02puppet] 07paladox opened pull request 03#919: Convert check_gdnsd_datacenters.py to python3 ref T3647 - 13https://git.io/fpKu8 [19:01:09] [02puppet] 07paladox closed pull request 03#919: Convert check_gdnsd_datacenters.py to python3 ref T3647 - 13https://git.io/fpKu8 [19:01:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKu4 [19:01:12] [02miraheze/puppet] 07paladox 03edc0b0a - Convert check_gdnsd_datacenters.py to python3 ref T3647 (#919) [19:01:13] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [19:01:15] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [19:02:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKuu [19:02:16] [02miraheze/puppet] 07paladox 036a06b6f - Update check_gdnsd_datacenters.py [19:02:26] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:03:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKuz [19:03:02] [02miraheze/puppet] 07paladox 03031c859 - Update check_gdnsd_datacenters.py [19:25:39] PROBLEM - bacula1 Bacula Private Git on bacula1 is UNKNOWN: NRPE: Unable to read output [19:26:15] PROBLEM - bacula1 Bacula Static on bacula1 is UNKNOWN: NRPE: Unable to read output [19:26:29] PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is UNKNOWN: NRPE: Unable to read output [19:26:35] PROBLEM - bacula1 Bacula Databases db4 on bacula1 is UNKNOWN: NRPE: Unable to read output [19:29:47] PROBLEM - bacula1 Puppet on bacula1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [19:34:07] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 3205 [20:26:30] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [20:26:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fpK2L [20:26:33] [02miraheze/puppet] 07paladox 033c82b25 - Convert check_bacula_backups to python3 [20:26:34] [02puppet] 07paladox opened pull request 03#920: Convert check_bacula_backups to python3 - 13https://git.io/fpK2t [20:40:05] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [20:56:24] RECOVERY - bacula1 Bacula Static on bacula1 is OK: OK: Full, 2691928 files, 200.5GB, 2018-12-02 20:22:00 (34.4 minutes ago) [20:56:36] RECOVERY - bacula1 Bacula Private Git on bacula1 is OK: OK: Full, 2076 files, 3.355MB, 2018-12-02 20:53:00 (3.6 minutes ago) [20:56:40] [02puppet] 07paladox synchronize pull request 03#920: Convert check_bacula_backups to python3 - 13https://git.io/fpK2t [20:56:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fpKaI [20:56:42] RECOVERY - bacula1 Bacula Phabricator Static on bacula1 is OK: OK: Full, 70959 files, 4.347GB, 2018-12-02 20:42:00 (14.7 minutes ago) [20:56:43] [02miraheze/puppet] 07paladox 030c3f95a - Update check_bacula_backups [20:57:04] RECOVERY - bacula1 Bacula Databases db4 on bacula1 is OK: OK: Full, 910097 files, 63.20GB, 2018-12-02 02:46:00 (18.2 hours ago) [20:57:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fpKat [20:57:28] [02miraheze/puppet] 07paladox 032e48c33 - Update director.pp [20:57:30] [02puppet] 07paladox synchronize pull request 03#920: Convert check_bacula_backups to python3 - 13https://git.io/fpK2t [20:58:24] [02puppet] 07paladox closed pull request 03#920: Convert check_bacula_backups to python3 - 13https://git.io/fpK2t [20:58:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fpKaq [20:58:27] [02miraheze/puppet] 07paladox 0382c2464 - Convert check_bacula_backups to python3 (#920) * Convert check_bacula_backups to python3 * Update check_bacula_backups * Update director.pp [20:59:58] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:00:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [21:00:18] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:04:43] PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is UNKNOWN: NRPE: Unable to read output [21:04:47] PROBLEM - bacula1 Bacula Private Git on bacula1 is UNKNOWN: NRPE: Unable to read output [21:05:11] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.37, 1.96, 1.51 [21:05:15] PROBLEM - bacula1 Bacula Databases db4 on bacula1 is UNKNOWN: NRPE: Unable to read output [21:06:41] RECOVERY - bacula1 Bacula Phabricator Static on bacula1 is OK: OK: Full, 70959 files, 4.347GB, 2018-12-02 20:42:00 (24.7 minutes ago) [21:06:47] RECOVERY - bacula1 Bacula Private Git on bacula1 is OK: OK: Full, 2076 files, 3.355MB, 2018-12-02 20:53:00 (13.8 minutes ago) [21:07:07] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 0.63, 1.42, 1.36 [21:07:13] RECOVERY - bacula1 Bacula Databases db4 on bacula1 is OK: OK: Full, 910097 files, 63.20GB, 2018-12-02 02:46:00 (18.4 hours ago) [21:16:57] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:16:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fpKVv [21:17:00] [02miraheze/puppet] 07paladox 0346f3580 - check_bacula_backup: Migrate from OptionParser to argparse.ArgumentParser OptionParser is currently deprecated [21:17:09] [02puppet] 07paladox opened pull request 03#921: check_bacula_backup: Migrate from OptionParser to argparse.ArgumentParser - 13https://git.io/fpKVf [21:25:03] [02puppet] 07paladox closed pull request 03#921: check_bacula_backup: Migrate from OptionParser to argparse.ArgumentParser - 13https://git.io/fpKVf [21:25:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKV4 [21:25:06] [02miraheze/puppet] 07paladox 031ed88ba - check_bacula_backup: Migrate from OptionParser to argparse.ArgumentParser (#921) OptionParser is currently deprecated [21:25:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [21:25:09] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:31:21] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:31:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fpKV1 [21:31:24] [02miraheze/puppet] 07paladox 0390aa294 - Convert check_varnishbackends.py to python3 [21:31:25] [02puppet] 07paladox opened pull request 03#922: Convert check_varnishbackends.py to python3 - 13https://git.io/fpKVM [21:31:30] PROBLEM - cp4 Puppet on cp4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [21:31:51] [02puppet] 07paladox closed pull request 03#922: Convert check_varnishbackends.py to python3 - 13https://git.io/fpKVM [21:31:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKVy [21:31:54] [02miraheze/puppet] 07paladox 031a17f9b - Convert check_varnishbackends.py to python3 (#922) [21:31:55] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:31:57] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [21:33:30] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:38:53] PROBLEM - cp4 Varnish Backends on cp4 is UNKNOWN: NRPE: Unable to read output [21:40:47] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [21:41:27] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:41:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fpKwt [21:41:30] [02miraheze/puppet] 07paladox 0391a61a6 - check_varnishbackends: Convert optparse.OptionParser to argparse.ArgumentParser [21:41:32] [02puppet] 07paladox opened pull request 03#923: check_varnishbackends: Convert optparse.OptionParser to argparse.Argu… - 13https://git.io/fpKwq [21:41:40] [02puppet] 07paladox edited pull request 03#923: check_varnishbackends: Convert optparse.OptionParser to argparse.Argu… - 13https://git.io/fpKwq [21:41:44] [02puppet] 07paladox edited pull request 03#923: check_varnishbackends: Convert optparse.OptionParser to argparse.ArgumentParser - 13https://git.io/fpKwq [21:45:39] [02puppet] 07paladox synchronize pull request 03#923: check_varnishbackends: Convert optparse.OptionParser to argparse.ArgumentParser - 13https://git.io/fpKwq [21:45:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fpKwl [21:45:42] [02miraheze/puppet] 07paladox 0395c6637 - Update check_varnishbackends.py [21:45:46] [02puppet] 07paladox closed pull request 03#923: check_varnishbackends: Convert optparse.OptionParser to argparse.ArgumentParser - 13https://git.io/fpKwq [21:45:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKw8 [21:45:49] [02miraheze/puppet] 07paladox 035e1adfd - check_varnishbackends: Convert optparse.OptionParser to argparse.ArgumentParser (#923) * check_varnishbackends: Convert optparse.OptionParser to argparse.ArgumentParser * Update check_varnishbackends.py [21:45:50] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:45:52] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [21:49:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKwE [21:49:20] [02miraheze/puppet] 07paladox 03bc54b8e - Update init.pp [21:53:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKwX [21:53:33] [02miraheze/puppet] 07paladox 03ef64860 - Switch to python3 for salt [22:32:46] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 3.41, 1.90, 1.06 [22:34:46] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 0.66, 1.36, 0.97 [22:44:21] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKoD [22:44:22] [02miraheze/dns] 07paladox 032e53d28 - Update admin_state [22:46:07] PROBLEM - bacula1 Bacula Daemon on bacula1 is CRITICAL: PROCS CRITICAL: 1 process with UID = 110 (bacula) [22:46:57] PROBLEM - bacula1 Bacula Static on bacula1 is UNKNOWN: NRPE: Unable to read output [22:47:09] PROBLEM - bacula1 Bacula Databases db4 on bacula1 is UNKNOWN: NRPE: Unable to read output [22:48:07] RECOVERY - bacula1 Bacula Daemon on bacula1 is OK: PROCS OK: 2 processes with UID = 110 (bacula) [22:48:55] RECOVERY - bacula1 Bacula Static on bacula1 is OK: OK: Full, 2691928 files, 200.5GB, 2018-12-02 20:22:00 (2.4 hours ago) [22:49:07] RECOVERY - bacula1 Bacula Databases db4 on bacula1 is OK: OK: Full, 910097 files, 63.20GB, 2018-12-02 02:46:00 (20.1 hours ago) [22:50:10] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKoh [22:50:12] [02miraheze/dns] 07paladox 030075ef8 - Revert "Update admin_state" This reverts commit 2e53d286abc37268eb4aee8f4e49e2649dde119e. [22:53:30] [02puppet] 07paladox synchronize pull request 03#912: Create a DB account for internetarchivebot - 13https://git.io/fpovz [22:53:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fpKKT [22:53:33] [02miraheze/puppet] 07paladox 03882a3d1 - Update mediawiki-grants.sql.erb [22:57:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fpKKq [22:57:25] [02miraheze/puppet] 07paladox 03e81e3c5 - Make mirahezerenewssl use python3 [22:57:27] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [22:57:28] [02puppet] 07paladox opened pull request 03#924: Make mirahezerenewssl use python3 - 13https://git.io/fpKKm [22:57:49] [02puppet] 07paladox closed pull request 03#924: Make mirahezerenewssl use python3 - 13https://git.io/fpKKm [22:57:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKK3 [22:57:52] [02miraheze/puppet] 07paladox 03edbbd52 - Make mirahezerenewssl use python3 (#924) [22:57:54] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [22:57:55] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [23:00:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKKC [23:00:42] [02miraheze/ssl] 07MirahezeSSLBot 035ce2c92 - Bot: Update SSL cert for antiguabarbudacalypso.com [23:00:55] the renewal bot works! [23:40:56] PROBLEM - cp5 Varnish Backends on cp5 is UNKNOWN: NRPE: Unable to read output [23:41:04] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [23:42:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:42:56] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [23:43:02] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:45:47] !log depool mw1 [23:45:58] HTTP CRITICAL - Unable to open TCP socket [23:46:14] PROBLEM - cp4 Varnish Backends on cp4 is UNKNOWN: NRPE: Unable to read output [23:46:22] PROBLEM - cp2 Varnish Backends on cp2 is UNKNOWN: NRPE: Unable to read output [23:46:56] PROBLEM - cp5 Varnish Backends on cp5 is UNKNOWN: NRPE: Unable to read output [23:48:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [23:48:56] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. b'mw1' [23:50:28] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:50:57] PROBLEM - cp5 Varnish Backends on cp5 is UNKNOWN: NRPE: Unable to read output [23:51:59] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 22922 bytes in 0.008 second response time [23:52:15] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [23:53:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKi8 [23:53:56] [02miraheze/puppet] 07paladox 03f44cd5d - Update check_varnishbackends.py [23:54:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [23:56:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:56:17] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [23:56:57] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [23:58:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpKi0 [23:58:08] [02miraheze/puppet] 07paladox 03271caf6 - Update check_varnishbackends.py