[00:03:06] PROBLEM - Puppet on misc3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:36] PROBLEM - Puppet on nfs1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:38] PROBLEM - Puppet on cp5 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:06] PROBLEM - Puppet on db4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:16] PROBLEM - Puppet on ns1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:24] Gc [00:04:26] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:56] PROBLEM - dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate dariawiki.org expires in 15 day(s) (Thu 12 Jul 2018 12:01:16 AM GMT +0000). [00:05:08] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4Qfy [00:05:10] [02miraheze/ssl] 07MirahezeSSLBot 0321fe5f6 - Bot: Update SSL cert for dariawiki.org [00:11:00] RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate dariawiki.org will expire on Sun 23 Sep 2018 11:05:01 PM GMT +0000. [00:11:20] RECOVERY - Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [00:11:40] RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [00:11:42] RECOVERY - Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [00:12:08] RECOVERY - Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:10] RECOVERY - Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:28] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:13:38] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [00:13:40] RECOVERY - Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:18:11] PROBLEM - kunwok.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate kunwok.org expires in 15 day(s) (Thu 12 Jul 2018 12:14:14 AM GMT +0000). [00:18:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4QJ9 [00:18:24] [02miraheze/ssl] 07MirahezeSSLBot 03e5c8ea8 - Bot: Update SSL cert for kunwok.org [00:22:03] RECOVERY - kunwok.org - LetsEncrypt on sslhost is OK: OK - Certificate kunwok.org will expire on Sun 23 Sep 2018 11:18:18 PM GMT +0000. [00:28:33] PROBLEM - pl.nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate pl.nonbinary.wiki expires in 15 day(s) (Thu 12 Jul 2018 12:25:48 AM GMT +0000). [00:28:49] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4QUR [00:28:51] [02miraheze/ssl] 07MirahezeSSLBot 036e933e6 - Bot: Update SSL cert for pl.nonbinary.wiki [00:29:17] PROBLEM - penalwiki.awiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate penalwiki.awiki.org expires in 15 day(s) (Thu 12 Jul 2018 12:26:58 AM GMT +0000). [00:32:27] RECOVERY - pl.nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate pl.nonbinary.wiki will expire on Sun 23 Sep 2018 11:28:39 PM GMT +0000. [00:41:18] RECOVERY - penalwiki.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate penalwiki.awiki.org will expire on Sun 23 Sep 2018 11:29:21 PM GMT +0000. [00:55:19] !log disabled https://phabricator.miraheze.org/H34 [00:55:21] Title: [ ☿ Autoclose requests for default extensions/skins ] - phabricator.miraheze.org [00:55:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:06:10] * NeoBogan coats MacFan4000 in #ffb2d0 paint [01:23:37] PROBLEM - Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-185.52.1.75-to-any-port-81] [01:24:07] again [01:24:08] meh [01:25:11] !log rebooting cp4 to clear ram [01:25:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:26:02] how you tried restarting varnish by any chance? [01:27:35] JohnLewis i tryed that before, it just dosen't free any of the ram [01:31:37] RECOVERY - Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [01:51:17] PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2504 [01:51:19] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2504 [02:04:37] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [02:11:53] how many servers, total, power miraheze’s operational/production infrastructure? [02:12:36] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:14:08] NeoBogan hmm [02:14:10] * paladox looks [02:14:39] NeoBogan 15 [02:14:52] that includes test1 [02:15:18] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [02:35:15] 15 servers, okay... [02:35:41] for an average day, you lot have an awful lot of warnings, errors, criticalities, and recoveries [02:55:17] PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5234 [02:55:19] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5234 [03:49:16] RECOVERY - JobQueue on mw2 is OK: JOBQUEUE OK - job queue below 300 jobs [03:49:18] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [05:51:02] Hello Centrist16! If you have any questions feel free to ask and someone should answer soon. [05:52:00] Does anyone here have experience with XML editor on Inkscape [06:02:01] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4Qn7 [06:02:03] [02miraheze/mw-config] 07Reception123 03bd2546f - license for akancyclopediawiki per T3265 resolves T3265 [09:58:06] Anyone that is authorized by Miraheze rules to lock accounts for impersonation please lock https://harrypotter.miraheze.org/wiki/Special:CentralAuth/FANDOMbot. It appears to be an impersonation of Wikia staff [09:58:15] Title: [ Global account information for FANDOMbot. - Harry Potter Wiki ] - harrypotter.miraheze.org [10:09:18] PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2437 [10:09:26] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2437 [10:33:18] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [10:57:05] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4QP2 [10:57:06] [02miraheze/mw-config] 07MacFan4000 036caf963 - Also search main ns on metawiki [11:09:11] MacFan4000: your last commit is redundant [11:23:33] [02miraheze/ManageWiki] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/f4QMk [11:23:35] [02miraheze/ManageWiki] 07JohnFLewis 039e93b8e - add to ManageWiki forms [11:23:36] [02miraheze/ManageWiki] 07JohnFLewis 0396afefe - Merge branch 'master' of github.com:miraheze/ManageWiki [11:24:21] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4QMY [11:24:22] [02miraheze/mw-config] 07JohnFLewis 032565bdc - Revert "Also search main ns on metawiki" This reverts commit 6caf963123c8d6f1473b54cd58333203e821d6c4. [11:27:15] But we weren’t searching the main NS by default [11:27:41] we are [11:28:10] Whenever I search the title of a main ns page, it never comes up, it just suggests pages in the user NS [11:28:17] JohnLewis: ^^ [11:28:32] I always get main ns [11:28:38] I never do [11:31:40] https://www.mediawiki.org/wiki/Manual:$wgNamespacesToBeSearchedDefault#Default_value [11:31:40] Title: [ Manual:$wgNamespacesToBeSearchedDefault - MediaWiki ] - www.mediawiki.org [11:32:55] Ok [11:44:37] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [11:52:37] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:48:07] PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.82, 1.52, 1.14 [12:50:07] RECOVERY - Current Load on misc1 is OK: OK - load average: 1.56, 1.42, 1.14 [13:23:27] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2625 [14:17:18] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [14:31:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f47vQ [14:31:06] [02miraheze/puppet] 07paladox 0391d055e - Increase fixed-size to 16gb cp4 and cp2 have 25gb and cp5 has 24.5gb. There's enough storage left for us to increase this to 16gb [15:14:38] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [15:22:37] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:58:02] [02mw-config] 07paladox synchronize pull request 03#2270: Switch to Swift for file uploads / viewing files - 13https://git.io/f4SjT [15:58:04] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/f47O1 [15:58:05] [02miraheze/mw-config] 07paladox 0375059b8 - Update LocalSettings.php [16:06:43] [02mw-config] 07paladox synchronize pull request 03#2270: Switch to Swift for file uploads / viewing files - 13https://git.io/f4SjT [16:06:46] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/f473y [16:06:48] [02miraheze/mw-config] 07paladox 037303bfb - Update FileBackend.php [16:13:11] [02mw-config] 07paladox synchronize pull request 03#2270: Switch to Swift for file uploads / viewing files - 13https://git.io/f4SjT [16:13:13] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/f47s5 [16:13:14] [02miraheze/mw-config] 07paladox 030f42333 - Update LocalSettings.php [16:14:52] [02mw-config] 07paladox closed pull request 03#2270: Switch to Swift for file uploads / viewing files - 13https://git.io/f4SjT [16:14:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/f47GT [16:14:55] [02miraheze/mw-config] 07paladox 039e583b2 - Switch to Swift for file uploads / viewing files (#2270) * Switch to Swift for file uploads / viewing files * Create FileBackend.php * Update FileBackend.php * consistency * Update LocalSettings.php * Update FileBackend.php * Update LocalSettings.php [16:15:16] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [16:15:18] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [16:38:15] hello! [16:38:57] hi [17:27:50] [02mw-config] 07thelevking opened pull request 03#2271: Newest by default in Flow / movestable for autoconfirmed in isvwiki - 13https://git.io/f474t [17:29:06] [02mw-config] 07paladox closed pull request 03#2271: Newest by default in Flow / movestable for autoconfirmed in isvwiki - 13https://git.io/f474t [17:29:07] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/f4743 [17:29:09] [02miraheze/mw-config] 07thelevking 03f891ce5 - Newest by default in Flow / movestable for autoconfirmed in isvwiki (#2271) * Sort Flow topics by newest by default * movestable for autoconfirmed in isvwiki [17:37:17] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 1686 [17:37:27] PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 1684 [17:51:18] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [17:51:28] RECOVERY - JobQueue on mw2 is OK: JOBQUEUE OK - job queue below 300 jobs [21:54:37] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [22:02:38] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:07:46] Someone could help me? the sidebar does not look good [22:07:53] https://pirategalaxy.miraheze.org/wiki/Pirate_Galaxy_Wiki [22:07:54] Title: [ Pirate Galaxy Wiki ] - pirategalaxy.miraheze.org [22:35:07] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f475h [22:35:08] [02miraheze/ManageWiki] 07paladox 03990a60d - Update populateWikiSettings.php [23:04:38] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [23:12:20] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f47bt [23:12:22] [02miraheze/mw-config] 07paladox 03cbe19f0 - Call swift backend miraheze-swift instead of relying on $wgDBname [23:12:36] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:19:27] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5273 [23:19:29] PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5270 [23:21:29] woulden't it be fun if someone wrote a script and just added one word to each user users page on all the wiki's i wonder how many jobs there would be [23:21:30] jk