[06:04:25] http://recommend.wmflabs.org/ is down. Can anyone look at it? [07:00:45] "Internal Server Error" nice error page [07:52:31] Cupid: :) [09:25:11] is down also for me [10:24:45] kart_: how important is that? [10:25:42] arturo: https://phabricator.wikimedia.org/T190014 [10:26:39] ok thanks Cupid do you have any more info about that system? any docs that could point me towards a fix? [10:26:47] not really [10:27:09] what I found on wikitect via search points the site towards a research thing [10:27:17] I can't find anything in wikitech [10:27:25] could you share the link? [10:28:07] https://wikitech.wikimedia.org/wiki/Research <--- this one? [10:28:29] ye [10:29:11] dsaez: this seems to be related to research, true? [11:02:18] !log tools reboot tools-exec-1408, to balance load. Server is unresponsive due to high load by some tools [11:02:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:37:24] !log recommendation-api reboot tool.recommendation-api.eqiad.wmflabs for T190014 [11:37:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Recommendation-api/SAL [11:37:26] T190014: https://recommend.wmflabs.org is down - https://phabricator.wikimedia.org/T190014 [12:08:31] arturo: I added https://phabricator.wikimedia.org/project/profile/2128/ [12:09:38] k Nemo_bis [12:14:32] cheers arturo thanks for looking into that but yeah not much we can do :) [12:15:26] np [13:56:39] !log phabricator recreating phabricator-stretch [13:56:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL [14:08:48] has anyone noticed that if you delete an instance and recreate the instance a week and a bit later, it still has the same puppet class and hiera values? [14:08:53] (using the same name) [14:24:35] paladox: will test that [14:24:41] chicocvenancio thanks [14:39:08] paladox: chicocvenancio I believe that's a known bug w/ a task somewhere [14:39:17] ah [14:39:20] thanks [15:25:21] !log recreated phabricator-stretch as phabricator-stretch2 then as phabricator-stretch3 [15:25:21] paladox: Unknown project "recreated" [15:25:30] !log phabricator recreated phabricator-stretch as phabricator-stretch2 then as phabricator-stretch3 [15:25:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL [15:46:31] !log phabricator recreating phabricator-stretch3 as phabricator-stretch4 [15:46:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL [16:42:59] !log deploy-prep ORES-beta has been down since January! [16:43:00] awight: Unknown project "deploy-prep" [16:43:21] !log deployment-prep ORES-beta has been down since January. [16:43:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [16:45:02] !log deployment-prep Put ORES-beta back onto master branch [16:45:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [16:57:32] !log deployment-prep ORES beta service is restored. [16:57:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [19:53:46] Krinkle: hey I put up patches for T189284 and tagged you and Ian. Let me know if/when that's all good to go! thank you [19:53:46] T189284: Stop serving slowparse logs from dumps distribution servers - https://phabricator.wikimedia.org/T189284 [19:54:32] legoktm: hey :) could you weigh in on https://phabricator.wikimedia.org/T189284#4057666 when you get a chance please? thank you! [20:03:53] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/420420 (owner: 10L10n-bot)