[01:32:31] Hey, can someone approve https://toolsadmin.wikimedia.org/tools/membership/status/511 please? [01:52:57] MaxSem, Bryan has done it [01:53:14] Thanks! [08:34:56] Hello, many of my Grid Crontab entries lately started to produce one of the following errors (sent to me by email): [08:35:00] OSError: [Errno 12] Cannot allocate memory [08:35:12] MemoryError [08:35:31] It started like 14 day ago and never occured before [08:35:35] What should I do? [08:37:17] (Whole tracebacks here: https://pastebin.com/uMRYKV5x) [08:39:16] It happens even for simple python scripts, that usually take miliseconds [08:39:42] Should I fill in a task or is it something temporary? [08:45:33] From 10 tasks last week 6 or 7 ended with this [08:46:38] Oh, bd is away? Better fill in a task then [08:53:39] Fixed paste:https://pastebin.com/WUgxTRQg [10:23:06] !log tools.quickcategories UPDATE command SET command_status = 0 WHERE command_batch = 236 AND command_status = 16; -- reset two commands that for some reason never ran from pending to planned [10:23:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [10:28:32] is anyone familiar with k8s online, whom I could ask a few questions? [10:42:04] (my issue reported here: https://phabricator.wikimedia.org/T222255) [11:25:28] !log tools.quickcategories UPDATE command SET command_status = 0 WHERE command_batch IN (38, 40, 41, 42, 43, 44, 115, 128, 129, 170) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned [11:25:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [11:32:48] !log tools.quickcategories UPDATE command SET command_status = 0 WHERE command_batch IN (123) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned [11:32:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [11:44:11] Dvorapa did you try increasing memory limit? [11:46:19] I will let you know when I see Dvorapa and I will deliver that message to them [11:46:19] @notify Dvorapa I replied in T222255 [11:46:19] T222255: Cron gridengine jobs are failing for memory errors - https://phabricator.wikimedia.org/T222255 [12:28:16] !log tools.quickcategories deployed fdfaa14666 [12:28:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [14:00:57] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @James_F & @mooeypoo - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:50:56] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @James_F & @mooeypoo - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:23:19] !log ores-support-checklist Ran "webservice --backend=kubernetes python restart" [15:23:19] halfak: Unknown project "ores-support-checklist" [15:23:19] halfak: Did you mean to say "tools.ores-support-checklist" instead? [15:23:24] Woops [15:23:28] !log tools.ores-support-checklist Ran "webservice --backend=kubernetes python restart" [15:23:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ores-support-checklist/SAL [15:23:34] <3 stashbot [17:53:14] !log deployment-prep deploying ores:52e9759 [17:53:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [20:55:47] !log tools.awmd-stats Deploying change set 52fe399 live on Toolforge [20:55:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.awmd-stats/SAL [20:56:32] !log tools.awmd-stats Restarting tool to pick-up latest deployed changes [20:56:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.awmd-stats/SAL [20:59:00] !log tools.awmd-stats Tool up and running, functioning as expected after the restart... [20:59:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.awmd-stats/SAL [21:33:56] !log tools.quickcategories deployed 7d2d1992bd (batch titles) [21:33:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [23:28:12] !log tools.quickcategories deployed 13006c471b (cache titles/summaries) [23:28:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL