[00:10:07] bd808: if phab search were as good as google [00:10:24] we would do that much rarely :) [00:10:34] I didn't even look this time ;) [00:10:54] I sometimes remember seeing a ticket years ago but it takes me forever to find them [01:26:49] search on Phabricator, well, let's just say it's worse than in MediaWiki ... [02:16:56] come online to fix one issue and then stumble onto another... [02:20:39] in case anyone else asks, codesearch is currently down because gerrit-replica is down: https://phabricator.wikimedia.org/T247182 [04:30:53] !log wm-bot Deleted jessie instances wm-bot-pg and wm-bot2 (T236564) [04:30:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wm-bot/SAL [04:30:55] T236564: "wm-bot" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236564 [11:59:14] !log tools.totoazero qdeled job 2029555 maj_articles_recents stuck since few weeks [11:59:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.totoazero/SAL [12:03:47] Hi, some of my grid job started by cron are regularly stuck for weeks, and thus -once param blocks new ones to be created. Any hint to automatically kill one-day old jobs or such? [12:04:55] preferably smarter than creating another job to kill older ones, and have this one stuck too :) [19:04:44] !log openstack restart apache2 to clear up horizon error on labweb1002 T247199 [19:04:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [19:04:48] T247199: Temporarily got HTTP 500 when trying to log into or switch projects in Horizon - https://phabricator.wikimedia.org/T247199 [21:08:29] anyone else @here having trouble with kubectl on labs? [21:09:11] runtime: failed to create new OS thread (have 39 already; errno=11) [21:09:23] (my ulimit is 100 i think, so it's not that) [21:09:37] but just pages and pages of go stack traces [21:10:23] errno 11 seems like maybe there's some degraded state? [21:12:16] ok fix is here: https://phabricator.wikimedia.org/T219070 (GOMAXPROCS=1) [21:14:00] wait, nvm, worked a couple times now kubectl just hangs