[08:29:03] !log tools cleanup disk in tools-sgebastion-09, was full of debug logs and unused apt packages [08:29:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:20:58] !log tools disable puppet in all servers to livehack tools-puppetmaster-01 to test T221225 [10:21:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:21:01] T221225: sssd integration needs to be updated to include sudo config from LDAP support - https://phabricator.wikimedia.org/T221225 [10:27:52] !log tools T221225 reboot tool-sgebastion-09 for testing sssd [10:27:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:27:55] T221225: sssd integration needs to be updated to include sudo config from LDAP support - https://phabricator.wikimedia.org/T221225 [10:42:16] Hi, login.tools.wmflabs.org says "The last Puppet run was at Mon Apr 29 10:10:24 UTC 2019 (30 minutes ago). Puppet is disabled. arturo T22122". Task T22122 is a) resolved b) irrelevant to Toolforge. What's up? :-) [10:42:16] T22122: Use of and tags in galleries - https://phabricator.wikimedia.org/T22122 [10:44:58] probably a missing digit in the task number? [10:45:29] Lucas_WMDE, true, but I don't know what the missing digit is :-) [10:45:34] me neither :) [10:45:49] that's why I'm asking in -cloud :-) [10:45:50] but T221122 is restricted, might be that one [10:45:52] (or might no, no idea) [10:45:58] let’s hope someone else knows ^^ [10:50:13] it's T221225, yes, missing digit :-P [10:50:14] T221225: sssd integration needs to be updated to include sudo config from LDAP support - https://phabricator.wikimedia.org/T221225 [11:11:57] thanks arturo [11:16:41] yw [11:22:16] !log tools T221225 re-enable puppet agent in all toolforge servers [11:22:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:22:18] T221225: sssd integration needs to be updated to include sudo config from LDAP support - https://phabricator.wikimedia.org/T221225 [13:20:21] !log tools.awmd-stats Deploying 0bc3c60 to the tool on Toolforge [13:20:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.awmd-stats/SAL [13:20:59] !log tools.awmd-stats Restarting tool after deploying 0bc3c60 so it applies [13:20:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.awmd-stats/SAL [13:23:19] !log tools.awmd-stats Tool up and running after restart complete [13:23:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.awmd-stats/SAL [13:37:53] !log tools.awmd-stats Deployed c276e36 and restarting tool to pick changes up. [13:37:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.awmd-stats/SAL [13:39:09] !log tools.awmd-stats Tool up and running after restart complete [13:39:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.awmd-stats/SAL [13:55:43] hi, I have a question re: labmon hosts, I see there are two, are they both active all the time or perform different duties or there's a standby etc? I'm asking because I'll be looking at upgrading prometheus to v2 there [13:56:02] T187987 that is [13:56:02] T187987: 100% of Prometheus traffic served by Prometheus v2 - https://phabricator.wikimedia.org/T187987 [14:04:07] !log deployment-prep add dsharpe user [14:04:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [15:45:11] !help - what's going on? https://usercontent.irccloud-cdn.com/file/Z2MvPeoO/image.png [15:45:11] RhinosF1: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [15:46:53] Full replag page source ---> https://dpaste.de/pjOK [15:50:35] ^^^ better view of it https://dpaste.de/q3HH [15:54:54] ^^ s1/3/8 0.9 seconds rest 0s now [15:58:27] https://quarry.wmflabs.org/query/35056 and https://quarry.wmflabs.org/query/35057 have just taken 760s (13 mins) compared to a ususal 1 -2 mins [16:14:09] hey RhinosF1 I'm not aware of any important issues right now in the DB [16:14:57] please open a phabricator bug/tasks if you see the issue is not just transient [16:19:14] Arturo, looks to be clear now [16:24:08] RhinosF1: lag of a few minutes is pretty common and usually clears quickly. Anything less than ~1 day lag is not something we would usually bug the DBA team about. [16:24:50] in the olden days on toolserver lag was really commonly 7+ days [16:35:43] bd08, never really see much of a lag ever. If anything it's normally s8 that's bad. Any reason the quarry querys would have taken so long to run [16:35:56] bd808, ^ [16:37:23] quarry is a shared worker queue for everyone, lots of things could cause it to take extra time to complete a given query (including locking for replication). [16:37:48] bd898, thx [16:37:56] bd808, ^ [23:25:33] Seeing python3 --version return 3.5.3 on tool forge and then 3.4.2 inside the webservice was misleading... I've upgraded my tool to django 2.2, developed agaist that, assuming python 3.5 is available on tool forge, and now I'm hit with python3.4:( This version asymmetry is really confusing. [23:26:24] How likely is it to have T219091 resolved in a couple of days? [23:26:24] T219091: toolforge: Update python image to Stretch - https://phabricator.wikimedia.org/T219091 [23:49:58] yurb: very unlikely in a couple of days. Quite possible in ~3 weeks. I'm hoping to make those changes happen during the Hackathon [23:50:18] bd808: okay, thanks for letting me know [23:50:57] I'll be there btw [23:51:40] cool! You can help me test then :)