[00:00:10] and bon apetite [00:02:45] 10PAWS, 6Research-and-Data: Create a mailing list for PAWS - https://phabricator.wikimedia.org/T129297#2101483 (10Legoktm) Erm, any reason we can't use the existing pywikibot-l mailing list? [00:04:27] 10PAWS, 6Research-and-Data: Create a mailing list for PAWS - https://phabricator.wikimedia.org/T129297#2110225 (10yuvipanda) I think the P in PAWS is no longer accurate, since it's just one of the things it does. There's db access, R, more languages coming... I also don't think the 'mailing list' was in the... [00:42:57] 6Labs: Disable instance running slate.wmflabs.org - https://phabricator.wikimedia.org/T129594#2110396 (10Peachey88) [03:30:12] 6Labs, 10Labs-Infrastructure, 6Operations: labnet1002 can't talk to webproxy.eqiad.wmnet:8080, puppet fails to install designateclient - https://phabricator.wikimedia.org/T129623#2110962 (10Dzahn) [06:23:41] 6Labs, 10Tool-Labs: Collect and display basic metrics for all tools (service groups) - https://phabricator.wikimedia.org/T129630#2111124 (10bd808) [07:15:48] 6Labs, 6Operations, 10ops-eqiad: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2111154 (10jcrespo) > @jcrespo can you help determine the way forward here? Do we need to pursue figuring out which disk is the faulty one here for replacment? We already have the replacement on rack, it... [07:50:54] 6Labs: Disable instance running slate.wmflabs.org - https://phabricator.wikimedia.org/T129594#2111178 (10Prtksxna) 5Open>3Resolved a:3Prtksxna Deleted the old [[ https://wikitech.wikimedia.org/wiki/Nova_Resource:Slatetest.design.eqiad.wmflabs | slate instance ]]. [08:34:52] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [08:54:35] (03PS1) 10Alexandros Kosiaris: Update maps private data structures to match production [labs/private] - 10https://gerrit.wikimedia.org/r/276704 [08:55:00] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Update maps private data structures to match production [labs/private] - 10https://gerrit.wikimedia.org/r/276704 (owner: 10Alexandros Kosiaris) [09:17:08] 6Labs, 10Tool-Labs, 10pywikibot-core: Tool Labs Pywikibot does not work with new shared Pywikibot config files - https://phabricator.wikimedia.org/T129406#2111220 (10zhuyifei1999) p:5Normal>3Unbreak! a:3zhuyifei1999 Happens on grid engine (not on bastion) regardless of of whether the setup is with /sha... [09:17:34] 6Labs, 10Tool-Labs, 10pywikibot-core: Tool Labs Pywikibot does not work with new shared Pywikibot config files - https://phabricator.wikimedia.org/T129406#2111223 (10zhuyifei1999) p:5Unbreak!>3High [09:20:29] Hello! I'm trying to play with lab VMs. How do I associate a security group to an instance? [09:23:58] 6Labs, 7Graphite: Puppet disabled on graphite-labs - https://phabricator.wikimedia.org/T129579#2111227 (10fgiunchedi) 5Open>3Resolved puppet is running again, the instance I'm using to test some production changes [09:24:06] gehel: I think they can only be added during instance creation, but not altered for existing ones, but not 100% sure, let me check [09:24:25] moritzm: I just found documentation that says the same. [09:24:40] moritzm: so you are probably right... [09:25:48] ok, Krenair and andrewbogott should be able to confirm when they're online later the day [09:26:28] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2111233 (10Nizil) Please consider to update this page too. https://en.m.wikipedia.org/wiki/Wikipedia:Pageview_statistics [09:29:34] moritzm: thanks for pointing me in the right direction! [11:08:15] Was there a deploy of login related things yesterday? [11:37:39] 6Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Cannot SSH to a few CI slaves - https://phabricator.wikimedia.org/T129640#2111492 (10hashar) The DNS entries are gone both instances. We had that issue on another instance previously. #labs is it possible to recreate the DNS entries for... [13:16:47] gehel, moritzm: hi [13:16:53] Hello ! [13:17:01] gehel, when you set up an instance, you choose the security group [13:17:07] it cannot be changed on wikitech afterwards [13:17:25] Krenair: yep, I finally saw that in the docs... [13:17:54] I'm cheating at the moment, so it works fine for what I'm trying to achieve [13:18:22] gehel, however, I believe you can change it in horizon [13:18:46] https://horizon.wikimedia.org/project/instances/ [13:19:11] log in, select the correct project at the top, then find your instance and use the actions dropdown to the right of 'Associate Floating IP' [13:19:17] you'll want 'Edit Security Groups' [13:19:38] Krenair: do you know what is this "totp token" ? [13:19:40] should be straightforward from there in theory [13:19:48] gehel, your two factor authentication code, same as with wikitech [13:21:43] I did not even realize I could enable 2-factor auth on wikitech... [13:22:06] If you don't have two-factor auth enabled you won't be able to log in to horizon [13:41:17] 6Labs, 7Shinken: Puppet is disabled on shinken-01 - https://phabricator.wikimedia.org/T127592#2111716 (10scfc) 5Open>3Resolved Puppet is no longer disabled on `shinken-01`. [13:44:33] 6Labs, 10Tool-Labs: shinken does not warn about tools-grid-master puppet staleness - https://phabricator.wikimedia.org/T122667#2111720 (10scfc) 5Open>3Resolved This seems to be working now (cf. http://shinken.wmflabs.org/host/tools-grid-master). [14:35:55] 6Labs, 10Labs-Infrastructure: change m1.gigantic type to 160 G HDD space - https://phabricator.wikimedia.org/T129581#2111830 (10Andrew) 5Open>3Resolved I'm not sure what it means to change a flavor that's already used for some isntances, so instead I created a new flavor called 'moregigantic'. [14:37:20] 6Labs, 10Labs-Infrastructure: change m1.gigantic type to 160 G HDD space - https://phabricator.wikimedia.org/T129581#2111833 (10Matanya) Thank you very much! [15:02:43] andrewbogott: morning [15:03:26] wanted to ask if we have some tool to expire virt-guests if some time passed [15:04:01] e.g, every 6 month, send an email to an instance owner and let him know he needs to ack his need of the instance within a week [15:04:31] if not responeded within the time frame, terminate the host and give it's resources back to the pool [15:04:52] there was such a tool on toolserver, iirc [15:21:44] there was account renewal on the toolserver [15:21:57] but there were no VMs that could have been terminated ;-) [15:23:45] we have a few tickets open for better ongoing management like that and a big needed one-time cleanup which is happening now out of necessity in part [15:42:57] o/ andrewbogott [15:43:21] In the ores project, we have a few "compute servers" that we use for doing batch CPU intensive work. [15:43:25] ‘morning halfak and matanya [15:43:37] It's great to have xlarge machines so we can parallelize [15:43:40] but I’m not really paying attention to this channel yet, will try to catch up later [15:43:46] But then the VMs stay idle most of the time [15:43:58] (OK no worries. Will post and let you catch scrollback) [15:44:27] Should I be getting rid of these VMs when they are likely to be unused for a few days or let them be? [15:45:10] for a few days of idle it seems not worth it man, at that point they really only preserve disk for the most part [15:45:36] shutting them down would be most pragmatic if we had a good solution for that :) [15:50:42] halfak: as far as I know RAM and CPU are juggled between VMs as needed, so an idle VM doesn’t really cost us anything unless it’s eating disk space. [15:51:19] andrewbogott, great to know. We don't need too much disk space for these VMs. [15:51:30] if we [15:51:34] halfak: pretty sure you’re harmless then [15:51:37] re not using it, it stays in the pool? [15:51:43] Great! :) [15:52:03] thanks for checking [15:53:41] so I was once at a place where we did a lot w/ a little in this way, shuffled boxes on and off for big projects [15:53:48] and it was really painful for updates and changes rolling out [15:54:00] there were always pockets of down VM's that were legit but stale in some way [15:54:16] and then there is a lot of scheduling and contention for which group of map hosts can be up when [15:54:23] hopefully we don't get that tight [16:12:58] 6Labs: proxy hostnames containing dots - https://phabricator.wikimedia.org/T129655#2111980 (10AlexMonk-WMF) 5Open>3Invalid Clearly, this was supposed to be a paste, not a task. I may have messed up when switching from one account in one profile to another. [16:18:49] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2111990 (10MusikAnimal) Hmm that page should probably be deleted, or completely rewritten. It seems devoted to stats.grok.se, when we wouldn't want it devoted to... [16:30:32] chasemp: the new horizon gui will let users shutdown and/or suspend their instances. But I’m not sure I like that since it means that they'll lose puppet updates and could fail to revive... [16:30:57] right it's a huge double edge sword [16:31:54] I suspect the right solution is just ‘have ample virt capacity' [16:35:33] better off destroying and having good spin up automation in 99% of cases rather than playing musical chairs even [18:11:40] anomie: Are you here? [18:11:51] Luke081515: Yes [18:12:12] since the deploy at thursday, I got a problem with my bot: action=edit throws badtoken [18:12:54] is this a known issue? or is this maybe my fault? [18:16:04] anomie: I'm getting this constantly: a:2:{s:8:"servedby";s:6:"mw1129";s:5:"error";a:3:{s:4:"code";s:8:"badtoken";s:4:"info";s:13:"Invalid token";s:1:"*";s:52:"See https://de.wikipedia.org/w/api.php for API usage";}}^Z [18:16:23] and I din't change anything in the scripts, the script uses csrf tokens [18:16:36] 6Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 10labs-sprint-118: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#2112306 (10Krenair) 5Resolved>3Open a:5Krenair>3scfc Sorry @scfc, I remembered earlier that I also created... [18:17:34] Luke081515: I can't think of anything in yesterday's deploy that would have changed any behavior there. Does fetching a new token then work? [18:19:31] anomie: this is the short function to save a page at my script (php), normaly the scripts tries to fetch a new token before every edit: https://phabricator.wikimedia.org/P2745 [18:20:31] Luke081515: Can you capture the HTTP request and response for the two calls in that paste, and put them in a private paste for me? [18:21:07] ok, wait a moment [18:21:50] (this paste is 'private' too, normaly I create my pastes only visible for CCs, but feel free to add some if needed ;)) [18:28:25] anomie: I created https://phabricator.wikimedia.org/P2746 [18:28:38] * anomie looks [18:29:35] Luke081515: You don't seem to have captured all the request and response headers for both requests there. [18:30:41] oh, ok [18:31:31] maybe I should let echo the output before the $website = trim(substr($website, strpos($website, "\n\r\n"))); command :D [18:33:59] anomie: I updated it, hope it is complete now [18:37:47] Luke081515: In your first response, I see it gave you a Set-Cookie header for dewikiSession=edXXXXX. Your second post, however, supplies dewikiSession=41XXXXX and dewikiSession=pcXXXXXX in the Cookie header, neither of which are correct. I see something similar with the centralauth_Session cookie. That's likely the cause of your problem. [18:38:40] anomie: Maybe you can take a short look at my login / get cookies methods? There are in php too [18:38:59] Luke081515: I can take a quick look [18:39:07] ok, thanks. wait a moment [18:40:05] anomie: https://phabricator.wikimedia.org/P2747 [18:40:19] hm, pasting things in phabricator doesn't keeps the formating [18:42:00] Luke081515: Ugh. You really should just use PHP's curl extension instead of doing it manually with fsockopen. That will handle cookies correctly for you. But a quick fix would be to call "GetCookies" on every response, not just the login responses. [18:42:36] anomie, ok, thanks [18:43:00] 6Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 10labs-sprint-118: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#2112478 (10Krenair) ```> $host = OpenStackNovaHost::getHostByPublicIP( '208.80.155.156' ); > $success = $host->de... [18:43:50] Luke081515: You'll also want to adjust your "GetCookies" so a cookie gets removed from $this->cookies whenever a Set-Cookie for it is seen (deleted or not). [18:44:46] 6Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 10labs-sprint-118: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#2112483 (10scfc) a:5scfc>3Krenair As I believe you are working on some UI improvements/migrations for the prox... [18:45:37] anomie: So as quick fix adding $this->GetCookies($website); after every token request is enough? [18:46:31] Luke081515: For the immediate problem, probably. Eventually your bot might be sending a gigantic "Cookie:" header on every request with hundreds of copies of dewikiSession though, due to the lack of deleting. [18:50:55] 6Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 10labs-sprint-118: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#2112514 (10Krenair) 5Open>3Resolved Thanks. ```sqlite> select id from project where name = 'openstack'; 69 sql... [18:54:09] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [18:55:05] PROBLEM - Puppet failure on tools-exec-1218 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:55:45] PROBLEM - Puppet failure on tools-exec-1207 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:56:59] PROBLEM - Puppet failure on tools-exec-1215 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:57:12] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:57:59] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:57:59] anomie: I still get invalidtoken :-/ [18:58:13] PROBLEM - Puppet failure on tools-exec-1219 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:58:37] Luke081515: Did you check that your cookies are right now? [18:59:11] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:59:43] 10Tool-Labs-tools-Database-Queries: Replication broken on c1 - https://phabricator.wikimedia.org/T129678#2112542 (10Superyetkin) [19:02:17] PROBLEM - Puppet failure on tools-exec-1213 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:06:16] PROBLEM - Puppet failure on tools-exec-1221 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:08:40] PROBLEM - Puppet failure on tools-exec-1206 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:11:04] PROBLEM - Puppet failure on tools-exec-1210 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:14:46] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:17:01] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:17:27] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:18:55] PROBLEM - Puppet failure on tools-exec-1217 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:18:55] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [19:19:53] ^failures are known issue atm [19:24:02] What's the best location for lazy-web "quarry query assistance" requests? Here, a specific talkpage somewhere, a phab task, other? [19:26:59] quiddity: depending on what you rae looking to do -research irc channel probably has the most battle worn users :) [19:27:32] ah, yes, ty :) (So many channels, so little coffee) [19:27:36] yup! [19:27:55] totally cool to drop a question here but hey go where the buffalo are kind of thing in reality [19:28:06] (03CR) 10ArthurPSmith: "recheck" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [19:39:18] !log tools fixed puppet runs on tools-exec (gerrit 276792) [19:39:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [19:39:56] !log tools should a tools-exec server be influenced by font packages on an mw appserver? [19:39:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [19:46:07] RECOVERY - Puppet failure on tools-exec-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [19:53:49] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [19:53:51] RECOVERY - Puppet failure on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0] [19:54:47] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [19:57:00] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [20:03:15] RECOVERY - Puppet failure on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [0.0] [20:04:09] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [20:04:20] that's due to that font package i added [20:04:45] i mean, that broke it and now it doesn't try to install that anymore [20:04:58] unless it's trusty or jessie [20:05:49] RECOVERY - Puppet failure on tools-exec-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [20:07:13] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [20:07:59] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [20:08:05] argg, and anther issue with that , the package conflicts with another font package [20:12:14] RECOVERY - Puppet failure on tools-exec-1213 is OK: OK: Less than 1.00% above the threshold [0.0] [20:16:16] RECOVERY - Puppet failure on tools-exec-1221 is OK: OK: Less than 1.00% above the threshold [0.0] [20:18:36] RECOVERY - Puppet failure on tools-exec-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [20:31:05] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:32:38] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:33:42] PROBLEM - Puppet failure on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:34:19] groooarr. and those on tools-webgrid are also font related [20:34:46] the common mistake, think something can be a 5-minute task [20:34:55] PROBLEM - Puppet failure on tools-exec-1403 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:35:12] specifically ttf-indic-fonts : Depends: ttf-gujarati-fonts fml [20:36:06] PROBLEM - Puppet failure on tools-bastion-05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:36:21] PROBLEM - Puppet failure on tools-bastion-mtemp is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:37:07] !tools more puppet issues due to font dependencies on trusty, on it [20:37:07] There are multiple keys, refine your input: tools-admin, toolsbeta, tools-bug, toolscors, tools-equiad, toolsmigration, tools-request, tools-status, toolsvslabs, tools-web, [20:37:12] !log tools more puppet issues due to font dependencies on trusty, on it [20:37:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [20:37:35] PROBLEM - Puppet failure on tools-exec-1401 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:38:05] PROBLEM - Puppet failure on tools-exec-1407 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:38:05] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1405 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [20:41:09] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:42:25] PROBLEM - Puppet failure on tools-exec-1406 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:43:10] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:44:55] PROBLEM - Puppet failure on tools-webgrid-generic-1405 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:45:31] PROBLEM - Puppet failure on tools-cron-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:46:06] 6Labs, 10Tool-Labs, 10DBA: What would be the preferred way to run a series of read queries on DB replica, each on two *_p databases ? - https://phabricator.wikimedia.org/T124805#2113042 (10Alphos) I really am looking for both. On another note, I'm in a bit of a tough time currently (for the past few weeks,... [20:46:17] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:47:23] PROBLEM - Puppet failure on tools-exec-1409 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:47:25] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:47:35] PROBLEM - Puppet failure on tools-bastion-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:53:44] recoveries should be there soon [20:57:02] !log tools reverted font changes - puppet runs recovering [20:57:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [21:01:47] shinken-wm: would you please [21:02:51] well, it looks good when i run puppet myself [21:06:06] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [21:07:44] RECOVERY - Puppet failure on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:07:44] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [21:11:04] RECOVERY - Puppet failure on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [21:13:35] Hm.. ssh/bash seems a bit sluggish on tools-login at the moment [21:13:40] RECOVERY - Puppet failure on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [21:13:43] taking a while to resolve cd commands and returning the promot [21:13:45] prompt* [21:14:54] RECOVERY - Puppet failure on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [21:17:31] RECOVERY - Puppet failure on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [21:21:15] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [21:22:27] RECOVERY - Puppet failure on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [21:23:11] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [21:24:55] RECOVERY - Puppet failure on tools-webgrid-generic-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [21:26:24] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [21:27:36] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [21:28:09] bd808: Do you know if there is a way to opt out some extensions from vagrant git-update? [21:30:29] 6Labs, 10MediaWiki-extensions-OpenStackManager, 13Patch-For-Review: Additions and removals of project members are very hard to decipher in the wiki diff - https://phabricator.wikimedia.org/T128001#2113210 (10Krenair) 5Open>3Resolved [21:32:26] RECOVERY - Puppet failure on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [21:45:54] question for the admins: if I want to install a newer version of Python (2.7.11) for one of my tools (HostBot) to use, should I install it in my user directory, or in HostBot's directory, after logging in as HostBot? [21:52:06] the answer is the last one [21:52:13] but also... why are you installing a newer version of python? [21:55:51] yuvipanda: I want to see if I can fix a bug by using the newer version. Some googling around tells me that requests_oauthlib doesn't play nicely with 2.7.6 [21:56:00] J-Mo: I see. [21:56:16] J-Mo: I'll say that building a new version of python is... not the easiest thing [21:56:28] J-Mo: so one recommendation is to try to use py3 but I'm sure you've been told that before [21:57:41] yuvipanda: yeah, I want to re-write my code in p3, but that takes time. Right now this bug is preventing me from running HostBot invites on cron using OAuth, and I'm getting tired of running the script manually every day [21:58:01] J-Mo: are you doing the '-l release=trusty' thing? [21:58:10] J-Mo: and are you running it manually in tools or locally? [21:58:52] I haven't tried -l release=trusty yet (and I just deleted my venv, so I have to re-build that now before I can test >_<…). I'm running the script manually, as HostBot [21:59:16] J-Mo: if running it manually works and crond oes not, there's a 90% chance that adding -l release=trusty will fix your issues [21:59:39] k. I'll make a new venv and try that out. Thanks yuvipanda! [22:00:00] J-Mo: np! [22:09:07] 10PAWS: Implement a sane way to access mysql replicas from PAWS - https://phabricator.wikimedia.org/T120471#2113359 (10yuvipanda) New containers will have access to the following environment variables: - MYSQL_HOST - MYSQL_USERNAME - MYSQL_PASSWORD for access to the replica databases atm. Working on a query ki... [22:12:12] Luke081515: if you switch an extension from master to another branch I don't think that git-update will update it any more [22:12:42] bd808: ok, thanks [22:15:42] 6Labs, 10Tool-Labs: Goal: Allow using k8s instead of GridEngine as a backend for webservices (Tracking) - https://phabricator.wikimedia.org/T129309#2113390 (10chasemp) p:5Triage>3Normal [22:26:27] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [22:32:31] (03CR) 10Ricordisamoa: "recheck" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [22:32:52] yuvipanda: I've got my venv set up again. where in the JSUB command do I pass in "-l release=trusty"? [22:34:20] J-Mo: just in the beginning [22:34:41] got it. job was submitted… fingers crossed [22:39:39] trusty worked, yuvipanda. Looks like my days of sending invites manually are over, for now at least. Thanks again! [22:39:45] :D [22:57:45] hello, is it possible setting the locale on jsub, or on crontab for all submissions? [23:03:02] bd808: I don't think so that git-update ignores all custom branches [23:03:03] ***** Remote not found in /vagrant/mediawiki/extensions/StaffEdits for branch 'custom' [23:03:48] Luke081515: that message would keep it from changing anything in the local clone I'm 90% sure [23:04:08] bd808: Yeah, but it displays errors anoyway ;) [23:04:12] *anyway [23:04:21] it's not going to silently ignore it if that's what you want. The point of git-update it to update all the things [23:04:21] have uncommited changes keeps the old things too [23:04:31] hm, ok [23:04:44] * Luke081515 can live with one small error [23:05:10] this is a development system, not really a long term server maintenance soluton [23:05:29] np [23:23:22] 10Tool-Labs-tools-wikiloves: Desenvolver aplicativo básico em Flask para a ferramenta - https://phabricator.wikimedia.org/T129712#2113596 (10Danilo) [23:32:29] 10Tool-Labs-tools-wikiloves: Desenvolver aplicativo básico em Flask para a ferramenta - https://phabricator.wikimedia.org/T129712#2113643 (10Danilo) Vamos criar uma homepage semelhante a do ptwikis ou melhor fazer uma a partir do zero? Basear no ptwikis vai poupar tempo, mas um design um pouco melhor também ser... [23:35:36] bd808: ping? [23:52:08] SMalyshev: pong [23:53:21] bd808: hi! I'm trying to figure out a problem in my vagrant setup... somehow xdebug remote does not work anymore for me. So I wonder - how the port forwarding for xdebug on vagrant is supposed to be working? [23:53:59] hmmm... good question. [23:55:11] SMalyshev: tgr might be a better person to ask. I never use xdebug/ide integration stuff [23:55:22] bd808: ok, thanks :) [23:56:04] SMalyshev: xdebug is somewhat unreliable with hhvm [23:56:12] tgr: I use zend, not hhvm [23:56:59] in that case, it should work out of the box [23:58:10] the vagrant box connects back to your IP, which is the IP of the host machine in the virtual network [23:59:01] tgr: hmm... it looks like for me it connects to localhost, not host ip [23:59:15] at least it never gets to IDE... [23:59:37] maybe you disabled xdebug.remote_connect_back? [23:59:44] but if I make nc listen to local port, I get connection. so somehow it doesn't get from local guest port to host port