[00:00:10] message is too long [00:08:45] Krinkle: we just changed the *.labsdb DNS to point to the new wiki replica cluster. It looks like https://tools.wmflabs.org/guc/ still works and I think it is a *lot* faster [00:16:25] bd808: I changed it to the new replicas last week! (Or I thought I did anyway) [00:18:04] ah. you changed the code then? [01:21:48] is there any issue with toollabs? [01:21:53] seems to me that Magnus's tools are having difficulties pulling information from the dataservers in toollabs; web interface is fine ... example https://tools.wmflabs.org/mix-n-match/#/list/61/auto ... haaaangs [01:22:19] has some similarity to issues yesterday [01:25:38] the problem seems to be related to to tools.labsdb, "mysql --defaults-file=$HOME/replica.my.cnf -h tools.labsdb" returns "ERROR 2005 (HY000): Unknown MySQL server host 'tools.labsdb' (0)" [01:26:20] I have a tool that is returning that same error [01:29:42] danilo: that sounds like something that might be related to https://lists.wikimedia.org/pipermail/cloud-announce/2017-December/000013.html ? [01:34:13] tools.labsdb don't resolve [01:36:04] I forgot it was today, I saw the messages above and the status in the topic now [01:36:21] I don't understand why it doesn't, given it's listed at https://gerrit.wikimedia.org/r/#/c/397256/7/modules/profile/files/openstack/base/pdns/recursor/labsdb.zone [01:36:25] zhuyifei1999_: see the andrewbogott message above [01:36:28] bd808: [01:36:34] danilo: ik [01:37:41] bd808: tools.labsdb has a cname for tools.db.svc.eqiad.wmflabs.labsdb [01:37:48] which looks wrong to me — shouldn't it be tools.db.svc.eqiad.wmflabs [01:38:05] andrewbogott: crap. Yes [01:38:12] easy fix? [01:38:19] the missing dot? [01:38:33] zhuyifei1999_: no, it has the wrong tld [01:38:39] or, a surplus tld [01:38:45] Yeah. There is a puppet file with the data in it [01:39:02] I mean in the patch [01:39:04] I’m out driving. No laptop [01:39:15] bd808: ok, I'll search and replace :) [01:39:21] And I hope you aren't actually driving [01:42:36] I pulled over :) [01:42:52] zhuyifei1999_: is right, there's a single missing . in that patch. Fixing... [01:44:02] https://gerrit.wikimedia.org/r/#/c/398201/ <- just waiting for gerrit [01:44:04] bd808: Yeah, I ended up switchig it with a hack - https://github.com/wikimedia/labs-tools-guc/commits/master [01:44:17] bd808: once I merge what do I need to do to force refreshes? [01:46:17] ok, merged, applied on labservices… looks better to me. zhuyifei1999_, danilo, sDrewth, better? [01:46:30] lgtm [01:46:35] yes, it is working :) [01:47:02] cool — sorry for the hiccup [01:47:20] #magic thanks for the fix [01:47:56] meh, hiccups happen, and as long as they correspond one to one with fixes, I am happy [01:48:05] (I checked that files for other dangling domains and there weren't any, so hopefully this is a one-off.) [02:07:30] Stupid period [09:45:54] Hi! I'm developing a tool named WikiCV. Before starting, I wanted to get inputs from the user. It would be great if you guys could tell that what are your expectations from such a tool? Or what all would you like to show in your WikiCV? Also, Is there someone who is working in a similar field? And can anyone point me to users who would be interested to give inputs? Thanks! [09:52:47] megha213_: What's the WikiCV tool all about? [09:53:12] Its a contribution summary tool that's optimized for presenting one's activity in a CV-like manner. [09:53:41] Here it is, [09:53:54] https://phabricator.wikimedia.org/tag/wikicv/ [09:54:58] megha213_: Ohh, is it an Outreachy project? And you are working on it for this round? [09:55:48] yes, this is an outreachy project [09:56:05] I've been selected in Round 15 to work on this! [09:56:19] Okay! How about making a survey with some questions and publishing out to get feedback? [09:56:48] megha213_: Maybe this platform (IRC) won't really be the best place to get feedback? Also, there was an Advice IRC meeting yesterday on #wikimedia-tech [09:57:08] That would have also been a good platform as there are people focused on the meeting [09:57:16] But I think a survey will do. [09:57:30] 1. Create survey with some questions you would want feedback on [09:57:42] 2. Publish the survey on as many mailing list as possible etc. [09:58:02] 3. Receive feedback from the community and analyze. [09:58:12] That's my humble oppinion :) [09:58:17] megha213_: What do you think? [09:58:50] I did make a survey and discussed it with my mentors.. [09:59:17] Ohh nice! Did you get some reasonable amount of feedback? I didn't see the survey! [09:59:22] Can you share the link here? [10:17:02] megha213_: Still waiting... for the link to the survey! [10:20:28] d3rlck: Sorry for the delay, [10:20:33] here's the link [10:20:34] https://goo.gl/forms/rSTH3B2hfKTXgzsW2 [10:24:49] d3rlck: the reason i didnt float it earlier was because I though personal conversation would give better results and looks more personalized ☺ [10:25:58] Ohhh okay! But you can always try both options ;) [10:26:27] Maybe a post on mailing lists for a broader reach and personal conversation for detailed discussions etc.. [10:26:37] Just my humble opinion :) [10:27:08] megha213_: I suggest you make a post on some mailing lists to get feedback from a wider part of the community! [10:28:22] megha213_: A quick questions! [10:29:03] Will your tool tap into other private wikis like (Wikitech for example) to get your contributions there too? Will this feature be part of the tool? Or will it only be public wikis using a unified account? [10:29:28] Maybe the tool should have an option to tap into private wiki under the permission of the user? [10:29:38] * d3r1ck is filling megha213_ survey now! [10:32:09] d3rlck: Yes, I've subscribed to some of the mailing lists. Waiting for the approvals. [10:32:53] For now we are targetting only the public ones. But yes this feature can probably be lined up for the next release ☺ [10:34:58] Okay! [10:36:05] megha213_: I've submitted my response. Thanks for sharing the survey! [10:36:56] Thankyou for filling it :) [10:41:14] megha213_: yw [10:42:00] :) [10:49:37] megha213_: Good luck with your project! :) [10:53:23] d3rlck: Thankyou so much :) [13:32:28] puppet is broken until I fix it hold on [14:22:49] If there a recommended way to rotate out the uwsgi.log created by the python k8s webservice? I've been trying to use logrotate but: if I create a new file uwsgi doesn't continue logging until `webservice restart`; if I use copytruncate uwsgi keeps writing where it left off (leaving the beginning of the log file as null and an unchanged filesize). I can put in `webservice restart` as a post rotate script but I'm [14:22:49] concerned that I'll get regularly caught by T140415 [14:22:49] T140415: `webservice restart` does not always wait for service to stop before trying to start again - https://phabricator.wikimedia.org/T140415 [14:25:09] tarrow: it's a weakspot honestly, I think teh best thing to do would be to try a weekly rotate and T140415 should be very rare afaiu [14:25:23] we have looked at more robust in place solutons and tehy are all fraught with issues like you describe [14:26:10] truncated files on a shared filestore but services farmed out over a cluster still holding open the file in some processes means truncation isn't effective without restart and orchestrated distributed restart is a nightmarish problem at times [14:26:21] I wish I had a better answer at [14:26:23] atm even [14:26:26] ok :) [14:26:58] the right solution is to not do local logging onto NFS and have syslog or syslog equivalent [14:27:06] That's fine; I understand. [14:27:18] glad you asked and are thinking about it [14:27:53] Cool, I'm happy that its as good a solution as possible right now. Thanks! [16:58:38] !log tools running clush -w @all 'sudo puppet agent --test' from tools-clushmaster-01.eqiad.wmflabs due to https://gerrit.wikimedia.org/r/#/c/394572/ being merged [16:58:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:16:47] (03CR) 10Merlijn van Deen: [C: 04-1] If its not working break it more. (031 comment) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/398143 (owner: 10Zppix) [17:19:34] (03PS3) 10MacFan4000: If its not working break it more. [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/398143 (owner: 10Zppix) [17:19:48] (03CR) 10MacFan4000: [C: 031] "Fixed" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/398143 (owner: 10Zppix) [17:20:19] (03Draft2) 10Zppix: Rename ORES hosts for warning messages [labs/icinga2] - 10https://gerrit.wikimedia.org/r/398287 (https://phabricator.wikimedia.org/T182880) [17:26:01] (03CR) 10Merlijn van Deen: [C: 032] If its not working break it more. [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/398143 (owner: 10Zppix) [17:26:28] (03Merged) 10jenkins-bot: If its not working break it more. [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/398143 (owner: 10Zppix) [17:26:30] (03CR) 10Awight: "That's great, just two missing. If it's possible to display the entire hostname by default, that might be even better." (032 comments) [labs/icinga2] - 10https://gerrit.wikimedia.org/r/398287 (https://phabricator.wikimedia.org/T182880) (owner: 10Zppix) [17:26:37] (03CR) 10jenkins-bot: If its not working break it more. [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/398143 (owner: 10Zppix) [17:26:52] Pings for me everywhere cant keep up... [17:26:59] !log tools.wikibugs Updated channels.yaml to: 54e2d754583aa29c5fd3d79fad64fae407c78924 If its not working break it more. [17:27:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [17:28:31] Zppix: well, hopefully it works now ;-) [17:28:59] (03PS3) 10Zppix: Rename ORES hosts for warning messages [labs/icinga2] - 10https://gerrit.wikimedia.org/r/398287 (https://phabricator.wikimedia.org/T182880) [17:29:07] (03CR) 10Zppix: [V: 032 C: 032] Rename ORES hosts for warning messages [labs/icinga2] - 10https://gerrit.wikimedia.org/r/398287 (https://phabricator.wikimedia.org/T182880) (owner: 10Zppix) [17:30:23] Lol ill test in a sec valhallasw`cloud [17:36:32] (03Draft1) 10Paladox: Rename ORES hosts for warning messages [labs/icinga2] - 10https://gerrit.wikimedia.org/r/398294 (https://phabricator.wikimedia.org/T182880) [17:36:34] (03PS2) 10Paladox: Rename ORES hosts for warning messages [labs/icinga2] - 10https://gerrit.wikimedia.org/r/398294 (https://phabricator.wikimedia.org/T182880) [17:37:06] (03CR) 10Paladox: [V: 032 C: 032] Rename ORES hosts for warning messages [labs/icinga2] - 10https://gerrit.wikimedia.org/r/398294 (https://phabricator.wikimedia.org/T182880) (owner: 10Paladox) [17:42:47] !log git deployed [[Gerrit:398294]] to denote ORES Experimental Hosts for #wikimedia-ai alerts. T182880 [17:42:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [17:42:51] T182880: Rename hostnames in warnings to denote non prod - https://phabricator.wikimedia.org/T182880 [17:44:31] Paladox ^ [17:44:36] thanks [17:54:14] (03CR) 10BryanDavis: [C: 032] Update for diffusion.repository.search API change [labs/striker] - 10https://gerrit.wikimedia.org/r/395924 (https://phabricator.wikimedia.org/T182142) (owner: 10Legoktm) [17:56:00] (03Merged) 10jenkins-bot: Update for diffusion.repository.search API change [labs/striker] - 10https://gerrit.wikimedia.org/r/395924 (https://phabricator.wikimedia.org/T182142) (owner: 10Legoktm) [17:58:37] (03PS4) 10BryanDavis: Enhance workflow for tool labs membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/374104 (https://phabricator.wikimedia.org/T168027) [18:05:53] (03CR) 10BryanDavis: [C: 032] Enhance workflow for tool labs membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/374104 (https://phabricator.wikimedia.org/T168027) (owner: 10BryanDavis) [18:07:33] (03Merged) 10jenkins-bot: Enhance workflow for tool labs membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/374104 (https://phabricator.wikimedia.org/T168027) (owner: 10BryanDavis) [18:07:39] (03PS3) 10BryanDavis: Add Referrer-Policy header [labs/striker] - 10https://gerrit.wikimedia.org/r/387067 [18:49:11] (03CR) 10BryanDavis: [C: 032] Add Referrer-Policy header [labs/striker] - 10https://gerrit.wikimedia.org/r/387067 (owner: 10BryanDavis) [18:50:23] (03Merged) 10jenkins-bot: Add Referrer-Policy header [labs/striker] - 10https://gerrit.wikimedia.org/r/387067 (owner: 10BryanDavis) [19:22:11] is there some trouble in labs? I am trying to reach https://wikidata-wdsearch.wmflabs.org/ and getting service unavailable [19:22:17] I can get there by ssh [19:22:49] SMalyshev: maybe not helpful, but it works for me. I get a wiki main page [19:23:12] hmm that actually helps, that means maybe problem on my side or wmf network side [19:23:46] I end up at https://wikidata-wdsearch.wmflabs.org/wiki/Main_Page [19:23:55] yes that what should happen [19:24:12] i am getting an error... [19:25:14] fwiw site loads for me too at https://wikidata-wdsearch.wmflabs.org/wiki/Main_Page [19:25:35] looks like network issue not labs one then... [19:26:02] thanks for checking1 [21:47:26] Is it a tool problem or a general one ? https://phabricator.wikimedia.org/T182916 [21:47:29] ping bd808 [21:48:27] framawiki: $ mysql --defaults-file=$HOME/replica.my.cnf -h s7.web.db.svc.eqiad.wmflabs [21:48:30] works for me [21:48:45] maybe hitting a limit on concurrency [21:54:30] ok, thanks [22:14:56] (03PS1) 10Ayounsi: BirdLG: Add fake python secret session key [labs/private] - 10https://gerrit.wikimedia.org/r/398380 [23:52:22] (03CR) 10Ayounsi: [V: 032 C: 032] BirdLG: Add fake python secret session key [labs/private] - 10https://gerrit.wikimedia.org/r/398380 (owner: 10Ayounsi)