[00:21:02] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3007849 (10Reguyla) I really appreciate you folks considering my request. If I may suggest, a lot of apps including Flickrtocommons and Commons Helper among others use Mediawiki as the source wiki... [01:48:50] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3008015 (10bd808) >>! In T157342#3007849, @Reguyla wrote: > I really appreciate you folks considering my request. If I may suggest, a lot of apps including Flickrtocommons and Commons Helper among... [01:49:32] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3008016 (10bd808) [02:04:20] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [02:05:20] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3008064 (10bd808) Switching the wiki contacted for the OAuth handshake would really be a game of whack-a-mole. Today someone is affected by a meta ban, tomorrow it will be someone affected by a mw... [02:08:50] JustBerry: so where are you stuck exactly? [02:08:56] bd808: continuing discussion from -operations [02:09:00] probably a better channel lol [02:09:48] you want to import a module into your app.py? [02:14:52] JustBerry: I'm getting called away for dinner. I'll probably be back online in a hour or so. [02:15:21] bd808: Sounds good, thanks. [02:16:08] if you can describe your problem I'll see it in backscroll when I return (and maybe someone else will solve your problem in the meantime) [02:16:19] Yeah, still debugging a bit here [02:16:26] But will do [02:53:20] bd808: bad interpreter: No such file or directory (when I do 'pip') [02:53:38] bash: /data/project/TOOL/www/python/venv/bin/pip: /mnt/nfs/labstore-secondary-tools-project/TOOL/www/python/venv/: bad interpreter: No such file or directory [02:54:02] "TOOL" ? [02:54:18] bd808: TOOL = the tool I'm using [02:54:33] just replaced so people understand what I mean [02:55:29] so... is the venv created for use on Kubernetes? If so are you inside the webservice shell? [02:57:12] bd808: did source ~/www/python/venv/bin/activate already [02:57:15] in (venv) [02:57:32] inside the webservice shell? [02:57:48] webservice --backend=kubernetes python shell? did that [02:58:38] bd808: just logged out from webservice and back to bastion [02:58:47] pip worked [02:58:52] may need to modify that part (step 2) [02:58:58] pip worked on bastion? [02:59:02] pip's not on webservice --backend=kubernetes python shell [02:59:10] bd808: tools.spiarticleanalyzer@tools-bastion-03 [02:59:12] pip is inside the vene [02:59:13] yes but in (venv) [02:59:17] I'm in [02:59:20] source ~/www/python/venv/bin/activate [02:59:22] so [02:59:27] but in bastion tho [02:59:28] not webmaster [02:59:51] ok that means your venv is built for trusty (the os on the bastion) and not jessie (the os on kubernetes) [03:00:24] that will break everything in the kubernetes container [03:00:27] bd808: probs because of instructions from before [03:00:30] well... [03:00:34] how should I resolve? [03:00:37] I used the instructions from before [03:00:40] should I start from scratch [03:00:47] and get this tools account reset? [03:00:58] we can fix it :) [03:02:11] rm ~/www/python/venv -- then start from -- https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/My_first_Flask_OAuth_tool#Create_a_Python_virtual_environment_for_the_application.27s_external_library_dependencies [03:02:45] bd808: used rm -rf (in case you're adding this to a tutorial) [03:02:51] doing the steps now [03:07:15] bd808: finally got a test message (so that part worked at least) [03:08:24] progress! [03:09:46] source $HOME/www/python/venv/activate [03:09:47] -bash: /data/project/spiarticleanalyzer/www/python/venv/activate: No such file or directory [03:09:48] bd808: [03:09:55] ~/www/...? [03:10:15] JustBerry: darn. missing a /bin/ in there [03:10:20] bd808: ah [03:10:51] I'll fix it [03:12:21] should be fixed now JustBerry [03:12:34] $HOME/www/python/src/requirements.txt < EOF [03:12:34] bash: EOF: No such file or directory [03:12:35] bd808: [03:13:03] more typos! [03:13:06] fixing [03:13:23] bd808: ah [03:14:38] {{done}} -- needed << EOF [03:23:24] (venv) tools.spiarticleanalyzer@interactive:~$ webservice --backend=kubernetes python shell [03:23:24] bash: webservice: command not found [03:23:25] bd808: hm [03:23:35] o wait [03:23:47] meant to say [03:23:49] (venv) tools.spiarticleanalyzer@interactive:~$ webservice --backend=kubernetes python start [03:23:49] bash: webservice: command not found [03:23:54] did webservice --backend=kubernetes python shell [03:23:59] and source $HOME/www/python/venv/bin/activate already [03:24:01] (which both worked fine) [03:24:07] "r@interactive" means you are already in the webservice shell [03:26:43] bd808: I know [03:26:52] but webservice --backend=kubernetes python start doesn't work though [03:26:57] I meant to type start not shell [03:27:46] ah. yes. You have to exit out of the webservice shell to actually start the service [03:28:12] bd808: edit instructions or shall I ;p [03:28:52] Did I miss a "$ exit" on one of the steps? [03:29:31] bd808: [03:29:32] Now restart the webservice: [03:29:33] $ webservice restart [03:29:33] Restarting webservice... [03:29:35] Once the webservice has restarted, you should be able to go to https://tools.wmflabs.org// in your web browser and see the new 'Goodnight moon!' message. [03:29:41] exit before webservice restart probs [03:29:42] probably worth mentioning in the problems? section at least [03:29:49] there too [03:29:56] so both places then probably [03:30:29] there is an exit at the end of the "Add mwoauth to the virtual environment" command block [03:30:38] maybe you missed that one? [03:35:02] bd808: should SECRET_KEY: $(python -c "import os; print repr(os.urandom(24))") be literally inserted [03:35:09] or gened and then copy the string in [03:35:11] wb jackmcbarn [03:35:42] JustBerry: literally will work just fine. [03:36:08] bd808: what's the point of secret key if it's not checked anywhere tho [03:36:18] The "$(...)" wrapper is a bit of exec magic [03:36:23] like consumer key and secret are checked @ oauth [03:36:43] that value is used by Flask internally to sign the HTTP cookies that are used for session storage [03:36:51] bd808: aigh [03:37:30] http://flask.pocoo.org/docs/0.12/quickstart/#sessions [03:38:23] bd808: CONSUMER_KEY: 'key' or CONSUMER_KEY: key [03:38:36] either will work [03:39:36] bd808: i just ended up using touch instead of cat ;p [03:40:49] bd808: clicked 'login' on tool page [03:40:51] Internal Server Error [03:40:51] The server encountered an internal error and was unable to complete your request. [03:41:38] check ~/uwsgi.log to see what went wrong [03:42:03] quite possible I fed you another typo somewhere in the python code [03:42:52] bd808: https://phabricator.wikimedia.org/P4908 [03:43:26] * bd808 looks for the missing return [03:44:04] JustBerry: was there something before that about "mwoauth.initiate failed"? [03:45:01] bd808: updated paste, looks like it [03:45:36] this is the missing return -- https://wikitech.wikimedia.org/w/index.php?title=Help:Tool_Labs/My_first_Flask_OAuth_tool&diff=1474532&oldid=1474531 [03:46:51] bah "OAUTH_MWURI". That was a config value in the longer code I extracted this from [03:47:04] * bd808 updates tutorial [03:47:32] bd808: ah [03:47:44] bd808: clicking login does nothing [03:49:04] bd808: updated at very bottom https://phabricator.wikimedia.org/P4908 [03:50:11] JustBerry: https://wikitech.wikimedia.org/w/index.php?title=Help:Tool_Labs/My_first_Flask_OAuth_tool&diff=1474533&oldid=1474532 [03:50:47] * JustBerry checks [03:51:06] bd808: still same bug [03:51:11] checking traceback [03:51:47] you will need to restart the app after fixing the config I think with `webservice restart` [03:51:49] bd808: bottom (https://phabricator.wikimedia.org/P4908) [03:51:50] did [03:51:53] another bug [03:52:26] cool. and by the way thanks for testing all of this. I obviously did not :/ [03:53:03] np [03:53:45] hmmm... so you are getting a complete failure from the OAuth handshake... [03:54:08] bd808: heh [03:54:13] key was recently updated [03:54:17] like few days ago [03:55:01] do you mind if I log in and look at the logs myself? [03:55:12] bd808: don't mind [03:56:48] bd808: I wonder if adding / to the end of the URL you updated in the tutorial might help [03:56:54] "Error: An error occurred in the OAuth protocol: Invalid signature" -- your key and/or secret aren't right [03:56:56] https://meta.wikimedia.org/w/index.php/ instead of https://meta.wikimedia.org/w/index.php [03:56:59] bd808: really? [03:57:11] FACEPALM [03:57:13] wait [03:58:03] bd808: aha! [03:58:05] worked [03:59:39] nice! [04:16:43] bd808: runs on python 2.7, right? [04:16:47] not recognizing unicode() for some reason [04:17:34] python --version [04:17:34] Python 2.7.6 [04:17:34] heh [04:20:30] yuvipanda: know of a better method? [04:29:18] JustBerry: in the kubernetes container it will be running python 3.something [04:29:35] bd808: hm [04:30:30] bd808: \n doesn't work with flask eh ;p [04:30:45]
;p [04:31:06] Python 3.4.2 [04:33:00] there's python2 which gives you python2 [04:33:09] bd808: i'm rebuilding container images now and pushing to get the webservice fix in [04:35:45] bd808: getting a form setup (so that the script is two-way) [04:36:35] JustBerry: all the code in the tutorial should work on python 2 as well. You would need to wipe out the venv again and then use "python2" in all of the webservice commands. [04:36:54] Creating the initial venv would be slightly different too [04:37:00] eh [04:37:05] I'll keep python 3 for now [04:37:17] it's the future! [04:37:29] and the past :) [04:39:35] bd808: seems like I'll have to worry about cookies if I want the script to run independently for each user [04:39:59] I'm not sure what that means [04:41:00] bd808: do I really need to worry about session cookies for now? [04:41:08] each user will use the flask form, submit, output... [04:41:22] well, probably [04:41:29] otherwise each user will be riding over each other [04:41:30] hm [04:41:38] the existing code you copied uses cookie based sessions [04:42:19] flask.session['access_token'] has the oauth token for a user after they have logged in [05:04:56] 06Labs, 10Tool-Labs: Upgrade bootstrap-vz version for tools docker builder - https://phabricator.wikimedia.org/T157526#3008191 (10yuvipanda) [05:28:03] !log tools disable puppet on all k8s nodes in preparation for docker upgrade [05:28:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [05:28:13] !log tools drain pods from tools-worker-1027.tools.eqiad.wmflabs for docker upgrade [05:28:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [05:40:34] bd808: still around? [05:40:44] yep [05:41:05] bd808: working on this atm https://github.com/JustBerry/SPIArticleAnalyzer/blob/master/SPIArticleAnalyzer/app.py [05:41:20] bd808: search function seems to be acting up ;p [05:42:08] I think the problem right now is that there should be something to address get [05:42:17] i.e. when the request method is GET, then display a form [05:42:29] do I have to create a form template in flask? been reading the docs [05:43:25] yeah, where you have "return request.method" you need to render an html page containing the form inputs [05:43:49] bd808: that was mainly for debugging but yeah [05:43:53] this may help -- http://exploreflask.com/en/latest/forms.html [05:43:57] heh [05:44:02] yeah I looked through that [05:44:09] the problem there is they use flask_wtf Forms [05:44:14] instead of wtforms Form [05:44:21] different from http://flask.pocoo.org/docs/0.12/patterns/wtforms/ bd808 [05:45:10] slightly, sure. in the one you linked you are missing something like "return render_template('register.html', form=form)" [05:47:27] bd808: would I need to make a template for the form? or, is there a default one available based on form class definition [05:48:09] I'm pretty sure you have to make one. (/me is not actually a frequent Flask user) [05:48:51] that http://flask.pocoo.org/docs/0.12/patterns/wtforms/ page looks like it describes all the things you need [05:49:15] templates just go in the www/python/src/templates directory [05:55:49] what does "queue instance continuous@tools-exec-1419.tools.eqiad.wmflabs dropped because it is overloaded" mean? [05:56:04] i have a jsub job that keeps just not finishing [05:56:39] it dies when it tries to convert a jp2 to a jpg with `convert` [05:56:40] samwilson: it just means that one exec host isn't going to handle your job [05:56:57] you are probably hitting the memory limit [05:57:14] i thought that might be it [05:57:29] bd808: what can I do? [05:58:01] try adding "-mem 1g" to the job description [05:58:15] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid#Allocating_additional_memory [05:58:34] ah cool! I forgot that was possible. [05:58:41] will try [06:05:31] bd808: brill! thanks :) djvu produced correctly [06:05:54] yw :) [06:08:13] bd808: NameError: name 'request' is not defined [06:08:16] imported flask already through [06:08:17] import flask [06:08:32] from flask import request wouldn't make sense... [06:08:59] ou have to do that *or* use it as flask.request [06:09:32] I tend to prefer the latter for clarity but it is a bit more typing [06:22:04] !log tools drain tools-worker-1026 for docker upgrade [06:22:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [06:26:30] bd808: got this logic down for two search methods so far... https://github.com/JustBerry/SPIArticleAnalyzer/blob/master/SPIArticleAnalyzer/app.py [06:37:33] PROBLEM - Puppet run on tools-exec-1414 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:17:33] RECOVERY - Puppet run on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [07:27:16] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3008324 (10Marostegui) All the tables have been imported to labsdb1009. Stopping and starting mysql worked fine, doing a SELECT over all the tables worked... [07:34:09] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3008328 (10Marostegui) All the tables have been imported to db1095. Stopping and starting mysql worked fine, doing a SELECT over all the tables worked and... [07:40:22] PROBLEM - Puppet run on tools-worker-1004 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:42:46] 10Tool-Labs-tools-LTA-Knowledgebase, 15User-Samtar: Localization - https://phabricator.wikimedia.org/T155738#3008332 (10Samtar) [07:43:06] 10Tool-Labs-tools-LTA-Knowledgebase, 15User-Samtar: Install Python - https://phabricator.wikimedia.org/T157465#3008333 (10Samtar) [07:47:45] RECOVERY - Puppet staleness on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [3600.0] [07:49:31] 06Labs, 10Tool-Labs, 10Prod-Kubernetes, 10Tools-Kubernetes, and 2 others: Set-up live-restore for docker containers - https://phabricator.wikimedia.org/T157180#3008337 (10yuvipanda) This is now done, and future docker upgrades are non-invasive :D [07:50:53] 06Labs, 10Tool-Labs, 10Prod-Kubernetes, 10Tools-Kubernetes: Unify docker code and configuration between tools and prod kubernetes - https://phabricator.wikimedia.org/T156852#3008350 (10yuvipanda) 05Open>03Resolved a:03yuvipanda This is done. [07:51:05] 06Labs, 10Tool-Labs, 10Prod-Kubernetes, 10Tools-Kubernetes, and 2 others: Set-up live-restore for docker containers - https://phabricator.wikimedia.org/T157180#3008353 (10yuvipanda) 05Open>03Resolved a:03yuvipanda [07:54:22] RECOVERY - Free space - all mounts on tools-worker-1004 is OK: OK: tools.tools-worker-1004.diskspace._var_lib_docker.byte_percentfree (No valid datapoints found) [07:55:22] RECOVERY - Puppet run on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [10:22:45] 06Labs, 10Tool-Labs, 06Security-Team: Allow tools to have their own ".tools.wmflabs.org" subdomain - https://phabricator.wikimedia.org/T125589#3008644 (10zhuyifei1999) [10:28:26] yahoy! I'm trying to connect to labs throw filezilla, but can't figure out what instance should i add, acording to docs -> https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_a_graphical_file_manager [10:29:38] alchimista: o/ would you be able to pastebin the settings you've got so far (omitting personal details!) [10:30:03] I've not used filezilla personally, but I've connected to labs with WinSCP :-) [10:30:37] I imagine the instructions at https://wikitech.wikimedia.org/wiki/Help:Access_to_instances_with_PuTTY_and_WinSCP would apply to filezilla too [10:31:05] samtar, i've got the proxycommand configurated, i can enter on console with ssh primary.bastion.wmflabs.org [10:31:19] but i'm tryng to connect by nautilus or filezilla [10:31:35] alchimista: are you accessing tool labs or other labs projects? [10:32:06] zhuyifei1999_, tool labs. i could get in, but /data/ foloder, well, had no data [10:32:38] for tool labs don't use proxycommands and other unnecessary complexities [10:32:54] connect to tools-login.wmflabs.org directly [10:34:15] https://wikitech.wikimedia.org/wiki/Help:Access_to_ToolLabs_instances_with_PuTTY_and_WinSCP [10:35:21] i'm in. with tools-login i got there superfast, why does the docs include the proxycommand? [10:35:39] proxycommand is for labs outside tool labs [10:36:08] so no safety problem on using login directly? [10:36:28] or any other problems? [10:36:38] the nautilus server address is something like sftp://tools-login.wmflabs.org/ [10:36:40] nope [10:37:08] (03CR) 10Volans: "-1 this is not the right way to generate cassandra dummy secrets, there is a cassandra-ca-manager tool for that." [labs/private] - 10https://gerrit.wikimedia.org/r/336462 (owner: 10Andrew Bogott) [10:37:13] that's the easy and usual way [10:38:13] well, many thanks, now filezilla and nautilus work just fine with labs [10:38:20] k np [10:40:47] alchimista: sorry I stepped away, you in? :) [10:49:06] (03PS2) 10Volans: Add missing dummy secrets from production [labs/private] - 10https://gerrit.wikimedia.org/r/335643 [10:52:38] (03PS3) 10Volans: Add missing dummy secrets from production [labs/private] - 10https://gerrit.wikimedia.org/r/335643 [10:54:15] yes samtar [10:55:23] Good to hear :-) [11:07:14] (03CR) 10Filippo Giunchedi: [C: 031] Add missing dummy secrets from production [labs/private] - 10https://gerrit.wikimedia.org/r/335643 (owner: 10Volans) [11:08:51] (03CR) 10Volans: [V: 032 C: 032] Add missing dummy secrets from production [labs/private] - 10https://gerrit.wikimedia.org/r/335643 (owner: 10Volans) [13:29:55] 10Tool-Labs-tools-LTA-Knowledgebase, 15User-Samtar: Install Python - https://phabricator.wikimedia.org/T157465#3009003 (10Samtar) @DatGuy is this still needed per the IRC conversation the other day? [13:37:09] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3009048 (10Marostegui) data has been sanitized in all the hosts (sanitarium2, labsdb1009,10,11). Triggers have been enabled on commonswiki on sanitarium2... [13:40:40] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3009080 (10jcrespo) > create the views - @chasemp / @yuvipanda is that something you will take care of? > add the grants for commonswiki BTW, we said that... [13:58:06] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3009126 (10Marostegui) Replication has been started on db1095 and db1064 and so far it is looking good (they are both catching up after 2 days stopped) [14:00:30] 06Labs, 10Tool-Labs: tools-manifest cannot be build with pdebuild - https://phabricator.wikimedia.org/T157411#3009129 (10scfc) 05Open>03declined I had a look at `sbuild` and judging from the first few lines in the output it also tries to execute `clean` (and thus create the source package) in the host syst... [14:13:57] 06Labs, 10Tool-Labs: tools-manifest cannot be build with pdebuild - https://phabricator.wikimedia.org/T157411#3004336 (10hashar) `clean` is indeed done before the build and if you require dependencies there is no way to define prebuilt dependencies. So one typically has to first install the build deps before... [14:20:17] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3009194 (10Marostegui) We also need to compress commonswiki on all the hosts. [14:35:47] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3009252 (10chasemp) >>! In T153743#3009048, @Marostegui wrote: > data has been sanitized in all the hosts (sanitarium2, labsdb1009,10,11). > Triggers have... [14:44:52] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3009323 (10Reguyla) I totally understand I just offered that as a possibility but it makes sense that it was used for early adopters. I hadn't realized that. [14:55:02] (03CR) 10Andrew Bogott: "My patch made the puppet compiler work, which was all I really cared about. Thank you for improving things though!" [labs/private] - 10https://gerrit.wikimedia.org/r/336462 (owner: 10Andrew Bogott) [15:08:05] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3009402 (10chasemp) > maintain-views --all-databases --replace-all --debug on all three, small note labsdb1010 had that same issue which needed a flush pr... [15:50:02] chasemp hi, im wondering what does this https://phabricator.wikimedia.org/rOPUP1c6ef25d5bf3bcd63a8de55d3b4765db2da2a7a0 do? [15:50:18] Should we disable it on labs for phabricator (instance)? [15:51:22] paladox: I'm tied up atm but that's the ssh service that phab uses for git/diffusion iirc [15:52:19] Ok thanks, i am trying to figure out if that was needed, but sounds like it dosent since on labs if we give a public ip that will work. :), i had to fix that for labs in https://gerrit.wikimedia.org/r/#/c/333358/31 [16:00:37] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3009591 (10Marostegui) >>! In T153743#3009402, @chasemp wrote: >> maintain-views --all-databases --replace-all --debug > > on all three, small note labsdb... [16:18:14] anyone know if Ocaasi uses IRC? [16:21:31] 06Labs, 10Analytics-Tech-community-metrics: http://korma.wmflabs.org/ got erased - https://phabricator.wikimedia.org/T156253#3009632 (10Aklapper) 05Open>03declined Thanks Luis for the explanation / how-to! I've decided that we should not spend time fixing this (will not re-set up the legacy korma.wmflabs.... [16:22:55] PROBLEM - Puppet staleness on tools-docker-builder-04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [43200.0] [16:27:53] samtar: no idea, but if they have a phab profile often ppl put their irc name on there [16:28:11] Had a look, seems not :( no worries! [16:31:50] chasemp: that "nova fullstack testing" change is quite nice :} [16:32:05] yesterday I was looking whether openstack itself reports such metrics [16:34:01] and nodepool has some metrics regarding how long it takes to delete an instance https://grafana.wikimedia.org/dashboard/db/nodepool-tasks?panelId=27&fullscreen&from=now-7d&to=now :) [16:48:29] 06Labs, 10Tool-Labs: Tool Labs users .bashrc file does not exist for tools accounts - https://phabricator.wikimedia.org/T131561#3009749 (10zhuyifei1999) [[https://github.com/wikimedia/operations-puppet/blob/49ee1c7c5ceb9f11bdd49361710566220a0f4471/modules/role/files/labs/db/maintain-dbusers.py#L305|`maintain-d... [16:50:56] 06Labs, 10Tool-Labs, 07LDAP: User tools.admin is in group project-tools - https://phabricator.wikimedia.org/T138182#3009757 (10zhuyifei1999) [17:03:04] yeah thanks hashar and btw the nodepool state check is there but silenced till tomorrow to make sure it's sane [17:12:35] 06Labs, 10Tool-Labs: Make tools-instances that don't need NFS not have NFS - https://phabricator.wikimedia.org/T111716#3009807 (10zhuyifei1999) [17:12:39] 06Labs, 10Tool-Labs, 07Wikimedia-Incident: Unmount unneeded NFS mounts from tool labs hosts - https://phabricator.wikimedia.org/T136222#3009810 (10zhuyifei1999) [17:21:22] 06Labs, 10Tool-Labs: virtualenvs set umask to 022 - https://phabricator.wikimedia.org/T152759#3009815 (10scfc) Note to self: Next time write down the name of the tool where you see something strange so you can check again :-). [17:25:20] 06Labs, 10Tool-Labs: Problem with utf-8 on Grid - https://phabricator.wikimedia.org/T143691#3009866 (10zhuyifei1999) [17:25:23] 06Labs, 10Tool-Labs: jsub and utf8 - https://phabricator.wikimedia.org/T60784#3009864 (10zhuyifei1999) [17:30:46] 06Labs, 10Tool-Labs: Help a new volunteer dev get his timeline script on Tool Labs - https://phabricator.wikimedia.org/T110557#3009872 (10zhuyifei1999) 05Open>03declined Closing as Declined: old ticket and possibly will never have any progress per coren above. [17:35:47] 06Labs, 10Tool-Labs: Update static tools host to be jessie - https://phabricator.wikimedia.org/T139743#3009909 (10bd808) [17:42:31] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: keystone admin api occasionally refuses connection - https://phabricator.wikimedia.org/T156337#3009928 (10Andrew) Hm, interestingly I see the same symptom (periodic connection failures) with the public API, but it errors out differently, like this: ('Connec... [17:57:35] 06Labs, 10Tool-Labs: Create a fonts CDN for use on Tool Labs - https://phabricator.wikimedia.org/T110027#1566983 (10zhuyifei1999) A while ago I wrote this for another project I was working on, might have some use here: {P4911} [18:00:29] chasemp: awesome. Thank you very much :-} I am off place is closing [18:02:07] 06Labs, 10Tool-Labs: qstat missing on exec hosts - https://phabricator.wikimedia.org/T103968#3010054 (10zhuyifei1999) [18:02:09] 06Labs, 10Tool-Labs: exec hosts are not allowed to view sge queue (qstat) - https://phabricator.wikimedia.org/T104611#3010056 (10zhuyifei1999) [18:06:45] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#3010067 (10zhuyifei1999) 05stalled>03declined Closing as Declined as the tool has basically been de-facto abandoned, without a license. It's very unlikely that other developers will fix a no-license tool. Feel free to reop... [18:08:08] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#3010072 (10zhuyifei1999) [18:08:10] 06Labs, 10Tool-Labs: kmlexport perl script memory usage - https://phabricator.wikimedia.org/T99236#3010071 (10zhuyifei1999) 05Open>03declined [18:12:00] 06Labs, 10Tool-Labs: Tool Labs users .bashrc file does not exist for tools accounts - https://phabricator.wikimedia.org/T131561#3010081 (10zhuyifei1999) [18:12:04] 06Labs, 10Tool-Labs: Get rid of toolwatcher, use skeleton homedirs instead - https://phabricator.wikimedia.org/T91235#3010084 (10zhuyifei1999) [18:13:35] 06Labs, 10Labs-Infrastructure, 10Tool-Labs, 10DBA: Make (redacted) log_search table available on ToolLabs - https://phabricator.wikimedia.org/T85756#3010092 (10zhuyifei1999) [18:15:05] 06Labs, 10Tool-Labs: Rewrite /usr/local/bin/crontab in python; fix bugs - https://phabricator.wikimedia.org/T156174#3010099 (10zhuyifei1999) {T86446} would be another bug [18:19:22] yuvipanda: Is https://quarry.wmflabs.org/query/16265 the right way to access a database on tools-db ? [18:26:34] yuvipanda: For when you get back or the lurkers, I want to do https://tools.wmflabs.org/multichill/queries/wikidata/unmatched_rkdartists.sql in quarry and get https://tools.wmflabs.org/multichill/queries/wikidata/unmatched_rkdartists.txt [18:27:17] multichill: quarry has no access to tools-db [18:27:28] (also I'm on vacation I guess) [18:27:34] (so I'll be gone soon) [18:27:55] yuvipanda: go away we'll call you if you are needed :) much love say hi to your friend [18:28:10] will do chasemp thanks [18:28:27] No tools-db access :-( [18:28:35] yuvipanda: Have a really nice vacation! [18:29:06] 06Labs, 10Tool-Labs: Clean up list of projects on Tool Labs home page and add Tomcat tools - https://phabricator.wikimedia.org/T51937#3010187 (10zhuyifei1999) 05Open>03Resolved > Clean up list of projects on Tool Labs home page They are moved to ?list > add Tomcat tools ?status contains them > Categorizati... [18:32:00] 06Labs, 10Tool-Labs, 06Tool-Labs-standards-committee: Request to be added as maintainer to abandoned bibleversefinder/ tool - https://phabricator.wikimedia.org/T91585#3010209 (10zhuyifei1999) [18:38:58] 06Labs, 10Tool-Labs: create procedure to show all granted permission to foreign user for own user databases - https://phabricator.wikimedia.org/T69552#3010243 (10zhuyifei1999) 05Open>03declined Closing as Declined: With the change that the default grant no longer includes a `WITH GRANT OPTION`, users canno... [18:47:42] 06Labs, 10Tool-Labs: qstat missing on exec hosts - https://phabricator.wikimedia.org/T103968#3010285 (10zhuyifei1999) [18:47:45] 06Labs, 10Tool-Labs: Make gridengine exec hosts also submit hosts - https://phabricator.wikimedia.org/T123270#3010288 (10zhuyifei1999) [19:00:28] 06Labs, 10Tool-Labs: Sorting by CPU/VMEM columns doesn't sort by their value on http://tools.wmflabs.org/?status - https://phabricator.wikimedia.org/T69737#3010317 (10zhuyifei1999) 05Open>03Invalid I don't think this is sortable right anymore, so this is moot. Feel free to reopen if I'm wrong or the situat... [19:01:51] 06Labs, 10Tool-Labs: Limit number of jobs users can execute in parallel - https://phabricator.wikimedia.org/T67777#3010333 (10zhuyifei1999) [19:01:54] 06Labs, 10Tool-Labs: Make gridengine exec hosts also submit hosts - https://phabricator.wikimedia.org/T123270#3010336 (10zhuyifei1999) [19:10:21] 06Labs, 10Tool-Labs: Add 'file a bug' link to tool labs error pages - https://phabricator.wikimedia.org/T90570#1062129 (10zhuyifei1999) -1. Right now the "no webservice" page (eg https://tools.wmflabs.org/a/) is imo quite informative on who's the maintainer and asks the user to contact the maintainer. The de-f... [19:30:35] chasemp: We're playing around with OpenRefine and Wikidata. Needs quite a bit of number crunching power while running. Do we already have something for short lived vm's like that? [19:31:27] not really, we don't have a fast track for temp projects or anything but depends on what quite a bit means [19:31:39] can you quantify in terms of like large instances? [19:31:46] Just thinking out loud here [19:31:50] sure np [19:32:59] The first cloud I build was about 7 or 8 years ago. The use case was researcher who were doing some sort of program on their laptop/desktop and that was taking too long. This could just P2V (physical to virtual) their machine and run it as a VM [19:33:04] 10Tool-Labs-tools-Xtools, 06Community-Tech: XTools: Articleinfo - https://phabricator.wikimedia.org/T157602#3010525 (10MusikAnimal) [19:33:19] But the VM would be on some big server. Big in that direction. So say 4 times more power than your average laptop. [19:33:38] we wouldn't run an image from outside, and that's...not really a measurement I can work with easily [19:33:54] we could definitely go for one xlarge instance which is def more than my laptop persay by a lot [19:33:57] and see how it goes for you [19:34:14] No outside image in this case, that was just what feels the same [19:34:19] gotcha [19:34:48] make a task for a new temp project outlining and ask for things in concrete terms using our large instance as a unit of measure [19:34:54] odds are we can accomodate [19:35:07] esp if it's not super long lived [19:35:52] multichill: https://phabricator.wikimedia.org/T76375 [19:35:52] I'm just going to let that sink in a bit. I happen to have access to a bit of resources at work for testing purposes so might spin up a VM there to see how that goes ;-) [19:36:01] sure [19:38:06] Would be cool to have something like PAWS. Those vm's are not up all the time,right? [19:39:18] chasemp: https://github.com/OpenRefine/OpenRefine is the tool btw [19:40:54] PAWS spins off a container per user multichill on kubernetes [19:40:57] it's a very different model [19:41:10] 10Tool-Labs-tools-Xtools, 06Community-Tech: XTools: Articleinfo - https://phabricator.wikimedia.org/T157602#3010574 (10Matthewrbowker) > Hopefully we can add tests? I don't think there is a test suite in place, but it might be nice to add one. It is possible, symfony comes with a built in test framework based... [19:41:57] chasemp: Container would be fine, probably even better, just depends on how much resources it can consume [19:43:15] 10Tool-Labs-tools-Xtools, 06Community-Tech: XTools: Articleinfo - https://phabricator.wikimedia.org/T157602#3010588 (10MusikAnimal) [19:49:28] multichill: that's a bigger thing to bite off, and would need ot run in tools and have a lot of testing I imagine. If you want to get a project to get things going just let us know :) super neat tho, I poked about the openrefine project a bit [19:49:29] 10Tool-Labs-tools-Xtools, 06Community-Tech: XTools: Articleinfo - https://phabricator.wikimedia.org/T157602#3010604 (10MusikAnimal) [19:51:57] 10Tool-Labs-tools-Xtools: xtools on Tool Labs: Rep Lag High - https://phabricator.wikimedia.org/T156345#3010620 (10Matthewrbowker) 05Open>03Resolved Actually, the xtools rewrite already fixes this issue. So I'm going to close this ticket as resolved, it will be fixed when this rewrite is deployed later this... [20:06:35] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Improve documentation for setting up XTools on a local machine - https://phabricator.wikimedia.org/T157609#3010730 (10MusikAnimal) [20:06:58] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Improve documentation for setting up XTools on a local machine - https://phabricator.wikimedia.org/T157609#3010730 (10MusikAnimal) a:03MusikAnimal [20:10:42] 10Tool-Labs-tools-Xtools, 06Community-Tech: XTools: Articleinfo - https://phabricator.wikimedia.org/T157602#3010805 (10MusikAnimal) [20:10:46] 10Tool-Labs-tools-Xtools, 06Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#3010806 (10MusikAnimal) [20:26:03] 10Tool-Labs-tools-LTA-Knowledgebase: Add discussion area - https://phabricator.wikimedia.org/T157223#3010873 (10DatGuy) p:05Triage>03Normal [20:26:57] 10Tool-Labs-tools-LTA-Knowledgebase: Add option for tool admins to change write access - https://phabricator.wikimedia.org/T156181#3010887 (10DatGuy) p:05Triage>03High a:03DatGuy [20:27:33] 10Tool-Labs-tools-LTA-Knowledgebase: Create search function - https://phabricator.wikimedia.org/T157464#3010891 (10DatGuy) p:05Triage>03High [20:27:36] 10Tool-Labs-tools-LTA-Knowledgebase: Add option for tool admins to change write access - https://phabricator.wikimedia.org/T156181#2966635 (10DatGuy) p:05High>03Normal [20:27:46] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Empty default security group for newly created project - https://phabricator.wikimedia.org/T136871#3010894 (10Andrew) 05Open>03Resolved Should be fixed by the above ugly patch [20:29:34] chasemp: Installing it on a *nix system is super easy. Clone, compile, run. But bigger datasets (the 100.000+ ones we have on WIkidata) consume a lot of resources. It's a shame Google isn't offering the web based version anymore [20:32:06] hi there. Some users at dewiki reported issues with toollabs: They get a timeout when they are trying to load tools.wmflabs.org [20:32:21] it works for me, but seems like it does not work for all. are there any known issues? [20:36:50] bd808, andrewbogott yuvipanda madhuvishy ? [20:37:01] It's nothing that we've heard much about. [20:37:10] Is it something that happens 100% of the time for certain users, and not at all for others? [20:37:13] Or intermittently for all users? [20:38:51] andrewbogott: at https://de.wikipedia.org/w/index.php?title=Wikipedia:Administratoren/Anfragen&curid=6974524&diff=162451525&oldid=162451422#tools.wmflabs.org we have 5 users talking about it, for 3 it works, 2 get an timeout every time [20:38:54] Sagan: it would also be useful to have timeframe information, like, "This worked last week but broke today" etc. [20:39:12] the report was from 13 minutes ago [20:41:39] Ok… this sounds like probably a network/ssl troubleshooting situation. what do those users have in common, have they tried using different browsers, are they running older operating systems, are they behind corporate VPNs, etc. [20:41:57] But it would also be good to know if it just now stopped working and they didn't change anything, or if it never worked from their specific laptop/location/etc [20:43:45] probably best to get that in a ticket [20:43:58] ok, I will tell them [20:44:31] Sagan: the short answer is that this is almost certainly not an issue on our end, other than a possible "we're using technology that doesn't work with your weird setup" [20:44:38] Sagan: kind of a headscratcher, has to be some common element there. So far I can't get it to fail. But I'm in the u.s. [20:45:09] be interested to know if http and https behave the same for them [20:45:11] Sagan: if OTHER wmf services (like wikipedia) are also failing for them then something more interesting might be happening [20:45:47] andrewbogott: at least wikipedia works, since the reported it there ;) [20:45:53] but thx for you answers :) [20:45:53] true [21:05:37] andrewbogott: ok, according to the last comment 5 minutes ago, it works again for one of those 2 users [21:05:49] the other one was an IP, so probably we don't get an answer [21:05:53] That's mysterious :( [21:10:11] * Platonides notes that tools.wmflabs uses a different tls configuration than wikipedia [21:12:26] tls issues typically result in 'connection reset', not in a timeout [21:22:54] valhallasw`cloud: yeah, good point. I don't have any good theory for what was happening. [21:23:31] could just be a temporary routing issue [21:23:44] yeah, although I'd expect that to break access to wikipedia as well [21:24:50] hm. yes, indeed [21:33:38] andrewbogott: the IP replied now as well, it works for them again [21:33:46] huh [21:33:57] so I would say it's resolved actually, isn't it? :D [21:34:18] If it stays fixed, then yes! [21:35:46] Change on 12www.mediawiki.org a page Wikimedia Labs/Tool Labs/Design was modified, changed by 101.99.52.237 link https://www.mediawiki.org/w/index.php?diff=2389807 edit summary: [21:38:39] I have one Flask route defined as @app.route('/search', methods=['GET', 'POST']) def searchHome(): {{code}} and another defined as @app.route('/search/', methods=['GET', 'POST']) def search(article_name): {{code}}. When I do return redirect(url_for('search', article_name=articlename)) in searchHome(), I get BuildError: Could not build url for endpoint 'search'. Did you forget to specify values ['article_name']?. [21:38:39] articlename being None or not is irrelevant because the BuildError comes before entering the first line in the searchHome() method [21:38:40] seems like an app init/compilation type of error [21:39:01] Flask module for python btw ^^ [21:46:56] JustBerry: might be the _, but I'm not entirely sure [21:50:32] JustBerry: #python and #pocoo could be good places to ask as well [21:50:41] valhallasw`cloud: asked in pocoo already [22:08:13] Change on 12www.mediawiki.org a page Wikimedia Labs/Tool Labs/Design was modified, changed by Shirayuki link https://www.mediawiki.org/w/index.php?diff=2389827 edit summary: Reverted edits by [[Special:Contributions/101.99.52.237|101.99.52.237]] ([[User talk:101.99.52.237|talk]]) to last revision by [[User:Giftpflanze|Giftpflanze]] [22:13:15] Sagan: I need your help. I just fucked up. [22:14:46] Sagan: essentially I just accidentally dropped a critical table on my DB. [22:15:02] I was wondering if there is a backup that can be implemented [22:16:00] andrewbogott: yuvipanda ^ [22:17:59] Anyone??? [22:18:00] Cyberpower678: I don't know if there are backups. Best bet would be to email jynus or wait for him to start work tomorrow (he's in Spain). [22:18:31] jynus: he's the DB goto? [22:19:32] andrewbogott: ^ [22:19:41] I thought Sagan was in charge of the DB. [22:19:49] Cyberpower678 did you backup the db? [22:20:11] Zppix: I was working on backing it up, but I accidentally dropped it instead. [22:20:35] I was going to run a DB maintenance script on it. [22:20:42] Cyberpower678 nice one, hmm is it a bot of yours? [22:20:49] InternetArchiveBot [22:21:12] Cyberpower678: yes, jynus. [22:21:22] Cyberpower678 mangodb? [22:21:40] Zppix: what's that? [22:21:40] question, how do I get removed from being a tool admin/maintainer? [22:21:52] Fredddie wikitech service groups [22:22:10] Cyberpower678 an alt to sql [22:22:32] Okay, I don't understand what it has to do with my situation. [22:22:35] another user added me, but didn't really explain what to do [22:22:44] Fredddie: go to https://tools.wmflabs.org/?list, find the tool, click the [manage maintainers] link, remove yourself. [22:22:51] so it's kinda useless to me [22:22:53] thanks [22:23:22] Fredddie: and come back to help with something where there are better docs! :) [22:23:53] Zppix: why mention mangodb? [22:24:03] Cyberpower678 im trying to determine if theres some undo function [22:24:13] It's the tools-db DB [22:24:18] Cyberpower678: I think you may be confusing S.agan with 'Sean' who was our head DBA before jynus [22:24:32] I just got another reminder to migrate to Trusty, although suggestbot is no longer listed at https://tools.wmflabs.org/precise-tools/ Is there any action needed on my part (e.g. reply to the email)? [22:24:37] Cyberpower678 nope your screwed unless manger of dbs has access to repair it [22:24:56] FUCK [22:24:56] Nettrom be sure webservices arent on precise either (if applicable) [22:25:14] Cyberpower678 well you took it better than i thought. [22:25:19] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic, and 2 others: Remove support for precise OGE exec hosts - https://phabricator.wikimedia.org/T94792#3011272 (10Matthewrbowker) [22:25:22] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Migrate XTools from Ubuntu Precise to Trusty - https://phabricator.wikimedia.org/T157123#3011269 (10Matthewrbowker) 05Resolved>03Open Since I just received another warning email, this appears to not be resolved. Reopening. [22:25:25] Zppix: based on my reading of the documentation and checking of the manifest, that’s not a problem, thanks though :) [22:25:36] Zppix: you have no idea. [22:25:55] Cyberpower678 oh i can imagine i would of set my pc on fire [22:26:14] Nettrom: if you are not showing on https://tools.wmflabs.org/precise-tools/ you are probably good. The script that is sending the emails doesn't have as much logic to check for changes in the last 7 days as the web service does. [22:26:20] Zppix: I'd rather set my house on fire. I need my PC. :P [22:26:30] Cyberpower678 wouldnt your pc be in your house? [22:26:41] I'l just bring it outside. [22:26:43] bd808: that’s what I suspected, thanks! [22:27:13] Cyberpower678 is there anything i could do to help? [22:27:44] I mean i love IAB i cant imagine a day w/o it [22:27:46] Zppix: like what? If you can magically undrop the DB, I would be ecstatic. [22:28:03] *table [22:28:04] Cyberpower678 if i had that magic power trust me i wouldnt be here right now [22:28:18] mangodb or mongodb? [22:28:22] https://www.google.com/search?q=mongodb+%2B+%22never+use%22 [22:28:30] mutante try tools-labs-db [22:28:31] Fortunately a safety kicked in and didn't drop the other tables. [22:28:42] Cyberpower678 so IAB is functional? [22:28:46] But all enwiki meta data is gone. [22:29:34] Zppix: it can function, but the table is important for it to function right. The bot will be slow as fuck without it. [22:30:28] Zppix: It's a table with more than 20 million rows. [22:30:34] Cyberpower678 and it can recover eventually by itself? [22:30:36] cant* [22:30:49] After many months perhaps. [22:31:27] what it all consist of [22:31:40] Zppix: in order to rebuild the entire table, the bot has to analyze all 5 million articles on Wikipedia. [22:32:01] that sounds like fun.... [22:32:06] Oh hey the global meta data is still here. :D [22:32:17] That's most critical table. [22:32:18] Cyberpower678 thats a good thing [22:32:25] Cyberpower678 i see how it is :P [22:33:13] The enwiki table is important in that it optimizes the bot's performance, and allows to not need to relentlessly assault the DB for information. The DB is 20GB [22:33:26] couldn't you query enwiki's db for articles IAB has analysed before then copy and paste that data into iab's db [22:33:39] how long will it take to analyze 5 million articles? [22:33:47] mutante months [22:33:50] ..wow [22:34:09] Without it, since the global data is still there it will run reasonably faster but hit the DB harder as it rebuilds the enwiki table [22:34:10] 1 article analysed by the bot is 5 new articles created [22:34:25] so enwiki will take a performance hit ? Cyberpower678 [22:34:40] mutante: Unless I restore all 28 active enwiki workers. [22:34:51] Then it can take a month. [22:34:57] And not months/years [22:35:07] Cyberpower678 if theres anything i can do either onwiki or whatever tell me [22:35:35] I would like to be able to restore the table from a backup if possible. If not possible, the global data will help to rebuild the table. [22:36:31] Zppix: the consequence now is you can't see what pages any given URL is on with the IABot Management Interface. https://tools.wmflabs.org/iabot [22:36:42] Cyberpower678 well here have a couple packs of every liquor known to man [22:38:16] * Cyberpower678 airhorns jynus out of bed. [22:38:44] Zppix: I don't drink liquor. How about some nice brews? [22:39:01] Cyberpower678 but i already bought the liquor... [22:40:25] Oh well. [22:40:27] :p [22:40:29] Cyberpower678: https://en.wikipedia.org/wiki/Malt_liquor [22:40:38] half beer, half liquor [22:40:47] sorry, i dont know if there are any backups or i would help [22:40:51] feels with you [22:40:53] Sounds fun. [22:48:50] Cyberpower678: I've been asking the other admins for opinions. Sadly we don't have good news for you. We are 99% sure that there are no periodic snapshots of tool/user databases. [22:49:13] Well that sucks. [22:49:20] There *may* be replication to a secondary server, but that would replicate the table drop too. [22:49:43] there is and I believe and yes [22:49:52] bd808: is it possible to checK? [22:50:15] chasemp: ^ [22:50:37] Cyberpower678: check for a replica? [22:50:44] Cyberpower678 it depends if puppets already caught up to the drop or not [22:50:58] teh replication is nearly instant and not through puppet [22:51:04] its not about puppet. its about mariadb replication streams [22:51:12] oh, i thought puppet did that? [22:51:20] lag is on the order of milliseconds [22:52:37] Zppix: puppet is probably involved in configuring the mariadb replication streams. the actual replication is a function of the db server itself. [22:53:51] :-( [22:54:07] I wish a DB drop wasn't so damn easy. [23:02:26] bd808: o/ [23:02:38] hey JustBerry [23:13:18] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#3011431 (10Andrew) [23:24:20] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Improve documentation for setting up XTools on a local machine - https://phabricator.wikimedia.org/T157609#3010730 (10MusikAnimal) 05Open>03Resolved Done at https://wikitech.wikimedia.org/wiki/Tool:Xtools#Installation_on_local_machine I tried this from sc... [23:31:08] 10Tool-Labs-tools-Xtools, 06Community-Tech: XTools: Articleinfo - https://phabricator.wikimedia.org/T157602#3011510 (10MusikAnimal) [23:36:16] musikanimal: PMed btw [23:38:44] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#3011547 (10Andrew) [23:39:28] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Migrate XTools from Ubuntu Precise to Trusty - https://phabricator.wikimedia.org/T157123#3011553 (10MusikAnimal) 05Open>03Resolved Madhumitha got back to me and said the script they were using may be out of date, and that if our tools don't show up at http... [23:39:31] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic, and 2 others: Remove support for precise OGE exec hosts - https://phabricator.wikimedia.org/T94792#3011555 (10MusikAnimal) [23:43:05] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: keystone admin api occasionally refuses connection - https://phabricator.wikimedia.org/T156337#3011599 (10Andrew) I've convinced myself that this is indeed a result of the uwsgi switch-over. We could always go back... [23:45:39] 10Tool-Labs-tools-Xtools: Bugs section on articleinfo returns incorrect results - https://phabricator.wikimedia.org/T148046#3011637 (10Matthewrbowker) [23:45:42] 10Tool-Labs-tools-Xtools, 06Community-Tech: XTools: Articleinfo - https://phabricator.wikimedia.org/T157602#3011638 (10Matthewrbowker) [23:45:52] 10Tool-Labs-tools-Xtools: Bugs section on articleinfo returns incorrect results - https://phabricator.wikimedia.org/T148046#2713184 (10Matthewrbowker) a:05Matthewrbowker>03None [23:51:51] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: keystone admin api occasionally refuses connection - https://phabricator.wikimedia.org/T156337#3011653 (10bd808) >>! In T156337#3011599, @Andrew wrote: > I've convinced myself that this is indeed a result of the uwsgi switch-over. We could always go back...... [23:52:22] 10Tool-Labs-tools-Erwin's-tools: Unknown Error/MySQL errors - https://phabricator.wikimedia.org/T140421#3011655 (10Akoopal) [23:52:24] 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from Tool Labs - https://phabricator.wikimedia.org/T94790#3011656 (10Akoopal) [23:54:00] 10Tool-Labs-tools-Erwin's-tools: Unknown Error/MySQL errors - https://phabricator.wikimedia.org/T140421#2464064 (10Akoopal) With the current project to phase out precise, this should get more focus.