[00:03:45] bd808: I'm trying to install frozen stuff but there's lots of errors, which is getting pretty frustrating. Is there a way to install them as you go through instead of caching them and installing them all at the end? [00:03:56] frozen pypi requirements that is* [00:05:39] DatGuy: hmmm... so you are doing something like `pip install -r freeze.txt` and not liking how that orders things? [00:06:07] I can't think of a lot of options that would affect how pip does what it does for that [00:06:21] It's mostly just missing requirements [00:06:42] https://pastebin.com/SCpgvFjK [00:07:38] bd808, thanks for that, i'll make a second tool account [00:07:57] bd808, i'll just open a request for deleting the -beta tool once i'm done testing :) [00:08:11] Sveta: sounds like a plan :) [00:08:21] DatGuy: I don't see "GeoIP-Python" in pypi [00:08:53] well I assume it's just GeoIP https://pypi.org/project/GeoIP/ [00:08:56] but under a different name [00:09:38] '10,000+ projects for "geoip-python"' -- https://pypi.org/search/?q=geoip-python [00:10:06] https://pypi.org/search/?q=%22geoip-python%22 is a bit better, only 5 hits [00:10:09] is there a way to ignore any errors [00:10:26] and maybe if I see them later effecting anything then I can manually fix them? [00:10:37] https://stackoverflow.com/questions/22250483/stop-pip-from-failing-on-single-package-when-installing-with-requirements-txt [00:11:10] cheers, I should study up on my google-fu :p [00:11:13] looks like the suggestion there is really manually installing each thing [00:11:35] my google search was: pip install ignore errors [00:12:16] yeah mine were along the lines of 'pip install requirements as you go' [00:39:10] whats the stretch login host? [00:46:36] Betacommand: login-stretch.tools.wmflabs.org [00:46:48] it is listed on https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation#SSH_to_the_Stretch_bastion [00:47:10] bd808: thanks, it was a copy paste fail that confused me. I left out the .tools. [00:47:49] ah, yeah. We are trying to stop polluting the top level domain with tools stuff :) [00:47:54] bd808: thats what Im reading thru, getting the ball started on switching over [00:48:01] cool [00:48:24] * bd808 is impressed that the nag emails are getting a response [00:48:38] bd808: why? [00:49:23] good maintainers try to keep abreast of the issues [00:50:32] Betacommand: true enough! There was very little response to the first announcement on cloud-announce last month so I was a bit worried that folks had stopped reading emails from me ;) [00:51:04] "Oh, that's just bd808 breaking all my stuff again. Ugh" [00:51:16] bd808: no, its the lets wait a little and let them iron out the bugs [00:51:52] we've killed off quite a few. I'm fighting an ugly one right now with pip and pyyaml [00:52:05] bd808: I tend to find them regardless of when I get my hands dirty though [00:52:29] bd808: thats why people tend to wait a little [00:55:12] bd808: is there a stretch host I can test run scripts on without sending them to the grid yet? [00:56:15] Betacommand: I have not built a second stretch bastion, but you can use one of the stretch grid exec nodes. You can see all of them and how loaded they are at https://tools.wmflabs.org/sge-status/ [00:56:51] bd808: how would I go about switching hosts? [00:58:10] bd808: most of my sshing is into the box I want to work on, not jumping boxes once logged in :) [00:58:10] Betacommand: the easy mode way is `ssh tools-sgeexec-0902.tools.eqiad.wmflabs` from the stretch bastion [00:58:20] ah thanks [00:59:51] you can do that after becoming your tool too which is sort of like pretending you have submitted a job to the grid [01:00:31] thanks, prefer to test one step at a time to find out whats broken [01:01:01] *nod* hopefully your issues will be few and easy to solve :) [01:02:13] my biggest headache I suspect will be the webservice [01:07:06] !log tools Creating tools-sgebastion-07 [01:07:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [02:00:17] bd808: I cant start my webservice [02:10:03] Betacommand: do you have any error messages or ? [02:10:29] bd808: (configfile.c.1154) source: /var/run/lighttpd/betacommand-dev line: 70 pos: 1 parser failed somehow near here: (EOL) [02:11:53] Betacommand: ooh that sounds new and fun. what's the tool name? I can stare at the lighttpd conf and see if I can spot the problem [02:12:07] bd808: Betacommand-dev [02:16:51] Betacommand: are you trying to run this on the grid or in kubernetes? [02:17:39] bd808: grid [02:17:40] I think we can use 'lighttpd -t -f ...' to test the generated config but there are slightly different versions on the grid vs in kubernetes [02:18:34] kub has issues with python that I dont feel like debugging at the moment [02:21:10] hmmm... `/usr/sbin/lighttpd -t -f .lighttpd.conf` says OK, so I guess the problem must come from the combined config... [02:23:01] Duplicate array-key '.txt' [02:25:00] Betacommand: here's the validation test -- https://phabricator.wikimedia.org/P8062 [02:25:29] line 70 is `$HTTP["url"] =~ "/$" {` [02:26:51] bd808: what is the solution? [02:28:01] that I don't know! I don't think I've ever used the $HTTP[...] construct. [02:28:33] I can poke around in google, but right now I'm being called to dinner so it will be a bit before I can dive in [02:28:46] ok [02:31:00] bd808: like I said I always find the find errors [02:31:08] *fun [03:04:09] Betacommand: the error seems to be related to the mimetype.assign block. If I comment that out then the syntax checker is happy [03:05:06] more specifically the ` ".txt" => "text/plain;charset=UTF-8",` line is the problem [03:05:33] which matches up with the "Duplicate array-key '.txt'" error message [03:07:01] this clashes with the '".txt" => "text/plain",' mime setting that /usr/share/lighttpd/create-mime.assign.pl generates into the runtime config [06:32:16] !log tools.jouncebot Migrated bot from Trusty grid to Kubernetes [06:32:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jouncebot/SAL [11:08:47] !log tools flip tools-prometheus.wmflabs.org to tools-prometheus-02 - T215272 [11:08:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:08:51] T215272: Upgrade Prometheus to 2.6 in deployment-prep and tools - https://phabricator.wikimedia.org/T215272 [11:26:14] bd808: Ive gone ahead and removed that line for now, but I am running into encoding issues with txt files now [12:25:55] !log tools install aptitude in tools-sgebastion-06 [12:25:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:49:18] !log tools upgraded all packages in SGE cluster [13:49:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:56:12] !log admin running `maintain-views --all-databases --replace-all` on labsdb1009 [14:56:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:02:37] !log ores staging ores-wmflabs-deploy:5bd6b61 [16:02:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [16:08:20] Amir1, looks like revscoring 2.3.3+ breaks all of the models we are currently running because they expect to get timestamp_str [16:08:44] halfak: yes, I'm rebuilding all models [16:08:44] So we can't complete this deployment until all of the models are rebuilt. [16:08:56] OK. [16:08:57] the editquality is already up for review [16:09:10] Right. Did you take a pass through the fitness stats? [16:09:22] I get error for other ones, e.g. dratfquality requires admin user+pass [16:09:23] Any big changes jump out at you? [16:10:23] Betacommand: it seems like this may be a behavior change from older versions of lighttpd (not allowing a key in a config map to be overwritten), but I have not tried to investigate in their upstream repo/bug tracker [16:11:08] Woops. I'm getting into ORES stuff in the wrong channel. [16:11:12] * halfak --> -ai :) [17:03:17] bd808: understood, do we have a solution then? Right now my UTF-8 txt files are being parsed with the โ€œWesternโ€ encoding causing some mangled text [17:25:52] Betacommand: I do not have a solution for you at this time. It would probably be a good idea to write up a bug report in phabricator so we can at least keep track of the problem. Ideally that bug report would give some minimal way to recreate the user facing part of the problem (incorrect encoding detection by the browser) [18:08:38] bstorm_: Came across https://phabricator.wikimedia.org/T166798, if I understand correctly, this was effectively decided on already right (with the _compat views). [18:10:26] ๐Ÿ‘€ [18:11:03] Yes. We basically did it, rolled it back and put them in the _compat views. [18:11:21] And I never actually saw that ticket the whole time ๐Ÿ˜ [18:14:18] cool, well, it'll be a nice close then. Those are the best ones, right? tickets you find you already did :D [18:15:00] Yup! [18:15:11] closed as invalid [18:15:16] with a note as to why [18:32:52] !log tools Stopped webservice for `tools.quentinv57-tools` for T210829. [18:32:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:32:56] T210829: Archive/Delete quentinv57-tools - https://phabricator.wikimedia.org/T210829 [18:51:43] Hi. I have received some trusty deprecation warnings due to some cronjobs. Is migrating them that simple of just logging to loging-stretch and issue the commands from there? [18:54:04] hauskatze: https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation#Move_a_cron_job [18:54:36] bd808: yeah, but is really that simple? Do I have to delete the cronjob from login-trusty afterwards? [18:55:19] bd808: forget it, 'crontab -r' [18:55:20] hauskatze: yes, the `crontab -r` step removes the crontab from the legacy Trusty cron server [18:55:24] :) [18:55:57] bd808: also, I restarted the 'stewardbots' webservice using kubernetes last time but it uses PHP 5, would it work if I use PHP 7.2 ? [18:55:58] hauskatze: documentation updates welcome. I did not add a lot of description in those sections [18:56:31] I also saw that kubernetes can be used to schedule jobs, but it looked kinda complex to me that day :) [18:56:51] I'll try to migrate mabot's now [18:57:22] hauskatze: it will probably work. there are some breaking changes between php5 and php7 that might require fixes. [18:58:30] I'm not ready to recommend folk moving their cron jobs to Kubernetes scheduled tasks yet. As you saw it is a bit different and mostly untested in our environment [18:58:58] maybe in 3-6 months we will have that figured out :) [18:59:26] okay so I'll just use jsub crontab commands from login-stretch [18:59:34] I'm on it [19:00:08] ๐Ÿ’ฏ thanks for your efforts [19:03:01] tools.mabot should be done [19:03:24] though when I do crontab -e and see the file name it is stored in a tmp/ file [19:03:41] always have been though, and while it works I don't care much [19:08:10] hauskatze: yeah. we do 'interesting' things with cron. When you use `crontab` on a bastion you are actually using a python script that communicates with tools-cron-02 over ssh. The remote crontab is scp'ed over to the bastion, edited, and then scp'ed back to the cron host (or at least that's the easy way to explain it) [19:08:40] oh [19:11:18] bd808: jobs working fine for now in stretch :D [19:11:24] for mabot [19:11:36] * hauskatze is happy [19:17:42] !log tools Stopped webservice of `tools.sulinfo` which redirects to `tools.quentinv57-tools` which is also unavalaible [19:17:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:18:01] hope that this means no more warnings for both of them [19:19:45] hauskatze: we will probably get nagged again next Thursday for those ones you have shutdown, but not the week after that. My "things that have moved" check script doesn't know how to deal with things that have been shutoff entirely [21:19:17] if i have a list of cloud projects how do i get from there to "an email to the owners of all these projects", how do you guys do the notifications [21:19:42] i want to push for deprecating / replacing a puppet role [21:19:47] that uses outdated modules [21:20:00] and openstack-browser tells me which projects are affected [21:20:48] mutante: I'm guessing bd808 has written some tooling for this sort of thing for the upgrade notifications [21:21:32] i think this might also bring some inactive ones to surface that can be closed [21:22:14] what i really want is fix "role simplelamp" to use httpd instead of apache and mariadb instead of mysql [21:22:14] !log wikidataconcepts deleting VMS and project as per T185430 [21:22:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidataconcepts/SAL [21:22:18] T185430: Remove wikidataconcepts project once migrated to wmde-dashboards - https://phabricator.wikimedia.org/T185430 [21:22:24] mutante: we usually do phab tasks and try to match the maintainers up to Phab identities or trick quiddity into helping us use mass-message on wikitech [21:22:28] both the mysql and apache class are supposed to be removed [21:23:18] bd808: ah, ok.. i could do that myself then. somehow i thought you get emails from LDAP or so [21:23:20] mutante: harej may be able to help you actually if he's not swamped with other things right now [21:23:33] well, i could look those up too [21:23:40] bd808: ok, thanks [21:24:07] makes a ticket to start with [21:24:08] yeah, I did that for the recent toolforge email whine but we try not to abuse direct email [21:24:23] *nod* [21:25:08] I have a hunch that all things using simplelamp are actually kind of broken. If I'm remmbering correctly the apache config wipes out vhost things not managed by puppet on each run [21:25:21] Kre-nair came up with a bash one-liner; hold on while I retrieve it [21:25:48] mutante: is simplelamp actually used in prod, or is it cloud vps only? [21:25:58] $ ldapsearch -x cn=project-graphql member -LLL | grep -o "uid=.*,ou=people" | sed -e 's/uid=//' | sed -e 's/,ou=people//' | xargs -I% -n1 ldapsearch -LLLx uid=% mail | grep mail: | sed -e 's/^mail: //' [21:26:05] replace graphql with the project of your choice [21:26:41] bd808: cloud vpd only [21:26:51] This produces a list of email addresses you can copy into an email client [21:27:15] bd808: but in this case i want to be able to delete mysql/apache modules entirely from the repo and all the prod stuff is converted now [21:27:29] i left simplelamp to the end because of the affected users [21:27:44] harej: awesome:) [21:27:56] mutante: it might be good actually to open a ticket about deprecating and removing that role entirely, add people we can find to the discussion, and figure out if they can all use mediawiki-vagrant or something else instead [21:27:57] well, how about i match them in phab but the emails help me [21:28:25] bd808: already have the create form open :) [21:28:27] !log tools.zppixbot moving Cron from trusty to stretch [21:28:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [21:28:32] mutante: https://tools.wmflabs.org/contact/ is pretty good for finding same-name phab accounts for people [21:28:42] bd808: i want to just fix the class to use the new modules but tell users about it [21:28:50] and in case i break anything while doing so [21:29:04] ok, thanks for that tool, didnt know [21:29:54] * bd808 should figure out how expose the SUL-LDAP-Phab account links from Striker to tools.contact [21:33:11] harej, mutante: this produces a list of members [21:33:18] rather than just admins [21:33:32] aren't they usually one in the same, except for the tools project? [21:33:48] certainly very wrong for the tools project but likely quite a bit out for others too [21:33:50] it varies quite a bit from project to project [21:34:21] I would guess that tools and deployment-prep are the outliers with many members who are not admins though [21:34:25] i'm not sure what value a non-admin membership is, unless you have more finely grained permissions/security groups i suppose [21:34:54] from what I've seen cloud projects aren't all that sophisticated. as bryan says, those two are definitely outliers [21:36:01] harej, with just membership you can SSH in and usually sudo to root [21:36:14] can't modify instances, domains, etc. in horizon [21:36:20] whole level of control less [21:36:23] huh, that's what admin means [21:37:10] i would not have intuitively made that association. for instance i am a member of tools and this doesn't give me any special powers other than access to toolforge [21:37:22] I think in deployment-prep it's like 50-50 [21:37:29] yeah [21:37:35] tools has different sudo rules than anywhere else [21:37:52] sudo is restricted to a special subset of members [21:38:05] i have a few bones to pick with horizon's UI but i don't think that's a priority for me [21:38:09] hence the 'usually' [21:40:01] !log tools.bash Restarted webservice [21:40:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bash/SAL [21:40:07] Reedy: ^ [21:40:10] seems better [21:40:22] whee [21:40:26] first one that appeared was me [21:40:58] https://tools.wmflabs.org/bash/search?p=0&q=reedy [21:41:21] this is one of my favorites -- https://tools.wmflabs.org/bash/quip/AU7VV5IC6snAnmqnK_1E [21:41:29] nothing like a good regex quip [21:42:35] :) [21:47:28] ticket i made https://phabricator.wikimedia.org/T215662 ticket i already made in the past and forgot https://phabricator.wikimedia.org/T202574 .. bunch of stuff linked :) [21:48:54] officetools.sugarcrm :p [21:49:30] just saw discussion on installing another crm _not_ in office, heh [21:51:51] contact info tool works great. but that sugarcrm project has no users at all.. looks like a strong candidate to delete [21:52:37] harej: should i make a ticket for that ^ ? [21:52:48] deleting project that looks unused [21:53:15] mutante: I'd say that the sugarcrm one is a leaked resource for a deleted project actually [21:53:21] https://tools.wmflabs.org/openstack-browser/project/sugarcrm [21:53:34] ooh, heh [21:53:39] yeah in my last round of nags i don't recall no sugarcrm project [21:53:47] well https://tools.wmflabs.org/openstack-browser/puppetclass/role::simplelap [21:53:52] this is where it is listed [21:53:56] so probably just a dangling config in the databsase for the puppetmaster [21:54:02] if this means the class is used by fewer than it seems.. good :) [21:54:40] on the other hand.. if i cant trust openstack-browser to know where it is actually used that would be :( [21:55:09] its the best/worst/only source of 'truth' that we have [21:55:19] yep, ok [21:56:07] but we do know that the puppet settings for instances sometimes leak on instance deletion [21:56:24] andrewbogott cleans them up every once in a while [21:56:41] gotcha.alright. i just striked it off the list [21:56:52] eh.. "stroke" ? [21:57:11] struck is the past tense I think... [21:57:25] english is crazy [21:57:58] ah :) [22:07:14] " [22:07:14] account-creation-assistance [22:07:22] this one has many contacts [22:15:25] andrewbogott: I'm moving Rillke's bot (https://commons.wikimedia.org/wiki/User:CommonsMaintenanceBot) to the new stretch grid, but apparently the botpassword it uses is allowed only in the eqiad region, not eqiad1, and it can't login. any way to workaround this besides waiting likely a few months for Rillke to 'periodically reappear'? [22:15:42] zhuyifei1999_: oh yuck [22:15:57] I really dislike the ip restrictions on botpasswords [22:16:01] +1 [22:16:28] I wonder if there is any way for stewards to update those? [22:16:59] I'm going to guess not actually. Would probably have to be done directly in the database [22:18:37] ^ [22:19:21] Should be easy to fix as it's just json [22:19:31] this is actually something we should add to the FAQ for the migration [22:19:42] I bet it will bite others [22:20:56] select * from oauth_registered_consumer where oarc_restrictions LIKE '%"IPAddresses":["10.%'; [22:21:18] {"IPAddresses":["10.0.0.0/8"]} [22:21:24] {"IPAddresses":["10.68.0.0/16","10.196.0.0/16","2620:0:860::/46"]} [22:21:32] {"IPAddresses":["10.0.0.0/0"]} [22:21:36] o_0 [22:21:52] zhuyifei1999_: so you are a co-maintainer of that tool right? I think I can hack the db if you open a phab task requesting the change and explaining why [22:21:56] Probably some more with regexes [22:22:10] yeah he added me as co-maintainer a few months ago [22:22:21] oarc_restrictions LIKE '%"10.%'; [22:22:24] That gives 15 rows [22:22:28] it's not logged or anything though [22:23:38] we implicitly trust the LDAP group membership [22:24:22] I guess I'll wait a few days for rillke to (hopefully) reappear before doing that phab request [22:24:28] It does look like most are {"IPAddresses":["0.0.0.0/0","::/0"]} [22:24:36] hacking db is evil :P [22:25:00] It's not hacking, I have legit access [22:25:01] ;) [22:25:16] legit hacking :P [22:25:39] foreachwiki fixAllTheThings.php --now --do-it [22:26:21] Looks like 15 oauth grants that have 10.% in the restrictions [22:26:29] Some with some of the public ip ranges too [22:29:41] I notified here https://commons.wikimedia.org/wiki/User_talk:Rillke/Discuss/2016#CommonsMaintenanceBot_move_to_new_grid_on_Debian_Stretch [22:44:54] !help - Why did djvm vanish as an external tool on the tools server? [22:44:54] ShakespeareFan00: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [22:49:31] ShakespeareFan00: it looks like that binary comes from the djvulibre-bin package on the old Ubuntu hosts. Let me check in Puppet to see if there is a note about that package on the new Debian hosts [22:50:06] !log wikistats - shutting down and deleted instance T210008 once made for testing krypton upgrade but broken somehow [22:50:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [22:50:09] T210008: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 [22:51:52] ShakespeareFan00: /usr/bin/djvm exists on the Debian Stretch bastion. [22:52:18] !log wikistats - creating throw away instance to test if T128642 is still true or not [22:52:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [22:52:20] T128642: role::simplelamp fails to start mysql due to apparmor - https://phabricator.wikimedia.org/T128642 [22:52:33] ShakespeareFan00: Where did you expect to find it also installed? [22:55:19] ShakespeareFan00: it is also installed on the exec grid and web grid [22:55:41] It is almost certainly not installed in the Kubernetes Docker containers [22:57:47] bd808- https://tools.wmflabs.org/ia-upload/log/peerageofbritish00westrich [22:57:56] That said it couldn't find it [23:02:58] ShakespeareFan00: that whole tool seems to still be running on the old job grid. No idea what if anything would have changed there [23:07:18] Okay [23:07:27] So no immediate answers then [23:07:29] :( [23:08:14] bd808: can you do a 2fa reset for wikitech for me? https://phabricator.wikimedia.org/T215676 [23:08:21] or point me to someone who can? [23:09:04] hexmode: yeah, I can check and do it. wrapping up a meeting, it may take me a few minutes [23:09:14] :) tyvm! [23:13:29] hexmode: {{done}} [23:14:05] saweet! [23:14:10] works [23:33:53] zhuyifei1999_: I added https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation#BotPassword_or_OAuth_grant_does_not_work_from_new_job_grid [23:34:21] ok [23:35:06] thanks [23:44:25] so i think we can estables that role(simplelamp) cant be used for anything new and projects that use it are lucky that it once worked and are doomed to never upgrade their OS or replace the instance .. or stop using it [23:44:33] establish [23:45:01] can i still add new paramaters to a puppet class in horizon? [23:50:08] Detail: undefined local variable or method `config_file' .. i need to set a local variable and add it as a parameter to the existing class [23:50:43] in wikitech UI you could add classes and parameters by name..afair [23:51:01] would something like mysql::config::config_file: 'test' work? [23:51:12] in Hiera you mean? [23:51:21] but the class isnt even looking in Hiera i think .. :/ [23:51:47] paladox: i have "mysql::params::config_file: test.cfg" in Hiera already [23:51:55] oh [23:53:08] look if you select the class role(simplelamp) now on a random instance and you click Apply .. you get 2 parameter form fields you can fill out before you hit Apply again [23:53:15] that kind of parameter [23:53:40] back in the old days there was no hiera and only those and you could click to add parameters.. right [23:54:06] it has "mysql_datadir" and "mysql_local" but it does not have the config_file thing [23:54:21] would need "add parameter" button next to it [23:54:42] or convert this to look in Hiera hrmm