[00:15:31] Hi I have this problem: $ ssh -I ~/tr8dr.labs tr8dr@login.tools.wmflabs.org dlopen /Users/james/tr8dr.labs failed: dlopen(/Users/james/tr8dr.labs, 0x0002): file too short /Users/james/tr8dr.labs tr8dr@login.tools.wmflabs.org: Permission denied (publickey,hostbased). [00:16:09] do I need to regenerate my ssh private key? [00:16:57] also, while I am here, do you support sqlite4? https://lsm-db.readthedocs.io/en/latest/quickstart.html [00:17:49] hello? [00:20:16] or https://github.com/RaRe-Technologies/sqlitedict ? [00:20:35] i think you want ssh -i (lowercase) [00:20:40] -I expects a shared library [00:20:49] which is why you're getting the strange dlopen error [00:21:12] oh thank you [00:21:55] ha yeah that works. Is sqlite3 or 4 standard on the labs toolforge cloud? [00:22:29] as far as i know, sqlite4 doesn't really exist. so my guess is 3. but i'm not sure [00:23:00] https://sqlite.org/src4/artifact/56683d66cbd41c2e [00:23:10] yeah, I think 4 got folded back into 3, so I can use https://github.com/RaRe-Technologies/sqlitedict Thank you so much for your kind and generous help and hospitality, and happy 18th birthday to Wikipedia! [00:23:20] no prob [00:23:41] later! [00:45:26] I will let you know when I see tr8dr and I will deliver that message to them [00:45:26] @notify tr8dr Using sqlite on Toolforge will probably be slower than using ToolsDB (hosted mariadb) -- https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases [11:58:10] !log openstack shutdown vmbuilder-trusty VM instance (ubuntu deprecation) [11:58:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [14:10:18] !log wikidata-dev wikidata-shex changing from github to gerrit [14:10:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [14:25:56] !log wikidata-dev wikidata-shex enable automatic git updates via systemd timer [14:25:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [14:41:37] !log admin T214369 deployed new jessie and stretch VM images [14:41:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:41:40] T214369: Update OpenStack images for jessie/stretch for CVE-2019-3462 - https://phabricator.wikimedia.org/T214369 [14:45:25] hi, does anyone know how to run MediaWiki-Vagrant on Toolforge? [14:46:02] I don't know if you can run that inside tools [14:46:21] can vagrant work inside a container? [14:49:00] Krenair: i'm not sure, i'm new to toolforge [14:49:07] afaik vagrant needs root [14:49:32] oh yeah you probably can't do that in tools [14:51:29] damn, that sucks... :( [14:52:11] do you guys just do your dev locally on vagrant or create a mediawiki instance every time in toolforge [14:52:32] or maybe you guys have ur own cloud vps instance [14:54:59] "This package requires php >=5.6.99 but your PHP version (5.5.9) does not satisfy that requirement." epic [14:56:09] I don't think people do run MediaWiki in tools? [14:56:25] I've always just used ordinary instances outside the tools project [14:57:07] ah i see [14:58:14] yes, people might run beta/test instances (which I haven't seen) but I don't think we allow production wikis on Toolforge [14:58:35] https://wikitech.wikimedia.org/wiki/Help:Toolforge#Installing_MediaWiki_core [14:59:41] gtirloni: i tried that, but mediawiki/core required deps that are uptodate [14:59:46] composer complains [14:59:58] daegontaven: if you need newer PHP (e.g. 7.2), it's available in the new grid.. let me try to find some docs [15:00:07] i'm not sure if we made the new grid public yet [15:00:35] there's a new grid? whoa O_o [15:00:56] in any case, I think the rationale for only allowing beta/test mediawiki installs in Toolforge is because Toolforge is a shared platform and a long-running wiki without proper care can invite a lot of spammers which could impact other users of Toolforge [15:01:29] yeah my usecase is a beta/test mediawiki install [15:01:52] https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation [15:03:07] * daegontaven clicks link like a maniac [15:03:08] any feedback is welcome. like I said, I'm not sure this public yet so it might be a bumpy ride (thanks for helping us test it!) :) [15:04:05] np, this is cool [15:04:08] :) [15:04:54] "2019-01-11: Availability of Debian Stretch grid announced to community" [15:05:05] I guess it's public already, I just can't find the email about it.. but I remember reading it [15:05:17] it was definitely sent, I read it too :) [15:05:49] gtirloni: https://lists.wikimedia.org/pipermail/cloud-announce/2019-January/000122.html [15:06:06] ah there it's, thanks! [15:21:44] daegontaven: hey [15:21:49] oyye :) [15:22:17] just a comment on your reply in discourse just now [15:22:17] oh wait...i remember you from discourse!! [15:22:23] lol jinx [15:22:31] * chicocvenancio laughs [15:22:51] Is it possible to identify bot owner by IP? [15:22:58] daegontaven: there is no way to run vagrant on toolforge [15:23:08] revi: possibly? [15:23:10] bot runnign without logging in for few days on WD, https://www.wikidata.org/w/index.php?diff=839995450&oldid=839994385&rcid=876217328 [15:23:13] running* [15:23:47] daegontaven: but we can run mediawiki on toolforge, it is just slow (unless things changed a lot) [15:24:12] chicocvenancio: yeah, it's too bad ..i decided to test out manually first and write a script in python to automate db creation for me [15:24:21] vargant would have been nice tho [15:24:25] *vagrant [15:24:37] there is a mariadb instance for use as well [15:24:40] regarding revi’s question: what I did was ssh into Toolforge, run `host $IP` to do an rDNS lookup, SSH into the worker host it returned, and listed processes there [15:24:47] would be curious if anyone has a better suggestion :) [15:24:57] presumably there are several containers running on each of those IPs at any given time revi [15:25:00] I wouldn't count on it [15:25:07] daegontaven: you could create and drop databases there (from toolforge or vps) [15:25:46] but I'm not sure about the xdebug thing, and how much toolforge nfs slowness will affect your use case [15:25:57] chicocvenancio: well that's a pain to have to manually type out sql statments into the mariadb console [15:26:21] I imagine the script could be adapted [15:26:52] I will echo my first comment, toolforge is a lot easier to start and has quicker aproval [15:27:00] I'd recommend you request it and try [15:27:27] if you still need things that are not available there, then apply for VPS and see [15:27:38] * chicocvenancio is going to lunch now [15:27:45] yeah i actually closed the cloud vps thing myself, gonna test on toolforge for now and if it's too hard then gonna request cloud vps [15:27:57] daegontaven: cool [15:28:03] ^_^ [15:29:50] Krenair: in this case there seem to be only two processes running as any tools.* user, both with the same user [15:30:05] the rest is root, sgeadmin, nagios and stuff like that [15:30:06] well that simplifies matters [15:30:17] though I think this is the new grid and probably isn't up to normal load yet [15:30:22] true [15:30:34] on the old grid we may not be so lucky [15:30:39] or the new grid in future [15:31:19] alex@alex-laptop:~$ dig -x 172.16.1.252 @labs-ns0.wikimedia.org +short [15:31:19] tools-sgeexec-0930.tools.eqiad.wmflabs. [15:31:19] hm [15:31:45] sgeexec hosts are the new grid right gtirloni ? [15:31:57] “Son of Grid Engine” I’d assume [15:32:57] * chicocvenancio 's lunch was delayed [15:33:02] pretty sure that's the new thing [15:33:12] sgeexec is the new grid [15:33:38] Krenair: right [15:35:48] tools.yifeibot 13964 11.5 7.0 715748 576568 RNl Jan13 1-00:49:44 \_ /usr/bin/python2.7 /shared/pywikipedia/core/pwb.py /data/project/yifeibot/pywikibot-shared/addbot-worker.py [15:37:01] revi and Lucas_WMDE: right now, you could do: [15:37:16] https://www.irccloud.com/pastebin/pNZ9iXWS/ [15:37:35] ah [15:37:37] neat [15:37:39] That doesn't guarantee that's what was running at the time [15:37:42] zhuyifei1999_, Steinsplitter [15:38:04] Yup ^^ [15:38:04] * Lucas_WMDE adds that command to his bash history [15:38:55] O_O [15:39:00] though it abbreviates the username :/ [15:39:06] It may or may not be that tool that was what you were looking for because the original one could have stopped. However, that could be it. It was running at the time on there [15:39:35] tools.yifeibot is the full name. Abbreviation is an issue with "q" commands. Often you can tell it to output xml with the full names. [15:39:51] https://www.irccloud.com/pastebin/t4gb1pp9/ [15:40:11] Those anon edits from 172.16.0.0/12 are due to a typo in a config change I made 3 months ago :/ Posted a patch to fix it [15:40:13] Like that ^^ [15:40:37] bstorm_: thanks [15:45:52] anything wrong with my bot? [15:45:55] also thanks to bd808 of course :) [15:47:31] zhuyifei1999_: apparently bd808 fixed it, though I don’t know where the patch is [15:47:53] ah, presumably https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/485843 [15:48:08] well, okay, not exactly [15:48:14] iiuc zhuyifei1999_ that change would fix your bot [15:48:20] Yeah, no changes to anyone's bot. Just a fix for the bad config I wrote :/ [15:48:22] would *block your bot [15:48:24] sorry [15:48:37] but zhuyifei1999_ I think you still need to fix your bot so it doesn’t try to edit anonymously? [15:49:43] assuming it is zhuyifei1999_'s bot [15:50:10] my bot only reads from wd, no writes [15:50:20] hm [15:50:27] okay… [15:50:31] hrm. Yeah. It could have been something else that was running on the host at the time [15:50:51] Depending on the timestamps. [15:51:14] The bot that is there now was running there then...but something could have stopped running there? [15:51:39] bstorm_: really probable actually. A lot of "bots" are cron tasks [15:52:07] oh, I only just noticed the last edits were last night [15:52:14] I thought it had edited right until it was blocked just now [15:52:25] yeah, in that case it might well be something else :/ [15:52:27] (sorry, I'm at work rn so responding kinda slow) [15:53:02] It would be something on this list -- https://tools.wmflabs.org/sge-jobs/ [15:54:14] getting the bug in the prod config fixed will fix the problem from the point of view of the wikis. Then hopefully the bot author will find out that they have some mistake in their auth code [15:57:11] revi: could you create a phabricator task to identify that tool? [16:02:00] nvm [16:02:11] * chicocvenancio read scrollback [16:03:30] sorry, was on a meeting [16:03:42] so you wanna fill a task or not? [16:03:58] the cron entries for Jan 21 19:40 are: [16:04:06] I think the mistery's solved [16:04:11] https://www.irccloud.com/pastebin/btOfLGc1/ [16:04:37] you're awesome chicocvenancio [16:10:34] * chicocvenancio didn't solve it [16:10:44] none of these looks like what I'd expect [16:11:14] As I understand it, bd808 has figured out the config error that is allowing anon edits [16:11:42] 19:41 are: [16:11:50] https://www.irccloud.com/pastebin/Spw4AvWb/ [16:12:37] And has proposed a fix to the config that would block it [16:12:54] yep I think it's dexbot [16:13:10] 'summary': u"Bot: Adding label and sitelink from [[w:"+lang+u":"+page.title()+u"]]", [16:14:06] revi, Lucas_WMDE: [16:14:26] then Amir1 [16:14:48] what's wrong [16:14:51] Dexbot is mine [16:15:10] your bot seemed to be editing while logged out [16:15:24] shoot [16:15:37] .... [16:15:44] lol [16:15:51] already fired a banhammer on the IP [16:17:15] ~tools.dexbot/new_iw14.err has a long history of being blocked 'by MediaWiki default' [16:17:23] It's so fun that the bot just randomly logs out [16:22:45] I disabled all codes running compat [16:23:01] I really need to spend a weekend and clean up all of my crontab [16:23:17] Amir1: the section is https://www.wikidata.org/wiki/Wikidata:Administrators%27_noticeboard#Special:Contributions/172.16.1.252 fwiw [16:24:05] Thanks [16:25:36] Amir1, maybe considering using the assertion feature [16:26:20] yeah, it's handled in pywikibot itself but this code is so old that doesn't have anything (even runs pywikibot compat) [16:26:46] ah [17:06:39] thanks all for figuring it out [17:06:43] I was away, sorry [17:06:58] (and my client doesn’t support the /away IRC feature, though I have no idea if anyone checks that anyways ^^ ) [17:32:13] !log starting roll out of apt package updates to all projects' [17:32:13] jbond42: Unknown project "starting" [17:34:37] jbond42: do "!log admin ...." [17:35:12] bd808: thanks was just investigating that [17:35:35] !log admin starting roll out of apt package updates to [17:35:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:35:41] "admin" is a special project that we tend to use for global operations here [17:35:57] good to know thanks [17:36:58] yeah it has no instances of its own AFAIK [18:37:08] !log admin T214299 reimaging cloudnet1003 as debian stretch [18:37:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:37:11] T214299: cloudvps: neutron: upgrade jessie -> stretch - https://phabricator.wikimedia.org/T214299 [18:40:31] !log admin T214299 manually delete from neutron agents from cloudnet1003 (must be added again after reimage, with new uuids) [18:40:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:57:31] !log tools Changed deb-tools.wmflabs.org proxy to point to tools-sge-services-03.tools.eqiad.wmflabs [18:57:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:13:24] * arturo adds a mental note: document that in wikitech ^^^ [19:19:47] !log admin T214299 stretch cloudnet1003 is apparently all set [19:19:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [19:19:50] T214299: cloudvps: neutron: upgrade jessie -> stretch - https://phabricator.wikimedia.org/T214299 [19:35:06] uhm... "Failed to apply catalog: Cannot allocate memory - /usr/bin/dpkg-query -W" on my cloud VPS.. hmm [19:35:50] running dpkg-query -W manually is no problem [19:36:00] but somehow it fails when puppet tries it [19:41:08] what is the proper way to install ruby gems in toolforge k8s? [19:43:56] That's a good question. I'm not sure it's come up that much. I wonder what the Ruby equivalent of virtualenv is. [19:44:13] (random thought: ruby is a gem, python is a snake, why can't we install snakes) [19:44:18] rvm I guess [19:44:30] harej: ^ [20:00:56] chicocvenancio: probably bundler, but I'm not actually sure that we have any good docs or practices for ruby [20:03:35] bd808: after searching I'm concluding we don't [20:04:21] hey, do you remember using "ifttt" (if this then that) [20:04:25] ifttt: Set up Wikimedia IFTTT channel service using puppet on labs [20:04:51] i see there are 2 roles, one "staging" and one "production" in puppet repo [20:04:59] chicocvenancio: there is a task... https://phabricator.wikimedia.org/T141388 and also https://phabricator.wikimedia.org/T127685 [20:04:59] but i dont see it on https://tools.wmflabs.org/openstack-browser/puppetclass/ [20:05:24] does that mean it's not used anymore or that i dont see where it is used [20:05:56] git log tells me Madhu wrote it [20:06:08] bd808: makes sense, I peeked into some ruby tools to get a sense [20:07:21] mutante: I'm not sure where the ifttt stuff is running these days. That's one of slaport's projects [20:09:25] bd808: aha!, good to know he was involved. thanks [20:16:42] i wonder if "not listed in openstack-browser" is enough to say it's not [20:18:55] mutante: I wouldn't consider that a guarantee [20:20:58] hmm, ok. .. i wish there was a way to find out.. happens a lot to me that trying to determine that is more work than an actual code change [20:21:24] mailing legal is probably not worth it for a small puppet code improvement [20:21:34] !log tools published new docker images (all) [20:21:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:36:50] mutante, re: ifttt: it's definitely working /somehow/, because I'm still getting weekly emails from https://ifttt.com/applets/306827p-automatically-get-a-weekly-vocabulary-list-of-wikipedia-words-of-the-day-in-your-email which I signed up for years ago! (i often have to repress the "it's Wiktionary!" twitch...) Beyond that idk. I suggest emailing slaporte directly perhaps at his volunteer address :) [23:38:28] quiddity: ah! maybe that could just be ifttt.com itself without needing puppet, but maybe not :) it's ok, i will not change anything there [23:48:33] Hello! starting sometime in the past few days, I sometimes get "1205 Lock wait timeout exceeded; try restarting transaction" when writing to my user database. I see with `SHOW FULL PROCESSLIST` there are queries stuck in the "updating" or "query end" states. They are very simple INSERTs and UPDATEs that are normally very fast (0.01 sec) [23:48:46] Could this be an issue with tools.db.svc.eqiad.wmflabs? I'm not sure how to debug further, as the issue is intermittent, and the code that writes to my user database hasn't changed in a long time [23:54:19] cloudcontrol1003 - Toolforge instance distribution - CRITICAL: worker class instances not spread out enough [23:59:22] mutante: our new check works :) We need to move a couple of vms to make it happy. I'll open a ticket.