[03:45:37] is Daminz there? [03:45:49] * Jasper_Deng is wondering whether we have a global title blacklist for the beta wikis [03:45:54] Damianz* [05:49:22] Good morning petan, Good morning Coren [06:08:28] well my continuous job died again [06:39:41] [bz] (8NEW - created by: 2Tilman Bayer, priority: 4Unprioritized - 6normal) [Bug 50556] Total number of Wikipedia articles is ca. 11 million too high - https://bugzilla.wikimedia.org/show_bug.cgi?id=50556 [08:10:25] AWAY [08:11:20] ? [08:18:09] @notify petan [08:18:09] I'll let you know when I see petan around here [08:28:58] @seen petan [08:28:59] zhuyifei1999: Last time I saw petan they were quitting the network with reason: Ping timeout: 246 seconds N/A at 7/2/2013 3:44:54 AM (04:44:04.9883400 ago) [08:50:25] @seen petan [08:50:25] zhuyifei1999: Last time I saw petan they were quitting the network with reason: Ping timeout: 246 seconds N/A at 7/2/2013 3:44:54 AM (05:05:31.0559730 ago) [09:39:00] petan: you're finally here [09:44:35] petan: Can wm-bot (another instance) run on linux? [11:22:19] is everything back up? [11:30:10] [bz] (8NEW - created by: 2se4598, priority: 4High - 6normal) [Bug 50498] "Error opening index" for File: search on Labs - https://bugzilla.wikimedia.org/show_bug.cgi?id=50498 [11:44:01] Beetstra hi [11:44:09] Beetstra: I installed some of packages you needed [11:44:57] Petan, let me try [11:46:03] Some = which? [11:46:21] BSD::Resource seems still to be missing [11:46:42] !sal [11:46:42] https://wikitech.wikimedia.org/wiki/Labs_Server_Admin_Log [11:47:42] * Beetstra lookifies log [11:49:00] ah .. libdata-dumper-simple-perl libhtml-html5-entities-perl libirc-utils-perl libtask-weaken-perl libobject-pluggable-perl libpoe-component-syndicator-perl libpoe-filter-ircd-perl libsocket-getaddrinfo-perl libpoe-component-irc-perl libxml-simple-perl [11:50:51] petan: could you add libbsd-resource-perl [11:52:41] sure [11:54:07] So, why has my script stopped working since the kasplasm? :( [11:56:20] done [11:57:34] !log bots deleting exec node #@ [11:57:36] Logged the message, Master [11:57:37] !log bots deleting exec node #2 [11:57:38] Logged the message, Master [11:57:40] meh [12:09:29] petan, I need package libwww-mechanize-perl as well [12:10:48] installed [12:12:43] Hmm .. the bot still complains that WWW::Mechanize is not there .. [12:13:19] really? :o [12:13:23] try again... where you run it? [12:13:29] ah .. wait [12:13:37] maybe a bit too fast? [12:14:08] while you're on it, please install MediaWiki::API HTML::TreeBuilder URI::Escape Bot::BasicBot::Pluggable::Module DateTime::Format::Strptime DateTime::Format::Duration DateTime::Format::Human::Duration [12:14:19] if that's at tools box that is [12:15:37] gry, can you find which packages are these in? [12:15:45] I really know little perl [12:15:55] Beetstra does it work already? [12:16:06] it seems to have run .. but crashed (?) [12:16:21] hmm... reason? maybe memory limit? [12:16:27] how much memory did you give it? [12:16:50] no .. the xlinkbot.err does not seem to have finished [12:17:01] but I can find it running on tools-exec-01 [12:17:09] interesting [12:18:00] packages -- libmediawiki-api-perl libdatetime-format-strptime-perl libbot-basicbot-perl libdatetime-format-duration-perl -- and those in cpan only: DateTime::Format::Human::Duration URI::Escape Bot::BasicBot::Pluggable::Module [12:18:14] thanks [12:19:20] !log tools petrb: installing packages -- libmediawiki-api-perl libdatetime-format-strptime-perl libbot-basicbot-perl libdatetime-format-duration-perl [12:19:22] Logged the message, Master [12:19:40] [bz] (8UNCONFIRMED - created by: 2se4598, priority: 4Unprioritized - 6normal) [Bug 50499] Missing thumbs: Error at scaling server: couldn't connect to host - https://bugzilla.wikimedia.org/show_bug.cgi?id=50499 [12:20:01] gry: I will need to debianize these from cpan... [12:21:14] it's undesirable to use cpan and you will only install them using a .deb? :) [12:22:09] gry, I tried to install stuff on my account using cpan .. some plainly fail, some work half, others go perfectly .. I don't know if it is different as root, but it seemed funny [12:22:28] gry: yes it is very undesirable to use cpan [12:24:19] petan .. xlinkbot.err in my dir does not say that the job finished (did not print my 'Bot crashed!'-line) .. but I can't find xlinkbot, nor is it on IRC .. [12:25:01] Beetstra it may happened that it died on OOM [12:25:09] Beetstra how much memory did you give it? [12:25:18] did not set anything for memory [12:25:25] you really should start on high value, like 1gb and then check how much it needs [12:25:38] what param do I need for that? [12:25:39] if you didn't set any memory, then it gave it only 256mb, which is not enough for perl [12:25:44] !toolsdoc [12:25:52] !toolsdoc alias toolsdocs [12:25:54] Unable to create alias for toolsdocs because the target is alias, but not a key, if you really want to create this broken alias do !toolsdoc force-alias toolsdocs [12:26:00] !toolsdoc alias tooldocs [12:26:01] Created new alias for this key [12:26:04] !toolsdocs [12:26:05] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [12:26:28] -mem value [12:26:46] jsub -mem 1g [12:32:04] xlinkbot is running [12:38:51] cool :) [12:40:44] Hi all [12:41:06] ohai magnus__ :) [12:41:08] I'm trying to start a php script via jsub, but I get "libgcc_s.so.1 must be installed for pthread_cancel to work" [12:41:25] 32/64 lib issue? [12:41:55] probably [12:42:01] we don't have ia32-libs installed? [12:42:03] petan: ^ [12:42:34] but it runs directly from command line [12:44:51] magnus__: possible that it is installed on -login but not on exec nodes? [12:46:57] So I can't run php in jsub? That would be ... odd. [12:48:27] i know that other php scripts run without issue... [12:48:35] petan: can you check if ia32-libs is installed on exec nodes? [12:49:53] back [12:50:01] YuviPanda sure [12:52:32] no they are not [12:52:55] can you install? [12:53:00] sure [12:54:20] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Normal - 6minor) [Bug 45868] [OPS] [worked around] let search indexer access the squid public ip - https://bugzilla.wikimedia.org/show_bug.cgi?id=45868 [12:57:20] !log deployment-prep Updating iptables rule that workaround NAT issue in beta. Applied on deployment-searchidx01 and deployment-upload : iptables -t nat -I OUTPUT --dest 208.80.153.219 -j DNAT --to-dest 10.4.1.133 See also {{bug|45868}} [12:57:22] Logged the message, Master [12:58:02] !log deployment-prep installing iptables on deployment-upload [12:58:04] Logged the message, Master [13:00:23] petan: installed? [13:00:31] also -login is 32 bit but exec nodes are 64? [13:01:58] not yet [13:02:00] it's lot of packages [13:02:13] petan: Or is there another way to run php in jsub? [13:02:50] magnus__ idk how php-cli is configured on exec nodes, I could try to run it. TBH I didn't even know that php-cli requires ia32 [13:03:23] liangent runs php stuff on exec nodes, right? [13:04:46] magnus__: give me a second :) [13:05:33] I use php and jsub [13:05:44] https://github.com/addshore/addwiki/blob/master/cron/tools-login-addbot [13:06:52] magnus__ = Magnus Manske ? [13:07:03] yup [13:07:18] magnus__: Darf ich dir nen Bug melden? [13:07:27] immer drauf :-) [13:07:52] magnus__: also, any ideas for using Redis? :) [13:08:20] YuviPanda: Was thinking about a new version of my tool pipeline thingy [13:08:35] YuviPanda: Using Redia as temporary storage for page lists [13:08:43] magnus__: http://tools.wmflabs.org/commonshelper/ Weist du dass, der commonshelper auf tools defekt ist oderhast du den noch nicht vollstendig germerged? [13:08:46] :-) [13:08:47] YuviPanda: s/Redia/Redis/ [13:08:58] magnus__: so I already have two bots running that use that kindofa pattern [13:09:20] https://github.com/yuvipanda/SuchABot/blob/master/suchabot/stream-receiver.py reads gerrit events, puts them on a redis queue [13:09:43] https://github.com/yuvipanda/SuchABot/blob/master/suchabot/github-sync.py reads from that redis queue, publishes them to github [13:09:58] Steinsplitter: A bisserl präzieser wär schon nicht schlecht... [13:09:59] !log deployment-prep pointed deployment-upload thumb handler to the varnish cache text instead of squid. Done by editing /data/project/upload7/scripts/thumb-handler.php [13:10:01] this way, if any other tool wants to reliably just read events from Gerrit, it can just register with my tool and easily receive them [13:10:02] Logged the message, Master [13:10:12] !log deployment-prep removed iptables 'nat' rule from deployment-upload [13:10:14] Logged the message, Master [13:10:20] this 'registering' can be done with a simple secret-key mechanism, implemented at https://github.com/yuvipanda/SuchABot/blob/master/suchabot/register-stream-client.py [13:10:46] I'd want to implement IRC RC Feed this way at some point, so writing new tools based on that will become very simple [13:11:29] magnus__: Die sprachen ins leere, z.B. wenn du Deutsch anklickst ;-) [13:11:59] The requested URL /commonshelper/commonshelper.php was not found on this server. [13:12:51] Steinsplitter: Repariert, danke. [13:13:27] petan: tools-db is still a virtualised server right? [13:13:37] yes [13:14:08] magnus__: ging ganz schnell. Danke [13:14:09] okay, I am looking to create a new db for my bot (meaning I can get of bots-bsql01 :)), I guess I will stick it on the repdb :) [13:23:41] addshore: I implemented external commands in pidgeon, now I can /grep blah :D [13:29:35] BTW, isn't "libgcc_s.so.1 must be installed for pthread_cancel to work" the error message when PHP runs out of memory? [13:31:19] scfc_de: Script barely uses any memory [13:31:37] magnus__: Do you have the job ID? [13:32:57] scfc_de: we should have those put up somewhere :) [13:32:58] well, it died so quickly... [13:33:16] petan: Now I get a new error : PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php5/20090626/mysql.so' - /usr/lib/php5/20090626/mysql.so: cannot open shared object file: No such file or directory in Unknown on line 0 [13:33:17] magnus__: try starting it with 1G of RAM? [13:33:28] YuviPanda: Just tried 4GB [13:33:32] 20090626?! [13:33:39] as in 'php version from four years ago'? [13:33:52] magnus__: I am still installing ia32 [13:33:55] okay, so definitely it was a RAM die, not a 32bit issue, I guess [13:33:57] it's really big [13:34:01] or is that just ubuntu being weird [13:36:04] petan: Yay, now it runs! Thanks!! [13:36:10] petan: Can wm-bot (another instance) run on Linux? [13:36:23] magnus__: The last job I see in accounting is 496503, that was submitted and started Thu Jun 27 and ended Mon Jul 1 03:35:41. [13:37:46] petan: we should put ia32-libs on puppet, then? [13:38:31] !log deployment-prep restarted mw-cgroup upstart service on apaches box. That recreated the wgCgroup directory /sys/fs/cgroup/memory/mediawiki [13:38:34] Logged the message, Master [13:39:52] So the problem was that php5-cli depends on ia32-libs, but doesn't declare that dependency? [13:41:33] petan, YuviPanda: ^ [13:41:58] scfc_de: so ia32-libs is now just a sortof metapackage that installs 32-bit versions of 'some popular libraries' [13:42:27] it's only needed if there exist some libraries that need 32 bit libs as dependencies to work [13:43:03] ... php5-cli doesn't (or at least /shouldn't/ require ia32-libs) [13:43:55] So what does? [13:45:09] some other library that magnus__ was using, I suppose? [13:45:13] We have, hypotheically, pure 64-bit installs so nothing should. ia32-libs is useful only for stuff that was compiled on a 32-bit box and is only distributed in binary form. [13:45:16] the specific error was pthreads... [13:45:27] magnus__: were you using any extra libraries? [13:45:47] Coren: are -login and -dev 64bit too? [13:46:00] YuviPanda: Yep. [13:46:14] ah, okay. thought they'd be 32, since they have less RMA [13:46:15] RAM [13:47:14] In retrospect, I might have been a bit cheap on the ram side for the login boxen. [13:47:28] zeljkof: thumbnails were broken on beta since yesterday [13:47:32] Coren: true, not sure why we aren't doing min 4GB :D [13:47:38] zeljkof: had to restart some security service on the apaches ( mw-cgroups ) [13:47:39] any particular reason they can't be upped now? [13:47:42] But then again, once I replace cron, that's not supposed to be an issue anymore. [13:48:05] hashar: thanks for letting me know [13:48:05] YuviPanda: Instance resizing is flaky and unreliable on OpenStack still, afaik. We'd need to check with Ryan. [13:48:50] Setting up new instances on the other hand is quick :-). [13:48:56] magnus__: You're still around? [13:49:20] scfc_de: Yes, but disruptive when you plan on replacing -login and friends. [13:49:48] scfc_de: Might be worthwhile to consider it come next planned outage. [13:50:03] scfc_de: But, ideally, I want my cron-replacement to be ready by then first. [13:50:04] Just a friendly push to -dev (and -dev-002 and ...) ... [13:50:09] YuviPanda: No extra library (except mysqli, which should come with php these days...) [13:50:25] scfc_de: yup [13:50:28] hmm, perhaps we've a polluted 32bit mysqli somewhhere? [13:50:59] Coren: (BTW, need to make sure that that doesn't break with Puppet's cron jobs for system users.) [13:51:30] scfc_de: It won't be named 'cron / crontab' to avoid confusion. [13:51:45] Aren't the MySQL (or MariaDB) packages homebrewn? [13:52:16] scfc_de: Not homebrews, the mariadb packages come from the mariadb official repo. All 64 bits though. [13:52:39] magnus__: I can't find your issue in scrollback. Care to give me the tl;dr summary to catch me up to speed? [13:55:21] meh, github's down [13:55:23] Coren: Tried "jsub php somescript.php", died. But, worked directly from command line. [13:55:32] not anymore [13:55:50] Coren: Original error was "libgcc_s.so.1 must be installed for pthread_cancel to work" [13:56:07] magnus__: That is 100% sure to be an OOM error. [13:56:07] Coren: petan installed 32bit libs, works now. [13:56:14] dafu? [13:56:22] Coren: OOM? [13:56:27] Out of Memory. [13:56:59] * Coren boggles a little. [13:57:27] Coren: How can /data/project/glamtools/viewcache_demon.php run out of 256MB (default jsub mem)? [13:59:07] magnus__: Php is a bit of a glutton on startup, though it remains fairly stable over time (because vmem includes the size of mapped libraries). [14:00:13] Coren: It is, but that'd be a little extreme, and it works now with unchanged settings? [14:00:46] magnus__: That's the part that confuses me. Perhaps php is "smart" enough to fall back to 32 bit libs if they are present -- but that would make the code considerable less efficient. [14:01:27] Coren: Maybe 32bit libs are only half the size? /me ducks [14:01:50] magnus__: Not /half/, but ia32 libs do tend to have a smaller footprint. :-) [14:03:21] Coren: And the data structures they create? [14:03:47] scfc_de: That's what tends to make the footprint smaller, yes. :-) [14:05:10] At any rate, I don't think I like that solution -- having 32 bit libraries stealthily used may well end up screwing with performance of code a lot and that it happens would be completely opaque to anyone. [14:05:26] magnus__: Is your script running in a job now? [14:06:59] (I'd much rather raise the default vmem limit to be big enough to encompass typical use cases) [14:07:09] (php somescript.php -> ur-typical) [14:10:12] Coren: Did get restarts/coredumps as jstart, so did jstart -mem 1g. Seems to run now as 562570. [14:10:55] maxvmem=265.277M [14:11:04] Heh. Just /barely/ above the default limit. :-) [14:13:32] Coren: Ugh. And I used to count CPU cycles and single byte memory waste ;-) [14:13:44] magnus__: Ah. That's saner than I thought. The process isn't actually using any 32-bit libraries, it's the -mem that did it. [14:15:37] magnus__: Do remember that this is the total of /mapped/ memory, which resembles and is linked to your actual memory usage, but includes all the system overhead. [14:16:26] magnus__: Of "real" resident memory, your process uses 12M [14:17:18] vmem includes all the shared libraries (including glibc), the executable, etc. All the stuff that, is almost always shared between processes and doesn't cost actual ram -- but that *could* be the only one used by your process in a worst-case scenario. [14:17:55] So one doesn't have to worry about being wasteful with a -vmem of 300m, say. [14:19:01] Coren: But the grid engine would still think it uses 1G and not start other jobs, right? [14:19:19] Coren: we should maybe change the default limit to 512m, maybe? since python also seems to need slightly over 256... [14:19:36] YuviPanda: Yeah, that's not unreasonable. [14:19:57] does SGE base scheduling on memory limits set by jobs or actual memory used? [14:20:42] magnus__: The way vmem is allocated is designed to take that difference in consideration; the total allocable vmem is equal to the sum of ram and swap so that, in practice, swap is never touched by vmem allocation (but the worse case remains covered) [14:21:41] YuviPanda: It bases it on the request, which is why you want to request enough, but not so much that you'll end up penalized. 512m seems sane as a default since it will cover most simple case without being overly generous. [14:21:59] agreed [14:27:35] Who is the owner of Selenium User? [14:28:00] magnus__ does it work now? [14:28:33] Steinsplitter: chrismcmahon [14:29:07] petan: Would you please answer my question? [14:29:18] thy, YuviPanda [14:33:52] zhuyifei1999_ hi [14:34:00] zhuyifei1999_: yes wm-bot is running on linux [14:34:11] where did you think it run o.O [14:34:14] it's running on labs [14:34:17] But the extension is .exe [14:34:19] we don't have windows here [14:34:41] you can rename it to .bin if you like... it's mono / .net executable (interpreted binary just as .jar) [14:34:48] zhuyifei1999_: http://www.mono-project.com/Main_Page [14:35:13] whelp, http://www.mono-project.com/Special:Version [14:35:20] petan: Yes, thanks. [14:35:25] magnus__: cool o/ [14:35:34] @notify Beetstra [14:35:34] I'll let you know when I see Beetstra around here [14:36:22] Thanks. [14:37:11] And how can I add admins when there's no admin existing? [14:45:01] zhuyifei1999_: you can create a file admins in config folder [14:45:14] zhuyifei1999_ it has syntax of |level [14:45:27] for example: zhyufei.*|admin [14:45:43] that is a list of global admins, like people who have global access to all channels [14:45:57] you should be quite carefull about the regex there [14:48:39] Ok thanks, it's working. [14:50:11] yw [14:51:01] YuviPanda that skin on mono is incredible [14:51:12] yeah, fairly nice [14:51:17] outdated, though. pre-RL [14:51:23] it really looks as a regular website [14:51:36] indeed. would not be recognizable without the url structure [15:00:08] YuviPanda do you have any experience with c# on android? [15:00:16] xamarin has support for it [15:00:18] mono for android is not free [15:00:21] so, haven't [15:00:23] wish it were, tho [15:00:33] yes, I saw... there are some restrictions [15:00:43] but I think only regarding some p/invoke calls [15:00:46] it's useless for all practical purposes without buying a license. [15:00:49] err, no. [15:00:52] ah... [15:01:01] how does it work? [15:01:20] is there some backend running on linux level, or there is interpretor running in java? [15:01:24] petan: https://store.xamarin.com/ [15:01:37] petan: no, they have it running on linux level, calling into java with IKVM whenever needed [15:01:50] ah [15:01:57] that's cool it is possible on android [15:02:04] their 'indie' is $300 per developer [15:02:07] without having to root your phone :P [15:02:14] yes that price suck... [15:02:32] yeah [15:02:40] true true, you can write for android in a variety of languages [15:02:43] python, go, whetver. [15:02:53] of course, if you want it to look 'native' you have to use Java [15:10:05] petan: why doesn't @logon work? [15:10:19] you are probably missing the logging module [15:10:32] http://bots.wmflabs.org/~wm-bot/db/%23wm-bot.htm << have look at bottom [15:10:42] eh [15:10:45] nvm [15:10:50] http://bots.wmflabs.org/~wm-bot/db/systemdata.htm [15:10:54] this one [15:11:13] LOGS (1.8.0) [15:13:44] How shall I add? [15:14:00] [bz] (8NEW - created by: 2Tim Landscheidt, priority: 4Unprioritized - 6normal) [Bug 50585] Silence the qacct transfer jobs and monitor them with Icinga instead - https://bugzilla.wikimedia.org/show_bug.cgi?id=50585 [15:18:10] !log deployment-prep deleting deployment-searchidx02 , not being used [15:18:12] Logged the message, Master [15:22:10] petan: How shall I add the modules? [15:22:42] zhuyifei1999_: you need to compile it, rename it to *.bin and copy to folder modules [15:26:35] petan: where's the modules source? There's only tow [15:26:45] it's in folder plugins [15:26:50] hold on [15:27:17] https://github.com/benapetr/wikimedia-bot/tree/master/plugins/wmib_log [15:29:29] YuviPanda: right [15:29:47] (for YuviPanda: liangent runs php stuff on exec nodes, right?) [15:31:32] Errors at plugin.cs line 28, 35, 40 [15:32:12] The type or namespace 'config' could not be found [15:33:24] [bz] (8ASSIGNED - created by: 2Niklas Laxström, priority: 4High - 6major) [Bug 48203] Purging does not work on deployment-prep / beta labs - https://bugzilla.wikimedia.org/show_bug.cgi?id=48203 [15:36:42] zhuyifei1999_ you need to copy bin/Debug/wmib.exe to bin/Release as well... [15:47:33] petan: but how to solve the error? [15:47:51] I'm not good at C# [15:48:13] zhuyifei1999 the error is because it can't resolve the function [15:49:02] it is linked to binary in Release which is missing now [15:49:56] Ok, I'll try again tomorrow. [15:51:13] zhuyifei1999 it will need that binary there tommorow as well :P [16:06:03] petan: just had an issue where a jsub shell script couldn't find /usr/bin/mysql with full path given. Maybe some execution hosts can't see the parts of the file system? [16:10:12] magnus__: It doesn't seem to be on any exec host. Huh! [16:13:43] /usr/bin/mysql is part of mariadb-client-core, and that is not in *any* toollabs manifest, so it must have been installed manually. I'll submit a patch to install it as part of exec_environ. [16:16:22] scfc_de HUH, afaik there is some mysql client on exec nodes [16:21:20] scfc_de are you adding it to puppet? [16:21:27] use mariadb-client, not mysql [16:21:36] I will install it on exec nodes for now.... [16:22:36] petan: Yes; mariadb-client or mariadb-client-core? [16:23:19] I think mariadb-client sounds like the correct package to install client :> [16:23:30] -core is one of its dependencies... [16:23:38] idk what all is part of whole -client [16:23:51] but there is probably a reason for it to exist [16:24:28] !log tools petrb: installed maria to all nodes so we can connect to db even from sge [16:24:30] Logged the message, Master [16:24:37] It installs several other programs beside /usr/bin/mysql, and that's what's asked for, I believe. [16:24:40] I gtg [16:38:59] petan, scfc_de: It doesn't look like there is a mariadb-client-core, just mariadb-client-core-5.5. Looks like the core package provides just /usr/bin/mysql and /usr/bin/mysqlcheck, while mariadb-client-5.5 provides all the other /usr/bin/mysql* utils. Note you can see what files are in an installed package with dpkg -L packagename; if you don't have it installed, download it and dpkg -c package.deb. [16:44:46] anomie: Yep; and the version number in the package name made me uneasy, so I went with mariadb-client in https://gerrit.wikimedia.org/r/#/c/71636/. [16:49:00] IIRC mariadb-client is just a virtual package to track dependency; mariadb-client-5.5 provides it. [16:51:46] Yes, and the virtual package made more sense to me in Puppet for example when 5.5.1 is released :-). [16:52:02] YuviPanda: I just found https://bugzilla.wikimedia.org/show_bug.cgi?id=48894 , I hope you didn't file another bug report? [16:52:15] That one's waiting for a response, though. [16:52:26] wolfgang42: yeah, no. I found that and left it as is [16:52:32] [bz] (8NEW - created by: 2Addshore, priority: 4High - 6enhancement) [Bug 48894] Include pagecounts dumps in datasets - https://bugzilla.wikimedia.org/show_bug.cgi?id=48894 [16:52:33] forgot to cc myself tho [16:52:34] * YuviPanda does [16:53:30] hah [16:53:47] addshore: hah? [16:53:58] * addshore got pinged :< [16:54:07] scfc_de: It's already up to 5.5.31 ;) But true for 5.6 (or 6.0, or whatever). [16:54:50] wolfgang42: poke Ryan_Lane with it when he comes online in a little while :) [16:55:27] * Coren commented on said bug. Why *is* http access to the pagecounts not adequate? It's not going to be any significantly slower than over, say, NFS. [16:56:21] Coren: they're huge, and you don't want multiple tools getting them over and over again [16:56:24] (I think?) [16:56:37] also HTTP is not significantly slower than NFS?! o_O [16:57:02] YuviPanda: You'll get the exact same network traffic if it's over NFS, and probably a bit less (NFS is chatty because it has to implement POSIX file semantics) [16:57:05] Coren: If everybody has their own copy, that means N times as much space. [16:57:05] Coren: Also, dumps.wikimedia.org is rate-limited, so it *is* slower [16:57:18] wolfgang42: Ah, didn't know about rate limits. [16:57:23] Coren: yeah, but NFS is over an internal network. dumps. is not [16:57:28] wolfgang42: You sure they apply on the local network? [16:57:41] I thought there was strict separation of labs and production? [16:58:27] Coren: Why would you want everyone to have to store their own local copy, though? [16:58:41] wolfgang42: I wouldn't. Why would they want or need to? [16:59:00] marc@tools-login:~$ tracepath dumps.wikimedia.org [16:59:00] 1: 10.4.0.1 2.242ms [16:59:00] 2: ae0-105.cr1-sdtpa.wikimedia.org 1.403ms [16:59:00] 3: dataset2.wikimedia.org 1.855ms reached [16:59:07] Looks like the local network to me. :-) [16:59:12] Coren: Because it's not available mounted to the local filesystem. [16:59:33] Coren: You need to put them somewhere while you're working on them. [16:59:36] wolfgang42: What does that change? Reading a file over HTTP is no harder than from the "local" filesystem (which isn't) [17:00:05] Coren: Also, would that not apply to the stuff currently in /public/datasets/public ? [17:00:14] Coren: I can't `cat` a URL. [17:01:46] Coren: you also can't 'seek' HTTP [17:01:57] wolfgang42: Yes you can. It's called 'curl -o - URI' [17:03:05] YuviPanda: Every scripting language I know allows you do. [17:03:07] to* [17:03:26] Coren: curl != cat . I have scripts that assume they're stored on the local filesystem, and I'd rather not change them if possible. [17:05:29] Duplicating 3.1T of data around to allow it to be used with one set of functions rather than another seems to be a fairly poor use of limited resources to me, especially since there is likely to be no performance gain. [17:06:22] Hell, I'd rather install a fuse HTTP filesystem than copy the data around. :-) [17:06:49] ... which isn't a half-bad idea now that I think about it. [17:07:31] * Coren tries it. [17:07:46] Or, as per Marc's comment #4, see if there's a way to share a single copy. [17:08:16] The problem with HTTP is that it doesn't map to directories well--it's difficult to get a directory listing. [17:09:45] * Coren is Marc. :-) [17:10:13] but is Marc Coren? :) [17:10:15] I think HTTP is not quite as "reliable" as NFS; you have to account for temporary network failures & Co. Also the "Unix way" usually means files in memory or as a temporary file, that can duplicate efforts. Not to speak of the amounts of developers that will each create their own special toolset to access the data with HTTP thus with their own little problems they blame on Tools instead of one working public dataset. [17:11:16] Coren: re: https://bugzilla.wikimedia.org/show_bug.cgi?id=48894 [17:11:36] * wolfgang42 pokes Ryan_Lane with https://bugzilla.wikimedia.org/show_bug.cgi?id=48894 [17:11:52] Oh, nevermind. You're already looking at it. [17:12:10] Coren: we can't really share it from the snapshots directly [17:12:23] and filesystem access would be better [17:12:50] we have way more than enough space, so making a copy doesn't hurt [17:13:11] I'd like to put this kind of stuff on the second set of NFS servers [17:14:48] Ryan_Lane: I'm looking at httpfs2 for that specific use case, and that would seem sane to me. [17:15:05] bleh [17:15:10] Ryan_Lane: It's on the local network; it'd work. [17:15:26] except that we'd be stealing basically all of its bandwidth [17:15:42] I don't see the problem with copying it [17:15:51] How is that different from the same data being transfered over NFS? [17:15:58] Except that we save the copy? [17:16:02] it only needs to be transferred once [17:16:23] once, to the NFS server. Actual access will also use the same network bandwidth. [17:16:39] to a different system [17:16:42] HTTP isn't really built for file access. You can't really just open() it [17:17:06] YuviPanda: You can, with the right libraries, but with httpfs2 you wouldn't even need to know it's over http. :-) [17:17:22]