[00:13:03] Reedy: I'm still busy deploying the change I just merged into wmf/1.21wmf1 [00:13:17] Rebuilding l10n cache because it touches MessagesEn.php [00:13:36] Funtimes [00:13:45] I only changed 1 file in Collection... [00:13:51] Non-i18n? [00:13:54] yup [00:13:56] [[Tech]]; Rjd0060; /* Sidebar link script */ new section; https://meta.wikimedia.org/w/index.php?diff=4199286&oldid=4163795&rcid=3570809 [00:13:58] OK go ahead and sync that hten [00:14:06] This cache rebuild is taking forever [00:14:21] I waited about 90 minutes for it all on monday [00:14:40] Oh I'm not using scap [00:15:01] I'm running mw-update-l10n followed by sync-l10nupdate-1 [00:15:11] to just do the l10n cache [00:15:24] Then syncing my changes separately. It's a one-line change in MessagesEn plus a JS change [00:21:40] * RD wonders where tomsaz is [00:21:55] oh, found him lol [00:34:02] RoanKattouw: just curious, if a change involves stopping using message key X and start using (newly added) message key Y (either as the main change or as part of a larger change) [00:34:18] RoanKattouw: in what order to deploy that, or needs scap? [00:36:23] What he's doing is pretty much teh same thing [00:36:38] it just doesn't go through checking everything else [00:38:33] You'd just run scap [00:38:46] (assuming a magical world where scap doesn't take forever) [00:38:51] Otherwise, you'd sync the l10n stuff before the code [00:45:00] RoanKattouw: because i18n takes longer than sync ? [00:45:15] No, because you want the new i18n first [00:45:21] Although, I guess you removed the old message, huh [00:45:28] Then you'd just sync the whole thing at the same time [01:10:43] gn8 folks [03:01:34] wikimediafoundation.org hung there for a few seconds on page save, but seems to be fine now. [03:11:52] Why do SVG links have &page=1 in them now? [03:11:55] Are SVGS paginated? [05:38:48] Brooke: I think they can be [08:07:21] apergos: There? Can I request that the add/change dumps be added together with the rsync to the publicdata project mount on Gluster? Please? [08:08:19] yes, and I think I need to see what's going on with that too (once the media space emergency is over). [08:09:06] can you send me an email with that in it? and if you hear nothing after a week, ping me [08:09:07] hmm? It looks like its working nicely to me [08:09:15] sure [08:09:18] or want a bug? [08:09:28] oh, it is? I know I kicked it but wasn't sure if it actually recovered [08:09:45] but was there a problem that even happened at all? [08:10:11] well a while back the mount went away [08:10:29] oh, I had a bug filed [08:10:34] Ryan fixed it for me [08:10:40] so its probably working now [08:10:45] ah that would be it [08:11:22] yeah email will be better I think, I look at that every day [08:11:25] sure [08:14:40] btw Hydriz [08:14:56] * Hydriz is listening [08:14:56] do you know that any output from your cron jobs on the labs instances goes to root and then to the ops team? [08:15:16] /bin/sh /home/hydriz/cron this produces output every run i guess [08:15:18] oh shyt [08:15:31] can I change that anywhere? [08:15:32] Already up-to-date. [08:15:47] Its running very very often [08:15:53] well could you haveit log the output to a file or someting, if you really need it? [08:16:03] > /dev/null? [08:16:07] or if you don't want the output, have it go to /dev/null [08:16:09] yeah [08:16:17] oh god, I am sooo sorry [08:16:20] heh [08:16:47] I must have spammed you guys 3600 mails per hour [08:16:56] no, 60 mails I meant [08:17:05] if you are working with anyone else on labs, remind them to look at their cron jobs for the same reason [08:17:10] yep, 60 an hour, that's right [08:17:19] yeah, I have one running per minute [08:17:20] over a day, that's... [08:17:26] too many :-D [08:17:38] I was still wondering why I don't get emails from cron at all [08:17:53] while I get spammed by it from the Toolserver [08:17:59] yeah well we get them [08:18:04] do you have any idea who runs [08:18:18] bots-apache-test.pmtpa.wmflabs [08:18:23] cause they have a pretty noisy job too [08:18:36] yeah, I can ping [08:18:40] ok [08:18:53] we get one from them every 5 mins so... just let em know the same thing if you don' tmind [08:18:56] thanks for that info! Need to add this to the docs [08:19:00] ...on labs [08:19:03] ah yes,good point [08:19:11] but do you know where to change this? [08:19:22] *the email address [08:19:27] oh [08:19:41] well if you want all the emails to you, you can put a line at the top of the crontab [08:20:09] http://www.cyberciti.biz/faq/linux-unix-crontab-change-mailto-settings/ look for MAILTO [08:20:14] apergos: Are there any other instances involved? [08:20:16] (it was the first hit in google for that) [08:20:27] for which? [08:20:39] the cron mails or something [08:21:07] well just deal with your own crontab on your own vm [08:21:35] sure, no problem [08:21:50] No longer getting mails from incubator-apache now, I presume? [08:26:12] dunno, it's buried in thenoise if I am [08:26:28] heh [08:26:36] Just sent the mail you needed, btw [08:28:45] I have it, thanks [08:30:27] np [09:20:03] Reedy: do you have anything to do with the toolserver? [10:39:48] !log gave myself an account on sanger's exim so that I can send outbound mail through it [10:40:00] Logged the message, Master [10:42:44] SPF? [10:44:22] I use smtp.gmail.com :-) [10:48:08] yes, for SPF [10:54:56] heh, just checking my mailing list posts from earlier today [10:55:37] mailman doesn't check SPF, it just remails the posts, so my posts from fastmail got an SPF pass despite the new policy [11:05:00] [[Tech]]; Ruslik0; /* Sidebar link script */; https://meta.wikimedia.org/w/index.php?diff=4200884&oldid=4199286&rcid=3571609 [11:51:08] does anyone know what is needed to move a bot from toolserver to bots.wmf.org? [14:02:20] !log Manually ran sync-common on srv247 [14:02:30] Logged the message, Master [20:38:09] rmoen|afk: ping me when you're back, if you have a moment.. [20:54:12] ori-l: Howdy [20:54:26] rmoen: i'll PM [20:54:38] ori-l: k [21:33:09] anyone have an idea why account throttling would still throttle even though throttle.php has been updated ? [21:33:47] LeslieCarr: It's likely been cached as over the limit in memcached [21:33:48] soooooooo [21:34:01] should just need to purge the relevant key [21:34:20] ah [21:34:27] so… reboot all memcaches ? [21:34:27] ;) [21:34:53] I'm sure Asher would buy you a pint for that :p [21:35:03] you mean a pint of my own blood ? [21:37:02] Reedy: got an idea how to purge the relevant key ? [21:37:11] That's the easy part [21:37:16] I'm just trying to find out what said key is [21:37:30] Have you tried both commons and enwiki? [21:39:05] * @warning Requires memcached */ [21:39:05] $wgAccountCreationThrottle = 0; [21:39:08] heh [21:39:34] $key = wfMemcKey( 'acctcreate', 'ip', $ip ); [21:39:53] okay [21:39:59] havne't tried on commons [21:40:16] enwiki:acctcreate:ip:12.183.19.7 [21:40:20] commonswiki:acctcreate:ip:12.183.19.7 [21:41:17] LeslieCarr: Both purged, try it now [21:42:51] yay [21:42:54] thanks Reedy [21:42:54] :) [21:43:47] back to helping women make awesome articles on awesome computer scientists! [21:43:58] and thanks mutante [21:44:23] sounds like either fun or hell [21:44:40] ah hah [21:44:42] Damianz: [21:45:14] you have a bots-apache1 cron job that sends mail every 5 mins [21:45:27] win [21:45:31] um [21:45:36] every minute actually [21:45:38] I'll &> /dev/null that in a sec [21:45:41] thanks! [21:46:40] the entire ops team will be grateful (cron jobs from lab projects spam all our mailboxes) [21:47:03] lol [21:47:17] You mean you don't want to know the users and related ids in bots every min? ;D [21:47:48] Should be quiet for now, I'll fix my puppet stuff to mute it on new hosts also [21:48:24] * apergos watches the clock for a minute [21:49:22] apergos: are you changing filebackend.php? [21:49:32] not today! [21:49:32] tomorrow [21:49:37] yes, I expect so [21:49:52] can you put the change in gerrit and what not? [21:50:04] yes, I was planning to do all that [21:50:09] standard check in and merge [21:50:29] but I mean in advance of deployment [21:50:36] not just right before [21:50:39] I can put it in the morning my time [21:50:59] (it's 1 am here, I am not doing any real work at this our) [21:51:11] hour! [21:54:21] the only thing I planned to change in there literally is upload6 -> upload7 [21:54:27] AaronSchulz [21:54:51] so that's what we are calling it? [21:54:56] creative [21:55:03] lol [21:55:14] * apergos would growl but is too tired for that [21:59:47] apergos: are the netapps mirroring now? [22:00:10] mirroring? [22:00:13] ms7 [22:00:30] or is it slowly falling out of date? [22:00:32] I must be too sleepy, don't get the question [22:00:48] well there's one, no two more rsyncs running [22:00:57] so some bits are current and some bits are out of date [22:01:28] there's no cron job or anything, this is still the first push of data [22:01:58] I restarted a job earlier today which will walk through a bunch of the dirs but not all [22:02:14] and it likely won't complete by the window [22:02:32] too many files :-P [22:04:29] if you have a special request, and asher isn't buried alive in other things, he might be able to run somthing for you [22:07:41] Damianz: bots-apache-test, is that you also? [22:08:08] yes [22:08:09] same thing? [22:08:22] every 5 mins [22:08:24] yeah [22:08:37] 1sec I'll fix puppet [22:08:41] thanks [22:14:06] Those should be sorted now [22:14:30] Do you get any from bots-sql{1..3}? I think their crons don't output anything but only run once a day. [22:14:35] ok, I'll watch the clock again [22:14:41] * Damianz might as well do them also but one of the boxes is down at the min so will have to later. [22:15:02] I don't see those but I might have already tossed those emails [22:16:43] Seems I've done them previously so should be good. [22:16:53] yay [22:19:08] looks like the every 5 mins are gone now [22:20:04] cool [22:20:50] apergos: What time tomorrow? [22:21:01] 11 am utc to 2 pm utc [22:21:03] it's in the email [22:22:02] shouldn't take til 2 but... [22:22:10] better safe than sorry [23:17:25] gn8 folks [23:27:32] tfinc wants me to upload some files to download.wikimedia.org…. i think i finally need my shell back :) [23:27:39] who do I poke for that these days? [23:28:39] mutante, can you help brion? [23:28:53] * brion breaks out the puppy dog eyes [23:29:34] make sure you get UID 500 [23:29:44] heh [23:29:47] there's still a whole lot of files that are owned by 500 with no name [23:29:54] ah, old times [23:31:54] Eloquence: brion , yea [23:32:03] whee [23:32:49] brion: do you have files on fenari? [23:32:59] i might have old files there, unsure [23:33:12] check for an old /home/brion or something in case it got kept [23:33:17] mutante: in brion's case, % match against cardboard cutout is a valid form of authentication for security purposes. [23:33:26] hehehe [23:33:29] we kept brion's bed made [23:33:39] i think i've gained a few pounds since the cutout, i may not match 100% ;) [23:33:40] he's got 220MB of files in /home/brion [23:33:44] /home/brion/keep/: directory [23:33:46] awww [23:33:52] you guys [23:44:09] running puppet on fenari.... [23:45:17] whee! [23:48:06] OS X supplemental update… "Addresses an issue that may prevent systems with more than 64 GB of RAM from starting up" -- wish I had that problem! :P [23:48:24] Oct 4 23:44:55 fenari puppet-agent[11559]: (/Stage[main]/Accounts::Brion/Unixaccount[Brion Vibber]/User[brion]/ensure) created [23:48:35] but..did not add your key yet..looking [23:48:43] whee! yeah still won't let me log in yet [23:59:52] !log manually fixed brion's ssh key on fenari. session opened for user brion. welcome back :)