[00:07:33] <Darkoneko>	 zzz
[00:41:20] <scream>	 How do I password recovery from
[00:41:21] <scream>	 https://lists.wikimedia.org/mailman/admin/simple-admins-l
[00:44:00] <scream>	 RobH, I'm not sure if you are the right person, but ping. :)
[00:47:47] <p858snake|l>	 scream: you probably can't, your best bet is probably to open a bz request and get someone to do it for you
[00:49:28] <nagios-wm>	 PROBLEM - Puppet freshness on snapshot4 is CRITICAL: Puppet has not run in the last 10 hours  
[00:49:46] <scream>	 I don't understand the reference... bz.
[00:50:26] <p858snake|l>	 bugzilla <http://bugzilla.wikimedia.org>, you would want to be filing under Wikimedia -> Mailing lists
[00:50:37] <scream>	 k
[02:00:11] <logmsgbot>	 !log LocalisationUpdate failed (1.18) at Mon Dec 19 02:00:11 UTC 2011
[02:00:24] <morebots>	 Logged the message, Master
[03:56:30] <nagios-wm>	 PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours  
[06:10:06] <nagios-wm>	 PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No  
[07:01:00] <apergos>	 !log thumb cleaner awake for the day. poor thing, slaving away but soon it will be able to retire
[07:01:10] <morebots>	 Logged the message, Master
[09:38:32] <nagios-wm>	 PROBLEM - Puppet freshness on bast1001 is CRITICAL: Puppet has not run in the last 10 hours  
[09:38:32] <nagios-wm>	 PROBLEM - Puppet freshness on fenari is CRITICAL: Puppet has not run in the last 10 hours  
[09:38:32] <nagios-wm>	 PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours  
[09:56:42] <nagios-wm>	 RECOVERY - MySQL slave status on es1004 is OK: OK:  
[10:09:19] <nagios-wm>	 PROBLEM - Disk space on db9 is CRITICAL: DISK CRITICAL - free space: /a 10743 MB (3% inode=99%):  
[10:11:37] <apergos>	 !log a few more binlogs on db9 gone.  eeking out another 12 hours or so
[10:11:45] <morebots>	 Logged the message, Master
[11:24:43] <nagios-wm>	 PROBLEM - Puppet freshness on snapshot4 is CRITICAL: Puppet has not run in the last 10 hours  
[11:27:10] <gerrit-wm>	 New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1618
[11:27:10] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1618
[11:31:06] <gerrit-wm>	 New review: Dzahn; "what about "misc::jenkins"?" [operations/puppet] (production); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/1611
[11:33:18] <gerrit-wm>	 New review: Dzahn; "sure, if the dir is missing anyways..." [operations/puppet] (production); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1620
[11:33:19] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1620
[11:36:43] <gerrit-wm>	 New review: Hashar; "misc::jenkins already exist and is a different project not related to mediawiki continuous integrati..." [operations/puppet] (production) C: 0;  - https://gerrit.wikimedia.org/r/1611
[11:48:39] <gerrit-wm>	 New review: Dzahn; "yep, we want to move stuff into separate files..." [operations/puppet] (production); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1611
[11:48:40] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1611
[11:50:59] <gerrit-wm>	 New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1617
[11:51:00] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1617
[11:59:13] <gerrit-wm>	 New review: Dzahn; "ok, convinced :)" [operations/puppet] (production); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1619
[11:59:14] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1619
[12:03:38] <mutante>	 would you guys also review me now?:)
[12:03:48] <mutante>	 1607 f.e.
[12:05:27] <gerrit-wm>	 New review: Dzahn; "just a tiny thing: "ensure=>installed;" would be nicer as "ensure => installed;"" [operations/puppet] (production); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/1612
[12:06:46] <gerrit-wm>	 New review: Dzahn; "Should "Nighties" be "Nightlies"? (nightly)?" [operations/puppet] (production); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/1615
[12:21:59] <nagios-wm>	 RECOVERY - Puppet freshness on bast1001 is OK: puppet ran at Mon Dec 19 12:21:46 UTC 2011  
[12:29:59] <nagios-wm>	 RECOVERY - Puppet freshness on fenari is OK: puppet ran at Mon Dec 19 12:29:31 UTC 2011  
[12:36:59] <nagios-wm>	 RECOVERY - Puppet freshness on snapshot4 is OK: puppet ran at Mon Dec 19 12:36:41 UTC 2011  
[12:47:52] <gerrit-wm>	 New patchset: Hashar; "basic class to install Android SDK prerequisites" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1612
[12:47:52] <gerrit-wm>	 New patchset: Hashar; "directory to host WikipediaMobile nightly builds" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1615
[12:47:52] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1615
[12:51:41] <gerrit-wm>	 New patchset: Hashar; "basic class to install Android SDK prerequisites" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1612
[12:51:54] <gerrit-wm>	 New patchset: Hashar; "directory to host WikipediaMobile nightly builds" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1615
[12:52:05] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1612
[12:52:05] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1615
[13:00:39] <gerrit-wm>	 New review: Dzahn; "i think you want the 2 links in their own <li> tag, so they are listed below each other. and how wil..." [operations/puppet] (production); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/1615
[13:27:04] <gerrit-wm>	 New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1612
[13:27:04] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1612
[13:33:39] <gerrit-wm>	 New patchset: Dzahn; "logrotate: add "managed by puppet" headers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1621
[13:34:19] <gerrit-wm>	 Change abandoned: Dzahn; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1621
[13:57:57] <gerrit-wm>	 New patchset: Hashar; "misc::contint::ant to ensure we have ant1.8" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1622
[14:00:31] <gerrit-wm>	 New patchset: Hashar; "misc::contint::ant to ensure we have ant1.8" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1622
[14:02:56] <gerrit-wm>	 New review: Dzahn; "one more } on line 64" [operations/puppet] (production); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/1622
[14:02:58] <gerrit-wm>	 New patchset: Hashar; "misc::contint::ant to ensure we have ant1.8" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1622
[14:09:34] <gerrit-wm>	 New review: Hashar; "Having all blogs under the same class looks nicer." [operations/puppet] (production) C: 1;  - https://gerrit.wikimedia.org/r/1607
[14:11:01] <gerrit-wm>	 New review: Hashar; "(no comment)" [operations/puppet] (production) C: 1;  - https://gerrit.wikimedia.org/r/1606
[14:15:18] <gerrit-wm>	 New review: Hashar; "Both links are part of the same sentence so they deserve to be in the same <li></li> block. " [operations/puppet] (production) C: 0;  - https://gerrit.wikimedia.org/r/1615
[14:17:01] <gerrit-wm>	 New patchset: Dzahn; "misc::contint::ant to ensure we have ant1.8" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1622
[14:19:16] <gerrit-wm>	 New patchset: Dzahn; "misc::contint::ant to ensure we have ant1.8" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1622
[14:19:21] <apergos>	 !log thumb cleaner to bed for the night... for the last time? 
[14:19:30] <morebots>	 Logged the message, Master
[14:20:53] <gerrit-wm>	 New review: Dzahn; "How do you like this? Putting it in generic::packages , so it can be used in other places too." [operations/puppet] (production); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/1622
[14:22:01] <gerrit-wm>	 New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1615
[14:22:02] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1615
[14:23:47] <gerrit-wm>	 New review: Dzahn; "yes, not using new classes anywhere initially, only once we are sure nothing is missing." [operations/puppet] (production); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1607
[14:23:47] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1607
[14:29:35] <gerrit-wm>	 New patchset: Hashar; "move misc::bugzilla::crons to misc/bugzilla.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1623
[14:29:48] <gerrit-wm>	 New patchset: Hashar; "misc::contint::ant to ensure we have ant1.8" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1622
[14:29:59] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1623
[15:10:34] <nagios-wm>	 PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours  
[15:31:02] <gerrit-wm>	 New review: Dzahn; "Yea, thanks. I was planning to do just this, too." [operations/puppet] (production); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1623
[15:43:33] <gerrit-wm>	 New patchset: Catrope; "Rewrite Nagios job queue check using new getJobQueueLengths.php script" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1624
[15:45:51] <RoanKattouw>	 mutante: ----^^
[15:47:40] <Nemo_bis>	 RoanKattouw, isn't this worth a !log so that someone looks into it? http://ganglia.wikimedia.org/?r=week&c=Miscellaneous+esams&h=maerlant.esams.wikimedia.org
[15:47:43] * Nemo_bis  is curious

[15:48:08] <RoanKattouw>	 Whoa that's a nasty curve in the load graph
[15:48:56] <^demon>	 maerlant? Is that a TS box?
[15:49:37] <RoanKattouw>	 !log maerlant is fried, load avg is 500+, linearly increasing since Friday. Rejects SSH login attempts
[15:49:46] <morebots>	 Logged the message, Mr. Obvious
[15:49:55] <RoanKattouw>	 ^demon: I thought it was the IPv6 gateway box
[15:50:04] <RoanKattouw>	 Whatever is running on it, it's not puppetized
[15:50:12] <RoanKattouw>	 It's only got "require standard" in puppet
[15:50:26] <^demon>	 Fun :)
[15:52:24] <notpeter>	 where the hell is maerlant?
[15:52:31] <^demon>	 esams.
[15:52:59] <notpeter>	 ty
[15:53:04] <notpeter>	 needed fqdn...
[15:53:28] <RoanKattouw>	 maerlant.esams.wikimedia.org , see the Ganglia URL
[15:53:38] <notpeter>	 yeah
[15:53:41] <notpeter>	 did you reboot it?
[15:53:56] <RoanKattouw>	 91.198.174.122
[15:53:58] <RoanKattouw>	 I didn't touch it
[15:53:59] <notpeter>	 oh man, popping a shell is taking forever!
[15:54:00] <notpeter>	 ok
[15:54:26] <notpeter>	 well, we'll just let that be and see if a shell ever appears...
[15:54:31] <RoanKattouw>	 I was just responding to Nemo_bis pointing out the worrying Ganglia graph
[15:54:36] <notpeter>	 kk
[15:54:49] <notpeter>	 I'm going to try to be really patient and see if I can geta  shell and some psaxu
[15:54:51] <RoanKattouw>	 If it does appear, renice it
[15:54:57] <RoanKattouw>	 (and ssh in as root)
[15:55:12] <notpeter>	 yeah, I'm coming through the console
[15:55:21] <RoanKattouw>	 Ah OK
[15:57:41] <gerrit-wm>	 New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1624
[15:57:42] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1624
[15:58:08] <mutante>	 RoanKattouw: looked good to me :) thx
[15:58:17] <RoanKattouw>	 Yay
[15:58:20] <RoanKattouw>	 Let's see if it runs correctly
[15:58:27] <RoanKattouw>	 I've tried it on fenari but you never know
[15:58:46] <RoanKattouw>	 (I also deployed the required MW script to spence, BTW)
[15:59:28] <mutante>	 ah, yes, the change to use mwscript was there before and it worked. trying to run puppet on spence
[16:00:33] <RoanKattouw>	 No, not mwscript
[16:00:38] <RoanKattouw>	 I mean getJobQueueLengths.php
[16:00:56] <RoanKattouw>	 That's not a pre-existing script, I wrote and deployed it today
[16:03:39] <notpeter>	 what runs on maerlant?
[16:04:24] <RoanKattouw>	 I don't know
[16:04:30] <RoanKattouw>	 Whatever it is it's not puppetized
[16:04:41] <RoanKattouw>	 I *think* it's the IPv6 gateway for upload.wm.o but I'm not sure
[16:05:07] <RoanKattouw>	 * 11:08 mark: Setup quick log rotation of ipv6and4 on maerlant
[16:05:15] <RoanKattouw>	 * 18:20 Ryan_Lane: moved ipv6 proxy from iris to maerlant
[16:05:15] <mutante>	 RoanKattouw: ohh.ok! gotcha
[16:05:28] <notpeter>	 awesome
[16:05:34] <mutante>	 RoanKattouw: we'll have to wait for spence..eventually picking up the change.. checking again later
[16:05:38] <notpeter>	 well.... I think rebooting is in order...
[16:05:44] <notpeter>	 I've been waiting 10 minutes for a shell
[17:01:46] <gerrit-wm>	 New patchset: Dzahn; "make service retry and check intervals configurable - still missing a line" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1625
[17:02:13] <gerrit-wm>	 New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1625
[17:02:14] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1625
[17:06:58] <gerrit-wm>	 New patchset: Dzahn; "fix path to check_job_queue" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1626
[17:07:23] <gerrit-wm>	 New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1626
[17:07:24] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1626
[17:53:06] <gerrit-wm>	 New patchset: Hashar; "move misc::bugzilla::crons to misc/bugzilla.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1623
[17:53:18] <gerrit-wm>	 New patchset: Hashar; "misc::contint::ant to ensure we have ant1.8" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1622
[17:55:45] <gerrit-wm>	 New patchset: Hashar; "move misc::bugzilla::crons to misc/bugzilla.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1623
[17:55:58] <gerrit-wm>	 New patchset: Hashar; "generic::packages::ant18 to ensure we have ant1.8" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1622
[18:15:22] <gerrit-wm>	 New review: Catrope; "(no comment)" [operations/puppet] (production) C: 1;  - https://gerrit.wikimedia.org/r/1622
[18:20:05] <Nikerabbit>	 RoanKattouw: going to scap :o
[18:20:25] <RoanKattouw>	 Ok go ahead
[18:31:58] <logmsgbot>	 !log nikerabbit synchronizing Wikimedia installation... : I18ndeploy r106667 and new extensions on mediawiki.org
[18:32:07] <Nikerabbit>	 Copying to fenari...rsync: failed to set times on "/usr/local/apache/common-local/php-1.18/cache/l10n": Operation not permitted (1)
[18:32:08] <morebots>	 Logged the message, Master
[18:34:38] <logmsgbot>	 sync done.
[18:37:27] <logmsgbot>	 !log nikerabbit synchronized wmf-config/InitialiseSettings.php  'Translate needs tables'
[18:37:35] <morebots>	 Logged the message, Master
[18:38:45] <Nikerabbit>	 just a little oops
[18:56:56] <Nikerabbit>	 can someone help me to apply some new tables?
[18:59:22] <jeremyb>	 Nikerabbit: as in add tables to all the dbs (or even a few) in prod?
[18:59:47] * jeremyb  was having trouble understanding "apply"

[18:59:48] <Nikerabbit>	 one production wiki
[18:59:57] <Nikerabbit>	 add new tables
[19:00:54] <jeremyb>	 i'm sure binasher can but he seems to be appearing just because i said he wasn't here
[19:01:11] <jeremyb>	 not sure who else
[19:01:13] <binasher>	 hah
[19:01:15] <binasher>	 can what?
[19:01:21] <jeremyb>	 add tables to one prod wiki
[19:02:05] <binasher>	 adding tables is painless…. bugzilla / some sort of documented request? 
[19:03:11] <jeremyb>	 binasher: you'll have to ask teh Nikerabbit... he just asked the wind in here
[19:03:18] <Nikerabbit>	 binasher: l10n team requests Translate on mediawiki.org
[19:03:55] <RoanKattouw>	 Nikerabbit: That's easy
[19:04:03] <Nikerabbit>	 RoanKattouw: yes you can slap me now
[19:04:05] <RoanKattouw>	 mwscript sql.php --wiki=mediawikiwiki path/to/foo.sql
[19:07:12] <Nikerabbit>	 okay, let's try again
[19:08:34] <logmsgbot>	 !log nikerabbit synchronized wmf-config/InitialiseSettings.php  'Translate with tables'
[19:08:42] <morebots>	 Logged the message, Master
[19:14:51] <nagios-wm>	 RECOVERY - check_job_queue on spence is OK: JOBQUEUE OK - all job queues below 10,000  
[19:20:23] <apergos>	 what does sql.php do that's clever (why use it instead of sql wikiname < blot.sql) ?
[19:25:14] <jeremyb>	 different creds/privs?
[19:25:56] <RoanKattouw>	 apergos: /*$wgDBname*/
[19:26:08] <RoanKattouw>	 In practice that's not needed for us
[19:26:23] <RoanKattouw>	 But for us, it also automagically connects to the right DB server (which cluster? which box is the master?)
[19:26:44] <apergos>	 sql would connect to the right cluster
[19:26:54] <jeremyb>	 master is in DNS?
[19:26:55] <apergos>	 I don't know if it connects to the master but I bet it does
[19:26:57] <apergos>	 I should check that
[19:27:36] <RoanKattouw>	 Yes, the sql command works too
[19:27:40] <apergos>	 wfGetLB()->getServerName(0)
[19:27:41] <RoanKattouw>	 jeremyb: DNS? No?
[19:27:42] <apergos>	 yup
[19:27:47] <RoanKattouw>	 Just in db.php
[19:28:03] <apergos>	 so fur us either is fine.  ok, thanks...
[19:28:05] <jeremyb>	 RoanKattouw: oh, sql!=mysql
[19:28:12] <apergos>	 no.it isn't.
[19:28:40] <apergos>	 it does nice things like know the password for you, figure out the right host
[19:28:43] <RoanKattouw>	 sql is a wrapper script around mysql
[19:28:45] <RoanKattouw>	 yeah
[19:28:52] <apergos>	 well really that's all it does :-D but that's enough
[19:28:55] <jeremyb>	 yeah, i forgot
[19:42:07] <gerrit-wm>	 New patchset: Asher; "install new mysql pkgs on db9/10" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1627
[19:42:54] <gerrit-wm>	 New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/1627
[19:42:54] <gerrit-wm>	 Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1627
[19:47:47] <gerrit-wm>	 New patchset: Asher; "Revert "install new mysql pkgs on db9/10"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1628
[19:48:33] <gerrit-wm>	 New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/1628
[19:48:33] <gerrit-wm>	 Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1628
[20:11:23] <logmsgbot>	 !log catrope synchronized php-1.18/extensions/ClickTracking/ApiClickTracking.php  'r106681'
[20:11:32] <morebots>	 Logged the message, Master
[20:14:00] <nagios-wm>	 RECOVERY - Puppet freshness on maerlant is OK: puppet ran at Mon Dec 19 20:13:35 UTC 2011  
[20:14:40] <nagios-wm>	 RECOVERY - SSH on maerlant is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0)  
[20:25:05] <logmsgbot>	 !log catrope synchronized php-1.18/extensions/ArticleFeedbackv5/ 
[20:25:14] <morebots>	 Logged the message, Master
[20:37:46] <logmsgbot>	 !log catrope synchronized wmf-config/InitialiseSettings.php  'Enable ArticleFeedbackv5 on enwiki'
[20:37:54] <morebots>	 Logged the message, Master
[20:39:36] <logmsgbot>	 !log catrope synchronized wmf-config/InitialiseSettings.php  '...and disable for now'
[20:39:45] <morebots>	 Logged the message, Master
[20:43:17] <Prodego>	 so what did you guys do to javascript
[20:43:23] <RoanKattouw>	 ?
[20:43:25] <Prodego>	 now if it doesn't work it seems to just not get loaded at alll?
[20:43:28] <Prodego>	 -l?
[20:43:37] <RoanKattouw>	 What do you mean?
[20:43:51] <Prodego>	 I made some changes to a script of mine - http://en.wikipedia.org/w/index.php?title=User:Prodego/Sandbox.js&diff=466745747&oldid=426993321
[20:43:55] <Prodego>	 probably made some errors
[20:44:06] <Prodego>	 but now, so far as I can tell chrome doesn't load that page at all
[20:45:21] <Prodego>	 which means I can't see any error message of course
[20:45:23] <RoanKattouw>	 missing ( before condition
[20:45:25] <RoanKattouw>	 [Break On This Error] if !isUserBlocked(user) { 
[20:45:34] <RoanKattouw>	 Chrome's console *should* show you a similar error
[20:45:55] <Prodego>	 RoanKattouw: the page disappears from the 'loaded scripts' page
[20:45:58] <Prodego>	 in chrome
[20:46:05] <RoanKattouw>	 That's strange
[20:46:18] <RoanKattouw>	 How is it loaded?
[20:46:38] <Prodego>	 its just included in my monobook.js
[20:46:39] <RoanKattouw>	 Cause this is probably loaded through action=raw or somethnig
[20:46:46] <logmsgbot>	 !log catrope synchronized php-1.18/extensions/ArticleFeedbackv5/ 
[20:46:55] <morebots>	 Logged the message, Master
[20:47:09] <RoanKattouw>	 Yeah
[20:47:14] <Prodego>	 ah
[20:47:17] <Prodego>	 this is chrome's fault
[20:47:17] <RoanKattouw>	 So we didn't actually mess with that kind of script loading at all
[20:47:19] <RoanKattouw>	 Yes
[20:47:34] <Prodego>	 it seems it just silently drops the page if it doesn't like it
[20:48:23] <logmsgbot>	 !log catrope synchronized wmf-config/InitialiseSettings.php  'Re-enable ArticleFeedbackv5, hopefully fixed now'
[20:48:32] <morebots>	 Logged the message, Master
[20:48:36] <RoanKattouw>	 !log ...on enwiki that is
[20:48:45] <morebots>	 Logged the message, Mr. Obvious
[20:51:43] <Jarry1250>	 How long should it take AFv5 to show?
[20:52:37] <RoanKattouw>	 Jarry1250: It should only be on pages in [[Category:Article Feedback 5]]
[20:52:41] <RoanKattouw>	 And it should be visible now-ish
[20:52:53] <Jarry1250>	 Yup, I figured that :P But I can't see it?
[20:52:56] <RoanKattouw>	 The AFTv5 people are testing right now, they're over in #wikimedia-dev
[20:53:17] <RoanKattouw>	 I see it
[20:53:21] <RoanKattouw>	 https://en.wikipedia.org/wiki/1935_Ohio_State_Buckeyes_football_team
[20:54:01] <nagios-wm>	 PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours  
[21:01:59] <Prodego>	 RoanKattouw: can I send any request to the api as a post request, or only those that require post request?
[21:02:14] <RoanKattouw>	 POST is accepted for everything
[21:04:30] <logmsgbot>	 !log catrope synchronized php-1.18/extensions/ArticleFeedbackv5/ 
[21:04:38] <morebots>	 Logged the message, Master
[21:04:53] <Prodego>	 thanks... hmm I can't figure out why this doesn't work
[21:04:58] <Prodego>	 but then again I really have no idea what I'm doing
[21:06:57] <Prodego>	 ohhh
[21:07:05] <Prodego>	 helps to actually send the request :(
[21:09:31] <gerrit-wm>	 New patchset: Lcarr; "Adding in mysql-client to requirements for xtrabackup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1629
[21:11:43] <gerrit-wm>	 New patchset: Lcarr; "Adding in mysql-client to requirements for xtrabackup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1629
[21:11:55] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1629
[21:15:40] <apergos>	 !log started an scp (to make ds1 work a tiny bit harder) of some files from ds2. running on ds2 in screen session as root. 
[21:15:48] <morebots>	 Logged the message, Master
[21:16:03] <apergos>	 !log note that dataset1 appears to be keeping time ok
[21:16:11] <morebots>	 Logged the message, Master
[21:16:12] <apergos>	 I have now jinxed it but oh well
[21:16:41] <gerrit-wm>	 New patchset: Lcarr; "Adding in mysql-client to requirements for xtrabackup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1629
[21:21:22] <gerrit-wm>	 Change abandoned: Lcarr; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1629
[21:28:42] <logmsgbot>	 !log asher synchronized wmf-config/db.php  'pulling db43 which died'
[21:28:51] <morebots>	 Logged the message, Master
[21:44:50] <logmsgbot>	 !log awjrichards synchronizing Wikimedia installation... : Deploying ContributionReporting fixes to use summary tables (r106696), disabling ContributionReporting everywhere except test and foundationwikis
[21:44:59] <morebots>	 Logged the message, Master
[21:46:41] <logmsgbot>	 sync done.
[22:33:57] <sumanah>	 Ryan_Lane: maplebed: LeslieCarr: hi, I heard that our Bugzilla database is getting to the edge of falling over and we might need to take it down for maintenance, today or sometime this week.  Can you confirm or deny this rumour? :)
[22:34:19] <Ryan_Lane>	 binasher: ^^
[22:36:19] <binasher>	 yes
[22:37:16] <sumanah>	 binasher: do you have a sense of when this downtime might happen?  I'd like to let people know if I can
[22:37:30] <binasher>	 sumanah: i'll email you in advance 
[22:37:43] <sumanah>	 binasher: you'll email wikitech-l/mediawiki-l?
[22:38:01] <binasher>	 y
[22:38:29] <sumanah>	 thanks binasher 
[22:41:38] <gerrit-wm>	 New patchset: Asher; "new pmtpa dbs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1636
[22:42:53] <gerrit-wm>	 New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/1636
[22:42:53] <gerrit-wm>	 Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1636
[22:52:45] <gerrit-wm>	 New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/1622
[22:52:46] <gerrit-wm>	 Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1622
[22:57:39] <gerrit-wm>	 New patchset: Hashar; "comment about ant => latest not being latest ant!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1637
[22:57:51] <gerrit-wm>	 New patchset: Hashar; "move misc::bugzilla::crons to misc/bugzilla.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1623
[22:58:38] <gerrit-wm>	 New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/1637
[22:58:38] <gerrit-wm>	 Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1637
[23:07:15] <nagios-wm>	 RECOVERY - DPKG on db17 is OK: All packages OK  
[23:10:34] <DaBPunkt>	 gn8 folks
[23:18:54] <gerrit-wm>	 New patchset: Hashar; "Create WikipediaMobile/nightly parent directory" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1640
[23:21:26] <gerrit-wm>	 New review: Lcarr; "LGTM" [operations/puppet] (production); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/1640
[23:23:11] <gerrit-wm>	 New patchset: Hashar; "Create WikipediaMobile/nightly parent directory" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1640
[23:23:55] <gerrit-wm>	 New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/1640
[23:23:56] <gerrit-wm>	 Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1640