[00:08:50] <gerrit-wm>	 New patchset: Asher; "extend check_mysql_slave_delay to support pt-heartbeat monitoring by server_id in a multi-tier replication tree + nrpe hooks." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1920
[00:09:05] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1920
[00:19:48] <Ryan_Lane>	 !log powercycling formey
[00:19:50] <morebots>	 Logged the message, Master
[00:21:09] <malafaya>	 hi. what does "Node-count limit exceeded" mean?
[00:21:25] <Ryan_Lane>	 where are you seeing this?
[00:21:53] <Reedy>	 Ryan_Lane, did it swap die?
[00:21:57] <Ryan_Lane>	 maybe
[00:22:02] <malafaya>	 at the bottom of https://pt.wiktionary.org/w/index.php?title=Wikcion%E1rio:Lista_de_l%EDnguas
[00:22:03] <Ryan_Lane>	 I have no clue
[00:22:09] <Ryan_Lane>	 Reedy: did you do something?
[00:22:13] <Reedy>	 No
[00:22:14] <Reedy>	 I've seen it a few times in the last few weeks
[00:22:17] <Ryan_Lane>	 ah. ok
[00:22:18] <Ryan_Lane>	 really?
[00:22:21] <Ryan_Lane>	 swap death?
[00:22:24] <Ryan_Lane>	 what was causing it?
[00:22:34] <Reedy>	 One time Tomasz killed process for me when I noticed them
[00:22:43] <Reedy>	 viewvc.cgi processes that max the cpu and cause a lot of swapping
[00:22:46] <Reedy>	 run for 20-30 minutes+
[00:23:05] <Reedy>	 wonder if there's some exploit in viewvc
[00:23:43] <FT2>	 hi, is something up with server lag?
[00:23:44] <FT2>	 The last 24 hrs I've had issues where I make edits, click to edit the page further and I'm getting old revisions lacking the most recent edits (which appear on history and isn't fixed by purge)
[00:23:45] <nagios-wm>	 PROBLEM - Puppet freshness on srv191 is CRITICAL: Puppet has not run in the last 10 hours  
[00:26:46] <gerrit-wm>	 New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/1920
[00:32:30] <gerrit-wm>	 Change abandoned: Ryan Lane; "something went wrong." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1920
[00:33:17] <gerrit-wm>	 New patchset: Asher; "extend check_mysql_slave_delay to support pt-heartbeat monitoring by server_id in a multi-tier replication tree + nrpe hooks." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1921
[00:33:47] <gerrit-wm>	 New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/1921
[00:33:47] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1921
[00:33:49] <gerrit-wm>	 New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/1921
[00:33:52] <gerrit-wm>	 Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1921
[00:53:14] <Ryan_Lane>	 !log stopping pdns on virt1 again to test dns
[00:53:16] <morebots>	 Logged the message, Master
[01:00:23] <Ryan_Lane>	 !log brought pdns on virt1 back up
[01:00:24] <morebots>	 Logged the message, Master
[02:04:47] <logmsgbot>	 !log LocalisationUpdate completed (1.18) at Sat Jan 14 02:04:46 UTC 2012
[02:04:49] <morebots>	 Logged the message, Master
[02:05:16] <Reedy>	 Ontime
[02:14:51] <logmsgbot>	 !log LocalisationUpdate failed: SVN update of extensions failed
[02:14:53] <morebots>	 Logged the message, Master
[02:15:58] <Reedy>	 !log That was me testing something
[02:16:00] <morebots>	 Logged the message, Master
[02:27:14] <saper>	 Reedy: can you file rt tickets?
[02:27:23] <Reedy>	 yeah
[02:27:37] <Reedy>	 why?
[02:28:26] <saper>	 I need !bug 33509 to be filed there, since that's not a typical problem we should deal with in bugzilla
[02:28:56] <Reedy>	 Ah
[02:28:56] <Reedy>	 yeah
[02:29:07] <gerrit-wm>	 New patchset: Bhartshorne; "added query duration statistics" [operations/software] (master) - https://gerrit.wikimedia.org/r/1922
[02:29:54] <gerrit-wm>	 New review: Bhartshorne; "(no comment)" [operations/software] (master); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1922
[02:29:57] <Reedy>	 done
[02:30:58] <Reedy>	 long query was long
[02:31:30] <saper>	 thanks
[02:31:40] <saper>	 wonder who could possibly handle this
[02:32:40] <Reedy>	 Depends where the entry is actually hosted
[02:32:50] <Reedy>	 I know Rob did some for the labs domains earlier today
[02:33:12] <tr|nn|>	 Hi, anyone can tell me how much data memcached can deal on mw-servers?
[02:33:21] <Reedy>	 alot
[02:33:37] <Reedy>	 errm
[02:33:39] <tr|nn|>	 gigabytes then?
[02:33:49] <Reedy>	 http://noc.wikimedia.org/conf/highlight.php?file=mc.php
[02:33:51] <Reedy>	 79 hosts
[02:34:53] <tr|nn|>	 Ooh, now I need a manual to translate that
[02:35:00] <Reedy>	 $wgMemCachedInstanceSize = 2000;
[02:35:17] <Reedy>	 http://svn.wikimedia.org/viewvc/mediawiki/trunk/debs/memcached/debian/memcached.conf?view=markup
[02:35:20] <Reedy>	 2G each
[02:35:34] <Reedy>	 so at least 158GB
[02:35:38] <tr|nn|>	 ah yeah, thats it :)
[02:36:08] <Reedy>	 possibly more
[02:36:11] <Reedy>	 but can't be far off
[02:37:06] <tr|nn|>	 158GB should be enough :)
[02:37:56] <DaBPunkt>	 gn8 folks
[02:39:44] <saper>	 Reedy: instra.com, I think it's pretty old
[02:39:51] <tr|nn|>	 now I need to lower the cache for mysqld and get memcached installed...
[02:40:06] <saper>	 Reedy: we host the site so no changes are needed besides DNS registrar bureaucracy
[02:40:24] <Reedy>	 Most of WMF stuff now are on godaddy
[02:40:37] <saper>	 this one is so old that it isn't
[02:41:17] <saper>	 They should have had a password since the change was made from Florida to San Franisco
[02:41:27] <saper>	 *cisco
[02:41:46] <Reedy>	 heh
[02:43:20] <nagios-wm>	 PROBLEM - Host sq46 is DOWN: PING CRITICAL - Packet loss = 100%  
[02:52:59] <nagios-wm>	 RECOVERY - Puppet freshness on srv191 is OK: puppet ran at Sat Jan 14 02:52:36 UTC 2012  
[02:56:19] <tr|nn|>	 Reedy, $wgMemCachedTimeout is in seconds?
[02:58:20] <tr|nn|>	 umph, google is my friend -- sorry
[02:58:26] <Reedy>	 Heh
[02:58:35] <Reedy>	 Most likely
[02:58:50] <tr|nn|>	 did not expect to find sth via google
[02:59:11] <tr|nn|>	 http://www.mediawiki.org/wiki/Manual:$wgMemCachedTimeout
[02:59:38] <Reedy>	 Most of our globals are documented to some level or another
[03:00:00] <tr|nn|>	 that is exemplary :)
[03:00:47] <Reedy>	 I wonder why $wgMemCachedInstanceSize isn't
[03:02:07] <Reedy>	 it's not in core
[03:02:13] <Reedy>	 I wonder if it's legacy/invented
[03:02:24] <Reedy>	 as it'll be set in the memcached config rather than by mw
[03:03:44] <tr|nn|>	 well at least size is defined in the memcached.conf as 2000
[03:12:49] <tr|nn|>	 I am wondering if there is a max "timeToLive" for a memcached object -- cannot find $wgMemc's Class -- no manualpage :(
[03:14:41] <Reedy>	 Look at the code
[03:15:32] <Reedy>	 max ttl will be set in memcached itself I'd guess
[03:15:38] <tr|nn|>	 Well, it tells me I can set a ttl, but I am just wondering if there is a max value behind that
[03:16:42] <Reedy>	 usually most people aren't caching data that is that static
[03:16:52] <Reedy>	 http://www.google.co.uk/webhp?sourceid=chrome-instant&ix=tea&ie=UTF-8&ion=1#sclient=psy-ab&hl=en&site=webhp&source=hp&q=memcached%20max%20ttl&pbx=1&oq=&aq=&aqi=&aql=&gs_sm=&gs_upl=&fp=4c26890357e371b1&ion=1&ion=1&bav=on.2,or.r_gc.r_pw.r_cp.,cf.osb&fp=4c26890357e371b1&ion=1&biw=1366&bih=667
[03:16:56] <Reedy>	 Google suggests 30 days
[03:17:41] <tr|nn|>	 whew, 30 days is alot
[04:15:32] <nagios-wm>	 RECOVERY - Disk space on es1004 is OK: DISK OK  
[04:16:52] <nagios-wm>	 RECOVERY - MySQL disk space on es1004 is OK: DISK OK  
[04:39:28] <nagios-wm>	 PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No  
[04:59:57] <Saiba>	 PHP fatal error in /usr/local/apache/common-local/php-1.18/includes/objectcache/MemcachedClient.php line 912:            Allowed memory size of 125829120 bytes exhausted (tried to allocate 533926 bytes)     
[04:59:57] <Saiba>	 on http://commons.wikimedia.org/w/index.php?title=Special:RecentChanges&limit=5000
[05:00:09] <Saiba>	 get some RAM! ;)
[05:00:49] <Saiba>	 2500 works
[05:01:27] <jeremyb>	 you shouldn't be permitted to get 5000 i thought?
[05:01:34] <jeremyb>	 unless you have some special rights?
[05:01:47] <jeremyb>	 bot or apihighlimits or sysop?
[05:01:51] <Saiba>	 I didn't get them  - got the error instead ;)
[05:01:52] <Saiba>	 sysop
[05:02:15] <Saiba>	 but.. em.. 5000 list entries... not that much?!
[05:02:44] <jeremyb>	 most ppl are limited to 500 i think...
[05:02:47] <Saiba>	 not 5 million ;)
[05:02:53] <Saiba>	 don't think so
[05:02:58] <Saiba>	 500 is the GUI limit ;)
[05:02:59] <jeremyb>	 although that might not apply to this particular query
[05:03:10] <Saiba>	 you can get more by URL manipulation
[05:03:21] <Saiba>	 nearly sure
[05:03:38] <jeremyb>	 500 is the limit for the api across the board unless you have apihighlimit(s?)
[05:03:45] <Saiba>	 hmm.. really?
[05:03:46] <jeremyb>	 which sysops and bots get automatically
[05:03:54] <jeremyb>	 but your link wasn't the api
[05:04:07] <Saiba>	 ah
[05:04:08] <Saiba>	 sure
[05:04:15] <jeremyb>	 anyway, i got the same error as a lowly autoconfirmed
[05:05:33] <Saiba>	 in any way:  that error isn't a graceful deny ;)
[05:06:24] <jeremyb>	 sure, it's trying to actually fulfill and it dies
[05:06:30] <Saiba>	 yup
[05:06:52] <jeremyb>	 probably it's doing too much in memory and should be instead flushing to the client periodically
[07:29:29] <nagios-wm>	 PROBLEM - Puppet freshness on virt1 is CRITICAL: Puppet has not run in the last 10 hours  
[10:06:24] <nagios-wm>	 PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 445028 MB (3% inode=99%):  
[10:08:24] <nagios-wm>	 PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 436164 MB (3% inode=99%):  
[10:09:34] <nagios-wm>	 PROBLEM - Puppet freshness on sodium is CRITICAL: Puppet has not run in the last 10 hours  
[10:37:04] <nagios-wm>	 RECOVERY - MySQL slave status on es1004 is OK: OK:  
[16:30:30] <DarkoNeko>	 zz
[16:54:29] <gerrit-wm>	 New review: Petrb; "(no comment)" [operations/puppet] (production) C: 1;  - https://gerrit.wikimedia.org/r/1918
[17:39:00] <nagios-wm>	 PROBLEM - Puppet freshness on virt1 is CRITICAL: Puppet has not run in the last 10 hours  
[19:06:39] <alot_of_mike>	 I'm trying to reset the password for an account on testwiki. The email is sent OK, and I can log in with the temp password and set a new password OK, but when I try to use that new password, I'm told "Login Error for ACP1 Incorrect password entered. Please try again." What is ACP1, and how can I log into my account/
[19:09:11] <petan>	 alot_of_mike: hi, does it happen on other sites too? or only on test wiki
[19:09:27] <Reedy>	 Isn't that the account creation project?
[19:09:42] <jeremyb>	 yes, sounds like acct creation project
[19:09:50] <alot_of_mike>	 I'm testing on meta now - the message is slightly different (doesn't mention ACP1) but the effect is the same - I cannot log in with the new password
[19:10:05] <petan>	 alot_of_mike: ok, just try to reset pw on meta
[19:10:07] <Reedy>	 what username?
[19:10:15] <alot_of_mike>	 Reedy: Perlwikibot testing
[19:10:48] <alot_of_mike>	 petan: I did. I still get a password error after changing the password from the temp one, it just doesn't mention ACP1
[19:11:04] <petan>	 ok, let's wait what Reed y find out
[19:12:13] <alot_of_mike>	 If it makes a difference, the account is locked via centralauth. It used to keep users from logging in, but hasn't done that in years, unless behaviour was changed since I've been gone.
[19:14:26] <Reedy>	 It's blocked indefinitely on enwiki/testwiki
[19:15:04] <alot_of_mike>	 on testwiki too O.o
[19:15:35] <alot_of_mike>	 Reedy: Should that affect logging in? My recollection is that it should not.
[19:15:50] <Reedy>	 I've no idea
[19:15:59] <alot_of_mike>	 oh, I did that
[19:16:00] <Reedy>	 In theory it shouldn't
[19:16:02] <alot_of_mike>	 I wonder why
[19:17:00] * alot_of_mike  figured it out

[19:17:09] <alot_of_mike>	 we don't need to edit for the tests that use this account :)
[19:17:16] <alot_of_mike>	 But we do need to log in... :\
[19:18:28] <Reedy>	 I can try unblocking/unlocking it to see if that helps
[19:18:50] <Reedy>	 not sure why it needs to be locked and blocked from editing
[19:18:54] <alot_of_mike>	 I wouldn't bother, I'm quite certain it has no effect on logging in.
[19:19:16] <alot_of_mike>	 Reedy: I didn't have sysop on all wikis, and the account shouldn't be permitted to edit, so I had pathoschild lock it.
[19:22:55] <alot_of_mike>	 huh, apparently being locked /does/ keep you from logging in now. I guess it'd be less evil if that were explained when your login fails
[19:23:15] <alot_of_mike>	 which is why we asked the devs to change it way back long ago -_-
[20:07:01] <Ryan_Lane>	 !log stopping pdns on virt1
[20:07:03] <morebots>	 Logged the message, Master
[20:19:23] <nagios-wm>	 PROBLEM - Puppet freshness on sodium is CRITICAL: Puppet has not run in the last 10 hours  
[20:29:48] <Ryan_Lane>	 !log stopping opendj on virt1
[20:29:50] <morebots>	 Logged the message, Master
[21:10:20] <nagios-wm>	 PROBLEM - LDAP on virt1 is CRITICAL: Connection refused  
[21:12:50] <nagios-wm>	 PROBLEM - LDAPS on virt1 is CRITICAL: Connection refused  
[23:23:17] <Ryulong>	 how come I can't edit from latest diffs anymore
[23:23:25] <Ryulong>	 *edit sections
[23:37:06] <Ryan_Lane>	 !log shutting down virt1
[23:37:07] <morebots>	 Logged the message, Master
[23:41:42] <nagios-wm>	 PROBLEM - Host virt1 is DOWN: PING CRITICAL - Packet loss = 100%  
[23:50:02] <nagios-wm>	 PROBLEM - Host virt1.wikimedia.org is DOWN: PING CRITICAL - Packet loss = 100%