[00:01:20] We got an image error at Commons: http://commons.wikimedia.org/wiki/Commons:Village_pump#Internal_error "[b6200044] 2012-06-26 23:56:49: Fatal exception of type MWException" That is the only case up to now. [00:01:48] not really "image error" but rather a page error ;) [00:08:13] in case that matters: "Served by srv285 in 0.142 secs" from source [00:09:26] Happens regardless of logged in / not logged in and regardless of http or https [00:09:53] (tested logged in only with https) [00:16:36] well, now at https://bugzilla.wikimedia.org/show_bug.cgi?id=37978 [00:27:44] strange [02:24:59] !log LocalisationUpdate completed (1.20wmf5) at Wed Jun 27 02:24:59 UTC 2012 [02:25:08] Logged the message, Master [02:48:51] !log LocalisationUpdate completed (1.20wmf6) at Wed Jun 27 02:48:51 UTC 2012 [06:00:48] !log tstarling synchronized wmf-config/CommonSettings.php [06:04:30] !log tstarling synchronized wmf-config/CommonSettings.php [07:40:28] domas: lol, nice blog post ;) [07:43:01] i like how you destroy their claims: "..., but as I get to choose my own benchmark I claim that MemSQL is 500x slower at single-thread durable transaction rate. Now that we established MySQL superiority (ha ha) ..." [07:43:15] [11:44:41] !log reedy synchronized php-1.20wmf6/extensions/WikimediaShopLink/ [11:50:46] Reedy! morebots is dead [11:51:00] lol [11:54:31] !log reedy synchronized php-1.20wmf6/extensions/WikimediaShopLink/ [11:56:47] !log reedy synchronized wmf-config/ 'WikimediaShopLink' [11:57:10] >:( [11:58:02] I did ask in #wikimedia-operations [11:58:04] I can't start it [12:01:03] * closedmouth cuddles morebots [12:06:24] sorry, but I was writing code, I don't follow the channel closely sometimes when doing that [12:09:54] oh, i didnt see the quit/part [12:11:19] /unignore #wikimedia-tech JOINS PARTS QUITS NICKS [12:30:48] Reedy: someone said that morebots has been renamed lessbots [12:30:52] (that was Damianz ) [12:31:19] less > more and he had left us so he was more less than more... wait that doesn't work [12:32:12] would like to use more to display bot-documentation though [12:38:54] [12:44:41] !log reedy synchronized php-1.20wmf6/extensions/WikimediaShopLink/ [12:39:03] [12:54:31] !log reedy synchronized php-1.20wmf6/extensions/WikimediaShopLink/ [12:39:07] argh [12:39:21] !log WikimediaShopLink is deployed to testwiki/test2wiki [12:39:26] Logged the message, Master [12:40:08] Reedy: when you have time, can you review my changes in mediawiki-config please ? https://gerrit.wikimedia.org/r/#/q/is:open+owner:hashar+project:operations/mediawiki-config,n,z [12:40:16] need them for 'beta' :-] [13:04:50] !log reedy Started syncing Wikimedia installation... : Rebuild message cache for WikimediaShopLink [13:04:55] Logged the message, Master [13:05:05] slllllllllllooooooooooooooowwwwwwwwwwwwwwwww [13:08:01] > MySQL’s InnoDB (I don’t assume other engines are usable) is durable by default [13:08:04] Heh. [13:09:56] > Even “SELECT MAX(id)” does crawl whole table. [13:10:00] How is that possible... [13:16:11] apparently skiplists are one way in their implementation! [13:22:17] !log installing apache2.2-bin-dbgsym on mw1 [13:22:23] Logged the message, Master [13:28:44] !log reedy Finished syncing Wikimedia installation... : Rebuild message cache for WikimediaShopLink [13:28:49] Logged the message, Master [13:29:27] 24 minutes [13:29:28] wow [13:31:36] Reedy: don't you have a pending change to raise the dsh fork limit? [13:31:49] yes [13:31:50] also Ryan proposed that we gzip the files, send them and unzip them on apaches [13:31:56] (or we could use rsync --compress) [13:32:10] but --compress will make fenari compress over and over the same files [13:33:00] fenari doesn't send the files [13:33:05] they get pulled from nfs [13:35:17] still the rsync daemon on fenari will have to compress the files for apaches [13:35:22] (if using rsync --compress) [14:10:39] having a hard time in my coworking place http://imgur.com/NlcpH [14:11:19] slacker! [14:12:10] our hosts are really awesome people [14:14:01] the venue http://www.cantine-nantes.org/index.php?option=com_content&view=article&id=6&Itemid=11 [14:14:20] sitting on the red couch right now [14:22:54] ideally we'd use 7zip rather than gzip [14:23:06] and I'm not a fan of rsync with --compress [14:25:21] 7zip is deprecated [14:25:29] lzma replaced it [14:25:33] and then xz replaced lzma [14:51:53] !log reedy synchronized php-1.20wmf6/extensions/ExtensionDistributor/ 'ED to trunk' [14:51:58] Logged the message, Master [14:56:26] we need an "history of compression utilities" on wikipedia [14:57:15] чя, [14:57:18] *xz? [14:57:23] I loved .arc http://en.wikipedia.org/wiki/ARC_(file_format) [14:58:51] hashar: you make me feel old [14:58:59] :-] [14:59:04] I mean young [14:59:19] just live with your time, don't look at the past [14:59:28] No [14:59:34] the only difference is our date of birth and there is nothing we can do about or be proud of ;-] [14:59:36] I'd rather go with living in future [14:59:40] so never "feel young" or "feel old" [14:59:48] just enjoy present (and the future) [15:00:15] vvv: well you are technically in the future already. No more USSR ;-] [15:00:23] and we have internet access !! [15:00:41] I'm not sure about USSR [15:00:54] Wikipedia is an outright communist idea [15:01:26] !log reedy synchronized php-1.20wmf6/extensions/ExtensionDistributor/ [15:01:31] Logged the message, Master [15:01:33] * vvv wonders how ignorant the world was when there was no Wikipedia [15:02:21] that is the point of our project I guess [15:02:25] free knowledge to everyone [15:02:48] I hope the next ten years will see that knowledge spread to global south [15:03:01] I know of some projects in Africa to run copy of wikipedia out of solar panels [15:03:21] (with solar/bicycle powered WIFI) [15:03:48] How to get fit and work in IT - work in africa [15:04:20] I am sure one day I will get there to get some stuff done [15:04:45] probably whenever I reach the age of 50 (when people actually understand they are going to die) [15:04:54] <^demon> Damianz: Or buy one of these: http://blog.shop-denali.com/wp-content/uploads/2011/04/treadmill-desk.jpg [15:05:12] I'll take the monitor stand off that [15:05:49] Also [15:05:53] I just read our code [15:06:03] We still store user passwords MD5-hashed [15:06:11] indeed [15:06:13] there's a bug about it [15:06:15] fix it ;) [15:06:21] !log reedy synchronized php-1.20wmf5/extensions/ExtensionDistributor/ [15:06:24] Is there? [15:06:26] Logged the message, Master [15:06:29] I just wanted to file one today [15:06:39] <^demon> Damianz: This one's better. http://www.signaturetreadmilldesks.com/IMAGES_S/lady_treadmill_desk.jpg Look how energetic she feels. [15:06:49] lawl [15:08:06] !log reedy synchronized php-1.20wmf6/extensions/ExtensionDistributor/ [15:08:11] !log ExtensionDistributor now works from git on mediawiki.org [15:08:11] Logged the message, Master [15:08:16] Logged the message, Master [15:11:48] vvv: https://bugzilla.wikimedia.org/show_bug.cgi?id=28419 [15:11:57] Reedy: thanks [15:12:42] Oh, that's a long bug [15:13:43] it's over 2 years old [15:14:15] And everyone is arguing there instead of just switching to some new algo [15:14:27] there's a patch [15:14:49] 7 months out of date now, so likely don't apply [15:16:13] Reedy: ExtensionDistributor works now for 1.19? [15:16:23] for trunk [15:16:33] 1.19 in svn, yes [15:16:44] 1.19 in svn? not git? [15:16:52] yup [15:17:02] we don't have any procedure of branching extensions [15:17:10] (in git) [15:17:17] so theres therefore no way to expose this via ED [15:17:47] But we probably have tags [15:17:56] we have nothing [15:17:58] <^demon> 1.19 was already branched in svn before the migration. Since (almost) nobody maintains the branches, I didn't bother pulling them over for extensions. [15:18:17] <^demon> If someone wants to come up with some sort of sane branching/tagging solution for extensions, that'd be awesome :) [15:18:32] https://bugzilla.wikimedia.org/show_bug.cgi?id=37946 Add support for git branches [15:18:39] Timo has started thinking a bout it [15:19:11] submodules are so painful [15:19:37] <^demon> Submodules *could* work but they're a PITA, right. [15:19:44] <^demon> Not easy for the average drive-by user to work with. [15:22:02] I can't work with the [15:22:04] them [15:22:14] still have some extensions checked out separately [15:22:25] I think we should try repo or something [15:23:16] OMG, that password hash bug is really involved [15:23:32] And the patch does something weird with the result [15:25:50] Reedy: "started thinking" reminds me http://www.youtube.com/watch?v=yR0lWICH3rY (German Coast guard Berlitz ad) [15:27:24] Ich liebe Radar! [15:30:01] :) it's actually norwegian ad they say [15:30:26] that's SO old… ;) [15:34:42] DaBPunkt: of course [15:35:50] (but I would not gurantee that it is not the truth…) [15:36:12] it's a joke but I like the accent [15:40:03] great ad [16:23:31] AFTv5 is down @ enwiki [16:23:42] PHP fatal error in /usr/local/apache/common-local/php-1.20wmf5/extensions/ArticleFeedbackv5/ArticleFeedbackv5.render.php line 494: [16:23:46] Call to a member function exists() on a non-object [16:24:40] lol [16:24:43] useful [16:25:42] code isn't same as trunk [16:25:42] ffs [16:31:10] https://gerrit.wikimedia.org/r/#/c/13060/ [16:31:16] Update to trunk [16:31:26] https://gerrit.wikimedia.org/r/#/c/13060/ [16:31:28] Reverted [16:31:30] Revert abandoned [16:31:33] * Reedy facepalms [16:33:11] is Matthias Mullie in Europe? [16:33:18] oh yeah Belgium [16:33:45] and french speaking apparently \O/ [16:34:29] Reedy: . o O (uninstall AFT?) [16:39:51] hashar, I just discovered codesniffer repo [16:40:04] \O/ [16:40:18] yet another project of me I have no motivation to concentrate on [16:40:25] that is on github isn't it? [16:40:32] that's on our git repo [16:40:37] ohhh [16:40:41] mediawiki/tools/codesniffer [16:40:43] https://github.com/hashar/MediaWiki-CodeSniffer indeed [16:40:44] that's why I found it [16:40:47] \O/ [16:41:09] the github history explains the old timestamps, though [16:41:30] I can't even remember how I used it [16:41:43] but the idea was to migrate style.php and our own linter to CodeSniffer [16:41:47] it just seems to provide a "global variables must begin with wg check" [16:41:50] which is output is supported by jenkins [16:42:00] (plus a svn_eol-style which is no longer relevant) [16:42:03] yeah :( [16:42:04] that is merely a skeleton to build upon [16:42:16] you know about check-vars.php, do you? [16:42:26] the ruleset.xml imports rules shipped by CodeReview [16:42:29] yeah [16:42:35] my idea was to replace check-vars.php [16:42:40] with a CodeSniffer rule [16:42:57] ERR> (the ruleset.xml file imports rules shipped by *CodeSniffer*) [16:43:03] so that project test a bit more entries [16:43:18] what's the output format you want? [16:43:24] seems easier to adapt check-vars.php [16:43:39] !log reedy synchronized php-1.20wmf5/extensions/ArticleFeedbackv5/ [16:43:45] Logged the message, Master [16:43:46] TB|Away: how is it now? [16:44:02] Same error [16:44:11] Platonides: example output : http://dpaste.org/aO1wX/ [16:44:22] Platonides: well CodeSniffer is probably a bit more powerful [16:45:03] Platonides: it comes with a lot of additional rules we could use. Overall it can possibly do a lot more stuff than check-vars.php [16:45:14] yeah, sure... [16:45:20] ...when you code them :) [16:45:33] or someone else submit new rules :-] [16:45:38] will be happy to merge them [16:45:44] And we're also amazingly slow [16:45:50] how is that format easy to parse?? [16:46:32] Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes. [16:46:37] Platonides: CodeSniffer provides several output formats: full, xml, check style, csv, emacs, source, summary, gitblame [16:46:47] Server kitties are being lazy again [16:46:52] which one is imported by jenkins? [16:47:06] it is starting to get very slow on frwiki as well [16:47:09] mediawikiwiki is down [16:47:20] all the Apache servers are beginning to fail [16:47:24] grrr [16:47:32] the AFT problem above? [16:47:35] Oh darn [16:47:43] or another memcache problem ? :( [16:47:51] now enwiki is down [16:47:53] Apparently [16:47:58] down [16:48:12] commons is down [16:48:16] en.wikipedia is down [16:48:21] Hi :) [16:48:23] Request: POST http://fr.wikipedia.org/w/index.php?title=Utilisateur:Superjuju10/Mod%C3%A9le/Maladresse&action=submit, from 91.198.174.52 via sq75.wikimedia.org (squid/2.7.STABLE9) to () [16:48:23] Error: ERR_CANNOT_FORWARD, errno [No Error] at Wed, 27 Jun 2012 16:48:03 GMT :( [16:48:24] panic ensues [16:48:28] from 10.64.0.138 via cp1015.eqiad.wmnet (squid/2.7.STABLE9) to () [16:48:30] Error: ERR_CANNOT_FORWARD, errno (11) Resource temporarily unavailable [16:48:35] look at #wikimedia-operations [16:48:56] wow [16:49:02] That's rather spammy [16:49:03] 240, 258, 275, 279 seem to have a sudden spike [16:49:33] I'd reboot them [16:50:04] Jasper_Deng nice channel [16:50:05] same symptoms [16:50:10] so this is how serers panic? [16:50:22] this is the biggest breakdown I've ever see [16:50:27] Platonides: Somewhat broken was far nicer :P [16:50:35] Totally need to fix the memcache issue though if it's that again [16:50:54] Jasper_Deng: not so unusual? [16:51:08] I haven't been on that channel for that long [16:51:30] On the bright side, stuff in squid is up =/ [16:52:18] hey, are there any statistics about the IPv6 part of the total Wikimedia traffic? [16:52:33] this isn't really the time to ask that [16:52:37] Macropus: ask again post outage [16:52:46] ok :) [16:57:33] commons is partly alive again for me [16:57:56] yeah, mw.org and enwiki are back up [16:58:11] Some boxes are still getting fixed but it's working to a basic level it seems. [16:59:33] fr.wikipedia restarts :) [17:00:24] Thank for repairs. Bye and good evening ! ;) [17:03:52] :'( [17:04:22] ...shhhh [17:06:34] okay, moving here? [17:06:36] moving here :)_ [17:06:50] ok, i'm rebooting srv287 [17:06:53] well, I kind of meant, the non-outage related chatter [17:06:53] so [17:06:53] !log rebooting srv287 [17:06:58] * Ryan_Lane shrugs [17:06:59] Logged the message, Mistress of the network gear. [17:06:59] there's a big danger ahead [17:07:12] we now have 3 more dead boxes, 270, 277, 287 [17:07:13] http://nagios.wikimedia.org/nagios/cgi-bin/extinfo.cgi?type=2&host=spence&service=check_all_memcacheds [17:07:13] I'll deal with AF later [17:07:19] by the rate we are powercycling them [17:07:21] <-- memcached IPs [17:07:25] soon we'll be running out of a cold cache [17:07:38] which I presume is bad bad bad [17:07:44] erm [17:07:50] paravoid: well, kind of [17:07:51] we could ulimit the apaches [17:07:55] better off to restart apache if you can get ahead of servers going non-responsive [17:08:10] did someone already take rebooting 270 277, 287? [17:08:12] indeed. sometimes it's best to just restart all apaches [17:08:22] in the meantime 211 crawled back from the grave completely [17:08:29] maplebed: did 287 [17:08:31] taking 287. see login [17:08:40] * maplebed takes 270 [17:08:41] ok [17:08:44] I wonder if some extension is leaking memory [17:09:09] Platonides: well, why would any extension need that much memory? [17:09:11] that would explain the reoccuring problems after a few days [17:09:24] vvv, a few non-freed bytes after each request? [17:09:25] if apache is eating all the memory, a restart should clear that up [17:09:26] binasher: do you think doing a dsh on the apache group of "Service apache2 restart" would be useful ? [17:09:26] hmm [17:09:27] no, the memory spikes [17:09:28] "504 Gateway Time-out" [17:09:34] PHP in vast majority of cases just resets all memory when session is over, at least I tihnk so [17:09:35] no [17:09:39] no/ [17:09:49] in one of the occasions the memory spiked from 4.3 to 6.6G in 10' [17:09:55] it's not a long-running leak [17:09:59] and stays that way? [17:10:06] no, then the box dies :) [17:10:10] swaps to death [17:10:14] I vote for swapoff -a :) [17:10:17] do those apaches handle uploads? [17:10:45] I think only some of them did [17:10:48] Platonides: IIRC, those are mostly apache + memcached, though some of them had a job runner [17:10:51] high memory usage is persisting across requests [17:11:01] vvv, upload as in Special:Upload [17:11:02] !log powercycling srv277 (had to, frozen console) [17:11:08] Logged the message, Master [17:11:20] srv238 looks fairly unhealthy [17:11:20] Reedy: could you review https://gerrit.wikimedia.org/r/#/c/13181/ [17:11:30] Reedy: eventually merge and cherry pick in wmf branch ? [17:11:31] !log powercycled srv270 [17:11:36] Logged the message, Master [17:11:38] Reedy: need to move out coworking space :-D [17:11:40] checking the metadata of a file could have all kind of errros [17:11:41] root@srv211:~# pmap 6519 | tail -1 [17:11:42] total 1061052K [17:11:45] will be back in roughly 25mins [17:11:58] binasher: what process is that? [17:12:04] indeed. it's a spike.... [17:12:05] currently left in Nagios: 277 and 287 [17:12:15] http://ganglia.wikimedia.org/latest/?r=week&cs=&ce=&m=&c=Application+servers+pmtpa&h=srv238.pmtpa.wmnet&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS [17:12:19] week looks like month view [17:12:28] vvv: it was an apache process [17:12:46] srv287 looks ok to me now... [17:13:00] remind me where the sampled squid logs are [17:13:01] the year view is interesting: http://ganglia.wikimedia.org/latest/?r=year&cs=&ce=&m=&c=Application+servers+pmtpa&h=srv238.pmtpa.wmnet&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS [17:13:02] 00007f9c15af7000 481208K rw--- [ anon ] [17:13:14] mark: ? [17:13:26] half the memory allocated anon [17:13:32] ah [17:13:51] what about 287? [17:13:53] what does it mean? a mmap() allocation? [17:14:05] paravoid: emery:/var/log/squid/sampled-1000.log [17:14:17] maplebed: thanks a lot [17:15:01] It's back up for me. [17:15:06] MEMCACHED OK - All memcacheds are online [17:15:10] !! [17:15:10] A red exclamation mark (!) in the recent changes or on your watchlist means that edit is not patrolled yet. Read more about patrolling at https://www.mediawiki.org/wiki/Help:Patrolled_edits and configuration at https://www.mediawiki.org/wiki/Manual:$wgUseRCPatrol [17:15:20] wtf [17:15:48] that's a message intended for #mediawiki [17:15:50] !help [17:15:50] http://www.mediawiki.org/wiki/Help:$1 [17:15:50] !(stalk|ignore|unstalk|unignore|list|join|part|quit) [17:15:59] memory usage is dropping on all srvs. [17:16:04] !stalk [17:16:05] !(un)stalk wiki (page|user|summary|log|all) [pattern] [17:17:39] please don't play with the bots... [17:18:02] Sorry... [17:18:49] Platonides: [anon], as far as I understand, means memory not mapped to any file, i.e. heap or stack [17:19:42] apache is still down on a few of these machines [17:19:58] !log restarting apache2 on srv258 [17:20:04] Logged the message, Mistress of the network gear. [17:20:07] they should come back after doing a sync, Leslie [17:20:16] ok [17:20:29] for detials see #wikimedia-overflow [17:20:54] \join #wikimedia-overflow [17:24:31] vvv, heap memory can come from sbrk() or an anonymous mmap() [17:25:01] in fact, if they were mapped files, it would be unlikely to cause a swap death [17:44:52] !log preilly synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess 'update for landing page' [17:44:57] Logged the message, Master [17:45:32] !log preilly synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess 'update for landing page' [17:45:38] Logged the message, Master [17:46:19] !log asher synchronized wmf-config/mc.php 'disabling wgMemCachedPersistent; lowering wgMemCachedTimeout to 2x client default from 30x default' [17:46:19] !log preilly synchronized php-1.20wmf6/extensions/ZeroRatedMobileAccess 'update for landing page' [17:46:24] Logged the message, Master [17:46:29] Logged the message, Master [18:09:42] binasher: 30X? ;) [18:10:55] Reedy: ready when you are to start the deploy [18:12:38] AaronSchulz: there was some kind of fault/outage that raised it iirc [18:18:56] AaronSchulz: yeah! it defaults to 100000us in MemcachedClient.php and was set to that in DefaultSettings.php, but was set to 3000000 a few months ago in response to timeouts [18:20:55] Reedy: ping [18:22:36] AaronSchulz: would you mind starting the deploy? I'm not sure when Reedy is going to be back [18:22:58] gah, I guess [18:23:57] let's get commons out of the way now [18:29:04] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved en(wikibooks|wikinews|wikiquote|wikisource|wikiversity|wiktionary) to 1.20wmf6 [18:29:11] Logged the message, Master [18:30:41] lol [18:31:19] So much or commons ;) [18:31:47] yeah...shows how much influence I have around here :-P [18:32:19] the home pages for everything that AaronSchulz just deployed looks fine [18:32:33] back around [18:33:15] Reedy: want to take over? commons now? [18:33:15] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved special wikis to php-1.20wmf6 [18:33:21] Logged the message, Master [18:33:28] heh [18:34:33] wait...when did commons move over? [18:34:38] http://commons.wikimedia.org/wiki/Special:Version [18:36:02] ah....nm [18:37:46] ohh ^demon reviewed wikibugs :-] [18:37:52] <^demon> Some of it :) [18:37:56] ^demon: I should probably write tests for that [18:38:15] ^demon: I know of perl tap, not sure how to make it test a whole cli script [18:38:28] <^demon> not a clue, I'm not a perl guy. [18:38:35] what is perl? [18:39:37] something uglier than LUA :-] [18:39:49] Lua is not ugly [18:39:51] and hackier than OCAML [18:40:03] It is in fact has a very beautiful syntax design [18:40:22] (yes, I have a highlight on that word) [18:40:26] vvv: my point :-] I like the look'n feel of LUA code (though I prefer ruby) [18:40:30] <3 Perl [18:40:42] PHP Notice: AggregateMessageGroup::exists[agg-WMF_Blog_Posts]: Group is empty in /usr/local/apache/common-local/php-1.20wmf6/extensions/Translate/Groups.php on line 649 [18:40:45] perl is a great toolbox to. But I am never going to build any project based on it [18:40:55] well maybe I should just rewrite wikibugs using python [18:41:01] !log synchronized payments cluster to fundraising/1.20 de0256084a [18:41:05] <^demon> hashar: That'd work too :) [18:41:06]