[00:00:37] Reedy: You shouldn't have told me that, 'cause now I want to test it... [11:05:51] hi [11:08:49] I have to do a little talk about dependable systems in university and wonder if there is documentation about how wikipedia deals with problems like disc failure or broken webservers... [11:09:29] https://wikitech.wikimedia.org/wiki/Hurricanes [11:09:41] We ALWAYS have documentation. [11:10:20] Hmm, now that I think of it, the RFP for data centre doesn't include a requirement about hurricanes, does it [11:11:27] and some docs about the architecture? how many db servers, how the replication is done, load balancing, ... [11:12:27] It's all on that wiki, follow the links [11:12:44] Summarise your discoveries on https://meta.wikimedia.org/wiki/Wikimedia_servers [11:21:15] thanks a lot! [11:27:26] one question: each "thing" in this graph https://upload.wikimedia.org/wikipedia/commons/d/d8/Wikimedia-servers-2010-12-28.svg is a server? [11:32:38] lbenedix: i think that yes, every symbol is a single physical machine [11:32:55] lbenedix: but if that's really from 2010, then it's certainly hopelessly outdated [11:33:10] are there more actual graphs? [11:33:20] no idea [11:33:37] I searched in the commons categories... no luck [11:33:38] there's some data on the wikitech wiki [11:33:48] but it's all not very well arranged for casual reading [11:34:36] thank you pointing me to wikitech wiki [11:34:55] ganglia.wikimedia.org [11:35:28] lbenedix: there's https://wikitech.wikimedia.org/wiki/Category:Servers and https://wikitech.wikimedia.org/wiki/Category:Clusters [11:36:19] but that apparently only contains the machines interesting enough to be named :P [11:37:13] I think wikitech is the right starting point to understand how wikipedia performs so well [12:23:04] if only I could access my wikitech account.. [13:46:32] Nemo_bis: who did you talk to about it? ryan? i can poke him when i see him again (but he has been on since you had that problem) [13:47:30] Nemo_bis: there's an implicit no hurricane clause because they say west of or at chicago. but then you can have tsunami [13:47:42] Nemo_bis: anyway, ops isn't so worried about hurricanes [13:48:21] Nemo_bis: they just assume that datacenters will fall off the grid once in a while and hurricanes are only one reason and we need multiple primaries with full copies of everything read to take over [13:49:30] earthquake clause :) [13:53:21] jeremyb: bug is filed [13:53:30] and I was kidding about the hurrican [13:53:31] e [13:54:07] ok, but did you talk to him about it? [13:54:43] remind me the bug #? [13:54:54] https://bugzilla.wikimedia.org/show_bug.cgi?id=56114 [13:55:07] I suppose I should just create another account [13:56:05] ewww, no [15:54:14] edsu: wikipulse seems to be broken again, do you have an idea what the reason is? [15:56:29] lbenedix: agreed [16:07:58] hm https://wikipedia-edits.herokuapp.com/ [16:49:58] Nemo_bis: is there an api where I can get the number of edits in a given time? [16:50:18] lbenedix: you could process the rc feeds [16:50:24] I know [16:50:47] I did that a while ago: http://page.mi.fu-berlin.de/benedix/pediameter/ [16:53:10] Reedy: PHP fatal error in /usr/local/apache/common-local/php-1.23wmf1/includes/PoolCounter.php line 91: [16:53:10] Class 'PoolCounter_Client' not found [16:53:14] on wikidata [16:53:16] editing my talk page. [16:53:37] but my edit went through [16:54:00] PoolCounter is enabled on wikidatawiki [16:54:40] reedy@tin:/a/common$ mwscript eval.php wikidatawiki [16:54:40] > echo class_exists( 'PoolCounter_Client' ); [16:54:40] 1 [16:54:42] wtf [16:54:59] heh. [16:55:10] There's a load of them [16:55:11] well, it went away. [16:55:13] :o [16:56:16] Numerous wikis [16:56:18] Numerous apaches [16:56:35] One Reedy [16:57:45] Scapping now [16:58:00] They're still happening [16:58:13] Numerous mw versions [16:59:13] of wtf [16:59:14] oh [16:59:21] there is a pool counter client extension [16:59:28] yeah [16:59:30] i assume wikidata has it enabled [16:59:33] for cirrus [16:59:35] everywhere does/should [16:59:40] k [16:59:44] For Lucene and article view [17:00:18] Ah [17:00:22] It's dying in Cirrus code [17:00:25] oh [17:00:30] ping ^d manybubbles [17:00:40] #0 /usr/local/apache/common-local/php-1.23wmf1/includes/PoolCounter.php(91): PoolCounter::factory() [17:00:40] #1 /usr/local/apache/common-local/php-1.23wmf1/includes/PoolCounter.php(148): PoolCounter::factory('CirrusSearch-Up...', '_elasticsearch') [17:00:40] #2 /usr/local/apache/common-local/php-1.23wmf1/includes/PoolCounter.php(290): PoolCounterWork->__construct('CirrusSearch-Up...', '_elasticsearch') [17:00:40] #3 /usr/local/apache/common-local/php-1.23wmf1/extensions/CirrusSearch/includes/CirrusSearchUpdater.php(197): PoolCounterWorkViaCallback->__construct('CirrusSearch-Up...', '_elasticsearch', Array) [17:00:47] reading [17:01:33] Reedy: which branch is this? [17:01:39] Any [17:01:49] 1.22wmf22 and 1.23wmf1 are showing it [17:02:50] maybe have to increase the pool counter limit? [17:02:54] Reedy: just so I'm clear, is everyone seeing this or just wikidata? [17:02:54] i know that was done already [17:03:16] manybubbles: i'm not sure we can tell, but very likely just wikidata [17:03:25] due to the size, etc. and it is indexing [17:03:28] <^d> Wikidata is only running as secondary. [17:03:33] <^d> Also, it's still indexing :\ [17:03:38] but still updates on pages save [17:03:43] <^d> Also, we do need to bump the maxqueue. [17:03:55] probably [17:04:18] <^d> Convenient....I wrote a patch last night for this ;-) [17:04:22] :) [17:04:25] <^d> https://gerrit.wikimedia.org/r/#/c/92808/ [17:04:28] looks familiar [17:04:47] <^d> Prolly way too high tho [17:05:09] 600 (lucene) is for all the wikis combined? [17:05:13] per host? [17:05:26] <^d> Per apache. [17:05:31] <^d> Number's too high [17:05:32] ok [17:05:40] <^d> I copied from lsearchd :p [17:05:44] yeah [17:07:32] ^d and aude: that error Reedy posted makes it look like pool counter isn't enabled in that wiki [17:07:55] <^d> Should be enabled for all wikis. [17:08:10] the line it is blowing up on is: new $class( $conf, $type, $key ); [17:08:16] looks like it is enabled everywhere [17:08:24] and would be strange for the problem to just be appearing now [17:08:35] since we have been indexing / enabled for 2 days [17:08:48] certainly odd, reay [17:08:51] yeah [17:09:00] i would try upping the limits [17:09:05] see if it helps [17:13:51] ^d: do you want to merge that commit and see if that helps? [17:16:35] i am trying to get the error myself [17:16:47] if it's really a fatal error, that's odd [17:19:16] Reedy: are you sure that the pool counter extension is being properly put everywhere? this seems really odd [17:19:25] it is in special version [17:19:44] in common settings it appears pool counter is enabled first [17:19:48] not that order should matter [17:20:29] <^d> We can merge. [17:20:36] <^d> I dunno if it'll fix things. [17:24:29] reproduced [17:25:10] Class 'PoolCounter_Client' not found [17:25:29] aude: did you get that by performing an update? [17:25:36] saving a page [17:25:46] but i saved a bunch without problems [17:25:59] it's like i'm hitting certain apache or something [17:26:16] what else is pool counter client used for besides cirrus? [17:26:21] lucene? [17:27:09] <^d> aude: Lucene, Cirrus, page views [17:27:22] hmmmm [17:27:37] if somehwo a scap went bad? [17:27:50] some files corrupt somewhere [17:27:56] * aude can't imagine and has no idea [17:30:18] I had a look at mw1201 which seems to be showing symptoms but I don't see anything wrong with the files [17:30:34] the timing seems to correspond with config changes deployed [17:30:44] yeah [17:30:53] 16:52 <+logmsgbot> !log reedy synchronized wmf-config/ [17:31:14] after https://gerrit.wikimedia.org/r/#/c/92897/ and https://gerrit.wikimedia.org/r/#/c/92898/ [17:31:25] but dont' see how they can cause this [17:32:23] well, the version specific list of extensions is new, right? [17:32:31] yeah but [17:33:08] it looks like it should skip fine, it the file doesnot exist [17:33:20] and cirrus / pool are not in the specific lists [17:35:17] hmmmm [17:35:34] if ( $wmgUserBetaFeatures ) { require_once( "$IP/extensions/BetaFeatures/BetaFeatures.php" ); [17:35:46] :( [17:35:46] * marktraceur fixes [17:35:58] i suppose that is not a problem? [17:36:24] It seems like it is, but I'll confirm [17:36:29] k [17:37:22] looks like wmgUseBetaFeatures is true for only 1.23wmf2 wikis [17:37:33] which has that stuff, i think [17:39:15] oh, looks like it stopped? [17:39:51] nope, nevermind [17:41:45] reedy has lost internet access if that matter [17:42:09] :( [17:42:33] ub [17:44:27] wait [17:44:31] aude: Where do you see that? [17:44:53] Oh [17:45:00] Was "wmgUser" your typo? [17:45:21] oh, probably [17:45:27] 'kay [17:45:32] Gave me quite the scare there :) [17:45:40] aude: Enabling BetaFeatures is intentional [17:45:42] yeah [17:45:53] i think beta, etc. looks fine to me [17:46:00] and can't see how it's related [17:46:09] Huh? [17:47:08] oh [17:47:14] i got the error on test.wikidata [17:47:22] yeah, so, errors not going away, Reedy still have connectivity or is his cell coverage gone? [17:47:32] test wikidata is still on 1.23-wmf1 [17:48:08] Yeah, AFAIK we haven't pushed wmf2 out [17:48:17] ok [17:48:24] right, but, those errors started as soon as that wmf-config change was pushed [17:48:39] actually, no, wmf2 has been pushed, just nothing should be pointing to it yet [17:48:43] i would revert them, but dont' really know the implications [17:48:55] 16:29 logmsgbot: reedy synchronized php-1.23wmf2 'Staging php-1.23wmf2' [17:49:01] they try one by one to apply them again [17:49:08] ok [17:49:26] i'd think something scapped incomplete or something, but again no idea [17:49:49] Reedy: you debuging or...? [17:50:07] * aude tries wikisource [17:50:14] aude: How do you repro on wikidata? [17:50:31] edited a page [17:50:35] though i edited a bunch fine [17:50:45] first edit on test.wikidata, boom* [17:50:56] Ah, yeah [17:51:01] just my user page [17:51:15] Quoth the server, "Edit conflict." [17:51:45] well, i can edit wikisource fine [17:52:01] oh, I just saw that hashar said Reedy lost net access [17:52:05] yeah [17:52:05] alright, so, crap [17:52:17] ^d: what do you think? [17:52:21] he is looking for wifi [17:52:25] yeah [17:52:31] iirc I've seen it on other wikis [17:52:36] it only affects editing, which is bad [17:52:42] but not good obviously [17:53:04] fortunately the wikidata community is patient with us, though we did screw up the js last night for 30 minutes [17:53:12] unrelated incident [17:53:16] if you have an issue with something and you can do it: revert :D [17:53:22] and figure it out later on :] [17:53:23] * aude doesn't really know [17:53:29] symlinks etc [17:53:34] if it is not that important, you can always wait a bit more and give you some more time [17:53:36] hahh [17:53:40] that is some docroot issue ? [17:54:01] Are there wikibase changes going out that are causing this, or is it an issue with something else? [17:54:03] not something i am qualified to know [17:54:06] marktraceur: no [17:54:45] * aude tries wikisource more [17:54:59] hashar: https://gerrit.wikimedia.org/r/#/c/92898/ ? [17:55:12] yeah those are scary [17:55:22] mutante: think we can revert that? [17:55:23] Normal edits work fine (in the user namespace) so it's not like all editing is borked [17:55:27] ^d: is going to revert those wmf-config changes [17:55:28] nop [17:55:33] marktraceur: i am editing normal pages on wikidata [17:55:34] (doesn't mean i know, didn't merge docroot change on apache) [17:55:35] user / talk [17:55:36] that is for wmf1.23wmf2 deployment [17:55:39] Oh? [17:56:14] there is also https://gerrit.wikimedia.org/r/#/c/92897/ Add version specific extension-list [17:56:20] yeah [17:56:31] i think that affects only i18n update [17:56:41] unless that failed [17:56:49] and then stuff was incomplete [17:57:13] ahh fatal spam hmm [17:57:29] aude: eh, escalated to platform, there's already talk at office [17:57:38] lbenedix: i'm not sure, it worked fine for months and months ... hmm [17:57:48] lbenedix: let me take a look at the logs [17:57:54] [31-Oct-2013 18:57:38] Fatal error: Class 'PoolCounter_Client' not found at /usr/local/apache/common-local/php-1.22wmf22/includes/PoolCounter.php on line 91 [17:57:55] :( [17:58:16] though that one is in open search api query [17:58:38] ^d: manybubbles: the opensearch API queries yield a bunch of Fatal error: Class 'PoolCounter_Client' not found [17:59:03] yeah, I was just saying that on wikimedia-operations [17:59:04] might or not be related [17:59:07] mutante: ok [17:59:14] <^d> Tell me something I don't know. [17:59:21] * aude leaving a note for the community if i can save a page [17:59:21] hashar: it is the same thing, I'm pretty sure [17:59:28] I guess [17:59:32] which i can sometimes [17:59:36] aude: 11:02 <+logmsgbot> !log demon synchronized wmf-config/ 'Cluster to known good state' [17:59:46] good [17:59:49] can someone figure out when it started exactly ? [17:59:58] https://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&title=MediaWiki+errors&vl=errors+%2F+sec&n=&hreg[]=vanadium.eqiad.wmnet&mreg[]=fatal|exception>ype=stack&glegend=show&aggregate=1&embed=1 [18:00:04] started about 16:52 [18:00:09] same time as the config changes about [18:00:58] yep [18:01:01] PHP fatal error in /usr/local/apache/common-local/php-1.23wmf1/includes/PoolCounter.php line 91: [18:01:02] Class 'PoolCounter_Client' not found [18:01:26] brion: yep [18:01:38] ahh [18:01:56] this is editing on mediawiki.org [18:02:12] ah, so not specific to wikidata [18:02:16] as suspected [18:02:17] alright, chad just synced out the revert [18:02:24] let's see.... [18:02:31] same [18:02:34] yay my edit saved this time [18:03:09] brion: yeah, but it was intermittent [18:03:15] agh [18:03:17] still seeing fatals :/ [18:03:38] lbenedix: well, a restart got it going again, unfortunately the heroku log didn't have history long enough to see what the problem was [18:03:52] thx [18:04:24] http://page.mi.fu-berlin.de/benedix/wikidatameter/ is using the wikipulse-"api" right now [18:05:01] got the error again [18:05:10] but was able to save a bunch of times also [18:05:20] aude: aren't you missing the PoolCounter extension on wikidata maybe ? [18:05:30] and on mediawiki and test wikidata? [18:05:43] it's enabled everywhere and ins pecial version [18:06:30] hashar: it is coming up an cawiki too [18:06:33] its pretty crazy [18:06:43] like something just decided it hates us [18:07:06] did we up the limits? [18:07:09] for pool counter? [18:07:16] so [18:07:16] i can't see how it's related, quite [18:07:38] Cirrus has limits, yeah [18:07:42] but they aren't super high [18:07:46] but that wouldn't cause this issue [18:07:50] agree [18:08:18] csteipp: the changes https://git.wikimedia.org/commitdiff/operations%2Fmediawiki-config/049259d18ddf17294bdbcf0a222d4d874b5a3acf and https://git.wikimedia.org/commitdiff/operations%2Fmediawiki-config/ecfa294d4b158aa5c44c166fa883d8ceef7d357d [18:08:22] eek, long urls [18:08:33] https://gerrit.wikimedia.org/r/92793 and https://gerrit.wikimedia.org/r/92897 [18:08:51] i can't see what/how but then i don't have shell access to look at anything [18:09:23] cirrus has been running fine the past few days on wikidata [18:11:48] reedy should be back in ~ half an hour [18:12:05] k [18:13:20] lbenedix: if are finding yourself relying on it, it should be pretty easy to run [18:13:54] lbenedix: off of heroku, i mean :-) [18:14:23] can we just try syncing PoolCounter client again? [18:14:37] the extension you mean ? [18:14:45] yeah [18:14:57] wild guess but maybe it's corrupt somewhere [18:15:00] corrupt file [18:15:06] some apache [18:15:10] but not all [18:15:57] doing so [18:16:07] k [18:16:40] I was wondering whether it is an issue in the autoloader [18:17:05] still fataling [18:17:07] i don't think anything changed there [18:17:24] not in pool counter, not in mediawiki (as deployed) [18:19:15] argh [18:19:45] I can't reproduce from the command line with eval [18:19:45] :( [18:19:51] edsu: I think I will run a modified version on a little old netbook [18:20:50] hashar: yeah, we were trying that on a couple of the apaches to see if it was there... :/ [18:20:51] lbenedix: cool, how did you plan on modifying it? anything worth rolling back to git? [18:21:16] I only need the "api" and only for wikidata [18:21:29] hashar@tin:/a/common$ mwscript eval.php --wiki=wikidatawiki [18:21:29] > return $wgVersion; [18:21:30] 1.23wmf1 [18:21:31] > $e = new PoolCounter_Client(); [18:21:32] PHP Warning: Missing argument 1 for PoolCounter_Client::__construct(), [18:21:35] :( [18:21:44] no changes, just commenting out stuff I dont need [18:21:46] requires arguments [18:21:48] greg-g: do you have people at the office looking at it ? [18:21:52] hashar: yeah [18:21:56] chad's on it right now [18:21:58] aude: it is irrelevant, it manage to find the class :-] [18:22:05] lbenedix: oh i see [18:22:06] greg-g: good. He is smarter than me :-] [18:22:10] hah [18:22:13] Cha to the dth power. [18:22:24] hashar: right :) [18:22:34] reedy incoming [18:22:38] yay [18:22:42] * marktraceur braces for impact [18:23:19] hashar: see -operations, btw [18:23:37] lbenedix: speaking of wikidata when did denny go to google? [18:23:51] lets switch to operations so [18:23:52] Reedy: when you get back online, check in with chad, he's at the helm, as it were [18:23:53] I think 2 month ago [18:24:08] lbenedix: is he able to continue working on wikidata in his new position? [18:24:53] I'm not sure, Lydia is project manager now [18:25:41] daniel kinzler is still there? [18:25:47] yes [18:26:15] here is the team: http://wikimedia.de/wiki/Mitarbeiter#WIKIDATA [18:26:47] edsu: he's not [18:26:52] i mean not right now [18:28:02] aude: daniel? [18:28:16] edsu: i see you are asking generally [18:28:19] aude: oh you mean, denny can't work on wikidata right now? [18:28:26] thought you were asking if he was here right now [18:28:32] * edsu is asking too many questions [18:28:36] denny is a community member now :) [18:28:37] aude: oh heh, yeah [18:28:42] he can edit, etc [18:28:48] is an admin [18:28:49] mais oui :) [18:29:31] just curious to know if google is willing to support it as a project, esp given how it's role w/r/t freebase [18:30:00] which is something i guess he *is* working on [18:30:12] google spend a lot of mony for wikidata... [18:30:22] lbenedix: yup [18:30:38] lbenedix: are they still? [18:30:58] I dont think so [18:31:03] edsu: no [18:31:47] is it just wikimedia de now? [18:32:31] pretty much [18:32:52] who else? [18:32:53] details are in the annual plan for wmde [18:32:59] we got a donation from yandex [18:33:09] no money from the foundation? [18:33:26] i think it's complicated [18:33:39] money is complicated in general ;) [18:33:55] with wmde doing fundraising, etc. [18:33:59] * aude doesn't understand 100% [18:34:20] oh, wmde is fundraising independently of the foundation? [18:34:29] * lbenedix thinks so [18:34:34] that does sound complicated :) [18:35:08] i could understand why some people might not want to have servers running in the US these days [18:35:29] edsu: servers are still in the us [18:35:36] oh, ok [18:35:57] i don't think that is changing anytime soon, since there are favorable aspects still to us law for wikipedia [18:36:18] although caching outside the us is okay [18:37:39] wikipedia and the foundation profit by wikidata, so in my opinion its strange that the funding comes only from wmde [18:38:15] not only profit by it, but are somewhat dependent on it too [18:38:32] i don't pretend to understand the politics involved though [18:38:39] me neither [18:38:53] my little exposure to how the foundation works left a bad taste in my mouth [18:38:57] I think they host it [18:39:20] lbenedix: We're actually going to help get bits of it going, though I think the details of that are still getting hashed out. See https://www.mediawiki.org/wiki/Multimedia "Implement structured data on Commons and integrate it with Wikidata [18:39:24] " [18:51:03] edsu: :( [18:53:08] edsu: well, all (or many) chapters strive to do a certain portion of stuff that benefits the wider community [18:53:16] e.g. hosting wikimania in dc [18:53:40] if wmde was more needy, i'm sure wmf would give the funding :) [18:54:15] and not 100% sure of all the details anyway [19:15:04] Damn it [19:15:32] welcome back! [19:15:38] you missed the fun [19:15:57] Found a brasserie with free internet [19:15:57] Won't let me SSH out etc [19:18:29] do we get a second post-mortem today? :P [19:18:30] did my scap run/finish? [19:18:41] i don't think so [19:18:55] we don't know what happened [19:19:04] heh [19:20:34] Sigh [19:20:53] So where do things lay? [19:22:11] I've still got to drive to lier [19:22:37] everything is okay now [19:22:55] we reverted some config stuff, changed some cirrus pool config [19:23:09] and disabled the multimedia /beta extensions everywhere and the issues are gone [19:23:37] no hurry, but still need to determine exactly what the problem is and scap/re-deploy stuff [19:24:04] okay for now [19:24:05] Currently it's still the deployment window [19:24:10] yeah [19:24:20] greg-g: ^ [19:24:39] I can't VPN out, I can't SSH out, i can't Remote Desktop out [19:24:43] i think we can wait until you have better access [19:24:57]