[00:01:43] even most of the core ones suck [00:02:51] https://strategy.wikimedia.org/wiki/Special:Log/thanks [00:04:26] Hello Elsie. [00:07:50] Hi. [00:21:27] Krenair: use the strapping skin [00:21:35] Krenair: then use bootswatch with one of the dark themes [00:26:11] Ryan_Lane, bootswatch? [00:27:50] Krenair: http://bootswatch.com/ [00:28:06] cyborg and slate are pretty dark [07:02:33] thanks ops, much prettier now :) https://gdash.wikimedia.org/dashboards/reqerror/ [07:11:30] Elsie: I see people are not ungrateful. [10:04:04] akosiaris: any reason for this? http://etherpad.wikimedia.org/robots.txt [10:12:31] Nemo_bis: it is the default example robots.txt as proposed by the etherpad devs. We have seen no reason to change it up to now. [10:24:05] akosiaris: well, it makes the two (well known) worst problems with etherpad worse: 1) not indexed by search engines, 2) not archived by wayback machine [10:24:40] at least whitelisting the wayback machine would be a good idea, so that there is less pressure/stress about losing everything contained in it [10:37:59] for 2) I must say etherpad.wikimedia.org never was intended for permanent storage. Preservation of a pad is up to the people interested in preserving that pad in another format. The software is well known to corrupt pads (hopefully the latest issues are resolved with 1.3.0 but we never know when others might show up) and restoring a pad from database backups is neigh to impossible. For 1) we have no idea how etherpad will react to being sc [10:46:33] akosiaris: that's what I'm saying :) if we don't plan to make archives, let's let others do so [10:47:47] we are on the same page. Let's see than if a) it can be done and etherpad won't start having horrible hiccups whenever a spider is crawling it, b) if people indeed want that [10:48:13] akosiaris: so for (b) I should file a bug? [10:49:29] I 'd say an email in wikitech-l might have some more visibility, but file a bug in any case so we have some accounting [10:52:07] why would we have search engines index pads [10:52:32] it'd have to be a really smart search engine too [10:52:55] it's all javascript [10:57:02] google reflects javascript [11:01:14] I don't see any other pads being indexed at google [11:01:31] if you find any, we can reevaluate, but until then frankly i see no point [11:05:33] sounds like Russel's teapot [11:06:44] https://bugzilla.wikimedia.org/show_bug.cgi?id=56893 [11:43:53] brion: can you help me out with https://noc.wikimedia.org/cgi-bin/report.py ? i can't find a good way to see what's going on on wikidata.org [11:44:37] we have a pretty serious performance issue with page rendering [11:44:43] YuviPanda: ---^ [11:45:01] hmm [11:45:19] brion, YuviPanda: https://www.wikidata.org/wiki/Q38 seems to time out [11:45:53] I... have no idea what that means. ugh. sorry! [11:46:02] no MaxSem [11:46:04] interesting... i see blank page output at https://www.wikidata.org/wiki/Q38 [11:46:07] ori-l is asleep, supposedly [11:46:14] seen https://wikitech.wikimedia.org/wiki/Server_admin_log#November_10 ? [11:46:22] usually a timeout or crash should show us an error from the proxy layer [11:46:23] https://wikitech.wikimedia.org/wiki/Server_admin_log#November_10 [11:46:28] * Fatal error: Allowed memory size of 201326592 bytes exhausted (tried to allocate 72 bytes) at /usr/local/apache/common-local/php-1.23wmf2/extensions/WikibaseDataModel/DataModel/Entity/Entity.php on line 130 (47 such fatals) [11:47:00] o_O [11:47:22] brion: oh, it's a *memory* issue? wtf?? [11:47:30] how do i profile that... [11:47:47] still a memory issue should push error to output shouldn't it? [11:47:51] or did we change our error reporting... [11:48:27] https://www.wikidata.org/wiki/Q38?action=raw <- ok this at least returns something (an application-level error for action=raw handler) [11:49:28] brion: that's by design. action=raw is expected to return wikitext. there is none [11:49:35] yeah [11:49:50] just confirming that the site doesn't explode purely by virtue of loading [11:49:57] https://www.wikidata.org/w/index.php?title=Q38&oldid=84393964 <- old rev shows [11:50:06] brion: you can get the content via the api or export, no problem [11:50:17] yea, i tried several old revisions. [11:50:28] as you get closer to current, more of them time out [11:50:32] err, error out [11:50:53] there doesn't seem to be a sharp boundary, though - some of them work "sometimes". [11:50:55] yeah i'm seeing the same [11:51:32] i wish we had a real in-production debugger that let us set breakpoints and step through [11:52:03] i find i'm really appreciating non-shitty debuggers in front-end and mobile app development, now i miss it when doing server-side :D [12:02:22] brion: i hear ya [12:04:39] brion: a stack trace of the out of memory error would already help [12:05:00] srsly [12:05:03] * brion hmms [12:05:08] xdebug can do that [12:05:15] but iirc it slows things down which is why it's not generally deployed [12:22:33] why are we having repeated problems with saving larger pages at the moment? [12:22:52] is it something that may be related to the West Coast servers? [12:22:55] sDrewth: on wikidata? or elsewhere? [12:23:12] ommons, enWS, enWP [12:23:23] not WD, they are little [12:23:32] little edits [12:24:15] plus other wikis [12:25:41] sDrewth: don't you get a Wikimedia error? or can you do a ping or something to find out what servers you're hitting [12:26:03] though akoopal has had the same problem yesterday and he's in another continent :) [12:26:21] get an error [12:26:48] brion: i think i found the issue. we are loading the content of all the *referenced* items as well as the item itself. [12:26:53] seems to be larger pages (60k - 80k) [12:26:54] will try to fix that now. [12:27:14] aho [12:27:19] that'll do ya [12:29:42] * akoopal looks up [12:43:39] anyone else seeing slow downs in europe ? [12:44:00] i had it last night as well, and now again. some bits requests take multiple seconds [12:59:00] [[Tech]]; 112.163.168.172; /* Batch/offline editing of Wikimedia contents */ new section; https://meta.wikimedia.org/w/index.php?diff=6328152&oldid=6307065&rcid=4660161 [13:31:52] Could someone review my bugs? https://gerrit.wikimedia.org/r/#/c/94598/ and https://gerrit.wikimedia.org/r/#/c/94609/ [13:31:55] Thanks in advance! [13:33:34] Hi. I just received an email of someone (no wikimedian) who posted an email to a public mailing list thinking it was a private address, and he included private contact info. Who should he contact to get the mail removed from the archive? [13:33:54] (and yes, I know the admins hate this kind of stuff) [13:35:23] hmm [13:37:21] effeietsanders, email to ops-requests@wikimedia.org [13:37:31] that'll create you a RT ticket [13:37:52] MaxSem: can I forward the whole email, including the private data to there? [13:38:00] (he included his phone number again...) [13:38:04] yes [13:38:10] or will that become a public ticket? :) [13:38:19] effeietsanders: i think you can also email -owner@lists.wikimedia.org [13:38:28] effeietsanders: RT tickets are "semi-private" [13:38:32] MatmaRex, it's not a ML [13:38:38] MaxSem: I'm a listowner [13:38:50] but I don't think I can do anything from my end [13:39:19] MatmaRex, RT is not public enough for that purpose [13:39:19] ehm, that was to MatmaRex [13:39:38] ah, okay [13:42:10] thanks MaxSem , I forwarded it [14:49:02] Could anyone check a page for me, I think got a bug. [14:49:20] https://en.wikipedia.org/wiki/Wikipedia_talk:Articles_for_creation/pierre_d_avoine half the text is appearing, click on edit.. [14:49:21] "check a page" [14:49:59] It's a page text issue [14:50:07] Numerous unclosed tags [14:50:17] Okay [14:50:19] Excellent [14:50:24] 5 opening, non closing [14:50:33] Ah right, I missed that. [14:50:59] Thanks Reedy [19:23:38] hi [19:23:45] please see https://meta.wikimedia.org/wiki/Meta:Babel#Problem_with_editing_-_edit_token [19:23:53] and https://meta.wikimedia.org/wiki/Wikimedia_Forum#Error:_unable_to_save_edit_while_logged_in [19:23:57] What is happening? [19:24:03] (need to go now, sorry) [19:24:27] gah [19:24:35] well, that came up before [19:24:45] with the same user (kubof) [19:24:55] i tried troubleshooting with him and then he disappeared IIRC [19:25:13] don't see him in #wikipedia-en [19:35:34] somethng wrong with Wiki? [19:35:51] tell me more? [19:35:57] https://simple.wikipedia.org/wiki/Iskander_Mirza [19:36:06] oh never mind [19:36:09] * Romaine gets pages without sidebar, hearder, etc [19:36:12] only text [19:36:23] WFM [19:36:26] Check your JS console? [19:36:38] I have lags [19:36:53] Reedy: watchmouse was saying perf issues with bits [19:37:05] Reedy: yeah, i too had lost css a few secs ago [19:37:07] And DangerMouse says it's broken [19:37:09] yeah, I just got a page with no CSS [19:37:09] i assume that's what "static assets" means [19:37:10] 503's on bits downloads [19:37:12] now it's back [19:37:19] (refreshed a few times) [19:37:35] Hm laggy [19:37:36] specifically on RL bits assets [19:37:41] https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&s=by+name&c=Bits%2520application%2520servers%2520eqiad&tab=m&vn=&hide-hf=false [19:37:46] Bump on CPU, drop in traffic [19:37:51] ok [19:37:56] Thanks [19:39:54] cpu seems back to normal levels. That was one heck of a spike [19:40:26] I wonder if changing all smaller projects in one go isn't the best idea [19:40:33] {{resolved fixed}} Thanks. [19:40:44] gone again... [19:40:45] 588 wikis in one go [19:41:02] this time on mw.org [19:41:18] [Error] Failed to load resource: the server responded with a status of 503 (Service Unavailable) (load.php, line 0) [19:41:48] on the bits RL for all skins and gadget resources [19:41:52] !log Switching 588 wikis to 1.23wmf3 in one go seems to have upset the bits appserver pool [19:42:04] hah [19:42:08] Logged the message, Master [19:42:37] Still slightly concerned about the drop in traffic [19:42:43] It's not obviously recovering [19:43:07] load avg jumped up again [19:43:19] yikes [19:43:22] the caches don't care [19:43:31] and network is back a bit too [19:46:35] doesn't seem to be reflected in https://graphite.wikimedia.org/render/?title=HTTP%205xx%20Responses%20-1hours&from=-1hours&width=1024&height=500&until=now&areaMode=none&hideLegend=false&lineWidth=2&lineMode=connected&target=color%28cactiStyle%28alias%28reqstats.500,%22500%20resp/min%22%29%29,%22red%22%29&target=color%28cactiStyle%28alias%28reqstats.5xx,%225xx%20resp/min%22%29%29,%22blue%22%29 [19:46:59] jeremyb: the error or the recovery? :) [19:47:06] Nemo_bis: either [19:47:41] hmm, interesting. I remember that over the past couple of weeks we had a few times of unstyled content reports before. Perhaps we were already nearing the edge on previous deploys, but now it is much more visible because we truly went over the edge ? [19:48:58] I've been having that for a while [19:49:19] but it's intermittent with local problems of my browser (?!) [19:49:34] we need more metrics (or more visible metrics) on that stuff.... it happens too often. [19:50:57] yeah, something like a big red speedmeter 95 % of 10 Gb link, 98 % ... hey you'll get your license revoked [19:59:20] Reedy: there is also a bit of a spike in pmtpa application servers traffic now [19:59:32] but that might be normal i guess [19:59:33] pmtpa? [19:59:36] They should be dead [20:00:14] In.. [20:00:24] I hadn't synced any large files.. [20:00:35] well their served traffic went from 3 to 8Mb just now on ganglia [20:00:48] MB/s [20:01:20] maybe someone changed DNS overrides, who knows [20:02:51] umm [20:03:04] ? [20:03:42] gettin errors [20:03:45] "Error: 503, Service Unavailable" [20:04:16] yes, it should getting better. [20:04:37] you might have to bypass your browser cache https://en.wikipedia.org/wiki/Wikipedia:BYPASS [20:05:27] :) [20:09:35] OlEnglish: 503's ? what location are you in ? [20:09:44] and what's the result of "host en.wikipedia.org" [20:09:47] vancouver, bc [20:10:12] seems to be working now [20:10:26] after i bypassed cache [20:12:10] speaking of weird stuff. analytics cluster just went from 0MB/s to 500MB/s :D [20:12:13] LeslieCarr: well, https://gdash.wikimedia.org/dashboards/reqerror/ doesn't look too food [20:12:24] thedj: I think it sometimes has peaks of several PB, fwiw [20:12:42] to reports are probably not so trustable [20:12:55] Nemo_bis: ok, good to know :D [20:13:05] Nemo_bis: but the reqerrors started too long ago [20:13:07] i think [20:13:58] jeremyb: they were fixed [20:14:10] i don't get it [20:19:29] jeremyb: it was fixed, now it's broken again (or differently broken) [20:19:57] for some definition of "it", "fixed" and "again" :) [20:20:13] useful! [20:20:22] yeah, am I not always [20:24:59] Nemo_bis just likes logging [20:27:16] * thedj installing ie8 xp Virtual box instance [20:27:37] * Nemo_bis greets the martyr and offers whip for flagellation [20:27:47] see if i can help those poor souls fix that issue with broken en.wp pages [20:29:01] hoo: how are your patches doing ? I hope i will have the time to take a look again this week. [20:29:49] thedj: Mostly working on non-patchy stuff now... but I've got a new patch for mobile frontend [20:30:27] Mobile team disagrees with me a bit over there (whether we need keyboard navigation in mobile) [20:30:53] of course :D [20:32:04] thedj: Would be great, if you could comment on https://gerrit.wikimedia.org/r/93490 a bit... I really want to get that don without having to write up a bazillion patch sets :P [20:32:37] ok, starred it. [22:02:25] [[Tech]]; MF-Warburg; /* Batch/offline editing of Wikimedia contents */ unsigned; https://meta.wikimedia.org/w/index.php?diff=6333521&oldid=6328152&rcid=4660647 [22:39:06] wgContentLanguage is broken on the incubator. it's showing "en" for everything. anyone know what happened? [22:42:51] wgContentLanguage used to be the language of the test project [22:50:02] !summon SPQRobin [22:51:22] Reedy: fatals seem still and constantly higher, don't they https://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&title=MediaWiki+errors&vl=errors+%2F+sec&x=0.5&n=&hreg[]=vanadium.eqiad.wmnet&mreg[]=fatal|exception>ype=stack&glegend=show&aggregate=1&embed=1 [22:52:13] 1 Fatal error: Access to undeclared static property: <::$instance in on line 340 [22:52:21] since 19.30 UTC or so AKA deploy [22:52:45] and that's in production? O_o [22:53:22] Nov 11 22:48:18 10.64.0.60 apache2[30045]: PHP Fatal error: Access to undeclared static property: <::$instance in on line 340 [22:53:57] [11-Nov-2013 22:20:25] Fatal error: Access to undeclared static property: ::$instance at on line 340 [22:53:57] [11-Nov-2013 22:48:18] Fatal error: Access to undeclared static property: <::$instance at on line 340 [22:54:55] #1 /usr/local/apache/common-local/php-1.23wmf3/includes/filebackend/FSFile.php(114): <::() [22:55:22] $magic = MimeMagic::singleton(); [22:55:54] public static function &singleton() { [22:55:54] if ( self::$instance === null ) { [22:55:54] self::$instance = new MimeMagic; [22:55:54] } [22:55:54] return self::$instance; [22:55:54] } [22:55:58] 340 being the second line [22:57:08] https://git.wikimedia.org/history/mediawiki%2Fcore/454b21318deced7e26fb7b1e0efa9b2e45800034/includes%2Ffilebackend ? [22:59:59] FSFile code hasn't changed since December 2011 [23:00:55] 340 [23:02:25] Reedy: is that call coming from subclass? [23:02:44] No.. [23:02:51] (doesn't self does some weird shit in subclasses?) [23:02:56] hm. welp [23:03:11] The code hasn't changed in ages [23:03:28] I see some weirdness in the stack traces though [23:03:29] #16 ▒(0): IndexPager->getBody() [23:03:29] #17 /usr/local/apache/common-local/php-1.23wmf3/includes/SpecialPage.php(631): [23:03:29] ->▒▒▒(NULL) [23:03:42] did you update php? :P [23:03:42] #16 독일 마르크(16): IndexPager->getBody() [23:03:42] #17 /usr/local/apache/common-local/php-1.23wmf3/includes/SpecialPage.php(631): [23:03:42] BGO->DDM(NULL) [23:04:32] uh [23:04:45] let's hope it's not related to the language converter bug [23:07:34] TimStarling: Did you fix that weird "BST" time error? [23:07:42] [11-Nov-2013 19:46:22] Fatal error: Allowed memory size of 220200960 bytes exhausted (tried to allocate 196083941 bytes) at /usr/local/apache/common-local/php-1.23wmf3/includes/GlobalFunctions.php on line 2928 [23:07:42] [11-Nov-2013 19:46:38] Fatal error: Maximum execution time of 180 seconds exceeded at /usr/local/apache/common-local/php-1.23wmf2/includes/diff/DifferenceEngine.php on line 794 [23:07:42] [11-Nov-2013 21:47:12] Fatal error: Call to undefined method Wikibase\RepoLinker::repoItemUrl() at /usr/local/apache/common-local/php-1.23wmf3/extensions/Wikibase/client/WikibaseClient.hooks.php on line 749 [23:07:42] [11-Nov-2013 19:47:13] Fatal error: Maximum execution time of 180 seconds exceeded at /usr/local/apache/common-local/php-1.23wmf2/includes/diff/DifferenceEngine.php on line 794 [23:07:42] [11-Nov-2013 19:47:34] Fatal error: Allowed memory size of 220200960 bytes exhausted (tried to allocate 190841061 bytes) at /usr/local/apache/common-local/php-1.23wmf3/includes/GlobalFunctions.php on line 2928 [23:07:58] Nemo_bis: Most of the errors look to be OOM or execution time exceded [23:08:19] I never isolated it properly [23:08:22] [11-Nov-2013 20:30:54] Fatal error: Call to undefined method Wikibase\RepoLinker::repoItemUrl() at /usr/local/apache/common-local/php-1.23wmf3/extensions/Wikibase/client/WikibaseClient.hooks.php on line 749 [23:08:22] [11-Nov-2013 19:31:20] Fatal error: Maximum execution time of 180 seconds exceeded at /usr/local/apache/common-local/php-1.23wmf2/includes/parser/Parser.php on line 2389 [23:08:36] I set date.timezone, but I couldn't see any way that that could actually fix the problem [23:08:45] I wonder why one of them is showing 2 hours out [23:09:10] what log file is this? [23:09:12] Nemo_bis: we should filter OOM/timeouts to their own fatal type [23:09:16] fatal.log? [23:09:19] /a/mw-log/fatal.log on fluorine [23:09:20] yeah [23:09:30] hmmm [23:10:35] Nemo_bis: Nothing that worries me in the fatal log. Other than if 1.23wmf3 was using a lot more memory... [23:10:40] this is amusing https://ishmael.wikimedia.org/more.php?host=db1021&hours=24&checksum=9929373781665888807# [23:11:10] lol [23:11:22] argh [23:11:29] two other top queries on db1021 (chosen at random) are also wikibasebasebasebase [23:11:39] half a dozen extensions have date_default_timezone_set() calls [23:11:50] the logpager query otoh was 1.1% of queries, 72% of time, when I looked at some point today or yesterday [23:11:56] HAH! [23:12:12] ok, actually 2 extensions [23:12:21] RSS and SemanticForms [23:13:36] If this was Wikitech... we could possibly blame SemanticForms [23:13:46] But that means it's possibly RSS [23:15:48] anybody willing to handle an Education_Program extension deployment on el.wp? The reporter wants to use it in two days for some students... https://bugzilla.wikimedia.org/show_bug.cgi?id=56771 [23:15:58] protected $date = "Y-m-d H:i:s"; [23:16:41] actually there is one other date_default_timezone_set() call [23:16:55] date_default_timezone_set( $wgLocaltimezone ); [23:17:01] in Setup.php [23:17:13] and you know what $wgLocaltimezone is? [23:20:56] I'll just use strftime() [23:21:43] Nemo_bis: What's the m unit supposed to be? [23:22:56] milli [23:23:12] Meaning it's actually showing very few errors [23:23:34] Max 1 per second [23:23:50] and 0.3/s for fatals [23:25:03] why is it not using ISO 8601? [23:25:32] is that for consistency with some other log?