[00:16:35] !log aaron synchronized php-1.20wmf2/includes/User.php [00:16:38] Logged the message, Master [00:27:10] !log aaron synchronized php-1.20wmf2/includes/User.php [00:27:13] Logged the message, Master [00:27:43] !log aaron synchronized php-1.20wmf2/includes/User.php 'removed some temp logging' [00:27:47] Logged the message, Master [00:29:30] !log aaron synchronized php-1.20wmf2/includes/User.php 'logging hack tweaks.' [00:29:33] Logged the message, Master [00:29:48] !log aaron synchronized php-1.20wmf2/includes/User.php 'removed logging hack tweaks.' [00:29:52] Logged the message, Master [00:40:38] gah, didn't actually remove [00:40:45] !log aaron synchronized php-1.20wmf2/includes/User.php [00:40:48] Logged the message, Master [00:48:48] !log aaron synchronized php-1.20wmf2/includes/User.php [00:48:51] Logged the message, Master [00:49:13] !log aaron synchronized php-1.20wmf2/includes/User.php [00:49:16] Logged the message, Master [00:51:56] Hello Houston, we have a problem ! [00:53:13] Is there any system administrator here ? [00:54:48] Nnemo: What's up? [00:56:20] !log tstarling synchronized php-1.20wmf2/includes/User.php 'header log' [00:56:24] Logged the message, Master [01:04:23] There is a problem on a server, it must be two days now [01:04:56] Nnemo: Please describe the problem and where you're seeing it. [01:05:04] On which wiki, at which URLs, etc. [01:05:38] Try to edit the last section of the article en.wikipedia.org/wiki/Montreal [01:05:45] make a change, and save [01:05:53] this gives an error : [01:05:54] Request: POST http://en.wikipedia.org/w/index.php?title=Montreal&action=submit, from 91.198.174.41 via sq78.wikimedia.org (squid/2.7.STABLE9) to 10.2.1.1 (10.2.1.1) Error: ERR_ZERO_SIZE_OBJECT, errno [No Error] at Fri, 11 May 2012 00:52:56 GMT [01:06:29] Can someone repair that please ? [01:07:13] RoanKattouw: ^^^^^^ :-) [01:07:23] I'm working on that [01:07:29] cool [01:07:42] Nnemo: I discovered the same bug earlier this evening. ;-) [01:07:49] With a different page on en.wikipedia.org. [01:11:02] OK, I go in quest for food, and in the meantime please tell me when it's fixed, thanks ! [01:24:29] anyone experienced with creating namespaces? [01:24:47] Romaine: You have to file a shell request for that [01:25:11] I want first to know what the conditions of it are [01:25:35] Community consensus [01:25:47] No collision with existing namespaces / aliases / interwiki prefixes [01:25:48] I remember from earlier creations that the wanted namespace-prefix must not be used before the creation, is that still the case? [01:25:57] Well it's better if it's not [01:26:07] But it's not needed, there's a script to fix that specific case [01:26:32] Oh, it was just waiting for the client to hang up [01:26:39] Got a segfault now, let's look at the backtrace [01:26:50] First 40 frames: #38 0x00007ffff7b53598 in ?? () from /lib/libpcre.so.3 [01:26:57] Of course :S [01:27:44] Looks like infinite recursion in pcre [01:27:49] Got 7600 frames already [01:31:24] Surprise! [01:31:26] #18179 0x00007ffff576f1a3 in zif_preg_replace [01:36:45] Aha, I have a backtrace [01:36:48] It's PageTriage [01:42:44] $text = preg_replace( '/\{\{[^\{]((?!\{\{).)*?\}\}/is', '', $text ); [01:42:50] Let's see what that regex does [01:44:22] OK so that strips template invocations [01:44:47] Most likely the problematic part is that it does a negative lookahead for '{{' , the ((?!\{\{).)*? part [01:46:02] WTF [01:46:06] This snippet function is evil [01:46:13] It strips templates using a regex that makes PCRE segfault [01:46:38] Then it parses it using the message parser [01:46:46] Then it truncates to 150 characters [01:48:10] PCRE is not meant to segfault anymore [01:48:54] now that we have pcre.recursion_limit [01:49:13] Well it sttill does :S [01:49:19] ah, it's set to 100k [01:49:22] lol [01:49:34] (gdb) bt -1 [01:49:36] #18206 0x00007ffff7fc1350 in main (argc=2, argv=0x7fffffffe778) at /build/buildd/apache2-2.2.14/server/main.c:742 [01:49:38] that's the thing to fix then [01:49:40] So 100k is not enough :) [01:49:52] we expose PCRE regexes to users in various places [01:50:12] e.g. AbuseFilter [01:50:20] so there shouldn't be any way to make them segfault [01:50:23] right [01:50:33] * RoanKattouw stops trying to 'fix' the PageTriage code [01:50:39] Ack, did our crawler start causing issues when we switched it back to using its old login? [01:50:48] Where is pcre.recursion_limit and how do we set it to a lower value? [01:51:03] shenl_: The segfault stuff we're discussing is not related to that [01:51:35] RoanKattouw: I'd hope not, but we got an email from Patrick asking about it, so I'm checking if we were causing separate issues [01:51:47] it's in php.ini [01:51:59] Oh, which is in puppet [01:52:10] it's 100000 by default [01:52:48] so it's probably not set in php.ini at all [01:54:08] Yeah, I got it [01:54:15] * RoanKattouw wonders why php.ini.appserver has DOS line endings [01:55:01] Is 1k a reasonable value? [01:55:09] yes, I think so [01:55:19] OK [01:58:14] !log catrope synchronized php-1.20wmf2/extensions/UserDailyContribs/UserDailyContribs.hooks.php 'Deploy 3c45831ffe1817f3dc18f06644db46b1b74173e7' [01:58:21] Logged the message, Master [02:12:21] RoanKattouw: how did debugging go? [02:12:36] I found the segfault [02:12:54] It was infinite recursion in PCRE, of course [02:13:05] Triggered by an evil regex in PageTriage [02:13:26] there is no stack depth limit? [02:13:31] So Tim and I have put a change in puppet that actually sets pcre.recursion_limit , it wasn't set before [02:13:38] Well, it was set to the default but that's 100k [02:13:46] And my stack trace was 'only' 18k frames deep [02:15:21] TimStarling: is there a simple way to make CA more resistant against G fetcher scenarious? [02:16:08] Aaron|home: CA? [02:16:13] CentralAuth [02:16:25] ah did we end up accidentally killing CA? [02:16:44] shenl: I'm just writing an email about that [02:17:40] sent [02:19:14] got it, sorry for the trouble, I'm rolling back that change now... [02:19:54] Aaron|home: I don't remember the reason for that invalidateCache() call at the moment... [02:20:09] * Aaron|home likes documentation [02:20:09] TimStarling: Was it because they weren't picking up the cookies from the Set-Cookie header? [02:20:11] but I seem to remember asking why it was there and being satisfied with the answer [02:20:18] hehe [02:20:31] RoanKattouw: yes [02:20:40] !log LocalisationUpdate completed (1.20wmf2) at Fri May 11 02:20:39 UTC 2012 [02:20:43] Logged the message, Master [02:21:04] we could just have a single session ID globally and not use $_SESSION, that would fix it [02:22:30] there's no other reference to $_SESSION['globalloggedin'] [02:22:59] so the only reason for it is to avoid User::invalidateCache() calls on every page view [02:24:09] But why does it need to invalidate the user cache there in the first place [02:24:42] well like I say, I don't recall, but I'm sure there was a good reason [02:24:47] I'm git blaming [02:24:53] Actually I think we do pick up the cookies, the problem is that we tried to take advantage of CA, so we have one site's session, instead of individual site sessions [02:25:50] which would explain the empty enwiki_session [02:26:21] So you were only picking up certain cookies and not others? [02:26:44] yeah, basically it would sign in on the first WMF site it tried to crawl, so it would have the session for that site but not others [02:26:58] because it would try to share the cookie instead of issuing more login requests [02:27:15] Right [02:27:17] Which works [02:27:46] and i guess there's a second part to that where it should have picked up cookies from the individual sites as well? [02:27:49] Except that you weren't picking up the cookies that are automatically issued when you show up with only a global cookie but no local cookie [02:27:51] Yeah [02:27:54] "* Invalidate the user's cache info, so he doesn't see already-cached anon pages" [02:28:12] by brion, April 2008 [02:29:07] yeah it definitely wasn't picking up the local cookies, I wasn't aware of that so I didn't give it that ability [02:30:14] so how do you tell if the user has anonymous page views in their browser cache without using cookies? [02:30:32] maybe it needs a throttle [02:30:48] if a user logs in, then assume they are still logged in for 5 minutes or something [02:33:03] we can use memcached [02:33:25] All right, all rolled back, we shouldn't be using the logged in user anymore [02:33:34] sorry for the trouble [02:33:57] * Aaron|home looks at user_touched uses [02:34:49] * Bsadowski1 touches user_touched [02:36:06] * Logan_ is touched by this conversation. [02:38:05] yeah, memcached is starting make sense [02:38:57] mostly user_touched is for suppressing 304 responses [02:39:20] maybe we should replace it with a cookie [02:39:29] on logout, we set a cookie which has the same function [02:39:31] * Aaron|home was looking at OutputPage [02:39:40] suppressing 304 responses before the time of logout [02:39:46] and if the user clears it, it's their fault ;) [02:39:53] we could use a cookie on login as well [02:39:59] then we wouldn't need to hit the database [02:40:08] a cache epoch cookie [02:40:33] serving both needs [02:42:19] sounds bugzilla worthy [02:43:25] Hey guys, how's it going? [02:44:35] Oooh, is this to suppress a 304 if the page hasn't been touched since the IMS, but the user's preferences have been? [02:44:53] Or the user has gone from logged out to logged in I guess [02:46:56] eh, how do we ask for an extension installed on a WMF wiki to be updated to the latest HEAD? [02:47:18] Ask nicely [02:47:22] Or wait for the next deployment [02:47:26] RoanKattouw: or prefs changes to appearance and such [02:47:34] zzz, there is a bugfix in it [02:47:47] but we are waiting for it to get updated for more than a week now [02:47:51] RoanKattouw: yes [02:48:04] going from logged out to logged in is by far the most common use [02:48:11] By default it won't get pushed till the next round of the deployments from head [02:48:13] you probably benefit from it all the time without realising it [02:48:22] Hydriz: A new deployment branch will be cut on Monday, and it'll be deployed over the course of the next two weeks [02:48:39] sounds good [02:48:46] is there a timeline available? [02:48:52] or will it get created next week [02:48:53] Yes, lemme grab it [13:00:23] Hello [13:00:39] Is there a system admin here ? [13:01:51] We have a problem ! [13:06:25] just specify the problem. depends which system [13:19:04] mutante: looks like the problem is his/her irc-connection ;) [13:21:56] heh,ok. i actually installed some upgrades on irc.wikmedia server earlier, but it shouldn't be that:) [14:48:23] this is still the problem Joan mentioned a while ago: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Problem_on_a_server [16:25:21] saper: you online? [17:05:05] Hi all! We're starting office hours with Sue Gardner in #wikimedia-office if anyone wants to join. [18:03:52] !log catrope synchronized php-1.20wmf2/extensions/UploadWizard/ 'Deploy 4b5df1a1151ac80e309d396102e5e2a8d0c27ccb' [18:03:55] Logged the message, Master [19:49:06] !log ran apache-graceful-all [19:49:10] Logged the message, Master [20:48:19] nighty~ [21:52:22] !log preilly synchronized wmf-config 'add wikimedia to zero image disable list' [21:52:25] Logged the message, Master [22:10:53] !log preilly synchronized wmf-config 'add wikimedia to zero image disable list fix header' [22:10:56] Logged the message, Master [22:13:03] gn8 folks