[00:01:25] 500 ISE https://meta.wikimedia.org/wiki/Special:GlobalAllocation [00:01:29] white screen of death [00:02:14] The hell is that page? [00:02:34] "global allocation" tab for centralnotice [00:02:43] centralnotice? huh. [00:04:26] jeremyb: ok; I'll look in a second [00:04:57] 21 00:04:31 < jeremyb> no clue how big a deal that is or when it broke [00:05:20] Krenair: i guess you got the same result? [00:05:41] yup... [00:08:17] I'm guessing it's throwing warnings again and so it dies; but I'll need to take a further look -- distracted by some queue stuff in the backend [00:09:32] well you can answer whether it's a big deal better than i can [00:09:41] don't want to steal your weekdend [00:09:46] weekend* [00:12:15] i was initially wondering if it was something about DB/indexes [00:12:21] because of the long wait for the 500 [00:16:14] possibly; but seemingly unlikely [00:18:08] ya; it's running out of memory -- too many warnings being generated somewhere [00:19:02] why are they being buffered? we're not displaying warnings to users, right? or which warnings do you mean? [00:19:29] is white screen of death normal for us on OOM? [00:19:40] ya; because it throws a fatal uncatchable exception [00:19:57] we can munge 500 in varnish if we want [00:20:01] synthg [00:20:15] synthetic* [00:20:35] in general that probably isn't a bad idea [00:21:07] ya; because it throws a fatal uncatchable exception [00:21:10] what are you talking about? [00:21:17] 21 00:01:25 < jeremyb> 500 ISE https://meta.wikimedia.org/wiki/Special:GlobalAllocation [00:21:20] 21 00:01:29 < jeremyb> white screen of death [00:21:21] 21 00:18:07 < mwalker> ya; it's running out of memory -- too many warnings being generated somewhere [00:21:25] if php runs out of memory -- we can't catch it with the last chance handler [00:21:40] mwalker: 21 00:19:01 < jeremyb> why are they being buffered? we're not displaying warnings to users, right? or which warnings do you mean? [00:21:48] if by exception you mean fatal error [00:21:53] and if by throw you mean raise [00:22:01] yes; that :) [00:22:32] and if by uncatchable you mean handled by our extension wmerrors [00:24:52] I'm not aware of a method that can handle an oom error in PHP (outside of the engine itself) -- but yes [00:25:34] and by "white screen of death" you mean a blank page? [00:25:45] yes! [00:26:08] how long does it take for the blank page to be delivered? [00:26:22] ~12 secs [00:26:40] ~15 this time [00:26:54] Content-Length: 0 [00:27:13] although, huh [00:27:16] Have you tried clicking that link TimStarling? [00:27:23] well, wmerrors.timeout is 10 [00:27:23] Server: is not apache [00:27:31] and no mention of php in headers [00:27:55] from the logs: "[21-Oct-2013 00:26:59] Fatal error: Allowed memory size of 201326592 bytes exhausted (tried to allocate 71 bytes) at /usr/local/apache/common-local/php-1.22wmf21/extensions/CentralNotice/special/SpecialGlobalAllocation.php on line 423" [00:28:14] basically there are two things that can happen when wmerrors intercepts an OOM [00:28:26] either it can write out an error message and abort() [00:28:42] or it can hit a deadlock in some C library (which is entirely possible) [00:28:44] so, i now see Server: is nginx for any normal (non-error) HTTPS req. maybe we should switch that to pass through the Server [00:28:52] in which case it waits for 10 seconds and then aborts without writing anything [00:29:33] (anyway, separate issue) [00:29:35] yeah, from HTTP I see a squid error message [00:29:41] from nginx I see nothing, which is a bug [00:30:04] it should show an error message if the connection is closed without responding [00:31:06] squid is also a 503 vs. nginx is a 500 [00:32:35] anyway, have you checked fatal.log on fluorine? [00:33:27] TimStarling: mwalker pasted something that looks like it's from there? [00:33:33] ~10 lines up [00:33:49] (actually squid is really varnish in this case) [00:33:53] yes, that is probably it [00:33:54] yep -- that's from fluorine -- I've been trying to recreate it on my lcoal [00:34:04] fatal.log is generated by wmerrors [00:34:27] PHP by itself does not have the ability to generate a backtrace of a fatal error and send it off to a UDP socket, that is our own addition [00:34:34] interesting -- where is that located? I would like to add that code to some other projects of my own [00:34:43] *the catching of fatal errors [00:34:55] mediawiki/php/wmerrors.git [00:34:59] and it's not catching [00:35:09] because it's not an exception [00:35:14] it's an error hook [00:35:20] does php.net not want that? [00:35:49] unlikely, it only works in non-threaded SAPIs [00:36:00] uhuh [00:36:02] I have considered submitting it to PECL, haven't gotten around to it yet [00:38:18] fatal error handling in PHP is somewhat inelegant [00:38:54] especially timeout handling, which involves calling longjmp() from a signal handler and then returning control to the webserver as if nothing happened [00:39:16] potentially leaving all sorts of libraries with global state with mutexes held [00:39:27] e.g. libc does this for all output functions [00:39:50] holds mutexes [00:40:10] gerrit has intentional downtime in case anyon ewas wondering. see SAL [00:40:18] anyone was* [00:40:32] TimStarling: does this module do anything to help with that? [00:41:04] *extension [00:41:16] yes, it kills the current process with an unconditional abort() after a timeout [00:41:51] so, not very friendly to threaded SAPIs [00:42:19] but if you use a threaded SAPI and you get a timeout in libc, you are pretty screwed [00:42:51] i assume you mean something like mod_php/php-fpm vs. traditional cgi (not fcgi) [00:44:55] I mean mod_php with a threaded MPM like "worker" [00:45:20] I think php-fpm starts separate processes rather than using threads [00:45:28] ok, so mom_php can be threaded or not [00:45:36] yes [00:46:05] and processes are reused but at a given point in time it's only one req per process [00:46:13] * jeremyb will have to read up on all that [00:46:41] sounds like wmerrors would be useful for php-fpm then [00:47:00] i wonder how this all fits in with hhvm [00:47:16] I was just reading about that :p [00:47:56] yeah, I'm not sure how hhvm does it [00:48:00] mwalker: feel free to drop me links if they're handy :) [00:48:24] the general architectural question is: what do you do when a timeout occurs when a library call is in progress? [00:48:40] jeremyb: I'm not really finding anything yet about how it handles requests [00:48:59] mwalker: it=? [00:49:04] hhvm [00:49:10] k [00:49:33] do you: set a flag and then check for that flag on return from the function? immediately kill the thread? or do a longjmp() and possibly deadlock? [00:49:54] option 1 means the code could run for a long time after the timeout [00:50:09] TimStarling: wouldn't the ideal case be kill the thread and hope that the library was not holding global state? [00:50:13] but you can have a second timeout in case that happens, that's the approach we take in Lua [00:50:39] well, if you kill the thread, then it's hard to write out sensible error messages, backtraces, etc. [00:50:52] maybe not impossible [00:51:04] that's basically what we do with PHP+wmerrors [00:51:10] not always successfully [00:53:39] hmm; thinking on this -- how does php's memory handling actually work? does it have it's own allocator or is it deferring to the kernel? -- basically I'm wondering how a third party module would be affected by php memory limits [00:59:27] in any case though; TimStarling; unless you think we need to remove access to that page right now; I'll poke at it futher and hopefully have a bug fix tomorrow to deploy [01:00:01] it's OK to leave it [01:00:59] kk [02:13:12] re: hiphop, https://github.com/facebook/hhvm/blob/4eb05b745fd3018a6d9e51464cae06a4465ee142/hphp/runtime/base/execution-context.cpp#L701 [02:17:20] but i don't know what is happening underneath the high-level calls [04:27:25] robla: just to be sure, signing up for architecture thing at this stage doesn't commit you to going does it? (january's a long time from now. at least for me) [04:28:19] jeremyb: nope, not at this point [04:28:31] k, danke [04:28:41] a white van will pull up and masked men in wikimedia-branded outfits will put a bag on your head [04:28:58] next thing you know, you're on a flight to SFO, handcuffed to the seat. [04:29:04] and they'll serialize you? [04:29:20] json_encode [04:30:28] serializing someone with non-free code? that's the worst. [04:31:03] but it's for good, not evil. [04:31:26] hah, "my baby" (manybubbles) [04:31:51] you could use the google rewrite [04:32:25] hrmmm, not finding that right now. maybe misremembered [06:03:15] Reedy: why did you leave this resource loading? that's not a WMF wiki https://sa.wikipedia.org/w/index.php?title=%E0%A4%AE%E0%A4%BF%E0%A4%A1%E0%A5%80%E0%A4%AF%E0%A4%BE%E0%A4%B5%E0%A4%BF%E0%A4%95%E0%A5%80%3ACommon.js&diff=252205&oldid=240084 [09:03:27] hi! On sv-wiktionary we'd like to have a simple blue link, that when you click on it, plays a video. Is there already a way to do that, or are we better off adding a bit of JS to our Common.js? [09:30:59] hey, how does an account have contributions without having a registered username? [09:31:00] https://vi.wikipedia.org/wiki/%C4%90%E1%BA%B7c_bi%E1%BB%87t:%C4%90%C3%B3ng_g%C3%B3p/TDogg310?setlang=en [09:31:21] Imports? [09:31:35] oops wait. don't click that if your language isn't English. left a ?setlang=en in [09:31:38] Reedy: oh. duh. [09:48:21] Does anyone know what is going on here? http://en.wikipedia.org/w/api.php?action=query&prop=info|flagged&inprop=protection&titles=Template%3AUser&format=xml [09:48:45] the page seems to be doubly protected [09:49:22] https://bugzilla.wikimedia.org/show_bug.cgi?id=28751 [09:50:22] Reedy: so, can/should I ask a steward to remove that script? https://sa.wikipedia.org/w/index.php?title=%E0%A4%AE%E0%A4%BF%E0%A4%A1%E0%A5%80%E0%A4%AF%E0%A4%BE%E0%A4%B5%E0%A4%BF%E0%A4%95%E0%A5%80%3ACommon.js&diff=252205&oldid=240084 [09:50:32] thanks Reedy, will check it out [09:51:41] Nemo_bis: I'll do it [09:52:13] thanks [09:53:56] anyone available in pm to check something possibly important? [09:54:15] :/ [09:54:17] (and possibly utterly unimportant :) [12:22:31] I have a persistant error [12:22:42] Request: POST http://en.wikipedia.org/w/index.php?title=Template:Syrian_civil_war_detailed_map&action=submit, from 208.80.154.8 via cp1010.eqiad.wmnet (squid/2.7.STABLE9) to 10.64.0.141 (10.64.0.141) [12:22:43] Error: ERR_READ_TIMEOUT, errno [No Error] at Mon, 21 Oct 2013 12:10:03 GMT [12:23:34] often get this error with different posts, for this post it has been about 10 times [12:24:05] this template is pretty big [12:25:17] ping ? [12:40:16] what is surprising is that the status link indicates that all servers are operating normally [14:51:35] hi andre [14:51:44] hi ragesoss [14:52:22] Andrew Green, our new contract developer for the education program extension, would like access to editing bugs and assigning them on bugzilla. [14:52:51] ragesoss, does he use a @wikimedia.org address in Bugzilla? If yes he gets such access for free. If not, I need his email ID in bugzilla [14:53:13] I just sent you his email. [14:53:32] done [14:53:39] The account has been added to the canconfirm, editbugs groups. [14:53:49] andre__: thanks much! [14:53:54] np [14:58:36] ragesoss: did he register on translatewiki.net already? :) [15:00:33] Nemo_bis: I don't think so. I'll nudge him that way. [17:16:19] how can I find list of only new articles in the mainspace (without redirections), created by some user? [17:17:08] the user has many edits, so just browsing list of all contribs is unwieldy. [17:22:31] https://tools.wmflabs.org/xtools/pages/index.php?name=UserName&lang=en&wiki=wikipedia&namespace=0&redirects=noredirects BartlomiejB [17:26:01] twkozlowski: thanks. [18:05:31] somebody have a look at bug 55947 please :) if possible [18:09:26] kunall_: Why did the email address get unattached? [18:10:28] somebody took control of it, i just don't know how, one year ago [18:11:37] Did you report that? [18:11:50] Because "Someone took over my account" seems like way more urgent news. [18:12:30] Yes I did, it was blocked on wikinews iirc, and then I started using this alternate account [18:13:19] Hm. [18:14:00] Reedy, you're the first person who comes to mind, what's the procedure for reverting changes to an account (like email address)? [18:14:10] Maybe CC James_F. [18:14:19] Got some brilliant Brits on the case. :) [18:14:24] ? [18:14:32] would that be an OTRS issue? [18:14:58] That sounds right [18:15:17] James_F: kunall_ here has lost control of his account, wants to reset the password, but whoever has it has changed the email address [18:16:25] yup, there's no email linked to the account, and seemingly they only changed the password, because no edits were made after I lost access to it [18:17:16] If there's no email linked to an account, it's hard to verify ownership [18:18:35] Reedy: Revert to the old email then? Or is there not enough history stored for that to work? [18:19:05] Hmm, all I have is the cloak I use in here, and some sort of confirmation between the link of that account and the one I use now (which used to be my account for use on public computers) [18:19:37] lol, enough history? [18:19:56] We don't track email changes and such [18:20:12] Actually... [18:20:30] [NickServ] kunall_ is not registered. [18:20:31] [NickServ] kunall is not registered. [18:20:34] What nick/cloak? [18:20:37] Hah. [18:20:45] He is cloaked. [18:20:51] But not registered. [18:20:52] Which is odd. [18:20:56] wtf [18:21:00] yup, i haven't registered this nick just jet [18:21:02] *yet [18:21:04] You can use non-registered nicks with cloaks [18:21:10] just nick to something that isnt registered [18:21:30] this one is registered, i think [18:21:31] As part of getting a cloak, don't we normally have the user leave a message somewhere? [18:21:45] Yes. [18:21:55] Ala https://meta.wikimedia.org/wiki/User_talk:Reedy#IRC_cloak_request [18:22:08] I don't remember that [18:22:13] Oh, right [18:22:16] marktraceur: DAMN NEWBIES [18:22:17] I'm still on FSF cloak. [18:22:56] marktraceur: WHERE IS YOUR LOYALTY! [18:23:01] [19:22:55] [NickServ] diegogrez is not registered. [18:23:05] lol [18:23:05] This is going well [18:23:07] YuviPanda: It's over there, at the FSF [18:23:09] hmm [18:23:11] that's strange [18:23:11] lol [18:23:15] [11:20:51 AM] kunall_ is logged in as MisterWiki [18:23:15] hold on [18:23:29] [11:23:19 AM] -NickServ- Information on MisterWiki (account MisterWiki): [18:23:30] [11:23:19 AM] -NickServ- Registered : Dec 11 18:54:28 2009 (3 years, 44 weeks, 6 days, 23:28:50 ago) [18:23:30] [11:23:19 AM] -NickServ- Last addr : be145082@wikimedia/Diego-Grez [18:23:41] is that account global or local anyways [18:24:03] Reedy, would you be able to revert the last change you made to Heb Voy which causes this bug - https://bugzilla.wikimedia.org/show_bug.cgi?id=55501 ? [18:24:39] WikiJunkie: Yeah... [18:25:30] partial revert [18:27:51] well, i have just registered this nick to my account [18:28:03] and I'm trying to find that irc cloak thingy on wiki [18:29:50] unfortunately it doesn't add to confirming the old account now [18:30:28] diegogrez: i'd try on https://meta.wikimedia.org/wiki/OTRS if you can think of other ways to confirm it was really you, add it all [18:33:53] okay; it's just that, besides that wikinews diff, i have no more proof i was the one who owned that account u.u [18:34:10] but will try anyway, how long would it take for it to be processed or something? :) [18:34:13] btw, thank you all [18:35:31] diegogrez: ask freenode staff for pw reset. [18:35:49] no. wat (my eyes......) [18:37:42] haha [20:43:30] mwalker: did you fix globalallocation? seems to work but i'm having trouble tracking down the commit [20:44:18] no; not yet -- Jeff is working on setting up our QA box so that I can work with the live data [20:45:19] mwalker: but why does it work then? :) [20:45:55] are there just less active banners or something? [20:46:01] because of what the error is -- it's running out of memory while attempting to generate the minimum covering set of active campaigns [20:46:23] ish -- it's a function of the number of active campaigns [20:46:25] yesterday FDC was active and now it's not [20:46:45] or i thought it wasn't but i do see it on the global page [20:47:12] anyway, ok, danke [20:47:21] is there a bug? should i make one? [20:47:24] mwalker [20:47:39] no bug; but go ahead and create one [20:49:12] mwalker: summary? [20:49:16] isn't it https://bugzilla.wikimedia.org/55973 ? [20:49:32] i guess so [20:49:37] oh; yes it is [20:51:14] * jeremyb gave it to mwalker :)