[00:26:04] gn8 folks [02:26:23] hey guys, just tried to use the notifytranslators extension and got the following error: [02:26:26] Request: POST http://meta.wikimedia.org/wiki/Special:NotifyTranslators, from 82.28.223.192 via amssq35.esams.wikimedia.org (squid/2.7.STABLE9) to 91.198.174.46 (91.198.174.46) [02:26:28] Error: ERR_READ_TIMEOUT, errno [No Error] at Thu, 14 Mar 2013 02:24:03 GMT [02:27:03] any thoughts? [02:30:30] what's a good source for the exact pope-related announcement times? [02:31:07] TimStarling: maybe the vatican uses twitter? [02:31:48] oh [02:31:50] habemus papam [02:31:51] crazy [02:31:52] TimStarling, how exact? [02:32:16] http://www.guardian.co.uk/world/2013/mar/12/papal-conclave-to-choose-new-pope-day-one-live-coverage is roughly what I want, I think [02:32:24] even has times in GMT [02:33:00] well, the day 2 version of that anyway [02:35:09] so white smoke at 18:06, then the announcement at 19:13 [02:36:46] http://www.bbc.co.uk/news/world-europe-21777747 [02:36:54] bbc pins the annoucement at 19:!" [02:36:58] 19:12 [02:39:06] c.f. http://ganglia.wikimedia.org/latest/graph.php?c=LVS%20loadbalancers%20eqiad&m=bytes_out&r=custom&s=by%20name&hc=4&mc=2&cs=03%2F13%2F2013%2017%3A00%20&ce=03%2F13%2F2013%2022%3A00%20&st=1363228610&g=network_report&z=medium [02:40:18] are you writing a paper? [02:40:31] no, just working out why the site went down [02:40:54] "Timing the pope election announce by analyzing behavior of a crowd with rrdtool", Tim, 2013. [02:41:36] :r( [02:42:23] 18:21 PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 8.59004514493 (gt 8.0) [02:42:47] so the network issues started well before the announcement [02:43:00] it's a miracle! [02:43:47] 18:42 PROBLEM - LVS HTTP IPv4 on appservers.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:44:50] ganglia is missing data for the LVS loadbalancers in esams. Could we had all the european traffic redirected to the us? [02:45:04] it's a bit sad for the servers to be brought down by white smoke [02:45:37] :-] [02:45:41] ganglia is missing data from the period when the esams/eqiad network was saturated [02:46:07] Leslie fixed it [02:46:12] also if you look at the network traffic for search, we had traffic moved from eqiad to pmtpa http://ganglia.wikimedia.org/latest/graph.php?c=Search%20eqiad&m=cpu_report&r=custom&s=by%20name&hc=4&mc=2&cs=03%2F13%2F2013%2017%3A00%20&ce=03%2F13%2F2013%2020%3A00%20&st=1363228964&g=network_report&z=medium&r=custom [02:46:26] yeah, that was coincidental [02:46:36] hmm, yeah [02:47:08] the apaches came down when search was moved to pmtpa [02:48:25] so maybe white smoke had nothing to do with the downtime [02:49:34] and we also switched all wikis to 1.21wmf11 apparently [02:49:52] 18:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable scribunto EVERYWHERE' [02:49:53] 18:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf11 [02:51:02] which eventually send some bits caches to swap : ( http://ganglia.wikimedia.org/latest/?r=day&cs=03%2F13%2F2013+17%3A00+&ce=03%2F13%2F2013+20%3A00+&m=cpu_report&s=by+name&c=Bits+caches+eqiad&h=&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 [02:51:19] ah no just niobium.wikimedia.org [03:02:44] anyone with safari handy? [06:26:08] jeremyb_: Yes. [06:26:32] too late but thanks anyway [06:26:51] well, maybe [06:27:07] Susan: you've tested the new SSL cert with safari? [06:27:11] enwiki [06:27:34] * jeremyb_ remembered he had a mac laying around and tested. but maybe you have a different version [06:28:08] didn't find any problems. this is for an OTRS ticket where someone was saying they had errors but they didn't say what the error was [06:28:23] I see a certificate that expires January 20, 2016. [06:28:34] but no errors, etc. [06:28:39] I don't think so. [06:28:47] so i sent them some screenshots from my safari (using pa.us.wikimedia.org as an example bad cert) and asked for more details [06:30:18] 2013031310000678 in case anyone feels like reading it [06:30:38] * jeremyb_ falls asleep [14:44:05] There's an error when I'm trying to open http://id.wikipedia.org/wiki/Imlek [14:44:19] "MathRenderer::writeDBEntry". ... "1048: Column 'math_outputhash' cannot be null (10.64.16.23)". [14:44:54] I edited the page anyway via http://id.wikipedia.org/w/index.php?title=Imlek&action=edit, but when I submitted the change, there's another error [14:44:56] A database error has occurred. Did you forget to run maintenance/update.php after upgrading? See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script Query: REPLACE INTO `math` (math_inputhash,math_outputhash,math_html_conservativeness,math_html,math_mathml) VALUES ('�0��p(��o%HQ0',NULL,'0','','') Function: MathRenderer::writeDBEntry Error: 1048 Column 'math_outputhash' cannot be null (10.64.16.23) [14:46:42] anybody looking into this? [14:49:27] <^demon> See https://bugzilla.wikimedia.org/show_bug.cgi?id=46042 [14:51:50] all fixed. Thanks ^demon [15:06:52] I just got a gateway timeout errro [15:06:56] error* [15:37:29] hi, on plwiki is smth wrong with configuration of time zones, the first edit about new pope is 20:13, but it should be 19:13 - http://pl.wikipedia.org/w/index.php?title=Franciszek_%28papie%C5%BC%29&diff=35228530&oldid=35115531 [15:37:46] (view of diff as anon) [15:42:31] Leinad_: I see it as in 19:13 [15:43:33] Base-w: as not logged-in? [15:45:42] Leinad_: as logged in [15:47:21] hmm, anon view is wrong, I see 20:13 and the same one of Polish journalist who just asked me about it, weired... :/ [15:48:10] it plwiki in UTC or in CET? [15:48:23] *do [15:48:26] *does [15:49:16] > 'plwiki' => 'Europe/Warsaw', [15:49:36] CET [15:49:53] shows 19:13 for me [15:50:02] In my setting it is UTC [15:50:06] oh, i'm logged in [15:50:13] log out please ;) [15:50:48] as not logged in [15:50:52] so all right [15:50:56] Wersja z 20:13, 13 mar 2013 (pokaż źródło) [15:51:23] yes, now you see the same - incorrect time [15:51:25] it is 20 for anonymous as CET and it is 19 for me as UTC [15:51:48] is it servers cache issue or something like that? [15:52:19] you mean that it was done in 18 UTC? [15:52:54] it was done 19:13 CET, so it was 18:13 UTC [15:53:30] ah i can remember the white smoke started when I was wthcing news at 1+1 chanel it is 19:30 EET + [15:53:41] *watching [15:54:23] so it is 18:30 CET+ [15:55:05] hm [15:56:56] I'm sure, it was 19:13 on plwiki ;) [16:24:42] hum.. how's #wikimedia-operations beeng used? Server stuff errors should be reported there or here? [16:25:40] here [16:26:07] ok, here it goes :P [16:26:09] seems to be having a problem opening pt.wikipedia pages with the math tag. a user reported me that opening the page http://pt.wikipedia.org/wiki/Pi for example, gets the errror Error: 1048 Column 'math_outputhash' cannot be null (10.64.16.23) [16:26:25] this was reported from brasil, i'm in portugal, and i can open it without problems [16:26:53] location of requester shouldn't matter [16:28:15] Ciao!, I'm here to notice a problem with Portuguese Wikipedia, the page http://pt.wikipedia.org/wiki/Pi return a database error : Error: 1048 Column 'math_outputhash' cannot be null (10.64.16.23) [16:28:30] pages withou the math tag, seems to be working, right jonas_agx? [16:28:47] Alchimista: Yes! [16:29:01] i can see them without problems :P [16:29:41] Alchimista: it only happens on Brazil area, I can access it perfectly from USA [16:30:37] Alchimista, jonas_agx this is https://bugzilla.wikimedia.org/show_bug.cgi?id=46042 [16:47:54] Reedy: Have you seen this? --^^ ([2013-03-14 16:26:04] seems to be having a problem opening pt.wikipedia pages with the math tag. a user reported me that opening the page http://pt.wikipedia.org/wiki/Pi for example, gets the errror Error: 1048 Column 'math_outputhash' cannot be null (10.64.16.23)) [16:48:00] RoanKattouw: Yes [16:48:06] There's been a bug logged for a few days about it [16:48:09] Excellent, just making sure [16:48:11] Oh, OK [16:48:26] Clearly I have no idea what's going on any more now that I'm away from work 2 days per week [16:48:35] https://bugzilla.wikimedia.org/show_bug.cgi?id=46042 [16:49:20] the formula has an error: http://pt.wikipedia.org/w/index.php?title=Usu%C3%A1rio%28a%29%3AAlchimista%2Ft1&diff=34437352&oldid=34437208 [16:50:23] Somebody thought it'd be great idea to set null in the code when the column is not null [16:51:30] ah, so it's not an error on the math expression? [17:12:05] Reedy: I assume those math equations used to render correctly? [17:12:18] At somepoint [17:12:27] I think the code has been changed relatively recently [17:12:31] maybe even for 1.21wmf11 [17:13:12] hrm [17:13:54] strange, it doesn't even allow to use

as a string on section titles
[17:17:44] The code appears in this commit: https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/extensions/Math.git;a=commit;h=926db7c3bdcf8911b5314ffd5c1c7d0064d2734f
[17:18:35] physikerwelt is the author, the one who can't reproduce it
[17:18:41] "This commit only deals about the restructuring of the math module design."
[17:18:44] the commit summary is a lie
[17:18:50] it's a huge commit
[17:19:16] I've not really looked at the code so much as to see where the hash comes from
[17:19:18] We do run a sort of custom texvc build on the cluster..
[17:19:36] totally should have been broken up into smaller commits, in my opinion
[17:19:52] greg-g: Either way, we shouldn't be setting null for a column that's not null. We should deal with that casing in the code, not letting it propogate to being a database error
[17:20:06] right
[17:20:40] https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/extensions/Math.git;a=commitdiff;h=20192a1ab87d092963af732fdd8c95e5e60509a8
[17:20:49] the hash is the md5 of the content of the tags
[17:20:53] if I remember correctly
[17:20:56]^ That merged commit should fix it presumably
[17:21:30] let's try and fix it on the cluster then
[17:21:31] hmm
[17:22:48] * greg-g  wonders how the aft stuff going....
[17:22:52] +is
[17:24:20] just see this 16:26 logmsgbot: mlitn synchronized wmf-config 'Prepare AFTv5 config for deployment new features'  and nothing after
[17:24:40] mlitn: how's the aft stuff going, if you don't mind me interupting :)
[17:26:15] greg-g: all is well, so far - I'm running a maintenance script to merge data before syncing new code
[17:26:30] great
[17:28:34] zeljkof: would you +2 for us? https://gerrit.wikimedia.org/r/#/c/53772/
[17:29:37] chrismcmahon: will do in a minute, have to reboot now
[17:32:29] Reedy: do you have a test page to, uh, test?
[17:35:28] http://en.wikipedia.org/wiki/User:Greg_G/Math_test
[17:37:26] That's better than a database error ;)
[17:37:48] touche
[17:38:24] greg-g: Reedy the simple case http://test2.wikipedia.org/wiki/User:Cmcmahon(WMF)
[17:39:42] greg-g: on the same page, i saw that kind of warning, but another user, from a diferent continent wasn't able to see it. the page couldn't be loaded
[18:24:28] i'm looking at the math issue
[19:05:21] Ryan_Lane: so... what's the oldest OS release we'll support (or try to) for root CA store coverage?
[19:05:35] does digicert have a list of stores they're in?
[19:06:27] we got an OTRS ticket from someone running an OS point release which is having an 8th birthday in less than a month
[19:06:39] legoktm: ^
[19:06:59] jeremyb_: Windows XP is older than that
[19:07:14] Betacommand: and XP sp3 is how old?)
[19:07:21] or sp2 even
[19:07:53] * jeremyb_  has to read the ticket some more
[19:08:04] hah, he sent tiffs!
[19:08:05] jeremyb_: SP2 August 25, 2004
[19:08:19] o.O
[19:08:25] jeremyb_: ticket #?
[19:08:43] legoktm: same one you said in #-otrs last night
[19:08:51] XP itself came out August 24, 2001
[19:09:23] oh that one
[19:09:25] ahahaha
[19:11:38] which is the same one i was discussing with Susan ~12.7 hours ago
[19:29:02] hi. certificate patrol often alerts me because of certificate changes (even the CA are different). is there a list of cerfiticate fingerprints used by the wikimedia projects?
[19:29:52] zg1: no idea, but the certificates were changed just recently
[19:29:57] zg1: yesterday, i think
[19:30:05]