[00:04:19] <p858snake|l>	 wmflabs.org is chucking security warnings at me now...
[00:07:16] <legoktm>	 i keep running into
[00:07:17] <legoktm>	 Not Found
[00:07:18] <legoktm>	 The requested URL /w/index.php was not found on this server.
[00:07:18] <legoktm>	 Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
[00:07:30] <legoktm>	 once i try a few times it works
[00:07:40] <jeremyb_>	 p858snake|l: what domain?
[00:07:46] <Reedy>	 jeremyb_: wmflabs.org
[00:07:48] <Reedy>	 he said
[00:07:56] <jeremyb_>	 Reedy: which never worked?
[00:08:00] <jeremyb_>	 afaik
[00:08:07] <jeremyb_>	 i thought he meant a subdomain
[00:08:15] <p858snake|l>	 well clearly yes
[00:08:26] <jeremyb_>	 it doesn't even resolve
[00:08:39] <jeremyb_>	 legoktm: what domain for you too
[00:08:50] <legoktm>	 enwiki
[00:08:54] <legoktm>	 using https
[00:09:35] <intgr>	 The requested URL /w/index.php was not found on this server.
[00:10:17] <intgr>	 Got that twice from https://en.wikipedia.org/w/index.php?title=RC4&curid=25831&diff=543711773&oldid=539790940
[00:10:54] <jeremyb_>	 legoktm: get the headers when it happens?
[00:11:09] <legoktm>	 sure
[00:11:36] <legoktm>	 umm
[00:11:40] <legoktm>	 how do i do that in firefox?
[00:11:56] <intgr>	 Ctrl+Shift+K
[00:12:03] <stwalkerster>	 headers: https://gist.github.com/stwalkerster/2bdaf678b9e1f0eb5fa5
[00:12:13] <Susan>	 Hi.
[00:12:21] <Susan>	 I'm intermittently getting 404s on en.wikipedia.org via HTTPS.
[00:13:40] <Reedy>	 Susan: You're late to the party
[00:14:55] <Susan>	 And yet.
[00:15:05] <Susan>	 Also, surely I'm fashionably late.
[00:17:20] <Magog_the_Ogre>	 what's up with the intermittent 404s? Have they been reported here?
[00:17:40] <Cyberpower678>	 Same here.
[00:17:45] <jeremyb_>	 Magog_the_Ogre: 3-4 people before you :)
[00:17:55] <Cyberpower678>	 My bots just corrupted as a result of those.
[00:17:58] <jeremyb_>	 Susan was fashionably late. the rest of you are really late
[00:18:05] <jeremyb_>	 corrupted?
[00:18:17] <Magog_the_Ogre>	 yeah my bot is going haywire too
[00:18:22] <legoktm>	 same
[00:18:36] <Magog_the_Ogre>	 'Missing dependency: \`( < MediaWiki 1.15)\`
[00:18:42] <stwalkerster>	 I'd recommend you suspend your bots until this is fixed
[00:18:50] <Krenair>	 MediaWiki 1.15?
[00:19:30] <Magog_the_Ogre>	 yeah it's probably because it's 404'ing so the bot can't grab the MW version
[00:19:54] <Cyberpower678>	 Mine can't read the api
[00:20:03] <Reedy>	 Just wait?
[00:20:40] <Cyberpower678>	 At first I thought I bugged up my framework while updating it.
[00:21:28] <Emw>	 i've gotten intermittent 404's on two different pages in the last few minutes
[00:21:34] <legoktm>	 welcome to the club
[00:21:37] <Cyberpower678>	 Yep.
[00:21:37] <Cncmaster>	 I'm getting a series of 404 errors on pages that I know exist
[00:21:39] <Krenair>	 You're late to the party, Emw :)
[00:21:51] <Emw>	 i always am
[00:21:52] <Krenair>	 Really late.
[00:22:01] <Cyberpower678>	 Cncmaster, welcome to the club.  We're all here to complain.
[00:22:23] <Ryan_Lane>	 the first rule of complaining club is: no complaining
[00:22:27] <Ryan_Lane>	 :)
[00:22:38] <Ryan_Lane>	 thanks for the reports, btw
[00:23:13] <Cyberpower678>	 If you guys are going to break everything, at least warn us first. :p
[00:23:26] <Cyberpower678>	 Or keep the API running.
[00:23:52] <Susan>	 Cyberpower678: That's not helpful.
[00:24:12] <Cyberpower678>	 Susan, it was joke.
[00:24:29] <Susan>	 Oh. In that case, that's not funny. ;-)
[00:25:00] <Cyberpower678>	 Test wikipedia is completely 404ed
[00:25:07] <Cyberpower678>	 Nothing is working there.
[00:25:53] <jeremyb_>	 Cyberpower678: we don't need more reports of more sites. *all* sites should be at least sometimes broken now.
[00:25:58] <Magog_the_Ogre>	 EVERYONE PANIC NOW
[00:26:08] * jdelanoy  runs around screaming

[00:26:32] * Cyberpower678  is lighting the torches and waving it around in panic.

[00:28:26] <Cyberpower678>	 en.wikipedia is working.  test is still down. :D
[00:30:16] <Cyberpower678>	 .....aaaand we're back up again.  Yay.
[00:33:24] <Cncmaster>	 finally
[00:33:41] <Cyberpower678>	 And my bots are functioning again.
[00:40:53] <DaB|Eingeschneit>	 gn8 folks
[01:02:22] <Ahonc>	 was mediawiki updated today?
[01:02:28] <Reedy>	 no
[01:02:33] <Ahonc>	 there something wrong in uk.wikipedia
[01:02:49] <Krenair>	 what's wrong?
[01:03:03] <Ahonc>	 sorting in category: http://uk.wikipedia.org/wiki/Категорія:Станції_Південно-Західної_залізниці
[01:03:20] <Ahonc>	 yesterday it worked fine 
[01:03:29] <Ahonc>	 after fixing bug 45444
[01:03:34] <Ahonc>	 but now it is broken
[01:04:05] <Reedy>	 Well, see how it is in 24 hours or so
[01:04:10] <Ahonc>	 there should be headers for each letter
[01:04:19] <Ahonc>	 but I look only letter Y there
[01:04:20] <Reedy>	 there's another script going to be run on it to fix some other issues
[01:04:47] <Ahonc>	 and other categories wrong too
[01:14:08] <jeremyb_>	 LeslieCarr: that discussion reminded me of https://rt.wikimedia.org/Ticket/Display.html?id=4143 :)
[01:22:52] <Ahonc>	 in pl.wiki is similar issue: http://pl.wikipedia.org/wiki/Kategoria:Prezydenci_Wenezueli
[01:22:59] <Ahonc>	 category sorting is broken
[01:24:09] <jeremyb_>	 Ahonc: i doubt the answer will change. ask again in 24 hours
[01:24:27] <jeremyb_>	 (won't change tonight that is)
[01:25:13] <LeslieCarr>	 jeremyb_: :)
[01:30:08] <Ahonc>	 and similar issue in pt-wiki
[01:30:08] <Ahonc>	 http://pt.wikipedia.org/wiki/Categoria:Presidentes_por_país
[01:30:27] <Ahonc>	 all these wikis had changed collation in categories
[01:31:33] <jeremyb_>	 Ahonc: i hope you don't want an answer?
[01:43:45] <Ahonc>	 I don't want answer. I want hear that problem is fixed :)
[01:48:32] <jeremyb_>	 Ahonc: so, set an alarm for 86000 secs from now
[01:49:06] <Ahonc>	 :)
[01:57:08] <Alchimista>	 has been any mw update? on pt.wp, cats are having non latin letters, and the alphabetic order is a mess: http://pt.wikipedia.org/wiki/Categoria:!Predefini%C3%A7%C3%B5es_da_p%C3%A1gina_principal
[01:58:29] <Ahonc>	 I asked similar question above
[01:58:44] <Alchimista>	 ya, that's what i've just noticed
[01:58:53] <Ahonc>	 such problems are at least in pl.wiki and uk.wiki too
[01:59:13] <Alchimista>	 is there any open bug?
[02:43:46] <p858snake|shower>	 Alchimista: yes, It's due to a update to the category sorting system (so we can actually do it properly in some languages) so there is a script running and updating causing the "appearance" of some mess whilst it does its magic
[03:20:52] <TimStarling>	 I'm not convinced that it is just a temporary update problem
[03:22:46] <TimStarling>	 no, I really don't think so
[03:44:36] <robla>	 TimStarling: what are you referring to?
[03:44:55] <robla>	 spose I should dig up the backlog
[03:45:15] <TimStarling>	 the garbage in the h3's on https://pl.wikipedia.org/wiki/Kategoria:Prezydenci_Wenezueli
[03:45:25] <TimStarling>	 I'm most of the way to fixing it now, I hope
[03:46:56] <robla>	 pretty unicode characters!  :)
[04:08:04] <TimStarling>	 Alchimista, robla, p858snake: should be fixed now
[04:22:39] <jeremyb_>	 fixed for me
[04:22:50] <jeremyb_>	 after a hard refresh
[04:24:38] <robla>	 ditto
[09:36:07] <DanielK_WMDE>	 hashar: got a minute for a gerrit oddity? I can't merge https://gerrit.wikimedia.org/r/#/c/46686/
[09:36:11] <DanielK_WMDE>	 i just don't get the button
[09:36:24] <DanielK_WMDE>	 even though it sais "Can Merge: Yes"
[09:36:30] <hashar>	 sure
[09:36:35] <DanielK_WMDE>	 is jenkins supposed to auto-merge this?
[09:36:45] <DanielK_WMDE>	 or is it because oif jeroen's (outdated) -2?
[09:37:51] <hashar>	 yeah I guess Jenkins attempted to submit the change
[09:37:56] <hashar>	 but it got blocked caused of CR-2
[09:38:06] <DanielK_WMDE>	 hrm
[09:38:12] <DanielK_WMDE>	 ok, i'll remove and re-add jeroen :)
[09:38:17] <hashar>	 ;-)
[09:38:24] <hashar>	 Zuul should gives out a message about it
[09:38:37] <DanielK_WMDE>	 hashar: but if CR-2 blocks, why does it say "Can merge: yes"?
[09:38:41] <DanielK_WMDE>	 that's just broken :/
[09:39:36] <DanielK_WMDE>	 hashar: seems like if there's a CR+2 and a CR-2, it *sais* it can merge, but doesn't let you.
[09:39:53] <hashar>	 na can-merge just mean the code could potentially be merged
[09:39:58] <hashar>	 regardless of the code review status
[09:40:08] <hashar>	 something like : " it is technically possible to merge the change"
[09:40:10] <DanielK_WMDE>	 o_O
[09:40:11] <DanielK_WMDE>	 really?
[09:40:14] <DanielK_WMDE>	 that's confusing
[09:40:41] <DanielK_WMDE>	 anyway, thanks for looking
[09:40:50] <hashar>	 here is Gerrit message:
[09:40:51] <hashar>	 error: blocked by Code-Review
[09:40:52] <hashar>	 one or more approvals failed; review output above
[09:40:57] <DanielK_WMDE>	 and yes, it would be good if it would tell me that it was unable to merge for some reason
[09:40:58] <hashar>	 that is only in the debug log though :(
[09:41:04] <DanielK_WMDE>	 :/
[09:41:35] <DanielK_WMDE>	 anyway, merged it now
[09:41:37] <DanielK_WMDE>	 thanks!
[09:44:57] <hashar>	 DanielK_WMDE: opened bug https://bugs.launchpad.net/zuul/+bug/1154507 :-]
[09:54:31] <DanielK_WMDE>	 hashar: thank you
[11:18:09] <Danny_B>	 !log reenabled mail delivery to wikibugs
[15:20:20] <Cyberpower678>	 Can a techie give me some help?
[15:20:44] <sumanah>	 Cyberpower678: if you describe your problem I can try
[15:20:50] <sumanah>	 I may not know the answer though.
[15:21:03] <Cyberpower678>	 createaccount on API seems to hangup/
[15:21:25] <Cyberpower678>	 When I give it the parameters and the token, it creates the account and stops.
[15:21:48] <Cyberpower678>	 It's definitely not an issue with the program.
[15:22:10] <sumanah>	 When did this start? Were you able to use it successfully before?
[15:22:10] <Cyberpower678>	 sumanah, any ideas
[15:25:19] <Cyberpower678>	 sumanah, since the beginning.  I just didn't notice at first.  When I use createaccount to get the token, the API returns results.  When I then use the createaccount, it successfully creates the account and hangs up, causing my bot to hangup.
[15:28:44] <petan>	 Cyberpower678 define "hang up"
[15:29:25] <petan>	 api are being used over the http protocol, its connection is terminated after every request
[15:32:07] <CP678|iPhone>	 petan, all API queries return something.  The createaccount function doesn't.
[15:34:17] <Cyberpower678>	 petan, when the apiQuery function is running in my framework, it gets stuck there.  All other queries work fine.
[15:38:20] <Cyberpower678>	 petan, according to the documentation, a successful account creation should return something along the lines result=success.
[15:42:48] <petan>	 CP678|iPhone OK if it doesn't return anything it probably crashed :o (you should get 500)
[15:43:06] <petan>	 you should never get a blank page
[15:43:58] <Nemo_bis>	 O_o https://bugzilla.wikimedia.org/show_bug.cgi?id=43998#c1
[15:44:29] <petan>	 lol
[15:44:42] <petan>	 haha
[15:44:55] <petan>	 Nemo_bis we needed moar bots :D
[15:45:06] <Cyberpower678>	 petan, But it's giving me a blank.  My bot is waiting for a return reply and the API is just not giving it anything.
[15:45:27] <petan>	 Cyberpower678 ok if it's giving blank page, then it's bug in mediawiki -> bugzilla is what you want :P
[15:45:30] <petan>	 !bugzilla
[15:45:30] <wm-bot>	 https://bugzilla.wikimedia.org/$1
[15:45:32] <petan>	 meh
[15:45:36] <petan>	 without $1
[15:45:51] <petan>	 !bugzilla    
[15:45:51] <wm-bot>	 https://bugzilla.wikimedia.org/
[15:45:52] <petan>	 :P
[15:45:54] <petan>	 here we go
[15:46:07] <Nemo_bis>	 8 lines for a link
[15:46:10] <petan>	 :D
[15:46:17] <petan>	 lazyness is responsible
[15:46:22] <petan>	 not me
[15:47:05] <Cyberpower678>	 petan, I did a thorough search for a bug and it always stops at the query command for createaccount.
[15:47:11] <Cyberpower678>	 Thanks for the link.
[16:38:08] <siebrand>	 Wikimedia Language Engineering team office hours in #wikimedia-office in about 20 mins
[17:09:34] <Nemo_bis>	 Wikimedia Language Engineering team office hours started in #wikimedia-office
[18:33:51] <gwick|issues>	 IRC borken too :/
[18:46:46] <LeslieCarr>	 engineering - have any deploys happened in the last hour ?
[18:47:49] <hexasoft>	 something is broken
[18:49:44] <LeslieCarr>	 hexasoft: yes, we are investigating in #wikimedia-operations
[18:50:06] <hexasoft>	 ok
[18:53:57] <lars__>	 things are very slow
[19:20:23] <Alchimista>	 anyone knows if the cause is an increase of viewers?
[19:21:29] <saper>	 addshore: can you stop addbot on plwiki please? we have some performance issues
[19:25:05] <binasher>	 AaronSchulz: ping
[19:38:05] <Darkdadaah>	 Habemus Luam :D Thanks !
[20:29:08] <AaronSchulz>	 binasher: hm?
[20:30:09] <paravoid>	 AaronSchulz: hey
[20:30:26] <paravoid>	 AaronSchulz: did asher find you or should I do a preliminary recap?
[20:30:46] <AaronSchulz>	 I just got back from Chaat, and talked to rob a bit
[20:35:33] <binasher>	 AaronSchulz: https://graphite.wikimedia.org/render?from=-2months&until=now&width=800&height=400&target=drawAsInfinite%28deploy.scap%29&target=*.-total.count&yMax=&logBase=10&uniq=0.5222163577696607
[20:36:16] <binasher>	 that's the rate of backend php requests 
[20:36:51] <binasher>	 i think the first big increase starts with this deploy - February 16 01:11 logmsgbot: reedy synchronized wmf-config/filebackend.php
[20:38:19] <Susan>	 Blame the ginger.
[20:38:48] <binasher>	 though maybe not! 
[20:39:25] <binasher>	 but today we noticed that a pretty high percentage of requests hitting the apaches are action=render requests against commons sent by mediawiki from FileRepo 
[20:41:01] <binasher>	 a pope related traffic spike caused some site disruptions but the increase in apache requests over the last few weeks meant we had less capacity for it 
[20:41:09] <AaronSchulz>	 binasher: so wont that change result in internal https requests from apache to apache to render commons description pages on foreign wikis?
[20:41:26] <AaronSchulz>	 maybe it did that before though
[20:41:38] * AaronSchulz  looks at $urlprotocol

[20:43:23] * binasher  steals AaronSchulz for a site performance team 

[20:44:05] <AaronSchulz>	 seems like that's not new, though still wonky nonetheless
[20:45:09] <Susan>	 Wiki wonkiness.
[20:45:21] <binasher>	 AaronSchulz: am i right in thinking mediawiki is sending some of those requests to https://commons? 
[20:45:24] <AaronSchulz>	 it used to be $urlprotocol = '' before that, so I can't see Reedy's change hurting much
[20:45:53] <binasher>	 AaronSchulz: does that split the memcached keys?
[20:46:00] <AaronSchulz>	 binasher: yes, if the main requests itself was https
[20:46:11] <binasher>	 $key = $this->repo->getLocalCacheKey( 'RemoteFileDescription', 'url', $wgLang->getCode(), $this->getName() );
[20:46:24] <awjr>	 greg-g: due to the site outage, the mobile team needs to reschedule today's deployment - can we reschedule for tomorrow 3-5pm PDT?
[20:46:50] <binasher>	 memcached misses there result in render request, so splitting the key space would double the requests 
[20:46:57] <AaronSchulz>	 doesn't seem to split the cache
[20:47:04] <AaronSchulz>	 the url is not in the key
[20:48:45] <greg-g>	 awjr: yep, tomorrow 3-5 is open, I'm editing the wiki right now for some other things, so I'll update it
[20:48:53] <awjr>	 awesome thanks greg-g
[20:49:03] <greg-g>	 thank you :)
[20:49:25] <binasher>	 Ryan_Lane: paravoid: ^^ the above https requests to commons made by mediawiki hit the ssl servers so would increase traffic there 
[20:49:34] <Ryan_Lane>	 binasher: yep
[20:49:37] <Ryan_Lane>	 that was my thought
[20:49:42] <Ryan_Lane>	 binasher: though they also do ipv6
[20:49:48] <Ryan_Lane>	 so it's possible that was the increase, too
[20:49:50] <binasher>	 mediawiki hitting the ssl servers and squids to talk to mediawiki.. heh 
[20:49:59] <AaronSchulz>	 heh
[20:50:02] <Ryan_Lane>	 wait. mediawiki?
[20:50:07] <Ryan_Lane>	 -_-
[20:50:08] <binasher>	 yes, mediawiki
[20:50:20] <Ryan_Lane>	 why would mediawiki go back through the entire stack?
[20:50:46] <binasher>	 https user request -> ssl -> (plain text) squid -> mediawiki -> ssl -> (plain text) -> squid -> mediawiki 
[20:50:54] <Ryan_Lane>	 hahahaha
[20:50:57] <Ryan_Lane>	 that's rough
[20:51:12] <binasher>	 yo dawg
[20:51:14] <Ryan_Lane>	 why on earth does that happen?
[20:52:18] <paravoid>	 er, what?
[20:52:22] <paravoid>	 why mediawiki went through SSL?
[20:52:38] <binasher>	 paravoid: because mediawiki :D
[20:52:42] <paravoid>	 lol
[20:52:47] <greg-g>	 haha
[20:52:52] <paravoid>	 you read the discussion about the much larger SSL cert?
[20:52:58] <binasher>	 yup
[20:52:59] <paravoid>	 with numbers?
[20:53:58] <AaronSchulz>	 seems like confusion around getDescriptionUrl() (which seems to give public urls, which should ssl as needed) and getDescriptionText() just reusing that (which makes sense for instant commons, not for wiki farm)
[20:54:19] <AaronSchulz>	 I guess that was written back when pretty much everything was non-https
[20:54:20] <paravoid>	 so, multiple issues found
[20:54:30] <paravoid>	 is anyone planning to summarize those in mails?
[20:54:33] <paravoid>	 I'm already lost :)
[20:55:01] <^demon>	 AaronSchulz: Yes, that stuff was written a long time ago.
[20:55:12] <^demon>	 And hasn't changed really since ForeignApiRepo.
[20:56:10] <binasher>	 AaronSchulz: do you think we should look for other causes behind the increase in php requests? 
[20:59:14] <^demon>	 There's a reason it doesn't fetch directly from the database though...it wants to be parsed in the context of the source wiki.
[20:59:36] <^demon>	 But yes, there's better ways than going all the way through a request like that, don't disagree.
[21:00:14] <Susan>	 It wants?
[21:00:29] <binasher>	 it hungers for parsing 
[21:00:35] <AaronSchulz>	 wtf does the Http of all things not have profiling calls
[21:00:48] <AaronSchulz>	 *Http class
[21:02:58] <AaronSchulz>	 binasher: https://gdash.wikimedia.org/dashboards/filebackend/ heh
[21:04:57] <binasher>	 yay saturated networks!
[21:06:00] <AaronSchulz>	 lots of Memcached error for key "commonswiki:file:87fb66e3a88146210c175706a2f0c807" on server "10.64.0.190:11211": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
[21:06:56] <AaronSchulz>	 probably wasn't helping the stat cache and thus the HEAD req/sec
[21:07:32] <AaronSchulz>	 actually that would increase master traffic to
[21:07:40] <AaronSchulz>	 DB master
[21:08:48] <AaronSchulz>	 so the stream/stat increase suggests that lots of new thumbs were viewed
[21:08:56] <AaronSchulz>	 I wonder how much of that work was redundant
[21:09:07] * AaronSchulz  recalls mentioning pool counter and thumbnails before

[21:14:50] <AaronSchulz>	 binasher: the graphite data seems to be flaky around this time
[21:46:01] <awight>	 !log payments cluster updated to 61d3f1f
[21:46:06] <morebots>	 Logged the message, Master
[21:53:47] <multichill>	 Ryan_Lane: Noticed the certificate discussion. You know Nagios has a build in certificate check?
[21:54:36] <Ryan_Lane>	 for validity as well as expiration?
[21:54:41] <Ryan_Lane>	 and does it check the entire chain?
[21:55:42] <multichill>	 AFAIK you should be able to check that yes
[21:56:04] <multichill>	 I mainly use it to hunt down about to expire certificates, but it's much more extensive I think
[22:29:22] <Beau_>	 Hi, anyone familiar with error: https://pl.wiktionary.org/wiki/eigenvector?uselang=en from within function "MathRenderer::writeDBEntry". Database returned error "1048: Column 'math_outputhash' cannot be null (10.64.16.8)".
[22:30:12] <Reedy>	 Yes
[22:30:16] <Reedy>	 Known and logged bug
[22:30:26] <Reedy>	 With activity ongoing
[22:30:43] <Beau_>	 Ok, thanks. Can anyone save https://pl.wiktionary.org/w/index.php?title=gutta_cavat_lapidem_non_vi,_sed_saepe_cadendo_/_sic_homo_doctus_fit_non_vi_sed_saepe_studendo then?
[22:31:06] <Beau_>	 Error: ERR_ZERO_SIZE_OBJECT, errno [No Error] at Wed, 13 Mar 2013 22:30:56 GMT