[00:19:00] lesliecarr: lol [00:19:19] i'm not as smooth as you domas ;) [00:19:33] lesliecarr: "lol" is about the thought that one can get fired for fixing something [00:19:41] hehe [00:19:45] oh well, they were threatening to take away my wikimedia root access when I was fixing something [00:19:56] so maybe there is some practice there [00:21:20] and asher implied that someone should have his cluster access revoked for some package removals [00:21:27] you guys love tension, don't you? [00:21:35] thats probably because wikipedia is so important [00:22:05] domas: are you involved in that facebook hacking contest thingie that just started?:) [00:22:18] domas: don't worry, the only access i'd revoke is mine [00:22:38] what hacking contest? [00:22:45] https://www.facebook.com/hackercup [00:22:53] it is for smart people [00:23:02] I wouldn't stand a chance against these kids [00:23:55] somebody pasted this on #mediawiki, thats why http://dpaste.org/EQx7a/ [00:24:12] its supposed to be the first problem and a 10k$ price heh [00:27:35] first problem is easy [00:27:37] usually [01:23:32] hey - who here is able to tell people that purge requests are now working [01:23:36] like tell the community at large [01:26:46] is Tilman on IRC? [01:26:59] !seen tbayer [01:34:11] <^demon> LeslieCarr: Oh, that's fixed now? [01:34:17] <^demon> Awesome, I can reply on-wiki again. [01:35:20] ^demon: she had to leave office, but yeah it's fixed per SAL [01:35:24] <^demon> LeslieCarr: Looks like James Alexander already replied on-wiki about it. [01:35:28] <^demon> Ah, gotcha. [01:35:33] <^demon> Well, looks like all's well. [01:35:42] cool [01:36:10] <^demon> (Granted, I haven't been able to replicate since I hit eqiad) [02:38:22] ^demon: #mediawiki has trolls [03:24:45] hey [03:24:48] i am back in the office now [04:58:10] Hi, I noticed from the logs that LeslieCarr fixed a purging problem 3.5 hours ago.. [04:59:42] enwiki "Main Page" is still 3.5 hours stale, and the redirect from "Main page" is still 4 days stale, so I posted relevant HTTP headers and HTML source comments to enwiki:VPT.. [05:00:19] just in case it's not simply the job queue working through the backlog of purges.. [05:00:27] http://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Users_reporting_site_time_issues_and_delay_in_visible_update_of_edits [05:01:23] Hope that's helpful, sorry to bother you if that's expected delay/SNAFU :) [13:31:10] #wikimedia-ops [13:31:29] whoops [15:32:27] [[Tech]]; MarcoAurelio; /* Computer modern font for SVG */ resolved fixed; https://meta.wikimedia.org/w/index.php?diff=5132996&oldid=5114793&rcid=3856198 [16:19:55] [[Tech]]; MarcoAurelio; /* L'il question */ new section; https://meta.wikimedia.org/w/index.php?diff=5133275&oldid=5132996&rcid=3856237 [16:29:51] [[Tech]]; Ruslik0; /* L'il question */; https://meta.wikimedia.org/w/index.php?diff=5133325&oldid=5133275&rcid=3856244 [17:23:31] LeslieCarr: thanks for the feedback at enwiki.VPT. Purging still isnt fixing the stale pages - or, at least, test purges made 12 hours ago have still not propagated. [17:24:24] Try them again? [17:24:43] Old purgse won't propogate [17:24:49] eg http://en.wikipedia.org/wiki/Main_Page still showing sitenotice that was blanked 15hrs ago (timestamp 20130126011833) [17:25:31] should a purge take immediate effect or is it normal to be queued for perhaps several hours? [17:25:33] Indeed.. [17:25:49] If it doesn't take almost immediate effect, it's not going to [17:26:57] We know the traffic routing for the information was completely broken to the other data centres, which should be fixed.. Leslie was watching the traffic to check it went where it was supposed to [17:27:07] also http://en.wikipedia.org/wiki/Main_page (small "p", redirect) I purged last night and again just now, but when reloading Ctrl-R, Ctrl-Shift-R still shows 22 January [17:27:43] "I'm in the US and I had the outdated pages problem only when I was not logged in. When I was logged in, I didn't notice any problem." [17:27:46] I do love some of these comments [17:29:05] Reedy: ^^ snitch https://meta.wikimedia.org/w/index.php?diff=5133275&oldid=5132996&rcid=3856237 reports high job queue on meta, and enwiki queue is higher than usual, 400k - is there a backlog of purges? [17:29:18] No [17:29:20] Purges are instant [17:29:26] The job queue isn't used for purges [17:30:25] Around 1500 jobs on meta isn't high [17:31:30] Will have to see if Leslie comes online in a bit [17:31:38] Still a little early in SF (even more so as a weekend) [17:32:33] cool, nessun dorma :) [17:33:48] wondering if prob is with unrefreshed transclusions; eg MediaWiki:sitenotice transcluded on to all page (might explain Main Page lag), and DidYouKnow templates are transcluded.. [17:34:46] It works fine as a logged in user [17:37:02] yes, no probs when logged in [17:42:09] purge worked at http://en.wikipedia.org/wiki/Template:Now but not http://en.wikipedia.org/wiki/Main_page (small "p" redirect); maybe purge-failure applies only to pages which transclude date-sensitive templates - eg DidYouKnow on to Main Page - not to the pages themselves. [17:43:33] Could a purged page use stale transclusions?! [17:44:16] if it's cached in squid (ie for anon users), it caches the whole page [17:44:30] until that expires, or it's told to throw that copy away, it won't attempt to get a newer copy [17:45:20] i guess it would need a major parser bug to mess up transclusion caching [18:04:48] Richardguk: Is it a problem of anywhere outside the EU? [18:06:39] Reedy: I'm in the UK, i don't recall most users specifying their location, so Europe only would be my assumption [18:06:55] Europe-only, not EU-only [18:07:42] Krenair: Same difference for all it matters [18:09:17] I thought I'd seen someone in the US complaining.. But I might be wrong [18:10:14] Reedy: yes, i think i saw that too, prbably remembering the same comment you saw; but presumably routing etc could confound isolated cases [18:11:01] Reedy: you could ask on VPT for more systematic reporting: page, source timestamp (from the HTML comment), country, whether ip/regd [18:11:51] Not really needed [18:12:10] It's easily reproduced by anyone in Europe, so it's not a minority [18:13:28] Reedy: US export sanctions? :) [18:42:07] Is there a known issue with planet ? https://meta.wikimedia.org/wiki/Planet_Wikimedia#Not_updated_or_removed [18:43:47] Might be related to recent updates [18:43:50] mutante: ^^ [18:44:58] Frakir: you published it only 5 h ago, Daniel said it's updated every 24 h IIRC [18:46:35] Nemo_bis : each day now ? It was each hour before [18:47:17] Reedy: should history pages be being cached? i assume ?action=history&action=purge is not valid so no purging would be possible over the web [18:47:22] Frakir: I think every 2 h, but yes, I don't know why [18:48:16] Reedy: stale history pages are showing up; could be that purge requests are themslves being cached?! that might explain why some purges go through but other (subsequent) ones are being ignored [18:50:01] Reedy: i.e. if squid were caching /w/index.php?* instead of only /wiki/* [19:21:14] Richardguk: Logged in or out? [19:21:21] How would the purge requests be cached? :/ [19:23:21] Reedy: Logged out (I've had no probs logged in, though some VPT comments are from regd users) [19:25:45] Reedy: if squid found a prior purge request, it's conceivable in principle that it might wrongly assume that it could intercept subsequent ones and return the cached response (which for anon useres would be the confirmation button, but same principle could apply to result of that too) [19:26:40] [[Tech]]; MarcoAurelio; /* L'il question */ Thank you.; https://meta.wikimedia.org/w/index.php?diff=5134272&oldid=5133325&rcid=3856505 [19:40:36] Not really [19:40:49] Old purge requests wouldn't just appear [19:43:09] Reedy: but it's intentional that action=history is cached? (and presumably is purged whenever the page view is invalidated) [19:44:11] As much as possible is cached for anonymous users [19:58:04] i am here [19:58:13] hate hate hate hate hate hate hate [19:58:14] hate [19:58:20] stab stab [20:02:38] Reedy: woooooonderful comment [20:03:55] Reedy: does it mean that ops could maybe deploy the disabled pages update at least for all but Mostlinkedtemplates and Mostcategories? [20:04:14] Say what now? [20:04:50] https://bugzilla.wikimedia.org/show_bug.cgi?id=44348#c6 --> https://gerrit.wikimedia.org/r/#/c/33713/ [20:05:34] I've no idea [20:05:53] What was apparently breaking on frwiki now isn't, so I'm just reverting that to see how it goes tonihgt [20:07:01] Nemo_bis: Do you want a full copy of that log posting? [20:07:26] well, the log from the cronjob [20:07:32] Reedy: yes! [20:07:53] Also, while you're at it, why not see how long disabled pages take too? :) [20:08:40] ideally we need to pick one wiki to try it on [20:08:46] I'm not letting it run over every wiki [20:09:07] yes, on one wiki [20:10:19] Nemo_bis: https://noc.wikimedia.org/~reedy/updateSpecialPages.log [20:10:35] I'll attach it to the bugs [20:10:39] preetty [20:10:43] good [20:16:45] Reedy: like, de.wiki has its own DB so you wouldn't affect any other wiki, the impact on servers would be easily checked. It has less links than fr.wiki (probably) but still among the biggest [20:17:17] I'm sure people will come and shout at me for lagging out the slaves [20:17:23] well, singular [20:18:05] Am I setting wgDisableQueryPageUpdate to array() for dewiki, then running update special pages? [20:21:34] there's an ignore option [20:21:54] you could maybe explicitly exclude the non-disabled ones to save some DB sweating? [20:22:00] override [20:22:16] * Reedy just does it [20:23:31] Reedy: all or just the disabled? [20:23:44] All [20:23:44] https://noc.wikimedia.org/~reedy/dewikispecialpages.log [20:23:48] Keep refreshing :D [20:24:24] haha [20:24:37] otherwise it would be like mwscriptwikiset updateSpecialPages.php dewiki --override --only=AncientPages [20:24:46] well, not wikiset [20:25:06] mwscript [20:25:07] ok, it's already doing one that is supposed to melt the DB [20:26:00] @replag [20:26:06] Have we stil got a bot for that? [20:26:07] Apparently not [20:29:03] Krinkle|detached: ^ [20:29:28] * Nemo_bis keeps https://de.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=dbrepllag&sishowalldb open [20:29:31] still 0 [20:30:49] let me see if I can find out which slave it's running on [20:30:57] It should show up in professor too [20:31:39] what info does ishmael provide? https://ishmael.wikimedia.org/?host=db1039 [20:32:14] That's what I was meaning [20:32:25] Though, the 3 eqiad slaves aren't doing a great deal [20:32:50] aren't they [20:33:18] QPS: 622.77 [20:33:50] | 2153625467 | wikiadmin | 208.80.152.165:49764 | dewiki | Query | 593 | Copying to tmp table on disk | SELECT /* AncientPagesPage::reallyDoQuery */ page_namespace AS namespace,page_title AS title,rev_t [20:33:53] wth [20:34:00] Why is it doing the query against a pmtpa slave? [20:34:08] Is it because I'm running from fenari? [20:34:29] Is it bad? [20:34:50] Not exactly bad, they're pretty idle [20:35:25] they're playing chess with each other [20:35:29] As long as it doesn't try to write to the pmtpa slaves [20:36:00] hm [20:36:10] Do you not have a labs account? [20:37:52] Oh, it works. I didn't think it could actually mean it. :) [20:38:30] > echo getRealmSpecificFilename( "$wmfConfigDir/db.php" ) [20:38:31] /home/wikipedia/common/php-1.21wmf7/../wmf-config/db-pmtpa.php [20:39:04] 'db1039' => 0, [20:39:09] Guess it makes sense [20:39:35] who is 208.80.152.165? [20:39:55] ah, fenari [20:40:45] It's now sending data [20:40:49] It took over 900 seconds.. [20:40:59] Ancientpages got 2000 rows in 16m 16.06s [20:41:27] that should be an easy one [20:41:57] but de.wiki is the only one with en.wiki to have non-default wgQueryCacheLimit [20:43:10] and the winner is.... Wielandsberg‏‎ (23:36, 22 November 2004) [20:43:39] Winner? [20:43:55] 1. in Special:Ancientpages [20:50:39] db35 starts complaining [20:50:54] is it getting lagged? [20:52:01] 4 s according to dbtree [20:52:35] now 0 [20:54:43] anything dbXX won't affect production [20:56:00] not running faker! [20:57:40] Nemo_bis: It's upto lonelypages [20:57:41] Though [20:57:41] Deadendpages got 1 rows in 13m 43.91s [20:57:42] lol. [20:57:46] haha [20:57:53] Lvova must be happy [20:58:14] https://de.wikipedia.org/wiki/Spezial:Sackgassenseiten [20:58:20] https://de.wikipedia.org/wiki/Jachya_Freeth [20:58:31] Created half an hour ago [20:58:34] we should give a barnstar to the article [20:58:36] ah [20:58:44] won't last long [20:58:47] indeed [20:58:54] Though, it'll be in the list for a while.. [20:58:55] :D [20:59:15] fyi , i am on the phone with juniper [20:59:16] this is db35 http://www.ruttkowski68.com/wp-content/uploads/2011/06/jachya_freeth_icarus_view_web.jpg [20:59:24] so i am gonna be ignoring irc [20:59:34] LeslieCarr: Is it outsourced on a weekend? [20:59:38] but if anyone asks, if someone could tell them that i am working on it [20:59:47] no, but usually it's only the more junior folks [20:59:55] usually if it has to escalated it's to someone at home [21:00:00] and they are hesitant to escalate [21:00:13] but so far this guy seems like he has some base knowledge and has asked only intelligent questions [21:00:14] so yay [21:00:19] Yay [21:00:28] anyways, ignoring irc mode on, but working on the problem :) [21:00:29] Have you posted on VPT recently? [21:00:43] um, don't remember [21:00:48] I'll look and update if necessary [21:00:49] hmpf, VPT is en.wiki only, use bugzilla [21:00:57] :p [21:01:05] Thanks for volunteeering Nemo_bis [21:01:26] i just linked enwiki's VPT on plwiki's VPT for this [21:01:32] probably people on other wikis did the same [21:02:04] Mostcategories [21:02:23] MatmaRex: yes bu on bugzilla thet get an email where there's an update and spread the word :) [21:04:17] Nemo_bis: tbh, as long as we targetted the idle pmtpa slaves for this sort of thing, it might be doable say weekly or something [21:05:29] Reedy: I think so! [21:05:42] I've used idle eqiad slaves for silly queries before [21:05:46] in my patch it's once per year... [21:06:08] no, twice [21:15:07] lol [21:15:23] Certainly if we extract out the regularily run ones too [21:15:56] Nemo_bis: Running it on enwiki next might not be a bad idea [21:16:02] Then we've got the worst case too [21:20:25] Reedy: en.wiki has $wgQueryCacheLimit set to 1000, though; maybe fr.wiki, with 5000, is worse for some of those queries? [21:44:10] i'm on advanced techsupport now [21:44:14] fyi [21:44:20] and back to ignoring irc and working on routers [21:44:56] good! [22:21:37] Mostlinked got 2000 rows in 1h 7m 44.67s [22:21:38] Nemo_bis: Mostlinked got 2000 rows in 1h 7m 44.67s [22:21:42] Snap [22:21:43] :) [22:22:01] and I wasn't even refreshing [22:22:11] I'd just looked now too [22:22:26] :) [22:22:29] db55 Lag: 3 [22:22:41] heh, Germans: Hilfe:Personendaten‏‎ (476,122 links) [22:22:53] scoldings: Hilfe:Signatur‏‎ (228,845 links) [22:52:40] heh Rows_examined 1.91548e+08 [23:03:39] @replag [23:05:23] trolol [23:23:20] fyi, still working on the routers, still ignoring irc [23:38:59] @replag [23:39:01] Krinkle: No replag currently. See also "replag all". [23:39:07] @replag all [23:39:09] Krinkle: [s1] db1017: 0s, db1001: 0s, db1042: 0s, db1049: 0s, db1050: 0s, db1043: 0s; [s2] db1034: 0s, db1002: 0s, db1009: 0s, db1018: 0s; [s3] db1019: 0s, db1003: 0s, db1010: 0s, db1035: 0s [23:39:10] Krinkle: [s4] db1038: 0s, db1004: 0s, db1011: 0s, db1020: 0s; [s5] db1039: 0s, db1005: 0s, db1026: 0s, db1021: 0s; [s6] db1006: 0s, db1022: 0s, db1027: 0s, db1040: 0s; [s7] db1041: 0s, db1007: 0s, db1024: 0s, db1028: 0s [23:41:18] Reedy: Crap, it can't update itself [23:41:34] because wmf-config/db.php no longer exists after git-remote update [23:41:42] Reedy: Which one should it include? [23:41:43] L-/ [23:43:22] I guess it needs to be aware of both, but it can't force a request to go to a certain data center [23:46:52] For most purposes, just use the eqiad one [23:47:21] the pmtpa one currently has the eqiad masters as their masters