[01:17:13] !log tstarling synchronized wmf-config/CommonSettings.php [01:17:22] Logged the message, Master [01:27:09] !log tstarling synchronized wmf-config/CommonSettings.php [01:27:19] Logged the message, Master [02:22:47] !log LocalisationUpdate completed (1.20wmf11) at Wed Sep 19 02:22:47 UTC 2012 [02:22:59] Logged the message, Master [02:42:01] !log LocalisationUpdate completed (1.20wmf12) at Wed Sep 19 02:42:01 UTC 2012 [02:42:11] Logged the message, Master [02:48:03] !log reedy synchronized php-1.20wmf11/extensions/PageTriage [02:48:13] Logged the message, Master [02:48:57] !log reedy synchronized php-1.20wmf12/extensions/PageTriage [02:49:06] Logged the message, Master [03:50:55] Reedy: https://en.wikipedia.org/wiki/Wikipedia_talk:Page_Curation#Tag_bug.3F [03:51:56] bloody localisation cache [04:04:15] !log reedy Started syncing Wikimedia installation... : Rebuild localisation caches for pagetriage [04:04:25] Logged the message, Master [04:25:25] Reedy: is that why the wiki's so slow? [04:26:15] No idea [04:26:23] Not seen anyone complain about scap doing that before [04:28:36] for some reason the page load times are taking forever, and they appear to have started around 4:00 UTC [04:28:49] speeding up now [04:30:33] nevermind... page load times still quite slow [04:35:23] Reedy: see operations channel [04:46:16] !log reedy Finished syncing Wikimedia installation... : Rebuild localisation caches for pagetriage [04:46:26] Logged the message, Master [04:46:37] much faster now [08:01:10] test [09:51:41] devunt: read :) [09:51:47] ok with some huge latency [09:51:50] but still, it worked! [10:33:56] comment on https://blog.wikimedia.org/2012/09/18/server-decommissioning-donations-sept2012/#comment-148898 in case ssomeone here knows: [10:34:01] I love these donations! It’s very nice that you manage to invest time on this (quite tedious) process. [10:34:05] As a clarification, is the donation restricted to USA organizations because there are usually enough requests and it’s easier to manage this way, or is there some legal reason behind it? [10:34:50] shipping probably [10:35:20] also making sure you don't ship to restricted countries etc etc [10:36:25] restricted countries for what, embargo? [10:36:29] like Iran? [12:02:00] !log hashar synchronized wmf-config/InitialiseSettings.php '(bug 40163) Try to fix ltwiki import source for betawikiversity' [12:02:11] Logged the message, Master [12:13:46] !log hashar synchronized wmf-config/InitialiseSettings.php '(bug 39206) Namespaces configuration for se.wikipedia' [12:13:56] Logged the message, Master [12:18:12] !log hashar synchronized wmf-config/InitialiseSettings.php '(bug 38840) Namespaces configuration on uz.wikipedia' [12:18:22] Logged the message, Master [12:24:35] !log hashar synchronized wmf-config/InitialiseSettings.php '(bug 38840) Namespaces configuration on uz.wikipedia' [12:24:45] Logged the message, Master [12:28:17] !log hashar synchronized wmf-config/InitialiseSettings.php '(bug 39264) Add Tudalen: and Indecs: namespaces to cy.wikipedia' [12:28:26] Logged the message, Master [13:45:09] what would be the reason for not having one wikipedia attached to sul? [13:56:38] Danny_B|backup: private wikis cannot tie into SUL [13:57:06] RobH: i said wikipedia - that's not private ;-) [13:57:55] oh, sorry, read it as wiki, heh [13:57:59] n oclue. [13:58:01] no clue even [13:58:29] RobH: b40359 if you're interested [13:58:49] in a training class, not technically working ;] [13:58:55] i just chimed in cuz i could. [13:59:23] i see [14:46:39] Danny_B|backup: maybe it's a retortion after their template disasters [14:46:45] or database strike [14:47:21] uhu [14:49:35] Danny_B|backup: it's just you, closed invalid [14:49:48] join the crowd of users who lost their accounts [14:49:49] Nemo_bis: what? [14:49:57] you only have to pick one bug [16:38:17] l10n/i18n office hours in #wikimedia-office now [16:43:17] mutante: hey, wanted to get your advice on https://bugzilla.wikimedia.org/show_bug.cgi?id=33406 -- "Is the Bugzilla configuration (or patched sources) accessible somewhere?" [16:43:29] It's not. ought it be? [16:49:12] sumanah: seems a valid bug [16:49:25] there are some crontabs in puppet, I don't know anything else [16:54:07] <^demon> The customizations are all in SVN. [16:54:19] <^demon> The configuration isn't public because BZ doesn't store it in a config file. [16:54:22] <^demon> It's all done via the UI [16:54:53] but after it's done via UI isn't it stored someplace? [16:54:56] or is it in the db? [16:57:06] AaronSchulz: I think apergos was looking for you [16:57:17] I pinged him in another channel [16:57:23] ok, nm [16:57:34] thanks [16:58:08] <^demon> sumanah: I can't remember to be honest, it's been ages since I poked BZ. [16:58:21] ok, nm, thanks [17:26:21] !log aaron synchronized php-1.20wmf12/thumb.php 'deployed 7be7e496ae2d6e4b082278ab4ace8c1b4b08e433' [17:26:31] Logged the message, Master [17:30:09] !log py synchronized wmf-config/lucene.php [17:30:19] Logged the message, Master [17:33:50] Any BZ admins around? [17:34:01] https://bugzilla.wikimedia.org/show_bug.cgi?id=40319 and https://bugzilla.wikimedia.org/show_bug.cgi?id=40363 [17:38:37] !log py synchronized wmf-config/lucene.php [17:38:47] Logged the message, Master [18:06:58] what happened with testwiki? [18:07:32] * robla looks [18:08:20] Alchimista: nothing that I can tell. Is there something wrong? [18:09:29] robla: sorry, i went to http://testwiki.org/, and i got a message telling that it was down, and a new managment service where planed [18:10:05] oh, I thought you were talking about test.wikipedia.org . testwiki.org isn't ours (I don't think) [18:10:29] yap, i wanted test.wikipedia, but i'm not sure, i went to testwiki. [18:11:19] is there any wiki suitable to be used for newbies bot developers? test.wikimedia? [18:12:01] !log demon rebuilt wikiversions.cdb and synchronized wikiversions files: Swapping all non-pedias to 1.20wmf12 [18:12:10] Logged the message, Master [18:13:25] Looks rather uneventful [18:13:45] Alchimista: test2.wikipedia.org is a good choice. [18:13:48] sumanah: ehm, yeah. I guess the answer is a straight "kind of", as in there is no single place that has all the sources/config but when it has been patched the patches have been public on tickets, so one could find them and list them but that's not really convenient [18:13:56] (our naming is admittedly terrible) [18:14:22] testtoo [18:14:30] <^demon> We should really swap test and test2. [18:14:33] <^demon> :) [18:16:00] robla: how often does the content disapears? I'm creating some doc's to those who want to start in pywikipediabot, so it would be interesting if the pages didn't go away [18:17:07] Alchimista: for stuff that you'd like to keep around, mediawiki.org is your best bet. nothing is guaranteed on test* [18:18:43] ^demon: I'm thinking test should be called "staging", and test2 should be called "rc" (for "release candidate"), or maybe "gamma" (post beta) [18:19:42] <^demon> Perhaps. perhaps. [18:32:52] hey, i'd like to do an off-schedule sync-dir on the E3 experiments extension.. any objections? [18:34:15] <^demon> ori-l: Check we robla, we just did a 1.20wmf12 deploy to all non-wikipedias. [18:34:19] <^demon> s/we/with/ [18:34:54] ori-l: gimme a couple of min to check some homepages... [18:35:39] thanks robla, no rush [18:38:18] ori-l: go for it [18:38:42] robla: thanks! [18:39:42] Warning: apc_store() [function.apc-store]: Potential cache slam averted for key 'zhwiktionary:conversiontables:zh' in /usr/loc [18:39:42] al/apache/common-local/php-1.20wmf12/includes/objectcache/APCBagOStuff.php on line 59 [18:45:30] oh, that might be a good thing. is that the code Tim just added? [18:45:57] No idea [18:46:14] There's a few of these appearing too [18:46:17] Warning: require() [function.require]: GC cache entry '/usr/local/apache/common-local/php-1.20wmf11/includes/Sanitizer.php' (dev=2049 ino=2119300) was on gc-list for 601 seconds in /usr/local/apache/common-local/php-1.20wmf12/includes/AutoLoader.php on line 1150 [18:49:01] nope, doesn't appear to be that [18:50:27] binasher: ^ [18:51:05] did we ever fully figure out the issues from this past weekend? [18:51:05] ugh, i need to fix a mistake i made. not syncing yet, sorry. i'll ask again later when i'm actually ready; sorry. [18:56:15] robla: beyond it being zhwiki related template expansion memory leaks in libxml space a la ocwiki, no [19:04:31] Reedy: were you able to see which requests resulted in the zhwikitionary apc cache slam warning? (that is not from new tim code) [19:04:50] I didn't actually look for it in the logs [19:04:51] hang on [19:05:41] It's 12.04 [19:05:52] Sep 19 18:38:06 10.0.2.197 apache2[15221]: PHP Warning: apc_store() [function.apc-store]: Potential cache slam averted for key 'zhwiktionary:conversiontables:zh' in /usr/local/apache/common-local/php-1.20wmf12/includes/objectcache/APCBagOStuff.php on line 59 [19:05:58] damn you apache logs [19:06:26] 12.04 as in the apc build it comes with? [19:07:13] Presumably [19:07:29] Looks like these and the GC cache entry warnings are all coming from 12.04 boxes [19:08:42] apc.slam_defense => On => On [19:08:50] yeah.. the build on lucid doesn't have that at all [19:09:36] ah....that makes sense [19:10:00] yay for os upgrades [19:10:03] "Prior to APC 3.1.4, the default value was "0" (disabled)." [19:11:14] That sounds rather suspect ;) [19:11:41] "Deprecated by apc.write_lock" [19:11:42] heh [19:11:54] deprecated.. so turn it on! [19:12:03] fuck yeah [19:12:05] http://www.php.net/manual/en/apc.configuration.php#ini.apc.write-lock -- also defaults to on [19:12:35] yay for apt-cacher-ng [19:13:06] apc.write_lock is enabled on our lucid apaches too, and seems reasonable [19:13:44] did we ever figure where the extra bits traffic was coming from? http://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Bits+application+servers+pmtpa&m=load_one&s=by+name&mc=2&g=network_report [19:14:59] ouch [19:15:32] robla: its not extra bits traffic [19:16:17] Reedy: the GC cache entry warning pasted above could be a sign of apache procs dying / crashing / maybe just php killing execution after hitting memory limits [19:16:30] Fair enough [19:16:37] They just seemed to be new, and somewhat regular [19:16:58] apc doesn't decrement ref counts when a proc crashes and that can be logged in the gc code when "dead->value->ref_count > 0" [19:17:02] hmm [19:18:30] * robla heads out to lunch [19:18:47] the fatal log has a lot of max exec time exceeded messages all for the same arwiki request [19:18:58] its been going on for at least a few days [19:19:09] What is the request? [19:20:11] http://ar.wikipedia.org/wiki/أسماء_الله_الحسنى [19:22:18] Wheeee [19:22:28] Error: 1205 Lock wait timeout exceeded; try restarting transaction (10.0.6.47) for updating user_touched [19:22:33] "names of god" [19:23:10] Reedy: domas said yesterday that he opened a bugzilla ticket for that a while ago [19:23:59] heh, yeah [19:24:47] Reedy: siebrand is making a bugfix, can you deploy it for wmf12 if I backport? [19:27:26] Nikerabbit: sure [19:27:26] 110,091 bytes [19:27:48] binasher: that page is getting restricted by pool counter it seems [19:28:42] Oh, or just a customised error message.. [19:29:25] it takes an epic time to parse even w/o poolcounter in the mix [19:29:34] based on loading some oldids [19:30:29] Lol [19:30:36] binasher: it's a page about Allah! [19:30:41] That's why it has no pictures ;D [19:36:45] Reedy: uuh twn is slow, the relevant change is https://gerrit.wikimedia.org/r/24321 [19:37:44] binasher: I'm assuming the oldid loading is because of FlaggedRevs [19:38:05] whats the best way to nicely shut down a toolserver bot? [19:38:16] binasher: kickban? [19:38:22] that's where the zhwiki template edits appear to be coming from [19:38:27] ah [19:38:40] mediawiki has /kickban? ooh [19:38:48] what bot is it? [19:38:58] MerlIwBot [19:39:02] it can be locked by a steward (or maybe yourself) if it's causing emergency disruption [19:39:09] blocking the Ip is a bad idea [19:39:18] blocking by ip would block toolserver :) [19:39:39] We can take care of it. [19:39:41] well, only a few dozens bot :) [19:39:50] Nemo_bis: more like several dozens [19:39:51] *drums* [19:39:53] my wikipedia account isn't actually an admin [19:39:55] binasher: link? [19:40:01] http://en.wikipedia.org/wiki/User:MerlIwBot [19:40:58] afaik, only its activity on zhwiki needs to be stopped [19:41:04] that bot is blocked on enwiki hmmm [19:41:08] lemme see zhwiki [19:41:19] lol, blocked for edit warring [19:42:01] binasher: what exactly is it doing wrong? [19:42:56] Snowolf: removing global bot flag would probably stop it on zh.wiki until the owner comes back i think? [19:43:08] anyway lock is fine too [19:43:10] !log reedy synchronized php-1.20wmf12/extensions/CategoryTree 'Category tree to master' [19:43:19] Nemo_bis: a local block might be enough, depending on the issue [19:43:20] Logged the message, Master [19:47:17] binasher: could you please explain the issue on this page? -> https://de.wikipedia.org/wiki/Benutzer_Diskussion:Merlissimo#MerlIwBot.40zhwiki [19:47:37] Snowolf: some templates on zhwiki are triggering memory leaks and that bot appears to be doing null edits of templates in enough volume to cause infrastructure issues [19:48:02] sure, thanks [19:48:07] Thank you :) [19:48:17] Auf Deutsche, bitte. [19:48:19] binasher: i am not doing null edit [19:48:25] *derp* [19:48:40] Did I really miss that Merlissimo_ was here? Sorry xD [19:48:56] What is it doing then? ;) [19:49:13] * Snowolf leaves the matter to Merlissimo_ and the sysadmins -- lemme know when to unblock it or something [19:49:19] sigh i looked for Merlissimo withoud underscore [19:49:28] Merlissimo_: what are you doing? [19:49:43] Snowolf: i was just ping here in irc by your link to my talk page [19:49:54] still reading the log of this channel [19:49:56] Merlissimo_: I'm glad it did, I'm sorry I didn't check it before [19:51:02] binasher: simple interwiki bot running unmodified since long time [19:51:49] edits of templates on zhwiki such as [19:51:51] | 20120919111006 | 10 | Location_map_Australia_Victoria | [19:51:52] | 20120919110949 | 10 | Location_map_USA_Nevada | [19:51:53] | 20120919110657 | 10 | Location_map_Armenia | [19:52:04] are causing a lot of pain [19:52:38] binasher: job queue of what is the problem? [19:52:40] the root cause is likely one or more other templates on zhwiki included on articles the have to be reparsed as a result of those edits [19:53:42] Snowolf: do you have a documentation template on zhwiki that includes interwikis from a subpage? [19:54:02] Merlissimo: he's not from zhwiki [19:54:03] Merlissimo: I have no idea, I'm unfamiliar with zhwiki [19:54:30] I blocked thru my steward powers while the stuff gets figured out given the request from a sysadmin [19:55:00] I could have locked but figured it would be better to get a local block for a moment than lock the whole thing [19:57:26] Reedy: thank you! [20:00:20] binasher: how should i change my bot? [20:00:41] !log Deleting php-1.20wmf9 from mediawiki-installation [20:00:51] Logged the message, Master [20:01:45] Merlissimo: I don't know that your bot is doing anything wrong or needs to be changed.. we just need it paused until we can fix other issues with zhwiki [20:02:07] and why only zhwiki, my bot is editing on all wiki except enwiki? [20:02:34] Merlissimo: it's probably something wrong with the templates on the pages your bot is editing on zhwiki [20:04:40] robla: i don't think so, i think its something wrong with zhwiki native templates that are included by pages that also include templates that bot is editing [20:05:12] the plot thickens :) [20:05:26] job queue 771003 on zhwiki - wow [20:05:40] just a bit [20:05:45] TRUNCATE TABLE [20:07:12] Snowolf: i have disabled editing template namespace on zhwiki in bot code [20:07:19] the site issues over the weekend that are only partially solved by a hackish workaround (killing apache children after only 4 requests) were caused by general zhwiki traffic, with no particular article standing out.. some zhwiki parses just leak a couple mb at a time [20:07:27] Merlissimo: thank you [20:08:28] binasher: editing article namespace is ok, isn't it? [20:08:55] Merlissimo: yeah, i think so [20:10:32] Merlissimo: removed the block, sorry again that I didn't poke you first, didn't realize you were arond [20:11:45] Merlissimo: generally speaking, on big wikis, I imagine that editing in the template namespace with a bot has the potential to trigger thousands of reparses at once [20:12:25] (or at least, thousands to get dropped into the job queue) [20:13:10] robla: on dewiki all templates included more than 1000time have a subpage for interwikis [20:14:09] subpages included at the noinclude part so not invalidate template usages [20:14:29] is there a graph about job queue size over time? [20:14:48] yeah [20:14:58] I think we've got one for enwiki, but I don't think we've got one for the others [20:15:06] I thought we had a total one? [20:15:10] my bot could monitor job queue size, but what are the critical values? [20:16:12] we hava a total job queue pop/pull graph, and an enwiki one based on whats actually in the db.. nothing specific to other projects though [20:17:35] * robla doesn't remember where the enwiki job queue one is [20:18:06] https://gdash.wikimedia.org/ [20:18:13] Merlissimo: https://gdash.wikimedia.org/dashboards/jobq/ [20:18:25] currently my bot is monitoring dbreplag only. the values is used to slow down editing speed on a hole cluster [20:18:40] I don't like maxlag [20:19:21] Reedy: i am not using maxlag [20:19:36] Maxlag comes from dbreplag [20:19:56] essentially, max( dbreplag ) [20:20:00] maxlag only throws away all submitted content [20:20:51] all my edit have a maxlag=12, but my bot slows down before it could hit this high value [20:21:37] Reedy: that graph is global job queue sum? [20:21:45] added/removed [20:21:53] we also have.. [20:22:17] http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Miscellaneous%20pmtpa&h=spence.wikimedia.org&v=770&m=enwiki_JobQueue_length&r=hour&z=small&jr=&js=&st=1348086106&z=large [20:22:27] Where's the other.. [20:25:47] how are job executed? cluster or wiki based? (each wiki has its own queue) [20:27:02] yeah, wikibased [20:32:08] !log Deleting php-1.20wmf10 localisation cache from mediawiki-installation [20:32:17] Logged the message, Master [21:00:08] gn8 folks [21:22:00] twinkle 1.0 seems to have stopped working. anyone know of a major change that might be causing this? [21:22:11] HTML5 maybe? [21:22:43] dunno, you are the expert :) [21:26:03] I should find AzaToth [21:26:04] asap [21:27:52] I'd go with HTML5 too [21:27:58] In fact, it almost certainly is [21:28:09] Given that Twinkle 1.0 broke the first time HTML5 was intrroduced laast year [21:40:52] would anyone object to me syncing extensions/E3Experiments to enwiki? [21:45:34] does someone know what patterns are accepted by Special:Nuke? [21:46:56] SQL wildcards, I believe [21:47:15] I seem to remember looking it up for someone and we couldn't work out the rationale [21:48:09] sigh [21:48:58] Reedy: it worked, thanks [21:49:07] %lol% [21:50:10] Reedy: you really saved me :) [22:07:27] !log olivneh synchronized php-1.20wmf11/extensions/E3Experiments [22:07:36] Logged the message, Master