[10:14:38] Tim-away: you can indeed change the ENUMs in that way [10:14:54] Alterations that modify only table metadata and not table data can be made immediately by altering the table's .frm file and not touching table contents. The following changes are fast alterations that can be made this way: [10:14:57] Changing the definition of an ENUM or SET column by adding new enumeration or set members to the end of the list of valid member values, as long as the storage side of the data type does not change. [10:17:46] that's a lot of caveats [10:18:51] it's obvious if you think of it [10:19:00] why is it obvious? [10:19:21] it uses numbers to store the enum items [10:19:51] if it needs to change from short int to long int, it can't just modify the metadata [10:20:27] adding them at the end comes because the numbers are always used in the orden in which they appear on the column definition [10:20:40] so ading items in the middle needs renumbering [10:20:58] (still, it shouldn't need to make a full table copy for that...) [10:21:19] the table definition could have the enums in both orders [10:21:45] what? [10:21:46] the order in which they were added, which is also the order they are stored, and secondly, the order they appear in the SHOW CREATE TABLE [10:22:06] yes, it could do that [10:22:13] then you could change the order in which they appear in the CREATE TABLE without changing the actual data [10:22:18] maybe I need to change my statement [10:22:29] it is obvious only if you know the way they did it :P [10:22:39] but I thought you meant the feature as a whole is obvious [10:23:07] i.e. easy [10:23:08] I refered to the restrictions [10:23:13] or unavoidable [10:23:38] which would be a funny statement since it took so long for them to introduce it, and even when they did, it is so limited [10:28:13] it seems to have been added in 3.x [10:28:36] although efficient appending was probably added in 5.1 [10:29:57] exposing the internal indexes to the user was probably a bad idea [10:32:01] it probably makes sense to do renumbering when you then run an order by for that column... [15:15:10] http://pt.planet.wikimedia.org isn't updating the feeds, new posts aren't shown there, what's happening? [15:16:13] I swear I've seen multiple discussions about planet but meh if I can remember [15:16:43] well, i know it will be replaced by planet2, but this one should be working [15:24:22] Hey guys, the database locked up again. [15:27:51] The lag is increasing and appears to have put enwiki in read only mode. [15:29:34] Found it, s3's having issues. [15:34:39] What makes you say s3? [15:35:29] I bet the s is correct [15:35:36] It's not [15:35:40] ! [15:35:40] ok, you just typed an exclamation mark with no meaning in the channel, good job. If you want to see a list of all keys, check !botbrain [15:35:41] it's teh enwiki master (s1) with issues [15:35:51] so the s was correct [15:36:08] ... [15:36:23] :) [15:37:43] (Cannot contact the database server: Unknown error (10.0.6.73)) [15:37:53] on en-wp [15:38:07] yup, we know [15:42:49] * Damianz blames Reedy :) [15:45:11] Sorry, busy. status.wikimedia.org said s3 was down [17:08:06] so, the static dumps are running again, finally. if the server load is too heavy, my cellphone number is in the user-agent :D [17:08:35] which static dumps? [17:09:36] HardDisk_WP: [17:12:23] apergos, remember when I bugged you two years ago? [17:12:34] * apergos tthinks back [17:12:37] er, not so much :-D [17:12:42] too many buggings... [17:13:11] lol :D anyway, i took up development again two days ago, when i saw that nothing had happened since then [17:13:42] which static dumps? [17:13:56] wikipedia html-dumps [17:14:02] dumpHTML? [17:14:07] yeah I don't know who owns that [17:14:11] nah, my own architecture [17:14:15] oh [17:14:22] where is it running, off-site? [17:14:41] yes. list generation on toolserver, html-fetching and processing on my own server [17:14:57] code's on github, https://github.com/msmuenchen/wiki-staticdumps [17:31:12] The DB seems to be back to normal, so you can update the topic. [17:32:21] better? [17:41:43] AaronSchulz: re: JobQueDB::claim() - have you tested for concurrency / innodb lock issues? [17:44:12] I've tested it with 16+ threads with normal jobs...tried 32 but it froze my box doing refreshLinks...it didn't give any DB errors though. I suppose one could just slam it with jobs that don't actually do anything and try 32+ threads. [17:44:28] torture testing :) [17:45:52] AaronSchulz: is the update in claim() autocomitted? [17:46:00] yes [17:46:24] same with ack() [17:46:59] ok, that explains it [17:50:37] AaronSchulz: is there a way to control how many concurrent jobs will run against any one wiki? [17:51:03] not really [18:02:11] Lol, AaronSchulz https://office.wikimedia.org/w/thumb.php [18:02:13] PHP fatal error in /usr/local/apache/common-local/php-1.21wmf1/thumb.php line 159: Call to a member function getTitle() on a non-object [18:17:05] Reedy: https://gerrit.wikimedia.org/r/#/c/27438/ [18:17:29] looks sensible [18:20:09] AaronSchulz: can there be a way? :) (per-wiki limit on concurrent jobs) [18:22:06] binasher: what do you want it for? [18:23:00] Oh. Deployment time [18:23:18] mostly just to prevent 500 job runners running select for update against the enwiki master at once.. although i know the new code does away with that [18:23:29] AaronSchulz: could really do with fixing Fatal error: Call to a member function getName() on a non-object in /usr/local/apache/common-local/php-1.21wmf1/includes/upload/UploadBase.php on line 558 too [18:28:29] mw16 aawiki nextJobDB::getPendingDbs 10.0.6.44 1146 Table 'fdcwiki.job' doesn't exist (10.0.6.44) [18:28:49] maybe that was ephemeral [18:29:49] Yeah, probably during the config earlier today [18:29:52] I wouldn't worry about it [18:34:53] notpeter: I forget, where is the job runner box list? [18:35:32] mw1-16 [18:35:34] in site.pp [18:36:04] I see a job fatal for srv231 [18:37:11] srv231 should no longer be running any jobs [18:37:16] that error may have been me stopping it [18:37:26] no it was an OOM [18:37:45] huh [18:37:48] when? [18:38:13] [10-Oct-2012 16:56:05] Fatal error: Allowed memory size of 157286400 bytes exhausted (tried to allocate 8388608 bytes) at /usr/local/apache/common-local/php-1.20wmf12/includes/parser/Preprocessor_DOM.php on line 1656 [18:38:15] Server: srv231 [19:05:43] Lots of stupid GC warning spams [19:14:16] <^demon> Reedy: About needing a suffix on shm_size? [19:14:28] nah [19:14:38] Warning: require() [function.require]: GC cache entry '/usr/local/apache/common-local/php-1.20wmf12/languages/Language.php' (dev=2049 ino=279663) was on gc-list for 601 seconds in /usr/local/apache/common-local/php-1.21wmf1/includes/AutoLoader.php on line 1150 [19:14:40] etc [19:16:26] what's gc-list? [19:16:50] <^demon> http://stackoverflow.com/questions/4656874/what-does-the-cryptic-gc-cache-entry-mean [19:19:47] <^demon> https://bugs.php.net/bug.php?id=58015 says more. Basically "it can happen if requests terminate early. If this is known/expected, the error is basically harmless" [19:20:02] <^demon> I concur with the guy who says it'd be nice if it was an E_NOTICE. [19:21:39] yeah [19:21:44] this warning noise is annoying [19:26:29] so, someone who writes php, maybe ^demon, would you be willing to alter the extdist cron to use the /mnt/originals nfs mount instead of '/mnt/upload6/private/E [19:34:13] <^demon> notpeter: http://p.defau.lt/?gzjGegY4_XtRX7Ki1C9r_w look right? [19:35:04] ^demon: oh, heh, probably :) [19:35:10] not sure how the script works, but yeah [19:35:58] I believe that ma_rk mentioned that the script acted via http, but if that's not the case, then yes, that should be perfect and I will deploy post-haste :) [19:37:35] <^demon> Well anyone in mortals can deploy this. https://gerrit.wikimedia.org/r/#/c/27443/ [19:37:49] <^demon> I would like if Reedy or RoanKattouw sanity checked me though. I haven't touched ExtDist in ages. [19:37:55] <^demon> Or AaronSchulz. [19:37:57] <^demon> Someone :) [19:38:09] * RoanKattouw is running away for lunch, will look after [19:38:15] Looks fine [19:38:21] as long as said files have been moved to those locations ;) [19:38:39] <^demon> Well, as long as apache can write to the location it can just make new files on demand :p [19:38:56] those files all exist [19:39:10] and are owned by apache:apache [19:39:22] (owners should all be the same as before move to netapps) [19:40:07] * AaronSchulz is still baffled by "Couldn't resolve host 'ms-fe.pmtpa.wmnet': Failed to obtain valid HTTP response" spam [19:40:11] it's easy enough to test [19:40:39] * Damianz wouldn't be surprised if there are some that aren't [19:41:42] * Reedy chowns Damianz [19:41:52] :o what do I own! [19:43:15] Damianz: something diferent than before! [19:43:24] and by that I mean I stole your bike.... [19:44:06] Oh, you where the dude who stole my bike ^.^ [19:57:02] AaronSchulz: one more question about the jobq refactor.. other than the schema migrations, is it just waiting on a review from tim? [19:58:37] yep [21:09:45] why does labsconsole have only 7 days-lasting cookies? is this a policy for nonstandard domains or whatever? [21:11:41] Ask in -labs? [21:12:52] Nemo_bis: because you can delete virtual machines with the credentials [21:12:55] and do other nasty things [21:13:07] Ryan_Lane: ok [21:13:18] 7 days is actually kind of long, really ;) [21:13:21] Reedy: I was too lazy to check if it's only in labs :p [21:13:32] it's just labs [21:14:30] thanks [21:15:00] and sure, I don't mind 7 days, I was just curious :) [21:36:11] binasher: http://ceph.com/wiki/Simple_Ideas_For_Auto_Insurance [21:36:58] meh [21:37:09] * AaronSchulz looked at the history ;) [21:37:19] quite spam-filled site [21:37:31] http://ceph.com/wiki/Thanks_Lai_was_lovely_to_meet_singlepeople.co.za_you_too_xx [21:37:39] that was me clicking random page :( [21:37:40] http://ceph.com/wiki/Special:RecentChanges [21:38:07] since I saw people had the emergency under control including knowing the causes [21:38:14] I am going to crawl into my bed [21:38:18] phone on vibrate [21:38:25] see folks tomorrow [21:38:43] bye [22:23:29] TimStarling: so is https://bugzilla.wikimedia.org/show_bug.cgi?id=39493 less severe now or something? [23:16:26] Merlissimo: ping [23:16:46] binasher: yes? [23:17:02] you run MerlIwBot, correct? [23:17:07] yes [23:17:11] uh oh :) [23:17:31] i thought you had stopped it from performing template edits on zhwiki? [23:18:11] i added a job queue size request [23:18:40] hm. what's its limit set to? [23:20:09] i have to look at the source, wait a minute [23:21:40] if (!isHumanMode && lp.getNs() == 10 && lp.getSite().getJobQueue() > 100000) return false; [23:24:39] that probably uses estimateRowCount() to get the job queue size [23:24:47] so it's probably off by a factor of 5 [23:25:21] it certainly isn't working [23:25:25] if am request job queue size using http://zh.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=statistics [23:25:41] heh [23:25:51] on zhwiki select max(rev_timestamp), rev_id from revision where rev_user_text = "MerlIwBot" == 20121010223412, and the jobq was several times 100k at the time [23:26:22] Merlissimo: how often does it check it? [23:26:52] of course even if it checks it after every edit, a single edit can make a 200k job queue, right? [23:27:01] TimStarling: where is the apache/squid conf that makes /w/ use /live-1.5/? I can't find it in the obvious places. [23:27:07] MerlIwBot is the biggest template editor on zhwiki by an order of magnitude and it does seem responsible for the edits that are filling the queue (i.e. rev_comment: 机器人添加：[[tl:Padron:Country data England]]) [23:27:09] always when requesting template source code, so its uptodate [23:27:18] AaronSchulz: symlink in common/docroot [23:27:30] I think it's just a symlink from docroot [23:27:31] * AaronSchulz didn't see a link there [23:27:33] or is that only for special domains [23:27:34] * AaronSchulz looks again [23:27:38] (source code is only requested if there is a pending edit, interwiki are read using api, not by source code) [23:27:41] AaronSchulz: I told you earlier today? [23:28:16] lrwxrwxrwx 1 brion wikidev 23 2006-02-13 13:41 w -> /apache/common/live-1.5 [23:28:38] nice to have usernames on those files now :) [23:29:18] binasher: babel editing shouldn't be a problem, but country data is heavily included [23:29:34] Krinkle: ? [23:29:46] TimStarling: which subdir is that in [23:29:47] * AaronSchulz greps [23:30:03] /home/wikipedia/common/docroot/wikipedia.org [23:30:19] AaronSchulz: for all I know there was a big gap between you asking and me saying, not sure. But I saw you asking earlier. [23:30:22] hm, http://zh.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=statistics is off by around 130k right now, though it varies a bunch.. i suppose it hits a random slave [23:30:44] TimStarling: I see, I was looking at www.wikipedia.org earlier ;) [23:31:04] yeah, that's just www [23:31:33] Merlissimo: country data template edits seem to be a big problem on zhwiki. i'm not sure what to do about that, since it must be related to how they're used throughout the wiki [23:32:33] binasher: yeah, $dbr = wfGetDB( DB_SLAVE ); $dbr->estimateRowCount( 'job' ); [23:32:40] binasher: a solution would be to move interwikis to subpage [23:33:08] didn't I tell Stevenliuyi to get rid of those country data templates? [23:36:12] * AaronSchulz likes how find/grep is the best documentation ;) [23:36:23] sure, discovery [23:36:45] binasher: my interwiki is only editing so much within template namespace, because it is the only bot who can edit in template namespace without any risk [23:36:48] back when I was doing ops work more regularly, I used to trust the documentation which I had personally written recently [23:36:59] now there isn't any of that, so I don't trust any documentation [23:37:06] I just hone my discovery skills [23:37:16] heh [23:38:00] still lots of pages in zhwiki with a node count around 1M [23:38:09] like this: https://zh.wikipedia.org/wiki/%E4%B8%B9%E5%A5%A5%E6%B3%BD [23:38:11] Merlissimo: well.. there is operational risk [23:38:15] so I guess it wasn't done [23:40:16] binasher: yes, but bot has no problems with subpages (it do not need any special local config like pwd) or include/onlyinclude parts. [23:41:21] binasher: we can probably just delete all those pages if they are a real problem [23:41:53] binasher: my log contain 11 failed edits on zh template namespace after 20 utc (the reason is not logged, but i think that's because of high job queue count) [23:42:16] TimStarling: are there lots of non-existing templates transcluded? [23:42:38] I don't think so [23:42:42] (with babel sounds likely) [23:44:42] or maybe we can expand all their templates, that would probably be better than deleting [23:46:35] would that help the job queue problem? [23:47:02] TimStarling: on which job queue size would you think that i should stop editing? [23:47:07] probably not [23:47:52] Merlissimo: [23:48:05] can you stop country data edits on zh entirely? [23:48:08] let me look at runJobs.log for a minute [23:48:35] right... [23:49:47] you should put the language links on the doc template [23:49:57] maybe first TimStarling could have a look if country data is really the reason [23:49:59] either that or don't put language links on templates at all [23:50:03] yes it's the reason [23:50:30] TimStarling: country data template have no doc subpage, so we have to take another one [23:50:37] template edits generally are the reason [23:51:03] the edits to {{Cite journal}} etc. are probably even more damaging [23:51:08] but i don't know the naming conventions on zhwiki. on dewiki we use /Meta for categories and interwikis [23:51:22] fine, so if a template has no doc subpage, don't edit it [23:51:37] gn8 folks [23:52:02] we could just ban MerlIwBot [23:52:42] binasher: :-P [23:52:46] you're adding a miniscule amount of value to the wiki for an enormous CPU cost [23:53:19] I don't think you will believe how much CPU time it uses, so maybe I should add it up [23:54:17] if i would know which subpage i could add something that my bot automatically creates these subpages, so that only one edit for each template is needed for the hole future [23:55:50] TimStarling: you should also measure it in CO2 :p [23:56:44] has sb added a graph for zhwiki job queue at the last time? [23:56:44] Merlissimo: on most english wikis they transclude cat and iw's from /doc (which is also transcluded, for the same reason: avoid forcing a purge of articles that use a template when non-template edits are made) [23:57:03] Citation/identifier is also a problem on zhwiki [23:57:38] wow, flood. brb, client buffer going crazy [23:58:04] Krinkle: yes i know, but not updating iws is not the solution, but auto creation of subpages could be one i think. [23:59:41] hmm, using sqlite for one of my testwikis is actually kind of cute, I wonder how fast it would be if imported a bunch of stuff