[00:00:51] PROBLEM - SSH on db29 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:01:54] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [00:03:31] !log db29 os installed, handed off to peter [00:03:33] Logged the message, RobH [00:03:33] RECOVERY - SSH on sq48 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [00:04:18] RECOVERY - SSH on db29 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [00:07:12] New patchset: RobH; "db61 in sandbox" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48976 [00:07:27] New review: Pyoungmeister; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/48862 [00:07:54] PROBLEM - Puppet freshness on tin is CRITICAL: Puppet has not run in the last 10 hours [00:08:31] New review: RobH; "Patch Set 1: Verified+2 Code-Review+2" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48976 [00:08:40] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48976 [00:09:28] New patchset: Pyoungmeister; "reviving db29 for use by pgehres" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48862 [00:09:33] RECOVERY - Frontend Squid HTTP on sq48 is OK: HTTP OK HTTP/1.0 200 OK - 631 bytes in 0.007 seconds [00:09:57] New review: Pyoungmeister; "Patch Set 3: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/48862 [00:10:07] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48862 [00:13:14] New patchset: RobH; "reclaiming copper, magnesium, and zinc for use elsewhere" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48977 [00:15:52] New patchset: RobH; "reclaiming copper, magnesium, and zinc for use elsewhere" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48977 [00:16:39] New review: RobH; "Patch Set 2: Verified+2 Code-Review+2" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48977 [00:16:51] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48977 [00:17:03] RECOVERY - swift-account-reaper on ms-be5 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-reaper [00:17:30] paravoid: ok, i just removed copper, magnesium, and zinc from site.pp and added to decom, so reclaimed =] [00:17:48] great, thanks [00:18:22] srv266, srv278? [00:18:31] and virt1004? [00:18:42] RECOVERY - swift-account-reaper on ms-be3 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-reaper [00:18:44] virt1004 is in decom so it shoudl come out of nagios [00:18:49] it'll come back later [00:18:51] cool, great [00:19:08] the srvs i have not gotten to yet but will, i wanna make sure i have a long history in each ticket so no one complains when i kill them [00:19:28] I don't think anyone will complain [00:19:36] they've been acked by mark too [00:24:21] New patchset: Hoo man; "Fix CORS for wikidata-test" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48979 [00:25:27] PROBLEM - Frontend Squid HTTP on sq48 is CRITICAL: Connection refused [00:28:45] PROBLEM - NTP on db29 is CRITICAL: NTP CRITICAL: Offset unknown [00:30:14] New patchset: Faidon; "Adjust knsq24 for its broken disk" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48980 [00:30:50] New review: Faidon; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/48980 [00:30:59] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48980 [00:32:12] RECOVERY - NTP on db29 is OK: NTP OK: Offset -0.007966518402 secs [00:35:52] New patchset: Hoo man; "Fix CORS for wikidata-test" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48979 [00:37:17] New patchset: Aaron Schulz; "Modified jobs-loop script to keep a fuller pipeline." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48981 [00:37:44] New review: Aaron Schulz; "Patch Set 1: Code-Review-1" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/48981 [00:38:44] New patchset: Hoo man; "Fix CORS for wikidata testing instances" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48979 [00:46:51] New patchset: Faidon; "Fix silly typo with previous commit" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48982 [00:47:20] New review: Faidon; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/48982 [00:47:28] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48982 [00:47:53] New patchset: Ryan Lane; "Add salt reactors for bootstrapping puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48983 [00:48:06] RECOVERY - Frontend Squid HTTP on sq48 is OK: HTTP OK HTTP/1.0 200 OK - 631 bytes in 0.007 seconds [00:52:28] http://test.m.wikipedia.org/ appears to be returning 503s [00:54:10] New patchset: Ryan Lane; "Add salt reactors for bootstrapping puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48983 [00:57:13] New patchset: Ryan Lane; "Add salt reactors for bootstrapping puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48983 [00:59:39] PROBLEM - Frontend Squid HTTP on knsq24 is CRITICAL: Connection refused [01:00:15] PROBLEM - Frontend Squid HTTP on knsq26 is CRITICAL: Connection refused [01:00:18] New review: Andrew Bogott; "Patch Set 1: Verified+2 Code-Review+2" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48941 [01:00:30] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48941 [01:00:51] PROBLEM - Frontend Squid HTTP on knsq20 is CRITICAL: Connection refused [01:01:27] PROBLEM - Backend Squid HTTP on knsq26 is CRITICAL: Connection refused [01:01:54] PROBLEM - Frontend Squid HTTP on knsq16 is CRITICAL: Connection refused [01:02:30] PROBLEM - Backend Squid HTTP on knsq20 is CRITICAL: Connection refused [01:02:48] PROBLEM - Frontend Squid HTTP on knsq22 is CRITICAL: Connection refused [01:02:57] PROBLEM - Backend Squid HTTP on knsq16 is CRITICAL: Connection refused [01:03:15] PROBLEM - Backend Squid HTTP on knsq22 is CRITICAL: Connection refused [01:03:51] PROBLEM - Frontend Squid HTTP on knsq27 is CRITICAL: Connection refused [01:04:13] that's me, fixing [01:05:30] PROBLEM - Backend Squid HTTP on knsq27 is CRITICAL: Connection refused [01:05:37] dammit [01:08:02] New patchset: Ryan Lane; "Add salt reactors for bootstrapping puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48983 [01:11:30] RECOVERY - Frontend Squid HTTP on knsq22 is OK: HTTP OK HTTP/1.0 200 OK - 821 bytes in 0.236 seconds [01:11:30] RECOVERY - Backend Squid HTTP on knsq20 is OK: HTTP OK HTTP/1.0 200 OK - 661 bytes in 0.358 seconds [01:11:39] RECOVERY - Backend Squid HTTP on knsq16 is OK: HTTP OK HTTP/1.0 200 OK - 661 bytes in 0.469 seconds [01:11:48] RECOVERY - Frontend Squid HTTP on knsq24 is OK: HTTP OK HTTP/1.0 200 OK - 1573 bytes in 0.525 seconds [01:11:57] RECOVERY - Backend Squid HTTP on knsq26 is OK: HTTP OK HTTP/1.0 200 OK - 1419 bytes in 0.237 seconds [01:11:58] RECOVERY - Backend Squid HTTP on knsq22 is OK: HTTP OK HTTP/1.0 200 OK - 659 bytes in 0.238 seconds [01:12:15] RECOVERY - Frontend Squid HTTP on knsq16 is OK: HTTP OK HTTP/1.0 200 OK - 819 bytes in 0.234 seconds [01:12:33] RECOVERY - Backend Squid HTTP on knsq27 is OK: HTTP OK HTTP/1.0 200 OK - 1419 bytes in 0.235 seconds [01:12:33] RECOVERY - Frontend Squid HTTP on knsq27 is OK: HTTP OK HTTP/1.0 200 OK - 1580 bytes in 0.237 seconds [01:12:45] RECOVERY - Frontend Squid HTTP on knsq26 is OK: HTTP OK HTTP/1.0 200 OK - 1580 bytes in 0.235 seconds [01:13:00] RECOVERY - Frontend Squid HTTP on knsq20 is OK: HTTP OK HTTP/1.0 200 OK - 819 bytes in 0.238 seconds [01:13:11] !log removing mysql & apache from emery(!) [01:13:14] Logged the message, Master [01:15:53] New review: Ryan Lane; "Patch Set 4: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/48983 [01:16:02] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48983 [01:17:11] !log updated OpenStackManager to git head on virt0. Previous state is preserved (temporarily) in the local 'premerge' branch. [01:17:12] Logged the message, Master [01:21:07] New patchset: Ryan Lane; "Firewall off ldap replication ports" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48988 [01:22:19] New patchset: Ryan Lane; "Fix salt_reactor argument" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48989 [01:22:39] andrewbogott: was there something unmerged? [01:22:44] I didn't think it had any live hacks [01:22:55] New review: Ryan Lane; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/48988 [01:23:04] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48988 [01:23:14] New review: Ryan Lane; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/48989 [01:23:21] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48989 [01:23:38] Ryan_Lane: The patch that merged netadmin and sysadmin was unmerged but the changes were there locally [01:23:54] And also the patch that created the default sudo policy wasn't merged for some reason [01:24:07] (the latter was what I was looking for) [01:24:34] andrewbogott: ah ok [01:27:55] New patchset: Ryan Lane; "Fix reference to moved directory for salt reactors" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48990 [01:28:28] New review: Ryan Lane; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/48990 [01:28:37] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48990 [01:42:43] New patchset: Tim Starling; "add cgroup to limit memory of sub processes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/40784 [01:42:49] New review: Tim Starling; "Patch Set 4: Verified+2 Code-Review+2" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/40784 [01:42:58] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/40784 [01:43:13] New patchset: Ryan Lane; "Fix typo for ldap_replication service" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48991 [01:43:30] New review: Ryan Lane; "Patch Set 1: Verified+2 Code-Review+2" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48991 [01:43:37] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48991 [01:43:54] TimStarling: merging in your cgroup stuff on sockpuppet [01:44:08] ok [01:46:13] PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours [01:48:02] New patchset: Ryan Lane; "Sort array entries for reactors" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48992 [01:48:33] New review: Ryan Lane; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/48992 [01:48:41] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48992 [02:03:41] New patchset: Tim Starling; "Fix MW cgroup installation" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48993 [02:04:58] New review: Tim Starling; "Patch Set 1: Verified+2 Code-Review+2" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48993 [02:05:06] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48993 [02:10:52] New review: Ryan Lane; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/47097 [02:11:01] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47097 [02:22:05] New patchset: Ryan Lane; "Set server explicitly for first puppet run" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48996 [02:23:39] New patchset: Ryan Lane; "Set server explicitly for first puppet run" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48996 [02:29:06] !log LocalisationUpdate completed (1.21wmf9) at Thu Feb 14 02:29:05 UTC 2013 [02:29:09] Logged the message, Master [02:30:29] RECOVERY - Puppet freshness on tin is OK: puppet ran at Thu Feb 14 02:30:00 UTC 2013 [02:32:56] drdee: locke's / is 100% full [02:34:09] New patchset: Catrope; "Add a symlink to the node_modules dir in the Parsoid config dir" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48998 [02:35:17] Ryan_Lane: --^^ [02:36:46] New review: Ryan Lane; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/48998 [02:36:55] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48998 [02:37:21] Yay thanks [02:37:27] yw [02:42:13] Hmm so that needs puppet runs on all of the minions then? [02:42:28] no [02:42:32] Just on tin? [02:42:38] just on sockpuppet [02:46:59] RECOVERY - Puppet freshness on locke is OK: puppet ran at Thu Feb 14 02:46:24 UTC 2013 [02:48:02] New patchset: Catrope; "LVS for Parsoid Varnish" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/38457 [02:48:19] New review: Catrope; "Patch Set 3:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/38457 [02:49:16] New patchset: Catrope; "LVS for Parsoid Varnish" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/38457 [02:50:25] New review: Catrope; "Patch Set 4:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/38457 [03:06:17] New patchset: Faidon; "mediawiki cgroup: cleanup the manifest" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48999 [03:07:23] I'll merge it later or tomorrow [03:09:32] what was wrong with the way I did it? I was just reading the puppet docs and couldn't work it out [03:11:54] the manual seems pretty explicit [03:12:08] class wordpress { [03:12:08] require apache [03:12:08] require mysql [03:12:08] ... [03:12:08] } [03:12:14] "The above example would cause every resource in the apache and mysql classes to be applied before any of the resources in the wordpress class. " [03:12:40] so you would think that there would be no way the service class could be applied before the conf file was created [04:01:44] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [04:04:33] New patchset: Tim Starling; "Use a cgroup for command execution" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/49000 [04:05:18] New patchset: Aaron Schulz; "Modified jobs-loop script to keep a fuller pipeline." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48981 [04:17:48] New review: Aaron Schulz; "Patch Set 2:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48981 [04:19:02] ugh [04:19:11] * Aaron|home fucking hates files/ssl/*.wikimedia.org.crt [04:25:20] https://bugzilla.wikimedia.org/44982 [04:25:31] ^ TimStarling Aaron|home [04:25:57] if anyone has any idea how to mitigate that issue, comments on the bug or such would be appreciated [04:27:03] it's one of the reasons we are developing lua [04:27:13] which is going to be deployed on enwiki early next week [04:27:34] good! [04:27:43] in the meantime, can we raise the time out? [04:28:01] it's impossible to save and have the page reload w/o error [04:28:11] you know this has been a problem for years, right? [04:28:14] yup, tried again [04:28:21] and 3 days before the fix, you want to raise the timeout? [04:28:22] yes but seems especially bad [04:28:42] the timeout is there to protect the servers [04:28:43] well, i don't know how magically lua will fix [04:28:51] i'm sure it will help [04:29:10] another 10 seconds or something might help [04:29:13] or 20 seconds [04:29:28] Request: POST http://en.wikipedia.org/w/index.php?title=New_York_City&action=submit, from 84.175.74.62 via amssq40.esams.wikimedia.org (squid/2.7.STABLE9) to 91.198.174.42 (91.198.174.42) [04:29:32] Error: ERR_READ_TIMEOUT, errno [No Error] at Thu, 14 Feb 2013 04:28:00 GMT [04:29:35] again [04:29:50] the fix we normally recommend is to modify the article to make it take less long to render [04:30:01] true [04:30:38] * aude tries different article [04:31:29] RECOVERY - MySQL disk space on neon is OK: DISK OK [04:31:51] TimStarling: why must we have that one file with a * in the name? [04:31:56] * Aaron|home kicks ntfs [04:32:19] you can change it can't you? [04:32:45] it's not just windows that is broken by filenames with asterisks in them [04:32:58] shell scripts can be made to do interesting things with such filenames [04:34:04] I'll have to wait till tomorrow when I'm on my linux box [04:35:09] * aude waits to see if i can save [[Barack Obama]] :) [04:35:20] New review: Aaron Schulz; "Patch Set 2: Code-Review-1" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/48981 [04:35:42] aude: that article is actually quicker than it used to be [04:36:01] it did save :) [04:36:05] slow-parse.log shows it takes 35-40s now [04:36:13] in december, it was 40-45 [04:36:28] ah [04:36:43] september shows it occasionally taking over 60 [04:36:58] TimStarling: I recall ryan saying that file was needed for something [04:37:09] for some hack, I can't recall [04:37:18] hmmm [04:37:36] i'll see what i can do to remove templates from [[New York City]] [04:37:44] and poke around more at other pages [04:38:04] * aude knows it's been an issue for years but seems especially bad recently [04:38:09] I measure 22 seconds with the removed [04:38:28] which may mean that there is a lot of non-reference fat to cut out [04:38:29] ah [04:39:01] porting cite/core to lua is the thing that will magically make everything faster after lua deployment [04:39:30] ah [04:39:36] * aude can't wait [04:39:55] I see it uses country data templates [04:40:06] * aude wants it on now :D [04:40:06] that's an obvious thing to target [04:40:11] sure [04:40:54] let's try removing that climate box, it looks a lot like some templates that were causing trouble elsewhere [04:41:08] sure [04:41:57] why climate templates have to be so complicated :P [04:42:19] maybe there is climate model software embedded in the templates [04:42:24] lol [04:42:31] PROBLEM - Puppet freshness on cp3022 is CRITICAL: Puppet has not run in the last 10 hours [04:43:11] oh no [04:43:26] but wouldn't be surprised if it converts units, etc. [04:43:36] there are many {{convert}} calls [04:43:42] not sure if they are the problem [04:43:53] ah, right [04:44:02] removing the climate box brings it down from 22 to 13 [04:44:27] I'm just using preview, you know, this is nothing advanced [04:47:02] i'm removing it [04:47:07] will leave a link to it [04:47:39] at least something that maybe will allow saving the page again [04:47:40] New patchset: Aaron Schulz; "Modified jobs-loop script to keep a fuller pipeline." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48981 [04:48:02] we can wait a week to maybe reintroduce the template :) [04:48:11] ah, I can reset it, I was forgetting to escape the * so it thought I was globbing [04:48:35] it saved! [04:48:40] magic [04:49:23] [04:49:31] slightly better, under the timeou [04:49:33] t [04:50:26] people who introduce these templates should be beaten over the head a bit [04:50:57] in many cases, they know what they are doing to parser performance, they just don't care [04:52:19] yeah [04:52:57] Why would an article about a city have information about its climate? [04:53:08] That's just outrageous for an encyclopedia. Don't these editors know this? [04:53:28] Why aren't they profiling the page for performance? [04:54:09] heh [04:54:22] my point is that they actually do profile the page for performance [04:54:27] New patchset: Tim Starling; "Fix sql script so it can run as wikidev" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49002 [04:54:41] they work on their templates until they start rendering in less than the timeout, then that's job done [04:54:52] sure [04:55:18] New review: Tim Starling; "Patch Set 1: Verified+2 Code-Review+2" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/49002 [04:55:25] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49002 [04:55:28] Yes, they've been trained to work below the ceiling. Raise the roof. [04:55:46] you're saying I should raise the roof? [04:55:51] like what aude wants? [04:56:01] increase the timeout to 60? [04:56:18] TimStarling: probably slightly higher would help [04:56:20] No, I commented on the bug. [04:56:33] but nothing insane [04:56:33] I think that bug is a dupe and I think Scribunto is the appropriate answer. [04:56:34] I think actually Susan just wants YouTube videos of you dancing. [04:56:44] Susan: feel free to mark it as a dup [04:56:48] Getting down, as it were. [04:56:56] * aude just got fed up [04:56:59] aude: You seemed frustrated, so I didn't want to antagonize. [04:57:00] While you bring it up. [04:57:05] sure :) [04:57:18] It's a legitimate bug, it just happens to be a known issue. [04:57:27] [[Q60]] has been my test item for wikidata, so having such a test page for wikipedia, .... [04:57:42] well it's not a good example for testing editing [04:57:43] I think Scribunto should come with global templates, so that we don't end up with {{convert}} on every wiki, but I got shouted down on that. [04:57:59] Susan: that would be cool [04:58:01] So... there will be an improvement, but the code fragmentation will almost certainly eventually be painful. [04:58:08] An improvement next week, it sounds like. [04:58:55] * aude notes the same article on italian wikipedia is quite fast [04:58:57] to edit [04:59:07] fewer templates.... [04:59:14] so before I got distracted by the sql script not working, I was about to tell you that there are now only 7603 pages that use {{Weather_box}} [04:59:23] presumably down from 7604 [04:59:27] oh [05:00:08] baby steps [05:00:14] heh [05:00:17] https://toolserver.org/~jarry/templatecount/index.php?lang=en&name=Template%3AWeather+box#bottom [05:00:27] TimStarling, did we ever make much progress on automatic translation of traditional wiki-templates to Lua? (I apologize if I somehow missed the answer on-wiki or a list) [05:00:36] dschoon: no [05:00:45] It's going to be a long tail. [05:00:50] That whole Turing Complete thing is a bitch. [05:00:57] But the main culprits should be pretty easy to knock out. [05:01:26] coord, cite, and convert, mostly, as I recall. [05:01:30] I guess it won't give us a list, eh? [05:01:31] The triple Cs. [05:01:39] TimStarling: would it complicate Scribunto deployment if I were to enable CodeEditor on metawiki tomorrow? [05:01:48] It'd be kind of interesting to see the histogram by views (perhaps via stats.grok.se) [05:02:13] It'd be more interesting if there were a real analytics infrastructure so we could stop using stats.grok.se. [05:02:54] ori-l: maybe you should first set $wgCodeEditorEnableCore = true [05:03:04] and deploy it to the wikis that it's running on already [05:03:31] Word, Susan. [05:03:40] currently mediawiki.org and test2.wikipedia.org [05:03:54] Out of curiosity, is there a literal asterisk template? https://toolserver.org/~jarry/templatecount/index.php?lang=en&namespace=10&name=Template%3A*#bottom or is that everything? [05:03:55] that would make things a bit simpler [05:04:03] OK, good idea. [05:04:05] (relatedly https://toolserver.org/~jarry/templatecount/index.php?lang=en&namespace=10&name=Template%3A.*#bottom ) [05:06:01] https://en.wikipedia.org/wiki/Template:* [05:06:07] It's a redirect for a bold middot. [05:06:45] Or bullet, I guess. [05:06:59] It's slowly being deprecated. [05:07:24] Huh. So there really is. [05:07:25] Yeah. [05:07:31] Just read the code. [05:07:48] (of the search) [05:10:06] TimStarling: in a7a74cf8 you added a comment advising not to enable it until it has been reviewed. It hasn't been formally reviewed yet but I figure that it's Brion's code and it has been running long enough to amount to something of a fact on the ground. [05:10:37] So I'm inclined to enable it, unless there was something specific that made you nervous. [05:13:08] have you tested it? [05:14:03] Yes, my dev instance has it enabled since I've been editing lots of JSON documents. [05:14:38] Periodically the syntax rules seem not to load and I haven't chased that down fully, but I haven't seen that affect anything else. [05:15:21] The comment didn't make much sense to me, because the extension is already deployed on a few production wikis, including mediawiki.org. [05:15:51] And the code didn't seem branched in a way that that disabled variable was stopping anything (additional) of consequence. [05:16:18] It's disabling the use of CodeEditor for .js and .css articles [05:16:25] the automatic use, rather. [05:16:56] Right. I didn't see how that made a difference in terms of code safety/deployment-readiness. [05:17:16] it took a fair amount of work to get it working sufficiently well to support lua editing [05:17:37] From where Brion left off, you mean? [05:17:44] yes [05:18:28] I hadn't tested it for JS or CSS editing, and I didn't really want to have to do that prior to the scribunto deployment [05:18:33] Well, it's available to be used by extensions, but the only extensions that make use of the hook are Scribunto and EventLogging, and the latter for functionality that is not enabled on mediawiki.org or test2wiki. [05:19:10] I could add the schema: namespace to test2, which would have the effect of enabling CodeEditor there. [05:19:16] so splitting the deployments seemed to make sense [05:19:38] I'd like CodeEditor to be enabled on Meta-Wiki for JSON, CSS, and JS. [05:19:47] I think Meta-Wiki is a good development/testing ground. [05:19:55] Because it's mostly experienced users. [05:20:10] generally a deployment is preceded by an announcement and some beta testing [05:20:11] And the extension needs more use to find more bugs and be ready for wider deployment. [05:20:19] and followed by gathering feedback [05:20:29] Heh, since when? :-) [05:20:30] TimStarling: Susan did.. [05:20:54] Most deployments in Wikimedia's history weren't preceded by anything more than an IRC comment saying "time to scap!" [05:21:01] http://meta.wikimedia.org/wiki/Meta:Babel?uselang=it#CodeEditor_extension_deployment_on_Schema: [05:21:25] Interesting that you went for Italian. [05:21:34] Blame the Googles. [05:21:44] if you're fine with taking responsibility for it, then I am fine with it being deployed [05:22:03] huge JavaScript applications are not my normal area of expertise [05:22:18] I'll just enable the Schema: namespace on test2 for now, I think. [05:22:58] I'm surprised EventLogging isn't already on there. [05:23:03] Or on test.wikipedia.org. [05:23:12] Then $wgCodeEditorEnableCore = true the week after that, and enable the extension on metawiki after that. [05:23:12] You were testing on Labs, I guess? [05:23:34] It's enabled on both test and labs, just not the component that manages a namespace. [05:23:50] Ah, weird. [05:23:50] And yes, on labs. [05:24:04] I thought the extension didn't do much of anything without the Schema namespace. [05:24:30] Yes, but rather than deploy and look after a bunch of schema articles scattered across Wikimedia wikis, they're all centralized on meta. [05:24:56] But data collection jobs on other WMF wikis can reference them. [05:25:03] Ah. [05:25:15] Yay for centralization. :-) [05:25:43] Well, it was where Dario, Faulkner & co were documenting data models anyhow. [05:39:46] New patchset: Ori.livneh; "Set $wgEventLoggingDBname on test2wiki to test2wiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/49003 [05:40:41] New review: Ori.livneh; "Patch Set 1: Code-Review-2" [operations/mediawiki-config] (master) C: -2; - https://gerrit.wikimedia.org/r/49003 [05:46:14] TimStarling: So my plan is to only enable CodeEditor for Schema: articles on test2 for now. I will report back if I notice anything funny that could affect Scribunto. Thanks for the advice. [05:47:37] ok, they didn't like me removing the climate box (lol) [05:50:17] there's a joke about global warming and climate science in here somewhere. [06:24:53] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [06:41:13] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 204 seconds [06:41:40] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 221 seconds [06:48:07] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 9 seconds [06:48:34] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 0 seconds [08:19:10] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [08:36:33] PROBLEM - Puppet freshness on stafford is CRITICAL: Puppet has not run in the last 10 hours [08:48:33] RECOVERY - MySQL disk space on neon is OK: DISK OK [09:10:50] New patchset: Catrope; "Fix stupid copy/paste mistake that broke Nagios checks for Parsoid Varnish" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49009 [09:38:10] dumps site is slow... downloading small rss file takes 10 secs... :-/ [09:43:57] there's a bug [09:45:09] https://bugzilla.wikimedia.org/show_bug.cgi?id=43647 [09:45:55] hm, for big files it eventually stabilises around some higher speed [09:46:44] network quite low http://ganglia.wikimedia.org/latest/?r=day&cs=&ce=&c=Miscellaneous+pmtpa&h=dataset2.wikimedia.org&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS [10:01:36] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [10:03:33] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [10:32:08] RECOVERY - MySQL disk space on neon is OK: DISK OK [10:34:12] apergos: apparently dumps.wm.o has some bandwidth / disk speed issue : https://bugzilla.wikimedia.org/show_bug.cgi?id=43647 ;D [10:34:16] out for acocunting [11:28:43] RECOVERY - Puppet freshness on ms1004 is OK: puppet ran at Thu Feb 14 11:28:12 UTC 2013 [11:31:58] !log ms1004 - full disk, deleted large HTCPpurger.log file, fixed puppet runs [11:32:01] Logged the message, Master [11:34:47] mutante: any chance I could get you to reinstall the two remaining lucid mobile varnish boxes to precise? :) [11:35:00] that would complete our varnish precise migration [11:39:23] mark: ah, what are their names? ok.. [11:41:02] ok, cp1043 and cp1044 i suppose [11:42:37] set to enabled: False in pybal first, right [11:43:20] yep [11:45:04] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [11:45:24] keep in mind that the backend varnish instances are still used by other frontends, independent of pybal [11:47:37] PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours [11:50:47] and do them one by one too [11:53:09] yea, one by one, about the other frontends, is this ./templates/varnish/mobile-frontend.inc.vcl.erb .. ? [11:54:10] no need to edit them [11:54:27] i just want you to be aware that depooling in pybal doesn't mean the box is then idle [11:54:38] but if you stop the backend varnish instance the others will notice and it's ok [11:54:57] ah, ok [11:56:37] PROBLEM - Puppet freshness on virt0 is CRITICAL: Puppet has not run in the last 10 hours [12:10:14] !log disabling cp1043 in pybal [12:10:16] Logged the message, Master [12:14:28] !log stopping varnish backend on cp1043 and starting reinstall [12:14:29] Logged the message, Master [12:16:06] !log Inserted varnish 3.0.3plus~rc1-wm7 into the precise-wikimedia APT repository [12:16:07] RECOVERY - MySQL disk space on neon is OK: DISK OK [12:16:07] Logged the message, Master [12:19:01] PROBLEM - Host cp1043 is DOWN: PING CRITICAL - Packet loss = 100% [12:24:49] RECOVERY - Host cp1043 is UP: PING OK - Packet loss = 0%, RTA = 26.51 ms [12:29:01] PROBLEM - Varnish HTTP mobile-backend on cp1043 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:01] PROBLEM - Varnish traffic logger on cp1043 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:10] PROBLEM - Varnish HTTP mobile-frontend on cp1043 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:29] PROBLEM - SSH on cp1043 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:46] PROBLEM - Varnish HTCP daemon on cp1043 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:31:07] RECOVERY - SSH on cp1043 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [12:43:52] RECOVERY - Varnish traffic logger on cp1043 is OK: PROCS OK: 3 processes with command name varnishncsa [12:44:17] New patchset: Mark Bergsma; "Update streaming range patch to M.B.Grydeland's updated version" [operations/debs/varnish] (testing/3.0.3plus-rc1) - https://gerrit.wikimedia.org/r/49031 [12:44:17] New patchset: Mark Bergsma; "varnish (3.0.3plus~rc1-wm7) precise; urgency=low" [operations/debs/varnish] (testing/3.0.3plus-rc1) - https://gerrit.wikimedia.org/r/49032 [12:44:19]