[00:51:09] PROBLEM - Puppet freshness on potassium is CRITICAL: Puppet has not run in the last 10 hours [01:42:09] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 260 seconds [01:43:21] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 271 seconds [01:49:48] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 658s [01:51:54] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 0 seconds [01:56:00] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 11s [01:56:36] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 16 seconds [02:01:13] New patchset: Jeremyb; "nycwikimedia: wgImportSources = (enwiki, metawiki)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/16804 [02:35:28] * jeremyb waves tfinc... want to do a deploy for me? ;-) [02:35:42] !g I39d1ca7cccb6 [02:35:42] https://gerrit.wikimedia.org/r/#q,I39d1ca7cccb6,n,z [03:12:03] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [03:17:09] RECOVERY - Puppet freshness on cp1020 is OK: puppet ran at Fri Jul 27 03:16:45 UTC 2012 [03:18:03] RECOVERY - Puppet freshness on db35 is OK: puppet ran at Fri Jul 27 03:17:42 UTC 2012 [03:18:39] RECOVERY - Puppet freshness on mw58 is OK: puppet ran at Fri Jul 27 03:18:30 UTC 2012 [03:26:36] RECOVERY - Puppet freshness on srv209 is OK: puppet ran at Fri Jul 27 03:26:30 UTC 2012 [03:36:03] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [06:06:35] PROBLEM - Puppet freshness on db63 is CRITICAL: Puppet has not run in the last 10 hours [06:46:19] PROBLEM - Puppet freshness on ms-be10 is CRITICAL: Puppet has not run in the last 10 hours [08:51:30] New patchset: Hashar; "lvsrealserver is no more a parameter" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16809 [08:51:35] any op around ? :-D [08:52:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16809 [08:52:28] apergos: mark: paravoid: could one of you please review/merge https://gerrit.wikimedia.org/r/16809 ? The role::applicationserver::jobrunner is broken because it uses the lvsrealserver parameter which got removed int he common class :/ [08:52:34] causes me err: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter lvsrealserver at /etc/puppet/manifests/role/apache.pp:77 on node i-0000031d.pmtpa.wmflabs [08:59:45] * hashar summons a !root [09:13:05] New review: Hashar; "https://bugzilla.wikimedia.org/show_bug.cgi?id=38749" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/16809 [09:40:01] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16809 [09:44:10] bah lvs_pool missing. Iwill just include the needed classes [10:51:32] PROBLEM - Puppet freshness on potassium is CRITICAL: Puppet has not run in the last 10 hours [11:32:44] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15445 [11:37:22] New patchset: Mark Bergsma; "Revert "Adds WLM api host config in misc-servers"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16812 [11:38:00] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16812 [12:38:05] mark or apergos or paravoid - any of you want to review an apache rewrite rule change? it's purpose is to allow swift to call the image scalers directly (bypassing ms5) for thumbnail generation. [12:38:09] https://gerrit.wikimedia.org/r/#/c/16743/ [12:38:15] yes [12:38:37] oh wow [12:38:42] very exciting [12:39:01] god I hate our apache configs [12:39:05] that reminds me, I should resume work on the short url thing rather and stop talking in irc [12:39:38] swift will take a URL like upload/wikipedia/en/thumb/a/a2/foo.jpg/120px-foo.jpg and turn it into en.wikipedia.org/thumb/a/a2/foo.jpg/120px-foo.jpg. The apache rewrite rule is to take that and hand it off to /w/thumb_handler.php [12:39:53] (all the upload/... -> en. happens in swift) [12:40:01] (and is almost ready for review but not quite) [12:45:09] so how does thumb_handler handle the original URL? [12:45:24] aaron's writing that part. [12:45:39] apparently there's very little munging necessary before thumb.php can take over. [12:46:13] as I understand it, thumb_handler basically just breaks up the URL into arguments like f=, w= (for filename and width) and passes it off to thumb.php [12:46:31] ok [12:47:04] wouldn't it be nicer to have mediawiki recognize the /thumb/ prefix then, and handle it internally? [12:47:43] there might be a higher setup cost to that... [12:48:24] I don't know mediawiki well enough to say. This method was aaron's suggestion. (though it's similar to the ShortURL thing that's also going on now) [12:48:27] so what will swift do exactly? it changes that url into that en.wikipedia.org url, and contacts which host exactly? [12:48:32] https://gerrit.wikimedia.org/r/#/c/16742/ [12:49:04] swift builds up a request for en.wikiyaddayadda and sets the rendening cluster LVS as a proxyhost. [12:49:15] ok [12:49:33] hmm [12:49:41] in that case we can restrict this change to only apply on the image scalers [12:49:54] we could, though I didn't think we handled the apache configs that way. [12:50:19] I would prefer it actually. less chance something else weird goes on. [12:50:26] yeah [12:50:39] we're not using the http://en.wikipedia.org/thumb/ path yet are we [12:50:43] nope. [12:50:49] sfaik. [12:51:06] just... the Thumb article will get less traffic [12:51:18] heh... [12:51:50] is thumb_handler already in place? [12:52:19] yes, though I think aaron needs to do more work to it. [12:52:57] can't we call /w/thumb_handler.php from swift directly? so we don't need this rewrite? [12:53:08] then we don't need the /thumb/ stuff either [12:53:50] I would say it doesn't matter to swift whether it generates en.wikipedia.org/thumb/a/a2/foo.jpg/bla or en.wikipedia.org/w/thumb_handler.php/a/a2/foo.jpg/bla [12:55:31] the only reason I can think of that aaron wanted it this way is that it provides mediawiki with the URL /thumb/a/a2/foo instead of having extra stuff on the beginning. [12:55:45] from swift's perspective, you're right; it doesn't matter. [12:56:06] he just needs to strip or replace the prefix doesn't he [12:56:24] that seems trivial compared to the other url processing he must be doing [12:56:45] i would prefer that since it doesn't further pollute the root level name space of the URLs [12:57:03] we already have /w for script entry points [12:57:24] I'll run that by him and see if it'll fly. [12:57:31] thank you [12:57:33] let me know what he thinks [13:13:04] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [13:36:53] mark: any reason we use rsyslog for remote-syslog but syslog-ng for the syslog servers? [13:36:56] or is it just legacy? [13:37:04] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [13:37:58] !g I39d1ca7cccb6 | site req deploy please? ;) [13:37:58] site req deploy please? ;): https://gerrit.wikimedia.org/r/#q,I39d1ca7cccb6,n,z [13:38:25] Change merged: Demon; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/16804 [13:42:22] paravoid: maybe cause rsyslog is now the default in ubuntu ? [13:53:44] New patchset: Jeremyb; "followup I39d1ca7cc (nycwikimedia wgImportSources)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/16817 [13:55:53] New review: Jeremyb; "meta worked, en didn't" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/16804 [14:05:24] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/16817 [14:07:38] !log srv281: rsync: write failed on "/apache/common-local/wmf-config/InitialiseSettings.php": No space left on device (28) [14:07:52] Logged the message, Master [14:07:59] New patchset: Bhartshorne; "amending swift's rewrite.py to allow it to talk directly to mediawiki image scalers instead of ms5" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16821 [14:08:09] mark and apergos: I've got another review request for you if you're game. ^^^^ [14:08:35] mark: it assumes /thumb/ at the moment but after I talk to aaron I'll change it to /w/thumb_handler.php (if it'll work). [14:08:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16821 [14:08:52] also I don't intend to submit this today but I would like the review. [14:22:58] maplebed: sure, will review today [14:23:16] tnx [14:28:17] where is the config for pdf renderers? not in git? [14:36:00] Reedy: around? [14:36:52] Reedy: so, http://test.wikipedia.org/s/4 redirects to /wiki/TestPageToTranslate [14:37:20] Reedy: /s/foo brings a "Short URL not found" page [14:37:36] That sounds good [14:37:53] jeremyb: it's whatever pediapress has set it to, it's not puppetised, and it's not in any of our repos. [14:38:07] https://github.com/pediapress [14:38:36] paravoid: looks good :) [14:38:51] /s/8bc that you mentioned yesterday does not work, but I suspect that's because of caching [14:39:02] Reedy: none of those repo names look like they'd have config in them [14:39:14] Reedy: do we have a pediapress contact? [14:39:18] probably because they don't [14:39:30] Best asking Tomasz (he's our main liason with them) [14:39:46] finc? [14:39:50] yes [14:39:51] yup [14:39:56] danke! [14:40:04] jeremyb: FYI, they compile nearly everything themselves from scratch... [14:40:23] yeah... i don't even want to get into that ;( [14:40:36] New review: Faidon; "Tested on srv193/test.wp.org and works :)" [operations/apache-config] (master); V: 0 C: 1; - https://gerrit.wikimedia.org/r/16742 [14:41:19] mark: do you foresee any other issue with supporting /s/foo (shorturls)? with caching or whatnot? [14:43:51] and I wonder how we deploy operations/apache-config :) [14:44:01] maplebed: saw that? works like a charm. [14:44:16] ah, https://wikitech.wikimedia.org/view/Apaches mentions how to deploy it [14:49:34] depends on what caching headers get sent for it [14:50:23] nice. [14:55:29] maplebed: I say let's push it on monday [14:55:35] +1 [14:55:59] (fwiw I'll be back in pacific time on monday) [14:57:05] PROBLEM - Host db63 is DOWN: PING CRITICAL - Packet loss = 100% [14:57:14] Reedy: the extension is enabled in production, right? [14:57:30] Only on the testwikis [14:57:45] oh [14:58:08] so, we push the apache config on Monday, we test it on test2 and then we tell you to enable it everywhere? [14:58:17] sounds good to me [14:58:18] does it sound sensible? [14:58:22] okay, great :) [14:58:24] are we enabling it "everywhere"? [14:58:33] I can't say I've actually read the email [14:58:34] s [14:58:43] maplebed: reviewed, comments inline [14:58:53] cool [14:59:42] Reedy: you're the RT requestor :-) but there's a bugzilla ticket that mentions only some wikis, so I guess the answer's no [14:59:43] Reedy: paravoid: there was talk of enabling it everywhere, but the extension doesn't actually have to be enabled; the rewrite rule on a wiki without it will just give adifferent 404 error than before (though still a 404) [15:00:14] I suspect the "issue" here will people not wanting the short url links to display on all pages on every wiki [15:01:09] bz says ta, or, zh and hi [15:01:40] right [15:01:46] there's also talks of using /r/ instead of /s/ [15:01:54] lols [15:03:06] I think I've broken my shredder [15:03:25] thanks mark. replied. [15:03:31] paravoid: I think the /r/ was rejected. [15:07:02] New patchset: Bhartshorne; "amending swift's rewrite.py to allow it to talk directly to mediawiki image scalers instead of ms5" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16821 [15:07:06] mark: ^^^ [15:07:38] sorry, hold on. must enable logging. [15:07:41] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16821 [15:08:20] RECOVERY - Host db63 is UP: PING OK - Packet loss = 0%, RTA = 0.52 ms [15:08:29] New patchset: Bhartshorne; "amending swift's rewrite.py to allow it to talk directly to mediawiki image scalers instead of ms5" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16821 [15:08:33] there we go. [15:09:07] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16821 [15:26:47] PROBLEM - Host db63 is DOWN: PING CRITICAL - Packet loss = 100% [15:32:20] RECOVERY - Host db63 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [15:43:26] New patchset: ArielGlenn; "See changelog for Version 0.2-pre for list." [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/16826 [15:50:59] New review: Hashar; "Is that still needed?" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/6905 [15:53:53] PROBLEM - Host db63 is DOWN: PING CRITICAL - Packet loss = 100% [15:59:26] RECOVERY - Host db63 is UP: PING OK - Packet loss = 0%, RTA = 1.75 ms [16:00:45] Change merged: ArielGlenn; [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/16826 [16:06:11] RECOVERY - SSH on db63 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [16:08:00] PROBLEM - Puppet freshness on db63 is CRITICAL: Puppet has not run in the last 10 hours [16:08:18] New review: Oren; "looks good - probably you would want to add a remove solr command" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/16732 [16:46:59] PROBLEM - Puppet freshness on ms-be10 is CRITICAL: Puppet has not run in the last 10 hours [17:02:54] preilly: ? [17:02:57] testing? [17:03:03] I pinged him as well half an hour ago [17:03:06] I'm at home, and would like to go into the office [17:03:16] so sooner, rather than later would be nice [17:03:19] I can provide support to Patrick if you'd like to leave. [17:03:29] isn't it late there? [17:03:30] I'm not sure /what/ I have to do though [17:03:44] they want to check requests via varnishlog [17:04:07] for a specific carrier, to ensure the vcl rules are applying correctly [17:05:33] he is not in office now [17:05:38] god damn it [17:07:32] Ryan_Lane: just go, I'll be around. [17:07:41] * Ryan_Lane grumbles [17:13:35] New patchset: J; "migrate wikimedia-job-runner to puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16501 [17:14:12] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16501 [17:19:54] j^: hi [17:20:47] j^: so, considering this change has a potential of breaking job runners and that it's Friday 8pm here, I'm not going to attempt merging it now [17:20:51] hope you understand :) [17:21:34] I can have a look on Monday, or you could convince someone who lives somewhere that it's morning now :) [17:22:02] <^demon> "Potentially breaking job runners" sounds like a bad idea on a friday regardless of timezone :\ [17:22:08] true that [17:22:09] PROBLEM - Host db63 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:38] anyone have experience with using git-svn to host a mirror of an svn repository? i'm trying to set up a git mirror (with WMF git repos) of an svn repo (hosted on the toolserver) for the purpose of better being able to manage deploying code from the repo to a box on the wmf cluster [17:24:13] awjr: I haven't used in a while, but everytime I used it it was quite straightforward [17:24:38] <^demon> awjr: Back in the day, Ævar maintained our svn -> git mirror. It wasn't very hard--just had a cron that checked for new svn commits and then pushed them over to git. [17:26:09] paravoid, ^demon: i was thinking about setting something similar up, with a cron job fetching svn commits into a mirror branch in the git repo, and then pushing code that we are comfortable deploying into a master branch [17:26:25] does that seem sane, or is there a better approach? [17:27:16] ask the toolserver folks to migrate to git? :) [17:27:34] heh [17:27:42] RECOVERY - Host db63 is UP: PING OK - Packet loss = 0%, RTA = 1.08 ms [17:31:16] awjr, we could run it on the same WLM server [17:32:20] awjr: ask them to use a repo in gerrit? [17:33:15] <^demon> Well, we made a repo in gerrit for this. [17:33:27] great, they need to use that, then :) [17:33:29] <^demon> But the development is happening in svn right now. The idea is to mirror the svn repo to git. [17:33:55] it's bad enough we're basing a production thing on toolserver [17:34:07] <^demon> Hey, I just made the repo. Don't blame me for the rest of it :p [17:34:13] I really dislike the idea of running production code from toolserver, too [17:36:51] Ryan_Lane, vice versa: we're going to run TS code on prod machine:P [17:37:01] yes [17:37:03] that's what I mean [17:37:05] I dislike that [17:37:23] put it into a gerrit repo and I'll be happier [17:37:43] <^demon> Let's please make Ryan happier. [17:37:52] <^demon> I have something I want to ask him later but I want him in a good mood first ;-) [17:37:56] hahaha [17:38:19] all happines will vanish once you see the code!!!1 [17:40:53] if it's the svn repo I think it is, it was created well before any git migration [17:41:22] and before InnoDB? :P [17:42:10] no [17:42:20] but the tables were already like that when I joined :P [18:33:31] !log authdns-update for tmh1/2 [18:33:39] Logged the message, RobH [18:33:43] cmjohnson1: ^ Ok, you are all set [18:34:00] oh yea... i need to change mgmt entries [18:34:01] doh [18:34:04] cmjohnson1: nm, lemme fix [18:37:51] !log another authdns-update, forgot to update asset tags to servernames for mgmt [18:37:59] Logged the message, RobH [18:38:29] yep, manual per the instructions they state in ticket [18:38:40] you wanna set the swap in a small raid1 and then the rest as a single large raid1 for / [18:47:58] New patchset: Cmjohnson; "adding tmh1 and tmh2 to dhcpd" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16837 [18:48:33] cmjohnson1: checking it out for you now [18:48:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16837 [18:49:58] New review: RobH; "Wrong FQDN, these are internal hosts, so they need to be server.pmtpa.wmnet not server.wikimedia.org" [operations/puppet] (production); V: 0 C: -1; - https://gerrit.wikimedia.org/r/16837 [18:52:37] New patchset: Catrope; "Update CSS class for Gerrit upgrade" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16841 [18:53:15] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16841 [19:00:49] !log allocating potassium and zirconium to rt 3342 (misc db use) and shutting them down until they are installed. [19:00:57] Logged the message, RobH [19:02:57] there's a mobile domain missing (NXDOMAIN) where the wiki already exists. who can make the mobile one for me? (mobile guys say it just needs to be created) [19:03:14] or should i make an RT [19:03:28] New patchset: Cmjohnson; "adding tmh1 and tmh2 to dhcpd" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16837 [19:03:54] (domain is wikimania2013.m.wikimedia.org) [19:04:06] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16837 [19:04:18] PROBLEM - Host potassium is DOWN: PING CRITICAL - Packet loss = 100% [19:05:12] PROBLEM - Host zirconium is DOWN: PING CRITICAL - Packet loss = 100% [19:09:09] cmjohnson1: checking now, sorry was not paying attentoin [19:09:49] New review: RobH; "all set" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/16837 [19:09:49] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16837 [19:10:06] cmjohnson1: it looks right. i am about to commit a bunch of changes for potassium and zirconium [19:10:15] so gimme five for that and i can roll them all live at the same time [19:11:41] New patchset: RobH; "swapping potassium and zirconium to internal vlan & adding to decom to remove invalid data from nagios" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16842 [19:12:18] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16842 [19:13:55] New patchset: RobH; "swapping potassium and zirconium to internal vlan & adding to decom to remove invalid data from nagios" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16842 [19:14:33] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16842 [19:15:35] New review: RobH; "self review ;]" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/16842 [19:15:36] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16842 [19:17:12] cmjohnson1: your changes are merged, dont forget to puppetd --test on brewster as root before you try to install [19:17:20] else it wont get the dhcp updates until the auto run [19:19:13] * jeremyb sent an RT but idk any way to get the #. search for me (or ask me) if you want to do a quick DNS fix ;) [19:21:50] Ryan_Lane: I just assigned you two misc dbs for use per rt#3107 (gerrit, openstack etc) [19:21:55] details in ticket [19:30:10] RobH: eh? [19:30:15] I thought we already switched over [19:30:25] were those temporary? [19:30:26] ..... [19:30:31] really? [19:30:39] we moved gerrit to eqiad [19:30:43] ....... [19:30:47] haven't you noticed it's about 924892733498274 times faster? [19:30:47] the ticket didnt mention it. [19:31:04] I know, right? [19:31:17] lint still takes forever, so nope. [19:31:20] so you dont need these? [19:31:25] (i hate old tickets with misleading info) [19:31:33] stealing back and removing the info [19:31:52] >>> math.log10(924892733498274) [19:31:52] 14.96609136738649 [19:32:37] New review: Demon; "Quick note: If we want to mark this as @external upstream so it will remain stable, the class is .ch..." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/16841 [19:32:47] Unuseable times 924892733498274 should == 0 times 24892733498274 :P [19:33:14] Damianz: huh? [19:33:26] Damianz: slow is always measurable. just maybe not by you [19:33:44] Slow is measurable as long as it doesn't time out the connection before rendering a response ;) [19:33:50] Damianz: so, $slow / 924892733498274 == $newspeed; [19:34:05] i don't think i've seen a timeout [19:35:36] Ryan_Lane: tyvm for you Dynamic Sidebar extension :) [19:35:55] hexmode: you've used it? [19:36:07] it probably needs to be updated [19:36:17] it's a neat extension, though [19:36:28] Ryan_Lane: yes, and added a patch [19:36:38] which is the real reason I'm here [19:36:47] Ryan_Lane: could you review/merge it? [19:36:54] cool [19:36:59] it looks like it works with 1.19, though [19:40:01] New review: Aaron Schulz; "Didn't mean to +1 yet...still looking." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/16821 [19:40:19] RECOVERY - SSH on potassium is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [19:40:28] RECOVERY - Host potassium is UP: PING OK - Packet loss = 0%, RTA = 35.46 ms [19:40:41] !log setting up yttrium per rt 3221 [19:40:49] Logged the message, RobH [19:42:07] RECOVERY - Host zirconium is UP: PING OK - Packet loss = 0%, RTA = 35.42 ms [19:46:01] RECOVERY - Puppet freshness on potassium is OK: puppet ran at Fri Jul 27 19:45:30 UTC 2012 [19:52:05] Logged the message, Master [20:12:47] New patchset: Ryan Lane; "Adding opera_mini to the xff list, and moving the acl" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16851 [20:13:01] Logged the message, Master [20:13:25] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16851 [20:14:47] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16851 [20:23:50] !log authdns-udpate for yttrium/wlm.w.o per rt 3221 [20:23:51] Change abandoned: Bhartshorne; "changed mediawiki so it wants /w/thumb_handler.php/stuff instead of /thumb/stuff so this change is n..." [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/16743 [20:23:57] Logged the message, RobH [20:40:19] New review: Hashar; "Please note patchset 1 and 2 are different approach to fix the problem. Patchset 1 is building up on..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/16661 [20:42:22] Logged the message, Master [20:52:31] New review: Ryan Lane; "I'm not a fan of the instance name in this. I'd much prefer this be fixed in some way that can be us..." [operations/puppet] (production); V: 0 C: -1; - https://gerrit.wikimedia.org/r/16661 [21:16:32] !log updated dns for mgmt on wtp1.mgmt [21:16:40] Logged the message, RobH [21:22:56] New patchset: J; "migrate wikimedia-job-runner to puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16501 [21:23:35] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16501 [21:46:00] New patchset: J; "migrate wikimedia-job-runner to puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16501 [21:46:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16501 [21:51:24] New patchset: awjrichards; "Adding wlm.wikimedia.org to site.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16858 [21:52:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16858 [21:53:31] if anyone's available for a quick review and merge ^^^^^ [22:29:23] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [23:13:51] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [23:31:29] New patchset: Platonides; "Remove the skipcaptcha rifht from autoconfirmed users in labs, for easy captcha testing." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/16866 [23:37:51] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [23:47:53] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours