[00:03:51] (03PS1) 10GWicke: Bug 55418: Exclude new Parsoid job types from default queue [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90271 [00:04:50] (03CR) 10jenkins-bot: [V: 04-1] Bug 55418: Exclude new Parsoid job types from default queue [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90271 (owner: 10GWicke) [00:08:41] odd timeout failure in the test setup [00:09:00] https://integration.wikimedia.org/ci/job/operations-mw-config-tests/4917/console [00:11:19] (03PS2) 10GWicke: Bug 55418: Exclude new Parsoid job types from default queue [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90271 [00:11:58] (03CR) 10GWicke: "Trivial change to see if the timeout in operations-mw-config-tests fixes itself" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90271 (owner: 10GWicke) [00:14:55] (03PS2) 10GWicke: Add job runners for new Parsoid job types [operations/puppet] - 10https://gerrit.wikimedia.org/r/90268 [00:22:56] * bd808 is learning to hate bots that touch zillions of pages [00:24:37] bd808: is this in response to the job queue length? [00:25:47] gwicke: My "purge all the things" manual job. It's bogged down in minwiki right now with a ton of bot related edits [00:26:20] hehe [00:27:00] we are slowly eating away https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Miscellaneous%20pmtpa&h=hume.wikimedia.org&v=823574&m=Global_JobQueue_length&r=hour&z=default&jr=&js=&st=1365625056&z=large [00:27:47] also largely bot-related [00:42:58] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [00:43:59] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [00:59:56] (03CR) 10Anomie: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88196 (owner: 10coren) [01:03:36] (03CR) 10coren: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88196 (owner: 10coren) [01:09:50] springle: https://en.wikipedia.org/wiki/Special:Log/Aaron_Schulz is still bad for me :/ [01:13:57] Aaron|home: ok. i havn't looked at that again since the email to you last week. will do [01:14:38] its the only remaining bad case I am *aware* of (getting all logs for a user with lots of them) [01:14:55] by type is fine [01:26:24] (03PS1) 10Ryan Lane: Remove hook definition calls on deployment server [operations/puppet] - 10https://gerrit.wikimedia.org/r/90277 [01:26:28] (03PS3) 10coren: Add templateeditor right, group, and restriction [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88196 [01:37:48] (03CR) 10Anomie: [C: 031] "Tested equivalent config changes in my local wiki, seems to work. +2 as far as I'm concerned, whenever someone is ready to deploy it." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88196 (owner: 10coren) [01:41:51] * Coren now tries to find documentation on how to push this to prod the right way. [01:43:55] Coren: Maybe look at https://wikitech.wikimedia.org/wiki/Deploy [01:44:19] Coren: or "ask Reedy" :) [01:45:20] bd808: Yeah, that pretty much only speak of deploying core or extensions, no so much for config. [01:45:30] But yeah, I asked Reedy. :-) [01:48:26] Coren: what config? [01:48:47] if it's just one file then it's just sync file once it's right on wherever (terbium?) [01:48:54] sync-file* i guess [01:49:30] unless there's some git-deploy but i don't think so? [01:50:01] (03CR) 10coren: [C: 032] "Simple change, consensus is there. Go go go." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88196 (owner: 10coren) [01:50:10] (03Merged) 10jenkins-bot: Add templateeditor right, group, and restriction [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88196 (owner: 10coren) [01:50:40] oh, *that* :P [01:51:44] !log marc synchronized wmf-config/InitialiseSettings.php 'Bug 55432: add templateeditor restriction level and user right to enwiki' [01:52:00] Logged the message, Master [01:55:04] Oh my. [02:10:40] ohhh myyy. [02:15:04] !log LocalisationUpdate completed (1.22wmf21) at Thu Oct 17 02:15:04 UTC 2013 [02:15:17] Logged the message, Master [02:28:15] !log LocalisationUpdate completed (1.22wmf20) at Thu Oct 17 02:28:15 UTC 2013 [02:28:28] (03PS1) 10Legoktm: Add MassMessage jobs to the high priority queue [operations/puppet] - 10https://gerrit.wikimedia.org/r/90280 [02:28:31] Logged the message, Master [02:31:29] Elsie: ^ [02:36:06] The extension determines the job name? [02:38:20] Elsie: "Oh my"? [02:40:24] Coren: It's a big step for en.wiki. [02:40:31] Preparing to replace libicu48:amd64 4.8.1.1-12 (using .../libicu48_4.8.1.1-12ubuntu0.1_amd64.deb) ... [02:40:32] And I opposed, FWIW. :P [02:40:46] I was just re-reading the discussion. [02:41:09] I think the whole idea kind of further ingrains an already awful page protection system at the expense of a quick fix. [02:41:16] Plus bureaucracy. [02:41:27] FlaggedRevs! [02:41:57] Elsie: As far as I'm concerned (wearing my volunteer hat) any change that puts more editing in the hands of users away from admins is a step in the right direction even if imperfect. [02:42:35] But from an ops/WMF point of view, this is a small change with clear consensus; so there really isn't any doubt it'd go live. [02:42:41] Right. [02:42:54] I knew all that when I voted. ;-) [02:43:15] It's not an unreasonable view, I just don't really like the direction. [02:43:33] Plus I think it requires a lot more overhead because of how it's implemented. [02:43:41] Where every page has to be specifically marked as template-protected. [02:43:46] I think? [02:43:53] I wonder if just applies to the whole namespace(s). [02:43:59] Does it include Module? [02:44:25] No, it's a new protection level; it can be applied to anything (though the proposal says it /should/ only be applied to templates and modules, IIRC) [02:44:42] And the really iffy templates can stay fully protected if necessary. [02:44:52] I can think of no sane reason to unprotect {{!}} for instance. :-) [02:45:28] I'm not worried about template vandalism. [02:45:30] Elsie: Yes, it's $wgJobClasses[] = 'massmessagesubmitJob'; or something [02:45:47] How does the job queue know which jobs to process? [02:45:51] The extension tells it? [02:45:57] $wgJobClasses['massmessageJob'] = 'MassMessageJob'; [02:45:57] $wgJobClasses['massmessagesubmitJob'] = 'MassMessageSubmitJob'; [02:46:00] Ah. [02:46:52] Coren: There's been a proliferation of user groups of the years. Perhaps it helps centralize power, but really it usually only seems to create bureaucracy. [02:47:09] And the underlying systems are still outside of any wiki's direct control. [02:47:14] I dunno, the whole thing is kind meh. [02:47:17] kinda * [02:47:40] Elsie: I suppose that's true, to a point, but then again the all-or-nothing approach of +sysop isn't really workable nowadays either. [02:48:09] Right. [02:48:21] I'd like more on-wiki control and more flexibility. [02:48:26] Similar to what we have with global groups. [02:48:42] I'm not sure I see the parallel? [02:48:55] Stewards can create arbitrary global groups with arbitrary user rights. [02:48:59] I think he means groups defined on the wiki [02:49:02] Local bureaucrats can't do the same. [02:49:03] Not in code [02:50:36] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Oct 17 02:50:36 UTC 2013 [02:50:43] Coren: I'm not being very coherent. My full thoughts were here: https://en.wikipedia.org/wiki/Wikipedia:Requests_for_comment/Template_editor_user_right [02:50:49] morebots is missing. [02:50:50] Again. [02:50:51] Logged the message, Master [02:50:52] I am a logbot running on tools-exec-08. [02:50:52] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [02:50:52] To log a message, type !log . [02:51:02] No, just unvoiced. [02:51:33] Different hostmask, I guess. [02:52:28] Elsie: Global rights don't actually work like that; you can indeed pick and chose the rights but only amongst existing one. [02:52:35] Right. [02:52:44] I don't think adding a new user right is necessary. [02:53:01] The overall problem here isn't strictly about who can edit templates. [02:53:09] Well, protection levels (and editing through them) does require a new protection level. [02:53:19] It's about how we let as many trusted people as possible do as much as possible. [02:53:26] The protection level is silly. [02:53:47] As I said on-wiki. There are better options for guarding against template vandalism. [02:54:02] Most of these pages are only protected due to high use. [02:54:19] Adding a protection level misses the much more obvious answer: gating updates. [02:54:24] Like we do with code review. [02:54:37] +2 LGTM [02:54:39] :-) [02:54:45] Well, yeah, but do you /really/ want to open a flagged revision can of worms? :-) [02:54:50] Well, [02:54:54] that's what I was about to say. [02:55:05] FlaggedRevs has such an awful reputation that... here we are. [02:55:09] FWIW, I fully supported (and support) FR. [02:55:16] Yeah. [02:55:38] I also remember you once had grand plans for a new world order, perhaps. [02:57:18] (03PS1) 10Jeremyb: bug 55796 - wgImportSources @ arwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90281 [02:57:22] (03CR) 10jenkins-bot: [V: 04-1] bug 55796 - wgImportSources @ arwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90281 (owner: 10Jeremyb) [02:58:00] Realism won out in the end. :-) [02:58:20] Heh, you joined WMF. :-) [02:58:54] Speaking of which... it's mid-October. [02:59:04] (03CR) 10Jeremyb: "recheck" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90281 (owner: 10Jeremyb) [02:59:06] I wonder what's happening with the ED search... [02:59:50] wtf ^^ [02:59:54] https://meta.wikimedia.org/wiki/Minutes_Meeting_19th_of_September [03:00:08] jeremyb: Jenkins troubles? [03:01:05] Elsie: well determinism is a good thing... [03:01:12] in this case at least! [03:01:43] i already mailed hashar about it [03:08:39] Elsie: Last I heard there were a couple of serious candidates in interview/meet mode. [03:09:24] Hmm, I wonder if Sue'll stick out the year, then. [03:16:41] She was pretty clear that she wouldn't leave before someone was at hand to take over and there was a reasonable cross over period. [03:24:08] PROBLEM - MySQL Processlist on db1027 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 324 statistics [03:25:08] RECOVERY - MySQL Processlist on db1027 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 0 statistics [03:33:43] Coren: Right, but I think the original timeline had been shooting for about now. [03:33:47] I think things went slower. [03:34:25] They did. Finding someone to fill those shoes was hard and the original timeline was overly optimistic. :-) [03:34:45] :-) [03:36:09] PROBLEM - MySQL Slave Running on db1027 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:36:09] PROBLEM - MySQL Slave Delay on db1027 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:36:18] PROBLEM - MySQL Idle Transactions on db1027 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:36:18] PROBLEM - MySQL Processlist on db1027 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:36:58] RECOVERY - MySQL Slave Delay on db1027 is OK: OK replication delay 0 seconds [03:36:58] RECOVERY - MySQL Slave Running on db1027 is OK: OK replication [03:37:08] RECOVERY - MySQL Idle Transactions on db1027 is OK: OK longest blocking idle transaction sleeps for 0 seconds [03:37:08] RECOVERY - MySQL Processlist on db1027 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 0 statistics [03:51:58] PROBLEM - SSH on lvs6 is CRITICAL: Server answer: [03:51:58] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [03:52:16] hrmmmm, lvs6 [03:52:50] Leslie and ma rk are both asleep i guess [03:52:58] RECOVERY - SSH on lvs6 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [03:53:08] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 28.50 ms [03:58:41] !log testing updateCollation on S6 with adaptive hash off [03:58:58] Logged the message, Master [04:07:16] PROBLEM - NTP on mw31 is CRITICAL: NTP CRITICAL: Offset unknown [04:12:16] RECOVERY - NTP on mw31 is OK: NTP OK: Offset 0.0007688999176 secs [04:16:38] I'm getting fatal errors. [04:16:49] http://meta.wikimedia.org/w/index.php?title=IRC/wikipedia/Guidelines&diff=prev&oldid=5724529 [04:16:59] > [f054a662] 2013-10-17 04:16:52: Fatal exception of type MWException [04:17:14] While logged out. [04:17:35] Logged in, too. [04:18:34] Elsie: its fine when I close the CN banner [04:21:30] I don't see a CN banner. [04:22:23] I did when I got the error [04:22:52] And now the diff loads? [04:23:07] A CentralNotice banner causing a fatal error would be unusual, but not impossible. [04:23:48] oh wait that was a diff >.> [04:23:53] nope [04:24:18] does the article use the translation extensionby any chance? [04:25:09] Lego found it. [04:25:12] https://bugzilla.wikimedia.org/show_bug.cgi?id=55808 [04:25:17] Bizarre. [04:25:39] yeah [04:25:41] bits is being slow for me [05:45:02] (03CR) 10Krinkle: [C: 04-1] "We're still waiting for operations to upgrade that server to modern standards (at the moment is has gone disconnected from regular mainten" [operations/puppet] - 10https://gerrit.wikimedia.org/r/37570 (owner: 10Jforrester) [05:46:30] mutante: any word on what will happen to the mail server that wikibugs runs on? I don't really understand how it works, but from what I understood when you explained it to me, it seems that server is outdated and/or needed to be upgraded to puppet? I don't think it makes sense though to have a server running software we can't configure. we've been waiting for over a year now to make a trivial ch [05:46:31] ange to it. [05:47:11] If operations can't make the upgrade happen, maybe we can have someone make the change manually? Surely we've change wikibugs in the past so it can't be blocked on this. [05:50:31] TimStarling: Ryan_Lane: paravoid: any of you know what the status is of that server and how we make changes there? [05:51:23] I haven't heard any updates [05:51:29] why are you waiting for things? [05:51:34] things don't happen just by waiting [05:52:16] Well, I'm more a curious observer in this case. [05:53:13] Wondering what went wrong. Was it puppetised at some point in the past? Presumably not, but then why is there merged code for it in puppet that is essentially dead and unused/untested? [05:53:40] If not, that means it's pre-puppet. How did we make changes back then? Can we still do that today? [05:54:37] it's never been puppetized [05:54:54] people merge strange things sometimes, that is not unusual [05:55:01] it can be changed by having someone with root access change it [05:55:47] I figured but it's nice to have that confirmed, thanks. [05:56:01] are we talking about the server that runs bugzilla? [05:56:27] you should make that change in the wikimedia/bugzilla/wikibugs project [05:56:42] the one that runs wikibugs and bugzilla mail, but could be the same as bugzilla itself I don't know. [05:57:28] TimStarling: There's a change in that repo waiting as well. But that repo doesn't contain the irc echoing afaik. [05:57:31] kaulen, yes? i puppetized it in may, but my patch was not up to standard https://gerrit.wikimedia.org/r/#/c/62404/ [05:59:20] what about moving wikibugs to labs like gerrit-wm? [05:59:23] just add that script [05:59:44] http://paste.tstarling.com/p/GiOxQA.html [06:01:06] that is /usr/local/bin/start-wikibugs-bot [06:02:23] How does that work for a client outputting to multiple channels (which is what Chad did in the operations/puppet version of wikibugs). I guess that's abtracted in the new puppet class for ircecho. [06:02:49] there's no such feature in the deployed version [06:04:02] puppet/**/ircecho/default.rb [06:04:38] udpmxircecho.py [06:04:40] hm.. [06:05:57] TimStarling: is that puppet code not deployed though for logbot? [06:06:06] Krinkle: the one deployed now is on mchenry i think [06:06:19] not kaulen [06:06:36] ah, mxircecho is for irc.wikimedia.org i guess. [06:06:48] not logbot or wikibugs [06:07:10] it does in fact have a mchenry hostmast [06:07:12] hostmask* [06:07:44] i was thinking that it somehow took advantage of being the same host where the bugzilla outbound mail was processed [06:07:52] but idk if that's true [06:08:05] jeremyb: don't squander a perfectly good opportunity for me to feel righteous indignation [06:08:19] the narrative works better if it's kaulen [06:08:24] yes, it does [06:08:31] jeremyb: Yes, it peeks in the mail directly afaik [06:08:32] if so then it would need an architecture change in order to run in labs [06:08:51] why do you want to run it in labs? [06:08:59] 17 05:59:20 < legoktm> what about moving wikibugs to labs like gerrit-wm? [06:09:23] jeremyb: all you need is a gmail account subscribed to wikibugs-l, and then a simple poll script that checks it [06:09:39] legoktm: ok, be my guest? :) [06:09:58] there are a number of irc bots written for reporting bugzilla bugs that can parse exactly that output [06:10:28] TimStarling: so more people can access it, easier to fix, doesn't need to be perfect to run in 'production' [06:10:48] but it will go down all the time [06:11:05] *all*! [06:11:10] heh [06:11:12] that's what labs does isn't it? [06:11:15] it'd still depend on that same mail server if its in labs [06:11:25] grrrt-wm doesn't go down that often any more [06:11:32] only hoop all over the internet first [06:11:51] "how do I change an unpuppetized file with no version control" [06:12:07] that is what you are having trouble with, right? [06:12:28] I mean, on the surface, you would think that it would be easier than changing a file which is puppetized [06:12:56] Well, it's a combination of that something else. Since we don't know exactly what change to be made because what is in version control is different. [06:13:56] well, I gave you the script you would have to change in a paste [06:14:18] a false start made by someone else shouldn't immediately stop all maintenance [06:15:10] sure [06:59:45] (03CR) 10Krinkle: "What issues were there? If it was just the duplicate bot, that was fixed by removing the start script (I9a3c6ac948be92567f), and needs som" [operations/puppet] - 10https://gerrit.wikimedia.org/r/60359 (owner: 10Dzahn) [07:43:39] (03CR) 10Nemo bis: "(There's also a discussion about this on #wikimedia-operations logs starting around 5.46 UTC today.)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/60359 (owner: 10Dzahn) [07:44:58] (03PS1) 10ArielGlenn: removing spence mgmt entry (rt 5440) [operations/dns] - 10https://gerrit.wikimedia.org/r/90292 [07:52:57] (03PS1) 10Mark Bergsma: Cleanup esams LVS checks [operations/puppet] - 10https://gerrit.wikimedia.org/r/90293 [08:00:51] (03PS2) 10Mark Bergsma: Cleanup esams LVS checks [operations/puppet] - 10https://gerrit.wikimedia.org/r/90293 [08:07:47] jeremyb: yes i was asleepies [08:07:51] jeremyb: i'm in athens right now at RIPE [08:10:34] (03PS3) 10Mark Bergsma: Cleanup esams LVS checks [operations/puppet] - 10https://gerrit.wikimedia.org/r/90293 [08:21:11] wow greek power is expanding [08:22:55] hehe [08:55:57] (03PS4) 10Mark Bergsma: Cleanup LVS checks [operations/puppet] - 10https://gerrit.wikimedia.org/r/90293 [08:56:32] (03CR) 10jenkins-bot: [V: 04-1] Cleanup LVS checks [operations/puppet] - 10https://gerrit.wikimedia.org/r/90293 (owner: 10Mark Bergsma) [09:00:02] (03PS5) 10Mark Bergsma: Cleanup LVS checks [operations/puppet] - 10https://gerrit.wikimedia.org/r/90293 [09:01:35] (03PS1) 10TTO: Fix stupid typo in eswikivoyage autopatrolled setup [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90296 [09:04:47] (03CR) 10Mark Bergsma: [C: 032] Cleanup LVS checks [operations/puppet] - 10https://gerrit.wikimedia.org/r/90293 (owner: 10Mark Bergsma) [09:05:38] mark: can you please check if /usr/local/bin/position-of-the-moon exsits on the servers? [09:06:09] on which servers? [09:07:08] mark: I see in puppetmaster.pp it should be absent (line 336) [09:13:52] (03PS1) 10Mark Bergsma: Fix misc_web LVS check [operations/puppet] - 10https://gerrit.wikimedia.org/r/90297 [09:14:33] (03CR) 10Mark Bergsma: [C: 032] Fix misc_web LVS check [operations/puppet] - 10https://gerrit.wikimedia.org/r/90297 (owner: 10Mark Bergsma) [09:14:52] matanya: i'm sure it is, feel free to remove that resource if you want that [09:15:18] Thanks, i will. [09:41:24] (03PS1) 10Mark Bergsma: Add ulsfo bits LVS monitoring [operations/puppet] - 10https://gerrit.wikimedia.org/r/90298 [09:42:04] (03PS2) 10Mark Bergsma: Add ulsfo bits LVS monitoring [operations/puppet] - 10https://gerrit.wikimedia.org/r/90298 [09:43:20] (03CR) 10Mark Bergsma: [C: 032] Add ulsfo bits LVS monitoring [operations/puppet] - 10https://gerrit.wikimedia.org/r/90298 (owner: 10Mark Bergsma) [09:49:13] (03Abandoned) 10Mark Bergsma: add LVS monitoring for ULSFO [operations/puppet] - 10https://gerrit.wikimedia.org/r/90257 (owner: 10Dzahn) [09:50:36] (03Abandoned) 10Mark Bergsma: add iptables accept from ulsfo for monitoring [operations/puppet] - 10https://gerrit.wikimedia.org/r/88724 (owner: 10Dzahn) [09:52:17] !log maxsem synchronized php-1.22wmf21/extensions/MobileFrontend/ [09:53:44] !log maxsem synchronized php-1.22wmf20/extensions/MobileFrontend/ [09:57:57] (03PS1) 10Mark Bergsma: Cleanup pmtpa LVS service monitoring as well [operations/puppet] - 10https://gerrit.wikimedia.org/r/90301 [09:59:06] (03CR) 10Mark Bergsma: [C: 032] Cleanup pmtpa LVS service monitoring as well [operations/puppet] - 10https://gerrit.wikimedia.org/r/90301 (owner: 10Mark Bergsma) [10:39:16] (03CR) 10Mark Bergsma: [C: 04-1] "(2 comments)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/88261 (owner: 10Dr0ptp4kt) [10:41:31] (03PS1) 10Mark Bergsma: Link IPv6 HTTPS LVS checks to the IPv6 host [operations/puppet] - 10https://gerrit.wikimedia.org/r/90304 [10:42:34] (03CR) 10Mark Bergsma: [C: 032] Link IPv6 HTTPS LVS checks to the IPv6 host [operations/puppet] - 10https://gerrit.wikimedia.org/r/90304 (owner: 10Mark Bergsma) [10:47:36] (03PS1) 10Mark Bergsma: Revert "Move ulsfo traffic back to eqiad" [operations/dns] - 10https://gerrit.wikimedia.org/r/90305 [10:48:15] (03CR) 10Mark Bergsma: [C: 032] Revert "Move ulsfo traffic back to eqiad" [operations/dns] - 10https://gerrit.wikimedia.org/r/90305 (owner: 10Mark Bergsma) [10:58:28] (03PS4) 10Matanya: ssh: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/90098 [11:02:53] (03PS1) 10Matanya: position-of-the-moon : absent on all servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/90307 [11:10:00] akosiaris: can you please review this ^ [11:11:00] (03CR) 10Akosiaris: [C: 032] position-of-the-moon : absent on all servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/90307 (owner: 10Matanya) [11:11:39] thank you [11:11:43] I had an idea for a better way to do this btw [11:11:48] :-) [11:11:53] I was telling akosiaris the other day :) [11:12:49] we should stop caring about the position of the moon :-) [11:12:49] basically, collect the keys on the puppet master and do file { '/etc/ssh/ssh_known_hosts': content => generate('/bin/cat /etc/ssh/ssh_known_hosts' } [11:12:59] for the rest of the servers [11:13:04] collect once, deploy everywhere [11:14:05] of course that will not allow for local known host additions [11:14:10] which may or may not be a plus [11:14:37] also you can just collect into the volatile file repo dir, and use that as a source [11:14:40] that is true [11:14:50] that might be better [11:14:52] esp. true for e.g. access to routers [11:14:59] yes [11:15:19] it has its pros and cons I guess [11:15:26] but then doing ssh key verification for our routers from every box we're access in them from is wrong too [11:15:36] because ssh_known_hosts is 1.6Mb and I have a bad feeling about increasing the catalog size for all servers by that amount suddenly [11:15:38] we should do this once [11:15:43] yup [11:15:48] akosiaris: hm, good point [11:16:02] it also needs clean up btw, as we don't clean up old hosts [11:16:32] meaning ? isn't generated by puppet ? [11:16:42] where all hosts in db are active hosts ? [11:16:43] additions are generated by puppet [11:17:00] puppet doesn't purge old hosts [11:17:02] resources that don't exist in puppet aren't purged from the file [11:17:10] I think there's a resource { ... } syntax to do that [11:17:21] but that would mean another O(n) scanning of the file [11:17:23] yes there is... naginator uses that... [11:17:41] I know naginator uses it, I'm not sure if it works for SSH though [11:17:45] probably it does [11:17:51] well is is generic [11:17:53] so yes [11:18:33] does it work generically enough for exported resources though ;) [11:19:58] http://docs.puppetlabs.com/references/latest/type.html#resources-attribute-unless_system_user [11:20:01] omg [11:20:28] architectural fail [11:23:27] note that you can define multiple known hosts files, so if we want to we can have multiple for different types [11:23:34] like have a static one for routers and a few other bits or something [11:23:41] or we can concatenate, but meh ;) [11:24:53] brb [11:25:53] I use five known host files [11:26:25] one for wmf (that I sync from bastion), one for wmfmisc, one for wmflabs, one for Debian and one for the rest [11:35:57] yep [11:51:37] paravoid: can't you just create a repo/db with known host that will be purged once in a while? [12:44:51] (03PS1) 10Springle: rename pt-query-digest arguments for v2.2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/90326 [12:46:27] (03CR) 10Springle: [C: 032] rename pt-query-digest arguments for v2.2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/90326 (owner: 10Springle) [12:55:40] (03CR) 10Springle: [C: 031] "Seems ok to me. From the perspective of coredb I think only the ganglia/icinga stuff could be affected." [operations/puppet] - 10https://gerrit.wikimedia.org/r/88666 (owner: 10Andrew Bogott) [13:00:27] (03PS1) 10Springle: rename pt-query-digest arguments for v2.2 ... in another place [operations/puppet] - 10https://gerrit.wikimedia.org/r/90328 [13:01:23] (03CR) 10Springle: [C: 032] rename pt-query-digest arguments for v2.2 ... in another place [operations/puppet] - 10https://gerrit.wikimedia.org/r/90328 (owner: 10Springle) [13:29:31] (03CR) 10Ottomata: "(1 comment)" [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/90028 (owner: 10Edenhill) [13:31:07] (03CR) 10Ottomata: [C: 032] Log failed Kafka message deliveries (issue #1) [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/90029 (owner: 10Edenhill) [13:32:05] (03CR) 10Ottomata: [C: 032] Provide some more detail when Kafka ..produce() fails. [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/90030 (owner: 10Edenhill) [13:34:30] (03CR) 10Ottomata: [C: 032] Added rate-limiting to (most) error logs generated by varnishkafka (issue #1) [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/90184 (owner: 10Edenhill) [14:01:53] (03PS1) 10Cmjohnson: removing decom'd servers: arsenic, niobium, palladium, strontium from file [operations/puppet] - 10https://gerrit.wikimedia.org/r/90344 [14:03:29] drdee: can you take a look at that ^ make sure I am not breaking anything..thx [14:12:09] done [14:14:48] thx [14:15:19] (03CR) 10Cmjohnson: [C: 032] removing decom'd servers: arsenic, niobium, palladium, strontium from file [operations/puppet] - 10https://gerrit.wikimedia.org/r/90344 (owner: 10Cmjohnson) [14:30:37] !log dist-upgrade & reboot payments* [14:30:49] Logged the message, Master [14:32:28] (03CR) 10Akosiaris: [C: 04-1] "(5 comments)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/88666 (owner: 10Andrew Bogott) [14:33:30] cmjohnson1: afaics the tampa redis boxes are fine to remove. the eqiad ones are still setup to replicate, but not doing so, obviously. so... *shrug* [14:33:41] Coren: akosiaris ^ any thoughts [14:34:04] context: https://rt.wikimedia.org/Ticket/Display.html?id=5969 [14:34:14] springle: That's my understanding as well, but my confidence level is only ~80% [14:35:01] springle...should we remove the entire stanza for redis or simply remove the old tampa boxes which will one day be replaced? [14:35:21] i really don't know guys... [14:36:47] the DNS don't even exist for these machines [14:37:06] nope...they're complete removed...the switch is gone [14:37:32] LeslieCarr: yeah, i was factoring that in [14:37:34] :) [14:37:37] in that case... fine to remove. [14:37:42] :P [14:37:57] i see no harm in removing the mcNN => mcNNNN defs from $redis_replication if those boxes are gone [14:38:23] but the array itself is still referenced in erb's [14:47:39] PROBLEM - Host payments1003 is DOWN: PING CRITICAL - Packet loss = 100% [14:50:19] RECOVERY - Host payments1003 is UP: PING OK - Packet loss = 0%, RTA = 0.74 ms [14:58:01] (03CR) 10Akosiaris: [C: 031] "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90268 (owner: 10GWicke) [14:58:05] (03CR) 10Andrew Bogott: [C: 04-1] "(2 comments)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90098 (owner: 10Matanya) [15:06:10] (03PS5) 10Matanya: ssh: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/90098 [15:06:29] hi andrewbogott. Thanks for all your help [15:06:37] thanks for yours! [15:06:48] Do you want me to talk you through setting up a test platform to verify that your patch is a no-op? [15:06:50] regarding labs, I don't have an env there for tesing [15:07:10] that would be nice andrewbogott [15:07:10] ok, let me add you to my project... [15:08:17] ok -- now you can create an instance in project 'puppet'. That's where I've been running all my puppet tests. [15:08:25] mark, did you have a moment to review https://gerrit.wikimedia.org/r/#/c/90095/ [15:08:31] Just a basic small instance, you shouldn't need to change any of the initial settings. [15:08:49] hashar helped with that patch, it should be just what is needed [15:12:04] (03CR) 10Andrew Bogott: "(3 comments)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90098 (owner: 10Matanya) [15:16:33] matanya, is that making sense so far? When I create a new instance I generally log in and watch it until I can get a clean puppet run before doing anything else [15:17:19] (03PS6) 10Matanya: ssh: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/90098 [15:17:55] ok, andrewbogott I'll log in and check out the instance [15:18:15] (03CR) 10Andrew Bogott: [C: 031] "Looks good! Let's run some tests before merging." [operations/puppet] - 10https://gerrit.wikimedia.org/r/90098 (owner: 10Matanya) [15:18:44] paravoid: not sure if mark is around - could you take a look please -- https://gerrit.wikimedia.org/r/#/c/90095/ [15:19:06] I really need this to start figuring out why ESI is not working [15:19:09] andrewbogott: how do i log into your project? [15:19:25] matanya, you don't need to log in, just add 'puppet' to the project filter up top. [15:19:32] Well, you need to log into wikitech, but I presume you've done that already [15:21:01] didn't quite get that andrewbogott [15:21:21] matanya: have you used labs/wikitech/etc before? [15:21:39] the labs shell only [15:21:49] Ah, so not the web interface? [15:22:11] i don't use gui match :P [15:22:17] yurik_: can you elaborate on the esi issue? is esi running in labs? [15:22:20] i did find the filter [15:22:36] matanya, yeah, ok, so visit https://wikitech.wikimedia.org/wiki/Special:NovaInstance [15:22:47] add 'puppet' to the filter, click 'set filter' [15:22:54] and then you should have a link to add an instance to the puppet project [15:24:38] i see, i must leave now. bbl. thanks for every thing andrewbogott [15:25:09] matanya: it takes a few minutes for the instance to spin up, so you should start yours before you go :) [15:26:00] ok done [15:30:42] (03CR) 10Mark Bergsma: [C: 032] "enable_esi" setting to the mobile beta labs varnish [operations/puppet] - 10https://gerrit.wikimedia.org/r/90095 (owner: 10Yurik) [15:31:12] (03PS3) 10Mark Bergsma: "enable_esi" setting to the mobile beta labs varnish [operations/puppet] - 10https://gerrit.wikimedia.org/r/90095 (owner: 10Yurik) [15:31:41] (03CR) 10Mark Bergsma: [C: 032] "enable_esi" setting to the mobile beta labs varnish [operations/puppet] - 10https://gerrit.wikimedia.org/r/90095 (owner: 10Yurik) [15:31:53] thx mark [15:48:26] ori-l: Bah! [15:48:27] reedy@tin:/a/common$ ./multiversion/checkoutMediaWiki [15:48:27] PHP Warning: require_once(./defines.php): failed to open stream: No such file or directory in /a/common/multiversion/checkoutMediaWiki on line 4 [15:48:27]