[00:08:54] New review: Tim Starling; "The code implies that this error occurs when the search *query* is empty, not the search results. Th..." [operations/debs/lucene-search-2] (master) C: -1; - https://gerrit.wikimedia.org/r/60860 [00:16:53] paravoid: ping [00:16:58] Guest83333: pong [00:17:15] paravoid: May I PM? [00:17:35] sure [00:17:39] you don't have to ask first :) [00:24:03] Change merged: Catrope; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60911 [00:26:05] WTF [00:26:16] Did someone mess with host SSH keys? [00:26:25] what key? [00:26:44] I'm getting "The authenticity of host 'mw1' couldn't be established" for every host when I try sync-file [00:27:51] I get [00:27:52] mw76: Warning: Permanently added 'mw76,10.0.11.76' (RSA) to the list of known hosts. [00:27:53] mw77: Warning: Permanently added 'mw77,10.0.11.77' (RSA) to the list of known hosts. [00:27:53] mw75: Warning: Permanently added 'mw75,10.0.11.75' (RSA) to the list of known hosts. [00:27:53] mw81: Warning: Permanently added 'mw81,10.0.11.81' (RSA) to the list of known hosts. [00:28:25] heh, we must have different SSH options then [00:28:45] But doesn't this mean something must be screwed up if we both get this for *all* hosts? [00:28:45] related to Leslie upgrading them? [00:29:04] Probably not [00:29:05] were their addresses changed? [00:29:16] Because it's also doing it for hume, bast1001 and searchidx1001 [00:29:26] no, i don't think so [00:29:48] strict host key validation would be useful if we had some way for securely updating ssh_known_hosts [00:30:01] but last time I checked, we didn't, that's why I have it disabled [00:31:20] ha ha [00:31:22] -rw------- 1 root root 2562362 Apr 26 00:11 ssh_known_hosts [00:31:27] helps if you can read the file [00:31:31] did stuff like ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R bayle [00:31:32] *sigh* [00:31:41] to remove the one for bayle..f.e. [00:32:06] TimStarling: Mind if I make that 644? [00:32:10] I did it already [00:32:12] OK [00:32:14] seems to work now [00:32:25] Syncing now [00:32:37] !log catrope synchronized wmf-config/CommonSettings.php 'Remove wmgVisualEditorHide' [00:32:39] !log on fenari: fixed permissions on /etc/ssh/ssh_known_hosts [00:32:45] Logged the message, Master [00:32:53] !log catrope synchronized wmf-config/InitialiseSettings.php 'Clean up VisualEditor config' [00:32:53] Logged the message, Master [00:33:01] Logged the message, Master [00:35:46] New review: Dzahn; "just adding comments and makes lint check happy:)" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/60851 [00:35:53] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60851 [00:42:20] Ryan_Lane: ping [00:42:26] sup? [00:42:35] Ryan_Lane: http://techcrunch.com/2013/04/25/facebook-parse/ [00:42:59] is that what you've been working on? :) [00:43:13] Ryan_Lane: no [00:43:21] Ryan_Lane: that means that Ben Hartshorne is now at Facebook [00:43:27] oooohhhhh [00:43:28] right [00:44:09] I forgot he moved there [00:44:11] Ryan_Lane: see https://www.parse.com/about/team [00:44:41] Ryan_Lane: click on Ben's head and read his bio — it's pretty cool [00:46:18] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:18] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:18] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:28] heh [00:48:29] New patchset: Jforrester; "wikibugs: Set up #mediawiki-visualeditor" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37570 [00:51:17] Change merged: Kaldari; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60947 [00:52:08] > [00:52:08] To apply via API, send a POST to: [00:52:09] https://www.parse.com/jobs/apply [00:52:09] > [00:52:10] Heh. [00:53:20] !log catrope synchronized php-1.22wmf2/extensions/VisualEditor/ 'Update VisualEditor to master' [00:53:28] Logged the message, Master [00:54:01] Susan: ha ha ha you should apply [00:54:12] Susan: the Facebook deal isn't closed yet ;-) [00:57:14] !log kaldari synchronized wmf-config/InitialiseSettings.php 'temporarily disabling WikiLove per bug 47457' [00:57:22] Logged the message, Master [00:59:08] you broke WikiLove... [01:02:50] odder: Broke? It's been disabled. [01:03:30] Susan: I know. [01:03:40] What do you see that's broken? [01:04:05] see bug :) [01:05:04] https://bugzilla.wikimedia.org/show_bug.cgi?id=47457 ? [01:05:10] I'm not following you. [01:14:03] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [01:54:24] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [01:55:25] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 13 seconds [02:10:04] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [02:12:39] !log LocalisationUpdate completed (1.22wmf2) at Fri Apr 26 02:12:38 UTC 2013 [02:12:47] Logged the message, Master [02:14:24] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication d [03:45:44] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 26 03:45:44 UTC 2013 [03:45:51] Logged the message, Master [04:58:15] why don't we use NAT? [04:59:17] in what way is a SOCKS proxy more secure or otherwise better than a specific firewall rule on a NAT server? [04:59:57] i suppose you could log requests or limit to certain domains/paths or something [05:00:25] harder to use on the client and a little more flexibility for the man in the middle. i think [05:03:16] I'm just considering the options for yet another internal/external relay [05:03:23] in particular IRC [05:04:29] it just seems like a waste of time, if we could have a working default next hop [05:05:59] what's this relay for? [05:15:21] it's for logmsgbot [05:15:54] the deployment scripts on tin have to send messages to freenode [05:20:18] oh, right [05:20:49] i suppose you could use udp (i think you mentioned that before) or salt or kafka [05:21:24] but maybe i'm mixing up irc bots [05:24:56] i guess i'm undecided about whether it's worth blocking outbound [05:29:46] there are many options [05:30:15] it's one of those things where you have to decide what camp you belong to before you can choose a solution :) [05:30:18] * jeremyb_ was limiting to stuff already in use :) [05:31:18] I don't have an opinion about the network setup, but I think there's a high-level reason for preferring the use of some message bus, and that's to allow people write software to react to what is happening on the cluster in useful and interesting ways [05:31:28] yeah, I knew you would like zmq [05:31:42] in fact I have just been reading its documentation for that reason [05:32:08] oh. you have to make it past the whole idiotic "it's a radioactive isotope that cures AIDS" nonsense [05:33:50] New patchset: Stefan.petrea; "Added packages needed for Jenkins environment" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60965 [05:34:02] of course IRC is also a message bus and a good one for certain applications, but it's also annoying to parse and annoyingly nontrivial to remain connected to [05:34:05] hello Operations [05:34:13] I'm from the Analytics team [05:34:15] please merge https://gerrit.wikimedia.org/r/60965 [05:34:23] I like that, straight & to the point [05:34:34] there are packages needed by the Jenkins environment on the Jenkins environment [05:34:56] hello average [05:34:59] hi TimStarling [05:34:59] do you have a name? [05:35:03] I do [05:35:07] Stefan Petrea [05:36:41] that's my name [05:36:47] if you wait another couple of hours, antoine will be awake and will be able to sort that out for you [05:36:57] or is it more urgent than that? [05:37:15] TimStarling: not very urgent, can wait [05:37:28] ori-l: for the IRC part of it, I've been looking at two solutions [05:37:36] just a bunch of packages that need to be present on the Jenkins [05:37:51] one is the python irclib package which we use already for this and various other things [05:38:06] average: You should add reviewers to your Gerrit changesets. [05:38:22] the other is supybot, which does a million things by default, but I've just confirmed that it can be configured to not do any of those default things, and just run a single plugin instead [05:39:20] What's zmq? [05:39:31] zeromq [05:39:32] yeah, I've written supybot plugins before. If you're just looking for "stay connected to IRC and behave yourself", you can look at Twisted's IRC lib, which IIRC supybot uses under the covers [05:39:35] supybot happens to have a plugin already which accepts IRC lines via a simple text protocol via TCP [05:39:35] it's a distributed message queue [05:39:39] Susan: ^^ [05:39:42] https://github.com/plq/zmq [05:39:47] That? [05:39:51] the protocol includes authentication [05:40:00] Susan: no [05:40:00] https://en.wikipedia.org/wiki/Zmq doesn't exist, so I'm kinda lost. [05:40:09] Susan: http://www.zeromq.org/ [05:40:13] Ah. [05:40:14] ottomata: hi sir :) [05:40:26] Susan wasn't that far off, that is a client for it [05:40:45] hi ori-l [05:40:53] hi average [05:41:01] anyway, either way, the IRC part is python [05:41:06] I thought everyone used XMPP. [05:41:23] now, you can do a simple TCP socket server in about 5 lines of code in python [05:41:35] Susan: uhm.. in the Analytics team we do IRC and Google Hangouts [05:41:35] probably a ZeroMQ listener is about the same number [05:41:45] Susan: not so much XMPP(or Gtalk) .. [05:41:50] they're equivalent from that perspective [05:42:11] the difference would mostly be in how hard they are to puppetize [05:42:17] and I think plain TCP would win there [05:42:18] ori-l: you know I though your isotope bit was sarcasm...and then it wasn't [05:42:33] average: but hangouts are XMPP? [05:42:37] maybe [05:42:38] *thought [05:43:13] Susan: I'm talking about a bot which is in this channel right now [05:43:18] it can't be trivially moved to XMPP [05:43:27] TimStarling: puppetizing zmq is package { 'python-zmq': ensure => present, } [05:44:18] but TCP works fine. the nice thing zero gives you is multiple subscribers and some in-memory buffering to smooth over brief disconnects [05:44:41] seems like overkill in this context [05:44:42] salt uses zeromq as its message bus and ryan deployed salt everywhere [05:44:46] i wonder how deploys get to graphite now [05:44:55] you could have graphite just be an extra listener [05:45:05] I am imagining a venn diagram of the features I need and the features various things will give me [05:45:28] TimStarling: as you said, it's the same python code. zmq mimics the familiar BSD socket API so they're not just both five lines, they're more or less the same five lines [05:45:39] and supybot and zeromq are a fair way towards overkill by that measure [05:46:43] supybot is, certainly [05:47:12] but I get paid if you use ZeroMQ, as the official Wikimedia cheerleader [05:47:37] then there is the question of whether to extend ircecho or to do my own new thing [05:47:51] what's ircecho? [05:47:57] ircecho annoys me in a number of ways, and I would be tempted to fix those things if I went my own way [05:48:04] Is the goal to get rid of the bots? [05:48:09] it's what logmsgbot uses at the moment [05:48:38] ori-l: https://wikitech.wikimedia.org/wiki/Ircecho [05:48:44] logmsgbot, gerrit-wm, and icinga-wm are all ircecho [05:48:47] it reads from a file like "tail -f" and writes to IRC [05:49:01] ircecho is just a way to pipe `tail -f` to a channel [05:49:04] it originally actually used tail -f | ircecho [05:49:09] Susan: thank you for adding Antoine to the ticket [05:49:17] but Ryan rewrote it to use inotify [05:49:21] No problem. [05:49:29] apparently not realising that tail -f also uses inotify now [05:49:31] I think getting rid of morebots would be nice. [05:49:32] that's actually kind of nice, I don't know why people hate pipes [05:49:40] If something could do all of that more ... directly. [05:49:48] But I guess there are different purposes. [05:50:22] anyway, it's got no configuration file [05:50:40] the configuration code that should be written in python is instead written in ruby [05:50:56] what's got no configuration file? [05:50:58] 40 lines of ruby code embedded in a puppet template [05:51:06] oh, the ruby i wrote? :) [05:51:07] ircecho [05:51:28] which eventually sets the command line parameters via the default file [05:52:09] i'm itching to write this now; are you sure you want to? [05:53:16] I am pretty sure that I don't want to, so go right ahead [05:53:38] hehe, TimStarling has people writing stuff for him left and right [05:53:45] first Coren, now ori-l [05:53:59] is Coren's thing deployed? [05:54:41] https://gerrit.wikimedia.org/r/#/c/58449/ [05:54:45] no, he still hasn't amended it since my review [05:56:25] has irclib been reliable? [05:56:40] yes, I think the ircbot part of it is quite reliable [05:57:03] ok, i'll just use that [05:57:31] and what else? zmq? [05:57:52] i'm contractually obliged to at least 'import zmq' in the header, but other than that i can use tcp [05:58:07] heh [06:04:02] TimStarling: you realize it was written in python before I made changes to it right? [06:04:10] and that it was broken in a number of ways [06:04:29] also, the ruby config thing I −1'd a number of times [06:04:30] ohi Ryan_Lane [06:04:39] I'd prefer a config file in yaml [06:04:49] stuck in /etc, installed by puppet [06:04:58] I saw you write that on another change, about YAML configuration [06:05:19] YAML did a bad thing to me when I was a child so my automatic reaction was to scoff at that [06:05:21] I chose to go the inotify route because I needed to read multiple files (gerrit) [06:05:24] hahaha [06:05:32] but then, I had to think, maybe configuration is the only thing YAML is actually useful for [06:05:34] well, you could go with json or paste, or a ini files, etc [06:05:45] I only like YAML for config [06:05:49] I use JSON for everything else [06:05:50] or XML [06:06:13] there used to be a YAML output format in the MW API, using a YAML library that had been imported without review [06:06:15] ini is a pain in the ass [06:06:26] paste is just a harder to use ini [06:06:28] and it turned out to be ridiculously full of bugs [06:06:39] heh [06:06:57] so I ripped it out, and now if you ask for format=yaml, it will give you JSON, since that is a strict subset [06:06:59] I remember this YAML library [06:07:22] I had to add it back into OpenStackManager because cloud-init only accepts YAML [06:07:29] maybe we're not using that anymore in it [06:08:09] it takes a lot of code to achieve beautiful output which actually round-trips correctly, that's my fundamental criticism of it [06:08:10] nah. still there. I hate cloud-init [06:08:40] JSON is a much better format [06:08:47] I like it for config because it's easier for humans to edit [06:09:08] it --> YAML [06:09:13] sorry [06:09:13] yes [06:10:07] anyway, yes, I can believe that YAML would work for configuration [06:10:08] anyway, I have no strong feeling about how tin -> a public host is implemented [06:10:28] I wrote on the gerrit change that we could just use socat [06:10:30] there's lots of ways to do any. none are necessarily superior to others [06:10:34] but that turns out to be wrong [06:10:38] Anyway, . [06:10:38] oh? [06:11:05] since that would additionally need a firewall to stop it from being an anonymous open proxy for connecting to freenode [06:11:33] right [06:11:44] but, I was just reading the socat man page, and it does actually have some authentication parameters in there [06:11:57] but it doesn't matter if ori-l is just going to do it in python [06:12:03] one thing I was doing in git deploy was writing the messages into redis to be picked up, since I was already storing things in redis [06:12:14] but in this case it's absurd overkill [06:13:07] yeah, if you're going to use something high-level, I wouldn't have thought redis would be at the top of the list [06:13:29] the positive about doing it in redis for git-deploy was that the messages wouldn't get lost [06:13:29] there's a lot of message queue servers [06:13:49] yeah, I was just reusing what was there, rather than using an additional thing [06:14:47] there's lots of better things to use otherwise [06:15:01] what do you think of my idea of using NAT? it would solve the general class of problems [06:15:05] zmq is an easy one since it's already installed everywhere [06:15:24] you'll have to get mark's opinion on that. I dislike NAT generally [06:15:45] I suppose he was the one who decided to not use it in the first place [06:16:16] I'm pretty sure we had NAT in the olden days [06:16:19] maybe on albert? [06:16:32] before my time :) [06:17:21] it's not very often the internal hosts need to connect to the outside world [06:17:33] this is the only really recent example I can think of [06:17:58] https://wikitech.wikimedia.org/w/index.php?title=Albert&oldid=12509 [06:18:35] maybe james day set it up... [06:19:34] who would be the best person to assign https://bugzilla.wikimedia.org/show_bug.cgi?id=30861 too? doesn't robh normally handle the domain names? [06:19:45] yep [06:19:52] there is an argument that a botnet client wouldn't be able to dial home if it has no network access [06:19:55] and sometimes mutante-away [06:20:15] but if it was a back door made for wikimedia, it could just dial home via one of the HTTP proxies [06:20:47] well, AFAIK our forward proxies only allow access to specific plaes [06:20:54] *places [06:20:58] this keyboard is killing me [06:21:08] p858snake|l: oh, *that* one [06:22:24] I can get any website out of brewster [06:22:36] hm. well, that's lame. [06:22:57] do you know the HTTP copy upload feature? I think that is enabled now [06:23:09] it really should only allow access to apt security, and a few other places [06:23:27] yeah, it's enabled on commons [06:23:33] ohhhhhh [06:23:38] I totally forgot about that [06:23:53] that's linne [06:23:56] yep [06:24:28] * Aaron|home dislikes the code for that [06:24:48] that's mostly there because we don't support large enough uploads, right? [06:25:27] or is it also there to get images and such from flickr and other locations? [06:26:48] I thought it was the later, the I guess it works for both [06:27:07] yeah, I'm not sure why it was enabled [06:27:47] does chunked upload work now? [06:28:20] it should [06:29:07] "a feature to import photographs from Flickr (started as a Google Summer of Code project)" is mentioned in the blog as a bullet point [06:29:23] !b 30861 | p858snake|l [06:29:23] p858snake|l: https://bugzilla.wikimedia.org/30861 [06:31:11] the only place that's enabled AFAIK is flickr [06:35:51] http://ankuranand.in/gsoc/wordpress/?p=83 [06:37:45] !log Restarted Jenkins. It was stuck in some while loop doing tons of stat() on unexisting file :( Would be up in roughly 40 minutes :( [06:37:53] Logged the message, Master [06:38:03] Hi hashar. [06:38:34] hashar: you've been added to a gerrit change by Susan for average [06:38:50] for average, [06:39:08] average_drifter of the analytics tea [06:39:09] m [06:39:23] it's me, average [06:39:26] sorry can't parse that, I am barely awake [06:39:33] hashar: my name's Stefan Petrea [06:39:48] bonjour. had coffee? [06:40:05] i also identify as "barely awake" [06:40:27] oh just saw Jenkins is being restarted [06:40:40] yeah had too sorry :( [06:40:54] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [06:40:54] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [06:41:04] it is causing me more and more headhaches [06:48:59] average: so I guess ops would be able to install the perl packages :-D [06:49:32] average: I don't have merge rights on ops/puppet [06:51:52] back to bed [06:53:46] * jeremyb_ figured the answer would be something like that... [07:17:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:18:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [07:31:19] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [08:12:59] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:44] New patchset: Hashar; "wikistats packages needed for Jenkins environment" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60965 [08:57:13] New review: Hashar; "PS2: moved the packages from `contint` module to the new misc::wikistats::package class." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/60965 [09:31:57] Change merged: MaxSem; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60943 [09:34:28] New patchset: Ori.livneh; "Add simple TCP->IRC forwarding bot" [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/60972 [10:00:01] MaxSem: what's the reason for the current 666 / 302 redirect handling in varnish for m/zero? [10:00:08] why isn't that done in mediawiki? [10:00:21] * MaxSem LOOKS [10:00:29] ehm, caps [10:00:31] :) [10:06:02] I guess because it doesn't use cache space this way [10:06:40] aha [10:06:51] I recall some of this [10:07:20] so the problem is that some carriers allow only particular language domains for free [10:08:42] thus, redirection has to be done sooner rarther than later to prevent people from paying for visiting other domains [10:09:44] but yeah, I explained Adam and Yuri that most of it, if not all, can be done in PHP (requires m. and sero. domain portals) [10:09:55] yeah [10:10:16] I believe we should make the apache/mediawiki layer working directly with m and zero urls [10:10:23] without all the rewriting in varnish [10:10:31] now of course, our apache config is a huge mess [10:10:33] that doesn't help [10:10:36] the issue is further complicated that changing even some details requires coordination with a lot of carriers [10:12:41] btw mark, I don't see any changes in varnish hit rate so far [10:13:49] MaxSem: https://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&c=Mobile+caches+eqiad&h=cp1041.eqiad.wmnet&v=65608808&m=varnish.fetch_length&jr=&js=&vl=N%2Fs&ti=Fetch+with+Length [10:13:56] backend fetches have gone down a bit [10:14:03] unfortunately the monthly graphs are fucked :( [10:14:16] so you know what the problem with the hit rate is? [10:14:25] the stupid purges are taken into account as well [10:14:34] and on the mobile cluster, there are nearly as many purge requests as normal requests [10:14:36] d'oh [10:14:39] so that skews the hit rate heavily :( [10:15:03] with squid that's not the case [10:15:10] and on upload, we have far more requests, so it's not as obvious [10:16:15] what was the problem with https://gerrit.wikimedia.org/r/#/c/58729/ ? [10:16:38] i reverted it because the nagios monitoring triggered [10:16:44] as it got a 301 instead of the expected pattern [10:16:50] but then I corrected that and put it back in place [10:16:58] but now I think it's related to the new 301 issue you're seeing [10:17:33] well, it's not causing it, it was hiding it before ;) [10:42:51] apergos: does replag in (0,1) interval get reported as 0 or 1 on dbtree? [10:43:18] I don't know [10:44:41] :( [10:44:57] is there a way to get a more precise reporting? ganglia maybe? [10:46:57] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:57] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:57] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [10:48:47] http://ganglia.wikimedia.org/latest/graph_all_periods.php?title=mysql_slave_lag&mreg[]=^mysql_slave_lag%24&hreg[]=db10%2807|24|28%29&aggregate=1&hl=db1007.eqiad.wmnet|MySQL%20eqiad,db1024.eqiad.wmnet|MySQL%20eqiad,db1028.eqiad.wmnet|MySQL%20eqiad [10:57:11] uh [10:57:11] I don't know how either of those gets its information [10:57:11] would have to go look at the code for the checks [11:14:47] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [11:19:12] New patchset: Mark Bergsma; "Support labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60977 [11:20:33] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60977 [11:21:24] New patchset: Aude; "(bug 47620) Exclude user and talk pages from wikidata features in clients" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60978 [11:22:10] New patchset: Ori.livneh; "Add simple TCP->IRC forwarding bot" [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/60972 [11:23:49] New patchset: Mark Bergsma; "Include the configuration class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60979 [11:24:25] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60979 [11:29:18] New patchset: Mark Bergsma; "Fix $override_hostname template lookup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60980 [11:30:15] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60980 [11:40:37] New patchset: Mark Bergsma; "Use the new ganglia monitor in Labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60982 [11:41:08] New patchset: Mark Bergsma; "Use the new ganglia module for monitors in Labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60982 [11:42:37] New patchset: Mark Bergsma; "Use the new ganglia module for monitors in Labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60982 [11:44:30] is mediawiki using Net_IPv4 or anything similar? [11:44:37] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60982 [11:58:40] Change abandoned: MaxSem; "Going the way of https://gerrit.wikimedia.org/r/#/c/60945/ instead" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60879 [12:10:07] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [12:21:55] New patchset: Mark Bergsma; "Disable the proxy for non pmtpa/eqiad hosts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60989 [12:23:27] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/25002 [12:23:44] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/38720 [12:25:06] New patchset: Mark Bergsma; "Disable the proxy for non pmtpa/eqiad hosts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60989 [12:25:49] alrighty, daily pleas for review time!:) [12:25:51] https://gerrit.wikimedia.org/r/60680 [12:26:00] https://gerrit.wikimedia.org/r/60945 [12:26:05] https://gerrit.wikimedia.org/r/60945 [12:26:12] pretty please:) [12:28:18] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60989 [12:31:22] New patchset: Mark Bergsma; "Create role::logging::eventlogging" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60680 [12:32:20] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60680 [12:36:00]