[00:00:07] and each different network can have a different network node [00:00:09] dan-nl: don't think so, as long as things aren't broken, I'm happy :) [00:00:20] I'll be honest in that I haven't looked at how to configure that [00:00:23] nothing is broken as far as i know :) [00:00:27] hm [00:00:30] that might not be so bad [00:00:35] is that possible now? [00:00:52] mutante: have you seen my mail about bugzilla audit mails? (another one just arrived) [00:01:04] I haven't used neutron at all [00:01:11] cool, david and i will test further later today [00:01:11] well, i think that the multiple rows is because we just didn't plan out and have servers randomly around ;) [00:01:21] it should be possible in nova-network, since it supports network node per compute node [00:01:22] thanks for all your help! [00:01:28] but... multiple network nodes would be more awesome [00:01:32] but we want to switch to neutron which does not currently support that [00:01:36] paravoid: no i havent, is it very annoying? [00:01:38] oh [00:01:46] paravoid: i can make sure to turn it off, it's on my plate [00:01:57] mutante: it is, it's also wrong, mails --@ and some other weird aliases like that [00:02:10] dan-nl: thank you. g'night! [00:02:11] it's pretty late in the game to be changing this kind of stuff up ;) [00:02:12] mutante: also mails RT and continues appending to a what is now a monstrous RT ticket [00:02:25] paravoid: sorry about that, its from non-prod , i'll stop it [00:02:38] my billable time might be a little low to investigate this [00:02:43] paravoid: the plan was once that it'd be smart about using RT like that because i have the tickets and it nags me [00:02:45] I'd say get mhoover on it [00:02:46] if there is an issue [00:03:03] (03CR) 10jenkins-bot: [V: 04-1] Collection Renderer (Now a module!) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 (owner: 10Mwalker) [00:03:25] there must be a difference to kaulen, because the same ones there have been running all the time [00:03:33] without doing this part wrong [00:03:47] so its me changing the bugzilla puppet to module [00:04:23] (03PS6) 10Mwalker: Collection Renderer (Now a module!) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 [00:04:47] so, if the vlan is split this means that traffic between instances will need to be routed through the network node [00:05:09] !log Reloading zuul to deploy I6df1f4acccd7a9 [00:05:23] Logged the message, Master [00:05:27] is it smart in the way instances are allocated? e.g. each tenant into one physical network/network node? [00:05:41] if so, I doubt we have much tenant-to-tenant traffic [00:06:02] right now we have one large flat network [00:06:24] and one vlan [00:06:51] eqiad's network design is very different than tampa's [00:06:53] it does not assign instances into a single physical network or network node [00:07:28] so [00:07:45] virt1000 & virt1009 are in A4 & A2 respectively [00:07:55] virt1001-virt1008 are in the same *rack* [00:07:55] virt1000 doesn't matter [00:07:58] not row, *rack* [00:08:18] assuming the compute nodes and the network node are in the same rack we're fine [00:08:42] or row [00:08:42] lose a switch = lose labs [00:08:42] scary [00:08:42] otoh, we have all of memcached in one rack, so... [00:08:42] (03CR) 10jenkins-bot: [V: 04-1] Collection Renderer (Now a module!) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 (owner: 10Mwalker) [00:08:52] if we lose any single rack we're likely fucked [00:09:23] and labs isn't currently designed with redundancy [00:09:32] ok [00:09:39] I have a feeling that what we're after is possible, but I haven't really looked into it [00:09:56] so I think the simplest solution for now would be to just swap virt1009 with one of the two unallocated boxes in B3 [00:09:57] (03PS7) 10Mwalker: Collection Renderer (Now a module!) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 [00:10:14] and be done with this [00:11:32] virt1000 is labsconsole, right? [00:11:35] yes [00:11:39] and ldap I guess [00:11:58] nova-scheduler, keystone, ldap, dns, mediawiki, mysql [00:12:00] way too many things [00:12:08] right [00:12:18] !log ongoing schema changes on slaves, indexing only, logging gerrit 85508 [00:12:20] yeah, it's slowly coming back :P [00:12:23] heh [00:12:33] Logged the message, Master [00:12:42] hm [00:16:32] (03CR) 10Mattflaschen: [C: 032] Create "Draft" namespace on the English Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 (owner: 10MZMcBride) [00:17:14] (03CR) 10Qgil: [C: 04-1] "The structure looks valid, good! I would say the font is also pretty close. Just curious, which font is it? However, at least the 16x16 ic" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102354 (owner: 10RashiqAhmad) [00:17:27] ori-l: if you have a free moment today; I'd appreciate you taking a look at https://gerrit.wikimedia.org/r/#/c/102352/ and giving me a +/-1 on if that's how I should create a puppet module [00:17:45] (03Merged) 10jenkins-bot: Create "Draft" namespace on the English Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 (owner: 10MZMcBride) [00:19:56] !log mflaschen synchronized wmf-config 'Deploy Draft namespace to English Wikipedia. Remove old Labs overrides that were used for testing this.' [00:20:12] Logged the message, Master [00:20:30] greg-g, done deploying. [00:22:17] springle: have you seen the alert of es1003 having a failed disk? [00:22:24] paravoid: yes [00:22:26] ok [00:22:28] getting there :) [00:22:39] ok, just making sure, sorry :) [00:24:24] paravoid: yeah, it may just be easier to move servers and maybe even designate a rack as labs [00:24:33] it's just one server to move :) [00:24:50] and there's space in the rack, plus WMFnnnn servers (unallocated0 [00:25:10] hehe [00:25:11] ok [00:25:34] know the funny part? i had andrew switch the ip of labnet1001 from row b to row c ;) [00:25:40] haha [00:25:51] oh right, I didn't check labnet [00:26:04] it's row c right now [00:26:06] right [00:26:09] so we have to move that too [00:26:39] that was silly anyway even without eqiad's design [00:26:41] b5 is basically empty [00:26:53] if we don't mind having something with the netapps [00:26:55] having 100% of labs' traffic being cross rack [00:33:16] LeslieCarr, Ryan_Lane, paravoid, reading backscroll... [00:33:32] So is labnet pointless until it's moved to a different rack? [00:34:07] mwalker: revewing [00:34:35] (03CR) 10RashiqAhmad: "Qgil: I used the font "PT Sans Bold", which comes pretty close to the original font." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102354 (owner: 10RashiqAhmad) [00:35:04] yeah... [00:36:02] welp [00:37:14] LeslieCarr: so you opened a ticket about expanding a vlan across racks… is that still the plan or do we instead need Chris to haul the box across the datacenter? [00:37:38] haul across the dc [00:37:43] if it's urgent we can do smarthands [00:37:55] cuz chris is out here all week [00:39:10] ok, so… when I originally requested this box did I need to specify 'should be able to connect to labs'? Does that not go without saying? [00:39:12] * andrewbogott is peeved [00:39:29] (03PS3) 10RashiqAhmad: Updated internal.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102354 [00:40:05] well [00:40:10] i think nobody was thinking about that [00:40:11] since labs is special [00:40:26] honestly i wasn't thinking about it until today [00:41:25] So, just to reconfirm -- the current agreement is that we should have all labs hosts in the same row? [00:41:41] Should I make an RT about that? [00:43:30] LeslieCarr, not mad at you, just… mhoover's contract is going to run out before we even get him hardware :( [00:43:51] oh sad [00:43:51] yes [00:45:25] What about e.g. labsdb and labstore? I feel like they are intentionally distributed right now. Do they need to be consolidated as well? [00:45:54] Coren? [00:46:19] Aye? [00:46:53] Coren: So, keeping in mind that I don't know a single thing about networking... [00:47:08] andrewbogott: to be clear, it's just virt1009 + labnet1001 that need to be moved to B3 [00:47:08] I'm told that labnet isn't useful because it's on a different row and vlan from the labs nodes. [00:47:23] and the former can probably wait :) [00:47:33] but labnet being far away isn't great [00:47:42] paravoid: the db and storage hosts can be elsewhere because… we don't care about latency so much for them? [00:48:24] andrewbogott: That latency wouldn't matter much, because it doesn't live /inside/ the networking stack. [00:48:39] 'k [00:49:04] labnet is a virtual "switch" that needs to be able to exchange frames a lot with the instances; its latency is literally multiplicative. [00:49:36] OK, renamed https://rt.wikimedia.org/Ticket/Display.html?id=6524 [00:50:09] I'll make another ticket for virt1009 [00:50:54] this has nothing to do with latency [00:51:04] it's all the same DC, latency doesn't matter [00:51:21] different row means cross row traffic (i.e. filling uplinks) [00:51:24] paravoid: Well, I'd have thought that was the primary issue. What is, then? [00:51:26] ok, then… why does sharing a row matter? [00:51:33] and for labnet, it also means that it's impossible due to eqiad's network design [00:51:41] since you can't have layer 2 cross-row traffic [00:51:52] (each row is a separate l2 domain) [00:51:54] paravoid: Really? Huh. [00:52:00] layer 2 = ethernet? [00:52:06] yes [00:52:32] And openstack happens at l2, it's not just plain old ip? [00:52:53] andrewbogott: It's an entire virtual network at that level. [00:52:55] labnet is a router, you need to be on the same broadcast domain to route [00:53:07] to be the default gateway, basically [00:53:32] paravoid: Well, it /could/ be worked around but it'd be a mostrosity. (read: happy fun GRE) [00:53:36] Hm, ok. I knew that it /acted/ like a virtual network but assumed that it was… virtual :) [00:54:16] Anyway, ok, tickets created. I guess there's no point in my nagging since we're waiting for Chris to return to Virginia? [00:54:24] yes, there are multiple workarounds, but we're talking about moving a single box two rows over :) [00:54:37] andrewbogott: if it's urgent, you could use smart hands [00:54:40] DC staff [00:55:04] and if you don't trust them to move a server, you could just ask them to lay a cable cross racks/rows for now [00:55:29] although I'd prefer you to not do that actually [00:55:36] on second thought :) [00:55:38] I don't know if it's urgent, will defer to ryan and mike. [00:55:42] they should be able to move a server [00:55:48] I'm not worried about trusting them with the server, it's empty now anyway [00:56:00] yeah, let's not have them potentially plug into a different port and bring the DC down [00:56:52] dunno, I've never seen eqiad smart hands and how smart they are :) [01:00:03] they're all right [01:03:21] (03PS1) 10Krinkle: role/gerrit.pp: Clean up extra line breaks [operations/puppet] - 10https://gerrit.wikimedia.org/r/102362 [01:03:59] (03PS1) 10Krinkle: role/gerrit.pp: Mirror VisualEditor as wikimedia/VisualEditor [operations/puppet] - 10https://gerrit.wikimedia.org/r/102363 [01:04:18] (03PS2) 10Krinkle: role/gerrit.pp: Mirror VisualEditor as wikimedia/VisualEditor [operations/puppet] - 10https://gerrit.wikimedia.org/r/102363 [01:06:05] ok, I've made some tickets and sent some emails. [01:06:24] If we decide we need the smarthands, how do I go about getting them to do stuff? Is that something Rob mediates? [01:07:07] who can get me admin wiki privileges on http://en.m.wikipedia.beta.wmflabs.org/ ? anyone? [01:07:13] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Mon 16 Dec 2013 06:59:39 PM UTC [01:07:19] i want to edit MediaWiki:Common.js [01:07:32] most anyone with ops can --- let me check the access list [01:08:18] (03PS1) 10Dzahn: temp disable bugzilla auditlog/metrics mails [operations/puppet] - 10https://gerrit.wikimedia.org/r/102365 [01:08:21] jdlrobson: on the wiki or on labs [01:08:30] just the wiki mutante [01:08:42] admin of wikipedia.beta, not admin of wikitech wiki [01:08:52] then i guess hashar is good to ping [01:08:55] correct mutante :) [01:09:02] or reedy maybe? [01:09:05] andrewbogott: ariel or faidon can [01:09:11] i can give daniel privileges as well [01:09:13] ok [01:09:24] I will wait for feedback from mhoover, then bug one of them if necessary. [01:09:55] This whole thing is kind of weird, I'm scrambling to get hardware ready for mike and ryan but am totally out of the loop in terms of their timeline. [01:10:16] yeah [01:10:21] oh ryan lane also has privileges [01:10:53] createAndPromote.php [01:11:05] (03CR) 10Dzahn: [C: 032] "paravoid, this disabled the annoying mails" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102365 (owner: 10Dzahn) [01:13:12] jdlrobson, done - why didn't you just ask me?:) [01:13:22] MaxSem: i didn't know - but thanks :) [01:13:28] Reedy: ah, i just remembered that from one single time.. wikivoyage wiki and making the first user ever:) [01:13:44] ah, it's resolved, thx Max [01:13:49] I seemingly can't login... [01:13:55] Presumably need a password from my vault [01:15:24] andrewbogott: hey andrew, thanks. im checking out your email [01:16:18] andrewbogott: as far as timeline, i'm not sure what the deadlines are. before xmas for the migration? or at least the network up? [01:16:50] mhoover: well, ideally the migration would be done before your contract runs out [01:17:00] (03CR) 10Reedy: "Caused bug 58612" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 (owner: 10MZMcBride) [01:17:28] andrewbogott: as far as the networking is concerned, i haven't mapped out or looked at eqiad in detail yet, so i can't be sure that moving this machine is a problem or not. i would think, if it's all on the same flat network, should be cool [01:18:12] mhoover: It has to be moved. [01:18:26] Currently it can't be used. [01:18:37] The question is when it being unusable will become a blocker for you. [01:18:41] andrewbogott: i would say leave the server move for the chris guy - if it needs to be moved before monday, we can go the smart hands route [01:18:55] (03CR) 10Ori.livneh: [C: 04-1] "Pretty good." (0320 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 (owner: 10Mwalker) [01:19:04] OK. I'll drop it for now. [01:19:10] mwalker: ^ [01:19:25] andrewbogott: i don't think it would be a blocker until next week [01:19:34] 'k [01:19:42] andrewbogott: we're still figure out the openstack-ish details [01:20:25] (03CR) 10Dzahn: [C: 032] Fetch only "Wikipedia" label from Netha Hussain's blog to Planet [operations/puppet] - 10https://gerrit.wikimedia.org/r/102210 (owner: 10Nemo bis) [01:20:44] (03PS2) 10Dzahn: Fetch only "Wikipedia" label from Netha Hussain's blog to Planet [operations/puppet] - 10https://gerrit.wikimedia.org/r/102210 (owner: 10Nemo bis) [01:20:44] ok -- please let me know if there's anything I can do to help. [01:21:47] for now I'm out. [01:21:57] (03CR) 10Dzahn: [C: 04-1] "I'm totally copying Ori on this review: "unrated changes are easily overlooked so i'm giving it a -1, even though it's more of a -0.5"" [operations/puppet] - 10https://gerrit.wikimedia.org/r/100760 (owner: 10Matanya) [01:22:22] heh [01:22:31] :) [01:22:37] it fit perfectly [01:22:38] andrewbogott: thank you for the help/heads-up andrew/leslie/paravoid :) [01:22:45] http://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&c=Mobile+caches+ulsfo&m=cpu_report&s=by+name&mc=2&g=mem_report hm ? [01:23:19] there's a spike in bytes_in http://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Mobile+caches+eqiad&m=cpu_report&s=by+name&mc=2&g=network_report [01:23:29] for mobile caches eqiad [01:24:13] PROBLEM - Puppet freshness on manutius is CRITICAL: Last successful Puppet run was Tue 17 Dec 2013 01:21:21 PM UTC [01:24:39] (03PS1) 10Reedy: Simplify Drafts related TitleQuickPermissions hook subscriber [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102366 [01:24:41] varnishkafka? [01:24:55] Looking at the Drafts problem [01:25:13] paravoid: that was my thought when i saw the memory spike, but the change in network traffic is in bytes *in* [01:25:35] ignore the first graph, ulsfo is not in production, this is just slowly rampup of page cache following an OOM coming from a netmapper bug [01:25:55] (03CR) 10Mattflaschen: [C: 04-2] "Not equivalent." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102366 (owner: 10Reedy) [01:25:57] superm401: Stupid typo I bet [01:26:03] ulsfo network is measured in kilobytes per second, if you look more closely [01:26:24] (03CR) 10Reedy: "Needlessly verbose?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102366 (owner: 10Reedy) [01:26:37] mem on eqiad mobile caches also kinda funky, though: http://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&c=Mobile+caches+eqiad&m=cpu_report&s=by+name&mc=2&g=mem_report [01:26:40] and the second graph, eqiad, hmm, let me see [01:27:11] that funkyness is ottomata messing up [01:27:25] cp3011.esams & cp4011.ulsfo appear in that group [01:27:31] as separate entries from the proper ones [01:27:41] http://ganglia.wikimedia.org/latest/?c=Mobile%20caches%20eqiad&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2 [01:27:46] probably manual gmetric invocation [01:28:12] so if you go to search and type cp3011 you'll see two entries [01:28:41] this would explain the network spike as well [01:28:55] heh [01:29:02] Reedy, is it possible that wmg variables can not be arrays? [01:29:17] hm, maybe [01:29:32] * ori-l has to run [01:29:32] I'm pretty sure I tested the exact same thing, just with a different name, on Labs. [01:29:32] ok [01:29:40] will investigate, although not too worried [01:29:43] thanks for the heads up :) [01:30:26] superm401: Nope... We do that, a lot [01:30:26] '+arwikiversity' => array( 0 => 1, 6 => 1, 10 => 0 ), [01:31:04] Yeah, there's even one right above. [01:33:53] It's not an obvious typo (it's the exact same string in both places) [01:34:07] noc.wikimedia.org looks right, so it doesn't seem like only one file got deployed. [01:36:24] (03PS1) 10Manybubbles: Cirrus config update [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102369 [01:41:26] (03PS2) 10Reedy: Simplify Drafts related TitleQuickPermissions hook subscriber [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102366 [01:41:32] (03CR) 10Reedy: Simplify Drafts related TitleQuickPermissions hook subscriber (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102366 (owner: 10Reedy) [01:42:33] superm401: var_dump( $wmgExemptFromUserRobotsControlExtra ); is right for enwiki and also for !enwiki [01:42:52] let's dig a little deeper [01:43:10] It's running way after the extract. [01:45:52] I wonder what it is if not an array [01:46:00] (03CR) 10Mattflaschen: [C: 04-1] "Equivalent now, but update the comment." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102366 (owner: 10Reedy) [01:46:49] !log reedy synchronized wmf-config/ 'Ensuring consistency...' [01:47:06] Logged the message, Master [01:49:38] Reedy, $wgExemptFromUserRobotsControl is right too, according to mwscript eval.php on enwiki. [01:50:04] And a random other wiki (ruwiki). [01:50:40] The other thing could be caching [01:50:42] ruwiki is just 0, of course. [01:51:41] * Reedy kicks things [01:52:02] !log reedy synchronized wmf-config/ 'touch' [01:52:16] Logged the message, Master [01:52:36] * Reedy waits [01:52:37] Reedy, it's not on every request, right? [01:52:44] If it were, it would be a lot more, I think. [01:52:48] Nope [01:53:34] tail has fallen silent [01:54:21] I do wonder if when we're syncing wmf-config and/or InitialiseSettings.php explicitly we should touch it [01:54:38] None since Dec 18 01:51:27 [01:54:55] I actually changed all four files, though. [01:55:18] Did you happen to check the mtime before you did the touches? [01:55:22] if ( @filemtime( $filename ) >= filemtime( "$wmfConfigDir/InitialiseSettings.php" ) ) { [01:55:43] -rw-rw-r-- 1 mflaschen wikidev 431778 Dec 18 00:18 InitialiseSettings.php [01:55:53] -rw-rw-r-- 1 mflaschen wikidev 431778 Dec 18 01:51 InitialiseSettings.php [01:55:57] That's UTC, right? [01:56:11] Yeah [01:56:15] The beginning of the lightning window was 00:00 on the dot. [01:56:18] I think we're on UTC [01:56:21] So that mtime should be fine. [01:56:27] Yup [01:56:28] Wed Dec 18 01:56:22 UTC 2013 [01:56:44] It fixed it though (coincidence?) [01:57:03] I've seen this happen a few times [01:57:29] Usually pulling in "older" changes [01:57:48] date +'%Z' confirms UTC [01:57:59] Blech, I hate when the problem stops but I have no idea why. [01:58:33] Thanks for helping my troubleshoot, and for fixing it if it was the touch. [01:59:42] I hate computers [02:02:12] Reedy, could it just be that some of the Apaches have really off time? [02:02:16] Are we monitoring that? [02:02:41] It could be the config caching then (BTW, didn't know about that). [02:05:34] Looking at the apache log entries and running date... [02:05:38] At most it's seconds [02:06:46] Yeah, never mind, the log entries do look about right. [02:08:48] Might almost be worth making sync-file touch by default [02:12:03] i actually did a sync-dir. [02:12:30] But yeah, it should DTRT [02:13:38] ori-l: Do I recall you saying something about js issues not needing files touched now to recache? [02:18:18] I think Krinkle's d3bdda3 change should have fixed that, so e.g. if you remove the oldest file in a module, it Just Works [02:25:17] !log LocalisationUpdate completed (1.23wmf7) at Wed Dec 18 02:25:17 UTC 2013 [02:25:35] Logged the message, Master [02:29:55] Reedy: ori-l: superm401: indeed. [02:41:17] (03Abandoned) 10Dzahn: add virtual language subdomain redirects for wikidata [operations/apache-config] - 10https://gerrit.wikimedia.org/r/65443 (owner: 10Dzahn) [02:42:19] longest abandon message ever [02:43:24] zero gerrit sounds like a pretty good goal [02:47:41] !log LocalisationUpdate completed (1.23wmf6) at Wed Dec 18 02:47:41 UTC 2013 [02:47:53] legoktm: tell me when you reach it >.> <.< [02:47:58] Logged the message, Master [02:48:07] p858snake|l: start revieiwng my patches plz [02:48:23] -1 for all \o. [03:09:14] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Dec 18 03:09:14 UTC 2013 [03:09:30] Logged the message, Master [04:03:29] (03PS1) 10Aquifacae: changed wiktionary/en.ico favicon to proper image from wikimedia commons and added 16x16, 32x32, and 48x48 resolutions to the file. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102382 [04:08:13] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Mon 16 Dec 2013 06:59:39 PM UTC [04:25:13] PROBLEM - Puppet freshness on manutius is CRITICAL: Last successful Puppet run was Tue 17 Dec 2013 01:21:21 PM UTC [04:35:58] Reedy: why -- are you finding that you do need to touch files? [05:05:42] (03PS1) 10Tim Landscheidt: Tools: Unify Tools and Toolsbeta configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/102385 [05:20:05] (03PS1) 10Aquifacae: Changed wiktionary/en.ico favicon to proper image and resolutions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102386 [05:20:59] (03Abandoned) 10Aquifacae: changed wiktionary/en.ico favicon to proper image from wikimedia commons and added 16x16, 32x32, and 48x48 resolutions to the file. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102382 (owner: 10Aquifacae) [05:21:34] (03PS2) 10Ori.livneh: Add scholarships.wikimedia.org [operations/dns] - 10https://gerrit.wikimedia.org/r/98849 (owner: 10BryanDavis) [05:23:12] (03CR) 10Ori.livneh: [C: 032] Add scholarships.wikimedia.org [operations/dns] - 10https://gerrit.wikimedia.org/r/98849 (owner: 10BryanDavis) [05:32:39] !log DNS update: scholarships.wikimedia.org CNAME misc-web-lb.eqiad [05:32:57] Logged the message, Master [06:04:08] (03PS1) 10BryanDavis: Configure twig cache directory for Scholarships [operations/puppet] - 10https://gerrit.wikimedia.org/r/102393 [06:05:38] (03CR) 10Ori.livneh: [C: 032] Configure twig cache directory for Scholarships [operations/puppet] - 10https://gerrit.wikimedia.org/r/102393 (owner: 10BryanDavis) [06:07:40] please don't deploy such changes without ops present (I'm hardly present) [06:07:49] it looks safe, but you never know [06:08:00] which? [06:08:05] the cache, or DNS? [06:08:05] dns for starters :) [06:09:03] well, okay. [06:09:47] at least do it on business hours when people are present [06:10:57] !log Reloading zuul to deploy Ib05ad8d180d5239ac [06:11:24] okay, my bad. noted. i thought it was trivial. [06:13:43] it is, true [06:16:33] (03PS2) 10Aquifacae: Changed wiktionary/en.ico favicon to proper image and resolutions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102386 [06:19:50] anything I can help with now? [06:20:42] bd808: ^ ? [06:20:55] i think we're good, but maybe bryan needs something [06:21:02] are you both working at this hour?! [06:21:10] I'd like a hot cup of cocoa :) [06:21:19] OTher than that I think I'm golden [06:21:20] I'm used to ori-l by now [06:21:21] paravoid: you're one to talk :P [06:21:47] :P [06:22:05] I have a deploy tomorrow and was still getting new requirements this afternoon [06:22:29] xmas messed up the schedule a bit [06:23:06] paravoid: I'd like to talk to you about logstash next week if you have some time [06:23:25] yup, I can do that [06:23:37] I'd love to [06:23:55] Mostly I'd like your thoughts about syslog and other data you would like to see in there [06:24:25] I leave on the 26th [06:24:36] so early next week I suppose [06:24:53] Sure. Whatever time you can find [06:34:02] the trivial ones are usually the surprising ones /me isnt here [06:34:13] mutante: availability bias :P [06:34:23] the surprising ones are the ones you remember [06:35:53] /me changes nick to schrodinger's mutante [06:48:23] (03PS1) 10Ori.livneh: Re-disable Scholarship app until scheduled deployment [operations/puppet] - 10https://gerrit.wikimedia.org/r/102397 [06:48:39] oh? [06:48:45] I hope that's not because of my comments [06:49:23] no, I pushed for us to test it in prod so that there are no nasty surprises tomorrow [06:49:29] ('today' for you) [06:49:50] it looks good, though [06:50:30] (03PS2) 10Ori.livneh: Re-disable Scholarship app until scheduled deployment [operations/puppet] - 10https://gerrit.wikimedia.org/r/102397 [06:51:55] (03CR) 10Ori.livneh: [C: 032] Re-disable Scholarship app until scheduled deployment [operations/puppet] - 10https://gerrit.wikimedia.org/r/102397 (owner: 10Ori.livneh) [06:55:39] paravoid: do you mind if i restart apache on zirconium so that it picks up the vhost drop? [06:56:19] it's etherpad, soon it will be bugzilla, now it's not the one in use [06:56:20] reload should suffice, but yes, go ahead [06:56:44] ok, reloaded [06:57:09] doesn't seem !log-worthy [06:57:44] and it's that civicrm stuff, meh [07:06:24] how do you debug the memory usage of a running process? [07:06:39] pmap + gdb? [07:09:13] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Mon 16 Dec 2013 06:59:39 PM UTC [07:26:13] PROBLEM - Puppet freshness on manutius is CRITICAL: Last successful Puppet run was Tue 17 Dec 2013 01:21:21 PM UTC [07:27:55] (03CR) 10Faidon Liambotis: "Didn't we explicitly agree to *not* do this for now?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102316 (owner: 10Yurik) [07:29:08] (03CR) 10Faidon Liambotis: [C: 032] "Thanks, the inconsistency was bugging me too." [operations/puppet] - 10https://gerrit.wikimedia.org/r/102333 (owner: 10Yurik) [07:29:29] paravoid, we had a meeting with dfoy yesterday, apparently HTTPS is a big blocker on his plate - hence wanted to get it in, as it won't fragment cache or add much to processing time [07:29:44] blocker for what? [07:30:06] it's not fragmenting the cache, but the config is already horrible to read & understand [07:30:36] for the partners that have already whitelisted us using ip -- mobile often forces HTTPS, which hides banners [07:30:50] we already have two of them (listed in that config [07:30:50] often? [07:31:15] "often" shouldn't happen [07:31:16] i don't have data - apparently we have forceHTTPS, which switches users [07:31:33] plus i think mobile really wants to push everyone to https by default [07:31:59] how did we get from "a big blocker" to "mobile really wants to push this!" ? [07:32:02] and this will provide a mitigating path - as we can set some "allow https" magic header [07:32:08] hehe :) [07:32:18] sorry, misspoke - dan asked for it [07:32:22] blocker is the wrong word [07:32:28] I'm all for https btw [07:32:40] but let's work on one piece at a time [07:32:46] and work on simplifying our config first [07:33:03] sure - but the only real simplification is ESI, right? [07:33:06]