[00:00:09] let's see, i'm gonna make some edit [00:00:59] Well, enwiki edits are pretty high frequency, right? :P [00:01:03] Can't join #en.wikipedia [00:01:33] greg-g, tanks for the heads up. [00:01:52] 208.80.152.178 [00:02:02] If it moved, surely its ip changed? [00:02:04] greg-g: Can I get an emergency SWAT addition for a JS error that I accidentally caused during SWAT? [00:02:13] Maybe? [00:02:18] DNS seems to be accurate [00:02:21] 178.152.80.208.in-addr.arpa name = ekrem.wikimedia.org. [00:02:31] ekrem.wikimedia.org has address 208.80.152.178 [00:02:53] Does the bot need restarting? [00:02:56] According to monitor 'IRC RecentChanges', the connect service on [00:02:58] 'irc.wikimedia.org' has been working again as specified since 2014-04-22 [00:02:59] 00:01:34. [00:03:02] RoanKattouw: nope, the channel hasn't been created although i made action on wiki [00:03:19] (03PS4) 10BBlack: Set bnx2x num_queues to match physical cores on lvs nodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/127827 [00:03:21] /usr/local/bin/start-ircbot [00:03:24] Down since 2:57pm PDT, that's when the Nimsoft alert first came up [00:03:32] Reedy: Yeah except that's already running. I'll restasrt [00:03:36] Wikitech suggests it might not be started automagically [00:03:39] Ah [00:03:44] If IRC was down in the first place [00:03:47] Chances are it needs rebooting [00:04:05] ouch [00:04:26] Reedy: No I think I got it, hold on [00:05:03] How about now? [00:05:11] I guess I should fire up a second IRC client or something [00:05:21] Yup, fixed [00:05:28] Awesome! [00:05:36] might wanna !log it [00:05:52] !log Restarted ircecho on ekrem, IRC working again now [00:05:55] Or get an IRC client that does multi network ;) [00:05:58] Logged the message, Mr. Obvious [00:06:25] just posted a wmfsf email asking if disabling ipv6 for the office for an extended length of time would be a burden for anyone -- I figured it impacts Ops/Dev more than most.. Wanted to ask in here too... [00:06:29] Concerns? [00:08:35] nice, works now [00:08:39] thanks, RoanKattouw [00:09:25] what is actually the reason for not to have the channels persistent? [00:09:32] I don't know [00:09:41] I have no idea how any of this IRC stuff is set up [00:09:52] after the restart bots can't reconnect to the channel because it doesn't exist [00:10:01] Danny_B: that shouldn't be the reason [00:10:05] It's just that https://wikitech.wikimedia.org/wiki/IRC happens to be perfectly accurate re how to start the service [00:10:09] a normal IRCd will create a channel if it doesn't exist [00:10:11] So it was very easy to bring it back up [00:10:24] although, setting mode +P would kinda fulfill what you want [00:10:27] (opers only) [00:10:28] Jasper_Deng: wmf irc is custom patched ircd [00:10:41] I know it's some charybdis-derived thing [00:10:43] I'm not sure it matters that much [00:10:46] It's rarely restarted [00:10:47] ^ [00:10:49] the channels are created only by irc bot [00:10:57] And rc-pmtpa is 99.999% there [00:11:02] no other user can create it [00:11:36] RoanKattouw: thanks for the SWAT! [00:12:19] greg-g: ping... ? [00:15:35] !log torrus (on manutius) is down [00:15:41] Logged the message, Master [00:16:10] just a heads up the ircd-ratbox not being in puppet has an rt ticket that has made slow progress [00:16:10] https://rt.wikimedia.org/Ticket/Display.html?id=4784 [00:16:30] so it's a weird manual thing but at least ticketed, thanks for doing it btw [00:17:01] We really need a better way of doing RC feed publishing and kill it completely [00:18:54] !log catrope synchronized php-1.24wmf1/extensions/MobileFrontend/javascripts/modules/editor/VisualEditorOverlay.js 'Fix JS error on save' [00:18:55] I think that's the idea all around, but at least packaging up properly what we have for the time being [00:19:01] Logged the message, Master [00:19:04] heh [00:19:04] greg-g: I went ahead and did it [00:19:09] chasemp: Did you sort your gerrit issues? [00:20:11] Reedy: if you mean for old ircd-ratbox repo -- I believe I did thanks, I haven't gotten a moment to try to sort it out properly yet tho [00:20:23] readding the 'right thing' I mean [00:25:44] Reedy: last time i heard there is xmpp rc feeding completely ready, but the only thing is to set up the xmpp server [00:26:59] I thought Daniel Kinzler gave up on that like 6 years ago? [00:27:09] https://github.com/wikimedia/mediawiki-core/tree/master/includes/rcfeed [00:27:52] good topic for hackathon ;-) [00:29:49] Bikeshed on what we could possibly use [00:29:50] ? [00:30:12] RoanKattouw: hey, thanks, sorry, was eating dinner [02:13:19] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3392 MB (3% inode=99%): [02:20:19] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3676 MB (3% inode=99%): [02:29:36] !log LocalisationUpdate completed (1.23wmf22) at 2014-04-22 02:29:34+00:00 [02:30:20] Logged the message, Master [02:43:03] !log LocalisationUpdate completed (1.24wmf1) at 2014-04-22 02:43:01+00:00 [02:43:10] Logged the message, Master [03:00:19] RECOVERY - Disk space on virt0 is OK: DISK OK [03:32:34] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 22 03:32:29 UTC 2014 (duration 32m 28s) [03:32:41] Logged the message, Master [03:50:01] oops [03:50:04] Dang [03:50:11] Lost the scrollback [03:50:19] this channel is logged [03:50:22] see the topic [03:50:26] hax [03:50:33] I did see it, Jasper_Deng [03:50:37] ... [03:59:45] werdna: I emailed you a few days about (maybe) changing your production login… any thoughts? [04:35:21] !log reactivate esams<->HE & eqiad<->HE peerings; issues are confirmed to be resolved [04:35:27] Logged the message, Master [04:39:48] paravoid: i imagine this involves grasping some orb that is filled with swirling plasma in both hands as bolts of lightning crack against the sky [04:40:03] lol what? [04:40:28] i don't know anything about peering except that it's important, so my imagination runs off [04:40:50] what would you like to know? [04:40:54] happy to explain :) [04:41:05] well, first of all, how you get the lightnings to appear on command like that [04:41:25] just kidding. i don't even know what to ask. i just thought what you said sounded cool :P [04:41:33] lol [04:42:29] so how does it actually work? [04:42:33] * ori googles it [04:43:01] there is a protocol called BGP [04:43:15] routers talk to each other using that protocol [04:43:59] can i pick it up on my computer? [04:44:10] you probably can't [04:44:12] networks are split into "autonomous systems", e.g. Wikimedia has two, one for Europe and another one for the US (although these days the lines have blurred a lot and you could consider it one) [04:44:15] BGP's at the core routers level ^ [04:44:31] we have two instead of one because trans-Atlantic transit is too expensive [04:44:34] each autonomous system has a number assigned to it (Wikimedia's US network is 14907, Europe is 43821) [04:44:55] (no it's not, but anyway) [04:45:54] Wikimedia also has some routes, e.g. 208.80.152.0/22 & 91.198.174.0/24 [04:46:04] #ko.wikinews @ irc.wikimedia.org says "no such channel" [04:46:15] and using BGP, we announce these routes as originated from our AS [04:46:26] Revi: known problem, scroll up [04:46:32] Ok [04:46:50] so we tell our peers (other AS), via BGP, that 208.80.152.0/22 is originated from AS 14907 [04:47:00] to give a perspective of how important BGP is, the Egyption government shutoff Internet access before the Arab spring for the entire country by turning off BGP announcements from Egypt's ISPs [04:47:31] some of them are just using this information for their internal networks (these we usually call "private peers", "IXP peers" or just "peers") [04:47:44] (this is really great by the way; i am rapt) [04:47:52] others provide us with connectivity to reach third-party networks [04:47:59] these are called "transits" [04:48:06] we usually pay for transit [04:48:28] to reach to be reached [04:48:29] yes, i remember this bit because you explained it to me when you were last in sf [04:48:42] *to reach to third-party networks and to be reached by third-party networks [04:49:35] www.google.com has address 74.125.228.83 [04:49:42] A Destination P Prf Metric 1 Metric 2 Next hop AS path [04:49:45] * 74.125.228.0/24 B 170 250 0 >208.80.154.194 15169 ? B 170 100 100 >209.48.42.49 2828 15169 ? B 170 100 0 >129.250.204.189 2914 15169 ? B 170 100 >130.244.6.242 1257 15169 ? [04:49:50] argh, whitespace breakage [04:49:53] A Destination P Prf Metric 1 Metric 2 Next hop AS path [04:49:56] * 74.125.228.0/24 B 170 250 0 >208.80.154.194 15169 ? [04:49:59] B 170 100 100 >209.48.42.49 2828 15169 ? [04:50:02] B 170 100 0 >129.250.204.189 2914 15169 ? [04:50:05] B 170 100 >130.244.6.242 1257 15169 ? [04:50:21] 15169 is Google's ASN (autonomous system number) [04:50:35] 2828, 2914 & 1257 are transits of ours [04:50:57] just like IPv4 addresses have been exhausted, 2-byte ASN's have been exhausted, so all new ASN's are 4-byte (32-bit) [04:51:12] we peer directly with google, and those three transits also peer directly with google [04:51:29] A network that does not have to rely on /any/ transit is a /tier 1 network/ [04:51:55] (well, that is, no transit to reach other tier 1 nets) [04:54:47] what command is that the output of? [04:54:56] show route 74.125.228.83 terse [04:55:00] from the juniper [04:55:05] one of them anyway [04:57:17] 'some of them are just using this information for their internal networks' [04:57:26] why do we peer with them? is it reciprocal somehow? [04:57:50] well, kind of [04:58:15] first of all, yes, it is usually reciprocal [04:58:43] i.e. when we peer, we usually reach them directly and they usually reach us directly as well [04:59:00] but the answer to your "why do we peer with them" question is [04:59:23] if we don't peer directly with those networks, we'd use transit to reach them (and they'd use transit to reach us) [04:59:40] this is (usually) more expensive for both parties [05:00:01] in bits flying over optics terms, what's the difference? [05:00:14] plus, peering directly means that you don't get random networks in between [05:00:15] ie: there's only so many cables going out of the DC [05:00:57] which can reduce latency and help with troubleshooting if there is an issue [05:01:19] greg-g: there are A LOT of cables going out of the DCs we're in [05:01:28] in the order of thousands probably [05:01:32] huh [05:01:34] well then [05:01:44] plus, you can multiplex traffic in many layers on top of a fiber [05:02:08] some (like optical multiplexing) gurantee you bandwidth, others are best effort/can get congested [05:02:13] given the advantages of peering, why wouldn't internet2.edu refuse to peer with us? they share equinix w/ us [05:02:25] why would* [05:02:56] peering is a complicated game [05:03:29] stupid scarce resource politics [05:03:43] internet2 is AS 11537; http://as11537.peeringdb.com/ indicates that they have a "selective" peering policy and that they are not adding more peers [05:04:19] why not? what's the practical cost? [05:04:54] one aspect is the maintenance overhead; dealing with hundreds of peers needs more human resources than dealing with e.g. three [05:04:56] having to do what you did earlier -- i.e., respond to failures in their network [05:04:57] ah [05:05:00] in terms of emails exchanged, monitoring sessions etc. [05:05:05] right [05:05:12] a second cost is that you lower the leverage you have with your transits [05:05:34] since you lower the bandwidth you push to them and hence you're essentially a smaller customer [05:05:44] weird [05:05:51] the third, more important aspect for the overall game is [05:06:05] often networks ask you to *pay* them to peer with you [05:06:31] remember, if two networks don't peer directly, they both pay their respective transits to reach each other [05:07:25] if one is substantially larger than the other, then it pays *less* to their transit per mbps (gets a better discount) and it matters less for their overall budget of their network [05:08:01] (When you're done with this current thread, I'm still confused on the physical layer: are there other ASNs that we are physically unable to peer with? To peer with someone must you share an exchange 1 hop away or something like that?) [05:08:02] then that network can basically ask the other network to pay them to reach to them, e.g. half of what they pay their transit [05:08:25] sometimes you can call this extortion [05:08:47] * ori is still working out the math [05:09:45] right, i can see how it is somehow unsavory [05:09:51] imagine you have a small isp called OriSP, that buys transit out of Level3 (3356) [05:09:53] greg-g: it's easiest when you share the exchange [05:10:02] (as w/ internet2.edu, hence why I asked about them) [05:10:07] since you're small, you probably don't get a good deal and you pay, say $10 per mbps [05:10:10] otherwise you need to make it so or get transit [05:10:44] wikimedia buys transit out of NTT (2914), but is larger so it pays, say, $5 per mbps (no, that's not a real figure) [05:11:24] 2914 & 3356 have a deal and peer with each other (undisclosed terms, but they're both tier-1s, so presumably for free) [05:11:44] so the network path is wikimedia <-> Level3 <-> NTT <-> OriSP [05:11:46] er [05:11:54] wikimedia <-> NTT <-> Level3 <-> OriSP [05:11:58] I meant [05:12:41] (03PS1) 10Springle: Switch s7-analytics-slave to db1007 (eqiad lvm/dump slave). RT #7330. Though apparently already on 12th floor, db68 has fallen offline during pmtpa 10th/12th floor changes for some reason. [operations/dns] - 10https://gerrit.wikimedia.org/r/127869 [05:12:44] it's surprising to me that there is any aspect of this that is 'naively' reciprocal -- i.e. where two parties agree to provide the same service for one another without considering their respective sizes [05:12:59] say OriSP's customers are very fond of wikipedia and hence the wikipedia to/from OriSP traffic is 100mbps [05:13:23] (03CR) 10Springle: [C: 032] Switch s7-analytics-slave to db1007 (eqiad lvm/dump slave). RT #7330. Though apparently already on 12th floor, db68 has fallen offline durin [operations/dns] - 10https://gerrit.wikimedia.org/r/127869 (owner: 10Springle) [05:13:24] i would have figured that by now you'd always be paid or be paying, depending on the exact proportion in which you are larger or smaller [05:13:27] ori: AT&T and some networks do what's called "settlement-free" peering [05:13:33] this means that your isp pays $1000 per month to Level3 to reach wikipedia, which may be a substantial figure in your budget [05:13:34] but let paravoid continue [05:14:07] the wmf pays $500 per month to reach orisp, which is a) lower, b) petty cash considering our overall figures [05:14:25] !log db68 down. s1-analytics-slave cname to db1007 [05:14:31] Logged the message, Master [05:14:34] the evil wmf can say to orisp "gimme $500 per month and I'll peer with you directly" [05:15:08] that's a net win of $500 for orisp, and +$1000 to wmf's bank account per month [05:15:32] now imagine that at scale [05:16:10] (insanity) [05:16:25] and add to that the fact that some of these networks you might want to ask to peer with are also transit providers (vendors) that have an incentive for small players to not peer at all :) [05:17:45] i.e. in the above example, imagine if the aforementioned small OriSP went to NTT instead of the WMF and said "hello, we have 100mbps per month with each other, maybe we should peer!" [05:17:49] (good luck with that) [05:18:57] is it still possible to graduate from a dinky isp to a being a major player? i guess if you are attracting a lot of traffic [05:19:31] it's even more complicated than that :) [05:20:04] attracting a lot of traffic by itself isn't always good for your peering partnerships [05:20:24] in the sense that if your traffic isn't balanced (in/out), you're still paying [05:20:28] think Netflix [05:20:52] the whole Netflix/Comcast/AT&T/Level3 deal is a peering issue [05:21:03] http://blog.level3.com/global-connectivity/chicken-game-played-child-isps-internet/ [05:21:06] http://www.attpublicpolicy.com/consumers-2/who-should-pay-for-netflix/ [05:21:12] http://blog.netflix.com/2014/03/internet-tolls-and-case-for-strong-net.html [05:21:30] greg-g: oh yeah, in terms of physical layers [05:21:53] we either do a private peering, i.e. we ask the DC provider to physically connect a fiber between our equipment and another network's equipment [05:22:21] which obviously has some costs, both to the DC provider and for the cost of the port on our respective equipment [05:22:47] so we do this only for very large peers of ours/important peerings [05:22:51] (and transits) [05:23:12] for all the rest, we connect with one or two ports to the local internet exchange point (IXP) [05:23:33] which is essentially a large, shared medium (a switch) where lots of companies connect to [05:23:36] then peer with each other [05:23:42] gotcha [05:24:06] http://as14907.peeringdb.com/ [05:24:08] top-right corner [05:24:43] ams-ix is one of the largest in the world (if not the largest), see https://www.ams-ix.net/connected_parties [05:24:51] or well, https://www.ams-ix.net/ [05:25:27] wow [05:25:55] alright, I have to craah, I just didn't want to miss this [05:26:08] crash [05:26:10] :) [05:26:13] * ori waves [05:26:19] the European IXPs are usually non-profits [05:26:19] thanks and g'night! [05:26:20] well, i was not wrong about this being full of high drama [05:26:20] associations [05:26:34] in the US, it's 1-2 companies dominating the market [05:26:40] (like Equinix) [05:26:54] the market being a metropolitan area? [05:27:11] the market being ALL metropolitan areas :) [05:27:20] so equinix dominates the whole uS [05:27:45] has the largest IXP in Ashburn and New York, as well as the San Jose and Palo Alto ones [05:28:00] and all the largest in between (Dallas, Chicago etc.) [05:28:53] so, last year, the three largest non-profit member-run european IXPs (and also the three largest IXPs in the world) decided to invade the US :) [05:30:03] AMS-IX (Amsterdam), DE-CIX (Frankfurt) and LINX (London) decided to fork to AMS-IX New York, DE-CIX New York and LINX NoVa (North Virgina/Washington DC) [05:32:10] basically so they don't have to pay equinix to reach american internet traffic? [05:32:29] they are just operating an "island" [05:32:31] just a switch [05:33:13] and asking networks (whether it's content providers such as wikimedia and google, or ISPs) to connect via them and exchange traffic with each other [05:33:15] right, but attractive to peering partners [05:33:24] instead of/in addition to connecting to Equinix [05:34:12] do the transits actually punish customers for "disloyalty" like that by reducing the quality of service or increasing prices? [05:34:33] well [05:34:37] it's not about disloyalty [05:34:49] it's about how much you pay [05:34:58] it's different to approach a vendor for 100gbps and different to approach them for 10gbps [05:35:16] typically, the price per mbps decreases the larger your order becomes [05:35:19] economies of scale [05:35:43] if you're aggressively peering, you lower your transit bandwidth needs and your price per mbps for transit increases, yes [05:38:22] but i take back what i said earlier, i think i can see now how mutualism / reciprocity can be sophisticated [05:38:42] it's quite complicated [05:39:20] fwiw, wmf's policy is to never charge for peering (a so-called "open" peering policy) [05:39:31] and we generally don't pay for private peerings either [05:39:55] (but we pay our transits) [06:44:29] !log db77 - revoke puppet cert,salt key,rm from monitoring [06:44:36] Logged the message, Master [06:56:40] (03PS1) 10Chad: Actually configure $wmgBetaFeaturesWhitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/127873 [06:57:49] (03CR) 10Chad: [C: 032] Actually configure $wmgBetaFeaturesWhitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/127873 (owner: 10Chad) [06:58:26] !log demon synchronized wmf-config/InitialiseSettings.php 'Unbreak $wmgBetaFeaturesWhitelist' [06:58:32] Logged the message, Master [06:58:51] (03Merged) 10jenkins-bot: Actually configure $wmgBetaFeaturesWhitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/127873 (owner: 10Chad) [07:04:46] (03CR) 10Chad: [C: 031] "Looks good. Feel free to merge whenever you're ready to babysit this live." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/127793 (owner: 10Manybubbles) [07:15:06] (03PS1) 10Giuseppe Lavagetto: Decommission pdu checks in sdtpa. [operations/puppet] - 10https://gerrit.wikimedia.org/r/127875 [07:17:00] (03CR) 10Giuseppe Lavagetto: [V: 032] Decommission pdu checks in sdtpa. [operations/puppet] - 10https://gerrit.wikimedia.org/r/127875 (owner: 10Giuseppe Lavagetto) [07:17:32] (03CR) 10Giuseppe Lavagetto: [C: 032] Decommission pdu checks in sdtpa. [operations/puppet] - 10https://gerrit.wikimedia.org/r/127875 (owner: 10Giuseppe Lavagetto) [07:18:02] (03CR) 10Dzahn: "thx, those should remove the "UNKNOWNs" at" [operations/puppet] - 10https://gerrit.wikimedia.org/r/127875 (owner: 10Giuseppe Lavagetto) [07:22:50] (03CR) 10Dzahn: "revoke puppet cert,salt key, rm from Icinga" [operations/puppet] - 10https://gerrit.wikimedia.org/r/127262 (owner: 10coren) [07:23:48] mornining mutante [07:23:55] now back finally [07:24:32] ACKNOWLEDGEMENT - Host labstore4 is DOWN: PING CRITICAL - Packet loss = 100% daniel_zahn already removed from puppet, but #7110 says to contact Coren [07:33:54] matanya: welcome [08:27:18] (03PS4) 10Faidon Liambotis: dataset: fix module path [operations/puppet] - 10https://gerrit.wikimedia.org/r/119212 (owner: 10Matanya) [08:28:11] (03CR) 10Faidon Liambotis: [C: 032] dataset: fix module path [operations/puppet] - 10https://gerrit.wikimedia.org/r/119212 (owner: 10Matanya) [08:28:21] thanks paravoid [08:28:55] (03PS2) 10Faidon Liambotis: url-downloader: fix module path [operations/puppet] - 10https://gerrit.wikimedia.org/r/119245 (owner: 10Matanya) [08:29:06] (03CR) 10Faidon Liambotis: [C: 032 V: 032] url-downloader: fix module path [operations/puppet] - 10https://gerrit.wikimedia.org/r/119245 (owner: 10Matanya) [08:47:58] !log upgrading Bugzilla to 4.4.4 [08:48:05] Logged the message, Master [08:50:36] (03Abandoned) 10Aklapper: Upgrade custom code to Bugzilla 4.4.2 [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/113685 (owner: 10Aklapper) [08:53:10] (03Restored) 10Aklapper: Upgrade custom code to Bugzilla 4.4.2 [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/113685 (owner: 10Aklapper) [08:53:39] nice @ 'restored' feature [09:04:11] !log hooper - revoked puppet cert [09:04:17] Logged the message, Master [09:04:27] apergos: and that was that remnant from your report [09:05:21] yep saw, thanks! [09:06:38] matanya: see the very bottom of that Tampa etherpad.. [09:06:45] new stuff [09:16:02] mutante: mind running nmap on the entire subnet? this will make sure we know all machines that are still up [09:17:35] matanya: apergos just did a ping scan too [09:17:51] ping might not say the whole truth [09:17:56] not a ping scan. [09:18:07] I jut did pings of specific boxes [09:18:32] <_joe_> matanya: how come ping might not say the truth? [09:18:34] well, what i did was nmap with -sn [09:18:41] in the past, but not today yet [09:18:58] * _joe_ wonders if we have machines that do not respond to ping [09:19:02] _joe_: if ping/icmp is blocked [09:19:15] <_joe_> matanya: by whom? :) [09:19:33] there may be a couple [09:19:35] certainly not many [09:20:07] there are machines there dated to the fedora era no? :P how knows what was before puppet on hardy ... [09:21:05] no there aren't [09:21:22] apergos: all "es" _except_ 7 and 10? [09:22:16] apergos: above it says es4 should stay, so changing "es3-6" to "3,5,6" [09:27:21] (03PS1) 10Dzahn: decom: remove es1,2,3,5,6,8,9 from DHCP [operations/puppet] - 10https://gerrit.wikimedia.org/r/127886 [09:28:07] matanya: i'll scan [09:31:32] (03CR) 10Matanya: [C: 031] decom: remove es1,2,3,5,6,8,9 from DHCP [operations/puppet] - 10https://gerrit.wikimedia.org/r/127886 (owner: 10Dzahn) [09:39:46] apergos: what is the fate of stat1? spare? [09:48:29] (03PS1) 10ArielGlenn: toss pmtpa mw hosts from api dsh group [operations/puppet] - 10https://gerrit.wikimedia.org/r/127890 [09:49:14] 2 Warning: Maximum number of allowable file uploads has been exceeded in Unknown on line 0 [09:53:05] mutante: thanks, yes 4 should stay [09:53:29] matanya: I think all the pmtpa hosts that are not special or in warranty are going to recycling [09:53:51] (03CR) 10ArielGlenn: [C: 032] toss pmtpa mw hosts from api dsh group [operations/puppet] - 10https://gerrit.wikimedia.org/r/127890 (owner: 10ArielGlenn) [10:03:58] (03CR) 10Dzahn: [C: 032] decom: remove es1,2,3,5,6,8,9 from DHCP [operations/puppet] - 10https://gerrit.wikimedia.org/r/127886 (owner: 10Dzahn) [10:12:31] apergos: regarding stat1 found 6144 it says move to eqiad [10:12:59] yes, but what is meant by that is 'move the service to eqiad' [10:13:08] stat100x is tht host (I forget which number) [10:13:59] matanya: otto shut it down "stat1 is powered down and ready for unracking and wiping." [10:14:03] stat1003 [10:14:08] thanks [10:14:39] mutante: what do we do with mgmt dns entries for decomed hosts? [10:15:33] matanya: in pmtpa, remove them [10:15:48] but dont do the same twice [10:15:55] means ? [10:16:08] apergos: [10:16:55] means probably 3 of us were preparing similar changes locally [10:17:04] oh, sure [10:17:08] /me picks another thing from the list [10:17:55] what is count:2 [10:18:19] !log harmon - delete salt key [10:18:25] Logged the message, Master [10:18:27] matanya: number of hits in apergo's grep [10:20:21] apergos: "lvs" being in dsh, imho it's a false positive [10:20:29] lvs1/2 just match amslvs1/2 [10:20:35] any clue what is tesla host ? not found in RT [10:20:41] matanya: pre-labs labs [10:20:59] should go? stay? [10:21:03] (03PS1) 10ArielGlenn: remove mgmt ips for more tampa boxes [operations/dns] - 10https://gerrit.wikimedia.org/r/127894 [10:21:05] (03PS1) 10Alexandros Kosiaris: Add IPv6 address to carbon [operations/puppet] - 10https://gerrit.wikimedia.org/r/127895 [10:21:31] that's not all of them [10:21:44] matanya: 3801, 2510, 674, 132 ...et al [10:22:01] if someone wants to eyebal that and +1 if there are no issues, I'll push that and leave the rest for the other two commit-preparing folks :-) [10:22:24] matanya: maybe 3801 needs reopen [10:22:34] * matanya thinks the same [10:23:26] matanya: the count"n stuff is when there are multiple files or entries, it will make the report format giant if I list them all [10:23:44] count:2 for dns usually means there's a prod ip and a mgmt ip (for example) [10:23:50] ok, i'll go over this [10:24:40] (03CR) 10Matanya: [C: 031] remove mgmt ips for more tampa boxes [operations/dns] - 10https://gerrit.wikimedia.org/r/127894 (owner: 10ArielGlenn) [10:24:56] okey okey [10:24:57] *dokey [10:25:18] (03CR) 10Dzahn: [C: 031] "yes, all of this has already had it's primary IPs removed and, in pmtpa, mgmt should also be removed now" [operations/dns] - 10https://gerrit.wikimedia.org/r/127894 (owner: 10ArielGlenn) [10:25:49] !log upgraded php-luasandbox to 1.9-1 on test.wikimedia.org [10:25:55] Logged the message, Master [10:26:57] (03CR) 10ArielGlenn: [C: 032] remove mgmt ips for more tampa boxes [operations/dns] - 10https://gerrit.wikimedia.org/r/127894 (owner: 10ArielGlenn) [10:27:05] (03PS3) 10Hashar: puppet-lint role/nova.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/127147 [10:28:13] (03CR) 10Hashar: "PS3 address most issues. I have left the selector inside a selector since I am never how bad it is going to interact. Also kept some arr" [operations/puppet] - 10https://gerrit.wikimedia.org/r/127147 (owner: 10Hashar) [10:34:09] i want a PS4 [10:34:45] what's the selling point(s) that gotcha? [10:35:09] is that a reply to hashar saying "PS3 address most issues" [10:35:35] !log upgraded php-luasandbox to 1.9.1 on beta (deployment-apache0{1,2}) [10:35:41] Logged the message, Master [10:38:30] PS4>XBONE! [10:40:54] (03PS1) 10Dzahn: remove even more Tampa mgmt IPs [operations/dns] - 10https://gerrit.wikimedia.org/r/127896 [10:44:10] cleanup \o/ [10:46:12] it's getting cold in Tampa :) Sensor A [10:46:13] is under threshold: 33C [10:52:00] (03PS2) 10Dzahn: remove even more Tampa mgmt IPs [operations/dns] - 10https://gerrit.wikimedia.org/r/127896 [10:58:00] (03CR) 10Steinsplitter: [C: 031] "now okay, pleas merge." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/127479 (owner: 10Gerrit Patch Uploader) [11:00:35] you call that cold? [11:02:25] heh, no:) just some sensor in pmtpa thinks so [11:03:52] it was a bit confusing that power consumption appeared to go up a little bit after we shut things down [11:04:11] right, Reedy [11:06:30] (03CR) 10ArielGlenn: [C: 031] remove even more Tampa mgmt IPs [operations/dns] - 10https://gerrit.wikimedia.org/r/127896 (owner: 10Dzahn) [11:07:49] (03CR) 10Dzahn: [C: 032] remove even more Tampa mgmt IPs [operations/dns] - 10https://gerrit.wikimedia.org/r/127896 (owner: 10Dzahn) [11:09:24] (03PS1) 10Aklapper: Port upstream Bugzilla 4.4.4 changes to our modifications [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/127898 [11:14:57] (03Abandoned) 10Hoo man: Increase apache MaxClients to 23 in order to have 40 more scaling slots [operations/puppet] - 10https://gerrit.wikimedia.org/r/127632 (owner: 10Hoo man) [11:23:17] andre__: did you see valhalla replacing wikibugs with pywikibugs? [11:23:30] mutante, uh, no [11:23:35] https://github.com/valhallasw/pywikibugs [11:23:46] 2. Bugzilla sends an e-mail to wikibugs-l@lists.wm.o [11:23:46]