[01:23:21] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: puppet fail [01:30:18] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: Puppet has 4 failures [01:40:07] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:04:33] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:31:36] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: Puppet has 9 failures [02:45:38] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [02:49:06] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.022 second response time [02:52:46] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.022 second response time [03:39:20] (03CR) 10MZMcBride: "dev, devs, developers?" [dns] - 10https://gerrit.wikimedia.org/r/181398 (owner: 10Dzahn) [04:11:22] Hi mark, are you there? ...and are you _the_ Mark Bergsma? (apologies of not, there are more than a few Marks around here...) [04:12:12] ...just looking for some help with tweaking the mobile_redirect in text-frontend.inc.vcl.erb [04:15:58] anyone who pings on "varnish"? varnish varnish varnish varnish varnish varnish varnish varnish varnish varnish [04:19:29] AndyRussG: Don't be annoying. [04:19:34] likely not at 05:19 in the morning [04:19:43] AndyRussG: And yes, that's Mark B. [04:20:16] * AndyRussG waves only slightly annoyingly at Fiona and Krenair :) [04:20:22] :-) [04:22:12] It's only 10:20 pm here, a fine time to stare at vcl files [04:23:11] CentralNotice causes some extra round trips on mobile by making background requests to non-mobile sites which get redirected to the mobile URLs [04:23:25] * AndyRussG wanna fix [04:31:38] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: Puppet has 2 failures [04:42:09] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [05:29:53] Morning/evening MaxSem, how's it going? [05:30:12] hey AndyRussG [05:31:13] MaxSem: Got a sec for a quick mobile and varnish quesiton? [05:31:23] sure [05:31:28] Thanks! [05:31:38] Basically CentralNotice causes some extra round trips on mobile by making background requests to non-mobile sites which get redirected to the mobile URLs [05:32:03] I'm trying to think of what the best solution is [05:32:49] CentralNotice makes background calls to Special:BannerLoader and Special:RecordImpression on meta [05:33:07] index.php?title=Special:CentralNotice [05:33:28] or whatever the page name is [05:33:44] ugly urls aren't redirected [05:34:03] yeah so varnish is redirecting [05:35:15] I guess one option would be to add a regex to exclude those from the redirect in text-frontend.inc.vcl.erb [05:35:29] But I'm not sure that's quite right... [05:35:30] just use ugly urls [05:36:26] Yeah, one solution would be to patch CentralNotice to use the mobile URLs when needed [05:36:53] ...mm then I was thinking, what does MobileFrontend actually to for similar situations, like inter-wiki links? [05:37:16] Maybe there's something in the mobile skin that's already doing something like this? [05:37:48] we rely on varnish to redirect interwikis to the mobile site [05:39:51] Ah hmmm [05:41:11] does the mobile skin know anything at all about the mobile-specific URL? Does it do anything with $wgServer? [05:41:32] MaxSem: ^ [05:42:02] we have a function that returns a mobile url, but it's guaranteed to work only for local wiki [05:42:14] so once again, just use ugly urls [05:47:13] MaxSem: OK! ...mmm can u point me to that function anyway? I should at least use similar terms, etc., maybe... [05:47:16] Also thanks much! :) [05:47:37] don't use it, don't look at it [05:47:53] <_< [05:47:56] >_> [05:48:05] that bad? [05:48:27] it is guaranteed to break for some sites linking to meta [05:50:57] Hmmm [05:58:49] * AndyRussG takes MaxSem's advice not to look [05:58:52] thanks again! [06:06:22] MaxSem: mmm... expected method from PHP or JS to detect that we're on the mobile URL (as opposed to the mobile or desktop view, which can be forced on either URL) ? [06:07:17] just make em ugly unconditionally? [06:08:14] on php side, the url is received from varnish rewritten to desktop [06:08:20] Maybe I didn't understand? By ugly, you mean fully qualified with the ".m." bit in them? [06:08:44] ugly is /w/indec.php?blah [06:10:07] MaxSem: aaaaaaaaaaaaaaaaaaah OK I see those don't redirect [06:10:17] * AndyRussG removes glasses to facepalm [06:11:11] K that's _much_ easier [06:12:14] ^ K now I see you did say that before, silly me [06:12:21] Reading is hard. [06:12:56] Fiona: yeah, it's been a long day [06:13:35] You know if everyone reading this gave just $3, [06:14:41] yes there are times when that coffee the banners promise would come in handy [06:18:02] PROBLEM - puppet last run on lvs2003 is CRITICAL: CRITICAL: puppet fail [06:28:42] PROBLEM - puppet last run on mw1240 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:23] RECOVERY - puppet last run on lvs2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:33:33] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: puppet fail [06:35:53] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:31] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures [06:37:12] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 1 failures [06:43:03] RECOVERY - puppet last run on mw1240 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:25] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:58] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:26] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:50:07] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:50:47] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: Puppet has 3 failures [07:04:27] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:08:00] (03PS1) 10Yuvipanda: toollabs: Do not rewrite /toolname to / by default [puppet] - 10https://gerrit.wikimedia.org/r/182053 [07:14:15] (03CR) 10Yuvipanda: [C: 032] toollabs: Do not rewrite /toolname to / by default [puppet] - 10https://gerrit.wikimedia.org/r/182053 (owner: 10Yuvipanda) [08:09:44] (03CR) 10Alexandros Kosiaris: [C: 032] "Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/181549 (owner: 10KartikMistry) [08:12:29] (03CR) 10Alexandros Kosiaris: [C: 032] "Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/181404 (owner: 10KartikMistry) [08:13:31] (03PS2) 10Yuvipanda: Tools: Fix typo in static nginx configuration [puppet] - 10https://gerrit.wikimedia.org/r/181989 (owner: 10Tim Landscheidt) [08:14:21] (03CR) 10Yuvipanda: [C: 032] "Not sure how this even worked..." [puppet] - 10https://gerrit.wikimedia.org/r/181989 (owner: 10Tim Landscheidt) [08:20:14] (03CR) 10Yuvipanda: "Not fully sure if this will work for labs graphite - nginx on labsproxy does termination, and nginx -> apache is just plain http." [puppet] - 10https://gerrit.wikimedia.org/r/181949 (owner: 10Hoo man) [08:25:40] (03CR) 10Alexandros Kosiaris: [C: 032] Change postgresql dir to have version and cluster [puppet] - 10https://gerrit.wikimedia.org/r/172659 (owner: 10Alexandros Kosiaris) [08:34:27] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: Puppet has 4 failures [08:36:07] PROBLEM - puppet last run on labsdb1006 is CRITICAL: CRITICAL: Puppet has 9 failures [08:41:35] (03PS1) 10Yuvipanda: Vastly simplify Shingen [software/shinkengen] - 10https://gerrit.wikimedia.org/r/182055 [08:41:39] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [08:41:58] hmm [08:42:01] (03PS2) 10Yuvipanda: Vastly simplify Shingen [software/shinkengen] - 10https://gerrit.wikimedia.org/r/182055 [08:42:11] I’m thinking I should just kill the package, move it all into one file, and put that in puppet [08:43:17] RECOVERY - puppet last run on labsdb1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:43:40] hehe [08:45:38] (03PS3) 10Yuvipanda: Vastly simplify Shingen [software/shinkengen] - 10https://gerrit.wikimedia.org/r/182055 [08:46:24] it’s so small... [08:46:28] now, at least [08:47:28] (03PS4) 10Yuvipanda: Vastly simplify Shingen [software/shinkengen] - 10https://gerrit.wikimedia.org/r/182055 [08:51:24] (03PS5) 10Yuvipanda: Vastly simplify Shingen [software/shinkengen] - 10https://gerrit.wikimedia.org/r/182055 [08:51:56] (03CR) 10Yuvipanda: [C: 032 V: 032] Vastly simplify Shingen [software/shinkengen] - 10https://gerrit.wikimedia.org/r/182055 (owner: 10Yuvipanda) [09:05:17] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 1 failures [09:18:51] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:29:43] (03PS1) 10Yuvipanda: shinken: Move shinkengen into ops/puppet [puppet] - 10https://gerrit.wikimedia.org/r/182059 [09:37:33] greetings [09:38:46] * YuviPanda waves at godog [09:39:54] hey Yuvi [09:42:55] heya :) [09:43:09] godog: wanna +1 ^? [09:43:17] well, CR [09:47:41] YuviPanda: sure, can you add me so I don't lose track of that? [09:47:48] godog: yeah, done :) [09:48:07] tyvm! [09:53:43] springle: around? [09:56:00] godog: did we end up creating a project for debs at all? [09:58:14] YuviPanda: in phab? not sure where did that go [09:58:21] hmm [09:58:25] let me file a request to create a package [09:58:26] err [09:58:28] to create a project [09:59:43] springle: when you get time, CX schema review please :) [10:00:40] godog: ok, created https://phabricator.wikimedia.org/T85443 [10:02:01] YuviPanda: Q: How can I add cronjob on Beta, is it just like Labs? [10:02:24] kart_: depends on what you want it to be for :) [10:02:29] yeah :) [10:02:33] kart_: betalabs, I suppose you should puppetize it [10:02:58] YuviPanda: need to run sql quries to get some statistics from DB [10:03:08] from the betacluster? [10:03:12] (and then email it) [10:03:15] Yes. [10:03:18] what for? [10:03:26] the data there isn’t really guaranteed to be anything useful... [10:03:54] YuviPanda: yes. I know. Content Translation usage. [10:04:10] (will be bit useful) [10:04:22] If things are complicated, I'll do manual run. [10:04:25] I still have a feeling at some point betacluster got confused with betafeatures. [10:05:06] betacluster :) [10:05:20] Can we rename betacluster to something else? ;) [10:05:40] too late, I htink [10:05:51] #ercluser [10:10:24] PROBLEM - check_puppetrun on db1008 is CRITICAL: CRITICAL: Puppet has 2 failures [10:15:24] PROBLEM - check_puppetrun on db1008 is CRITICAL: CRITICAL: Puppet has 2 failures [10:20:26] RECOVERY - check_puppetrun on db1008 is OK: OK: Puppet is currently enabled, last run 128 seconds ago with 0 failures [10:35:44] !log reboot ms-be2011, stuck while removing a LD, no console [10:35:52] Logged the message, Master [10:42:30] RECOVERY - very high load average likely xfs on ms-be2011 is OK: OK - load average: 5.38, 2.09, 0.77 [10:42:31] RECOVERY - RAID on ms-be2011 is OK: OK: optimal, 13 logical, 13 physical [10:54:02] (03CR) 10Filippo Giunchedi: shinken: Move shinkengen into ops/puppet (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/182059 (owner: 10Yuvipanda) [10:56:57] godog: good catch, it was a copypasta fail [10:56:57] (03PS2) 10Yuvipanda: shinken: Move shinkengen into ops/puppet [puppet] - 10https://gerrit.wikimedia.org/r/182059 [10:57:49] (03CR) 10Yuvipanda: shinken: Move shinkengen into ops/puppet (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/182059 (owner: 10Yuvipanda) [10:58:55] (03CR) 10Filippo Giunchedi: "is there a reference ticket for this? ditto for I2ffc6aab07fbdd" [puppet] - 10https://gerrit.wikimedia.org/r/181984 (owner: 10John F. Lewis) [11:03:04] YuviPanda: hehe, so also the exec in the puppet recipe can be removed? [11:03:13] godog: hmm? [11:03:16] YuviPanda: exec { '/usr/local/bin/shingen': [11:03:26] godog: the exec runs this script to generate the config, so I guess that should stay? [11:04:05] what's the difference between /usr/local/bin/shinkengen and /usr/local/bin/shingen ? [11:04:34] gaaah [11:04:37] typo [11:04:51] haha yeah realized it now, thought they were two different files instead [11:04:55] (03PS3) 10Yuvipanda: shinken: Move shinkengen into ops/puppet [puppet] - 10https://gerrit.wikimedia.org/r/182059 [11:04:56] godog: ^ [11:05:15] godog: I was going to call it ‘shingen’ but then it’s only one character different from ‘shinken’ and would’ve been confusing [11:05:56] indeed, good call [11:06:20] do you need to explicitly declare also hostgroups definitions or are they created on the fly? [11:07:05] godog: I was explicitly creating the hostgroups, but then it turns out shinken doesn’t require that [11:07:13] nice [11:07:16] godog: just put it in the hostgroups key for a host and it’s all good [11:07:20] so that simplified things [11:09:12] (03CR) 10Filippo Giunchedi: [C: 031] shinken: Move shinkengen into ops/puppet [puppet] - 10https://gerrit.wikimedia.org/r/182059 (owner: 10Yuvipanda) [13:40:11] hey, so I'm writing this python program to download webpages, so I was testing it on Wikipedia (http://www.wikipedia.org), turns out Wikipedia uses Chunked Transfer, now the size advertized on the top of the chunks is 008000 which comes to 32768, however when I download the page through curl, the size is 42387, because of this what's happening is that my [13:40:11] program thinks it's downloaded the entire page (according to the size specified in the first line in hex) and stops while the page isn't downloaded. I'm curious why is there this discrepancy in the advertises size and the actual size, turns out Chunked Transfers are also uses for two other resources on that page [13:40:11] (http://bits.wikimedia.org/meta.wikimedia.org/load.php?debug=false&lang=en&modules=ext.gadget.wm-portal&only=styles&skin=vector&* and http://bits.wikimedia.org/meta.wikimedia.org/load.php?debug=false&lang=en&modules=ext.gadget.wm-portal&only=scripts&skin=vector&*) and their correct size is advertized so what's the issue in the case of the text/html [13:40:44] (since this is to do with the headers, I thought of asking here, let me know if it's off-topic) [14:05:16] ACKNOWLEDGEMENT - puppet last run on ms-be2011 is CRITICAL: CRITICAL: Puppet has 3 failures Filippo Giunchedi sda broken [14:05:16] PROBLEM - Host db1021 is DOWN: CRITICAL - Plugin timed out after 15 seconds [14:05:33] RECOVERY - Host db1021 is UP: PING OK - Packet loss = 0%, RTA = 2.16 ms [14:29:30] (03PS1) 10Ottomata: [WIP] Add account for Ananth Ramakrishnan (contractor) and add to analytics-privatedata-users group [puppet] - 10https://gerrit.wikimedia.org/r/182070 [15:11:26] Does anyone know of a way to make phabricator search for all tickets in project A, and that also don't include project B? In my case I want to search for all operations project tickets needing triage, but exclude any labs tasks [15:12:14] its unwieldily to try to hit all operations need triage for ticket duty and such when labs are usually outside of that scope entirely [15:12:19] (but show up since they have operations tag) [15:13:01] so you have to go to advance search and add the projects you don't want to see [15:13:13] I dont see a 'do not include' option [15:13:19] robh: https://phabricator.wikimedia.org/maniphest/ [15:13:19] only a 'in projects' option [15:13:19] in the "Not in Projects" [15:13:33] oh, im in the top left search [15:13:36] not the right sidebar search [15:13:39] did i mention i fucking hate that? [15:13:48] why are there two different query windows? [15:14:15] (click the search icon in the top left, says 'advanced search' but doesn't show a third of the options [15:14:16] ) [15:15:01] robh: One searches tasks, another searches everything [15:15:24] incorrect, the top right searches everything [15:15:28] has checkboxes for them all [15:15:34] Document Types [15:15:38] lsits all of them [15:15:52] it just has a third of the query options in terms of entry textboxes [15:15:53] mutante: ping :D [15:16:21] marktraceur: it does default to task though you are right [15:16:37] hoo: I've been waiting to catch him online for a few days - going on 55 hours idle now :p [15:16:40] it just seems like a stupid difference to exclude advanced options when it says advanced, heh [15:16:53] JohnLewis: meh... [15:17:42] PROBLEM - Host db1055 is DOWN: CRITICAL - Plugin timed out after 15 seconds [15:18:02] RECOVERY - Host db1055 is UP: PING OK - Packet loss = 0%, RTA = 2.84 ms [15:19:00] (03PS2) 10Ottomata: Add account for Ananth Ramakrishnan (contractor) and add to analytics-privatedata-users group [puppet] - 10https://gerrit.wikimedia.org/r/182070 [15:22:44] heh, that a cosmetic 'click this search not this other search' is my biggest complaint about an entirely new ticketing system is pretty awesome. [15:23:03] ACKNOWLEDGEMENT - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 9.75134434482e-138 ottomata Ok, Christian wants to check this out before I fix it. [15:23:16] I'm fairly certain at no time at any company have I experienced quite so easy a transition between two bits of task mgmt software [15:29:02] *sigh* another week without SWAT [15:29:08] What am I going to do with myself [15:29:55] marktraceur: You can set something else ablaze surely :D [15:30:24] Nothing so spectacular [15:30:43] marktraceur: https://gerrit.wikimedia.org/r/181791 [15:30:53] Not deploying that is probably way more harmful when deploying it [15:30:59] Oh my. [15:31:22] * than [15:31:30] I really tried to get sleep [15:31:36] But I failed. Badly. [15:31:37] Heh [15:32:27] I'm with you there. Tossed and turned all night because I'm sick. [15:32:48] I'm at the 31C3... and ... [15:32:53] well, I guess it's obvious :d [15:33:08] Ah, yes. [15:33:24] (03CR) 10Ottomata: [C: 032] "I verified Ananth's ssh key via Google Hangout." [puppet] - 10https://gerrit.wikimedia.org/r/182070 (owner: 10Ottomata) [15:36:06] marktraceur: If you're bored... I have stuff to review in gerrit :D' [15:36:08] * :'D [15:36:15] hoo: Why the hell not [15:36:25] Provided it's something I *can* review [15:37:14] https://gerrit.wikimedia.org/r/181808 [15:37:22] that's pure moving stuff around [15:37:28] and the follow up is pure refactoring [15:37:35] nothing should change functionality wise [15:37:38] *should* [15:40:34] (03PS2) 10Filippo Giunchedi: graphite: introduce local c-relay [puppet] - 10https://gerrit.wikimedia.org/r/181080 [15:41:54] hoo: You inspire my confidence. [15:59:22] hoo: Who's maintaining TorBlock these days? [15:59:54] marktraceur: No one and everyone, I guess [16:00:29] Fun times. [16:00:45] hoo: +2'd [16:01:38] awesome :) [16:01:47] There's a follow up, if you got more time :D [16:02:10] I'm looking at it [16:02:17] hoo: I don't understand the "other user" business [16:02:39] It can't check other users, because it takes the IP from global state [16:02:45] Right [16:02:59] hoo: So why bother passing in the user if you're just checking wgUser anyway? :) [16:03:03] Doubt that really happens, but who knows [16:03:07] (03PS1) 10Ottomata: Use logster to parse varishkafka logs and send to statsd [puppet] - 10https://gerrit.wikimedia.org/r/182072 [16:03:16] Has anyone enabled 2FA for their phabricator account; if so, have you also reset it yet? (I have not enabled it, but I won't until I know the reset when lost procedures exist ;) [16:03:18] Oh, because hook. [16:03:20] Never mind [16:03:45] (03Abandoned) 10Ottomata: [WIP] Use logster to parse mobile vanrishkafka logs and send to statsd [puppet] - 10https://gerrit.wikimedia.org/r/177562 (owner: 10Ottomata) [16:03:54] (03CR) 10jenkins-bot: [V: 04-1] Use logster to parse varishkafka logs and send to statsd [puppet] - 10https://gerrit.wikimedia.org/r/182072 (owner: 10Ottomata) [16:04:01] robh: No, I expect my three-character password to be enough security. [16:04:31] (03PS2) 10Ottomata: Use logster to parse varishkafka logs and send to statsd [puppet] - 10https://gerrit.wikimedia.org/r/182072 [16:04:33] My paranoia and wish for 2FA is tempered by how annoying it is to get in without it [16:04:41] wikitech = i use root and reset my shit [16:04:50] though for non roots, it seems like a major pain [16:05:07] since it requires some method of verification to eliminate social engineering [16:05:15] (03CR) 10jenkins-bot: [V: 04-1] Use logster to parse varishkafka logs and send to statsd [puppet] - 10https://gerrit.wikimedia.org/r/182072 (owner: 10Ottomata) [16:05:28] robh: 2FA secretly has a small-print warning: "Not to be used unless you're root" :p [16:05:47] heh, no 2FA = no labs instance controls [16:05:58] but those are mostly roots + admins [16:06:10] which has a very high level of overlap, so yea, for the most part you are right ;D [16:06:38] (03PS3) 10Ottomata: Use logster to parse varishkafka logs and send to statsd [puppet] - 10https://gerrit.wikimedia.org/r/182072 [16:06:39] 'not to be used unless you realize someday you may have to google hangout or video conference to prove you are you' [16:07:09] hi robh, ping about namenodes, whats the haps? :) [16:07:12] 'and publically shame yourself by doing it in an ops meeting' :p [16:07:16] robh, I use 2FA on phabricator and wikitech [16:07:41] ottomata: these analytics1001/1002? [16:08:01] ottomata: the task has my last update that there is nothing hitting dhcp, i have it on my list today to try it again from scratch and reconfirm all my findings [16:08:07] and then escalate to someone with network rights [16:08:08] yes [16:08:19] robh, i thought the problem was the network boo boo that I made [16:08:22] that someone fixed. [16:08:30] hrmm, i vaguely recall that [16:08:33] missing semicolon in something [16:08:47] but no comments in task [16:08:50] bad robh [16:08:52] =P [16:09:01] ottomata: So its on my list to get back to today ;D [16:09:20] ok cool!, danke! [16:09:24] Krenair: good to know someone has at least confirmed it works =] [16:09:35] robh, I certainly won't get to do anytihng with them until next week, so no hurry [16:09:53] cool [16:10:16] hoo: Comments! [16:10:31] oh noes [16:10:37] (03CR) 10John Vandenberg: "+1" [puppet] - 10https://gerrit.wikimedia.org/r/176898 (owner: 10Nemo bis) [16:10:37] Small, though :) [16:10:42] It's a good patch otherwise [16:11:20] Oh, I see... those are stupid... it says editing and then it says they actually want to send an email [16:11:28] copy and pasted code, ftw [16:11:37] Heh [16:13:49] (03CR) 10Mdann52: [C: 031] FlaggedRevs configuration on de.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/180813 (owner: 10Glaisher) [16:16:06] (03CR) 10BBlack: [C: 031] Use logster to parse varishkafka logs and send to statsd [puppet] - 10https://gerrit.wikimedia.org/r/182072 (owner: 10Ottomata) [16:17:51] (03CR) 10Filippo Giunchedi: [C: 031] dsh: create files based on exported resources [puppet] - 10https://gerrit.wikimedia.org/r/179121 (owner: 10Giuseppe Lavagetto) [16:18:53] ... and out [16:21:32] (03CR) 10Filippo Giunchedi: "minor comment, the rest LGTM" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/182072 (owner: 10Ottomata) [16:24:28] (03PS4) 10Ottomata: Use logster to parse varishkafka logs and send to statsd [puppet] - 10https://gerrit.wikimedia.org/r/182072 [16:24:36] (03CR) 10Ottomata: Use logster to parse varishkafka logs and send to statsd (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/182072 (owner: 10Ottomata) [16:24:54] godog: do you think that is safe to merge today then? I am probably not going to work miuch more today [16:25:05] maybe I should limit it to just a few varnishes first? [16:25:08] mobiles maybe? [16:26:14] ottomata: yep if you can apply it incrementally I think it is safer, will you be around e.g. tomorrow? [16:27:01] i will be for a half day as well [16:27:07] and working full day friday [16:27:10] but off wed and thurs [16:28:32] ottomata: ack, yeah a few varnishes sounds good to me [16:28:41] k [16:30:27] I'll be around most of the day if we need to revert [16:30:32] ok cool. [16:31:01] (03PS5) 10Ottomata: Use logster to parse varishkafka logs and send to statsd [puppet] - 10https://gerrit.wikimedia.org/r/182072 [16:31:05] thanks yalls, going to include this on mobiles only for now. [16:31:47] ok [16:32:27] (03PS6) 10Ottomata: Use logster to parse varnishkafka logs and send to statsd [puppet] - 10https://gerrit.wikimedia.org/r/182072 [16:33:44] (03CR) 10Ottomata: [C: 032] Use logster to parse varnishkafka logs and send to statsd [puppet] - 10https://gerrit.wikimedia.org/r/182072 (owner: 10Ottomata) [16:46:04] (03PS4) 10Yuvipanda: shinken: Move shinkengen into ops/puppet [puppet] - 10https://gerrit.wikimedia.org/r/182059 [16:47:20] (03CR) 10Yuvipanda: [C: 032] shinken: Move shinkengen into ops/puppet [puppet] - 10https://gerrit.wikimedia.org/r/182059 (owner: 10Yuvipanda) [16:48:59] (03PS3) 10John F. Lewis: planets: remove SSL stanza [puppet] - 10https://gerrit.wikimedia.org/r/181984 [16:49:15] (03PS2) 10John F. Lewis: planet: change dns to misc-web [dns] - 10https://gerrit.wikimedia.org/r/181985 [16:49:51] godog ^^ [16:50:04] woot woot [16:50:17] 2fa works for me in phab too, this pleases me. [16:50:36] until it messes up and you're left without access [16:51:12] yea [16:51:16] we're still not sure how to reset that [16:52:09] * robh removes it while his phone is not dead [16:52:26] (03PS1) 10Ottomata: Fix for new varnishkafka logster job [puppet] - 10https://gerrit.wikimedia.org/r/182073 [16:52:42] (03PS2) 10Ottomata: Fix for new varnishkafka logster job [puppet] - 10https://gerrit.wikimedia.org/r/182073 [16:53:37] (03CR) 10Ottomata: [C: 032] Fix for new varnishkafka logster job [puppet] - 10https://gerrit.wikimedia.org/r/182073 (owner: 10Ottomata) [16:54:19] robh: https://secure.phabricator.com/book/phabricator/article/multi_factor_auth/#recovering-from-lost-fac ? [16:55:00] yep found it already [16:55:06] testing in progress =] [16:55:09] thx though! [16:55:36] im on that exact same page in fact, heh [16:55:55] a new meaning to 'we're on the same page' [17:00:22] yea that was super easy [17:00:28] i just enabled, tested worked, and disabled [17:00:46] via admin command, so its cool to use regularly now imo (though i dunno shit about our rollout plan ;) [17:02:04] PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: Puppet has 1 failures [17:02:21] PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: Puppet has 1 failures [17:05:02] robh: see what can be accomplished by looking into the docs when you go 'idk' :D [17:05:34] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:05:53] RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:06:16] yea but its a lot easier to just ping chase and make him answer (kidding, mostly) [17:06:33] so wikitech is updated for future clinic duty folks [17:07:11] (03CR) 10Filippo Giunchedi: [C: 04-1] etherpad: remove SSL stanza (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/181413 (owner: 10John F. Lewis) [17:07:40] (03CR) 10Filippo Giunchedi: [C: 031] planets: add Varnish statement [puppet] - 10https://gerrit.wikimedia.org/r/181419 (owner: 10John F. Lewis) [17:08:13] (03CR) 10Filippo Giunchedi: [C: 031] planets: remove SSL stanza [puppet] - 10https://gerrit.wikimedia.org/r/181984 (owner: 10John F. Lewis) [17:18:51] (03PS3) 10John F. Lewis: etherpad: remove SSL stanza [puppet] - 10https://gerrit.wikimedia.org/r/181413 [17:18:57] godog ^^ [17:19:38] (03PS1) 10Ottomata: Add cache type into statsd prefix for varnishkafka metrics [puppet] - 10https://gerrit.wikimedia.org/r/182076 [17:19:45] (03CR) 10jenkins-bot: [V: 04-1] Add cache type into statsd prefix for varnishkafka metrics [puppet] - 10https://gerrit.wikimedia.org/r/182076 (owner: 10Ottomata) [17:19:49] (03PS2) 10Ottomata: Add cache type into statsd prefix for varnishkafka metrics [puppet] - 10https://gerrit.wikimedia.org/r/182076 [17:20:40] (03CR) 10Ottomata: [C: 032] Add cache type into statsd prefix for varnishkafka metrics [puppet] - 10https://gerrit.wikimedia.org/r/182076 (owner: 10Ottomata) [17:23:21] is there a way for a labs node to determine what project it is a member of? facter doesn't know. [17:24:35] in ldap? [17:24:42] (03PS1) 10Ottomata: Emergency fix for last change. stdlib's delete did not do what I thought it did [puppet] - 10https://gerrit.wikimedia.org/r/182077 [17:24:45] JohnLewis: yep, looks good, I'll let mutante take a look too [17:24:50] (03CR) 10jenkins-bot: [V: 04-1] Emergency fix for last change. stdlib's delete did not do what I thought it did [puppet] - 10https://gerrit.wikimedia.org/r/182077 (owner: 10Ottomata) [17:24:52] (03PS2) 10Ottomata: Emergency fix for last change. stdlib's delete did not do what I thought it did [puppet] - 10https://gerrit.wikimedia.org/r/182077 [17:25:10] (03CR) 10Ottomata: [C: 032 V: 032] Emergency fix for last change. stdlib's delete did not do what I thought it did [puppet] - 10https://gerrit.wikimedia.org/r/182077 (owner: 10Ottomata) [17:25:36] godog: alright [17:25:56] hmm ottomata good idea, i'll check [17:26:43] OHMAHGOODNESS [17:26:58] stdlib's delete actually alters the variable you are trying to delete a string from [17:27:18] i wanted to extract 'mobile' from 'webrequest_mobile' in the $topic variable [17:27:27] delete($topic, 'webrequest_') [17:27:33] doesn't just return 'mobile' [17:27:40] it alters $topic so that it contains 'mobile' [17:27:46] haw [17:27:48] the functino in ruby actually does [17:27:48] puppet? [17:27:52] collection.gsub! item, '' [17:27:53] yes [17:27:55] that's horrible [17:27:55] (03CR) 10John F. Lewis: [C: 031] cache: install the planet SSL cert on misc-web [puppet] - 10https://gerrit.wikimedia.org/r/181415 (owner: 10Dzahn) [17:28:04] the ! means it will alter the argument [17:28:25] why would they do that?! [17:28:29] there is a perfectly normal gsub! [17:28:31] without ! [17:28:34] called [17:28:36] gsub [17:28:46] it looks like it doesn't do the same for arrays [17:28:57] it won't alter the array if you try to delete an element [17:29:14] geeezuuuuuzzz [17:30:20] jgage: I asked this once and coren pointed me to a file in etc [17:30:27] can't remember it tho :) [17:30:56] (03PS1) 10AndyRussG: DO NOT DEPLOY BEFORE https://gerrit.wikimedia.org/r/#/c/182074/ Ugly URLs to override mobile redirect for CentralNotice [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182078 [17:32:07] thanks chase, i will grep for it [17:32:36] (03PS1) 10Yuvipanda: shinken: Style fixes for shinkengen [puppet] - 10https://gerrit.wikimedia.org/r/182079 [17:32:43] jgage: /etc/wmflabs-project [17:32:44] (03CR) 10jenkins-bot: [V: 04-1] shinken: Style fixes for shinkengen [puppet] - 10https://gerrit.wikimedia.org/r/182079 (owner: 10Yuvipanda) [17:33:05] yep just found it. thanks! [17:33:26] jgage: there’s also $::instanceproject [17:33:27] in puppet [17:33:32] facter should know that, I think [17:33:42] (03PS2) 10Yuvipanda: shinken: Style fixes for shinkengen [puppet] - 10https://gerrit.wikimedia.org/r/182079 [17:34:07] facter does not seem to know about that [17:34:19] facter instanceproject returns nothing and facter | grep $projectname doesn't find it [17:34:33]