[00:02:26] (03CR) 10Dzahn: [C: 032 V: 032] "sorry it took so long, i actually checked the diffs of the /template/en/custom files now, i just saw one left in the index.html.tmpl, fixe" [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/119726 (https://bugzilla.wikimedia.org/61499) (owner: 1001tonythomas) [00:08:14] (03PS1) 10MaxSem: Fix Amazon Silk detection [operations/puppet] - 10https://gerrit.wikimedia.org/r/137838 [00:10:05] (03CR) 10Kaldari: [C: 031] Fix Amazon Silk detection [operations/puppet] - 10https://gerrit.wikimedia.org/r/137838 (owner: 10MaxSem) [00:12:11] crap, i broke something in Bugzilla, looking [00:16:02] !log fixed permissions on bugzilla's index.cgi, sry [00:16:07] Logged the message, Master [00:22:49] ACKNOWLEDGEMENT - RAID on es1006 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) Sean Pringle RT 7639 [00:24:47] mutante: I meant against the upstream project, which is me :) [00:27:24] bblack: which ticket system:) [00:29:16] https://github.com/blblack/gdnsd/issues?state=open [00:29:52] ok:) [00:33:40] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Thu 05 Jun 2014 21:33:08 UTC [00:33:50] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Fri Jun 6 00:33:42 UTC 2014 [00:38:33] !log nginx restarted on ssl* [00:38:37] Logged the message, Master [00:44:39] (03CR) 10BBlack: [C: 032 V: 032] Fix Amazon Silk detection [operations/puppet] - 10https://gerrit.wikimedia.org/r/137838 (owner: 10MaxSem) [01:13:40] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Thu 05 Jun 2014 19:12:41 UTC [01:38:40] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [01:51:23] ori: hmm. when I go to http://codepen.io/Krinkle/full/laucI I get the same thing, even with http:// [01:51:35] let me try disabling https-everywhere [01:51:50] or, I'll just use a different browser [01:57:46] well, it works in safari [01:57:47] weird [02:19:01] (03PS1) 10Yurik: LABS: Disabled wmgZeroBanner on zerowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/137854 [02:19:54] greg-g, can i sync ^ [02:20:47] (03CR) 10Yurik: [C: 032] "production NOOP" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/137854 (owner: 10Yurik) [02:20:53] (03Merged) 10jenkins-bot: LABS: Disabled wmgZeroBanner on zerowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/137854 (owner: 10Yurik) [02:40:29] (03PS6) 10Mwalker: Adding GELF support to LogStash [operations/puppet] - 10https://gerrit.wikimedia.org/r/137811 [02:43:32] !log LocalisationUpdate completed (1.24wmf7) at 2014-06-06 02:42:28+00:00 [02:43:37] Logged the message, Master [02:53:13] bblack, around? [03:01:31] (03PS1) 10Yurik: netmapper URL based on realm (prod/labs) [operations/puppet] - 10https://gerrit.wikimedia.org/r/137856 [03:02:50] bblack, please change the script to use "action=zeroportal" (works in prod too), and I submitted this to work on beta: https://gerrit.wikimedia.org/r/#/c/137856/ [03:13:22] !log LocalisationUpdate completed (1.24wmf8) at 2014-06-06 03:12:19+00:00 [03:13:26] Logged the message, Master [03:19:11] (03PS2) 10Ori.livneh: Puppetize /home/tstarling/.bashrc [operations/puppet] - 10https://gerrit.wikimedia.org/r/76678 (owner: 10Tim Starling) [03:19:38] (03CR) 10Ori.livneh: "Updated for Chase's new admin module pattern" [operations/puppet] - 10https://gerrit.wikimedia.org/r/76678 (owner: 10Tim Starling) [04:14:40] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Thu 05 Jun 2014 19:12:41 UTC [04:24:15] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 6 04:23:08 UTC 2014 (duration 23m 7s) [04:24:20] Logged the message, Master [04:35:37] (03PS1) 10Ori.livneh: Add my dotfiles [operations/puppet] - 10https://gerrit.wikimedia.org/r/137864 [04:37:23] ^ +1 ? [04:39:40] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [05:29:25] got akosiaris ? [05:52:30] hola cajoel [05:52:39] nighto [05:53:23] cloudy day at the acropolis [05:53:24] http://www.greece-athens.com/webcamera/acropolis.php [05:54:58] jgage: found from lsof that current active TCP sessions are initiated from inside the NAT [05:55:08] so I also need to PBR to the same ports outbound [05:55:12] but that's an easy fix [05:55:21] all ready to go [05:56:30] oh interesting [05:57:37] aside: Do you know anyone at deviantArt? [05:57:44] I know Chase worked there [05:57:56] one of our interns is interested in applying there -- IT [05:59:03] we have two other people on staff from DA i believe, but i don't personally know anyone there [06:00:40] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 03:00:13 UTC [06:04:44] i'm on the verge of zzz. are you planning to wait for akosiaris for that change? [06:05:00] i am here [06:05:08] good evening :) [06:05:46] hi hi [06:06:09] if you have some time, take a look at my notes? [06:06:48] would also be nice if you could lsof on sanger [06:06:54] wait, maybe I can login to sanger.. [06:07:16] hot damn -- I can [06:07:58] looking at the notes right now [06:08:48] java 1276 opendj 78u IPv6 5665 0t0 TCP sanger.wikimedia.org:41461->sanger.wikimedia.org:8989 (ESTABLISHED) [06:08:48] java 1276 opendj 79u IPv6 5655 0t0 TCP sanger.wikimedia.org:41454->sanger.wikimedia.org:8989 (ESTABLISHED) [06:08:48] java 1276 opendj 80u IPv6 48442688 0t0 TCP sanger.wikimedia.org:8989->216.38.130.189:44754 (ESTABLISHED) [06:08:48] java 1276 opendj 82u IPv6 48462983 0t0 TCP sanger.wikimedia.org:8989->sfo-intranet.corp.wikimedia.org:33394 (ESTABLISHED) [06:08:48] java 1276 opendj 83u IPv6 48442686 0t0 TCP sanger.wikimedia.org:8989->sfo-intranet.corp.wikimedia.org:33374 (ESTABLISHED) [06:08:57] connections are being initiated from SF [06:09:14] (03CR) 10Ori.livneh: "Giuseppe, I think you're right. I'll amend the change to undo that particular part." [operations/puppet] - 10https://gerrit.wikimedia.org/r/137470 (owner: 10Ori.livneh) [06:09:26] might work both ways, but current open ports are initiated on sf side [06:10:33] ok so my only point is for WAN in ${WANLIST} [06:10:35] I don't see WANLIST defined somewhere in that script [06:10:44] yeah, it's much higher [06:10:49] eth2 eth8 eth 9 etc.. [06:10:50] ah ok [06:10:59] I use it for other parts of the script [06:11:16] seems fine then [06:11:25] spent some time reading iptables, and it looks like you can specify multiple intefaces in one go, but that's for another time [06:11:39] I have James Alexandar online too [06:11:47] sugarcrm uses LDAP via labs [06:12:11] it uses dns, but he's ready to kick apache if it doesn't clean up [06:12:15] the labs LDAP is completely separate though [06:12:16] !log on osmium installed nodejs for testing [06:12:21] Logged the message, Master [06:12:26] (03PS8) 10Mwalker: Adding GELF support to LogStash [operations/puppet] - 10https://gerrit.wikimedia.org/r/137811 [06:12:35] I'm going to grab a glass of water and come back and get started. [06:12:39] ok [06:12:50] akosiaris: it's an instance running on labs that talks to OIT LDAP [06:12:53] for user auth [06:13:01] ah ok [06:13:07] gelf support for logstash! [06:13:14] yep; it now works too [06:13:20] just tested that final revision [06:14:21] i have recently done the same thing, for hadoop. nice to see your use of filters. [06:15:58] akosiaris: ok, ready to roll [06:16:08] ok [06:16:13] one at a time or both at once? [06:16:31] one at a time ? [06:16:58] jgage, how did you make it HA? we were considering LVS + pybal [06:17:00] ok, sfo-intranet1 first [06:17:59] first dns [06:18:03] (03CR) 10Mwalker: "Now tested in labs -- see https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/gelf_test" [operations/puppet] - 10https://gerrit.wikimedia.org/r/137811 (owner: 10Mwalker) [06:18:56] jgage, also if you want to +2 and puppet merge that would be really cool [06:19:10] (03PS3) 10Ori.livneh: mediawiki::php: restore php5-igbinary; delete some unused files [operations/puppet] - 10https://gerrit.wikimedia.org/r/137470 [06:20:06] dns looks ok to me [06:20:08] you? [06:20:20] ;; ANSWER SECTION: [06:20:20] sfo-intranet1.corp.wikimedia.org. 300 IN A 198.73.209.13 [06:20:27] (03PS1) 10Ori.livneh: remove unused files in files/apache [operations/puppet] - 10https://gerrit.wikimedia.org/r/137883 [06:20:29] mwalker, i haven't done anything with HA so far [06:20:41] mwalker, looks good. gonna +2 and merge. [06:20:42] I just saw the change in your ns, the recursors have not picked it up yet [06:20:47] (03CR) 10Gage: [C: 032] "Cool! I was about to make a nearly-identical change." [operations/puppet] - 10https://gerrit.wikimedia.org/r/137811 (owner: 10Mwalker) [06:21:13] 71 secs left, not purging, will wait it out [06:21:23] akosiaris: i'm happy to see any work on the apache pile-o'-crap, plenty of mess for everyone to get some action :) [06:21:36] akosiaris: I was seeing it on 4.2.2.1 [06:21:36] dig @4.2.2.1 sfo-intranet1.corp.wikimedia.org [06:21:48] mwalker, done! [06:21:53] whooo! [06:21:54] ori: yes indeed. let's clear that mess please :) [06:21:55] thanks :) [06:22:01] :D [06:22:19] ori: got a plan ? I just got started to try and solve some puppet issues on zirconium [06:22:30] but I am up for bigger changes [06:22:47] akosiaris: making PBR changes [06:22:49] cajoel: ok it changed on the recursors too [06:23:00] akosiaris: well, i don't think we're worried about keeping up with upstream, so i figure maybe start by removing all the pieces of that module that we aren't using [06:23:58] akosiaris: i don't really have a final architecture in mind, i usually get a handle on the problem by doing small lint-type commits [06:24:03] ori: ok so some housecleaning first. That reminds me of the scap approach. I am up for it [06:24:11] yep [06:24:42] that is mostly why I started it anyway. To understand what the @@#$ is going on with that set of classes [06:25:33] cajoel: SSL connection attempt from 198.73.209.2 (198.73.209.2) failed: Received close_notify during handshake [06:25:36] is that you ? [06:25:41] akosiaris: [05/Jun/2014:23:23:29 -0700] category=SYNC severity=SEVERE_ERROR msgID=14942389 msg=Replication Server 12440 sanger:8989 dc=corp,dc=wikimedia,dc=org has badly disconnected from this replication server 30547 [06:25:53] yeah, that's one of the IPs [06:26:28] ok, looks like opendj hasn't pick up yet the DNS change [06:26:32] I will restart my end [06:27:13] ok [06:28:09] Could not connect to any replication server on suffix dc=corp,dc=wikimedia,dc=org among the following RS candidates {12440=Url:sanger:8989 ServerId:12440, 12701=Url:sfo-aaa1.corp.wikimedia.org:8989 ServerId:12701, 30547=Url:sfo-intranet1:8989 ServerId:30547}, retrying... [06:28:19] it is complaining about the second one, not the intranet1 one [06:28:45] odd [06:28:59] checking rules [06:29:28] [06/Jun/2014:06:27:31 +0000] category=SYNC severity=NOTICE msgID=15138878 msg=Replication is up and running for domain cn=schema with replication server id 12701 sfo-aaa1.corp.wikimedia.org/216.38.130.188:8989 - local server id is 1243 - data generation is 8408 [06:29:35] ok so the cn=schema repl is working [06:29:44] it only complained about the actual tree [06:29:53] weird [06:29:54] port is open [06:30:00] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Fri Jun 6 06:29:55 UTC 2014 [06:30:18] shall we try a data change with the half brain? [06:30:23] or do the full conversion now? [06:31:35] ok it recovered so let's go on [06:31:52] ok [06:31:58] next dns [06:33:24] applied [06:33:31] 4.2.2.2 seeing it [06:33:57] (03PS1) 10Ori.livneh: ::apache: delete everything we're not already using [operations/puppet] - 10https://gerrit.wikimedia.org/r/137884 [06:35:19] yup it changed on the resolvers too [06:35:29] (03PS2) 10Ori.livneh: ::apache: delete everything we're not already using [operations/puppet] - 10https://gerrit.wikimedia.org/r/137884 [06:37:41] [05/Jun/2014:23:29:45 -0700] category=SYNC severity=NOTICE msgID=15138921 msg=SSL connection attempt from sanger.wikimedia.org (208.80.152.187) failed: Unrecognized SSL message, plaintext connection? [06:39:36] cajoel: yes it needed again a restart [06:39:44] it did not complain at all now [06:40:42] ok [06:40:53] want to do a app level test? [06:41:10] sure [06:41:36] changed my title [06:41:50] are you hitting queries on the command line? [06:41:57] on sanger? [06:42:34] I am dumping the entire ldif [06:42:55] cli ? [06:43:20] title: System and Network Engineer [06:43:31] ds-sync-hist: title:000001466fe935f157ef000044f8:repl:System and Network Engineer [06:43:47] so we are ok it seems, right ? [06:44:10] yes indeed [06:44:16] sudo -u opendj /usr/opendj/bin/export-ldif -l lala.ldif -n userRoot [06:44:26] if you are interested in the command [06:44:33] yes, please and thanks [06:45:06] well sweet [06:47:19] ok [06:47:28] confirmed that sugar+labs is functional [06:47:39] failed once and then connected (didn't need to kick apache) [06:47:53] _joe_: https://gerrit.wikimedia.org/r/#/c/137470/ <-- updated to remove the recursive dir, per yr suggestion [06:48:04] cajoel: cool. Seems like you can go to bed then ? [06:48:09] akosiaris: I think so! [06:48:18] <_joe_> ori: great [06:48:34] gotta be at work in 8.5 hours for a fireside chat! [06:48:37] <_joe_> sorry to be a pain sometimes, but those are paths I already went down in the past [06:48:45] akosiaris: thanks for your help [06:48:58] _joe_: i wouldn't have taken the advice if i didn't end up agreeing [06:49:04] cajoel: you are welcome. Have a nice night :-) [06:49:17] next is the pbx [06:49:23] do you have a sip phone? [06:49:27] I know faidon does [06:49:42] not tonight -- next week [06:49:48] yes I do [06:50:00] are you using dns names or ips? [06:50:06] dns names [06:50:13] asterisk also sends that helper address thingy [06:50:19] hardcoded in the asterisk config [06:50:58] externalIP in sip.conf [06:51:00] something like that [06:51:04] some other night [06:51:07] cheers! [06:51:13] cheers! [06:51:22] nice job dudes [06:52:10] thx gage [06:52:16] happy ulfso touring tomorrow [06:52:27] jgage: still up ? thanks :-) [06:53:40] (03PS1) 10Giuseppe Lavagetto: puppetmaster: acls only in apache for puppet 3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/137886 [06:53:56] akosiaris: sent a couple of patches your way. i'm permitted to merge changes that get a +1 from opsen, so if any look sane, i can shepherd them into prod and make sure they don't break [06:54:04] hi alex :) sure, happy to stick around and observe [06:55:31] ori: I can merge them too if you feel like going to sleep. I am looking at them now [06:56:02] <_joe_> um gerrit bot where art thou? [06:56:05] akosiaris: happy to have you merge them, but i'll be up either way, have to write up a document [06:56:16] <_joe_> oh I'm blind [07:00:09] (03CR) 10Alexandros Kosiaris: [C: 032] remove unused files in files/apache [operations/puppet] - 10https://gerrit.wikimedia.org/r/137883 (owner: 10Ori.livneh) [07:00:38] PROBLEM - Puppet freshness on mw1058 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 06:57:51 UTC [07:01:00] dur, the dependent patch has a typo [07:02:08] (03PS4) 10Ori.livneh: mediawiki::php: restore php5-igbinary; delete some unused files [operations/puppet] - 10https://gerrit.wikimedia.org/r/137470 [07:02:34] (03CR) 10Alexandros Kosiaris: [C: 032] Add my dotfiles [operations/puppet] - 10https://gerrit.wikimedia.org/r/137864 (owner: 10Ori.livneh) [07:02:38] PROBLEM - Puppet freshness on mw1058 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 06:57:51 UTC [07:03:00] (03PS2) 10Ori.livneh: remove unused files in files/apache [operations/puppet] - 10https://gerrit.wikimedia.org/r/137883 [07:03:18] (03CR) 10Ori.livneh: [C: 032 V: 032] "(rebase)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/137883 (owner: 10Ori.livneh) [07:03:47] puppet's fine on mw1058 [07:03:53] <_joe_> yeah it is [07:03:55] we've been having these false alarms, must be an snmp issue [07:03:56] <_joe_> look at the time [07:04:15] <_joe_> Last successful Puppet run was Fri 06 Jun 2014 06:57:51 UTC [07:04:22] <_joe_> so 7 minutes ago [07:04:28] <_joe_> I do hate that check [07:04:38] PROBLEM - Puppet freshness on mw1058 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 06:57:51 UTC [07:05:57] <_joe_> shut up you icinga [07:06:13] akosiaris: thank you [07:06:38] PROBLEM - Puppet freshness on mw1058 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 06:57:51 UTC [07:06:58] ah yes, that crappy thing [07:07:30] so, icinga gets the snmp fine, turns to OK and then again CRITICAL within minutes [07:07:55] <_joe_> yeah we need to change this [07:07:59] the one way I found out on how to solve it is to disable notifications, wait a bit, submit a passive ok check, wait some more time and then reenable notifications [07:08:09] <_joe_> grr [07:08:17] but I hate that check so much... [07:08:31] <_joe_> akosiaris: or, we can write the timestamp and the exit status of the puppet run to a file with a wrapper [07:08:37] <_joe_> and check that via nrpe [07:08:38] PROBLEM - Puppet freshness on mw1058 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 06:57:51 UTC [07:08:41] <_joe_> or whatever [07:08:59] <_joe_> that would work a LOT better, and won't tamper with manual runs [07:09:00] so I have had a python script almost ready, then some creeping featuritis came in through the window and I 've never finished it [07:09:09] <_joe_> :) [07:09:27] I am gonna go back and revisit that thing while closing the window at the same time [07:09:28] <_joe_> featuritis is a bad disease, I almost always suffer from it [07:10:38] PROBLEM - Puppet freshness on mw1058 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 06:57:51 UTC [07:10:49] (03PS1) 10Giuseppe Lavagetto: puppet: ensure defining $puppet_version does upgrade [operations/puppet] - 10https://gerrit.wikimedia.org/r/137889 [07:12:12] <_joe_> akosiaris: can I ask this couple of CRs? [07:12:32] <_joe_> when those are done, we are officially puppet3-enabled :) [07:12:38] PROBLEM - Puppet freshness on mw1058 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 06:57:51 UTC [07:14:38] PROBLEM - Puppet freshness on mw1058 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 06:57:51 UTC [07:15:17] ok [07:15:34] <_joe_> and I trust you as a reviewer :P [07:15:38] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Thu 05 Jun 2014 19:12:41 UTC [07:16:36] <_joe_> yeah that, I must ack [07:16:38] PROBLEM - Puppet freshness on mw1058 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 06:57:51 UTC [07:17:25] ACKNOWLEDGEMENT - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Thu 05 Jun 2014 19:12:41 UTC Giuseppe Lavagetto working on puppet 3 migration. [07:18:38] PROBLEM - Puppet freshness on mw1058 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 06:57:51 UTC [07:20:38] PROBLEM - Puppet freshness on mw1058 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 06:57:51 UTC [07:22:38] PROBLEM - Puppet freshness on mw1058 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 06:57:51 UTC [07:24:38] PROBLEM - Puppet freshness on mw1058 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 06:57:51 UTC [07:26:38] PROBLEM - Puppet freshness on mw1058 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 06:57:51 UTC [07:27:28] RECOVERY - Puppet freshness on mw1058 is OK: puppet ran at Fri Jun 6 07:27:27 UTC 2014 [07:29:38] PROBLEM - Puppet freshness on mw1058 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 07:27:27 UTC [07:30:37] (03CR) 10Alexandros Kosiaris: [C: 04-1] puppetmaster: acls only in apache for puppet 3 (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/137886 (owner: 10Giuseppe Lavagetto) [07:30:47] (03CR) 10Alexandros Kosiaris: [C: 032] puppet: ensure defining $puppet_version does upgrade [operations/puppet] - 10https://gerrit.wikimedia.org/r/137889 (owner: 10Giuseppe Lavagetto) [07:35:36] <_joe_> akosiaris: I did not think puppetmaster::passenger had any use outside of the puppetmaster class :) [07:35:55] (03CR) 10Giuseppe Lavagetto: "I agree completely." (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/137886 (owner: 10Giuseppe Lavagetto) [07:40:03] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [07:40:27] all this puppet freshness is giving me headaches [07:40:27] and I have and reviews today [07:40:41] <_joe_> oh the performance review process [07:40:51] <_joe_> I did not understand if I should participate [07:40:55] <_joe_> or how [07:41:06] <_joe_> I'm lazy as hell with those things [07:41:23] <_joe_> (still need to ask for the reimbursement for Athens) [07:42:17] ask then. I would expect you to not but don't take my word for it [07:46:34] there was a wmfall email about reimbursements before end of fiscal [07:46:56] _joe_: might want to be quick :) [07:48:20] <_joe_> when is the end of fiscal? [07:48:31] <_joe_> god I'm a trainwreck [07:48:33] june 30 i guess [07:48:37] <_joe_> oh ok [07:48:42] <_joe_> "I have time" [07:48:43] <_joe_> :P [07:48:46] heh [07:49:03] <_joe_> why waste time earning money when I can play with tech? [07:51:39] (03PS2) 10Giuseppe Lavagetto: puppetmaster: acls only in apache for puppet 3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/137886 [07:54:08] (03PS3) 10Giuseppe Lavagetto: puppetmaster: acls only in apache for puppet 3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/137886 [07:55:46] (03PS4) 10Giuseppe Lavagetto: puppetmaster: acls only in apache for puppet 3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/137886 [07:55:59] (03CR) 10jenkins-bot: [V: 04-1] puppetmaster: acls only in apache for puppet 3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/137886 (owner: 10Giuseppe Lavagetto) [07:57:15] <_joe_> wat? [07:57:24] <_joe_> that was after the rebase [07:57:33] RECOVERY - Puppet freshness on mw1058 is OK: puppet ran at Fri Jun 6 07:57:31 UTC 2014 [07:59:51] (03PS5) 10Giuseppe Lavagetto: puppetmaster: acls only in apache for puppet 3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/137886 [08:01:48] (03CR) 10Giuseppe Lavagetto: [C: 032] puppetmaster: acls only in apache for puppet 3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/137886 (owner: 10Giuseppe Lavagetto) [08:03:10] (03PS1) 10Yurik: LABS: set zerobanner to use lab's zerowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/137892 [08:05:15] good i can't remember [08:08:13] RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Fri Jun 6 08:08:02 UTC 2014 [08:08:58] <_joe_> \o/ [08:09:03] <_joe_> |o/ [08:10:21] (03CR) 10Yurik: [C: 032] "Another prod noop, updating zero config for labs." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/137892 (owner: 10Yurik) [08:10:27] (03Merged) 10jenkins-bot: LABS: set zerobanner to use lab's zerowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/137892 (owner: 10Yurik) [08:11:46] Just +2 my own wmf-config patch - only touches labs files ^ [08:12:16] no production sync (unless there are ops here who say its ok to do sync-dir on wmf-config at this time) [08:13:59] (03PS2) 10Giuseppe Lavagetto: puppet: ensure defining $puppet_version does upgrade [operations/puppet] - 10https://gerrit.wikimedia.org/r/137889 [08:21:38] (03CR) 10Alexandros Kosiaris: [C: 04-1] "Two files still actively referenced in the tree. The rest LGTM" (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/137470 (owner: 10Ori.livneh) [08:21:40] yurik: +1 :D [08:22:36] hashar, you think its a good idea to do sync-dir or even scap now? :) [08:23:14] yurik: when I do change to the -labs.php files I just rebase the repo on tine [08:23:19] tin [08:23:30] I don't bother running scap since that is a noop for prod [08:24:13] (03PS1) 10Giuseppe Lavagetto: puppet3: correct resource title for removal [operations/puppet] - 10https://gerrit.wikimedia.org/r/137893 [08:24:25] hashar, you mean i should just run git pull in wmf-config dir on tin? [08:25:13] yurik: git fetch then compare current head with whatever you fetched: git log HEAD..FETCH_HEAD [08:25:17] or something like that [08:25:21] then git rebase if you are happy :D [08:25:39] i haven't used git rebase without params [08:25:53] i.e. make sure you are not going to bring in another change that might have been merged but remains undeployed [08:25:56] (03CR) 10Giuseppe Lavagetto: [C: 032] puppet3: correct resource title for removal [operations/puppet] - 10https://gerrit.wikimedia.org/r/137893 (owner: 10Giuseppe Lavagetto) [08:26:21] (03PS5) 10Ori.livneh: mediawiki::php: restore php5-igbinary; delete some unused files [operations/puppet] - 10https://gerrit.wikimedia.org/r/137470 [08:26:50] <_joe_> ow, how stupid I am [08:26:52] akosiaris: ^ amended. we don't have any lucid snapshot hosts, so modules/snapshot/manifests/phpfiles.pp could be killed [08:27:08] ori: yes it could :-) [08:27:10] hashar, right, gotcha [08:27:16] akosiaris: good catch, thanks for that [08:29:21] (03PS1) 10Giuseppe Lavagetto: puppet3: remove 2.7 prefs, not the actual ones [operations/puppet] - 10https://gerrit.wikimedia.org/r/137895 [08:30:00] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] puppet3: remove 2.7 prefs, not the actual ones [operations/puppet] - 10https://gerrit.wikimedia.org/r/137895 (owner: 10Giuseppe Lavagetto) [08:35:39] ms100{1,2,4} ? what are those machines ? [08:35:53] site.pp is not really forthcoming [08:36:21] "maybe something" [08:37:02] I sure hope you are joking [08:37:58] cause they are running nginx and HTCPpurger and are supposedly serving thumbs ? [08:38:33] i am joking :) i don't know what they are [08:38:48] you had me for a minute there [08:39:21] heh, truth is not much better: per ganglia, ms is "miscellaneous" [08:40:09] sigh [08:40:27] well they are lucids, which is why I picked them up [08:41:00] I am starting to wonder what will happen if I power them down [08:41:33] (03PS1) 10Giuseppe Lavagetto: labs: default all clients to puppet 3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/137898 [08:42:08] (03CR) 10Giuseppe Lavagetto: "I'll leave it to labs people when and if to merge this." [operations/puppet] - 10https://gerrit.wikimedia.org/r/137898 (owner: 10Giuseppe Lavagetto) [08:42:21] maybe something :) [08:42:59] <_joe_> ori: are you still awake? so I can ask you something about rcstream [08:43:02] <_joe_> :) [08:43:03] https://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&title=&vl=&x=&n=&hreg[]=ms100%5B1-9%5D&mreg[]=bytes_%28in%7Cout%29>ype=stack&glegend=show&aggregate=1&embed=1&_=1402044083307 [08:43:12] I will be damn if those machines actually do anything [08:43:13] _joe_: what's up? [08:43:34] _joe_: You must be new here; ori goes to bed at 11 AM UTC [08:43:47] * twkozlowski winks at ori [08:43:48] <_joe_> ori: I want to introduce some performance instrumentation, and I was asking myself which could be the best way to do that [08:43:48] twkozlowski: shh, no outing [08:44:14] what do you want to measure? [08:44:15] <_joe_> twkozlowski: I share the pain of high insomnia with ori, so I always want not to keep him awake :) [08:45:13] <_joe_> ori: things like the number of changes dispatched, number of clients per subscription type, "response times" do not make that much sense probably [08:45:55] <_joe_> ori: I was thinking of storing counters in redis, it's easy to make counts per minute using it [08:46:21] are you sure it's not featuritis? i thought about that, but i don't see much value [08:46:32] <_joe_> it probably is :P [08:46:45] <_joe_> it's just that I love metrics [08:47:07] no shortage of things to instrument that really do need more instrumentation :P [08:47:12] <_joe_> but, if the number of changes dispatched drops [08:47:21] <_joe_> there is a problem somewhere [08:47:31] <_joe_> and we could set up an alert on that [08:47:43] <_joe_> I want to have a way to check that easily [08:48:57] mm, i'd prefer to wait for some indication that it is failure-prone [08:49:46] bytes_out in ganglia is probably as good indication as any [08:50:34] hashar, is there a way to specify an extra header in eval.php ? [08:51:04] i couldn't find any doc on eval.php for some reason [08:51:17] yurik: there is no doc for eval.php afaik. Read the source :D [08:52:06] are you trying to do a web request ? [08:52:21] <_joe_> ori: yeah it is [08:52:26] <_joe_> agreed :) [08:53:02] _joe_: the job queue is a black pit of darkness, btw [08:53:09] lots to instrument there [08:53:14] you've already done some good work there [08:53:16] <_joe_> yeah I kinda noticed [08:53:20] <_joe_> :P [08:53:35] <_joe_> ori: first I must give some love to the varnishes stats [08:53:58] (03CR) 10Alexandros Kosiaris: [C: 032] mediawiki::php: restore php5-igbinary; delete some unused files [operations/puppet] - 10https://gerrit.wikimedia.org/r/137470 (owner: 10Ori.livneh) [08:54:02] <_joe_> which is probably what I'm going to do staring today [08:54:37] paravoid: ping [08:55:37] akosiaris: thanks! [08:56:13] ori: thank you as well. [08:56:23] * akosiaris still baffled by ms100X [08:57:48] <_joe_> akosiaris: eheh ghosts form the past [08:58:49] for a moment I was like, heh swift [08:59:09] guess what, swift is not installed [08:59:26] but for some reason 24TB of spaces are used by thumbs [08:59:28] meh... [09:00:01] Is there some trick that I’m missing for making Ubuntu 14.04 work on a Dell 1950? [09:01:04] <_joe_> preilly: a poweredge 1950 seems quite old, maybe some drivers are not in the standard ubuntu kernel anymore? [09:01:14] <_joe_> (I know this is very generic) [09:01:23] 2006 machines ? [09:01:39] _joe_: well the installer works just fine just doesn’t find disk after reboot [09:01:44] 2007 model [09:02:17] <_joe_> preilly: as I said above, you're probably missing some module from the kernel [09:02:57] probably initramfs [09:03:23] well I’m loading megaraid_sas in initramfs [09:04:36] <_joe_> maybe support for that generation of percs has been moved to another module [09:05:03] PROBLEM - DPKG on vanadium is CRITICAL: DPKG CRITICAL dpkg reports broken packages [09:06:03] RECOVERY - DPKG on vanadium is OK: All packages OK [09:15:05] (03CR) 10Hashar: "_joe_ the beta cluster and integration labs projects have their own puppetmaster so we can probably attempt the migration to puppet 3 on b" [operations/puppet] - 10https://gerrit.wikimedia.org/r/137898 (owner: 10Giuseppe Lavagetto) [09:15:53] <_joe_> hashar: at this point, it's really up to labs people [09:16:06] <_joe_> I know too little about labs to make any kind of call [09:16:53] <_joe_> althoug it seems much more time given how much I'm annoying, I'm just here since two months :) [09:16:57] I know to little about puppet 2.7 -> 3.0 migration [09:17:08] <_joe_> hashar: should be seamless [09:17:29] <_joe_> it was seamless (mostly) in all cases where we tried [09:17:46] \O/ [09:17:58] I am going to break the integration labs project so ! [09:18:03] s/break/upgrade/ [09:18:05] <_joe_> it could still break things for individual machines [09:18:11] <_joe_> on friday? [09:18:12] <_joe_> mmmh [09:18:26] it is technically still thursday in hawai [09:18:32] so I can pretend I was working from there [09:18:33] <_joe_> You did not had plans for this weekend? [09:18:45] <_joe_> :P [09:18:54] beside working and the usual on call ? Not much planned :-D [09:18:59] it is ok, it just puppet! [09:21:15] speaking of puppet! I have also https://gerrit.wikimedia.org/r/#/c/136128/ up for review if somebody wants to take a stab :) (mini-dinstall for releases.wikimedia.org) [09:25:25] <_joe_> godog: I do have some time now [09:25:32] <_joe_> I can take a look [09:25:38] thanks _joe_ ! [09:29:09] <_joe_> godog: the first question that comes to mind is: why not reprepro? [09:29:15] <_joe_> :) [09:29:30] <_joe_> oh sorry this is the build pipeline [09:30:19] the main reason is that reprepro doesn't support multiple versions in a suite, however that was a requirement gwicke mentioned [09:32:13] <_joe_> so we're releasing mediawiki and the rest as debs and we want to have multiple versions in a single suite [09:32:56] <_joe_> which is not exactly something apt will care about unless someone pins the version [09:33:20] <_joe_> (another one of the uncountable reasons why debs are a bad way to distribute webapps) [09:33:29] <_joe_> but still, I do see the point [09:33:33] <_joe_> kind of [09:34:01] <_joe_> more software to support/install, yay! [09:39:49] _joe_: will do puppet 3 migration later on. My instances puppet are broken cause of some dupe definition i have to figure out :D [09:42:25] <_joe_> :) ok [10:00:01] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "Some small comments, apart from that LGTM." (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/136128 (owner: 10Filippo Giunchedi) [10:34:03] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 07:33:22 UTC [10:35:09] (03PS1) 10Hashar: mediawiki::packages learned to disable php-apc [operations/puppet] - 10https://gerrit.wikimedia.org/r/137910 [10:41:03] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [10:44:58] (03PS2) 10Hashar: mediawiki::packages learned to disable php-apc [operations/puppet] - 10https://gerrit.wikimedia.org/r/137910 [10:44:59] (03PS1) 10Hashar: Qualify $hostname in network manifest [operations/puppet] - 10https://gerrit.wikimedia.org/r/137911 [10:46:41] _joe_: puppet catalog compiler gave me an error: cp: cannot create regular file `/usr/local/bin/naggen': Permission denied :D [10:46:43] https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/59/console [10:46:56] <_joe_> yeah discard that [10:47:08] <_joe_> it means nothing [10:47:11] <_joe_> :) [10:47:25] :D [10:49:27] (03CR) 10Hashar: "I ran it via the puppet catalog compiler against node mw1042 but that is still failing:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/137911 (owner: 10Hashar) [10:49:51] _joe_: and apparently neither $hostname nor $::hostname is set in puppet http://puppet-compiler.wmflabs.org/change/137911/compiled/puppet_catalogs_2.7_137911/mw1042.warnings [10:49:53] PROBLEM - Disk space on analytics1015 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/f 74307 MB (3% inode=99%): [10:50:36] neither $hostname or $::hostname seems to be set :/ [10:50:37] <_joe_> hashar: probably missing some facts files [10:51:43] (03PS3) 10Hashar: mediawiki::packages learned to disable php-apc [operations/puppet] - 10https://gerrit.wikimedia.org/r/137910 [11:03:33] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Fri Jun 6 11:03:25 UTC 2014 [11:30:54] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:27:58 UTC [11:32:54] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:27:58 UTC [11:34:54] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:27:58 UTC [11:36:54] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:27:58 UTC [11:38:54] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:27:58 UTC [11:41:02] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:27:58 UTC [11:43:02] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:27:58 UTC [11:45:02] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:27:58 UTC [11:47:02] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:27:58 UTC [11:49:02] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:27:58 UTC [11:51:02] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:27:58 UTC [11:53:02] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:27:58 UTC [11:55:02] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:27:58 UTC [11:57:02] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:27:58 UTC [11:57:22] PROBLEM - Disk space on analytics1010 is CRITICAL: DISK CRITICAL - free space: / 10 MB (0% inode=92%): [11:59:02] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:27:58 UTC [11:59:02] RECOVERY - Puppet freshness on analytics1024 is OK: puppet ran at Fri Jun 6 11:58:56 UTC 2014 [11:59:22] RECOVERY - Disk space on analytics1010 is OK: DISK OK [12:01:02] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: Last successful Puppet run was Fri 06 Jun 2014 11:58:56 UTC [12:04:48] (03PS2) 10BBlack: netmapper URL based on realm (prod/labs) [operations/puppet] - 10https://gerrit.wikimedia.org/r/137856 (owner: 10Yurik) [12:04:55] (03CR) 10BBlack: [C: 032 V: 032] netmapper URL based on realm (prod/labs) [operations/puppet] - 10https://gerrit.wikimedia.org/r/137856 (owner: 10Yurik) [12:09:02] (03PS1) 10BBlack: simplify zero update url, remove outdated $zero_realm [operations/puppet] - 10https://gerrit.wikimedia.org/r/137916 [12:10:45] (03CR) 10BBlack: [C: 032 V: 032] simplify zero update url, remove outdated $zero_realm [operations/puppet] - 10https://gerrit.wikimedia.org/r/137916 (owner: 10BBlack) [12:28:35] RECOVERY - Puppet freshness on analytics1024 is OK: puppet ran at Fri Jun 6 12:28:26 UTC 2014 [12:29:54] (03Abandoned) 10Hashar: Qualify $hostname in network manifest [operations/puppet] - 10https://gerrit.wikimedia.org/r/137911 (owner: 10Hashar) [12:30:50] (03PS1) 10BBlack: zerofetch: action name s/zeroconfig/zeroportal/ (per Yuri) [operations/puppet/varnish] - 10https://gerrit.wikimedia.org/r/137919 [12:31:26] (03CR) 10BBlack: [C: 032 V: 032] zerofetch: action name s/zeroconfig/zeroportal/ (per Yuri) [operations/puppet/varnish] - 10https://gerrit.wikimedia.org/r/137919 (owner: 10BBlack) [12:37:26] (03PS1) 10Hashar: contint: reduce duplication with mediawiki::packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/137921 [12:40:44]