[00:01:10] (03CR) 10Dzahn: [C: 032] dsh: split the misc-servers into dc-specific files [operations/puppet] - 10https://gerrit.wikimedia.org/r/92599 (owner: 10Dzahn) [00:08:30] PROBLEM - twemproxy process on arsenic is CRITICAL: PROCS CRITICAL: 0 processes with UID = 65534 (nobody), command name nutcracker [00:11:50] PROBLEM - Host wikibooks-lb.pmtpa.wikimedia.org_ipv6 is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:860:ed1a::4 [00:12:11] RECOVERY - Host wikibooks-lb.pmtpa.wikimedia.org_ipv6 is UP: PING WARNING - Packet loss = 37%, RTA = 31.29 ms [00:12:29] Hey opsen, does anyone know why I still do not get SMS pages despite getting email pages? [00:12:47] hrmm, lemme take look at contact file [00:14:06] RoanKattouw: you are just set for notify by email [00:14:08] not sms [00:14:14] but i can change it now to both (like most ops) [00:14:16] sound good? [00:14:20] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [00:14:38] I thought that's how I was supposed to have been set up ... [00:14:39] heh [00:14:40] nope! [00:14:42] edits on https://wikitech.wikimedia.org/wiki/Tampa_cluster appreciated. we have moved more stuff than it says, i will also get back to editing it, but if you know about something, please go ahead and edit [00:14:47] so you want it changed right? [00:15:20] Yeah get me SMS pages [00:15:24] Do I have off hours configured? [00:16:03] yep [00:16:07] yer set for pdt awake hours [00:16:16] so it will email then but not page. [00:16:27] What are "awake hours" exactly? [00:16:55] Like, 8am-midnight or something? [00:17:06] its in public repo icinga [00:17:07] uhh [00:17:18] alias PDT 8 am till midnight, 7 days a week [00:17:19] yep [00:17:20] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:17:23] except its gmt [00:17:30] so its shuffled for daylight savings no? [00:19:18] What's the UTC offset they used? [00:19:35] Did they do 1500-0700 or 1600-0800 UTC? [00:19:59] RoanKattouw: you can have custom hours if you want to, it's public puppet files/icinga/timeperiods.cfg [00:20:23] 15 alias CET 8 am till midnight, 7 days a week 16 sunday 07:00-23:00 [00:20:27] Alright, was just wondering [00:20:48] Thanks for giving me the path [00:21:01] RobH: When I travel to a different timezone, do I notify someone to change my config? [00:21:54] Hah, it's 1600-0800, so the name is a lie, it's PST [00:22:41] RoanKattouw: you are the special case who gets paged but can't change it self-service, so i guess the answer is yes [00:22:48] RoanKattouw: its public repo [00:22:57] oh wait, its not [00:22:58] hrmmm [00:23:01] EST and EDT_early are correctly labeled, but I'm amused that we have no straight EDT or EST_early :) [00:23:04] it's public to define a new timezone [00:23:07] schedule is, but yea [00:23:12] it's not public to attach a timezone to a contact [00:23:14] but timezone is set in private file [00:23:15] RobH: The TZ defs are public so I can fix those [00:23:25] so while they can be added, assignment needs ops [00:23:32] Right [00:23:37] or ops can decide roan can do it himself for his own contact only [00:23:41] which im pretty ok with. [00:23:42] Shall I just drop a note to ops@ when I travel? [00:23:54] yea, atleast for the first time [00:23:55] RoanKattouw: they will appear a bit random because the way they have been added was pretty much everybody added the one they wanted for their contact :) [00:24:03] then i bet everyone just agrees for you to change your own then ;] [00:24:07] so feel free to have a better one [00:24:40] Yeah I figured that the reason we have EDT_early but not PDT_early is because there is one early bird person and they're back east [00:24:45] and yea, ops-requests "switch roan to roan_pdt" would be fine [00:24:59] We also don't have any Australia configs so I guess that means Tim is 24x7? [00:25:37] hmm, but i once used Australia too when i was in Darwin [00:25:52] maybe removed in some cleanup [00:26:00] if it wasnt applied to anyone it may have been [00:26:13] would be nice to define them in a bit better naming and order. [00:26:31] a little more systematic, yep [00:26:32] I'm already working on that a bit [00:26:37] cool [00:26:54] Just for the existing ones [00:27:56] Renaming PDT to PST because that's what it is, adding real PDT, adding EDT_early and EST to complement EST and EDT_early [00:28:04] Should probably add CEST as well [00:28:21] Also, this means our configs will now actually be DST-aware and we'll have to switch everyone's next week [00:29:05] i'm pretty sure we had more zones in the past .. [00:29:13] nice [00:29:19] i bet the pdt was right at one part [00:29:28] but then it changed when daylight ended [00:29:37] and whoever changed it was easier to change there than on every contact [00:29:43] * RobH did not do it, but that makes sense to him [00:32:58] Right [00:33:38] So DST ends on Sunday, so apparently no one minded their paging hours having been off by an hour since like March [00:34:15] But once my change is merged, it'll fix the definition of "PDT" to actually be correct, and so if it lands after Sunday we'll want to change PDT contacts to PST [00:35:58] (03CR) 10Legoktm: "Caused bug 56358." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90934 (owner: 10Amire80) [00:38:07] off for food, ttyl (or on a gerrit change) [00:39:21] RobH: Hey quick question (since mut ante just left): those aliases like "alias PDT 8 am till midnight, 7 days a week", are they ever used in the config file, or is it only the timeperiod_name like PDT_awake_hours? [00:39:37] I ask because some of the aliases have typos in them and I don't want to break anyone's config by fixing them [00:39:58] yea aliases are not used that i can see [00:40:00] all like host_notification_period PDT_awake_hours [00:40:24] yea, no aliases used [00:40:36] i suppose that was pulled for info elsewhere at some point, dunno. [00:40:40] OK [00:41:03] you should be able to treat that just like a comment here [00:41:10] /aways [00:44:47] (03PS1) 10Catrope: Clean up Icinga timezone definitions [operations/puppet] - 10https://gerrit.wikimedia.org/r/92604 [00:49:16] (03CR) 10RobH: [C: 031] "looks right to me, i just don't want to merge something and ensure it doesn't break when its this late in day." [operations/puppet] - 10https://gerrit.wikimedia.org/r/92604 (owner: 10Catrope) [00:54:04] (03CR) 10Catrope: "Yeah let's not merge changes to paging right before dinnertime :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/92604 (owner: 10Catrope) [00:56:30] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: No successful Puppet run in the last 10 hours [00:58:36] (03CR) 10Bsitu: "The span should be defined in HTML template inside email formatter" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90934 (owner: 10Amire80) [01:20:21] (03CR) 10Bsitu: "If this can not be defined inside the template, then we need to strip out the span tag for text version" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90934 (owner: 10Amire80) [01:32:03] (03PS1) 10Reedy: Remove codereview specific config file, collaps into CommonSettings.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92605 [01:36:50] (03PS1) 10Reedy: Remove $wgSubversionProxy [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92606 [09:53:24] (03PS1) 10ArielGlenn: remove dup wtp1003 entry [operations/dns] - 10https://gerrit.wikimedia.org/r/92622 [09:56:45] !log jenkins migrated mediawiki-core jobs from PHPUnit 3.7.24 to 3.7.28 {{gerrit|I8651f2fb0e09d4e868c2139b5a7d1b640de61784}} [09:57:44] paravoid: my sponsor is Michael Prokop (mika), he wrote jenkins-debian-glue a bunch of shell script to easily create package with Jenkins [09:58:00] yeah I know him [09:58:00] paravoid: so he did review, caught a few issues like description / copyright [09:58:12] and tried out some a rebuild on unstable :-] [09:58:33] he's the grml author [09:59:49] (03CR) 10ArielGlenn: [C: 032] remove dup wtp1003 entry [operations/dns] - 10https://gerrit.wikimedia.org/r/92622 (owner: 10ArielGlenn) [10:01:01] paravoid: and he is willing to setup a Gerrit/Zuul system as well :D [10:06:21] !log jenkins migrated mediawiki jobs from PHPUnit 3.7.24 to 3.7.28 {{gerrit|Ib8eb320ec5c2e9f8332f09c50e06cfed0d7f5fdf}} [10:06:29] bah morebot is dead again [10:07:05] apergos: mind restarting morebots on tools labs? doc at https://wikitech.wikimedia.org/wiki/Morebots#Example:_restart_the_ops_channel_morebot [10:29:09] huh well my attempt to log in over there just failed :-/ [10:29:25] I wonder if I am a member of the project [10:29:42] evry time they move these bots around something like that happens [10:30:13] you know, since this does logging for ops (a production service) maybe it shouldn't live in labs, just sayin... [10:30:59] I have the same feeling :] [10:31:12] though the idea is to offload us to volunteers whenever they can help [10:31:28] I am happy to have volunteers do it, but look... I'm doing it instead [10:31:34] :-/ [10:35:09] hi, we have a varnish package in our git/gerrit, that we have modified from the upstream. What's the better way to apply a patch that was implemented in the upstream? [10:35:59] yurik_: have a look at previous changes? https://gerrit.wikimedia.org/r/#/q/project:operations/debs/varnish,n,z [10:36:15] https://gerrit.wikimedia.org/r/#/c/87347/ [10:36:25] apparently just stick your patch under debian/patches/ [10:36:28] yurik_: debian/patches [10:36:37] "not a big fan of forking" -- a bit too late for that [10:36:54] -plus is a varnish fork anyway [10:37:04] and we need patches on top of -plus too [10:37:05] paravoid: if android was able to de-fork into mainstream kernel, so can we [10:37:11] sigh [10:37:18] life is just not easy, eh? [10:37:23] and they're well aware of our patches [10:38:57] paravoid: are you saying i should just copy/paste the patch into a file in that dir and build? [10:39:15] try it at least? :) [10:39:18] and here i am, trying to use the power of git to merge and do other crazy things :( [10:39:50] its not like i build deb packages everyday (or ever) [10:39:56] might take a few min [10:39:59] esp on win [10:40:00] (03PS1) 10Odder: en.planet - add Wikistaycation blog [operations/puppet] - 10https://gerrit.wikimedia.org/r/92628 [10:40:39] !log restarted morebots (failed to rejoin after netsplit) [10:41:02] Logged the message, Master [10:56:55] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: No successful Puppet run in the last 10 hours [10:59:40] yurik_: what you linked to is a 8 line patch [10:59:54] that seems much more trivial than a 3.0.4 upgrade :) [11:02:39] paravoid: i know, that's why i'm attempting the package build. Currently strugling with automake regenerating everything to 1.11.3, and then complaining about aclocal being 1.11.1 [11:02:45] and later -- lib/libvarnishapi/Makefile.am:35: HAVE_LD_VERSION_SCRIPT does not appear in AM_CONDITIONAL [11:06:54] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [11:08:54] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:56:03] paravoid or mark: when i run from varnish root - ./autogen.sh, ./configure, make, make install, do i need to do anything else to apply the extra patches? [13:00:07] debian/patches you mean? [13:00:20] debian/patches are used when you build the package [13:00:34] if you build manually, they won't be applied, obviously [13:02:52] paravoid: i thought they are included as part of one of the steps above. When should I apply them during the build process, and which tool should i usse? [13:24:10] paravoid: do you know who is admin for RT? [13:25:27] (03PS2) 10Krinkle: Remove old tampa search config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92016 (owner: 10Chad) [13:28:16] (03CR) 10jenkins-bot: [V: 04-1] Remove old tampa search config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92016 (owner: 10Chad) [13:42:50] oh my god [13:54:21] (03CR) 10Lcarr: "Those hosts did exist and were decommissioned. The IPs were temporarily moved to sq50, since they occupied several roles and pybal would " [operations/dns] - 10https://gerrit.wikimedia.org/r/92613 (owner: 10ArielGlenn) [13:54:27] (03CR) 10Lcarr: [C: 032] remove entries for sq87-106, these hosts never existed [operations/dns] - 10https://gerrit.wikimedia.org/r/92613 (owner: 10ArielGlenn) [13:59:46] (03CR) 10Lcarr: "FYI the bug has been fixed in pybal. was weird edge case" [operations/dns] - 10https://gerrit.wikimedia.org/r/92613 (owner: 10ArielGlenn) [14:07:05] has anyone changed gerrit's host key? i got a men-in-the-middle warning [14:07:13] http://pastebin.com/Y6aaudDx [14:07:44] (03PS1) 10Lcarr: decommissioned cp1-10 [operations/dns] - 10https://gerrit.wikimedia.org/r/92642 [14:08:10] hasn't changed as far as i know yurik [14:08:34] lemme see my key i have for that and compare [14:08:48] thx [14:10:29] i have a different fingerprint [14:10:34] lovelly [14:10:34] 1024 dc:e9:68:7b:99:1b:27:d0:f9:fd:ce:6a:2e:bf:92:e1 blah (RSA) [14:10:39] PROBLEM - Host wikidata-lb.pmtpa.wikimedia.org_ipv6 is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:860:ed1a::12 [14:10:41] PROBLEM - Host wikisource-lb.pmtpa.wikimedia.org_ipv6 is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:860:ed1a::5 [14:10:59] RECOVERY - Host wikidata-lb.pmtpa.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 35.43 ms [14:11:02] sorry took me a second, i needed to make the fingerprint from the pub key in my known hosts [14:11:05] LeslieCarr: should we nuke all my access right now just to be safe? [14:11:28] mark: got an opinion on that ? [14:11:29] RECOVERY - Host wikisource-lb.pmtpa.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 35.97 ms [14:11:39] mark will be happy to nuke my access ;) [14:11:45] hey [14:11:49] got pages [14:11:52] what's up? [14:12:18] (03CR) 10Lcarr: [C: 032] decommissioned cp1-10 [operations/dns] - 10https://gerrit.wikimedia.org/r/92642 (owner: 10Lcarr) [14:12:28] meh, pmtpa [14:12:36] paravoid: http://pastebin.com/Y6aaudDx -- is it safe? [14:12:36] and only v6 [14:12:55] did anyone do anything LVS related? [14:13:20] yurik_: huh, interesting [14:13:36] yurik_: as long as you never went through and logged in anyway, there shouldn't be a problem [14:13:53] well, define "logged in" [14:13:59] not like i use a password [14:14:14] ssh stopped you because it looks like a MitM [14:14:21] as long as you didn't override that it should be fine [14:15:15] i haven't, but the q remains if there is an MitM [14:15:35] run a tcptraceroute to that host/port [14:15:35] LeslieCarr: how did you convert known_hosts to fingerprints [14:15:57] ssh-keygen -lf .ssh/known_hosts [14:16:48] or above i had copied that specific key into its own file named "blah" (hence the "blah" in the line) [14:16:58] yurik_: you did paste a different fingerprint on pastebin [14:17:24] dc:e9:68:7b:99:1b:27:d0:f9:fd:ce:6a:2e:bf:92:e1 (the correct one) vs 83:fe:34:4b:16:2c:9e:95:1d:f6:d7:7d:ee:28:03:02 (the one you pasted) [14:17:33] akosiaris: correct [14:17:45] 2048 83:fe:34:4b:16:2c:9e:95:1d:f6:d7:7d:ee:28:03:02 ytterbium.wikimedia.org (RSA) [14:17:49] i didn't paste anything here [14:17:55] is what my known hosts says [14:18:05] 22 vs. 29418 probably? [14:18:52] lol [14:19:09] 1024 dc:e9:68:7b:99:1b:27:d0:f9:fd:ce:6a:2e:bf:92:e1 [208.80.154.81]:29418 (RSA) [14:19:15] yes that must be it. I have the same symptom [14:19:25] hrm hrm hrm [14:19:29] for ssh gerrit instead of ssh -p 29418 gerrit [14:19:32] stupid ports [14:20:32] heh [14:20:34] http://pastebin.com/bEBKRYRx [14:20:39] mark ^ tracert [14:21:00] no you need tcptraceroute, not a normal traceroute [14:21:01] so the old key must be manganese [14:21:13] dc:e9 is the key on port 22, 83:fe is the key on port 29418 (gerrit) [14:21:14] mark, i'm on windows, remember? :) [14:21:31] that's your problem :) [14:21:55] i tried running tcptraceroute on a virt host, but it only traced it to the host machine [14:22:04] paravoid: other way around [14:22:14] mark, forgive me, i thought security was yours :-P [14:22:46] * apergos grabs popcorn [14:22:51] hehe [14:23:08] ok, as i take it, no cornerns, i'm cleaning up my hosts file [14:23:23] :-) [14:27:20] oh I have concerns, just not about MitM attacks at the moment :P [14:28:51] hehe. btw, it allows me to do git pull, but refuses git review [14:28:54] with pubkey [14:29:11] ohh well, i will just reply with the patch :) [14:55:17] hashar: you mentioned getting a review for gear? [14:57:39] paravoid: yup you asked me earlier who was my sponsor for python-gear . [14:57:46] are these in svn? [14:57:49] then disappeared mysteriously :-] [14:57:50] yeah [14:58:00] got reviewed by some folks from the python module team as well [14:58:06] ok [14:59:20] paravoid: here it is http://anonscm.debian.org/viewvc/python-modules/packages/python-gear/trunk/debian/ [14:59:21] :-] [14:59:43] I know [15:00:11] does being listed in the Uploaders: field grant me some specific rights ? [15:00:16] or I should become a DD as well ? [15:00:39] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:00:41] no, uploaders doesn't get you upload rights, confusingly enough [15:01:04] there's an intermediate step before becoming DD though [15:01:07] I got used to be confused, the last encounter was figuring out 'any' versus 'all' [15:01:29] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 1 logical drive(s), 4 physical drive(s) [15:01:34] I am not ready to even apply as a DD, I got a ton of documentation to read still and would like to try out more packages before even thinking about becoming a dd [15:02:00] yeah, I wouldn't suggest you to go through the process at this time either [15:02:07] you can be a DM :) [15:02:33] that is, a Debian Maintainer, which is also different from being a maintainer of a package :P [15:02:47] https://wiki.debian.org/DebianMaintainer [15:03:53] ohhh [15:04:39] so noobie -> DM -> debian new members process -> DD -> social events -> root [15:04:45] that is going to take long [15:04:48] root? [15:04:55] @debian.org \O/ [15:05:07] yeah... no :) [15:05:18] maybe in ten years or so I will have escalated through the whole process to help with sysadmin [15:06:10] * jeremyb makes a hashar for DSA sign and puts it on a stick [15:06:30] in my closet :) [15:06:49] then I will be able to watch you poop, unlike the NSA [15:06:58] (hopefully the NSA doesn't have yet a webcam in our closets) [15:07:09] i said closet! i don't have a toilet in there [15:07:49] hashar: "social events" are not before DD though :P [15:09:27] hashar: DSA has multiple meanings btw. https://wiki.debian.org/Glossary#dsa (and it's an algorithm too) [15:09:51] jeremyb: in France we call the toilet "water closest" [15:10:04] jeremyb: more exactly we refer to them either by "toilettes" or "WC3 [15:10:06] WC [15:10:39] hashar: i meant storage area. so it can be stored until you're actually running :) [15:11:01] paravoid: so DM is essentially the same as a DD but on a restricted set of packages. Sounds good. [15:11:23] no voting for DM [15:11:30] what jeremyb said [15:11:44] it's an important difference [15:11:47] also DD comes in two flavors. uploading and non-uploading [15:12:59] does DM have porterbox or other machine access? alioth non -guest? email alias? [15:13:30] http://lists.debian.org/debian-newmaint/2006/02/msg00012.html :D [15:14:16] default access to alioth resources like collab-maint? [15:14:43] hah hashar [15:15:05] hashar: time flies [15:15:20] hashar: https://nm.debian.org/public/person/paravoid [15:15:27] can we stop talking about me? :P [15:15:43] well that explain the process [15:16:00] it's a lot simpler nowadays [15:16:07] it took me exactly a year [15:16:14] and it was one of the short ones back then [15:16:24] of the very few short ones [15:16:29] a year after I applied to become a DD that is [15:16:34] well there was a time when it was even shorter [15:16:56] some people in NYC have been DD since last century [15:26:02] jeremyb: that nm page is nice, we could generate a genealogy of the DDs [15:26:40] hashar: sure :) [15:40:25] !log Broke jenkins for a couple hours while doing some path migration. All jobs got updated in the process and failed jobs got retriggered. [15:40:39] Logged the message, Master [16:06:33] (03PS3) 10Hashar: Remove old tampa search config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92016 (owner: 10Chad) [16:11:33] <^d> mutante: Ping [16:13:29] So my VPS provider gave me 18446744073709551616 IPv6 addresses to play with. [16:14:09] And they also changed the way the addresses are added to the machine. Anyone who wants to help get the new translatewiki.net server IPv6-ed? [16:15:05] <^d> siebrand: Can I have 1817466888 ± 9551616 to use then? :p [16:15:35] ^d: ehr... I guess so (?) [16:16:01] <^d> I figured if you had 18446744073709551616 you could spare a few :p [16:18:11] so that's a /64 [16:18:29] i needed some help from bc :P [16:31:44] (03PS1) 10Dzahn: dsh cleanup. these servers are decom in racktables RT #6123 [operations/puppet] - 10https://gerrit.wikimedia.org/r/92660 [16:37:04] PROBLEM - Backend Squid HTTP on amssq32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:14] PROBLEM - Frontend Squid HTTP on amssq32 is CRITICAL: Connection timed out [16:37:55] RECOVERY - Backend Squid HTTP on amssq32 is OK: HTTP OK: HTTP/1.0 200 OK - 1423 bytes in 0.453 second response time [16:38:04] RECOVERY - Frontend Squid HTTP on amssq32 is OK: HTTP OK: HTTP/1.0 200 OK - 1408 bytes in 0.741 second response time [16:38:32] (03PS1) 10Dzahn: add arsenic to dsh groups, treat like terbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/92662 [16:39:07] (03PS2) 10Dzahn: add arsenic to dsh groups, treat like terbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/92662 [16:40:27] (03CR) 10Chad: [C: 031] add arsenic to dsh groups, treat like terbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/92662 (owner: 10Dzahn) [16:40:37] (03CR) 10Dzahn: [C: 032] dsh cleanup. these servers are decom in racktables RT #6123 [operations/puppet] - 10https://gerrit.wikimedia.org/r/92660 (owner: 10Dzahn) [16:41:07] (03CR) 10Dzahn: [C: 032] add arsenic to dsh groups, treat like terbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/92662 (owner: 10Dzahn) [16:41:41] ^d: waiting for verified [16:42:04] ^d: other thing i wonder, shouldn't terbium and arsenic both be in 'apaches' as well [16:42:19] but terbium wasn't either [16:42:37] that would make sync-apache work on them [16:42:51] <^d> But they don't need it? [16:43:03] <^d> They're not part of the apache pool, they're only meant for crons & one-off tasks. [16:43:15] ok, alright then [16:43:36] just cause you said 09:36 <^d> So /usr/local/apache/common/ isn't setup right [16:43:45] and that would be synced by that [16:44:10] but yep, it's like terbium, so ok [16:44:46] no jenkins? [16:45:16] <^d> Just finished running, just slow in responding it seems. [16:45:33] <^d> nvm, that was for 92660 [16:46:40] <^d> Hmm :\ [16:46:48] (03CR) 10Chad: "recheck" [operations/puppet] - 10https://gerrit.wikimedia.org/r/92662 (owner: 10Dzahn) [16:47:14] (03CR) 10Dzahn: "recheck" [operations/puppet] - 10https://gerrit.wikimedia.org/r/92660 (owner: 10Dzahn) [16:47:33] <^d> 92660 seems to have run, but not responded yet :\ [16:49:40] (03PS1) 10Andrew Bogott: Switch to using uwsgi for the proxy api [operations/puppet] - 10https://gerrit.wikimedia.org/r/92664 [16:55:53] <^d> mutante: I guess just go ahead and verify yourself? Jenkins isn't going to check the dsh files anyway :p [16:57:02] (03CR) 10jenkins-bot: [V: 04-1] add arsenic to dsh groups, treat like terbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/92662 (owner: 10Dzahn) [16:57:08] ^d: it just finished for the first one [16:57:10] and oops heh [16:57:15] (03CR) 10jenkins-bot: [V: 04-1] add arsenic to dsh groups, treat like terbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/92662 (owner: 10Dzahn) [16:57:19] <^d> Silly jenkins [16:57:46] and where was my merge on irc? hmm [16:57:51] of the other one [16:58:37] i see, fixing [17:00:55] (03PS3) 10Dzahn: add arsenic to dsh groups, treat like terbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/92662 [17:04:51] (03CR) 10Dzahn: [C: 032] add arsenic to dsh groups, treat like terbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/92662 (owner: 10Dzahn) [17:05:18] ^d: merged on sockpuppet [17:06:20] <^d> I guess we need to run puppet on tin and then scap? [17:09:29] ^d: sounds right, yea [17:09:35] want me to run on tin? [17:09:44] <^d> That'd be great. I can do scap myself [17:10:10] doing so [17:10:55] <^d> yurik: You've got a deploy window right now, what all you doing for that? [17:11:12] ^d: not using it [17:11:22] <^d> Ah cool, thanks. [17:11:25] np [17:11:27] mutante: do you have ability to help me with RT? [17:11:30] feel free to crash it [17:12:21] <^d> greg-g: Gonna run scap. No new code going out, got a new terbium-like box to get in sync. [17:12:26] aude: yea, give me a minute [17:12:30] mutante: ok [17:12:40] <^d> yurik's not using his window, so shouldn't be stepping on toes [17:13:19] * yurik_ 's toes have been hardened by ops ;) [17:13:28] sees grain-ensure , salt grains .. and finished catalog [17:13:46] ^d: awesome, go forth and such [17:13:58] I recommend steel toed boots [17:14:38] ^d: mediawiki-installation:arsenic on tin [17:15:00] <^d> Hmm? [17:16:02] ^d: saying it's done, dsh group updated on tin [17:16:19] <^d> Ah k, cool. [17:17:01] !log demon Started syncing Wikimedia installation... : No changes, making sure arsenic is synced [17:17:17] Logged the message, Master [17:17:36] <^d> !log aborted sync [17:17:51] Logged the message, Master [17:18:03] <^d> Copying wikiversions dat and cdb files to apaches...arsenic: rsync: mkdir "/usr/local/apache/common-local" failed: Permission denied (13) [17:18:03] <^d> arsenic: rsync error: error in file IO (code 11) at main.c(605) [Receiver=3.0.9] [17:18:05] <^d> mutante: ^ [17:18:56] <^d> cd /usr/local/apache && mkdir common-local && chown mwdeploy:mwdeploy common-local [17:18:58] <^d> Should do it [17:19:24] ^d: try again [17:19:46] 0 lrwxrwxrwx 1 root root 12 May 15 23:40 common -> common-local [17:19:49] 4.0K drwxr-xr-x 2 mwdeploy mwdeploy 4.0K Oct 30 17:19 common-local [17:20:31] !log demon Started syncing Wikimedia installation... : No changes, making sure arsenic is synced [17:20:45] <^d> Looks good, no errors so far. [17:20:45] Logged the message, Master [17:22:35] (03PS2) 10Andrew Bogott: Switch to using uwsgi for the proxy api [operations/puppet] - 10https://gerrit.wikimedia.org/r/92664 [17:23:00] <^d> Unrelated, but worth noting: [17:23:18] <^d> snapshot[24]: sudo: no tty present and no askpass program specified [17:24:19] <^d> arsenic: /usr/local/bin/mwversionsinuse: line 5: /usr/local/apache/common-local/multiversion/activeMWVersions: No such file or directory [17:24:19] <^d> arsenic: Unable to read wikiversions.dat or it is empty [17:24:19] <^d> ... [17:24:20] <^d> arsenic: install: cannot change owner and permissions of `/usr/local/apache/uncommon': No such file or directory [17:24:22] <^d> arsenic: Unable to create /usr/local/apache/uncommon, please re-run this script as root. [17:24:24] <^d> Blahhh [17:25:42] you need mkdir /usr/local/apache/uncommon ? [17:26:05] <^d> Perhaps. But also curious why all this wasn't done via puppet or something else. [17:26:18] <^d> Reedy: Did we have these problems when setting up terbium? :) [17:26:27] i created uncommon for you, mwdeploy:mwdeploy [17:27:33] <^d> Well the uncommon bit is less important. I'm curious why we're not getting multiversion [17:33:07] (03PS1) 10MaxSem: Enable mobile redirect for wikimania2014.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/92671 [17:33:30] * ^d bangs on arsenic with a wrench [17:36:45] (03CR) 10Dzahn: [C: 031] Enable mobile redirect for wikimania2014.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/92671 (owner: 10MaxSem) [17:38:49] !log demon Finished syncing Wikimedia installation... : No changes, making sure arsenic is synced [17:39:03] Logged the message, Master [17:39:11] (03PS2) 10Dzahn: delete search.wikimedia.org Apache config file [operations/puppet] - 10https://gerrit.wikimedia.org/r/91132 [17:39:36] (03CR) 10Dzahn: [C: 032] delete search.wikimedia.org Apache config file [operations/puppet] - 10https://gerrit.wikimedia.org/r/91132 (owner: 10Dzahn) [17:40:26] MaxSem: O NO U DIDNT [17:40:31] :D [17:40:34] <^d> dangit. [17:40:41] * ^d fumes [17:41:26] is arsenic not workin? [17:42:33] heh, daniel is updtin me [17:42:35] annoying issues [18:04:00] ^d: is search.wikimedia.org = "new" search? [18:04:12] <^d> No [18:04:31] ok, so do we care about monitoring for this ? http://search.wikimedia.org/?lang=en&site=wikipedia&search=foo&limit=5 [18:04:37] returning something [18:04:46] not anymore? [18:04:54] <^d> It works. [18:04:57] <^d> It's a thing. [18:05:07] <^d> Silly apple dictionary thing. [18:05:08] apple dictionary [18:05:14] ah, yea, true [18:05:23] dont stir the mountain (apple) [18:05:28] =P [18:05:34] <^d> mutante: new search is search.svc.eqiad.wmnet ;-) [18:05:36] ehm.. yea, damn, well there's an RT. now let's forget about it :) [18:05:49] Didn't Tim essentially re-write it? [18:05:58] yep [18:06:01] its across apaches now [18:06:02] yes, he did, so it moved to apache-config [18:06:12] and we deleted it from puppet/files/apache/sites [18:06:16] just now [18:06:42] just wanted to know if there is any relation between cirrus and the usage of search.wm [18:08:43] Requestor hashar added . Requestor dzahn deleted [18:21:44] who wants to help me deploy a robots.txt tweak? [18:21:56] https://gerrit.wikimedia.org/r/#/c/92552/ [18:25:35] <^d> RobH, mutante: Where are we on arsenic? [18:27:59] <^d> brion: Should just be a matter of pulling and sync-file, right? [18:28:09] ^d: yeah [18:29:02] (03CR) 10Chad: [C: 032] Whitelist API mobileview for robots.txt [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92552 (owner: 10Brion VIBBER) [18:29:03] ^d: uhhh, what do you mean? [18:29:17] whats wrong with it? [18:29:23] (it should be identical to terbium) [18:29:30] (03Merged) 10jenkins-bot: Whitelist API mobileview for robots.txt [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92552 (owner: 10Brion VIBBER) [18:29:32] <^d> puppet setup's right, and we added it to dsh. [18:29:42] <^d> But nothing's been sync'd to it [18:29:51] <^d> Other than wikiversions.txt|dat [18:29:54] hrmm [18:30:16] i didnt include all the terbium stuff, ie: left out maintaince script calls [18:30:16] perhaps those calls (set to disabled) install that other stuff. [18:30:38] im gonna install them set to false [18:30:40] see if it doesnt fix. [18:31:03] ...does that make sense orn o? [18:31:07] wheeeee [18:31:08] * RobH checks into what they install first [18:31:27] thx ^d [18:31:45] hahahaaha [18:31:51] ^d: i dont think anything requires mediawiki at present [18:32:03] since those maintainance scripts are what require it. [18:32:07] =P [18:32:27] mediawiki? pssh that's some bloated php crap. delete! [18:32:36]