[00:02:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.142 second response time [00:05:14] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:08:44] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 00:08:39 UTC 2013 [00:09:14] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:09:44] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 00:09:42 UTC 2013 [00:10:14] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:10:44] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 00:10:38 UTC 2013 [00:11:14] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:11:37] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 00:11:30 UTC 2013 [00:12:14] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:12:14] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 00:12:12 UTC 2013 [00:13:14] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:13:24] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 00:13:19 UTC 2013 [00:14:14] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:15:24] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [00:32:24] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [00:32:54] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 00:32:48 UTC 2013 [00:33:15] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:39:31] PROBLEM - Disk space on cp1041 is CRITICAL: Timeout while attempting connection [00:40:21] RECOVERY - Disk space on cp1041 is OK: DISK OK [01:03:35] New patchset: Alex Monk; "(bug 46990) Add the 'editor' restriction level on pl.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58038 [01:04:15] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [01:13:35] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [01:50:02] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [01:50:02] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [01:50:02] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [02:04:28] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [02:04:48] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [02:18:18] !log LocalisationUpdate completed (1.22wmf1) at Mon Apr 8 02:18:18 UTC 2013 [02:18:26] Logged the message, Master [02:25:07] !log LocalisationUpdate completed (1.21wmf12) at Mon Apr 8 02:25:07 UTC 2013 [02:25:13] Logged the message, Master [02:34:57] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [03:04:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [03:07:27] PROBLEM - Squid on brewster is CRITICAL: Connection refused [03:55:16] !log on all apaches: upgrading libpoppler [03:55:23] Logged the message, Master [03:59:30] RECOVERY - Squid on brewster is OK: TCP OK - 0.027 second response time on port 8080 [03:59:54] !log on brewster: root partition was full, removed squid access.log and store.log and started squid [04:00:01] Logged the message, Master [04:05:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:08:37] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 04:08:35 UTC 2013 [04:09:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:09:47] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 04:09:40 UTC 2013 [04:10:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:10:37] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 04:10:36 UTC 2013 [04:11:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:11:37] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 04:11:26 UTC 2013 [04:12:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:12:17] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 04:12:09 UTC 2013 [04:13:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:13:27] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 04:13:17 UTC 2013 [04:14:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:31:11] New patchset: Ori.livneh; "udp2log on fluorine: relay MW errors to vanadium" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58047 [04:32:04] New patchset: Ori.livneh; "udp2log on fluorine: relay MW errors to vanadium" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58047 [04:32:47] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 04:32:43 UTC 2013 [04:33:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:34:59] TimStarling: if you have a moment, could you look at https://gerrit.wikimedia.org/r/58047 ? it's a small change which adds a udp log filter on fluorine. [05:04:41] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [05:44:43] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [06:00:43] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [06:08:24] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [06:16:24] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [06:28:34] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 06:28:27 UTC 2013 [06:29:24] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [06:29:44] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 06:29:40 UTC 2013 [06:30:24] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [06:32:54] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 06:32:50 UTC 2013 [06:33:24] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [07:06:23] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [07:30:23] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [07:48:02] !log restarting Zuul for demo purposes :-) [07:48:10] Logged the message, Master [07:53:16] apergos: Hi, gerrit got stuck again and refuses to talk to zuul (which is needed for the gerrit/jenkins integration). Could you please restart the gerrit daemon on manganese? [07:53:24] σεψ [07:53:25] sec [07:55:04] qchris: I gave a bit of context on the bug report for history purposes [07:55:12] apergos: good morning :-] [07:55:18] hashar: Thanks [07:55:27] qchris: what struck me is that whenever stream-events is blocked, it is blocked for everyone else even a new connection. [07:55:32] done [07:55:34] please check now [07:55:50] apergos: Thanks \o/ [07:55:55] yay [07:56:02] whenever that happened, I tried establishing a new connecting with my account. It does not receive any new event either :( [07:56:06] hashar: Yes, there seems to be something blocked within gerrit [07:56:29] But I did not yet manage to reproduce reliable. [07:56:40] But that's on the agenda for this morning :-) [07:57:47] hashar: Is there some repository that we can use to periodically invoke "recheck" on that easy on Jenkins tests? [07:58:28] maybe test/mediawiki [07:58:32] (As fix until Chad joins us to install a new gerrit.war) [07:58:39] Ok. Thanks [07:58:44] bah it got deleted [08:00:11] qchris: you can create a test change under integration/zuul-config and spam recheck there :-] [08:00:27] Ok. I'll try that. Thanks. [08:00:52] It triggers a YAML linter job https://integration.wikimedia.org/ci/job/integration-zuul-config-yamllint/ [08:01:29] qchris: also I could not found the Gerrit source code we are using. It used to be under operations/gerrit.git [08:02:00] ^demon: Decided that we run vanilla upstream [08:02:12] okkk [08:02:23] In the bottom right of a gerrit page you'll find something like "2.6-rc0-144-gb1dadd2" [08:02:32] and so the production version shows up a sha1 of gb1dadd2 but that is not in upstream [08:02:40] So it's the commit starting in b1dadd2 [08:02:46] .. [08:02:49] g [08:03:03] not hexadecimal [08:03:18] 'b1dadd2cc209482f60ca0e52c47e76bc51b87ed7' [08:03:26] ^ is the full hash [08:03:48] The string stems from 'git describe' [08:04:12] And that prefixes the hash with 'g' as in /g/it [08:04:38] today I learned a new git command "describe" :-] [08:06:02] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:08:11] git log --oneline --no-merges 52fb5ae..b1dadd2 |wc -l [08:08:12] 38 [08:08:18] we are lucky, only 38 commits to look at :-] [08:08:22] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 08:08:12 UTC 2013 [08:09:02] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:09:22] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 08:09:16 UTC 2013 [08:10:02] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:29:35] qchris / hashar : Did you automate that recheck comment yet, or would you need some help? [08:30:02] siebrand: Done :-) [08:30:12] cool [08:31:22] PROBLEM - Puppet freshness on virt1000 is CRITICAL: No successful Puppet run in the last 10 hours [08:33:12] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 08:33:11 UTC 2013 [08:34:02] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:35:18] ah [08:35:19] Received disconnect from 208.80.154.152: 2: User idle has timed out after 600000ms. [08:35:20] :-D [08:35:27] that must be the message zuul can't parse [08:36:55] qchris: I will create you another repo for the 5 minutes ping [08:37:18] Ok. Great. [08:38:40] qchris: test/gerrit-ping owner is ldap/wmf [08:39:22] Thanks hashar. I am not in ldap/wmf (last time I checked) ... let's see if I can push anyways :-) [08:40:10] you should be able to create a change against it then add a comment like 'ping' [08:40:21] Ok. I'll try [08:43:47] it receives events :-] [08:44:01] hashar: Now gerrit-wm is spamming #mediawiki with the 'recheck's. Is that ok? [08:44:13] ahrgh [08:44:50] seems #mediawiki and #wikimedia-dev are the default ahah [08:44:55] will amend the hook [08:44:56] Is there some way for gerrit-wm to ignore me and my comments? [08:49:11] hashar: besides, jenkins-bot does not seem to act on test/gerrit-ping :-( See [08:49:14] https://gerrit.wikimedia.org/r/#/c/58060/ [08:49:27] yeah but Zuul receive events nonetheless [08:49:34] I can create a jenkins job if you want [08:49:40] Nono. [08:49:50] If zuul acts, it's ok [08:50:12] Zuul is just too quick for me to notice :-) [08:51:24] New patchset: Hashar; "gerrit: no IRC message for test/gerrit-ping" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58063 [08:52:18] apergos: sorry to interrupt again but we would need Gerrit to not send notification for test/gerrit-ping.git repository. The change is https://gerrit.wikimedia.org/r/58063 :-] [08:52:43] apergos: you can blindly trust me on this one :-] I think it just need merge + puppet run on manganese [08:55:53] please give me just two minuntes first [08:56:06] take your time 8) [08:58:55] morning [08:59:16] gooood morning ! [09:03:34] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58063 [09:04:01] apergos: Thanks [09:05:16] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [09:09:03] paravoid: I have officially joined the Debian python module team :-] [09:09:13] woo! [09:09:15] paravoid: will probably upload my debs tomorrow :-] [09:09:42] do you have a sponsor? [09:09:43] just have to figure out my credentials to access the subversion repo , tweak the maintainer field and add myself as uploader [09:09:53] aren't you my sponsor? ;-] [09:10:05] haha [09:10:06] yes I am :) [09:10:08] ;-] [09:10:11] so yes [09:10:16] you can't upload them yourself [09:10:26] you'll need to commit them to SVN [09:10:38] then I'll build and upload [09:10:49] will ping you whenever I have done the commit so [09:10:59] I though we could use dput to send the package to some public area [09:11:06] there's mentors.debian.net [09:11:10] but no need to [09:40:49] speaking of deb packages, can someone create operations/debs/libvpx i want to push a backport with a patch, anything except creating the repo can go via gerrit review i guess [09:59:22] j^: hi :-] I can create a git repo named operations/debs/libvpx [10:02:34] hashar: cool, can i push branches via gerrit(upstream, pristine-tar) or would i need push permissions for those? [10:02:49] hmm [10:02:54] http://packages.ubuntu.com/search?keywords=libvpx [10:03:04] there is a 1.1.0-1 package in ubuntu Quantal and Raring [10:03:08] maybe we can reuse those [10:03:47] I am not sure whether you need a git repo [10:03:59] hashar: yes i started off with the versoin from quantal imported via git-import-dsc --pristine-tar libvpx_1.1.0-1.dsc and added my patch [10:04:14] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [10:04:26] ah so that is upstream + a patch :-] [10:04:45] hashar: https://rt.wikimedia.org/Ticket/Display.html?id=4868 [10:05:14] yes, current release + one patch from git [10:06:38] was told I should use git here on #wikimedia-operations, so I put it in git, now just a question of where to push it to [10:09:32] creating creating [10:13:02] bah I don't know how to create the orphan branches upstream and pristine-tar :( [10:16:42] j^: I think I screwed it up [10:16:55] I created the branches master pristine-tar and upstream [10:17:02] but they all point to the first initial empty commit [10:17:04] not ideal [10:17:37] deleted them [10:20:33] New patchset: J; "Imported Upstream version 1.1.0" [operations/debs/libvpx] (master) - https://gerrit.wikimedia.org/r/58070 [10:21:14] yeah I think we need to be in the upstream branch don't we ? [10:21:50] ! [remote rejected] 60c5e53cf81d47a0f1d2f4333bc338a2fcf092f8 -> refs/for/upstream (branch upstream not found) [10:21:50] Change abandoned: J; "target should be upstream" [operations/debs/libvpx] (master) - https://gerrit.wikimedia.org/r/58070 [10:21:50] :( [10:21:57] I have no idea how to do it [10:25:06] so https://wikitech.wikimedia.org/wiki/Git-buildpackage only works if one can push changes, using gerrit to setup a repo does not quite work [10:25:44] * [new branch] 60c5e53cf81d47a0f1d2f4333bc338a2fcf092f8 -> upstream [10:25:45] Bus error: 10 [10:25:46] oh ho [10:26:37] j^: bah I have pushed your change as the first revision of the upstream branch [10:26:49] need to clean that out [10:29:43] j^: I have created dummy commits for the pristine-tar and upstream branches [10:29:51] j^: you can restore your change https://gerrit.wikimedia.org/r/58070 [10:30:04] and push its sha1 to refs/for/upstream [10:30:15] that might update the change to be against the upstream branch [10:30:52] !log gerrit: created operations/debs/libvpx for j^ . Initialized pristine-tar and upstream branches using empty commits and force push. [10:30:58] Logged the message, Master [10:31:16] j^: I can even try it for you :] [10:31:21] Change restored: Hashar; "(no reason)" [operations/debs/libvpx] (master) - https://gerrit.wikimedia.org/r/58070 [10:33:41] cherry picked [10:33:44] resending [10:34:00] all this seams way to complicated [10:34:02] New patchset: Hashar; "Imported Upstream version 1.1.0" [operations/debs/libvpx] (upstream) - https://gerrit.wikimedia.org/r/58071 [10:34:22] and that is a new change huuh [10:34:51] New review: Hashar; "I have made this change against upstream branch with https://gerrit.wikimedia.org/r/#/c/58070/" [operations/debs/libvpx] (master) - https://gerrit.wikimedia.org/r/58070 [10:35:05] j^: I must agree [10:35:25] j^: anyway your change is https://gerrit.wikimedia.org/r/#/c/58071/ [10:35:41] and you get dummy branches for master and pristine-tar [10:36:07] well that was the first of something like 5 commits that [10:36:39] git-review also keeps messing with my commit history and the branches are no longer in sync [10:38:42] not sure if getting it through gerrit/review is reasonable for deb packages [10:39:09] i can see that it should happen for the patch, but review for upstream tarball [10:40:09] in the rt ticket i have a link to the deb files. this is all to 'document' the changes. since i might have to push another package soon i would not mind figuring out a workflow that i can repeat [10:41:11] New review: Hashar; "Leslie : that is mostly harmless :-] The reason I added you as a reviewer is because I think you h..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55304 [10:48:28] j^: mortals can't push to operations/debs repositories. So if you want to update an upstream branch you have to submit a change that will be merged by ops [11:02:45] hashar: ok. do you know how i can push a review that is not for master? [11:04:06] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [11:06:40] j^: git push origin 123456ABCDEF:refs/for/upstream [11:06:44] I mean [11:07:02] git push :refs/for/ [11:07:05] so that would be: [11:07:12] git push origin upstream:refs/for/upstream [11:07:51] restarting comp brb [11:11:29] New patchset: J; "pristine-tar data for libvpx_1.1.0.orig.tar.bz2" [operations/debs/libvpx] (master) - https://gerrit.wikimedia.org/r/58072 [11:13:54] New patchset: J; "Imported Upstream version 1.1.0" [operations/debs/libvpx] (master) - https://gerrit.wikimedia.org/r/58073 [11:13:55] New patchset: J; "Import 1.1.0-1+wmf1" [operations/debs/libvpx] (master) - https://gerrit.wikimedia.org/r/58074 [11:26:09] all this does not work since git-buildpackage uses sha1 hashes to identify i.e. pristine-tar commits, once Change-Id is added they of cause change and building no longer works [11:34:08] j^, how so? [11:34:15] it just an extra line in the commit mst [11:35:08] Platonides: yes but the sha1 of the commit includes the commit msg so changing a commit will change its hash, git-buildpackage links to a commit hash in another commit [11:35:40] pass the hash of the changed commit to git-buildpackage ? [11:36:43] Platonides: yes I can manually do what git-buildpackage does, but at that point its not helping to 'use' git-buildpackage [11:37:07] i.e. git-import-dsc --pristine-tar ../libvpx_1.1.0-1.dsc [11:37:20] creates commits in 3 branches [11:38:15] does it obey git hooks? [11:38:37] if the hook adding the Change-Id is installed, I would expect it to work [11:38:38] tried that, does not look like it does [11:38:43] :( [11:50:05] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [11:50:05] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [11:50:05] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [11:52:44] <^demon> !log bringing gerrit down for urgent update [11:52:51] Logged the message, Master [11:57:13] <^demon> !log gerrit back, deployed 2.6-rc0-154-gfcdb34b which contains a temporary fix for the stream-events timeout issues. See bug 46917 for info. [11:57:19] Logged the message, Master [12:05:34] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:06:14] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [12:08:44] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 12:08:35 UTC 2013 [12:09:34] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:09:44] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 12:09:39 UTC 2013 [12:10:34] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:10:44] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 12:10:36 UTC 2013 [12:11:34] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:11:34] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 12:11:26 UTC 2013 [12:12:34] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:12:54] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 12:12:47 UTC 2013 [12:13:34] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:32:54] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Apr 8 12:32:49 UTC 2013 [12:33:15] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [12:33:34] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:35:51] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [12:39:33] bblack--: a little early isn't it [12:39:38] :) [12:39:54] I'm back on central US time at home now :) [12:40:44] :) [12:41:35] So, I got home Saturday night from the airport to find a gargantuan tree had been knocked over by a storm while I was gone last week, right across my driveway. Didn't hit anything important (but crushed a basketball goalpost). [12:41:52] ouch [12:41:56] Spent all day yesterday with a chainsaw and a couple friends cutting it up and getting it out of the way [12:42:34] So that was my exercise for the week, now I can just sit in this chair for days and not feel guilty [12:42:54] i've been laying stones/pavement yesterday [12:43:07] so I had trouble fitting my socks this morning [12:43:33] hah [12:56:11] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [13:05:47] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [13:21:37] New patchset: Nemo bis; "Global jobqueue check: mwscript path fix" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58079 [13:31:07] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [13:31:58] New patchset: Odder; "(bug 41745) Remove ptwiki, ptwikinews from EmergencyCaptcha" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58081 [13:32:57] New review: Nemo bis; "Per bug, nihil obstat." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/58081 [13:40:13] New patchset: Nemo bis; "Prevent gerrit logo from pushing the search bar outside the screen" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58082 [13:42:02] New review: Nemo bis; "I've not mentioned https://bugzilla.wikimedia.org/show_bug.cgi?id=36471 because this is only a very ..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58082 [13:42:35] ^demon: I hope that patch makes sense [13:48:10] PROBLEM - Varnish traffic logger on dysprosium is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [14:01:10] RECOVERY - Varnish traffic logger on dysprosium is OK: PROCS OK: 3 processes with command name varnishncsa [14:04:08] PROBLEM - Varnish traffic logger on dysprosium is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [14:04:38] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [14:15:48] hashar: about to merge the cowbuilder stuff [14:15:50] ack? [14:15:57] yeahhhh [14:16:08] New patchset: Faidon; "package-builder learned 'cowbuilder'" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56382 [14:16:09] paravoid: that is a bit messy though. [14:16:20] maybe I should have written a short shell script to generate the images :-] [14:16:30] but hey, it works [14:16:33] I didn't look closely [14:16:58] it's contint material and I trust you enough for that :) [14:17:09] ;-] [14:18:34] paravoid: and you need to merge :-] https://gerrit.wikimedia.org/r/#/c/56382/ [14:18:39] jenkins does not merge for ya [14:18:43] (on ops/puppet [14:19:08] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56382 [14:19:15] I know [14:19:28] I rebased and I was waiting for jenkins to give verified [14:20:37] merging the gerrit stuff on sockpuppet to [14:20:39] *too [14:23:58] \O/ [14:26:38] PROBLEM - Varnish HTTP mobile-backend on cp1041 is CRITICAL: Connection timed out [14:27:28] RECOVERY - Varnish HTTP mobile-backend on cp1041 is OK: HTTP OK: HTTP/1.1 200 OK - 634 bytes in 0.643 second response time [14:30:08] RECOVERY - Varnish traffic logger on dysprosium is OK: PROCS OK: 3 processes with command name varnishncsa [14:30:29] New review: Ottomata; "(5 comments)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50452 [14:31:07] I'm listening [14:31:08] :) [14:31:44] New patchset: Ottomata; "Adding puppet-merge for sockpuppet puppet merges." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50452 [14:31:50] hehe [14:32:04] first an easy one: [14:32:11] git clean -dffx -e private [14:32:12] s'ok? [14:32:18] private isn't in sockpuppet's working copy anyway [14:32:20] but just in case [14:32:22] I guess :) [14:32:34] ha, k [14:33:38] uhh, as for flock, sure! [14:34:23] shall I? [14:34:40] we can iterate [14:34:42] i guess it only helps, if someone else is runnign puppet-merge, then the second user shouldn't be able to [14:34:43] if you prefer that [14:36:04] hmm, either way I guess, i'm worried about locks being left behind if someone ctrl-cs or something (haven't used flock much) [14:36:34] yeah let's leave it for later [14:36:56] hmmk [14:37:05] I'll add a TODO comment in the script [14:37:13] have you tested this? [14:37:38] not a lot with the puppet repository, but in lots of different cases with my own test repo [14:37:49] removing, adding, modifying, etc. canceling, etc. [14:38:18] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [14:44:11] do we still need to build package for the `hardy` distribution ? [14:44:17] it does not have cowbuilder :-] [14:44:35] so I though we could remove hardy from the package building class [14:45:23] no, yes [14:46:10] New review: Faidon; "Seems reasonable, let's iterate if needed." [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/50452 [14:46:15] ottomata: ^ [14:46:40] COOOOOOLLLL [14:46:51] so, what should I do then, merge it and send an email to #ops explaining? [14:47:10] New patchset: Hashar; "pbuilder: get rid of `hardy` environnement" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58090 [14:47:16] merge it, let's start using it and yeah, inform the rest of ops [14:47:17] ;-] [14:47:27] woot [14:47:53] New patchset: Ottomata; "Adding puppet-merge for sockpuppet puppet merges." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50452 [14:48:08] ^ that's just a comment change [14:49:17] New review: coren; "The sense, you are making it." [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/58090 [14:50:29] New patchset: Ottomata; "Adding puppet-merge for sockpuppet puppet merges." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50452 [14:50:41] ^ and that was the rebase [14:51:09] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50452 [14:51:20] paravoid: removing hardy with https://gerrit.wikimedia.org/r/58090 :-] [14:54:06] coren: are you about? [14:54:17] cmjohnson1: I live! [14:54:45] great...so, chaning the cables for you now [14:54:55] cmjohnson1: /me dances. [14:55:13] you want both servers connected to both disk shelves...correct? [14:56:54] cmjohnson1: Exactly [14:57:06] cmjohnson1: Multi-server rather than multipath. :-) [14:57:24] okay [15:00:27] Coren: are you aware of any current issues with project storage? I had a read-only fs /home, and after reboot I can't log in [15:01:18] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [15:01:32] mark: It's not so much "current" as it is ongoing. That sounds like you just lost a brick. [15:02:07] I don't know if Ryan wrote down notes on what he does to fix that. I'll try to find if he did. [15:04:10] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [15:05:18] coren: should be good but plz check [15:06:27] hey paravoid, would you have a couple of minutes to look at the kafka external contractor position that I drafted (you should have received the link by email) [15:06:30] cmjohnson1: I go check now. [15:10:11] New patchset: Mark Bergsma; "Add device detection to mobile ResourceLoader requests" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56774 [15:11:57] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56774 [15:16:15] cmjohnson1: AFAICT, I only see one shelf from 1001. Lemme check something in the H800 doc. [15:16:18] thanks mark for your feedback! [15:18:11] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [15:20:57] coren: 1001 will only have 1 shelf...it's on a different rack....I will have to move it to the same rack as the others [15:21:10] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [15:21:17] cmjohnson1: Ah! Which ones did you tie then? [15:21:22] 1003/1002 [15:21:35] Well, perhaps I should check /those/ then. :-) [15:22:06] heh..if you want it all tied together ...i can do that...i will just have to move it to a different rack (i have the space) [15:22:32] cmjohnson1: No, that's allright, I need just the two, it's not really important /which/ two. [15:26:23] cmjohnson1: 1002 only sees one shelf too. :-( [15:26:34] ok [15:26:36] cmjohnson1: They both powered on? :-) [15:26:41]