[00:09:10] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 00:09:03 UTC 2013
[00:09:22] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 00:09:09 UTC 2013
[00:09:22] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[00:09:33] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[00:10:08] <TimStarling>	 !log installing the experimental libmemcached packages on srv193
[00:10:17] <morebots>	 Logged the message, Master
[00:13:30] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 00:13:29 UTC 2013
[00:14:11] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 00:14:00 UTC 2013
[00:14:11] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[00:14:21] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[00:14:51] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 00:14:48 UTC 2013
[00:15:32] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[00:17:40] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 00:17:39 UTC 2013
[00:18:10] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[00:18:20] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 00:18:10 UTC 2013
[00:18:20] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 00:18:12 UTC 2013
[00:18:20] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[00:18:30] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[00:20:21] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 00:20:17 UTC 2013
[00:20:31] <Ryan_Lane>	 !log upgraded mediawiki on wikitech-static
[00:20:38] <morebots>	 Logged the message, Master
[00:20:51] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 00:20:43 UTC 2013
[00:21:03] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 00:20:51 UTC 2013
[00:21:11] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[00:21:21] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[00:21:31] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[00:22:51] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 00:22:42 UTC 2013
[00:23:09] <AzaToth>	 "Exception in thread "pool-1-thread-1" com.facebook.buck.util.HumanReadableException: Unable to parse build file:" - thanks for that human readable error (File "<script>", line 6, in <module>)
[00:23:10] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 00:23:04 UTC 2013
[00:23:10] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 00:23:06 UTC 2013
[00:23:10] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[00:23:22] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[00:23:32] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[00:23:41] * AzaToth  could strangle people for less

[00:24:53] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 00:24:48 UTC 2013
[00:25:10] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[00:25:12] <paravoid>	 lol
[00:25:20] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 00:25:11 UTC 2013
[00:25:20] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[00:25:30] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 00:25:22 UTC 2013
[00:25:30] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[00:25:30] <paravoid>	 I'm sorry, I didn't expect this to be that horrible
[00:27:00] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 00:26:53 UTC 2013
[00:27:00] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 00:26:59 UTC 2013
[00:27:10] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[00:27:20] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[00:27:30] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 00:27:26 UTC 2013
[00:28:30] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[00:28:30] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 00:28:25 UTC 2013
[00:28:51] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 00:28:44 UTC 2013
[00:29:01] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 00:28:56 UTC 2013
[00:29:10] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[00:29:21] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[00:29:32] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[00:29:50] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 00:29:43 UTC 2013
[00:29:50] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 00:29:47 UTC 2013
[00:30:00] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 00:29:53 UTC 2013
[00:30:10] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[00:30:21] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[00:30:30] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[00:34:55] <AzaToth>	 paravoid: I must say buck is soo ugly I don't think it's feasable to use in any practical settings
[00:41:35] <mutante>	 how do i call a class/script that mails the mail alias file ?   misc::maintenance::mailmailaliasfile ..i don't like it:)
[01:01:42] <icinga-wm>	 RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset 0.0005390644073 secs
[01:08:08] <gerrit-wm>	 New patchset: Dzahn; "add a maintenance cronjob to mail the mchenry alias file to OIT (RT #5278)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68127
[01:32:50] <icinga-wm>	 RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset 0.004817247391 secs
[01:41:04] <icinga-wm>	 PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours
[01:41:04] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours
[01:41:04] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours
[01:41:04] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours
[01:41:04] <icinga-wm>	 PROBLEM - Puppet freshness on ms-be1 is CRITICAL: No successful Puppet run in the last 10 hours
[01:41:04] <icinga-wm>	 PROBLEM - Puppet freshness on mw1020 is CRITICAL: No successful Puppet run in the last 10 hours
[01:41:05] <icinga-wm>	 PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours
[01:41:05] <icinga-wm>	 PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours
[01:41:06] <icinga-wm>	 PROBLEM - Puppet freshness on sockpuppet is CRITICAL: No successful Puppet run in the last 10 hours
[01:41:06] <icinga-wm>	 PROBLEM - Puppet freshness on spence is CRITICAL: No successful Puppet run in the last 10 hours
[01:41:07] <icinga-wm>	 PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours
[01:41:07] <icinga-wm>	 PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours
[01:41:08] <icinga-wm>	 PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours
[01:51:26] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:53:14] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time
[02:07:39] <logmsgbot>	 !log LocalisationUpdate completed (1.22wmf6) at Wed Jun 12 02:07:39 UTC 2013
[02:07:48] <morebots>	 Logged the message, Master
[02:13:50] <logmsgbot>	 !log LocalisationUpdate completed (1.22wmf5) at Wed Jun 12 02:13:49 UTC 2013
[02:13:58] <morebots>	 Logged the message, Master
[02:22:21] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[02:23:11] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time
[02:27:19] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 12 02:27:19 UTC 2013
[02:27:27] <morebots>	 Logged the message, Master
[02:48:40] <icinga-wm>	 PROBLEM - SSH on mc15 is CRITICAL: Connection timed out
[02:49:40] <icinga-wm>	 RECOVERY - SSH on mc15 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0)
[03:24:19] <gerrit-wm>	 New patchset: Ori.livneh; "Set common rsync and dsh parameters in mw-deployment-vars" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57890
[03:48:33] <icinga-wm>	 PROBLEM - DPKG on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[03:49:23] <icinga-wm>	 RECOVERY - DPKG on mc15 is OK: All packages OK
[04:09:17] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 04:09:08 UTC 2013
[04:09:17] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[04:09:37] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 04:09:34 UTC 2013
[04:09:50] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 04:09:38 UTC 2013
[04:10:17] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[04:10:23] <gerrit-wm>	 New patchset: Catrope; "Point the Parsoid cache in labs to parsoid-spof rather than deployment-parsoid2" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68133
[04:10:37] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[04:11:58] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 04:11:47 UTC 2013
[04:12:08] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 04:11:59 UTC 2013
[04:12:08] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 04:12:05 UTC 2013
[04:12:17] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[04:12:17] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[04:12:38] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[04:13:39] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 04:13:28 UTC 2013
[04:13:48] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 04:13:44 UTC 2013
[04:13:57] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 04:13:49 UTC 2013
[04:14:18] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[04:14:18] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[04:14:37] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[04:14:57] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 04:14:47 UTC 2013
[04:14:57] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 04:14:54 UTC 2013
[04:15:07] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 04:14:57 UTC 2013
[04:15:17] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[04:15:17] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[04:15:37] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[04:15:59] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 04:15:54 UTC 2013
[04:16:08] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 04:16:06 UTC 2013
[04:16:17] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[04:16:17] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 04:16:10 UTC 2013
[04:16:17] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[04:16:38] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[04:17:07] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 04:16:59 UTC 2013
[04:17:07] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 04:17:04 UTC 2013
[04:17:18] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 04:17:07 UTC 2013
[04:17:18] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[04:17:18] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[04:17:37] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[04:17:57] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 04:17:52 UTC 2013
[04:17:57] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 04:17:56 UTC 2013
[04:18:08] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 04:17:58 UTC 2013
[04:18:17] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[04:18:17] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[04:18:37] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[04:22:27] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:23:17] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time
[04:24:47] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 04:24:44 UTC 2013
[04:25:17] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[04:28:47] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 04:28:42 UTC 2013
[04:29:17] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[04:30:17] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 04:30:08 UTC 2013
[04:30:38] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[04:33:03] <gerrit-wm>	 New patchset: Ori.livneh; "morebots: auto-link SHA1s and Gerrit change IDs" [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/68135
[04:39:30] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:41:24] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time
[04:51:42] <ori-l>	 TimStarling: Do you know if what Tyler Romeo said about '@file' on wikitech true? He claimed that it is useful only for associating a comment with an external file, but the Doxygen docs explicitly allow '@file' to be unparametrized, which suggests there is some use for it.
[04:51:53] <ori-l>	 I'm trying to save myself a ~400mb download.
[04:52:27] <TimStarling>	 why not run it and see what happens?
[04:52:41] <icinga-wm>	 PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server
[04:52:43] <ori-l>	 0 upgraded, 61 newly installed, 0 to remove and 68 not upgraded.
[04:52:44] <ori-l>	 Need to get 433 MB of archives.
[04:52:51] <ori-l>	 that's why :)
[04:54:34] <TimStarling>	 I can probably run it
[04:54:50] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 04:54:46 UTC 2013
[04:55:20] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[04:58:42] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 04:58:39 UTC 2013
[04:59:15] <TimStarling>	 it tries to run /usr/bin/php for some reason
[04:59:21] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[05:00:00] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 04:59:52 UTC 2013
[05:00:31] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[05:01:20] <icinga-wm>	 PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server
[05:03:36] <ori-l>	 http://noc.wikimedia.org/~olivneh/doxygen_testcases.tar.gz has four php files
[05:04:13] <ori-l>	 i'm guessing 1, 2 & 3 produce identical output, no idea about 4
[05:05:16] <ori-l>	 (the hypothesis is that a comment block doesn't need '@file' if it's freestanding and at the top of the file it's describing)
[05:07:30] <TimStarling>	 only case 1 and 3 are in the file list, case 2 and 4 are missing
[05:08:26] <TimStarling>	 this is with default configuration with doxygen 1.7.6.1
[05:09:32] <ori-l>	 ah, bummer. so we can't get rid of it. thanks for checking.
[05:14:31] <gerrit-wm>	 Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68133
[05:22:32] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:23:22] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.150 second response time
[05:40:27] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:41:17] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time
[06:00:27] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:02:17] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time
[06:06:42] <icinga-wm>	 PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[06:07:42] <icinga-wm>	 RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s)
[06:10:32] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:11:24] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time
[06:22:32] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:23:22] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time
[06:35:22] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 06:35:17 UTC 2013
[06:35:33] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 06:35:26 UTC 2013
[06:35:42] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 06:35:32 UTC 2013
[06:36:22] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[06:36:22] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[06:36:45] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 06:36:22 UTC 2013
[06:36:45] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 06:36:27 UTC 2013
[06:37:24] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[06:37:24] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[06:37:34] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[06:40:34] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:41:24] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time
[06:57:34] <hashar>	 morning
[06:57:44] <hashar>	 apergos: here I am :)
[07:01:22] <apergos>	 morning
[07:08:45] <hashar>	 !g I0dab6f527e590624fd354342df604a8cdadc04e5
[07:08:46] <wm-bot>	 https://gerrit.wikimedia.org/r/#q,I0dab6f527e590624fd354342df604a8cdadc04e5,n,z
[07:22:52] <icinga-wm>	 PROBLEM - SSH on nescio is CRITICAL: Server answer:
[07:23:53] <icinga-wm>	 RECOVERY - SSH on nescio is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0)
[07:31:02] <icinga-wm>	 PROBLEM - SSH on amslvs2 is CRITICAL: Server answer:
[07:33:32] <icinga-wm>	 PROBLEM - SSH on knsq24 is CRITICAL: Server answer:
[07:33:43] <icinga-wm>	 PROBLEM - SSH on knsq22 is CRITICAL: Server answer:
[07:34:02] <icinga-wm>	 RECOVERY - SSH on amslvs2 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0)
[07:34:32] <icinga-wm>	 RECOVERY - SSH on knsq24 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0)
[07:34:34] <icinga-wm>	 PROBLEM - SSH on amssq47 is CRITICAL: Server answer:
[07:34:34] <icinga-wm>	 PROBLEM - SSH on cp3021 is CRITICAL: Server answer:
[07:34:42] <icinga-wm>	 PROBLEM - Parsoid on wtp1021 is CRITICAL: Connection refused
[07:34:42] <icinga-wm>	 PROBLEM - SSH on amssq32 is CRITICAL: Server answer:
[07:35:32] <icinga-wm>	 RECOVERY - SSH on amssq47 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0)
[07:35:52] <icinga-wm>	 RECOVERY - SSH on knsq22 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0)
[07:37:18] <icinga-wm>	 RECOVERY - SSH on cp3021 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0)
[07:38:58] <icinga-wm>	 PROBLEM - SSH on nescio is CRITICAL: Server answer:
[07:39:08] <icinga-wm>	 PROBLEM - SSH on cp3009 is CRITICAL: Server answer:
[07:39:48] <icinga-wm>	 RECOVERY - SSH on amssq32 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0)
[07:40:08] <icinga-wm>	 RECOVERY - SSH on cp3009 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0)
[07:40:58] <icinga-wm>	 RECOVERY - SSH on nescio is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0)
[07:50:38] <icinga-wm>	 RECOVERY - Parsoid on wtp1021 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.005 second response time
[07:55:46] <gerrit-wm>	 New patchset: Hashar; "upload cache in labs now uses role::cache::configuration" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54863
[07:56:25] <gerrit-wm>	 New review: Hashar; "patch probably unneeded now." [operations/puppet] (production) C: -1;  - https://gerrit.wikimedia.org/r/54863
[08:00:03] <gerrit-wm>	 Change abandoned: Hashar; "got merged via https://gerrit.wikimedia.org/r/#/c/44709/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50064
[08:01:48] <icinga-wm>	 RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset 0.0002949237823 secs
[08:01:53] <gerrit-wm>	 Change abandoned: Hashar; "I am not working on this. Feel free to cherry pick this patch and create a new change number. Meanti..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53714
[08:03:08] <icinga-wm>	 RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.004890203476 secs
[08:05:20] <hashar>	 apergos: I am recreating the upload cache instance in beta.   The current one is running Lucid ( deployment-cache-upload03 )
[08:07:48] <apergos>	 yes, I saw that it was
[08:08:02] <apergos>	 I assume it will look just like the old one?
[08:08:32] <hashar>	 hopefully :-)
[08:08:48] <apergos>	 (I am gathering up my changes but haven't submitted a changeset yet and you will get to review them before anything happens, it's a fair number of changes)
[08:08:53] <apergos>	 ok cool
[08:10:02] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 08:09:53 UTC 2013
[08:10:12] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 08:10:08 UTC 2013
[08:10:24] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 08:10:16 UTC 2013
[08:10:25] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[08:10:25] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[08:10:52] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[08:12:12] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 08:12:03 UTC 2013
[08:12:12] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 08:12:08 UTC 2013
[08:12:12] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 08:12:11 UTC 2013
[08:12:22] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[08:12:22] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[08:12:52] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[08:13:12] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 08:13:09 UTC 2013
[08:13:22] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 08:13:16 UTC 2013
[08:13:22] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[08:13:22] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 08:13:20 UTC 2013
[08:13:22] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[08:13:52] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[08:14:05] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 08:13:55 UTC 2013
[08:14:12] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 08:14:06 UTC 2013
[08:14:12] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 08:14:07 UTC 2013
[08:14:22] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[08:14:22] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[08:14:53] <hashar>	 crazy spam
[08:14:54] <hashar>	 :/
[08:14:54] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[08:20:23] <gerrit-wm>	 Change restored: Hashar; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50064
[08:24:52] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 08:24:48 UTC 2013
[08:25:24] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[08:28:52] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 08:28:50 UTC 2013
[08:29:22] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[08:29:52] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 08:29:44 UTC 2013
[08:29:52] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[08:34:43] <icinga-wm>	 PROBLEM - Parsoid on wtp1009 is CRITICAL: Connection refused
[08:37:31] <gerrit-wm>	 New patchset: Hashar; "beta: adapt role::cache::upload" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50064
[08:38:53] <hashar>	 apergos: and I resurrected a fairly old patches for the upload caches https://gerrit.wikimedia.org/r/#/c/50064/ :D
[08:39:26] <apergos>	 hmm looking
[08:40:15] <hashar>	 I have created a new instance to test it out
[08:40:30] <apergos>	 ok
[08:41:16] <ori-l>	 hi hashar, apergos
[08:41:35] <hashar>	 ori-l: :-)
[08:41:54] <hashar>	 ori-l: were you expecting any reply from me to your "doxygen tags" thread?
[08:42:07] <ori-l>	 no, not really
[08:47:58] <icinga-wm>	 PROBLEM - Parsoid on wtp1018 is CRITICAL: Connection refused
[08:51:46] <gerrit-wm>	 New patchset: Hashar; "beta: adapt role::cache::upload" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50064
[08:52:27] <hashar>	 apergos: the upload cache rebuild is under bug https://bugzilla.wikimedia.org/show_bug.cgi?id=49470  if you want to cc yourself
[08:59:45] <icinga-wm>	 PROBLEM - Parsoid on wtp1007 is CRITICAL: Connection refused
[08:59:45] <icinga-wm>	 PROBLEM - Parsoid on wtp1011 is CRITICAL: Connection refused
[09:00:35] <icinga-wm>	 PROBLEM - Parsoid on wtp1019 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:02:45] <icinga-wm>	 RECOVERY - Parsoid on wtp1009 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.007 second response time
[09:02:48] <gerrit-wm>	 New patchset: Hashar; "beta: adapt role::cache::upload" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50064
[09:08:47] <gerrit-wm>	 New review: Hashar; "Fine :)" [operations/puppet] (production) C: 1;  - https://gerrit.wikimedia.org/r/64462
[09:08:52] <icinga-wm>	 RECOVERY - Parsoid on wtp1018 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.015 second response time
[09:09:51] <icinga-wm>	 RECOVERY - Parsoid on wtp1011 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.004 second response time
[09:10:11] <icinga-wm>	 PROBLEM - Parsoid on wtp1015 is CRITICAL: Connection refused
[09:12:11] <icinga-wm>	 RECOVERY - Parsoid on wtp1015 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.007 second response time
[09:12:41] <icinga-wm>	 RECOVERY - Parsoid on wtp1007 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.011 second response time
[09:13:50] <gerrit-wm>	 New patchset: Hashar; "beta: adapt role::cache::upload" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50064
[09:18:35] <gerrit-wm>	 New patchset: Hashar; "beta: adapt role::cache::upload" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50064
[09:18:41] <icinga-wm>	 PROBLEM - Parsoid on wtp1005 is CRITICAL: Connection refused
[09:30:16] <hashar>	 ah puppet is running
[09:30:41] <icinga-wm>	 RECOVERY - Parsoid on wtp1005 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.004 second response time
[09:31:10] <ori-l>	 scheduled run every 30 mins + billion-line site.pp == puppet is running
[09:32:26] <hashar>	 :-D
[09:32:42] <hashar>	 bah my varnish frontend gives me a 403 ..
[09:36:44] <hashar>	     if (client.ip !~ wikimedia_nets) {
[09:36:46] <hashar>	 ..
[09:38:49] <icinga-wm>	 PROBLEM - Parsoid on wtp1013 is CRITICAL: Connection refused
[09:41:09] <gerrit-wm>	 New review: Mark Bergsma; "This goes against what I was trying to do yesterday in making the storage backends more uniform." [operations/puppet] (production) C: -1;  - https://gerrit.wikimedia.org/r/50064
[09:41:57] <mark>	 ori-l: && puppet code quality
[09:46:06] <hashar>	 oh hello mark :-]
[09:46:11] <mark>	 hello
[09:46:41] <hashar>	 could use an advice for the varnish upload cache.  The backend restrict access to the wikimedia networks (public IP +  10/8).   On labs the backend are accessed locally on 127.0.0.1
[09:46:56] <hashar>	 the varnish::instance class has  wikimedia_networks => $network::constants::all_networks,
[09:47:08] <mark>	 then we should simply add 127.0.0.1 and ::1 to that I think
[09:47:17] <hashar>	 I am wondering how to magically append 127.0.0.1  ;)
[09:48:05] <ori-l>	 hi mark :P
[09:48:09] <mark>	 why magically?
[09:48:19] <gerrit-wm>	 New patchset: Nikerabbit; "ULS config for phase 2" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68146
[09:48:39] <icinga-wm>	 RECOVERY - Parsoid on wtp1013 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.006 second response time
[09:49:29] <gerrit-wm>	 New review: Nikerabbit; "Planned for 2013-06-18." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68146
[09:50:37] <gerrit-wm>	 New patchset: Hashar; "loopback to all_networks" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68147
[09:50:45] <hashar>	 mark: 127/8 :-)  https://gerrit.wikimedia.org/r/68147
[09:51:03] <mark>	 no, not there
[09:51:07] <hashar>	 haha
[09:51:08] <mark>	 i'll submit a patchset
[09:51:18] <hashar>	 sorry
[09:51:43] <hashar>	 another way would be to append 127.0.0.1 to the wikimedia_networks array passed to varnish::instance
[09:52:00] <gerrit-wm>	 Change abandoned: Hashar; "lame idea, mark has a better patch." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68147
[09:55:25] <gerrit-wm>	 New review: Hashar; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50064
[09:55:37] <gerrit-wm>	 New patchset: Mark Bergsma; "Add 127.0.0.0/8 and ::1 to the wikimedia networks Varnish ACLs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68148
[09:55:45] <mark>	 that's what I did
[09:55:53] <hashar>	 ah flatten
[09:56:31] <gerrit-wm>	 New review: Hashar; "thanks!" [operations/puppet] (production) C: 1;  - https://gerrit.wikimedia.org/r/68148
[09:57:21] <hashar>	 need to cleanup https://gerrit.wikimedia.org/r/#/c/50064/13/templates/varnish/upload-backend.inc.vcl.erb,unified  now
[09:57:35] <hashar>	 it has a bunch of if else if  that always  set req.backend = swift
[09:57:46] <hashar>	 not sure where it comes from
[09:58:48] <mark>	 as for those storage backends
[09:58:58] <mark>	 we really need to make beta have two storage backends, just like production
[09:59:02] <mark>	 with the exact same names
[09:59:11] <mark>	 and preferably the same locations, although that's not a strict requirement
[09:59:22] <mark>	 in VCL it should look NO different
[09:59:49] <mark>	 i've done part of that in the text varnish cluster
[09:59:59] <hashar>	 my trouble is that I barely know what our storage backend are
[10:00:04] <hashar>	 so I end up lamely copy pasting :-D
[10:00:05] <mark>	 what do you mean?
[10:00:16] <hashar>	 aren't we storing all files in swift nowadays ?
[10:00:22] <mark>	 huh?
[10:00:26] <mark>	 i'm talking about varnish storage backends
[10:00:35] <mark>	 the horrible vdb stuff
[10:00:47] <hashar>	 sorry
[10:01:07] <hashar>	 I have no idea how that could be fixed :(
[10:01:08] <mark>	 so for text-backend, i named the storage backends main1 and main2
[10:01:25] <mark>	 they used to be called main-sda3 and main-sdb3 before
[10:01:36] <mark>	 we can call them main1 and main2 in production and labs the same
[10:01:42] <mark>	 then varnish VCL refers to them as that
[10:01:44] <gerrit-wm>	 New patchset: Hashar; "beta: adapt role::cache::upload" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50064
[10:01:49] <mark>	 and we don't need to special case beta yet again
[10:02:03] <mark>	 of course this needs to be renamed in production, so it's not very easy
[10:02:06] <mark>	 but we'll do it anyway
[10:02:16] <mark>	 and we'll do it when the new boxes get installed (they arrive today I think)
[10:02:17] <gerrit-wm>	 New review: Hashar; "* rebased to bring https://gerrit.wikimedia.org/r/#/c/68148/ which allow 127.0.0.1" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50064
[10:02:56] <mark>	 so labs/beta only has /srv/vdb
[10:03:03] <mark>	 I would like to store two files in there, for two storage backends
[10:03:06] <mark>	 (or 4 in the case of upload)
[10:03:37] <hashar>	 I can't remember why I have named them differently
[10:03:49] <hashar>	 oh yeah because of the names 'main-sda3'
[10:03:53] <mark>	 yeah
[10:04:06] <mark>	 so let's change that instead of making it more horrible :P
[10:04:28] <hashar>	 I tried to not be intrusive in the production code
[10:04:43] <mark>	 yeah but beta is of no use to me if it's mainly cluttering up the puppet manifests and VCL
[10:04:46] <hashar>	 I am not confident in refactoring that for prod :(
[10:04:53] <mark>	 I'll help with that
[10:05:31] <mark>	 like I said, we're getting new boxes installed, so that's a good and easy opportunity to change it
[10:05:52] <mark>	 for text we can already do it today
[10:05:57] <mark>	 those are not in production yet
[10:06:02] <mark>	 upload and mobile are a bit harder
[10:06:26] <gerrit-wm>	 New review: Hashar; "Expected CNUM got ';'" [operations/puppet] (production) C: -1;  - https://gerrit.wikimedia.org/r/68148
[10:06:55] <mark>	 i'll take a first stab at it today
[10:07:34] <hashar>	 I got to take a look at the text cache in beta
[10:07:43] <hashar>	 I started with upload to clean it up for Ariel
[10:07:43] <mark>	 that's only part of it though
[10:07:45] <mark>	 we need more
[10:08:43] <gerrit-wm>	 New patchset: Hashar; "Add 127.0.0.0/8 and ::1/128 to the wikimedia networks Varnish ACLs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68148
[10:08:54] <mark>	 why /128?
[10:09:01] <mark>	 that's implicit
[10:09:02] <gerrit-wm>	 New review: Hashar; "added /128 to ::1. Varnish was complaining :/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68148
[10:09:12] <mark>	 strange
[10:09:13] <hashar>	 it complained with: Expected CNUM got ';'
[10:09:13] <hashar>	 (program line 307), at
[10:09:15] <hashar>	 ('input' Line 20 Pos 15)
[10:09:16] <hashar>	         "::1"/;
[10:09:16] <hashar>	 --------------#
[10:09:19] <mark>	 ok
[10:09:33] <mark>	 that's probably a bug in our vcl templates
[10:09:34] <hashar>	 with 3.0.3plus~rc1-wm11
[10:09:38] <mark>	 not varnish
[10:09:45] <hashar>	         "::1"/;
[10:09:45] <mark>	 anyway
[10:09:47] <mark>	 back after lunch
[10:09:49] <hashar>	 yeah that does not look nice :-)
[10:09:52] <hashar>	 have a good lunch
[10:10:58] <gerrit-wm>	 New patchset: Hashar; "beta: adapt role::cache::upload" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50064
[10:16:07] <icinga-wm>	 PROBLEM - Parsoid on wtp1010 is CRITICAL: Connection refused
[10:22:07] <icinga-wm>	 RECOVERY - Parsoid on wtp1010 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.005 second response time
[10:25:26] <gerrit-wm>	 New review: Nemo bis; "Config good, but not that phase 1 is not completed yet, as communication for next phase has not been..." [operations/mediawiki-config] (master) C: 1;  - https://gerrit.wikimedia.org/r/68146
[10:26:09] <icinga-wm>	 PROBLEM - Parsoid on wtp1023 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:28:37] <icinga-wm>	 PROBLEM - Parsoid on wtp1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:31:16] <hashar>	 nap time
[10:42:00] <icinga-wm>	 PROBLEM - Parsoid on wtp1003 is CRITICAL: Connection refused
[10:49:55] <gerrit-wm>	 New patchset: Mark Bergsma; "Rename text storage backends, add a second one in beta" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68149
[10:50:18] <gerrit-wm>	 New patchset: Mark Bergsma; "Rename text storage backends, add a second one in beta" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68149
[10:51:12] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68148
[10:53:13] <gerrit-wm>	 New review: Liangent; "I like it, but it seems some other people from zhwiki don't really do so." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68146
[10:56:34] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68149
[11:01:44] <gerrit-wm>	 New patchset: Mark Bergsma; "Prevent Puppet from making the Varnish storage files" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68151
[11:02:33] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68151
[11:09:09] <icinga-wm>	 RECOVERY - Parsoid on wtp1003 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.009 second response time
[11:15:49] <icinga-wm>	 PROBLEM - Parsoid on wtp1024 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:18:29] <icinga-wm>	 PROBLEM - RAID on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[11:19:20] <icinga-wm>	 RECOVERY - RAID on mc15 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0
[11:22:38] <gerrit-wm>	 New patchset: Mark Bergsma; "Rename upload storage backends to be device name independent" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68153
[11:24:31] <gerrit-wm>	 New review: Mark Bergsma; "This needs restarts of all backends" [operations/puppet] (production) C: -2;  - https://gerrit.wikimedia.org/r/68153
[11:27:40] <gerrit-wm>	 New patchset: Mark Bergsma; "Rename Parsoid storage backends, add a second storage backend in beta" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68155
[11:29:17] <gerrit-wm>	 Change abandoned: Mark Bergsma; "The storage backends are currently on inconsistent partitions. Let's fix on the new servers." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68155
[11:31:33] <gerrit-wm>	 New patchset: Mark Bergsma; "Rename Parsoid storage backends, add a second storage backend in beta" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68156
[11:32:12] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68156
[11:34:21] <gerrit-wm>	 New patchset: Mark Bergsma; "Rename Mobile storage backends, add a second storage backend in beta" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68158
[11:34:56] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68158
[11:36:39] <icinga-wm>	 PROBLEM - Parsoid on wtp1016 is CRITICAL: Connection refused
[11:42:00] <icinga-wm>	 PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours
[11:42:01] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours
[11:42:01] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours
[11:42:01] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours
[11:42:01] <icinga-wm>	 PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours
[11:42:01] <icinga-wm>	 PROBLEM - Puppet freshness on ms-be1 is CRITICAL: No successful Puppet run in the last 10 hours
[11:42:01] <icinga-wm>	 PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours
[11:42:02] <icinga-wm>	 PROBLEM - Puppet freshness on mw1020 is CRITICAL: No successful Puppet run in the last 10 hours
[11:42:02] <icinga-wm>	 PROBLEM - Puppet freshness on sockpuppet is CRITICAL: No successful Puppet run in the last 10 hours
[11:42:03] <icinga-wm>	 PROBLEM - Puppet freshness on spence is CRITICAL: No successful Puppet run in the last 10 hours
[11:42:03] <icinga-wm>	 PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours
[11:42:04] <icinga-wm>	 PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours
[11:42:04] <icinga-wm>	 PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours
[11:48:09] <icinga-wm>	 PROBLEM - Parsoid on wtp1017 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:48:59] <icinga-wm>	 RECOVERY - Parsoid on wtp1017 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.002 second response time
[11:53:28] <hashar>	 re
[11:53:39] <icinga-wm>	 PROBLEM - Parsoid on wtp1014 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:54:29] <icinga-wm>	 RECOVERY - Parsoid on wtp1014 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.008 second response time
[11:54:39] <icinga-wm>	 RECOVERY - Parsoid on wtp1016 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.005 second response time
[11:58:59] <icinga-wm>	 PROBLEM - Parsoid on wtp1017 is CRITICAL: Connection refused
[12:01:59] <icinga-wm>	 RECOVERY - Parsoid on wtp1017 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.006 second response time
[12:03:47] <gerrit-wm>	 New patchset: Mark Bergsma; "Rename $storage_backends var to the more descriptive $storage_partitions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68162
[12:04:48] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68162
[12:05:29] <icinga-wm>	 PROBLEM - Parsoid on wtp1014 is CRITICAL: Connection refused
[12:06:45] <gerrit-wm>	 New patchset: Mark Bergsma; "Cleanup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68163
[12:06:51] <mark>	 hashar: so I've added a bunch more storage backends in beta
[12:07:07] <mark>	 I didn't test it, restart anything, and also they might run out of space ;)
[12:07:23] <hashar>	 :D
[12:08:07] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68163
[12:09:42] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 12:09:34 UTC 2013
[12:10:04] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 12:09:58 UTC 2013
[12:10:12] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[12:10:12] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 12:10:06 UTC 2013
[12:10:12] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[12:10:14] <hashar>	 I am triggering to figure out what is wrong with my new upload cache instance :D
[12:10:43] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[12:12:22] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 12:12:15 UTC 2013
[12:12:32] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 12:12:27 UTC 2013
[12:12:42] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 12:12:32 UTC 2013
[12:12:42] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[12:13:12] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[12:13:12] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[12:13:51] <hashar>	 ahh
[12:14:42] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 12:14:41 UTC 2013
[12:14:52] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 12:14:51 UTC 2013
[12:15:14] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[12:15:22] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 12:15:13 UTC 2013
[12:15:42] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[12:16:12] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[12:16:12] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 12:16:08 UTC 2013
[12:16:22] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 12:16:17 UTC 2013
[12:16:22] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 12:16:19 UTC 2013
[12:16:43] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[12:17:13] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[12:17:14] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[12:17:34] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 12:17:29 UTC 2013
[12:17:34] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 12:17:30 UTC 2013
[12:17:42] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 12:17:35 UTC 2013
[12:17:42] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[12:18:12] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[12:18:12] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[12:18:32] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 12:18:23 UTC 2013
[12:18:33] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 12:18:29 UTC 2013
[12:18:33] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 12:18:31 UTC 2013
[12:18:42] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[12:19:12] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[12:19:12] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[12:19:22] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 12:19:12 UTC 2013
[12:19:22] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 12:19:18 UTC 2013
[12:19:23] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 12:19:20 UTC 2013
[12:19:42] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[12:20:09] <gerrit-wm>	 New patchset: Mark Bergsma; "Let's clutter up the old crap instead" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68164
[12:20:12] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[12:20:12] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[12:23:14] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68164
[12:25:32] <icinga-wm>	 PROBLEM - RAID on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[12:25:49] <hashar>	 apergos: I am tweaking the beta apache configs a bit
[12:25:52] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 12:25:48 UTC 2013
[12:26:12] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[12:26:20] <apergos>	 ok
[12:26:33] <icinga-wm>	 RECOVERY - Parsoid on wtp1014 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.005 second response time
[12:27:32] <icinga-wm>	 RECOVERY - RAID on mc15 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0
[12:28:52] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 12:28:45 UTC 2013
[12:29:12] <icinga-wm>	 PROBLEM - Parsoid on wtp1006 is CRITICAL: Connection refused
[12:29:42] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[12:29:52] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 12:29:44 UTC 2013
[12:30:12] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[12:30:45] <AzaToth>	 could someone kick icinga-wm
[12:30:49] <AzaToth>	 in da butt
[12:31:04] <gerrit-wm>	 New patchset: Mark Bergsma; "Remove the now pointless variable $varnish_fe_directors" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68166
[12:32:26] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68166
[12:34:54] <hashar>	 AzaToth: na we need it :D
[12:35:32] <AzaToth>	 hashar: just the puppet freshness is pretty irrelevant
[12:35:41] <AzaToth>	 escpecially as it can't count
[12:36:17] <hashar>	 then it need to be fixed
[12:36:38] <AzaToth>	 which is what I meant by kicking it in it's but
[12:36:43] <mark>	 its
[12:36:48] <AzaToth>	 whatevah
[12:37:04] <hashar>	 AzaToth: that is not fixing anything, it just hiding garbage under the carpet
[12:37:07] <AzaToth>	 mark die grammar nazi
[12:37:34] <AzaToth>	 hashar: anyone knows what's broken?
[12:38:05] <AzaToth>	 it's like it forgets what it just said
[12:38:22] <AzaToth>	 or that the clock is off a day or two
[12:42:48] <gerrit-wm>	 New patchset: Matthias Mullie; "(bug 49468) ArticleFeedback box shouldn't show on help pages on Portuguese Wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68167
[12:42:52] <gerrit-wm>	 New patchset: Mark Bergsma; "Make $varnish_be_directors tier based, not site (for text)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68168
[12:45:42] <gerrit-wm>	 New patchset: Mark Bergsma; "Make backend directors tier based, not site (for text)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68168
[12:46:00] <hashar>	 somehow my varnish cache does queries missing a  /  after the protocol :   http:/upload ….
[12:46:01] <hashar>	 :D
[12:48:58] <gerrit-wm>	 Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68168
[12:49:31] <gerrit-wm>	 New patchset: Hashar; "beta: adapt role::cache::upload" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50064
[12:49:43] <mark>	 ha
[12:49:48] <mark>	 so you can't remove patchsets in gerrit?
[12:50:05] <mark>	 i submitted a 2nd patchset, then decided the 1st one was better and wanted to merge that
[12:50:10] <mark>	 seems not possible
[12:50:48] <hashar>	 na you need to cherry pick the first and resend it :/
[12:53:10] <icinga-wm>	 RECOVERY - Parsoid on wtp1006 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.013 second response time
[12:53:15] <mark>	 then it's still not happy
[12:53:23] <mark>	 need to change the change-id
[12:53:38] <mark>	 oh well
[12:53:41] <gerrit-wm>	 New patchset: Mark Bergsma; "Make $varnish_be_directors tier based, not site (for text)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68169
[12:54:36] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68169
[12:56:19] <gerrit-wm>	 New patchset: Akosiaris; "Debianize Kafka" [operations/debs/kafka] (debian) - https://gerrit.wikimedia.org/r/68026
[13:01:42] <hashar>	 mark: so I get a bug in the upload frontend.  It some how drop a slash from the requested url :/
[13:02:02] <hashar>	 if I query the backend (listening on port 3128)  everything goes fine
[13:02:15] <hashar>	 varnisnsca  shows that the frontend query the wrong url
[13:02:47] <gerrit-wm>	 New review: Akosiaris; "So PS3 fixes:" [operations/debs/kafka] (debian) - https://gerrit.wikimedia.org/r/68026
[13:05:11] <hashar>	     if ( req.http.host == "upload.beta.wmflabs.org") {
[13:05:12] <hashar>	         set req.url = regsuball(req.url, "/{2,}", "/");
[13:05:12] <hashar>	 in upload-frontend.inc.vcl  , wondering why we do that
[13:06:24] <gerrit-wm>	 New patchset: Mark Bergsma; "Put the Text Varnish code in a separate sub class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68170
[13:06:31] <icinga-wm>	 PROBLEM - Parsoid on wtp1022 is CRITICAL: Connection refused
[13:07:04] <mark>	 hashar: to prevent double slashes in the URI from messing with the regsubsts?
[13:07:32] <hashar>	 that cause the frontend cache to rewrite http://upload….  to  http:/upload...
[13:07:36] <hashar>	 which then confuse the apache backend
[13:07:57] <hashar>	 apache gives me:  "GET http:/upload….  404
[13:07:57] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68170
[13:08:16] <mark>	 no it shouldn't
[13:08:23] <mark>	 the hostname is not part of req.url
[13:09:11] <mark>	 unless, perhaps, if it's sent as a HTTP proxy request
[13:09:20] <hashar>	 ahhh
[13:09:32] <hashar>	 curl -i -x 10.4.0.252:80 "http://upload.beta.wmflabs.org/wikipedia/en/a/ab/Polar_bear.jpeg?`date +%s`"
[13:09:35] <hashar>	 that is what I use :D
[13:09:39] <mark>	 that would do it
[13:10:12] <hashar>	 yeah that fix it
[13:10:42] <hashar>	 spent like an hour figuring that out :(
[13:10:58] <hashar>	 apergos: I got a patch for the varnish upload cache https://gerrit.wikimedia.org/r/#/c/50064/
[13:11:11] <apergos>	 ok I'll look in a little bit
[13:11:23] <hashar>	 I have it set up on uploadtest07.pmtpa.wmflabs  instance
[13:11:39] <hashar>	  upload.beta.wmflabs.org  points to the instance IP
[13:12:05] <mark>	 that's not going in like that
[13:12:08] <hashar>	 running curl gives me something :-]
[13:12:21] <hashar>	 now I need to update the storage objects names
[13:12:30] <mark>	 yeah
[13:14:04] <mark>	 and you can probably rewrite those regexes to not depend on the upload hostnames at all
[13:15:30] <hashar>	 mark:  you did the storage rename already
[13:15:30] <hashar>	 https://gerrit.wikimedia.org/r/#/c/68153/
[13:15:36] <hashar>	 so I should "just" rebase on top of that ?
[13:15:45] <mark>	 yeah but that's not going in for the next couple of days
[13:21:43] <gerrit-wm>	 New patchset: Mark Bergsma; "First attempt at refactoring role::cache Varnish clusters" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68171
[13:22:27] <gerrit-wm>	 New patchset: Hashar; "beta: adapt role::cache::upload" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50064
[13:22:42] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68171
[13:22:58] <mark>	 if I were you I'd holdoff for a few days
[13:23:03] <mark>	 as I'm going to refactor tons of stuff ;)
[13:23:05] <gerrit-wm>	 New review: Hashar; "Rebased on https://gerrit.wikimedia.org/r/#/c/68153/1 which makes the varnish storage backends name ..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50064
[13:23:12] <mark>	 trying to keep up with that for beta will be madness
[13:23:31] <hashar>	 cant you refactor after the patch lands ?
[13:23:32] <hashar>	 :D
[13:23:49] <hashar>	 this way I can upgrade the upload cache to precise and have it running out of puppet
[13:23:56] <hashar>	 that is currently a lucid instance with puppetmaster::self
[13:23:59] <mark>	 i don't like your patch as it currently is
[13:24:47] <hashar>	 well with the new storage backends name it is a bit simpler
[13:24:59] <mark>	 yes, that fixes that
[13:25:11] <gerrit-wm>	 New patchset: ArielGlenn; "make media storage in beta labs more like production" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68172
[13:25:46] <hashar>	 and there is now only one occurrence of 'vdb' :-)
[13:26:57] <gerrit-wm>	 New review: ArielGlenn; "this needs changes to deployment-cache-upload03 at the same time:" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68172
[13:27:02] <gerrit-wm>	 New review: Faidon; "Views are for collecting metrics over several hosts and presenting them in a single dashboard. That ..." [operations/puppet] (production) C: -1;  - https://gerrit.wikimedia.org/r/68044
[13:29:11] <gerrit-wm>	 New patchset: Mark Bergsma; "Move inclusion of standard/nrpe back to fix ganglia/nagios" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68173
[13:29:15] <gerrit-wm>	 New review: Hashar; "I wanted to regroup the most important contint metrics and though a ganglia view would be nice.  An ..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68044
[13:29:47] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68173
[13:30:02] <hashar>	 apergos: and I just discovered the upload cache is a squid :-)
[13:30:31] <icinga-wm>	 RECOVERY - Parsoid on wtp1022 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.003 second response time
[13:33:07] <gerrit-wm>	 New review: Hashar; "Go ahead :-)" [operations/mediawiki-config] (master) C: 1;  - https://gerrit.wikimedia.org/r/68172
[13:37:42] <hashar>	 paravoid: so should i just drop my ganglia view ? :D
[13:38:39] <paravoid>	 I dunno, it doesn't make much sense to me tbh
[13:43:51] <gerrit-wm>	 New review: Hashar; "So this is on hold till mark deploy the dependent change which is require all caches to be restarted." [operations/puppet] (production) C: -1;  - https://gerrit.wikimedia.org/r/50064
[13:44:00] <gerrit-wm>	 New review: Matthias Mullie; "Not to be merged before https://bugzilla.wikimedia.org/show_bug.cgi?id=49468 is cleared up" [operations/mediawiki-config] (master) C: -2;  - https://gerrit.wikimedia.org/r/68167
[13:44:18] <gerrit-wm>	 New patchset: Mark Bergsma; "Move upload varnish to a separate class, and inherit from varnish::two-tier" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68175
[13:45:13] <hashar>	 paravoid: will craft my own view on integration.wikimedia.org :-)
[13:45:54] <gerrit-wm>	 Change abandoned: Hashar; "Per discussion with Faidon, I will use a simple HTML page including Ganglia graphs and host it under..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68044
[13:46:20] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68175
[13:46:26] <paravoid>	 mark: iirc dash name in classes results in some weird puppet behavior
[13:47:05] <mark>	 i use them all the time
[13:47:59] <paravoid>	 http://docs.puppetlabs.com/puppet/3/reference/lang_reserved.html#classes-and-types
[13:48:05] <paravoid>	 unsupported in docs
[13:48:27] <paravoid>	 https://projects.puppetlabs.com/issues/5268#change-65981
[13:48:56] <paravoid>	 bleh
[13:49:06] <mark>	 i rest my puppet case ;p
[13:51:39] <mark>	 		$varnish_be_directors = {
[13:51:39] <mark>	 			"pmtpa" => {
[13:51:39] <mark>	 				"backend" => $lvs::configuration::lvs_service_ips[$::realm]['upload']['pmtpa']['uploadsvc'],
[13:51:44] <mark>	 this doesn't make a lot of sense to me
[13:52:09] <mark>	 so varnish upload backend caches in pmtpa, if they existed (well, beta), connect to the pmtpa upload cache cluster?
[13:52:45] <mark>	 I think i'll just change that to be the same as eqiad, i.e. swift
[13:53:48] <gerrit-wm>	 New review: Ottomata; "> CLASSPATH doesn't really belong in default, looks like init script material." [operations/debs/kafka] (debian) - https://gerrit.wikimedia.org/r/68026
[13:54:54] <gerrit-wm>	 New patchset: Mark Bergsma; "Make upload varnish $varnish_be_directors $cluster_tier based" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68177
[13:54:58] <mark>	 check that ^
[14:18:57] <hashar>	 :-)
[14:18:59] <paravoid>	 who, me?
[14:22:14] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68177
[14:25:39] <gerrit-wm>	 New review: Catrope; "You can't allocate 2x19GB for .persist files when the labs instance this runs on only has 20GB of st..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68156
[14:26:10] <mark>	 actually you can
[14:26:20] <mark>	 it will just break once it fills them up
[14:31:42] <mark>	 hey
[14:31:45] <mark>	 so varnish supports percentages
[14:31:52] <mark>	 why don't we just set a 50% (or slightly below that) for labs instances
[14:32:55] <gerrit-wm>	 Change merged: Andrew Bogott; [operations/software] (master) - https://gerrit.wikimedia.org/r/67898
[14:33:15] <mark>	 damn, not supported for persistent
[14:33:23] <andrewbogott>	 hashar, we can merge and deploy https://gerrit.wikimedia.org/r/#/c/68012/ if it looks right to you.
[14:35:29] <hashar>	 andrewbogott: let me check operations/software status :)
[14:36:45] <hashar>	 andrewbogott: there is still geturls/geturls that does not pass pep8
[14:36:49] <paravoid>	 AzaToth: heya
[14:37:16] <andrewbogott>	 oh, missing the .py.  OK, I'll fix that one
[14:37:27] <paravoid>	 AzaToth: I've kinda lost track, anything you need help with or someone to brainstorm with?
[14:37:45] <hashar>	 andrewbogott: well my pep8 version detects it since   'geturls' is listed in  filename
[14:37:54] <hashar>	 andrewbogott: we could even rename it to use a .py  :)
[14:38:12] <andrewbogott>	 I don't think it matters in this case
[14:42:41] <gerrit-wm>	 New patchset: Andrew Bogott; "Pep8 cleanup for geturls." [operations/software] (master) - https://gerrit.wikimedia.org/r/68180
[14:43:46] <hashar>	 ahh nice bug
[14:43:55] <hashar>	 there is nothing being triggered for that change
[14:44:05] <hashar>	 since there is no python file being changed :-)
[14:44:27] <icinga-wm>	 PROBLEM - Parsoid on wtp1001 is CRITICAL: Connection refused
[14:44:43] <hashar>	 and zuul enqueued a change with no jobs associated to it
[14:45:29] <icinga-wm>	 RECOVERY - Parsoid on wtp1001 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.003 second response time
[14:45:43] <AzaToth>	 paravoid: as you perhaps noticed I removed half of the deps, and I think it's fine as long we only use it to build gerrit, but offcourse I would need someone to confirm that
[14:46:20] <paravoid>	 ^demon would probabl know
[14:46:39] <AzaToth>	 https://gerrit.wikimedia.org/r/#/c/67999/ for reference
[14:46:45] <AzaToth>	 ^demon: pokipinkypine
[14:46:47] <gerrit-wm>	 New patchset: Hashar; "Pep8 cleanup for geturls and rename to geturls.py" [operations/software] (master) - https://gerrit.wikimedia.org/r/68180
[14:46:56] <^demon>	 AzaToth: What sort of dependencies were removed?
[14:47:02] <^demon>	 I'm just curious...
[14:47:04] <AzaToth>	 ^demon: https://gerrit.wikimedia.org/r/#/c/67999/1..2/debian/install
[14:47:18] <hashar>	 andrewbogott: I have renamed the script to geturls.py
[14:47:27] <paravoid>	 ^demon: not sure if you heard already, AzaToth is working on properly packaging gerrit
[14:47:31] <gerrit-wm>	 New review: Hashar; "good to me :-)" [operations/software] (master) C: 1;  - https://gerrit.wikimedia.org/r/68180
[14:47:42] <^demon>	 paravoid: I heard a rumbling :)
[14:47:43] <andrewbogott>	 that's one way to work around the zuul bug :)
[14:47:51] <hashar>	 will log a bug upstream
[14:47:52] <paravoid>	 maybe we'll even upload it to Debian if it's not too messy
[14:47:57] <AzaToth>	 ^demon: http://paste.debian.net/9714 thats what I found was availabel in debian/ubuntu
[14:48:21] <^demon>	 AzaToth: Gerrit's the only thing I ever planned to use buck for, so dropping useless dependencies is fine by me as long as everything builds :)
[14:48:28] <AzaToth>	 paravoid: well, as long it must be build with buck, I dont think it's possible to upload it to debian
[14:48:39] <gerrit-wm>	 New patchset: Mark Bergsma; "Use role::cache::varnish::two-tier as role::cache::parsoid ancestor" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68182
[14:48:41] <paravoid>	 so, buck is a build system, right?
[14:48:42] <AzaToth>	 not in the near time
[14:48:48] <AzaToth>	 paravoid: yes, for android
[14:48:58] <paravoid>	 why does it embed jython and whanot?
[14:49:01] <AzaToth>	 paravoid: http://facebook.github.io/buck/
[14:49:07] <AzaToth>	 becuase facebook has made it
[14:49:26] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68182
[14:49:27] <AzaToth>	 it actually spawns a python script to actually parse the BUCK files
[14:49:41] <paravoid>	 oh god
[14:49:52] <AzaToth>	 paravoid: hehe
[14:49:54] <paravoid>	 so, wait
[14:50:18] <paravoid>	 gerrit, a google project made to review code for android, uses a build system written by facebook, that was designed to build android, a google project?
[14:50:22] <andrewbogott>	 hashar, is that a tool that's only run by hand?  No need to change callers anywhere due to the rename?
[14:50:32] <^demon>	 paravoid: That's about right, yes.
[14:50:32] <hashar>	 oops
[14:50:36] <AzaToth>	 https://groups.google.com/forum/?fromgroups#!topic/repo-discuss/Xrp-HfkkBxw
[14:50:52] <hashar>	 andrewbogott: I am assuming it s a standalone tool :)
[14:51:30] <AzaToth>	 ^demon: if you want, a look into https://gerrit.wikimedia.org/r/#/c/67999/ is wanted from my part
[14:51:46] <gerrit-wm>	 Change merged: Andrew Bogott; [operations/software] (master) - https://gerrit.wikimedia.org/r/68180
[14:51:59] <AzaToth>	 it looked like it built it, but I never tested gerrit from the build
[14:52:04] <paravoid>	 okay, naive question: do we have to use buck?
[14:52:16] <^demon>	 paravoid: They ripped out maven support with a crowbar, so yes :(
[14:52:17] <paravoid>	 can't we just build gerrit ourselves via other means?
[14:52:37] <paravoid>	 fucking java people and their horrible terrible build systems
[14:52:39] <AzaToth>	 paravoid: gerrit commit: http://paste.debian.net/9926/
[14:52:54] <gerrit-wm>	 New patchset: Bene; "Bug 48354: exclude MediaWiki: namespace" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68183
[14:53:18] <paravoid>	 (no offence ^demon :)
[14:53:19] <AzaToth>	 so yes
[14:54:03] <gerrit-wm>	 New patchset: Mark Bergsma; "Use role::cache::varnish::two-tier as role::cache::mobile ancestor" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68184
[14:54:30] <AzaToth>	 paravoid: I would never be so evil of introducing buck to debian
[14:55:13] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68184
[14:56:30] <mark>	 ruby and java
[14:58:55] <paravoid>	 i know, ugh
[14:59:36] <andrewbogott>	 mark, before you punch out can you have a look at https://gerrit.wikimedia.org/r/68006 and https://gerrit.wikimedia.org/r/68011?  The first is trivial, the second will require some spoon-feeding I fear.
[15:01:14] <gerrit-wm>	 Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68006
[15:02:42] <gerrit-wm>	 New review: Mark Bergsma; "Looks good on first glance, but I only skimmed over it. I can review a bit more thoroughly tomorrow ..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68011
[15:03:21] <andrewbogott>	 mark:  thanks.  I'll see if I mutante can help me actually run a test.
[15:04:00] <mark>	 i'd say, diff the two expanded configs
[15:04:08] <mark>	 there are bound to be some differences that aren't relevant
[15:04:22] <mark>	 and it does help to have some exim knowledge for that
[15:04:46] <mark>	 i'm happy to help, but I need to run out soon
[15:05:13] <gerrit-wm>	 New review: Ottomata; "+log.dir=/var/lib/kafka/	" [operations/debs/kafka] (debian) - https://gerrit.wikimedia.org/r/68026
[15:05:44] <andrewbogott>	 Yeah, I have the expanded configs and there are a ton of diffs.
[15:05:50] <andrewbogott>	 I'll link them to the patch.
[15:08:27] <icinga-wm>	 RECOVERY - Parsoid on wtp1023 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.004 second response time
[15:08:27] <icinga-wm>	 RECOVERY - Parsoid on wtp1019 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.008 second response time
[15:08:37] <icinga-wm>	 RECOVERY - Parsoid on wtp1004 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.002 second response time
[15:08:37] <icinga-wm>	 RECOVERY - Parsoid on wtp1024 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.003 second response time
[15:08:54] <gerrit-wm>	 New patchset: Mark Bergsma; "Move ganglia monitoring into role::cache::varnish::two-tier" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68187
[15:08:58] <gerrit-wm>	 New review: Faidon; "I'm not entirely sure about what kafka does with the logdir. If it's just spooling data for consumpt..." [operations/debs/kafka] (debian) - https://gerrit.wikimedia.org/r/68026
[15:08:59] <mark>	 ok
[15:09:11] <gwicke>	 !log updated Parsoid to 058da0822
[15:09:21] <morebots>	 Logged the message, Master
[15:09:49] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68187
[15:16:59] <gerrit-wm>	 New patchset: AzaToth; "Initial debian build" [operations/debs/buck] (master) - https://gerrit.wikimedia.org/r/67999
[15:18:29] <AzaToth>	 ottomata: did you solve all java deps in kafka?
[15:18:51] <paravoid>	 AzaToth: akosiaris wrote Makefiles from scratch
[15:18:53] <ottomata>	 ha
[15:18:54] <ottomata>	 yeah
[15:18:55] <cmjohnson1>	 !log mw1041 powering down for h/w issue
[15:19:08] <morebots>	 Logged the message, Master
[15:19:08] <ottomata>	 he avoided using sbt (or maven, etc)
[15:19:13] <paravoid>	 and ditched sbt, which from what I'm hearing must be of very similar quality with buck
[15:19:13] <AzaToth>	 sbt?
[15:19:15] <ottomata>	 was able to link against some existing .deb java packages
[15:19:19] <paravoid>	 a scala build system
[15:19:22] <AzaToth>	 k
[15:19:27] <ottomata>	 (scala build tool, kafka is written in scala)
[15:19:36] <ottomata>	 but for those that didin't exist in debian/ubuntu
[15:19:37] <paravoid>	 scala being a JVM-based language fwiw
[15:19:41] <AzaToth>	 http://paste.debian.net/9930/
[15:19:45] <paravoid>	 AzaToth: https://gerrit.wikimedia.org/r/#/c/68026/
[15:19:47] <ottomata>	 we committed the .jars directly to the repo
[15:20:33] * paravoid  grumbles at the lost PS history

[15:20:50] <icinga-wm>	 PROBLEM - Host mw1041 is DOWN: PING CRITICAL - Packet loss = 100%
[15:21:12] <AzaToth>	 ottomata: buck repo is 50MiB and convenience jars are 32 MiB
[15:21:59] <AzaToth>	 it made an 28MiB gerrit.war
[15:22:03] <AzaToth>	 seems good
[15:22:09] <paravoid>	 fsvo goof
[15:22:11] <paravoid>	 good
[15:22:15] <AzaToth>	 fsvo?
[15:22:41] <paravoid>	 for some value of
[15:23:01] <AzaToth>	 ok
[15:23:52] <akosiaris>	 ahahaha
[15:25:48] <ottomata>	 aye cool
[15:31:11] <gerrit-wm>	 New patchset: Odder; "(bug 49328) Create Wikipedia in Tyvan language" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68188
[15:31:23] <icinga-wm>	 RECOVERY - Host mw1041 is UP: PING OK - Packet loss = 0%, RTA = 0.60 ms
[15:33:20] <icinga-wm>	 PROBLEM - DPKG on mw1041 is CRITICAL: Connection refused by host
[15:33:30] <icinga-wm>	 PROBLEM - twemproxy process on mw1041 is CRITICAL: Connection refused by host
[15:33:30] <icinga-wm>	 PROBLEM - Disk space on mw1041 is CRITICAL: Connection refused by host
[15:33:40] <icinga-wm>	 PROBLEM - RAID on mw1041 is CRITICAL: Connection refused by host
[15:36:47] <gerrit-wm>	 New patchset: Ottomata; "RT 5161 - erosen access to stat1001" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68189
[15:37:03] <gerrit-wm>	 Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68189
[15:38:15] <gerrit-wm>	 New patchset: Akosiaris; "Debianize Kafka" [operations/debs/kafka] (debian) - https://gerrit.wikimedia.org/r/68026
[15:42:34] <gerrit-wm>	 New review: Akosiaris; "Patch Set 4 solves:" [operations/debs/kafka] (debian) - https://gerrit.wikimedia.org/r/68026
[15:43:48] <akosiaris>	 ottomata: dare I say that with this we are almost done ?
[15:44:34] <ottomata>	 dare we might be!
[15:44:35] <ottomata>	 paravoid
[15:44:44] <ottomata>	 log.dir is kind of a spool, i guess, its more like the kafka data
[15:44:50] <ottomata>	 its a buffer
[15:45:32] <paravoid>	 if it's a buffer, it's a spool, no?
[15:45:41] <paravoid>	 akosiaris: I didn't exactly get the answer to CLASSPATH in d/default
[15:45:44] <paravoid>	 that's certainly wrong
[15:46:16] <akosiaris>	 paravoid: is is also used by /usr/bin/kafka command
[15:46:22] <paravoid>	 it shouldn't
[15:46:26] <akosiaris>	 and not just init script
[15:46:28] <paravoid>	 default is for the init script, period
[15:46:40] <icinga-wm>	 PROBLEM - NTP on mw1041 is CRITICAL: NTP CRITICAL: No response from NTP server
[15:47:18] <akosiaris>	 so what ? we keep it in 2 places with the danger of it running out-of-sync ?
[15:47:51] <paravoid>	 why do we need it in both?
[15:49:06] <akosiaris>	 well the init script i believe it is quite obvious why
[15:49:32] <akosiaris>	 the command is a wrapper for some java classes
[15:49:53] <akosiaris>	 which depend pretty much on the same CLASSPATH
[15:50:43] <ottomata>	 hm, welp, akosiaris, in my 0.7.2 effort
[15:51:00] <ottomata>	 i had the init script using /usr/sbin/kafka to start the broker
[15:51:11] <ottomata>	 which meant that the CLASSPATh was only in one place
[15:51:13] <ottomata>	 and not in default
[15:52:56] <ottomata>	 that also avoids issues with special JMX_PORT settings, because then they are all provided by default per service in /usr/sbin/kafka, not in etc/default/kafka
[15:53:07] <akosiaris>	 yeah but the init script we got now is a lot better that the start_stop commands in kafka command.
[15:53:16] <akosiaris>	 For starters it actually daemonizes...
[15:53:39] <ottomata>	 yeah, but I used upstart
[15:53:45] <ottomata>	 which deals with that
[15:55:39] <ottomata>	 and paravoid, i think I agree
[15:55:45] <ottomata>	 log.dir should be /var/spool/kafka
[15:55:55] <paravoid>	 I'm not sure even if *I* agree :)
[15:56:05] <paravoid>	 I'm not sure what exactly that data is
[15:57:11] <ottomata>	 its the event message data
[15:57:14] <ottomata>	 whatever gets produced to kafka
[15:57:21] <ottomata>	 is stored there in whatever the kafka binary format is
[15:57:47] <ottomata>	 but, it isn't deleted when it is consumed
[15:57:54] <ottomata>	 it keeps a sliding window of backlogs
[15:57:57] <ottomata>	 by default 1 week
[15:59:03] <icinga-wm>	 PROBLEM - search indices - check lucene status page on search1020 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 58845 bytes in 0.014 second response time
[15:59:03] <icinga-wm>	 PROBLEM - swift-account-auditor on ms-be1 is CRITICAL: Connection refused by host
[15:59:03] <icinga-wm>	 PROBLEM - swift-object-updater on ms-be1 is CRITICAL: Connection refused by host
[15:59:03] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[15:59:03] <icinga-wm>	 PROBLEM - search indices - check lucene status page on search1018 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 60534 bytes in 0.014 second response time
[15:59:04] <icinga-wm>	 PROBLEM - Disk space on virt4 is CRITICAL: Connection refused by host
[15:59:04] <icinga-wm>	 PROBLEM - RAID on mw1020 is CRITICAL: Connection refused by host
[15:59:13] <icinga-wm>	 PROBLEM - NTP on spence is CRITICAL: NTP CRITICAL: No response from NTP server
[15:59:25] <paravoid>	 hrm
[15:59:28] <paravoid>	 so, borderline
[15:59:57] <cmjohnson1>	 !log powering down carbon for h/w testing/reports
[16:00:05] <morebots>	 Logged the message, Master
[16:00:13] <icinga-wm>	 PROBLEM - Host mw1041 is DOWN: PING CRITICAL - Packet loss = 100%
[16:02:33] <icinga-wm>	 PROBLEM - Host carbon is DOWN: CRITICAL - Host Unreachable (208.80.154.10)
[16:03:41] <ottomata>	 yeah
[16:06:28] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[16:06:28] <icinga-wm>	 PROBLEM - swift-object-updater on ms-be1 is CRITICAL: Connection refused by host
[16:06:28] <icinga-wm>	 PROBLEM - search indices - check lucene status page on search1019 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 58869 bytes in 0.016 second response time
[16:06:28] <icinga-wm>	 PROBLEM - Solr on vanadium is CRITICAL: Average request time is 1081.9827 (gt 1000)
[16:06:38] <icinga-wm>	 PROBLEM - DPKG on ms-be1 is CRITICAL: Connection refused by host
[16:07:34] <akosiaris>	 hmmm i am correct in guessing we want buck for some CI stuff for mobile ?
[16:07:40] <cmjohnson1>	 paravoid is that you ^ (ms-be1)
[16:08:24] <ottomata>	 akosiaris, paravoid, what to do about CLASSPATH in default then?
[16:08:31] <icinga-wm>	 PROBLEM - search indices - check lucene status page on search1018 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 60534 bytes in 0.009 second response time
[16:08:31] <icinga-wm>	 PROBLEM - SSH on virt4 is CRITICAL: Connection refused
[16:08:31] <icinga-wm>	 PROBLEM - Disk space on virt4 is CRITICAL: Connection refused by host
[16:08:44] <ottomata>	 can we leave it?  or duplicate it?  or use /usr/sbin/kafka to start server?
[16:09:20] <paravoid>	 akosiaris: no, gerrit
[16:09:21] <icinga-wm>	 PROBLEM - RAID on mw1020 is CRITICAL: Connection refused by host
[16:09:40] <paravoid>	 cmjohnson1: no
[16:09:41] <icinga-wm>	 PROBLEM - SSH on spence is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:10:12] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 16:10:03 UTC 2013
[16:10:24] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 16:10:18 UTC 2013
[16:10:24] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[16:10:31] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 16:10:27 UTC 2013
[16:11:11] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[16:11:21] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[16:12:33] <icinga-wm>	 RECOVERY - Host carbon is UP: PING OK - Packet loss = 0%, RTA = 0.16 ms
[16:12:51] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 16:12:42 UTC 2013
[16:13:11] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 16:13:02 UTC 2013
[16:13:11] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[16:13:11] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 16:13:10 UTC 2013
[16:13:21] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[16:13:22] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[16:14:12] <icinga-wm>	 RECOVERY - DPKG on mw1041 is OK: All packages OK
[16:14:12] <icinga-wm>	 RECOVERY - Disk space on mw1041 is OK: DISK OK
[16:14:22] <icinga-wm>	 RECOVERY - Host mw1041 is UP: PING OK - Packet loss = 0%, RTA = 1.55 ms
[16:14:32] <icinga-wm>	 RECOVERY - RAID on mw1041 is OK: OK: no RAID installed
[16:14:41] <icinga-wm>	 RECOVERY - twemproxy process on mw1041 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker
[16:15:41] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 16:15:36 UTC 2013
[16:16:01] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 16:15:58 UTC 2013
[16:16:11] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[16:16:27] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[16:16:33] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 16:16:21 UTC 2013
[16:17:21] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[16:19:18] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 16:19:08 UTC 2013
[16:19:35] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 16:19:28 UTC 2013
[16:19:42] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 16:19:34 UTC 2013
[16:20:11] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[16:20:21] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[16:20:31] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[16:21:58] <Coren>	 Ooo.  BZ got a facelift?
[16:22:13] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 16:22:04 UTC 2013
[16:22:21] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 16:22:18 UTC 2013
[16:22:33] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[16:22:54] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 16:22:46 UTC 2013
[16:23:13] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[16:23:23] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[16:23:41] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:24:31] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time
[16:24:31] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 16:24:29 UTC 2013
[16:24:41] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 16:24:34 UTC 2013
[16:24:41] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 16:24:37 UTC 2013
[16:25:05] <akosiaris>	 ottomata: well let's just duplicate it and be done with it...
[16:25:11] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[16:25:14] <greg-g>	 Coren: sorry, I couldn't hear you over the icinga noise :)
[16:25:21] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[16:25:27] <greg-g>	 Coren: and yeah, thank mutante (and andre, I assume?)
[16:25:35] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[16:26:01] <icinga-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 12 16:26:00 UTC 2013
[16:26:11] <icinga-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: No successful Puppet run in the last 10 hours
[16:26:11] <icinga-wm>	 RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 12 16:26:06 UTC 2013
[16:26:11] <icinga-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 12 16:26:09 UTC 2013
[16:26:21] <icinga-wm>	 PROBLEM - Puppet freshness on lardner is CRITICAL: No successful Puppet run in the last 10 hours
[16:26:31] <icinga-wm>	 PROBLEM - Puppet freshness on wtp1 is CRITICAL: No successful Puppet run in the last 10 hours
[16:27:41] <icinga-wm>	 PROBLEM - NTP on carbon is CRITICAL: NTP CRITICAL: No response from NTP server
[16:31:28] <hashar>	 out for dinner
[16:33:01] <ottomata>	 akosiaris: fine with me
[16:33:24] <ottomata>	 maybe ummm, we can have it in one place in the debian/ stuff, and templtae it out in the init and kafka scripts?
[16:33:28] <ottomata>	 or whatever, just duplicate it
[16:35:28] <cmjohnson1>	 !log removing/replacing failed disks in slot5 and slot9 from ms-be1012
[16:35:35] <morebots>	 Logged the message, Master
[16:39:44] <gerrit-wm>	 New review: Andrew Bogott; "Here's what the new template-generated file looks like:  https://dpaste.de/yXsAw/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68011
[16:51:11] <icinga-wm>	 PROBLEM - Host carbon is DOWN: CRITICAL - Host Unreachable (208.80.154.10)
[16:51:52] <yurik>	 greg-g, about to start deployment of zero ext
[16:53:17] <greg-g>	 yurik: cool, godspeed good sir
[17:01:22] <LeslieCarr>	 anyone working on carbon
[17:01:22] <LeslieCarr>	 ?
[17:01:41] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:01:42] <cmjohnson1>	 i am..
[17:01:45] <LeslieCarr>	 ok
[17:02:31] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time
[17:05:53] <cmjohnson1>	 !log removing/replacing disk slot 9 ms-be1007
[17:06:02] <morebots>	 Logged the message, Master
[17:06:11] <icinga-wm>	 RECOVERY - Host carbon is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms
[17:15:49] <ottomata>	 paravoid, if we are on the fence about log.dir location
[17:17:54] <logmsgbot>	 !log yurik synchronized php-1.22wmf6/extensions/ZeroRatedMobileAccess/
[17:17:56] <ottomata>	 i think i'd like to keep it at /var/lib/kafka/data or /var/lib/kafka/log, that way I don't have to think about it kafka.csv.metrics.dir, which is currently at /var/lib/kafka/metrics
[17:18:03] <morebots>	 Logged the message, Master
[17:18:03] <ottomata>	 hm, actually
[17:18:04] <ottomata>	 hmmm
[17:18:07] <yurik>	 mw1020 has just given me lots of nasty messages about fingerprint, is that ok?
[17:18:14] <ottomata>	 maybe metrics dir could go to /var/log/kafka/metrics
[17:18:20] <ottomata>	 and log.dir to /var/spool/kafka
[17:18:23] <ottomata>	 i'd be fine with that
[17:18:25] <ori-l>	 yurik: !log it
[17:18:34] <yurik>	 ori-l, too many lines
[17:19:51] <ori-l>	 !log mw1020: Warning: Remote Host Identification Has Changed
[17:20:01] <paravoid>	 ottomata: so, cups for example stores a backlog in /var/spool
[17:20:02] <morebots>	 Logged the message, Master
[17:20:07] <yurik>	 !log mw1020 complained during depl about fingerprint 81:2e:56:16:83:fd:b1:65:08:7b:ad:f8:66:98:19:c9.
[17:20:16] <morebots>	 Logged the message, Master
[17:20:27] <yurik>	 thx ori-l
[17:20:30] <paravoid>	 ori-l, yurik: also logged yesterday by Roan
[17:20:38] <paravoid>	 I have to go, maybe e.g. mutante can have a look?
[17:20:53] <paravoid>	 ottomata: so /var/spool is the right one, I think
[17:20:56] <ottomata>	 ok paravoid, let's do /var/spool/kafka then, i'm ok with that
[17:20:57] <ottomata>	 ok
[17:21:04] <ottomata>	 i'm going to remove /var/lib/kafka altogether then
[17:21:07] <paravoid>	 I don't have a strong opinion, but I'm leaning towards that
[17:21:09] <ottomata>	 and set KAFKA_HOME to /var/spool/kafka
[17:21:10] <ottomata>	 is that ok?
[17:21:16] <paravoid>	 er?
[17:21:20] <ottomata>	 ok
[17:21:22] <paravoid>	 what about the metrics thing?
[17:21:23] <ottomata>	 better q:
[17:21:31] <ottomata>	 i'll put that in /var/log/kafka/metrics
[17:21:36] <paravoid>	 what is that?
[17:21:37] <paravoid>	 metric?
[17:21:45] <ottomata>	 well, its disabled by default
[17:21:46] <ottomata>	 so nothing
[17:21:49] <ottomata>	 but if you turn it on
[17:21:56] <ottomata>	 i think it is a csv of kafka stats
[17:22:07] <paravoid>	 then I think it's /var/lib material
[17:22:14] <ottomata>	 bwerr awww
[17:22:18] <paravoid>	  /var/lib/kafka/metrics/ sounds good
[17:22:54] <logmsgbot>	 !log yurik synchronized php-1.22wmf5/extensions/ZeroRatedMobileAccess/
[17:23:01] <ottomata>	 hm. FIIIINE
[17:23:04] <paravoid>	 okay, got to go now
[17:23:04] <morebots>	 Logged the message, Master
[17:23:14] <ottomata>	 ok thanks
[17:23:15] <ottomata>	 laters
[17:23:21] <paravoid>	 I'll be back later
[17:23:37] <paravoid>	 I think we can finish it and push it today or tomorrow
[17:26:48] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:28:38] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.136 second response time
[17:28:42] <cmjohnson1>	 !log powering down wtp1008 for h/w problem/testing
[17:28:54] <morebots>	 Logged the message, Master
[17:30:48] <icinga-wm>	 PROBLEM - Host wtp1008 is DOWN: PING CRITICAL - Packet loss = 100%
[17:47:54] <gerrit-wm>	 New patchset: AzaToth; "Initial debian build" [operations/debs/buck] (master) - https://gerrit.wikimedia.org/r/67999
[17:48:14] <AzaToth>	 paravoid: down to 6 bundled jars now
[17:48:48] <AzaToth>	 couldn't use system jython as it borked, and jackson-core seemed to be too old
[17:49:12] <AzaToth>	 ^demon: any ideas why system jython fnucked
[17:49:13] <AzaToth>	 ?
[17:50:07] <AzaToth>	 ^demon: http://paste.debian.net/9972/
[17:50:09] <^demon>	 AzaToth: Not a clue.
[17:50:46] <AzaToth>	 that's when using /usr/share/java/jython.jar instead of bundled jython-standalone-2.5.3.jar
[17:51:43] * AzaToth  likes the "HumanReadabeException" which occured in File "<script>", line 2, in <module>...

[17:52:18] <AzaToth>	 Ryan_Lane: there?
[17:56:07] <Ryan_Lane>	 yep
[17:56:34] <Ryan_Lane>	 AzaToth: what's up?
[18:06:57] <AzaToth>	 have you used jython?
[18:08:25] <icinga-wm>	 RECOVERY - Host wtp1008 is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms
[18:09:24] <Ryan_Lane>	 AzaToth: I haven't
[18:10:03] <AzaToth>	 k
[18:10:24] <gerrit-wm>	 New patchset: Asher; "test twemproxy on 10 prod api servers" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68207
[18:10:34] <gwicke>	 cmjohnson1: any findings from wtp1008?
[18:11:09] <cmjohnson1>	 gwicke: not yet..i had to pull a report for dell...can only be done via centos so it requires that reboot
[18:11:19] <binasher>	 Aaron|home: hey sicky, can you review https://gerrit.wikimedia.org/r/68207 if you're around?
[18:11:21] <cmjohnson1>	 hopefully know something in the next few days.
[18:11:27] <AzaToth>	 Ryan_Lane: anyway, buck is at https://gerrit.wikimedia.org/r/#/c/67999/ and needs someone to ±2 it
[18:11:37] <gwicke>	 cmjohnson1: OK, thanks!
[18:12:21] <gerrit-wm>	 New patchset: Pyoungmeister; "using xenon for testing node" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68208
[18:12:22] <Aaron|home>	 binasher: maybe, I'm looking at https://gerrit.wikimedia.org/r/#/c/62130/7/includes/db/DatabaseMysqlBase.php
[18:12:28] <Aaron|home>	 actually you can look at that too :)
[18:13:09] <binasher>	 Aaron|home: hey, then we can switch right to mysqlnd instead of mysqli
[18:15:57] <Ryan_Lane>	 AzaToth: oh cool. you got it working?
[18:16:22] <Ryan_Lane>	 AzaToth: thanks for your work on this, it's really great to get packaging help!
[18:18:40] <Ryan_Lane>	 AzaToth: for this, would it be better for BUCK_JAVA_ARGS to be defined in /etc/default/buck, or something along those lines? https://gerrit.wikimedia.org/r/#/c/67999/4/debian/bin/buck,unified
[18:18:51] <Ryan_Lane>	 otherwise it seems there's no way to override the defaults
[18:20:19] <AzaToth>	 Ryan_Lane: I've stipped of half of buck to minimize bundled jars and so, so it's only to build blody gerrit
[18:20:24] <Ryan_Lane>	 ah
[18:20:25] <Ryan_Lane>	 heh
[18:20:33] <Ryan_Lane>	 nevermind then :)
[18:20:36] <AzaToth>	 Ryan_Lane: in the gerrit.git, which is the branch you are using?
[18:20:43] <AzaToth>	 master or wmf?
[18:20:56] <Ryan_Lane>	 I actually don't know. ^demon: ^^
[18:20:57] <AzaToth>	 perhaps ^demon knows better
[18:21:40] <^demon>	 Neither, we're using the 2.6-rc0-154-gfcdb34b tag
[18:22:05] <gerrit-wm>	 Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68208
[18:22:11] <^demon>	 wmf branch is what I want to deploy, but haven't yet.
[18:22:32] <AzaToth>	 ok
[18:22:35] <icinga-wm>	 PROBLEM - NTP on wtp1008 is CRITICAL: NTP CRITICAL: Offset unknown
[18:22:46] <AzaToth>	 just felt a bit old
[18:22:48] <AzaToth>	 january
[18:24:15] <Ryan_Lane>	 AzaToth: for these jars that are being bundled, does the source also exist in the debian source package?
[18:25:26] <AzaToth>	 nope
[18:25:49] <Ryan_Lane>	 that's going to violate their licenses, right?
[18:25:57] <AzaToth>	 not really
[18:26:12] <Ryan_Lane>	 the license info just needs to be there?
[18:26:18] <gerrit-wm>	 New patchset: Ottomata; "Adding proper .md formatting for code block to README.md" [operations/puppet/zookeeper] (master) - https://gerrit.wikimedia.org/r/68210
[18:26:25] <AzaToth>	 depends on the license
[18:26:30] * Ryan_Lane  nods

[18:26:35] <Ryan_Lane>	 these are all Apache 2?
[18:26:36] <gerrit-wm>	 Change merged: Ottomata; [operations/puppet/zookeeper] (master) - https://gerrit.wikimedia.org/r/68210
[18:27:35] <icinga-wm>	 RECOVERY - NTP on wtp1008 is OK: NTP OK: Offset -0.0007456541061 secs
[18:28:33] <AzaToth>	 Ryan_Lane: buck itself are apache-2.0
[18:28:43] <Ryan_Lane>	 yep. saw that in the license info :)
[18:28:51] <AzaToth>	 what the license of all blody bundled jars I've no clue of
[18:28:58] <AzaToth>	 there are 6 bundled jars
[18:29:16] <Ryan_Lane>	 :D
[18:29:20] <AzaToth>	 jython was funky, but the rest should be possible to create packages for
[18:29:23] <Ryan_Lane>	 yeah, that's what I was worried about
[18:29:31] <Aaron|home>	 binasher: do you know for sure if the hash mapping lines up?
[18:29:39] <AzaToth>	 paravoid: ↑
[18:30:00] <Ryan_Lane>	 debian policy is usually to make the dependencies library packages, right?
[18:30:12] <AzaToth>	 yea
[18:30:21] <Ryan_Lane>	 I went through this with opendj and decided it was too much work to get it included in debian
[18:30:38] <AzaToth>	 opendj is?
[18:30:42] <Ryan_Lane>	 ldap server
[18:31:00] <AzaToth>	 language?
[18:31:02] <Ryan_Lane>	 java
[18:31:07] <Ryan_Lane>	 same set of issues
[18:31:09] <binasher>	 Aaron|home: i watched a prod instance get keys that were hits, then manually ran gets for them via twemproxy, all hashed to the same server and were also hits
[18:31:16] <Ryan_Lane>	 except they use an easier build system
[18:31:37] <AzaToth>	 Ryan_Lane: dependency wize, java is a hell
[18:31:42] <Ryan_Lane>	 yep
[18:31:54] <AzaToth>	 everyone is just bundling jars here and there ignoring the rest of the world
[18:31:55] <Ryan_Lane>	 I hate touching anything java related when it comes to packaging
[18:32:07] <Ryan_Lane>	 yeah. it's like ruby
[18:34:05] <ori-l>	 Ryan_Lane: why? just use sbt's maven plugin to pull your ant dependencies from your ivy-maven bridge! there's an eclipse plugin for that, surely.
[18:37:40] <Aaron|home>	 binasher: eval.php seems to confirm what you are saying
[18:37:44] <Aaron|home>	 I think its oK
[18:37:46] <Aaron|home>	 *OK
[18:38:08] <binasher>	 Aaron|home: cool, thanks for testing
[18:38:37] <AzaToth>	 ^demon: can you setup a branch on current?
[18:38:39] <AzaToth>	 for gerrit
[18:38:59] <AzaToth>	 tag jumping aint my game
[18:39:13] <^demon>	 Huh?
[18:39:16] <^demon>	 A branch of what?
[18:39:39] <AzaToth>	 2.6-rc0-154-gfcdb34b
[18:39:52] <AzaToth>	 or is it 2.6-rc0-322-geeed497?
[18:39:59] <Aaron|home>	 binasher: how is the TP restart staggered on scap/sync?
[18:40:11] <Aaron|home>	 just -F or something?
[18:41:03] <Ryan_Lane>	 ori-l: :D
[18:41:18] <Ryan_Lane>	 ori-l: you're killing me :)
[18:42:38] <binasher>	 Aaron|home: scap/sync doesn't restart it, currently restart-twemproxy has to be run explicitly
[18:43:04] <binasher>	 Aaron|home: restart-twemproxy does: dsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F8 -- "sudo /sbin/restart twemproxy"
[18:43:34] <Aaron|home>	 I see, F8
[18:45:59] <binasher>	 Aaron|home: the restart take ~50ms/server from kill to accepting connections
[18:47:54] <gerrit-wm>	 Change merged: Asher; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68207
[18:50:58] <logmsgbot>	 !log asher synchronized wmf-config/mc-eqiad.php  'use twemproxy on mw113* (10 api apaches)'
[18:51:06] <morebots>	 Logged the message, Master
[18:54:28] <binasher>	 Aaron|home: not related to twemproxy but looking at the memcached-serious.log is :(
[18:54:29] <binasher>	 2013-06-12 18:53:58 mw1106 commonswiki: Memcached error for key "commonswiki:pt-lock:File:U.S._Navy_Damage_Controlman_(DC)_2nd_Class_Jered_Sassen_and_DC3_Michael_Moody_practice_the_proper_removal_of_a_contaminated_advanced_chemical_protection_garment_in_a_decontamination_station_aboard_the_aircraft_100926-N-XL102-089.jpg" on server ":": A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE
[18:54:44] <binasher>	 Aaron|home: is that filebackend related?
[18:55:11] <Aaron|home>	 no, translate extension
[18:55:22] <Aaron|home>	 https://gerrit.wikimedia.org/r/#/c/67316/
[18:58:17] <AzaToth>	 ^demon: any news about the tags?
[18:58:25] <AzaToth>	 branches/tags/whatever
[18:59:33] <^demon>	 AzaToth: Oh yes, sorry. New branch called cur-deployed that points to fcdb34b97f3d684515668710bd5f1e3c681d187f (2.6-rc0-154-gfcdb34b)
[18:59:45] <AzaToth>	 ok
[19:02:33] <AzaToth>	 ^demon: you know why debs/gerrit has 2.6-rc0-322-geeed497??
[19:02:41] <AzaToth>	 no used I assume?
[19:02:47] <AzaToth>	 not*
[19:02:50] <^demon>	 Not used, correct.
[19:04:59] <AzaToth>	 ^demon: which version are next to be released?
[19:05:20] <AzaToth>	 becuase I notice 2.6-rc0-154-gfcdb34b is old and uses maven
[19:05:29] <AzaToth>	 s/released/used/
[19:05:30] <^demon>	 Yes, it was pre-switchover.
[19:05:43] <^demon>	 Next one I want to use is wmf branch, ef469acc8db351fc1d82d3ba657399482cebff4f
[19:05:47] <AzaToth>	 ok
[19:11:29] <AzaToth>	 ^demon: I just saw there is a wmf-deployed which points to something else
[19:11:46] <^demon>	 Yeah, I don't remember what that's for.
[19:11:48] <^demon>	 Something old prolly.
[19:21:10] <odder>	 I notice that the MariaDB accepts machines as donations
[19:21:11] <binasher>	 Aaron|home: the documented pecl memcached default config values at http://us1.php.net/manual/en/memcached.constants.php aren't correct :/
[19:21:15] <binasher>	 Aaron|home: http://pastebin.mozilla.org/2516560
[19:21:32] <odder>	 are there any plans for donating some of our unused machines?
[19:23:01] <binasher>	 we are going to need to add support to MemcachedPeclBagOStuff to change those, probably to what the docs claim are the default for use with twemproxy.  don't want the pecl client waiting 2 seconds to retry talking to twemproxy and seeing memcached as completely down in the interim
[19:24:42] <paravoid>	 binasher: we tested the new libmemcached/php-memcached patches yesterday btw
[19:24:52] <average_drifter>	 paravoid: hi
[19:24:55] <paravoid>	 I made a pull request to php-memcached too with the port
[19:25:15] <paravoid>	 also a launchpad bug to revert the api breakage
[19:25:39] <paravoid>	 anyway, I'm following up with pushing those upstream but in any case the packages are ready for consumption
[19:25:51] <gerrit-wm>	 New patchset: Ottomata; "Debianize Kafka" [operations/debs/kafka] (debian) - https://gerrit.wikimedia.org/r/68026
[19:25:58] <paravoid>	 not sure if you care, I'm guessing the twemproxy work kinda makes the bugfix redundant, no?
[19:26:40] <paravoid>	 ok, back afk :)
[19:28:09] <Aaron|home>	 binasher: so there needs to be a way to set OPT_SERVER_FAILURE_LIMIT, should be easy
[19:33:59] <ori-l>	 csteipp: what's the process for getting some advice re: a security question? this is regarding a proposed change, not code in production. should I e-mail you directly or file a bug or?
[19:34:12] <gerrit-wm>	 New review: Ottomata; "Patchset?5:" [operations/debs/kafka] (debian) - https://gerrit.wikimedia.org/r/68026
[19:34:37] <AzaToth>	 paravoid: http://paste.debian.net/9993/
[19:34:46] <csteipp>	 ori-l: Just looking for advice on how to do something? Or discussing a potential vulnerability?
[19:34:46] <AzaToth>	 got some troubles
[19:34:57] <AzaToth>	 can't figure out what's fnucing
[19:35:28] <csteipp>	 For advice, just send me an email. If it's a problem in some code, then open a bug.
[19:36:39] <ori-l>	 csteipp: ok, will e-mail. thanks
[19:41:08] <binasher>	 Aaron|home: https://gerrit.wikimedia.org/r/#/c/68222/ ?
[19:42:10] <Aaron|home>	 huh, I was doing the same thing
[19:42:18] <Aaron|home>	 anyway, you need to update the constructor docs
[19:42:22] <binasher>	 doh
[19:42:31] <binasher>	 you should probably proceed and i'll abandon
[19:42:34] <Aaron|home>	 no
[19:42:45] <Aaron|home>	 it's faster to just amend that
[19:43:18] <Aaron|home>	 if ( $params['retry_timeout'] ) {
[19:43:24] <Aaron|home>	 binasher: that needs isset()
[19:43:27] <Aaron|home>	 otherwise fine
[19:43:48] <binasher>	 Aaron|home: what about the existing if ( $params['persistent'] ) {
[19:44:08] <binasher>	 ah, applyDefaults sets to false
[19:44:25] <Aaron|home>	 yeah
[19:44:34] <Aaron|home>	 I suppose you override applyDefaultParams() and call super
[19:44:38] <Aaron|home>	 that would work too
[19:48:10] <binasher>	 Aaron|home: https://gerrit.wikimedia.org/r/#/c/68222/2/includes/objectcache/MemcachedPeclBagOStuff.php
[19:48:42] <Aaron|home>	 binasher: I think you failed to spell failed :)
[19:49:00] <binasher>	 llllll
[19:50:07] <binasher>	 Aaron|home: ok, /3 :)
[19:50:19] <mutante>	 Ohloh says per COCOMO model it took 13 years of combined effort to create operations/mediawiki-config
[19:50:23] <Aaron|home>	 hmm
[19:50:52] <Aaron|home>	 binasher: shouldn't it be more like "Time in seconds to wait before retrying a server marked as "down""?
[19:51:50] <binasher>	 Aaron|home: the pecl documents uses the language "The server will be removed after this many continuous connection failures." which is why i went with removed vs. down
[19:52:17] <binasher>	 though i agree that the pecl documentation is misleading there as well
[19:52:27] <Aaron|home>	 meh, whatever
[19:52:33] <Aaron|home>	 it's one the pecl docs
[19:52:37] <Aaron|home>	 *on
[19:54:42] * binasher  goes afk

[20:22:26] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:23:16] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time
[20:29:20] <paravoid>	 and back
[20:31:13] <paravoid>	 anyone needs anything from me?
[20:32:15] <ottomata>	 yup!
[20:32:25] <ottomata>	 howzit?
[20:32:25] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/68026/
[20:33:05] <paravoid>	 homedir can be /nonexistent
[20:33:13] <paravoid>	 it's fairly common
[20:33:16] <paravoid>	 nobody's ~ is that
[20:33:30] <paravoid>	 how come you didn't go the /var/lib/kafka/metrics way for metrics?
[20:34:34] <ottomata>	 i didn't want to create /var/lib/kafka for something that was going to be completely empty by default
[20:34:54] <ottomata>	 metrics is off by default, you configure it to be on if you want it
[20:35:01] <ottomata>	 i don't even know if we want it, its a new 0.8 feature (I think)
[20:37:31] <ottomata>	 paravoid: , i think i'm ok with kafka's home being /usr/share/kafka, shoudl it not be?
[20:38:56] <paravoid>	 doesn't matter
[20:39:11] <paravoid>	 the /nonexistent suggestion is because you moved it
[20:39:20] <paravoid>	 and if you're unsure where it should be, just put /nonexistent
[20:39:37] <mutante>	 but that should really not exist
[20:39:37] <ottomata>	 i moved it because alex's init.d script sets KAFKA_HOME there
[20:40:11] <paravoid>	 what is KAFKA_HOME used for?
[20:40:45] <ottomata>	 you know, i bet nothing.
[20:40:53] <ottomata>	 it was previously used for finding .jars and adding them to classpath
[20:41:00] <ottomata>	 but since CLASSPATH is manually specified now
[20:41:05] <ottomata>	 it probably isn't used at all
[20:41:13] <ottomata>	 i'll take it out altogether and set to /nonexistent
[20:41:55] <gerrit-wm>	 New patchset: AzaToth; "Update build to build gerrit" [operations/debs/gerrit] (master) - https://gerrit.wikimedia.org/r/68281
[20:43:04] <ottomata>	 paravoid, any other comments on this before I submit a new patchset?
[20:43:19] <gerrit-wm>	 New patchset: AzaToth; "Update build to build gerrit" [operations/debs/gerrit] (master) - https://gerrit.wikimedia.org/r/68281
[20:43:30] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:44:20] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.141 second response time
[20:44:21] <AzaToth>	 Ryan_Lane: there you go
[20:45:16] <paravoid>	 ottomata: nope
[20:45:20] <paravoid>	 I think it's ready to go
[20:45:26] <paravoid>	 do you have it built somewhere?
[20:45:33] <paravoid>	 I'd love to see a dpkg --contents kafka*deb
[20:45:45] <paravoid>	 but I can build it too
[20:45:52] <paravoid>	 other than that it LGTM
[20:46:02] <paravoid>	 but you should probably wait for alex's +2
[20:46:04] <ottomata>	 yes
[20:46:06] <ottomata>	 totally will wait
[20:46:14] <ottomata>	 umm, lemme build a new one with the homedir changed
[20:46:18] <ottomata>	 then will get you .deb
[20:46:39] <paravoid>	 nice work!
[20:47:03] <gerrit-wm>	 New patchset: Ottomata; "Debianize Kafka" [operations/debs/kafka] (debian) - https://gerrit.wikimedia.org/r/68026
[20:47:14] <ottomata>	 ja thanks!  most of credit is due to alex for his crazy scala makefile sleuthing!
[20:47:33] <gerrit-wm>	 New review: Ottomata; "Patchset 6:" [operations/debs/kafka] (debian) - https://gerrit.wikimedia.org/r/68026
[20:51:03] <mutante>	 ottomata: any news about ana1020?
[20:52:01] <mutante>	 paravoid: $sometime, if you feel like taking a look at the Bugzilla module Ori wrote
[20:52:03] <ottomata>	 I made an RT plea for help!
[20:52:04] <ottomata>	 https://rt.wikimedia.org/Ticket/Display.html?id=5281
[20:52:30] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:52:59] <mutante>	 ottomata: ah, nice summary,thanks
[20:53:20] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time
[20:53:52] <mutante>	 ottomata: giving it one more shot
[20:54:18] <odder>	 Ryan_Lane:
[20:54:24] <mutante>	 maybe less solar winds today :p
[20:55:03] <mutante>	 !log PXE booting analytics1020
[20:55:12] <morebots>	 Logged the message, Master
[20:55:19] <Ryan_Lane>	 odder: ?
[20:55:36] <Ryan_Lane>	 ugh. I fucking hate php
[20:55:39] <ottomata>	 haha
[20:55:45] <ottomata>	 thanks mutante
[20:56:19] <Ryan_Lane>	 $test = '';
[20:56:29] <Ryan_Lane>	 $test = (int)$test;
[20:56:35] <Ryan_Lane>	 guess what $test equals?
[20:56:40] <paravoid>	 1?
[20:56:40] <odder>	 Ryan_Lane: I noticed https://bugzilla.wikimedia.org/show_bug.cgi?id=49352 but have no idea where the config file is located
[20:56:43] <Ryan_Lane>	 0
[20:56:44] <ottomata>	 0
[20:56:51] <paravoid>	 0 is expected
[20:56:51] <odder>	 Ryan_Lane: I can upload the favicon instead, I guess...
[20:57:02] <paravoid>	 1 would impress me, but 0 looks alright to me
[20:57:10] <icinga-wm>	 PROBLEM - Host analytics1020 is DOWN: PING CRITICAL - Packet loss = 100%
[20:57:24] <odder>	 upload to the bug, ie. attach
[20:57:51] <Ryan_Lane>	 $test = "hello";
[20:57:58] <Ryan_Lane>	 $test = (int)$test;
[20:58:00] <Ryan_Lane>	 guess what $test equals?
[20:58:12] <mutante>	 favicon.ico is served by favicon.php
[20:58:15] <mutante>	 Krinkle: ^
[20:59:15] <ottomata>	 hmmmmm
[20:59:18] <ottomata>	 1?
[20:59:22] <Ryan_Lane>	 ottomata: 0
[20:59:27] <ottomata>	 haha
[20:59:28] <ottomata>	 http://www.mediawiki.org/wiki/User:Ottomata/PHP_False
[20:59:31] <ottomata>	 i should add int() to that
[21:00:02] <Ryan_Lane>	 now, what the fuck does $this->getOption( 'delete-var' ) return from the Maintenance class?
[21:00:07] <Ryan_Lane>	 if it isn't set?
[21:00:16] <mutante>	 Ryan_Lane: on the project wikis Krinkle does this via  mediawiki-config/w/favicon.php
[21:00:17] * Ryan_Lane  reads the code

[21:00:20] <Ryan_Lane>	 maybe it's null
[21:01:05] <Ryan_Lane>	 seems it's a bool
[21:03:30] <Ryan_Lane>	 no. it's null
[21:03:45] <odder>	 mutante: yeah, but labs code isn't there...
[21:04:11] <mutante>	 Ryan_Lane: if (is_bool($test) === true) { .. does not think it is boolean :)
[21:04:35] <mutante>	 ah, yea
[21:05:06] <ottomata>	 paravoid: fenari:/home/otto/kafka_deb/
[21:05:31] <mutante>	 odder: @fenari:/h/w/common/docroot/labs/w  favicon.php
[21:07:30] <icinga-wm>	 RECOVERY - Host analytics1020 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms
[21:08:19] <odder>	 mutante: I meant, the favicon isn't defined in any config files, and it isn't in bits/favicon
[21:09:33] <mutante>	 odder: aah, i found it , it's in /var/www/ on virt0
[21:09:58] <mutante>	 but the rest of the wiki is /srv/org/wikimedia/controller/wikis/
[21:10:11] <odder>	 which is where on gerrit? :-)
[21:11:27] <mutante>	 hmm, i don't know, there is /docroot/labs/ in mediawiki-config but that's not it
[21:11:47] <mutante>	 i guess this one file at least isn't in it
[21:12:01] <mutante>	 because /var/www/ has nothing but favicon.ico and default index.html
[21:12:37] <mutante>	 odder: what did you want to replace it with?
[21:13:11] <odder>	 mutante: wanted to fix https://bugzilla.wikimedia.org/show_bug.cgi?id=49352 and https://bugzilla.wikimedia.org/show_bug.cgi?id=49351
[21:13:29] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:13:34] <odder>	 but I guess it'll be easier if I just attach the icos to the bugs
[21:13:35] <mutante>	 yea, but do you have a better .ico made from one of https://commons.wikimedia.org/wiki/Category:SVG_Wikimedia_Labs_logos ?
[21:13:39] <odder>	 yes
[21:13:55] <mutante>	 does commons let you upload the .ico?
[21:13:58] <odder>	 both, 16 x 16, 32 x 32 and 48 x 48
[21:14:04] <odder>	 in multilayered icos
[21:14:19] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time
[21:14:26] <odder>	 mutante: nope
[21:15:11] <mutante>	 odder: well, mediawiki-config/docroot/labs/images/ doesn't sound totally wrong
[21:16:51] <paravoid>	 ottomata: looks good
[21:16:54] <paravoid>	 for our purposes anyway :)
[21:18:12] <odder>	 mutante: or I can simply upload them to bits?
[21:18:22] <odder>	 we do that for all the other projects
[21:18:22] <Krinkle>	 mutante: Though I'm not sitting in the servers handing favicons to users, I also didn't write that code. Tim did, though I do support the general idea behind it.
[21:18:45] <odder>	 s/upload to bit/place in docroot/bits/favicon/
[21:18:50] <Krinkle>	 (w/favicon.php)
[21:19:13] <paravoid>	 unterminated 's' command
[21:19:35] <paravoid>	 aka EPARSE
[21:19:44] <ottomata>	 awesome
[21:19:48] <ottomata>	 thanks paravoid!
[21:19:55] <paravoid>	 s;upload to bit;place in docroot/bits/favicon;
[21:19:58] <paravoid>	 maybe :)
[21:20:08] <ottomata>	 give that there change a +1 or +2 if you don't mind
[21:20:16] <ottomata>	 if we get yours and alex's in, we can merge it and call it done
[21:20:29] <gerrit-wm>	 New review: Faidon; "LGTM" [operations/debs/kafka] (debian) C: 1;  - https://gerrit.wikimedia.org/r/68026
[21:20:54] <paravoid>	 +1's description is "LGTM, but someone else must approve"
[21:21:01] <mutante>	 odder: eh, "like all other projects" sounds good, i dunno better then
[21:21:05] <paravoid>	 which is appropriate here, I'd like alex to approve too :)
[21:21:31] <paravoid>	 ottomata: remind me, what's next on your pipeline?
[21:21:33] <mutante>	 Krinkle: gotcha, the question was more if this should apply to wikitech wiki like all other wikis or not (yet)
[21:21:46] <paravoid>	 zookeeper module, kafka module, ...?
[21:21:48] <mutante>	 and .. i'm not sure about /docroot/labs/ on fenari
[21:21:59] <LeslieCarr>	 !log restarting caesium
[21:22:02] <paravoid>	 magnus is doing good process on varnishkafka I'm hearing
[21:22:07] <morebots>	 Logged the message, Mistress of the network gear.
[21:22:12] <paravoid>	 s/doing/makig/
[21:22:27] <ottomata>	 paravoid, probably a rework of the kafka module soon
[21:22:30] <ottomata>	 to match the .dewb
[21:22:36] <ottomata>	 but that's mostly been reviewed by you before
[21:22:41] <paravoid>	 and use the zookeeper module too, right?
[21:22:45] <ottomata>	 yes
[21:22:51] <ottomata>	 but for more reviewy heavy things
[21:22:53] <ottomata>	 ummm, up next is server side hadoop ecosystem stuff
[21:22:55] <paravoid>	 right it was simple enough too
[21:22:58] <ottomata>	 oozie server, hive server, etc.
[21:23:05] <mutante>	 odder: i think paravoid is right, check into operations/mediawiki-config/docroot/bits/favicon/ in gerrit
[21:23:06] <ottomata>	 yeah, i'm also working on reinstalling the hadoop nodes
[21:23:12] <ottomata>	 mutante:  ishelping with that
[21:23:15] <ottomata>	 since i'm having so much trouble
[21:23:17] <ottomata>	 with an20
[21:23:18] <Krinkle>	 mutante: Don't know, wikitech setup is mostly black magic to me. Ryan would know. Afaik it isn't running wmf-config. Or it might have its own copy of it
[21:23:23] <paravoid>	 mutante: I just corrected the regexp above
[21:23:29] <ottomata>	 we can reinstall the zookeeper nodes now too
[21:23:31] <paravoid>	 I have no idea about what you folks are talkinga bout
[21:23:33] <Ryan_Lane>	 ?
[21:24:01] <paravoid>	 ottomata: so we're going the reinstall piece-by-piece way rather than setting up a new cluster?
[21:24:04] <ottomata>	 once we get oozie, hive, hue, etc. pupetized, we can reinstall an27
[21:24:04] <paravoid>	 not that I mind
[21:24:09] <paravoid>	 just news to me :)
[21:24:31] <ottomata>	 yes?, shouldn't be, the idea was to not have to find so many TBs of space to store the existing hdfs data
[21:24:34] <mutante>	 paravoid: ok:) , but it fits
[21:24:37] <ottomata>	 so the hadoop data nodes go one at a time
[21:24:43] <ottomata>	 rebalancing data after each install
[21:24:55] <odder>	 mutante: I did, it isn't there
[21:25:14] <mutante>	 odder: i meant to put your new files there via a patch set
[21:25:40] <mutante>	 then at least they are already somewhere
[21:26:57] <gerrit-wm>	 New patchset: Odder; "(bug 49351) Fix Labs and Tool Labs favicons" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68291
[21:27:02] <odder>	 wee
[21:27:09] <icinga-wm>	 PROBLEM - RAID on analytics1020 is CRITICAL: Timeout while attempting connection
[21:28:29] <icinga-wm>	 PROBLEM - Host analytics1020 is DOWN: PING CRITICAL - Packet loss = 100%
[21:29:34] <gerrit-wm>	 New review: Dzahn; "the favicon.ico for wikitech appears to be:" [operations/mediawiki-config] (master) C: 2;  - https://gerrit.wikimedia.org/r/68291
[21:29:46] <gerrit-wm>	 Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68291
[21:30:29] <icinga-wm>	 RECOVERY - Host analytics1020 is UP: PING OK - Packet loss = 0%, RTA = 0.38 ms
[21:31:08] <mutante>	 odder: did you also do testwikidata.ico ?
[21:31:31] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:31:51] <odder>	 Sam did that, I think
[21:32:20] <odder>	 https://gerrit.wikimedia.org/r/#/c/67980/
[21:32:56] <mutante>	 odder: MS Windows icon resource - 3 icons, 16x16, 256-colors
[21:33:19] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time
[21:33:46] <odder>	 mutante: mmm?
[21:34:16] <mutante>	 !log deploying new favicon on wikitech
[21:34:19] <mutante>	 odder: https://wikitech.wikimedia.org/favicon.ico
[21:34:25] <morebots>	 Logged the message, Master
[21:34:28] <mutante>	 just copied it for now
[21:34:42] <mutante>	 looks better, thanks
[21:35:03] <odder>	 mutante: it just has three layers
[21:35:37] <mutante>	 odder: yea, i just pasted that to confirm "see it's on the servers, it has 3 in 1 .ico now"
[21:38:33] <gerrit-wm>	 New review: Dzahn; "the favicon.ico for wikitech appears to be:" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68291
[21:41:47] <gerrit-wm>	 New review: Faidon; "Initial comments :) Thanks for working on this!" [operations/debs/gerrit] (master) C: -1;  - https://gerrit.wikimedia.org/r/68281
[21:42:06] <icinga-wm>	 PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours
[21:42:06] <icinga-wm>	 PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours
[21:42:06] <icinga-wm>	 PROBLEM - Puppet freshness on ms-be1 is CRITICAL: No successful Puppet run in the last 10 hours
[21:42:06] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours
[21:42:06] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours
[21:42:06] <icinga-wm>	 PROBLEM - Puppet freshness on mw1020 is CRITICAL: No successful Puppet run in the last 10 hours
[21:42:06] <icinga-wm>	 PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours
[21:42:06] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours
[21:42:06] <icinga-wm>	 PROBLEM - Puppet freshness on sockpuppet is CRITICAL: No successful Puppet run in the last 10 hours
[21:42:06] <icinga-wm>	 PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours
[21:42:06] <icinga-wm>	 PROBLEM - Puppet freshness on spence is CRITICAL: No successful Puppet run in the last 10 hours
[21:42:07] <icinga-wm>	 PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours
[21:42:07] <icinga-wm>	 PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours
[21:43:57] <jdlrobson>	 mark: any chance you could find a moment to take a look at https://gerrit.wikimedia.org/r/#/c/68009/ ?
[21:47:56] <paravoid>	 jdlrobson: it's around midnight there, don't expect this today
[21:48:44] <jdlrobson>	 haha ok (always forget about time zones ;-))
[21:50:03] <paravoid>	 jdlrobson: but i'm not sure what this is
[21:50:09] <paravoid>	 could you clarify a bit?
[21:50:31] <paravoid>	 I don't see Cache-Control: no-cache, must-revalidate etc. on mobile responses
[21:51:01] <paravoid>	 for which responses is this exactly?
[21:51:09] <paravoid>	 jdlrobson, MaxSem: ^
[21:51:31] <ottomata>	 mutante:  any luck on an20?
[21:52:14] <MaxSem>	 paravoid, the code says for every mobile HTML hit coming from squids or varnishes
[21:52:22] <MaxSem>	 might be broken though
[21:52:24] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:52:35] <paravoid>	 that's not the case
[21:52:41] <paravoid>	 Cache-control: s-maxage=2678400, must-revalidate, max-age=0
[21:52:53] <MaxSem>	 hahahaha
[21:53:03] <paravoid>	 ?
[21:53:07] <MaxSem>	 it gets overridden by OutputPage
[21:53:10] <gerrit-wm>	 New review: AzaToth; "Well, I've not changed much of the logic ryan setup, didn't want to whack his carefully crafted code..." [operations/debs/gerrit] (master) - https://gerrit.wikimedia.org/r/68281
[21:53:14] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.137 second response time
[21:53:22] <MaxSem>	 was like this for a year then:):):)
[21:53:22] <paravoid>	 no Pragma: no-cache or Expires either
[21:53:33] <AzaToth>	 Ryan_Lane: :-P
[21:54:16] <MaxSem>	 paravoid, I see Expires:Sat, 26 Jul 1997 05:00:00 GMT however
[21:54:37] <paravoid>	 MaxSem: http://p.defau.lt/?af8cu4DnohLi9ql0W9GBSg
[21:54:48] <paravoid>	 that's the response for en.m/wiki/Main_Page
[21:55:05] <paravoid>	 the response from appservers, no varnish involved at all
[21:55:43] <paravoid>	 the request came from cp1041, so it should match the wfConfiguredProxy
[21:56:00] <paravoid>	 wfIsConfiguredProxy even
[21:56:55] <paravoid>	 same from cp1043 (which is under wikimedia.org rather than eqiad.wmnet -- I saw that it mattered for the first incarnation of that piece of code)
[21:57:36] <MaxSem>	 yes, but not the current one which uses IPs
[21:58:03] <paravoid>	 right, I saw that but I checked from cp1043 just in case
[21:58:38] <MaxSem>	 I checked from outside
[21:59:34] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:59:57] <MaxSem>	 so I assume that this code had been broken for quite some time
[22:00:11] <MaxSem>	 and if we accidentally fix it we will fry the cluster
[22:00:15] <paravoid>	 it'd be nice to know why
[22:00:21] <paravoid>	 fry? why?
[22:00:31] <paravoid>	 it'd probably be noop from varnish's perspective
[22:00:47] <MaxSem>	 by making everything non-cacheable
[22:01:08] <paravoid>	 oh, by fix it you mean fix that piece of code
[22:01:09] <binasher>	 Aaron|home: thanks for +2'ing that peclbagostuff change. does it need to be merged into wmf5/6?
[22:01:16] <paravoid>	 yeah, don't do that
[22:01:22] <MaxSem>	 :)
[22:02:24] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time
[22:04:14] <MaxSem>	 ok, thanks paravoid
[22:04:20] <mutante>	 ottomata: not yet, it did the same thing, next is going back to BIOS
[22:04:57] <Ryan_Lane>	 AzaToth: gonna take me a while to review the packages :)
[22:04:59] <mutante>	 odder: i talked to Coren, he copied for toollabs as well, http://tools.wmflabs.org/favicon.ico
[22:05:16] <paravoid>	 MaxSem: I +1ed
[22:09:54] <gerrit-wm>	 New patchset: Odder; "(bug 46244) Enable wmgUseVectorFooterCleanup on ilowiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68298
[22:10:05] <odder>	 mutante: awesome :-)
[22:17:48] <Aaron|home>	 binasher: yeah
[22:17:57] * Aaron|home  returns from late lunch

[22:18:14] <Aaron|home>	 binasher: doing that now
[22:18:28] <binasher>	 Aaron|home: thanks!
[22:23:15] <logmsgbot>	 !log aaron synchronized php-1.22wmf5/includes/objectcache/MemcachedPeclBagOStuff.php  'f6287b2e2faba20aea7379dae84ee7c5dd84983f'
[22:23:24] <morebots>	 Logged the message, Master
[22:25:28] <Aaron|home>	 binasher: did you do the config yet?
[22:25:34] <binasher>	 going to http://en.wikipedia.org/wiki/Main_Page as a logged in user presents "1 reader comment" above welcome to wikipedia, linking to http://en.wikipedia.org/wiki/Special:ArticleFeedbackv5/Main_Page?ref=article&filter=featured which shows one comment that says "it needs to be talking about what is the effect of the subject to the people"
[22:25:38] <binasher>	 thanks aftv5!
[22:25:43] <logmsgbot>	 !log aaron synchronized php-1.22wmf6/includes/objectcache/MemcachedPeclBagOStuff.php  '7f091934ceae0280b44292e5ce426b24c5a579a2'
[22:25:55] <morebots>	 Logged the message, Master
[22:26:07] <binasher>	 why is aftv5 enabled on the enwiki main page??
[22:27:14] <Aaron|home>	 surely that is some bug I hope
[22:27:27] <binasher>	 aftv5 is indeed a bug :)
[22:28:22] <paravoid>	 I don't see that
[22:29:00] <binasher>	 i still get it
[22:29:25] <Prodego>	 you can changed whether it is enabled or not with no log entry?
[22:29:28] <Prodego>	 change*
[22:29:40] <Prodego>	 I just went the the enwiki main page, and selected 'off', instead of 'on'
[22:29:53] <Prodego>	 there is no log of me doing that that I see
[22:30:08] <binasher>	 Prodego: selected 'off' where?
[22:30:29] <odder>	 not the first time you can do stuff in extensions without them being logged...
[22:30:32] <Aaron|home>	 binasher: what should OPT_SERVER_FAILURE_LIMIT be?
[22:32:39] <Prodego>	 binasher: page protection
[22:32:50] <Prodego>	 https://en.wikipedia.org/w/index.php?title=Main_Page&action=unprotect under article feedback
[22:33:00] <Prodego>	 someone probably turned it on on the main page, and that wasn't logged if turning it off isn't
[22:33:14] <binasher>	 Aaron|home: if using twemproxy, i think it should be 0
[22:33:36] <Aaron|home>	 I was looking at the docs to be sure exactly what 0 does
[22:33:56] <binasher>	 Aaron|home: the libmemcached docs?
[22:36:31] <icinga-wm>	 PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server
[22:37:41] <binasher>	 Aaron|home: " MEMCACHED_BEHAVIOR_SERVER_FAILURE_LIMIT - Deprecated since version 0.48"
[22:37:59] <mutante>	 http://www.slideshare.net/PuppetLabs/roles-talk
[22:38:04] <binasher>	 auto_eject_hosts is deprecated as of 0.48 as well..
[22:38:53] <binasher>	 but was only more recently added to the pecl client, hah
[22:39:42] <binasher>	 Aaron|home: MEMCACHED_BEHAVIOR_RETRY_TIMEOUT is still in 1.x and may be the most relevant
[22:40:40] <Aaron|home>	 the eject thing is probably deprecated given http://hoborglabs.com/en/blog/2013/memcached-php
[22:40:52] <Aaron|home>	 "OPT_RETRY_TIMEOUT is by default set to 2 seconds, not 0." heh
[22:41:13] <icinga-wm>	 PROBLEM - RAID on analytics1020 is CRITICAL: Timeout while attempting connection
[22:41:20] <binasher>	 looking at libmemcached src, setting retry_timeout to 0 should indeed result in the behavior i'd prefer
[22:42:41] <icinga-wm>	 PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server
[22:42:50] <icinga-wm>	 PROBLEM - Host analytics1020 is DOWN: PING CRITICAL - Packet loss = 100%
[22:43:54] <gerrit-wm>	 New patchset: Aaron Schulz; "Set server_failure_limit for twemproxy cache." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68301
[22:45:14] <binasher>	 Aaron|home: add retry_timeout = 0? also, spacing vs tabs
[22:45:37] <drdee>	 binasher: quick question, i see some "http://www.wikipedia.orghttp://www.wikipedia.org/" entries and variations thereof (very few)  in the requested uri field  in the web request log files with status code 200, any idea how that can happen?
[22:45:39] <binasher>	 it sucks that retry_timeout is measured in seconds and only takes an int
[22:45:40] <Aaron|home>	 yeah, tabs get turned to spaces in vim
[22:45:51] * Aaron|home  doesn't remember that happening before

[22:46:40] <Aaron|home>	 binasher: that's not redundant?
[22:46:55] <binasher>	 drdee: that occurs when a client sends "GET http://..." instead of relying on the host header
[22:47:34] <drdee>	 kool, thanks!
[22:47:35] <binasher>	 Aaron|home: not necessarily, since the server_failure_limit option the client is setting is deprecated in our version of libmemcached
[22:48:31] <binasher>	 drdee: see section 5.1.2 http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html
[22:48:32] <gerrit-wm>	 New patchset: Aaron Schulz; "Set server_failure_limit for twemproxy cache." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68301
[22:49:52] <gerrit-wm>	 Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68301
[22:54:50] <paravoid>	 drdee: it's https://www.varnish-cache.org/trac/ticket/1255
[22:54:57] <paravoid>	 drdee: fixed in varnish 4, not easily backportable
[22:57:08] <binasher>	 oh, lol.. i didn't realize at first that bug was opened by tim
[22:57:44] <binasher>	 then i read "see the terrible hacking workaround we implemented to allow purging from an LWP client" -> link to terrible thing i did
[22:58:48] <paravoid>	 more accurately, link to 404  :P
[22:59:20] <Aaron|home>	 Wed Jun 12 22:33:48 UTC 2013    hume    testwikidatawiki        Error selecting database testwikidatawiki on server 10.0.6.74
[22:59:21] <Aaron|home>	 Wed Jun 12 22:33:49 UTC 2013    hume    votewiki        Error selecting database votewiki on server 10.0.6.74
[22:59:28] <binasher>	 yay, death of gitweb
[22:59:43] <logmsgbot>	 !log asher synchronized wmf-config/mc-eqiad.php  'set (server_failure_limit, retry_timeout) = 0 for twemproxy test hosts'
[22:59:46] <paravoid>	 it'd be nice to rewrite urls, somehow...
[22:59:51] <morebots>	 Logged the message, Master
[23:05:13] <gerrit-wm>	 New review: Faidon; "A couple inline comments." [operations/debs/gerrit] (master) - https://gerrit.wikimedia.org/r/68281
[23:06:37] <paravoid>	 binasher: are you messing with memcached? do you mind if I ship an updated libmemcached/php5-memcached or should I wait until after you're done?
[23:06:49] <binasher>	 paravoid: go ahead
[23:07:08] <binasher>	 there's no conflict
[23:07:51] <gerrit-wm>	 New patchset: Faidon; "appserver: update for newer libmemcached" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68307
[23:08:58] <gerrit-wm>	 Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68307
[23:09:54] <paravoid>	 !log apt: updating libmemcached/php5-memcached (#5268)
[23:10:02] <morebots>	 Logged the message, Master
[23:12:57] <Aaron|home>	 binasher: when might you do the other servers?
[23:13:25] <binasher>	 i'd be ok doing it now
[23:16:25] <gerrit-wm>	 New patchset: Asher; "use twemproxy on all apaches" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68308
[23:16:29] <binasher>	 Aaron|home: ^^
[23:17:13] <gerrit-wm>	 New patchset: Asher; "use twemproxy on all apaches" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68308
[23:20:55] <Aaron|home>	 ^demon: https://gerrit.wikimedia.org/r/#/c/67880/
[23:21:25] <drdee>	 thanks paravoid for the link
[23:21:57] <gerrit-wm>	 New patchset: MaxSem; "Create a class to sync MediaWiki:Common.css etc on labs with production" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/68309
[23:24:26] <binasher>	 where are the scap scripts actually documented, i should add restart-twemproxy there i suppose
[23:24:57] <Ryan_Lane>	 binasher: they aren't AFAIK
[23:25:19] <ori-l>	 binasher: mostly files/scap in operations/puppet
[23:26:32] <binasher>	 ori-l: that's where restart-twemproxy is, i was wondering if i should document it
[23:26:56] <ori-l>	 well lookie here, we got ourselves a fancy "documentation" fella!
[23:27:10] <ori-l>	 terse shell scripts with obscure punctuation not good enough for ya?
[23:27:13] <binasher>	 nope, gave up :)
[23:27:37] <ori-l>	 if you add restart-twemproxy to sync-common-file and scap you cover all cases, IIRC
[23:27:49] <binasher>	 wait, i can document it via comments in mc-*.php files!
[23:28:14] <ori-l>	 don't forget @file
[23:28:16] <mutante>	 binasher: https://wikitech.wikimedia.org/wiki/Wikimedia_binaries
[23:28:17] <binasher>	 ori-l: don't want to run it except with the twemproxy configs actually change
[23:28:23] <binasher>	 *when
[23:28:39] <Aaron|home>	 binasher: https://wikitech.wikimedia.org/wiki/Het_deploy#Change_wiki_configuration
[23:29:03] <binasher>	 hey, that wikimedia_binaries page has actually been modified within the last month
[23:29:28] <Ryan_Lane>	 wow. holy crap. we have docs for this stuff?
[23:29:43] <mutante>	 as long as it's not in the "unsorted" section it should work :)
[23:29:57] <mutante>	 on tin !
[23:31:48] <gerrit-wm>	 New patchset: Asher; "use twemproxy on all apaches" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68308
[23:32:05] <logmsgbot>	 !log tstarling cleared profiling data
[23:32:20] <morebots>	 Logged the message, Master
[23:33:03] <gerrit-wm>	 Change merged: Asher; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68308
[23:33:12] <ori-l>	 morebots is absurdly slow since the last upgrade, but posting to the wiki hasn't gotten more expensive
[23:33:12] <morebots>	 I am a logbot running on wikitech-static.
[23:33:12] <morebots>	 Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log.
[23:33:13] <morebots>	 To log a message, type !log <msg>.
[23:33:24] <Aaron|home>	 ...
[23:33:40] <ori-l>	 it must be the timeout on the select loop or something
[23:34:11] <ori-l>	 i think it was touched recently, not sure. looking
[23:34:38] <logmsgbot>	 !log asher synchronized wmf-config  'use twemproxy for memcached access from all hosts'
[23:34:46] <morebots>	 Logged the message, Master
[23:34:57] <paravoid>	 oh!
[23:35:22] <Aaron|home>	 getting timed retry entries in the logs
[23:35:41] <binasher>	 Aaron|home: only from tampa, weird
[23:35:54] <Aaron|home>	 hmm, right
[23:38:24] <binasher>	 why oh why is just tampa screwy
[23:42:01] <logmsgbot>	 !log aaron cleared profiling data
[23:42:12] <morebots>	 Logged the message, Master
[23:43:55] <gerrit-wm>	 New patchset: Asher; "resync twemproxy-pmtpa.yaml with mc-pmtpa.php (10.0.12.15 had been pulled)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68316
[23:43:56] <Aaron|home>	 binasher: Setup.php-memcached.tavg on graphite is sexy
[23:44:25] <gerrit-wm>	 Change merged: Asher; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68316
[23:44:44] <binasher>	 Aaron|home: link?
[23:45:27] * paravoid  is watching silently

[23:45:28] <Aaron|home>	 https://graphite.wikimedia.org/render/?width=801&height=394&_salt=1371080615.671&target=Setup.php-memcached.tavg
[23:45:31] <paravoid>	 I should grab some popcorn
[23:45:50] <logmsgbot>	 !log asher synchronized wmf-config/twemproxy-pmtpa.yaml  'pull 10.0.12.15'
[23:45:59] <gerrit-wm>	 New patchset: Spage; "Add Campaigns extension" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68317
[23:46:00] <morebots>	 Logged the message, Master
[23:46:05] <paravoid>	 oh wow
[23:46:39] <paravoid>	 it's a shame we didn't get to measure the libmemcached change by itself first
[23:47:20] <logmsgbot>	 !log asher restarted twemproxy on all servers
[23:47:31] <morebots>	 Logged the message, Master
[23:47:59] <binasher>	 we can revert using twemproxy
[23:48:18] <gerrit-wm>	 New review: Spage; "Ife949daf51210d68a3328fc7b4bfef581afe80cd creates the Campaigns extension; don't deploy this config ..." [operations/mediawiki-config] (master) C: -2;  - https://gerrit.wikimedia.org/r/68317
[23:48:54] <binasher>	 which from the log, maybe we should
[23:49:03] <Aaron|home>	 DISABLED UNTIL TIMED RETRY in eqiad now :)
[23:51:06] <mutante>	 ottomata: so, the server itself must be jinxed, brewster or network can't be blamed:
[23:51:18] <ottomata>	 wha!
[23:51:25] <mutante>	 analytics1020.eqiad.wmnet has address 10.64.36.120 |  brewster dhcpd: DHCPACK on 10.64.36.120 to 90:b1:1c:12:98:d9 via 10.64.36.2
[23:51:28] <mutante>	 <--
[23:51:34] <mutante>	 as opposed to the other day
[23:51:38] <binasher>	 "SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY" has been due to libmemcached 1.0.x bugs unrelated to actual server timeouts
[23:51:45] <paravoid>	 oh?
[23:51:48] <paravoid>	 new bugs?
[23:51:54] <binasher>	 so this could be related to the new library.. or might not be!
[23:52:01] <paravoid>	 maybe it was a bad idea to upgrade both at the same time
[23:52:10] <paravoid>	 I hadn't realized you were about to deploy twemproxy today
[23:52:10] <binasher>	 probably
[23:53:20] <Aaron|home>	 Warning: Memcached::setOption(): error setting memcached option in /usr/local/apache/common-local/php-1.22wmf6/includes/objectcache/MemcachedPeclBagOStuff.php on line 67
[23:53:25] <Aaron|home>	 just starting eval.php
[23:53:41] <paravoid>	 it didn't do that yesterday on srv193
[23:53:58] <Aaron|home>	 and 71
[23:54:07] <Aaron|home>	 yeah, so that's both of those config options :)
[23:54:08] <binasher>	 OPT_RETRY_TIMEOUT
[23:54:18] <paravoid>	 that's new, right?
[23:54:35] <binasher>	 yup
[23:54:40] <gerrit-wm>	 New patchset: Asher; "Revert "use twemproxy on all apaches"" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68318
[23:55:11] <paravoid>	 and what's :71 ?
[23:55:33] <binasher>	 Aaron|home: so the question is, did it do that before the library upgrade?
[23:55:41] <Aaron|home>	 paravoid: line 71
[23:55:58] <gerrit-wm>	 Change merged: Asher; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/68318
[23:56:16] <Aaron|home>	 binasher: what version is it now?
[23:56:39] <binasher>	 1.0.17?
[23:56:49] <logmsgbot>	 !log asher synchronized wmf-config  'revert twemproxy'
[23:56:57] <paravoid>	 yes.
[23:56:59] <morebots>	 Logged the message, Master
[23:57:59] <paravoid>	   case MEMCACHED_BEHAVIOR_RETRY_TIMEOUT:
[23:58:00] <paravoid>	     if (data == 0)
[23:58:00] <paravoid>	     {
[23:58:00] <paravoid>	       return memcached_set_error(*ptr, MEMCACHED_INVALID_ARGUMENTS, MEMCACHED_AT,
[23:58:02] <paravoid>	                                         memcached_literal_param("MEMCACHED_BEHAVIOR_RETRY_TIMEOUT requires a value greater then zero."));
[23:58:05] <paravoid>	     }
[23:58:45] <binasher>	 is that new-ish?
[23:58:45] <paravoid>	 same in 1.0.8
[23:58:52] <Aaron|home>	 binasher: it does not like 0
[23:59:04] <Aaron|home>	 using a really high value works though
[23:59:08] <binasher>	 funny that the pecl extension claims to default to 0
[23:59:34] <Aaron|home>	 binasher: I wonder how high I can set it...that would be a work around :)
[23:59:48] <paravoid>	   case MEMCACHED_BEHAVIOR_SERVER_FAILURE_LIMIT:
[23:59:48] <paravoid>	     if (data == 0)
[23:59:48] <paravoid>	     {
[23:59:48] <paravoid>	       return memcached_set_error(*ptr, MEMCACHED_INVALID_ARGUMENTS, MEMCACHED_AT,
[23:59:51] <paravoid>	                                         memcached_literal_param("MEMCACHED_BEHAVIOR_SERVER_FAILURE_LIMIT requires a value greater then zero."));
[23:59:54] <Aaron|home>	 for server_failure_limit
[23:59:54] <paravoid>	     }
[23:59:57] <paravoid>	 too