[00:03:55] New patchset: Lcarr; "removing lvs monitors :(" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4598 [00:04:11] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4598 [00:04:12] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4598 [00:04:15] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4598 [00:04:21] New patchset: Bhartshorne; "moving iron to the new generic mysql class and throwing up a firewall." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4599 [00:04:28] binasher: ^^^ [00:04:36] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4599 [00:04:59] working. \o/ [00:09:15] New patchset: Bhartshorne; "moving iron to the new generic mysql class and throwing up a firewall." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4599 [00:09:31] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4599 [00:16:21] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4599 [00:16:23] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4599 [00:20:28] PROBLEM - Puppet freshness on search1004 is CRITICAL: Puppet has not run in the last 10 hours [00:28:34] PROBLEM - Puppet freshness on search1003 is CRITICAL: Puppet has not run in the last 10 hours [00:36:31] PROBLEM - Puppet freshness on search1001 is CRITICAL: Puppet has not run in the last 10 hours [00:44:28] PROBLEM - Puppet freshness on search1002 is CRITICAL: Puppet has not run in the last 10 hours [00:59:43] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [00:59:43] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [01:14:43] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [01:14:43] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [01:30:13] New patchset: Tim Starling; "Enable mod_auth on lists.wikimedia.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4603 [01:30:28] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4603 [01:31:34] New review: Tim Starling; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4603 [01:31:36] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4603 [01:33:16] !log on sodium: enabling mod_auth on lists.wikimedia.org by running puppet [01:33:19] Logged the message, Master [01:36:16] New review: Tim Starling; "Did you test this? I'm just wondering how long it was broken for." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/503 [03:49:11] oooh, it's a paravoid! good flight? [03:50:44] any swift ppl around? no maplebed [03:54:21] the swift issue is at https://commons.wikimedia.org/wiki/User_talk:Zscout370#File:Flag_of_Belarus_2012.svg [03:58:35] PROBLEM - Puppet freshness on sq34 is CRITICAL: Puppet has not run in the last 10 hours [03:58:59] !b 34755 [03:58:59] https://bugzilla.wikimedia.org/34755 [03:59:17] (discussion going about this now in #wikimedia) [04:05:09] New review: Krinkle; "Do you still want to remove this, now that https://gerrit.wikimedia.org/r/#change,4366 is in?" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/4364 [04:08:39] maplebed is SF? [04:21:45] ok, mailed ben+aaron [04:55:54] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [05:16:54] PROBLEM - MySQL Slave Running on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:26:08] PROBLEM - Puppet freshness on lvs1002 is CRITICAL: Puppet has not run in the last 10 hours [08:26:29] New review: Hashar; "Krinkle wrote:" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/4364 [09:06:52] PROBLEM - MySQL Slave Delay on db42 is CRITICAL: CRIT replication delay 281519 seconds [09:13:28] RECOVERY - mysqld processes on es1003 is OK: PROCS OK: 1 process with command name mysqld [09:28:37] PROBLEM - MySQL Slave Delay on es1003 is CRITICAL: CRIT replication delay 390763 seconds [09:30:25] !log restarted slaving on es1003, it will be a bit before it catches up. patience, young nagios [09:30:28] Logged the message, Master [10:02:06] RECOVERY - MySQL Slave Delay on es1003 is OK: OK replication delay 0 seconds [10:02:18] υαυ [10:02:19] er [10:02:21] yay :-P [10:21:45] PROBLEM - Puppet freshness on search1004 is CRITICAL: Puppet has not run in the last 10 hours [10:29:42] PROBLEM - Puppet freshness on search1003 is CRITICAL: Puppet has not run in the last 10 hours [10:37:39] PROBLEM - Puppet freshness on search1001 is CRITICAL: Puppet has not run in the last 10 hours [10:45:51] PROBLEM - Puppet freshness on search1002 is CRITICAL: Puppet has not run in the last 10 hours [11:00:51] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [11:00:51] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [11:15:52] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [11:15:52] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [11:16:12] New review: Dzahn; "yeah, appears to work for me. i do get a 401 - Unauthorized on " [operations/puppet] (production) - https://gerrit.wikimedia.org/r/503 [11:20:22] New patchset: Mark Bergsma; "Allow 3.0.2-2wm3 in the repository without upgrading existing installs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4611 [11:20:39] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4611 [11:21:54] New patchset: Mark Bergsma; "Allow 3.0.2-2wm3 in the repository without upgrading existing installs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4611 [11:22:10] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4611 [11:23:18] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4611 [11:23:20] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4611 [11:24:58] !log Imported varnish 3.0.2-2wm3 into the Wikimedia APT repository [11:25:00] Logged the message, Master [11:29:05] New review: Dzahn; "how come every single line is diff here?" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/4367 [11:32:04] New review: Reedy; "I actually have no idea..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/4367 [11:34:04] New patchset: Mark Bergsma; "Configure cp1029-1036 as eqiad Varnish servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4612 [11:34:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4612 [11:34:42] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4612 [11:34:45] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4612 [11:39:21] New review: Dzahn; "looks good though anyways. tested the new https links. the only thing left is the certificate warnin..." [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4367 [11:39:24] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4367 [11:39:40] New patchset: Mark Bergsma; "cp1029-1036 were installed with the wrong partitioning template" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4613 [11:39:55] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4613 [11:40:08] daniel? [11:40:14] hi mark [11:40:29] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4613 [11:40:30] hi [11:40:31] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4613 [11:40:36] do you have time to reinstall 8 servers? [11:40:43] cp1029-1036 were installed with the wrong partitioning [11:41:21] yea, well, at least i can start [11:41:41] thanks [11:41:49] it should be fully automatic [11:42:09] ok, cool, so does not involve partman recipes?;) [11:42:12] nope [11:42:26] you just need to do some handholding to get them to pxe boot [11:42:45] ok [11:42:57] they're not in production, you can do them all at once [11:43:05] we got your and my merge on sockpuppet right now, was at "fetch && diff" , right before merge [11:43:11] ok [11:43:15] i just merged it [11:43:18] cool [11:45:13] New patchset: Mark Bergsma; "Snapshot build" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4614 [11:45:13] New patchset: Mark Bergsma; "Compilation fixes" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4615 [11:45:14] New patchset: Mark Bergsma; "VCL support for the new chash director" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4616 [11:45:15] New patchset: Mark Bergsma; "Ran automake" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4617 [11:45:15] New patchset: Mark Bergsma; "Added chash director parser to vcc_compile.h" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4618 [11:45:16] New patchset: Mark Bergsma; "Add chash director to include/vrt.h" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4619 [11:45:17] New patchset: Mark Bergsma; "Fix wrong struct size being passed to qsort" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4620 [11:45:18] New patchset: Mark Bergsma; "Correct chash director author, styling and additional comments" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4621 [11:45:33] crap [11:45:52] Change abandoned: Mark Bergsma; "(no reason)" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4614 [11:46:04] Change abandoned: Mark Bergsma; "(no reason)" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4615 [11:46:15] Change abandoned: Mark Bergsma; "(no reason)" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4616 [11:46:26] Change abandoned: Mark Bergsma; "(no reason)" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4617 [11:46:37] Change abandoned: Mark Bergsma; "(no reason)" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4618 [11:46:48] Change abandoned: Mark Bergsma; "(no reason)" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4619 [11:46:57] Change abandoned: Mark Bergsma; "(no reason)" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4620 [11:47:07] Change abandoned: Mark Bergsma; "(no reason)" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4621 [11:48:06] New patchset: Mark Bergsma; "New version 3.0.2-2wm3: * Correct chash director author, styling and additional comments" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4622 [11:49:13] New review: Mark Bergsma; "(no comment)" [operations/debs/varnish] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4622 [11:50:18] !log pxe boot / reinstall cp1029 - cp1036 [11:50:20] Logged the message, Master [11:51:06] New review: Mark Bergsma; "(no comment)" [operations/debs/varnish] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4622 [11:51:08] Change merged: Mark Bergsma; [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4622 [11:52:02] PROBLEM - Host cp1029 is DOWN: PING CRITICAL - Packet loss = 100% [11:57:35] RECOVERY - Host cp1029 is UP: PING OK - Packet loss = 0%, RTA = 26.39 ms [12:01:29] PROBLEM - SSH on cp1029 is CRITICAL: Connection refused [12:02:23] PROBLEM - Host cp1032 is DOWN: PING CRITICAL - Packet loss = 100% [12:03:08] PROBLEM - Host cp1033 is DOWN: PING CRITICAL - Packet loss = 100% [12:04:56] PROBLEM - Host cp1029 is DOWN: PING CRITICAL - Packet loss = 100% [12:05:50] RECOVERY - SSH on cp1029 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [12:05:59] RECOVERY - Host cp1029 is UP: PING OK - Packet loss = 0%, RTA = 26.43 ms [12:07:20] PROBLEM - Host cp1034 is DOWN: PING CRITICAL - Packet loss = 100% [12:07:56] RECOVERY - Host cp1032 is UP: PING OK - Packet loss = 0%, RTA = 26.38 ms [12:08:41] RECOVERY - Host cp1033 is UP: PING OK - Packet loss = 0%, RTA = 26.59 ms [12:10:11] PROBLEM - Host cp1035 is DOWN: PING CRITICAL - Packet loss = 100% [12:11:32] PROBLEM - Host cp1036 is DOWN: PING CRITICAL - Packet loss = 100% [12:11:41] PROBLEM - SSH on cp1033 is CRITICAL: Connection refused [12:12:17] PROBLEM - SSH on cp1032 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:53] RECOVERY - Host cp1034 is UP: PING OK - Packet loss = 0%, RTA = 26.46 ms [12:15:17] RECOVERY - SSH on cp1032 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [12:15:44] RECOVERY - Host cp1035 is UP: PING OK - Packet loss = 0%, RTA = 26.38 ms [12:16:02] PROBLEM - Host cp1033 is DOWN: PING CRITICAL - Packet loss = 100% [12:16:47] PROBLEM - SSH on cp1034 is CRITICAL: Connection refused [12:17:05] RECOVERY - Host cp1036 is UP: PING OK - Packet loss = 0%, RTA = 26.39 ms [12:17:41] RECOVERY - SSH on cp1033 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [12:17:50] RECOVERY - Host cp1033 is UP: PING OK - Packet loss = 0%, RTA = 26.44 ms [12:19:29] PROBLEM - SSH on cp1035 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:26] PROBLEM - SSH on cp1036 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:11] RECOVERY - SSH on cp1035 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [12:24:08] RECOVERY - SSH on cp1036 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [12:26:14] PROBLEM - NTP on cp1029 is CRITICAL: NTP CRITICAL: Offset unknown [12:29:23] PROBLEM - NTP on cp1032 is CRITICAL: NTP CRITICAL: No response from NTP server [12:34:47] RECOVERY - NTP on cp1029 is OK: NTP OK: Offset 0.04394471645 secs [12:35:39] mark: cp1029 - puppet run: /Role::Cache::Upload/Varnish::Setup_filesystem[sdb3]/Mount[/srv/sdb3]: Could not evaluate: Execution of '/bin/mount -o noatime,nodiratime,nobarrier,logbufs=8 /srv/sdb3' returned 32: mount: /dev/sdb3: can't read superblock [12:35:48] cp1030: Role::Cache::Upload/Varnish::Instance[upload-frontend]/Exec[load-new-vcl-file-frontend]: Failed to call refresh: /usr/share/varnish/reload-vcl -n frontend returned 1 instead of one of [0] at /var/lib/git/operations/puppet/manifests/varnish.pp:128 [12:39:42] puppet runs finish though [12:40:00] PROBLEM - NTP on cp1033 is CRITICAL: NTP CRITICAL: No response from NTP server [12:41:48] PROBLEM - Varnish HTTP upload-frontend on cp1034 is CRITICAL: Connection refused [12:41:48] PROBLEM - Varnish HTCP daemon on cp1035 is CRITICAL: Connection refused by host [12:41:48] PROBLEM - Varnish HTCP daemon on cp1033 is CRITICAL: Connection refused by host [12:41:48] PROBLEM - Varnish HTTP upload-frontend on cp1036 is CRITICAL: Connection refused [12:42:15] PROBLEM - Varnish HTTP upload-backend on cp1029 is CRITICAL: Connection refused [12:42:15] PROBLEM - Varnish traffic logger on cp1036 is CRITICAL: Connection refused by host [12:42:15] PROBLEM - Varnish traffic logger on cp1034 is CRITICAL: Connection refused by host [12:42:15] PROBLEM - Varnish HTTP upload-backend on cp1035 is CRITICAL: Connection refused [12:42:24] PROBLEM - Varnish HTTP upload-backend on cp1033 is CRITICAL: Connection refused [12:42:42] PROBLEM - Varnish HTTP upload-frontend on cp1035 is CRITICAL: Connection refused [12:42:42] PROBLEM - Varnish HTTP upload-frontend on cp1029 is CRITICAL: Connection refused [12:42:51] PROBLEM - Varnish HTTP upload-frontend on cp1033 is CRITICAL: Connection refused [12:42:51] PROBLEM - Varnish HTCP daemon on cp1036 is CRITICAL: Connection refused by host [12:42:51] PROBLEM - Varnish HTCP daemon on cp1034 is CRITICAL: Connection refused by host [12:43:00] PROBLEM - Varnish HTTP upload-backend on cp1036 is CRITICAL: Connection refused [12:43:00] PROBLEM - Varnish traffic logger on cp1035 is CRITICAL: Connection refused by host [12:43:00] PROBLEM - Varnish HTTP upload-backend on cp1034 is CRITICAL: Connection refused [12:44:12] PROBLEM - NTP on cp1035 is CRITICAL: NTP CRITICAL: Offset unknown [12:44:57] PROBLEM - NTP on cp1036 is CRITICAL: NTP CRITICAL: No response from NTP server [12:46:00] RECOVERY - Varnish traffic logger on cp1035 is OK: PROCS OK: 2 processes with command name varnishncsa [12:47:57] RECOVERY - Varnish traffic logger on cp1036 is OK: PROCS OK: 2 processes with command name varnishncsa [12:49:00] RECOVERY - NTP on cp1032 is OK: NTP OK: Offset 0.02929723263 secs [12:49:54] RECOVERY - NTP on cp1035 is OK: NTP OK: Offset -0.04992544651 secs [12:51:51] RECOVERY - NTP on cp1033 is OK: NTP OK: Offset -0.01880443096 secs [12:52:09] PROBLEM - Host cp1034 is DOWN: PING CRITICAL - Packet loss = 100% [12:55:33] mutante: oh darn [12:55:38] that means they're installed wrong again [12:55:46] probably because puppet on brewster didn't update yet [12:57:03] ah, so maybe it updated while i was doing this. because on 1030 i did not see the "superblock" error [12:57:42] RECOVERY - Host cp1034 is UP: PING OK - Packet loss = 0%, RTA = 26.40 ms [12:57:50] ok [12:57:51] RECOVERY - NTP on cp1036 is OK: NTP OK: Offset -0.007479310036 secs [12:59:04] redoing 1031 and 1034. one showed superblock error during install, other i got disconnected from console and then didnt get any output. on second attempt they appear to install fine [13:05:30] RECOVERY - SSH on cp1034 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:06:49] arr, can't login with the new_install key now.. [13:07:55] nevermind:) [13:12:33] RECOVERY - Varnish traffic logger on cp1034 is OK: PROCS OK: 2 processes with command name varnishncsa [13:19:54] PROBLEM - Host cp1029 is DOWN: PING CRITICAL - Packet loss = 100% [13:25:27] RECOVERY - Host cp1029 is UP: PING OK - Packet loss = 0%, RTA = 26.45 ms [13:25:36] RECOVERY - SSH on search1001 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:25:54] RECOVERY - Varnish HTCP daemon on cp1033 is OK: PROCS OK: 1 process with UID = 998 (varnishhtcpd), args varnishhtcpd worker [13:26:12] RECOVERY - SSH on search1002 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:28:09] RECOVERY - Host search1005 is UP: PING OK - Packet loss = 0%, RTA = 26.62 ms [13:29:03] PROBLEM - SSH on cp1029 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:29:03] PROBLEM - Varnish HTCP daemon on cp1029 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:29:48] PROBLEM - Varnish traffic logger on cp1029 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:31:36] PROBLEM - SSH on search1005 is CRITICAL: Connection refused [13:31:54] RECOVERY - SSH on cp1029 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:32:48] PROBLEM - Varnish traffic logger on cp1034 is CRITICAL: Connection refused by host [13:33:15] RECOVERY - SSH on search1004 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:33:51] RECOVERY - Host search1007 is UP: PING OK - Packet loss = 0%, RTA = 26.61 ms [13:33:51] RECOVERY - Host search1006 is UP: PING OK - Packet loss = 0%, RTA = 26.46 ms [13:35:30] PROBLEM - Host search1005 is DOWN: PING CRITICAL - Packet loss = 100% [13:35:39] RECOVERY - Varnish traffic logger on cp1029 is OK: PROCS OK: 2 processes with command name varnishncsa [13:36:47] woo lookit all those search boxen come alive [13:37:37] <^demon> Did you throw a switch and lightning rushed down from the roof to fill them with surging energy? [13:37:45] RECOVERY - SSH on search1005 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:37:45] PROBLEM - SSH on search1007 is CRITICAL: Connection refused [13:37:54] RECOVERY - Host search1005 is UP: PING OK - Packet loss = 0%, RTA = 26.44 ms [13:38:19] PROBLEM - SSH on search1006 is CRITICAL: Connection refused [13:38:46] PROBLEM - NTP on search1005 is CRITICAL: NTP CRITICAL: No response from NTP server [13:39:49] PROBLEM - Lucene on search1005 is CRITICAL: Connection refused [13:40:02] ^demon: more like alchemy [13:40:12] i turned my blood and sweat into working servers for notpeter to install [13:40:34] <^demon> Ah, well as long as it was cooler than just pushing a button :) [13:41:19] PROBLEM - Varnish HTTP upload-frontend on cp1030 is CRITICAL: Connection refused [13:41:19] PROBLEM - Varnish HTTP upload-frontend on cp1032 is CRITICAL: Connection refused [13:41:19] PROBLEM - Varnish HTCP daemon on cp1031 is CRITICAL: NRPE: Command check_varnishhtcpd not defined [13:41:19] RECOVERY - Varnish HTCP daemon on cp1029 is OK: PROCS OK: 1 process with UID = 998 (varnishhtcpd), args varnishhtcpd worker [13:41:46] PROBLEM - Varnish HTTP upload-backend on cp1031 is CRITICAL: Connection refused [13:41:55] PROBLEM - Lucene on search1007 is CRITICAL: Connection refused [13:42:04] PROBLEM - Varnish HTTP upload-frontend on cp1031 is CRITICAL: Connection refused [13:42:04] PROBLEM - Varnish HTCP daemon on cp1030 is CRITICAL: NRPE: Command check_varnishhtcpd not defined [13:42:04] RECOVERY - Host search1008 is UP: PING OK - Packet loss = 0%, RTA = 26.37 ms [13:42:04] RECOVERY - Host search1009 is UP: PING OK - Packet loss = 0%, RTA = 26.79 ms [13:42:13] RECOVERY - Host search1011 is UP: PING OK - Packet loss = 0%, RTA = 26.43 ms [13:42:13] RECOVERY - Host search1010 is UP: PING OK - Packet loss = 0%, RTA = 26.39 ms [13:42:13] RECOVERY - Host search1012 is UP: PING OK - Packet loss = 0%, RTA = 26.42 ms [13:42:31] PROBLEM - Varnish HTTP upload-backend on cp1032 is CRITICAL: Connection refused [13:42:31] PROBLEM - Varnish HTTP upload-backend on cp1030 is CRITICAL: Connection refused [13:43:52] PROBLEM - Host search1006 is DOWN: PING CRITICAL - Packet loss = 100% [13:44:32] New patchset: Mark Bergsma; "Add cp1029 to the pool" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4627 [13:44:48] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4627 [13:44:55] RECOVERY - SSH on search1006 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:45:48] RECOVERY - Host search1006 is UP: PING OK - Packet loss = 0%, RTA = 26.43 ms [13:45:48] PROBLEM - SSH on search1008 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:48] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4627 [13:45:48] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4627 [13:45:48] PROBLEM - SSH on search1011 is CRITICAL: Connection refused [13:45:48] RECOVERY - SSH on search1007 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:45:49] PROBLEM - SSH on search1010 is CRITICAL: Connection refused [13:46:16] PROBLEM - SSH on search1012 is CRITICAL: Connection refused [13:46:25] PROBLEM - SSH on search1009 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:55] RECOVERY - Host search1015 is UP: PING OK - Packet loss = 0%, RTA = 26.45 ms [13:47:55] RECOVERY - Host search1014 is UP: PING OK - Packet loss = 0%, RTA = 26.59 ms [13:47:55] RECOVERY - Host search1013 is UP: PING OK - Packet loss = 0%, RTA = 26.51 ms [13:47:55] RECOVERY - Host search1016 is UP: PING OK - Packet loss = 0%, RTA = 26.41 ms [13:48:13] RECOVERY - SSH on search1008 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:50:19] PROBLEM - Lucene on search1008 is CRITICAL: Connection refused [13:50:37] PROBLEM - Lucene on search1011 is CRITICAL: Connection timed out [13:50:46] RECOVERY - SSH on search1009 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:50:55] PROBLEM - Lucene on search1010 is CRITICAL: Connection timed out [13:51:22] PROBLEM - Lucene on search1012 is CRITICAL: Connection timed out [13:51:31] PROBLEM - SSH on search1016 is CRITICAL: Connection refused [13:51:31] RECOVERY - SSH on search1011 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:51:40] RECOVERY - SSH on search1010 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:51:40] PROBLEM - SSH on search1015 is CRITICAL: Connection refused [13:51:40] PROBLEM - SSH on search1014 is CRITICAL: Connection refused [13:52:25] PROBLEM - SSH on search1013 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:52] RECOVERY - Varnish HTCP daemon on cp1031 is OK: PROCS OK: 1 process with UID = 998 (varnishhtcpd), args varnishhtcpd worker [13:53:19] PROBLEM - Lucene on search1006 is CRITICAL: Connection refused [13:53:37] RECOVERY - Host search1017 is UP: PING OK - Packet loss = 0%, RTA = 26.46 ms [13:53:37] RECOVERY - Host search1018 is UP: PING OK - Packet loss = 0%, RTA = 26.77 ms [13:54:49] RECOVERY - Varnish traffic logger on cp1034 is OK: PROCS OK: 2 processes with command name varnishncsa [13:54:58] RECOVERY - Varnish HTCP daemon on cp1034 is OK: PROCS OK: 1 process with UID = 998 (varnishhtcpd), args varnishhtcpd worker [13:54:58] RECOVERY - Varnish HTTP upload-backend on cp1029 is OK: HTTP OK HTTP/1.1 200 OK - 632 bytes in 0.053 seconds [13:54:58] RECOVERY - SSH on search1012 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:55:16] PROBLEM - Host search1013 is DOWN: PING CRITICAL - Packet loss = 100% [13:55:43] RECOVERY - Varnish HTCP daemon on cp1035 is OK: PROCS OK: 1 process with UID = 998 (varnishhtcpd), args varnishhtcpd worker [13:56:01] PROBLEM - NTP on search1007 is CRITICAL: NTP CRITICAL: No response from NTP server [13:56:19] RECOVERY - SSH on search1014 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:56:28] PROBLEM - Lucene on search1016 is CRITICAL: Connection refused [13:56:28] RECOVERY - Varnish HTCP daemon on cp1036 is OK: PROCS OK: 1 process with UID = 998 (varnishhtcpd), args varnishhtcpd worker [13:56:37] PROBLEM - Lucene on search1015 is CRITICAL: Connection refused [13:56:46] RECOVERY - SSH on search1013 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:56:46] PROBLEM - SSH on search1018 is CRITICAL: Connection refused [13:56:55] PROBLEM - SSH on search1017 is CRITICAL: Connection refused [13:56:55] RECOVERY - Host search1013 is UP: PING OK - Packet loss = 0%, RTA = 26.42 ms [13:57:13] PROBLEM - Lucene on search1009 is CRITICAL: Connection refused [13:59:46] PROBLEM - Puppet freshness on sq34 is CRITICAL: Puppet has not run in the last 10 hours [14:00:58] RECOVERY - Varnish HTCP daemon on cp1030 is OK: PROCS OK: 1 process with UID = 998 (varnishhtcpd), args varnishhtcpd worker [14:01:34] PROBLEM - Lucene on search1017 is CRITICAL: Connection timed out [14:02:10] PROBLEM - Lucene on search1018 is CRITICAL: Connection timed out [14:02:37] PROBLEM - Lucene on search1014 is CRITICAL: Connection refused [14:02:37] PROBLEM - NTP on search1008 is CRITICAL: NTP CRITICAL: No response from NTP server [14:03:22] PROBLEM - NTP on search1011 is CRITICAL: NTP CRITICAL: No response from NTP server [14:03:22] PROBLEM - NTP on search1010 is CRITICAL: NTP CRITICAL: No response from NTP server [14:04:07] RECOVERY - SSH on search1018 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:04:16] RECOVERY - SSH on search1016 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:05:28] PROBLEM - Lucene on search1013 is CRITICAL: Connection refused [14:05:37] RECOVERY - SSH on search1017 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:06:40] PROBLEM - NTP on search1006 is CRITICAL: NTP CRITICAL: No response from NTP server [14:08:55] RECOVERY - SSH on search1015 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:09:22] PROBLEM - NTP on search1016 is CRITICAL: NTP CRITICAL: No response from NTP server [14:09:49] PROBLEM - NTP on search1014 is CRITICAL: NTP CRITICAL: No response from NTP server [14:09:58] PROBLEM - NTP on search1015 is CRITICAL: NTP CRITICAL: No response from NTP server [14:10:07] PROBLEM - NTP on search1009 is CRITICAL: NTP CRITICAL: No response from NTP server [14:15:13] PROBLEM - NTP on search1017 is CRITICAL: NTP CRITICAL: No response from NTP server [14:15:13] PROBLEM - NTP on search1018 is CRITICAL: NTP CRITICAL: No response from NTP server [14:16:34] PROBLEM - NTP on search1012 is CRITICAL: NTP CRITICAL: No response from NTP server [14:17:22] !log search in eqiad is being reinstalled, no need to be alarmed (thats a pun!) [14:17:24] Logged the message, Master [14:18:04] PROBLEM - NTP on search1013 is CRITICAL: NTP CRITICAL: No response from NTP server [14:24:40] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 11.6225215 (gt 8.0) [14:33:31] RECOVERY - Host search1019 is UP: PING OK - Packet loss = 0%, RTA = 26.40 ms [14:36:31] PROBLEM - SSH on search1019 is CRITICAL: Connection refused [14:38:46] RECOVERY - Varnish HTTP upload-frontend on cp1030 is OK: HTTP OK HTTP/1.1 200 OK - 643 bytes in 0.053 seconds [14:39:13] RECOVERY - Host search1020 is UP: PING OK - Packet loss = 0%, RTA = 26.46 ms [14:39:13] RECOVERY - Host search1023 is UP: PING OK - Packet loss = 0%, RTA = 26.43 ms [14:39:14] RECOVERY - Host search1021 is UP: PING OK - Packet loss = 0%, RTA = 26.67 ms [14:39:14] RECOVERY - Host search1024 is UP: PING OK - Packet loss = 0%, RTA = 26.89 ms [14:39:14] RECOVERY - Host search1022 is UP: PING OK - Packet loss = 0%, RTA = 26.47 ms [14:39:40] RECOVERY - Varnish HTTP upload-backend on cp1030 is OK: HTTP OK HTTP/1.1 200 OK - 634 bytes in 0.053 seconds [14:40:36] PROBLEM - Disk space on search1021 is CRITICAL: Connection refused by host [14:40:36] PROBLEM - SSH on search1021 is CRITICAL: Connection refused [14:40:36] PROBLEM - SSH on search1024 is CRITICAL: Connection refused [14:40:36] PROBLEM - NTP on search1022 is CRITICAL: NTP CRITICAL: No response from NTP server [14:40:36] PROBLEM - NTP on search1023 is CRITICAL: NTP CRITICAL: No response from NTP server [14:40:54] PROBLEM - SSH on search1023 is CRITICAL: Connection refused [14:40:54] PROBLEM - NTP on search1020 is CRITICAL: NTP CRITICAL: No response from NTP server [14:41:21] PROBLEM - Disk space on search1022 is CRITICAL: Connection refused by host [14:41:21] PROBLEM - SSH on search1022 is CRITICAL: Connection refused [14:41:39] PROBLEM - DPKG on search1021 is CRITICAL: Connection refused by host [14:41:48] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 3.79167176471 [14:42:06] RECOVERY - Varnish HTTP upload-backend on cp1031 is OK: HTTP OK HTTP/1.1 200 OK - 634 bytes in 0.053 seconds [14:42:24] PROBLEM - RAID on search1022 is CRITICAL: Connection refused by host [14:42:24] PROBLEM - DPKG on search1022 is CRITICAL: Connection refused by host [14:42:33] RECOVERY - Varnish HTTP upload-frontend on cp1031 is OK: HTTP OK HTTP/1.1 200 OK - 643 bytes in 0.053 seconds [14:42:42] PROBLEM - SSH on search1020 is CRITICAL: Connection refused [14:42:42] PROBLEM - Lucene on search1019 is CRITICAL: Connection timed out [14:42:51] RECOVERY - Varnish HTTP upload-backend on cp1032 is OK: HTTP OK HTTP/1.1 200 OK - 634 bytes in 0.053 seconds [14:43:00] PROBLEM - RAID on search1021 is CRITICAL: Connection refused by host [14:43:09] RECOVERY - Varnish HTTP upload-frontend on cp1032 is OK: HTTP OK HTTP/1.1 200 OK - 643 bytes in 0.053 seconds [14:43:27] RECOVERY - Varnish HTTP upload-backend on cp1033 is OK: HTTP OK HTTP/1.1 200 OK - 632 bytes in 0.053 seconds [14:43:54] RECOVERY - Varnish HTTP upload-frontend on cp1033 is OK: HTTP OK HTTP/1.1 200 OK - 641 bytes in 0.054 seconds [14:44:30] RECOVERY - Varnish HTTP upload-frontend on cp1034 is OK: HTTP OK HTTP/1.1 200 OK - 643 bytes in 0.053 seconds [14:44:57] RECOVERY - Varnish HTTP upload-backend on cp1035 is OK: HTTP OK HTTP/1.1 200 OK - 634 bytes in 0.053 seconds [14:45:24] RECOVERY - Varnish HTTP upload-frontend on cp1035 is OK: HTTP OK HTTP/1.1 200 OK - 641 bytes in 0.054 seconds [14:45:51] RECOVERY - SSH on search1019 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:45:51] PROBLEM - Host search1020 is DOWN: PING CRITICAL - Packet loss = 100% [14:46:00] RECOVERY - Varnish HTTP upload-frontend on cp1036 is OK: HTTP OK HTTP/1.1 200 OK - 643 bytes in 0.053 seconds [14:46:54] PROBLEM - Lucene on search1021 is CRITICAL: Connection timed out [14:47:03] PROBLEM - Lucene on search1024 is CRITICAL: Connection refused [14:47:03] RECOVERY - Varnish HTTP upload-backend on cp1036 is OK: HTTP OK HTTP/1.1 200 OK - 634 bytes in 0.054 seconds [14:47:21] PROBLEM - Lucene on search1023 is CRITICAL: Connection refused [14:47:35] New patchset: Mark Bergsma; "Add cp1030-1032 to the pool" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4635 [14:47:39] PROBLEM - Lucene on search1022 is CRITICAL: Connection timed out [14:47:48] RECOVERY - SSH on search1021 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:47:51] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4635 [14:47:59] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4635 [14:48:02] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4635 [14:48:24] RECOVERY - SSH on search1020 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:48:33] RECOVERY - Host search1020 is UP: PING OK - Packet loss = 0%, RTA = 26.38 ms [14:49:54] RECOVERY - SSH on search1022 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:51:24] PROBLEM - Host search1023 is DOWN: PING CRITICAL - Packet loss = 100% [14:53:21] RECOVERY - SSH on search1024 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:54:06] RECOVERY - SSH on search1023 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:54:15] RECOVERY - Host search1023 is UP: PING OK - Packet loss = 0%, RTA = 26.43 ms [14:56:21] PROBLEM - Lucene on search1020 is CRITICAL: Connection refused [14:56:30] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [14:59:49] PROBLEM - NTP on search1021 is CRITICAL: NTP CRITICAL: No response from NTP server [15:00:06] PROBLEM - NTP on search1024 is CRITICAL: NTP CRITICAL: No response from NTP server [15:07:00] PROBLEM - NTP on search1019 is CRITICAL: NTP CRITICAL: No response from NTP server [15:12:32] New patchset: Mark Bergsma; "Add the remaining upload servers to the pool" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4639 [15:12:48] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4639 [15:12:54] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4639 [15:12:56] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4639 [15:13:27] PROBLEM - Host search1003 is DOWN: PING CRITICAL - Packet loss = 100% [15:15:33] RECOVERY - SSH on search1003 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [15:15:42] RECOVERY - Host search1003 is UP: PING OK - Packet loss = 0%, RTA = 26.44 ms [15:19:18] RECOVERY - Puppet freshness on search1001 is OK: puppet ran at Tue Apr 10 15:19:08 UTC 2012 [15:20:32] hi hexmode, do you have access to the bugzilla apache access or error log? [15:21:12] drdee: I don't have access to that, but maybe I can help you figure out what the problem is [15:21:33] ^demon: do you have access to the bz machine? [15:21:52] <^demon> I can get to it, but I don't have root, what's up? [15:22:15] ^demon: drdee wants to see what is in the error logs [15:22:26] he is trying to set up a bz client [15:22:30] and it isn't working [15:22:54] so maybe the xmlrpc endpoint is spewing errors or something [15:23:05] <^demon> Apache logs? [15:23:07] exactly [15:23:17] assuming that bugzilla runs on apache [15:23:31] drdee: what do you have in your set up for the endpoint? https://bugzilla.wikimedia.org/ ? [15:23:36] yes [15:24:06] RECOVERY - Puppet freshness on search1002 is OK: puppet ran at Tue Apr 10 15:23:37 UTC 2012 [15:24:11] drdee: Did you try https://bugzilla.wikimedia.org/xmlrpc.cgi as well? [15:24:32] <^demon> Hmm, well apache logs seem to be owned by root:adm, can't touch them :\ [15:24:33] yes but pivotal tracker says that you have to exclude xmlrpci.cgi [15:24:52] it could be that pivotal tracker does not support https [15:25:19] i have looked at the web console to look at the ajax requests but i only see the json error response [15:25:25] so that doesn't really help [15:26:15] drdee: is pivotal running on a machine you have access to? [15:26:35] no, it's a 3rd party service [15:27:28] <^demon> hexmode, drdee: I can't access /var/log/apache/ or /srv/org/wikimedia/bugzilla/, can't help I'm afraid :\ [15:27:51] drdee: so, maybe you can ask pivotal to tell you what is wrong? [15:28:00] ^demon: ty anyway :) [15:28:44] <^demon> I used pivotal before. Was not a fan ;-) [15:29:12] RECOVERY - NTP on search1001 is OK: NTP OK: Offset 0.0014564991 secs [15:29:21] RECOVERY - Puppet freshness on search1003 is OK: puppet ran at Tue Apr 10 15:29:11 UTC 2012 [15:29:51] demon: no? i am just looking for a decent scrum pm tool and it seemed okay [15:30:21] ^demon:thanks for trying, who does have access to those files? [15:30:35] <^demon> I think you'll need to track down a root. [15:30:35] FYI, puppet camp now scheduled for may 19 in LAX [15:30:43] <^demon> In the airport? Yuck. [15:30:52] ^demon: not afaik [15:31:17] Geneva and Dublin prolly conflict with wikimania [15:32:12]