[00:38:36] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [00:40:06] RECOVERY - Host bellin is UP: PING OK - Packet loss = 0%, RTA = 0.60 ms [00:57:39] PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours [01:03:39] PROBLEM - Puppet freshness on search17 is CRITICAL: Puppet has not run in the last 10 hours [01:07:33] PROBLEM - Puppet freshness on search15 is CRITICAL: Puppet has not run in the last 10 hours [01:14:36] PROBLEM - Puppet freshness on search14 is CRITICAL: Puppet has not run in the last 10 hours [01:14:36] PROBLEM - Puppet freshness on search19 is CRITICAL: Puppet has not run in the last 10 hours [01:14:36] PROBLEM - Puppet freshness on search16 is CRITICAL: Puppet has not run in the last 10 hours [01:24:39] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [01:41:18] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 252 seconds [01:46:51] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 0 seconds [03:29:36] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [03:31:51] RECOVERY - Host bellin is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [03:50:27] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [04:04:09] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [04:05:39] RECOVERY - Host bellin is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms [04:54:15] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [04:56:21] RECOVERY - Host bellin is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [06:41:29] PROBLEM - Puppet freshness on bellin is CRITICAL: Puppet has not run in the last 10 hours [07:24:27] PROBLEM - Puppet freshness on es1003 is CRITICAL: Puppet has not run in the last 10 hours [07:24:27] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [07:24:27] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [08:05:59] PROBLEM - Puppet freshness on search20 is CRITICAL: Puppet has not run in the last 10 hours [08:28:56] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [08:49:45] New review: Reedy; "(no comment)" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9717 [08:49:47] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/9717 [09:08:41] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [09:09:53] RECOVERY - Host bellin is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms [09:10:10] notpeter: ^^^ lol [09:13:11] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [09:13:38] RECOVERY - Host bellin is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms [09:22:29] New patchset: Asher; "adding db16 to decom" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9728 [09:22:50] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9728 [09:24:03] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9728 [09:24:05] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9728 [09:30:22] PROBLEM - Apache HTTP on srv221 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:34:07] PROBLEM - Backend Squid HTTP on knsq18 is CRITICAL: Connection refused [09:38:46] RECOVERY - Apache HTTP on srv221 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.151 second response time [09:40:43] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [09:41:46] RECOVERY - Host bellin is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [09:43:16] PROBLEM - Apache HTTP on srv221 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:57:31] RECOVERY - Apache HTTP on srv221 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 9.161 second response time [10:03:13] PROBLEM - Apache HTTP on srv221 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:04:25] RECOVERY - Apache HTTP on srv221 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 7.800 second response time [10:36:26] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [10:38:23] RECOVERY - Host bellin is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms [10:54:05] New patchset: Jdlrobson; "match varnish config with DeviceDetection.php (bug 36935)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9640 [10:54:28] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9640 [10:54:56] hi robh [10:58:47] PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours [11:04:47] PROBLEM - Puppet freshness on search17 is CRITICAL: Puppet has not run in the last 10 hours [11:08:41] PROBLEM - Puppet freshness on search15 is CRITICAL: Puppet has not run in the last 10 hours [11:15:45] PROBLEM - Puppet freshness on search14 is CRITICAL: Puppet has not run in the last 10 hours [11:15:45] PROBLEM - Puppet freshness on search16 is CRITICAL: Puppet has not run in the last 10 hours [11:15:45] PROBLEM - Puppet freshness on search19 is CRITICAL: Puppet has not run in the last 10 hours [11:22:27] RECOVERY - Host search18 is UP: PING OK - Packet loss = 0%, RTA = 0.50 ms [11:25:54] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [11:26:12] PROBLEM - SSH on search18 is CRITICAL: Connection refused [11:26:30] PROBLEM - Lucene disk space on search18 is CRITICAL: Connection refused by host [11:30:42] PROBLEM - Lucene on search18 is CRITICAL: Connection refused [11:42:51] PROBLEM - NTP on search18 is CRITICAL: NTP CRITICAL: No response from NTP server [11:58:18] RECOVERY - SSH on search18 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [12:07:54] RECOVERY - Puppet freshness on search20 is OK: puppet ran at Sat Jun 2 12:07:35 UTC 2012 [12:10:00] RECOVERY - Lucene disk space on search20 is OK: DISK OK [12:10:00] RECOVERY - Lucene disk space on search18 is OK: DISK OK [12:10:15] domas: db13 is on the decom list [12:13:25] whyyy [12:14:32] New patchset: Dzahn; "cronspam sprint - srv222 - use -ignore_readdir_race with find to avoid errors for missing files" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9752 [12:14:55] New patchset: Mark Bergsma; "Cron Spam sprint: silence varnish restart cron job" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9753 [12:15:17] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9752 [12:15:17] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/9753 [12:15:17] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9753 [12:15:28] domas: db13 might become an mha management host or something [12:15:33] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/9753 [12:15:33] =) [12:15:35] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9753 [12:15:47] thats a good use for 16 disk 32G server! [12:15:52] New review: Dzahn; "see man find for ignore_readdir_race" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9752 [12:15:54] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9752 [12:16:54] Jeff_Green: db26 would just need a "-c 1" option added to its snaprotate cron [12:17:10] ok [12:19:32] RECOVERY - Lucene on search20 is OK: TCP OK - 0.009 second response time on port 8123 [12:27:02] RECOVERY - NTP on search18 is OK: NTP OK: Offset -0.01420676708 secs [12:27:02] RECOVERY - NTP on search20 is OK: NTP OK: Offset -0.03076040745 secs [12:28:03] New patchset: Mark Bergsma; "Remove br1-knams (decommissioned)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9755 [12:28:24] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9755 [12:29:48] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/9755 [12:29:50] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9755 [12:38:17] New patchset: Jgreen; "modifying snaprotate job for db26 b/c it lacks space for snapshots, so we're keeping fewer now" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9759 [12:38:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9759 [12:39:19] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9759 [12:39:21] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9759 [12:39:28] New patchset: Bhartshorne; "porting over changes made only in puppet to bring the two back in sync." [operations/software] (master) - https://gerrit.wikimedia.org/r/9760 [12:39:28] New patchset: Bhartshorne; "the stack traces are all for normal errors. Instead of raising an exceptoin, log it and fail instead." [operations/software] (master) - https://gerrit.wikimedia.org/r/9761 [12:40:05] New review: Bhartshorne; "(no comment)" [operations/software] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9760 [12:40:08] Change merged: Bhartshorne; [operations/software] (master) - https://gerrit.wikimedia.org/r/9760 [12:42:46] New review: Bhartshorne; "(no comment)" [operations/software] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9761 [12:42:48] Change merged: Bhartshorne; [operations/software] (master) - https://gerrit.wikimedia.org/r/9761 [12:43:02] New patchset: Dzahn; "cronspam - adding 'missingok' to logrotate config to avoid mails for missing files" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9764 [12:43:23] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9764 [12:43:58] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9764 [12:44:00] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9764 [12:50:10] New patchset: Dzahn; "cronspam - locke - adding another 'missingok' for aft-udp2log" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9767 [12:50:30] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9767 [12:51:27] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9767 [12:51:32] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9767 [12:53:03] New patchset: Jgreen; "reduce lvm snapshot count to 1, (-c 1) for db10 b/c it lacks space for more" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9769 [12:53:25] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9769 [12:53:27] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9769 [12:53:30] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9769 [13:06:13] New patchset: Bhartshorne; "crosspam - pulling in changes to suppress unnecessary stack traces and just print an error message and return instead" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9771 [13:06:34] New patchset: Bhartshorne; "specifying pull path to htcp.pph" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9772 [13:06:55] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9771 [13:06:55] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9771 [13:06:55] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9771 [13:06:55] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9772 [13:10:05] New patchset: Faidon; "Remove redundant debian/ files" [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9773 [13:10:06] New patchset: Faidon; "Use dh_installinit for the default file" [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9774 [13:10:07] New patchset: Faidon; "Use dh_install to install sysctl.conf" [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9775 [13:10:07] New patchset: Faidon; "Use debian/dirs instead of mkdir" [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9776 [13:10:08] New patchset: Faidon; "Bump version to 0.05, build for precise" [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9777 [13:10:09] New patchset: Faidon; "Switch to dh" [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9778 [13:10:09] New patchset: Faidon; "Remove redundant comments" [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9779 [13:10:32] New patchset: Bhartshorne; "specifying full path to htcp.pph" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9772 [13:10:53] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9772 [13:10:53] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9772 [13:10:55] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9772 [13:11:50] [user] name = Faidon Liambotis email = faidon@wikimedia.org [13:12:01] fortunately not a password [13:12:26] New review: Faidon; "(no comment)" [operations/debs/wikimedia-lvs-realserver] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/9773 [13:12:38] New review: Faidon; "(no comment)" [operations/debs/wikimedia-lvs-realserver] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9773 [13:12:40] Change merged: Faidon; [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9773 [13:12:46] New review: Faidon; "(no comment)" [operations/debs/wikimedia-lvs-realserver] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9774 [13:12:48] Change merged: Faidon; [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9774 [13:13:04] New review: Faidon; "(no comment)" [operations/debs/wikimedia-lvs-realserver] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9775 [13:13:06] Change merged: Faidon; [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9775 [13:13:13] New review: Faidon; "(no comment)" [operations/debs/wikimedia-lvs-realserver] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9776 [13:13:15] Change merged: Faidon; [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9776 [13:13:30] New review: Faidon; "(no comment)" [operations/debs/wikimedia-lvs-realserver] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9777 [13:13:31] Change merged: Faidon; [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9777 [13:13:41] New review: Faidon; "(no comment)" [operations/debs/wikimedia-lvs-realserver] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9778 [13:13:42] Change merged: Faidon; [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9778 [13:13:52] New review: Faidon; "(no comment)" [operations/debs/wikimedia-lvs-realserver] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9779 [13:13:53] Change merged: Faidon; [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9779 [13:21:53] !log updated quotas on labstore1 for publicdata-proect [13:21:59] Logged the message, Master [13:23:00] New patchset: Pyoungmeister; "disabling pipeline_prefetch for squid" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9781 [13:23:21] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9781 [13:25:02] notpeter: [13:25:02] cat > /etc/apt/apt.conf.d/90disable-pipelining [13:25:03] Acquire::http::Pipeline-Depth "0"; [13:25:18] paravoid, wrong console? [13:25:23] Change abandoned: Pyoungmeister; "wrong-o" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9781 [13:26:01] Platonides: not this time! [13:26:47] New patchset: Faidon; "Release 0.05" [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9782 [13:27:33] New review: Faidon; "(no comment)" [operations/debs/wikimedia-lvs-realserver] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9782 [13:35:05] mark: wikimedia-lvs-realserver 0.05 is in precise-wikimedia/main now [13:36:40] paravoid: you rock [13:37:51] ah, right, you were interested in that too [13:41:57] New patchset: Hashar; "extensions separation per cluster" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/9790 [13:42:03] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/9790 [13:42:48] Reedy: https://gerrit.wikimedia.org/r/9790 [13:44:40] paravoid: do I need any magic to make that apt conf take? [13:46:20] !log rebuilding archives for fd-advisorygroup mailing list [13:46:24] Logged the message, Master [13:46:55] notpeter: no [13:47:05] paravoid: damnit. [13:51:37] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [13:52:51] New review: Hashar; "As discussed with Sam" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9790 [13:52:53] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/9790 [13:56:55] New patchset: Hashar; "disable CheckUser on labs" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/9796 [13:57:01] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/9796 [13:58:53] New patchset: Hashar; "disable CheckUser on labs" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/9796 [13:58:58] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/9796 [13:59:46] New review: Hashar; "As discussed with Sam" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9796 [13:59:48] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/9796 [14:03:16] New patchset: Pyoungmeister; "removing some old karmic-related code" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9797 [14:03:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9797 [14:08:07] RECOVERY - Puppet freshness on search15 is OK: puppet ran at Sat Jun 2 14:07:52 UTC 2012 [14:10:07] !log deploying some nasty configuration changes in wmf-config [14:10:12] Logged the message, Master [14:16:56] RECOVERY - Lucene disk space on search15 is OK: DISK OK [14:19:12] New patchset: Faidon; "Prepare selective-answer.py for a blacklist" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9798 [14:19:34] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9798 [14:19:36] mark: ^^^ [14:21:40] New patchset: Pyoungmeister; "making all apt files be in place before apt-get update runs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9800 [14:22:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9800 [14:22:14] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/9797 [14:22:16] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9797 [14:23:50] PROBLEM - MySQL Replication Heartbeat on db1005 is CRITICAL: CRIT replication delay 182 seconds [14:23:59] PROBLEM - MySQL Slave Delay on db1005 is CRITICAL: CRIT replication delay 183 seconds [14:27:08] RECOVERY - NTP on search15 is OK: NTP OK: Offset -0.01884806156 secs [14:28:01] New patchset: Jgreen; "removing cron/snaprotate from db10 b/c it lacks space to snapshot" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9801 [14:28:22] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9801 [14:28:33] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9801 [14:28:35] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9801 [14:29:58] New patchset: Bhartshorne; "adding history file timestamps to the default bash configs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9802 [14:30:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9802 [14:30:30] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9802 [14:30:33] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9802 [14:31:37] New review: Faidon; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/9800 [14:35:59] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/9800 [14:36:02] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9800 [14:41:59] RECOVERY - MySQL Replication Heartbeat on db1005 is OK: OK replication delay 0 seconds [14:41:59] RECOVERY - MySQL Slave Delay on db1005 is OK: OK replication delay 0 seconds [14:43:34] New patchset: Pyoungmeister; "Revert "making all apt files be in place before apt-get update runs"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9808 [14:43:55] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9808 [14:45:11] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/9808 [14:45:13] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9808 [14:53:09] can anyone fix rights in Gerrit for operations/debs/testswarm repos? I have created it this morning but failed adding a parent project so only admins can see it right now [14:53:50] RECOVERY - Puppet freshness on search17 is OK: puppet ran at Sat Jun 2 14:53:33 UTC 2012 [14:54:28] paravoid Ryan_Lane ^^^^ [14:55:51] mutante ^^^^ [14:57:17] RECOVERY - Lucene disk space on search17 is OK: DISK OK [14:57:35] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [14:58:47] RECOVERY - Host bellin is UP: PING OK - Packet loss = 0%, RTA = 0.39 ms [15:00:23] hashar: you wouldnt care if i just deleted and recreated it? [15:02:59] RECOVERY - Puppet freshness on search16 is OK: puppet ran at Sat Jun 2 15:02:48 UTC 2012 [15:03:08] * Reedy deletes hashar [15:03:14] RobH: those 480's sound kinda puny, i want routers that take up a full rack [15:03:26] TX Matrix [15:03:35] routing, serious bizness [15:04:29] RECOVERY - Lucene disk space on search16 is OK: DISK OK [15:05:23] RECOVERY - Puppet freshness on search14 is OK: puppet ran at Sat Jun 2 15:05:12 UTC 2012 [15:07:02] RECOVERY - Lucene disk space on search14 is OK: DISK OK [15:07:47] RECOVERY - Puppet freshness on search19 is OK: puppet ran at Sat Jun 2 15:07:33 UTC 2012 [15:08:59] RECOVERY - Lucene disk space on search19 is OK: DISK OK [15:10:48] mutante: sorry been distracted. Yeah feel free to delete operations/debs/testswarm git repo. It is empty [15:11:05] RECOVERY - Lucene on search13 is OK: TCP OK - 0.007 second response time on port 8123 [15:11:32] RECOVERY - Lucene on search18 is OK: TCP OK - 0.002 second response time on port 8123 [15:11:50] RECOVERY - Lucene on search15 is OK: TCP OK - 0.001 second response time on port 8123 [15:13:47] RECOVERY - NTP on search17 is OK: NTP OK: Offset -0.0145881176 secs [15:13:58] hashar: i take that back. can't delete projects [15:14:57] haha [15:15:28] Reedy: can you change the "Rights Inherit From:" ? demon can :) [15:15:58] I'm not sure [15:16:15] there's an Edit button, but can't ..hrmmm [15:16:36] I usually end up having to ask chad to do stuff [15:16:51] Admin->Projects->Access->Rights Inherit From: [15:17:11] I keep forgetting that menu is htere [15:17:23] Reedy: yea, thats what we figured .. gotta ask demon [15:17:32] hmm, possibly [15:17:45] What do I need to change? [15:17:59] oh, nope [15:18:09] project: operations/debs/testswarm, Rights Inherit From: [15:18:20] operations/debs instead of All-Projects [15:18:24] I can edit the stuff beneath it [15:18:29] i can click that Edit button but not change, ack [15:19:21] yea, can leave commit message (optional), but nothing to select [15:23:32] RECOVERY - NTP on search16 is OK: NTP OK: Offset -0.02201211452 secs [15:23:41] RECOVERY - Lucene on search17 is OK: TCP OK - 0.001 second response time on port 8123 [15:23:59] RECOVERY - Lucene on search14 is OK: TCP OK - 0.001 second response time on port 8123 [15:24:17] RECOVERY - Lucene on search16 is OK: TCP OK - 0.001 second response time on port 8123 [15:24:28] I am screwed :-D [15:24:35] RECOVERY - Lucene on search19 is OK: TCP OK - 0.001 second response time on port 8123 [15:24:53] RECOVERY - NTP on search14 is OK: NTP OK: Offset -0.02463829517 secs [15:27:53] RECOVERY - NTP on search19 is OK: NTP OK: Offset -0.01446068287 secs [15:34:03] New patchset: Mark Bergsma; "Support passing IPv6 addresses to ipvsadm" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9818 [15:34:04] New patchset: Mark Bergsma; "Add IPv6 address family and resolving support" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9819 [15:34:04] New patchset: Mark Bergsma; "Use .find, .contains does not exist" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9820 [15:34:05] New patchset: Mark Bergsma; "Fix spacing for ipvsadm arguments" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9821 [15:34:06] New patchset: Mark Bergsma; "Fix DNS resolving, it was disabled and broken" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9822 [15:34:06] New patchset: Mark Bergsma; "Merge branch 'master' into ipv6" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9823 [15:34:07] New patchset: Mark Bergsma; "Update monitors proxyfetch and idleconnection for IPv6" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9824 [15:34:08] New patchset: Mark Bergsma; "Resolve both IPv4 and IPv6 addresses" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9825 [15:34:09] New patchset: Mark Bergsma; "Use a set for storing multiple IP addresses" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9826 [15:34:09] New patchset: Mark Bergsma; "Implement asynchronous DNS resolving for both IPv4 and IPv6 addresses" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9827 [15:34:10] New patchset: Mark Bergsma; "Add addressFamily parameter to Server._lookupFinished, to use for inet_ntop" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9828 [15:34:11] New patchset: Mark Bergsma; "Merge resolved hostname log lines for IPv4 and IPv6" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9829 [15:34:11] New patchset: Mark Bergsma; "Ensure server.ready when pooling a server, add a few asserts" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9830 [15:34:12] New patchset: Mark Bergsma; "Fix the DNS resolving callback chain" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9831 [15:34:13] New patchset: Mark Bergsma; "Move creation of monitoring instances to after DNS resolving" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9832 [15:34:13] New patchset: Mark Bergsma; "Fix DeferredList callback in _configReceived, add log messages" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9833 [15:34:14] New patchset: Mark Bergsma; "Use a random ip4_addresses for idleconnection & proxyfetch, until Twisted's connectTCP supports IPv6" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9834 [15:34:15] New patchset: Mark Bergsma; "Catch DNSLookupErrors in ProxyFetch" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9835 [15:34:15] New patchset: Mark Bergsma; "Remove superfluous spaces" [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9836 [15:37:04] !log We ran out of beer, see {{bug|37307}} [15:37:08] Logged the message, Master [15:37:21] yes [15:37:25] JUST WHEN I NEEDED IT [15:37:37] whoever wants ipv6 better get me beer now [15:56:54] New patchset: Bhartshorne; "adding orion's ssh key for access to the swift hardware cluster" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9843 [15:57:15] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9843 [15:58:29] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9818 [15:58:35] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9843 [15:58:38] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9843 [15:58:59] New patchset: Mark Bergsma; "Remove CVS $Id$ from __skeleton__.py" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/9844 [16:04:54] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9844 [16:04:56] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/9844 [16:05:10] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9818 [16:11:27] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9818 [16:11:28] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9818 [16:11:47] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9819 [16:11:49] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9819 [16:12:44] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9820 [16:12:46] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9820 [16:13:15] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9821 [16:13:16] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9821 [16:13:43] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9822 [16:13:45] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9822 [16:14:38] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9823 [16:14:39] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9823 [16:15:18] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9824 [16:15:19] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9824 [16:16:01] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9825 [16:16:02] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9825 [16:16:35] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9826 [16:16:37] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9826 [16:17:43] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9827 [16:17:45] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9827 [16:18:22] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9828 [16:18:24] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9828 [16:18:47] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9829 [16:18:49] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9829 [16:19:12] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9830 [16:19:13] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9830 [16:19:44] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9831 [16:19:46] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9831 [16:20:20] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9832 [16:20:22] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9832 [16:22:54] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9833 [16:23:00] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9833 [16:23:47] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9834 [16:23:48] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9834 [16:24:36] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9835 [16:24:37] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9835 [16:24:59] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (ipv6); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9836 [16:25:00] Change merged: Mark Bergsma; [operations/debs/pybal] (ipv6) - https://gerrit.wikimedia.org/r/9836 [16:28:55] New patchset: Bhartshorne; "removing sumanah from manganese's sudo list after confirming with her that she does not need access." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9848 [16:29:16] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9848 [16:29:16] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9848 [16:31:14] New patchset: Mark Bergsma; "Merge remote-tracking branch 'gerrit/ipv6'" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/9849 [16:31:39] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9849 [16:31:41] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/9849 [16:33:55] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/9798 [16:37:02] New patchset: Faidon; "Add IPv6 support & release 0.06" [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9852 [16:37:34] New review: Faidon; "(no comment)" [operations/debs/wikimedia-lvs-realserver] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9782 [16:37:35] Change merged: Faidon; [operations/debs/wikimedia-lvs-realserver] (master) - https://gerrit.wikimedia.org/r/9782 [16:37:51] New review: Faidon; "(no comment)" [operations/debs/wikimedia-lvs-realserver] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/9852 [16:38:04] New review: Faidon; "(no comment)" [operations/debs/wikimedia-lvs-realserver] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9852 [16:42:22] PROBLEM - Puppet freshness on bellin is CRITICAL: Puppet has not run in the last 10 hours [17:25:25] PROBLEM - Puppet freshness on es1003 is CRITICAL: Puppet has not run in the last 10 hours [17:25:25] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [17:25:25] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [18:20:01] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 207 seconds [18:20:28] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 224 seconds [18:29:46] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [18:29:46] RECOVERY - MySQL Slave Delay on db1047 is OK: OK replication delay 24 seconds [18:30:13] RECOVERY - MySQL Replication Heartbeat on db1047 is OK: OK replication delay 9 seconds [19:40:37] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 183 seconds [19:41:40] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 224 seconds [19:47:40] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 204 seconds [19:53:13] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 214 seconds [19:55:46] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 182 seconds [20:01:19] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 181 seconds [20:01:37] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 196 seconds [20:09:52] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 184 seconds [20:11:04] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 221 seconds [20:43:14] New patchset: Mark Bergsma; "Prefix monitor reports with LVS service name" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/9867 [20:43:15] New patchset: Mark Bergsma; "Prefix Coordinator messages with lvsservice names" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/9868 [20:43:52] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9867 [20:43:54] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/9867 [20:44:24] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9868 [20:44:25] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/9868 [20:58:29] RECOVERY - MySQL Replication Heartbeat on db1047 is OK: OK replication delay 27 seconds [20:58:56] RECOVERY - MySQL Slave Delay on db1047 is OK: OK replication delay 3 seconds [20:59:50] PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours [21:26:50] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [22:19:25] PROBLEM - Backend Squid HTTP on cp1001 is CRITICAL: Connection refused [22:22:07] RECOVERY - Backend Squid HTTP on cp1001 is OK: HTTP OK HTTP/1.0 200 OK - 27407 bytes in 0.116 seconds [23:52:47] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours