[00:00:30] mark: only by ilom [00:00:38] I can have a quick look if you want [00:00:41] RECOVERY - Host cp3002 is UP: PING OK - Packet loss = 0%, RTA = 109.09 ms [00:00:44] if I canj't fix it we can leave it off [00:03:19] but I've been amazed that btrfs has worked this long on 48 separate drives ;) [00:05:41] alright, i'll go to bed then [00:05:47] good luck and good weekend [00:05:49] see you on sunday [00:05:58] doh got distracted by non chat [00:06:17] see you sunday [00:06:26] later mark [00:16:07] New patchset: Asher; "provide pt-heartbeat with socket location" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1994 [00:16:23] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1994 [00:22:59] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1994 [00:22:59] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1994 [00:26:59] New patchset: Bhartshorne; "updating AUTH string for a newly created account for the eqiad swift cluster" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1995 [00:27:14] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1995 [00:27:23] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1995 [00:27:59] binasher: did you merge in my commit too? [00:28:16] it was only one line. [00:28:41] doesn't look like it [00:28:42] files/mysql/pt-heartbeat.init | 2 +- [00:28:42] 1 files changed, 1 insertions(+), 1 deletions(-) [00:28:46] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1995 [00:28:47] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1995 [00:28:53] no, it doesn't. [00:28:58] I approved but didn't merge. [00:29:03] ::sigh:: [00:29:14] and there it is. [00:42:10] New patchset: Asher; "provision db /a volumes with correct default mount options" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1996 [00:42:25] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1996 [00:42:28] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1996 [00:42:28] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1996 [01:50:43] New patchset: Bhartshorne; "loosening the regex to allow Swift to function correctly; we were catching too little" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1997 [01:50:59] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1997 [01:53:14] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1997 [01:53:15] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1997 [02:25:23] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1924s [02:29:13] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 2154s [02:29:53] PROBLEM - Frontend Squid HTTP on knsq9 is CRITICAL: Connection refused [02:39:13] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:45:23] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [04:18:20] RECOVERY - Disk space on es1004 is OK: DISK OK [04:22:41] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:38:00] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [06:17:02] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [09:48:31] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 430663 MB (3% inode=99%): [09:55:58] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 399390 MB (3% inode=99%): [10:44:18] RECOVERY - MySQL slave status on es1004 is OK: OK: [16:26:50] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [17:45:08] Change abandoned: Catrope; "We don't need this at all if we use the Gerrit hooks plugin for Jenkins, because it has per-repo act..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1794 [19:32:58] huh holy crap, I just found some ancient (2005 nov) image dumps [19:41:58] nice [19:42:44] not for en pedia, for all the rest of the projects though [19:42:50] (except the pedias) [19:51:15] archive has one from 2005 for pedia... 75gb [19:51:18] it's so cute :-D [20:15:50] apergos, seems worth to get pushed into IA still :) [20:18:59] well my plan is to dig through the old stuff [20:19:06] put up a couple form each year that we have [20:19:10] and then let folks do what they will [20:26:58] New patchset: pugmajere; "Clone git-setup from the puppet repository and update it for the software repository." [operations/software] (master) - https://gerrit.wikimedia.org/r/1998 [20:27:00] New patchset: pugmajere; "Simplify the aliases for the simplified branch "tradition" in the software repo." [operations/software] (master) - https://gerrit.wikimedia.org/r/1999 [20:27:01] New review: gerrit2; "Lint check passed." [operations/software] (master); V: 1 - https://gerrit.wikimedia.org/r/1999 [20:28:16] New review: Lcarr; "(no comment)" [operations/software] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1999 [20:29:10] New review: Lcarr; "(no comment)" [operations/software] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1998 [20:29:10] Change merged: Lcarr; [operations/software] (master) - https://gerrit.wikimedia.org/r/1999 [20:29:10] Change merged: Lcarr; [operations/software] (master) - https://gerrit.wikimedia.org/r/1998 [20:32:12] New patchset: Lcarr; "1st edition of fw creation tool" [operations/software] (master) - https://gerrit.wikimedia.org/r/2000 [20:32:12] New review: gerrit2; "Lint check passed." [operations/software] (master); V: 1 - https://gerrit.wikimedia.org/r/2000 [20:32:25] New review: Lcarr; "(no comment)" [operations/software] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2000 [20:32:25] Change merged: Lcarr; [operations/software] (master) - https://gerrit.wikimedia.org/r/2000 [21:37:25] PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 39 MB (0% inode=60%): /var/lib/ureadahead/debugfs 39 MB (0% inode=60%): [21:40:04] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [21:47:34] RECOVERY - Disk space on srv219 is OK: DISK OK [21:50:04] RECOVERY - Disk space on srv220 is OK: DISK OK