[00:03:21] New patchset: Lcarr; "giving ori sudo access (approved by ct RT3152)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/14600 [00:03:55] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/14600 [00:04:04] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/14600 [00:04:33] ori-l: try sudoing again in 5 minutes (i'll bb in about 5 ;) ), puppet running now [00:12:18] ori-l: how's it look [00:13:34] LeslieCarr: i'm not sure which password my shell account is tied to [00:13:43] should be passwordles [00:13:43] s [00:13:56] LeslieCarr: nope [00:14:03] [sudo] password for olivneh: [00:14:25] does putting in nothing work? [00:14:44] oh doh [00:14:48] would help if i merged the change [00:15:37] :) [00:16:46] LeslieCarr: thanks again for doing this all on a friday evening [00:19:56] now, does this work ? [00:20:33] LeslieCarr: sorry, irc connection died [00:20:34] trying now [00:21:11] LeslieCarr: works! # > $ [00:21:21] thanks again! [00:21:35] yay [00:23:31] time for me to run :) [00:23:43] have a good night/weekend [01:04:16] PROBLEM - Puppet freshness on ms3 is CRITICAL: Puppet has not run in the last 10 hours [01:14:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [01:24:14] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [01:37:17] PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours [01:41:29] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 267 seconds [01:42:23] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 263 seconds [01:48:23] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 623s [01:50:20] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 0 seconds [01:52:53] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 51s [01:56:38] PROBLEM - swift-object-auditor on ms-be7 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [01:57:14] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 19 seconds [02:02:56] PROBLEM - swift-object-auditor on ms-be8 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [02:07:26] RECOVERY - swift-object-auditor on ms-be8 is OK: PROCS OK: 2 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [02:17:38] RECOVERY - swift-object-auditor on ms-be7 is OK: PROCS OK: 2 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [02:25:44] PROBLEM - swift-object-auditor on ms-be6 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [02:32:32] RECOVERY - swift-object-auditor on ms-be6 is OK: PROCS OK: 2 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [02:48:53] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [03:32:02] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [05:21:58] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [05:24:58] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [06:22:35] PROBLEM - Puppet freshness on db1029 is CRITICAL: Puppet has not run in the last 10 hours [08:38:29] PROBLEM - Puppet freshness on mw1102 is CRITICAL: Puppet has not run in the last 10 hours [08:52:35] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [09:51:19] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [10:14:00] PROBLEM - Host mw1126 is DOWN: PING CRITICAL - Packet loss = 100% [10:38:36] PROBLEM - swift-object-auditor on ms-be8 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [10:49:42] PROBLEM - swift-object-auditor on ms-be7 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [10:58:06] RECOVERY - swift-object-auditor on ms-be8 is OK: PROCS OK: 2 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [11:04:49] PROBLEM - swift-object-auditor on ms-be6 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [11:04:58] PROBLEM - Puppet freshness on ms3 is CRITICAL: Puppet has not run in the last 10 hours [11:08:34] RECOVERY - swift-object-auditor on ms-be7 is OK: PROCS OK: 2 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [11:13:40] RECOVERY - swift-object-auditor on ms-be6 is OK: PROCS OK: 2 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [11:15:01] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [11:24:55] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [11:37:58] PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours [12:50:11] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [13:32:57] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [15:23:02] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [15:26:45] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [16:23:54] PROBLEM - Puppet freshness on db1029 is CRITICAL: Puppet has not run in the last 10 hours [18:39:09] PROBLEM - Puppet freshness on mw1102 is CRITICAL: Puppet has not run in the last 10 hours [18:53:59] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [19:29:22] hi guys! [19:29:42] im here to fix your puppet... [19:29:59] it really needs to be refactored into modules :) [19:30:53] PROBLEM - swift-object-auditor on ms-be8 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [19:36:53] RECOVERY - swift-object-auditor on ms-be8 is OK: PROCS OK: 2 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [19:37:00] Hi jesusaurus. Are you familiar with Git and Gerrit? [19:37:19] with git, not with gerrit [19:38:04] I guess you've seen [[mw:Git/Workflow]] [19:38:13] yeah [19:38:40] and roan linked me to the labsconsole page to git, and the developer access page too [19:41:05] PROBLEM - swift-object-auditor on ms-be7 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [19:45:26] RECOVERY - swift-object-auditor on ms-be7 is OK: PROCS OK: 2 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [19:46:02] PROBLEM - swift-object-auditor on ms-be6 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [19:52:02] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [19:52:11] RECOVERY - swift-object-auditor on ms-be6 is OK: PROCS OK: 2 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [21:05:55] PROBLEM - Puppet freshness on ms3 is CRITICAL: Puppet has not run in the last 10 hours [21:16:34] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [21:25:43] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [21:38:37] PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours [22:50:46] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [23:33:49] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours