[00:20:59] <gerrit-wm>	 New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1613
[00:21:00] <gerrit-wm>	 Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1613
[01:43:08] <nagios-wm>	 PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours
[02:02:08] <nagios-wm>	 PROBLEM - Puppet freshness on bast1001 is CRITICAL: Puppet has not run in the last 10 hours
[02:04:08] <nagios-wm>	 PROBLEM - Puppet freshness on fenari is CRITICAL: Puppet has not run in the last 10 hours
[02:25:38] <nagios-wm>	 PROBLEM - Memcached on magnesium is CRITICAL: Connection refused
[04:31:18] <nagios-wm>	 PROBLEM - Puppet freshness on snapshot4 is CRITICAL: Puppet has not run in the last 10 hours
[07:37:47] <nagios-wm>	 PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No
[08:36:06] <nagios-wm>	 PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours
[08:38:48] <apergos>	 so I see that my cert req for dataset1 wound up on stafford
[08:39:05] <apergos>	 but it can't be signed there.  so.... how is it supposed to make it over to sockpuppet?
[12:28:08] <nagios-wm>	 RECOVERY - MySQL slave status on es1004 is OK: OK:
[13:01:30] <nagios-wm>	 PROBLEM - Puppet freshness on bast1001 is CRITICAL: Puppet has not run in the last 10 hours
[13:01:30] <nagios-wm>	 PROBLEM - Puppet freshness on fenari is CRITICAL: Puppet has not run in the last 10 hours
[13:01:30] <nagios-wm>	 PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours
[15:54:15] <nagios-wm>	 PROBLEM - Puppet freshness on snapshot4 is CRITICAL: Puppet has not run in the last 10 hours
[16:27:27] <gerrit-wm>	 New patchset: Catrope; "Fix for r1516: also remove files from manifest" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1617
[16:30:23] <gerrit-wm>	 New patchset: Catrope; "Remove stray slash" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1618
[16:37:45] <gerrit-wm>	 New review: Catrope; "You cherry-picked this without cherry-picking https://gerrit.wikimedia.org/r/1048 , which introduced..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1558
[16:42:31] <gerrit-wm>	 New patchset: Catrope; "Put mysql::client in its own class, and install it on bastion hosts too." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1619
[19:57:22] <nagios-wm>	 PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours
[22:21:54] <gerrit-wm>	 New patchset: Catrope; "Remove olddir directive from the l10nupdate logrotate config." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1620
[22:51:39] <RobH>	 !lof removed older binlogs on db9 again to kick it back to a bit more free space to last the weekend.
[22:51:52] <RobH>	 !log removed older binlogs on db9 again to kick it back to a bit more free space to last the weekend.
[22:52:00] <morebots>	 Logged the message, RobH
[22:52:06] <RobH>	 99 to 95% =P
[22:52:33] <RobH>	 !log Anytime db9 hits 98 or 99% someone needs to remove binlogs to bring it back down to 94 or 95%
[22:52:41] <morebots>	 Logged the message, RobH
[23:03:30] <Platonides>	 RobH, add a crontab?
[23:04:03] <RobH>	 i am not comfortable doing that since we are kililng logs that are very very recent.
[23:06:04] <Platonides>	 I don't mean removing it on each hour, but perhaps checking each hour if it reached >= 98% full