[00:02:00] <gerrit-wm>	 New patchset: Ryan Lane; "Password login inside of labs is bad. Let's ensure this is only set for production." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1764
[00:02:40] <gerrit-wm>	 New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1764
[00:02:40] <gerrit-wm>	 Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1764
[00:10:09] <nagios-wm>	 PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours
[01:28:02] <nagios-wm>	 PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL -  Seconds_Behind_Master : 881s
[01:31:52] <nagios-wm>	 PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL -  Seconds_Behind_Master : 991s
[02:05:02] <nagios-wm>	 PROBLEM - ps1-d2-sdtpa-infeed-load-tower-A-phase-Z on ps1-d2-sdtpa is CRITICAL: ps1-d2-sdtpa-infeed-load-tower-A-phase-Z CRITICAL - *2413*
[02:53:54] <nagios-wm>	 RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Sat Dec 31 02:53:44 UTC 2011
[04:42:33] <nagios-wm>	 PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No
[06:58:02] <nagios-wm>	 RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK -  Seconds_Behind_Master : 0s
[07:03:42] <nagios-wm>	 RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK -  Seconds_Behind_Master : 0s
[10:06:45] <nagios-wm>	 RECOVERY - MySQL slave status on es1004 is OK: OK:
[10:19:25] <nagios-wm>	 PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours
[19:41:11] <nagios-wm>	 PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours
[20:28:50] <nagios-wm>	 PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours
[22:20:34] <nagios-wm>	 RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Sat Dec 31 22:20:24 UTC 2011
[23:29:44] <nagios-wm>	 PROBLEM - Disk space on search6 is CRITICAL: DISK CRITICAL - free space: /a 2568 MB (1% inode=99%):
[23:30:04] <nagios-wm>	 PROBLEM - Lucene on search6 is CRITICAL: Connection timed out
[23:30:30] <OrenBochman>	 how do I start apache
[23:32:35] <nagios-wm>	 PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out