[00:00:40] <nagios-wm>	 PROBLEM - SSH on srv276 is CRITICAL: Server answer:
[00:00:50] <nagios-wm>	 RECOVERY - DPKG on srv262 is OK: All packages OK
[00:30:20] <nagios-wm>	 RECOVERY - SSH on srv276 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0)
[00:30:20] <nagios-wm>	 RECOVERY - DPKG on srv276 is OK: All packages OK
[00:32:50] <nagios-wm>	 RECOVERY - Disk space on srv276 is OK: DISK OK
[00:38:20] <nagios-wm>	 RECOVERY - Apache HTTP on srv276 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.041 second response time
[00:38:30] <nagios-wm>	 RECOVERY - RAID on srv276 is OK: OK: no RAID installed
[00:46:59] <tfinc>	 !log killing long running show_bug.cgi procs on kaulen
[00:46:59] <morebots>	 Logged the message, Master
[00:47:01] <tfinc>	 :D
[00:47:16] <Reedy>	 I was just about to to that, cheers
[00:47:32] <Reedy>	 Saves me having to write an email/try and find some ops about
[01:29:49] <nagios-wm>	 PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours
[02:58:45] <nagios-wm>	 RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Sun Jan 8 02:58:38 UTC 2012
[03:13:13] <nagios-wm>	 PROBLEM - Puppet freshness on cp1043 is CRITICAL: Puppet has not run in the last 10 hours
[03:21:23] <nagios-wm>	 PROBLEM - Puppet freshness on cp1044 is CRITICAL: Puppet has not run in the last 10 hours
[03:34:23] <nagios-wm>	 RECOVERY - Puppet freshness on db1003 is OK: puppet ran at Sun Jan 8 03:34:11 UTC 2012
[03:35:23] <nagios-wm>	 PROBLEM - Puppet freshness on db22 is CRITICAL: Puppet has not run in the last 10 hours
[03:49:13] <nagios-wm>	 PROBLEM - MySQL replication status on db1025 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL -  Seconds_Behind_Master : 1401s
[03:57:34] <nagios-wm>	 RECOVERY - MySQL replication status on db1025 is OK: CHECK MySQL REPLICATION - lag - OK -  Seconds_Behind_Master : 0s
[04:21:24] <nagios-wm>	 RECOVERY - Disk space on es1004 is OK: DISK OK
[04:22:14] <nagios-wm>	 RECOVERY - MySQL disk space on es1004 is OK: DISK OK
[04:37:04] <nagios-wm>	 PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No
[06:57:08] <nagios-wm>	 PROBLEM - Squid on brewster is CRITICAL: Connection refused
[10:06:07] <nagios-wm>	 PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 442897 MB (3% inode=99%):
[10:09:57] <nagios-wm>	 PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 414797 MB (3% inode=99%):
[10:24:07] <nagios-wm>	 RECOVERY - MySQL slave status on es1004 is OK: OK:
[13:23:06] <nagios-wm>	 PROBLEM - Puppet freshness on cp1043 is CRITICAL: Puppet has not run in the last 10 hours
[13:31:06] <nagios-wm>	 PROBLEM - Puppet freshness on cp1044 is CRITICAL: Puppet has not run in the last 10 hours
[13:44:37] <nagios-wm>	 PROBLEM - Puppet freshness on db22 is CRITICAL: Puppet has not run in the last 10 hours
[14:08:37] <nagios-wm>	 PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours
[14:15:52] <petan>	 could someone fix brewster?
[14:15:58] <petan>	 apergos: ^
[14:19:23] <apergos>	 it's news to me that there is any squid process on brewster
[14:26:52] <apergos>	 I don't see off hand why nagios thinks it should be doing something else
[14:39:17] <apergos>	 ah well I see the root of the problem
[14:39:20] <apergos>	 but...
[14:51:11] <apergos>	 !log cleared out some very large squid logs on brewster, (basically all of them) plus lighty logs, disk was full. restarted squid manually
[14:51:13] <morebots>	 Logged the message, Master
[14:52:34] <nagios-wm>	 RECOVERY - Squid on brewster is OK: TCP OK - 0.002 second response time on port 8080
[14:57:15] <apergos>	 !log removed old puppet lockfile on brewster, ran by hand
[14:57:16] <morebots>	 Logged the message, Master
[15:11:59] <petan>	 apergos: without it puppet is broken
[15:12:12] <petan>	 I couldn't do puppetd -tv
[15:23:48] <apergos>	 on what host?
[16:43:49] <nagios-wm>	 RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Sun Jan 8 16:43:48 UTC 2012
[23:00:14] <Prodego>	 what's going on with this http://en.wikipedia.org/wiki/Special:Contributions/208.80.154.52
[23:03:40] <Prodego>	 what do you guys use that IP for?
[23:06:11] <Prodego>	 anyone around?
[23:10:48] <Prodego>	 Ryan_Lane: you around?
[23:11:06] <Reedy>	 Prodego, it resolves as being a squid box
[23:11:12] <Reedy>	 cp1042.wikimedia.org
[23:11:48] <Prodego>	 well obviously we have some sort of problem if people are editing from it
[23:12:02] <Prodego>	 this could be an XFF issue, or a misconfiguration issue
[23:12:11] <Prodego>	 someone needs to investigate this
[23:12:13] <Reedy>	 I think it's an xff issue
[23:12:14] <Reedy>	 hangon
[23:12:23] <Reedy>	 yeah
[23:12:24] <Reedy>	 it's not listed
[23:13:40] <Prodego>	 ok, so are all xff headers being respected or something?
[23:13:58] <Prodego>	 perhaps anyone who is on a host where xff headers are respected can just add their own xff headers as well?
[23:14:30] <Reedy>	 Are there any other hosts around that IP also showing up as editing?
[23:15:28] <Prodego>	 that IP is the only one I saw, but I was just shown it and haven't looked
[23:15:30] <Prodego>	 let me ask
[23:16:35] <Reedy>	 looks like all it's ip neighbours are allocated too
[23:24:23] <Reedy>	 Prodego, that's all the eqiad deployed cp10** listed in XFF now
[23:25:23] <Prodego>	 so that server should be passing IPs correctly now?
[23:25:42] <Reedy>	 cp1001-cp1042 will be, yes
[23:26:24] <Prodego>	 so I'l let you know if we see anything more
[23:27:19] <Prodego>	 thanks Reedy
[23:32:18] <nagios-wm>	 PROBLEM - Puppet freshness on cp1043 is CRITICAL: Puppet has not run in the last 10 hours
[23:40:18] <nagios-wm>	 PROBLEM - Puppet freshness on cp1044 is CRITICAL: Puppet has not run in the last 10 hours
[23:54:18] <nagios-wm>	 PROBLEM - Puppet freshness on db22 is CRITICAL: Puppet has not run in the last 10 hours