[02:21:41] <nagios-wm>	 PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL -  Seconds_Behind_Master : 1728s
[02:23:21] <nagios-wm>	 PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL -  Seconds_Behind_Master : 1828s
[02:31:31] <nagios-wm>	 RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK -  Seconds_Behind_Master : 0s
[02:33:11] <nagios-wm>	 RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK -  Seconds_Behind_Master : 0s
[04:16:55] <nagios-wm>	 RECOVERY - MySQL disk space on es1004 is OK: DISK OK
[04:24:25] <nagios-wm>	 RECOVERY - Disk space on es1004 is OK: DISK OK
[04:37:03] <nagios-wm>	 PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No
[06:31:14] <nagios-wm>	 PROBLEM - Puppet freshness on sodium is CRITICAL: Puppet has not run in the last 10 hours
[07:23:29] <nagios-wm>	 PROBLEM - Squid on brewster is CRITICAL: Connection refused
[09:55:54] <nagios-wm>	 PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 441882 MB (3% inode=99%):
[10:01:52] <nagios-wm>	 PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 415270 MB (3% inode=99%):
[10:27:45] <nagios-wm>	 RECOVERY - MySQL slave status on es1004 is OK: OK:
[11:03:05] <nagios-wm>	 PROBLEM - RAID on db13 is CRITICAL: CRITICAL: Defunct disk drive count: 1
[13:20:45] <petan>	 apergos: hi
[13:21:10] <petan>	 what is in wikimedia-ssl-backend.conf in apache on prod?
[13:21:27] <petan>	 ./usr/local/apache/conf/wikimedia-ssl-backend.conf
[13:21:41] <petan>	 it isn't on noc, does it contain private data?
[13:47:54] <petan>	 someone from ops is around?
[16:31:27] <jeremyb>	 petan: was ref'd in the initial commit of the puppet repo (so it was there from before the move to git)
[16:31:30] <jeremyb>	 https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=blob;f=files/apache/apache2.conf.appserver;h=c85ddb6dd33046d99cedfc5b986029ed3b243bfd;hb=de059228933681b6b0f97a818a561cde22901e1e#l333
[16:32:24] <petan>	 and is it there? :P
[16:32:28] <petan>	 I wanted to see content heh
[16:32:45] <petan>	 Reedy: can you check what is in /usr/local/apache/conf/wikimedia-ssl-backend.conf on prod?
[16:32:59] <petan>	 or how is ssl configured on prod?
[16:33:13] <petan>	 that's only thing which isn't public
[16:33:19] <petan>	 right, not only
[16:33:29] <petan>	 another :>
[16:34:57] <Reedy>	 I don't see anything in it that looks secret
[16:36:21] <jeremyb>	 treasure map?
[16:37:05] <petan>	 yay
[16:37:40] <nagios-wm>	 PROBLEM - Puppet freshness on brewster is CRITICAL: Puppet has not run in the last 10 hours
[16:37:40] <petan>	 Reedy: can you check how is ssl configured at least?
[16:37:55] <Reedy>	 What are you wanting from the config
[16:37:56] <Reedy>	 ?
[16:39:23] <petan>	 hm, when I try to use ssl in standart way using guide from apache it start warnings related to set up from noc
[16:39:40] <petan>	 because virtual host has * instead of *:port
[16:40:21] <Reedy>	 you realise it's ssl terminated, then goes to apache?
[16:40:41] <petan>	 ah, so apache isn't configured for ssl?
[16:40:50] <Reedy>	 nope
[16:40:52] <petan>	 right I forgot there is squid
[16:40:54] <Reedy>	 nginx is, i think
[16:41:03] <petan>	 true
[16:41:24] <jeremyb>	 yes, it's nginx
[16:44:04] <nagios-wm>	 PROBLEM - Puppet freshness on sodium is CRITICAL: Puppet has not run in the last 10 hours