[00:19:08] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [00:35:07] gn8 folks [01:38:09] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [01:58:39] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [02:04:23] !log LocalisationUpdate completed (1.18) at Sun Dec 25 02:04:23 UTC 2011 [02:04:33] Logged the message, Master [02:16:49] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1277s [02:22:09] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 709s [02:26:29] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:32:09] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:40:49] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Sun Dec 25 02:40:20 UTC 2011 [04:01:10] RECOVERY - ps1-d2-pmtpa-infeed-load-tower-A-phase-X on ps1-d2-pmtpa is OK: ps1-d2-pmtpa-infeed-load-tower-A-phase-X OK - 1200 [04:35:09] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [06:07:17] zzz [07:08:20] PROBLEM - RAID on search10 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:12:30] PROBLEM - Squid on brewster is CRITICAL: Connection refused [07:19:20] RECOVERY - RAID on search10 is OK: OK: 1 logical device(s) checked [09:51:30] RECOVERY - MySQL slave status on es1004 is OK: OK: [11:30:41] Damianz: around? [11:53:23] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [12:08:02] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [13:38:29] a random fact: "merry christmas" is still widely used in enwp :) https://docs.google.com/spreadsheet/ccc?key=0Atpvz6IuOetOdHM2TzFSX01hZXdzR1Z3a25QeFB1bFE&hl=en_US#gid=0 [13:47:01] orly [13:58:39] it's scary to see how many warnings are sent to people on enwp [13:58:46] https://docs.google.com/spreadsheet/ccc?key=0Atpvz6IuOetOdGxIbnRITFllNjZaLTJjQS1iVWNDdWc&hl=en_US#gid=0 [14:14:16] pagecounts are zero: http://dumps.wikimedia.org/other/pagecounts-raw/2011/2011-12/ [14:55:14] PROBLEM - mobile traffic loggers on cp1041 is CRITICAL: PROCS CRITICAL: 7 processes with args varnishncsa [14:55:14] PROBLEM - mobile traffic loggers on cp1042 is CRITICAL: PROCS CRITICAL: 7 processes with args varnishncsa [15:14:44] RECOVERY - mobile traffic loggers on cp1041 is OK: PROCS OK: 3 processes with args varnishncsa [15:14:44] RECOVERY - mobile traffic loggers on cp1042 is OK: PROCS OK: 1 process with args varnishncsa [16:45:18] PROBLEM - check_minfraud3 on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:18] PROBLEM - check_minfraud3 on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:18] PROBLEM - check_minfraud3 on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:18] PROBLEM - check_minfraud3 on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:08] PROBLEM - Puppet freshness on brewster is CRITICAL: Puppet has not run in the last 10 hours [16:50:18] RECOVERY - check_minfraud3 on payments4 is OK: HTTP OK: HTTP/1.1 200 OK - 8571 bytes in 0.380 second response time [16:50:18] RECOVERY - check_minfraud3 on payments1 is OK: HTTP OK: HTTP/1.1 200 OK - 8571 bytes in 0.365 second response time [16:50:18] RECOVERY - check_minfraud3 on payments3 is OK: HTTP OK: HTTP/1.1 200 OK - 8571 bytes in 0.311 second response time [16:50:18] RECOVERY - check_minfraud3 on payments2 is OK: HTTP OK: HTTP/1.1 200 OK - 8571 bytes in 0.314 second response time [19:35:56] !log reedy synchronized wmf-config/flaggedrevs.php 'bug 29742 Some tweaks to FR config for fiwiki' [19:36:08] Logged the message, Master [22:02:28] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [22:17:28] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [23:28:29] gn8 folks