[00:54:50] Reported that wikitech won't let AndyRussG create a new instance... [00:55:28] Also analytics project seems not to have any instances? [00:58:59] (in my case, cause found: project was over the quota) [01:04:24] Search still borked [01:10:15] Yeah [01:46:28] PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:47:28] RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [02:13:18] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 2793 MB (2% inode=99%): [02:19:19] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3718 MB (3% inode=99%): [02:23:04] !log LocalisationUpdate completed (1.23wmf21) at 2014-04-12 02:23:02+00:00 [02:23:11] Logged the message, Master [02:31:58] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [02:31:58] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [02:31:58] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [02:31:58] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [02:42:09] !log LocalisationUpdate completed (1.23wmf22) at 2014-04-12 02:42:07+00:00 [02:42:16] Logged the message, Master [03:00:18] RECOVERY - Disk space on virt0 is OK: DISK OK [03:21:48] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 12 03:21:43 UTC 2014 (duration 21m 42s) [03:21:54] Logged the message, Master [03:50:28] PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:52:18] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:52:28] PROBLEM - MySQL Recent Restart on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:53:18] RECOVERY - MySQL Recent Restart on db1047 is OK: OK 1485639 seconds since restart [03:53:38] PROBLEM - MySQL Slave Running on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:54:18] RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [03:54:28] RECOVERY - MySQL Slave Running on db1047 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [03:58:08] RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [04:06:28] PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:08:18] RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [04:12:18] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:13:08] RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [04:16:28] PROBLEM - MySQL Recent Restart on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:17:18] RECOVERY - MySQL Recent Restart on db1047 is OK: OK 1487079 seconds since restart [04:23:28] PROBLEM - Disk space on lvs3001 is CRITICAL: DISK CRITICAL - free space: / 1572 MB (3% inode=97%): [04:25:28] RECOVERY - Disk space on lvs3001 is OK: DISK OK [04:46:18] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:48:28] PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:50:08] RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [04:59:18] RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [05:01:13] (03PS1) 10Ori.livneh: Replace $channel with $variant; make it Beta-only [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/125556 [05:03:06] (03CR) 10Ori.livneh: [C: 032] Replace $channel with $variant; make it Beta-only [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/125556 (owner: 10Ori.livneh) [05:03:13] (03Merged) 10jenkins-bot: Replace $channel with $variant; make it Beta-only [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/125556 (owner: 10Ori.livneh) [05:03:36] !log ori updated /a/common to {{Gerrit|I5f900190c}}: Replace $channel with $variant; make it Beta-only [05:03:42] Logged the message, Master [05:32:58] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [05:32:58] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [05:32:58] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [05:32:58] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [05:46:28] PROBLEM - MySQL Recent Restart on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:47:18] RECOVERY - MySQL Recent Restart on db1047 is OK: OK 1492479 seconds since restart [07:48:28] PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:51:19] RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [08:33:58] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [08:33:58] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [08:33:58] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [08:33:58] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [09:41:18] PROBLEM - MySQL InnoDB on db1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:41:18] PROBLEM - MySQL Idle Transactions on db1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:43:08] RECOVERY - MySQL InnoDB on db1016 is OK: OK longest blocking idle transaction sleeps for 0 seconds [09:43:08] RECOVERY - MySQL Idle Transactions on db1016 is OK: OK longest blocking idle transaction sleeps for 0 seconds [10:10:18] PROBLEM - MySQL InnoDB on db1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:10:18] PROBLEM - MySQL Idle Transactions on db1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:12:08] RECOVERY - MySQL InnoDB on db1016 is OK: OK longest blocking idle transaction sleeps for 0 seconds [10:12:08] RECOVERY - MySQL Idle Transactions on db1016 is OK: OK longest blocking idle transaction sleeps for 0 seconds [11:34:58] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [11:34:58] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [11:34:58] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [11:34:58] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [14:35:58] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [14:35:58] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [14:35:58] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [14:35:58] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [16:05:08] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [17:36:58] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [17:36:58] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [17:36:58] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [17:36:58] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [18:01:08] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [19:01:08] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.00111111111111 [20:01:08] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.000277777777778 [20:01:48] 0.00028? sounds terrible! [20:37:58] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [20:37:58] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [20:37:58] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [20:37:58] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [21:01:08] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [21:04:29] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 12 Apr 2014 08:59:45 PM UTC [21:06:29] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 12 Apr 2014 08:59:45 PM UTC [21:08:29] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 12 Apr 2014 08:59:45 PM UTC [21:10:29] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 12 Apr 2014 08:59:45 PM UTC [21:12:29] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 12 Apr 2014 08:59:45 PM UTC [21:14:29] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 12 Apr 2014 08:59:45 PM UTC [21:16:29] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 12 Apr 2014 08:59:45 PM UTC [21:18:29] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 12 Apr 2014 08:59:45 PM UTC [21:20:29] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 12 Apr 2014 08:59:45 PM UTC [21:22:29] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 12 Apr 2014 08:59:45 PM UTC [21:24:29] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 12 Apr 2014 08:59:45 PM UTC [21:26:29] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 12 Apr 2014 08:59:45 PM UTC [21:28:29] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 12 Apr 2014 08:59:45 PM UTC [21:30:09] RECOVERY - Puppet freshness on mw1097 is OK: puppet ran at Sat Apr 12 21:30:06 UTC 2014 [21:32:29] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 12 Apr 2014 09:30:06 PM UTC [22:00:37] RECOVERY - Puppet freshness on mw1097 is OK: puppet ran at Sat Apr 12 22:00:31 UTC 2014 [22:27:57] PROBLEM - MySQL Idle Transactions on db1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:28:07] PROBLEM - MySQL InnoDB on db1016 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:28:47] RECOVERY - MySQL Idle Transactions on db1016 is OK: OK longest blocking idle transaction sleeps for 0 seconds [22:28:57] RECOVERY - MySQL InnoDB on db1016 is OK: OK longest blocking idle transaction sleeps for 0 seconds [23:38:17] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [23:38:17] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [23:38:17] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [23:38:17] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC