[00:06:35] (03PS2) 10Dzahn: revert the SSL chain file change on gerrit [operations/puppet] - 10https://gerrit.wikimedia.org/r/90676 [00:07:53] (03CR) 10Dzahn: [C: 032] "partly reverts Change-Id: I12c58752b5ae3b07941530d3cc3fb19c17a9076c" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90676 (owner: 10Dzahn) [00:15:46] (03CR) 10Dzahn: "was fine for planet and metrics, just not for gerrit. therefore partly reverted for now. reason in https://gerrit.wikimedia.org/r/#/c/9067" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84901 (owner: 10Dzahn) [00:24:27] (03PS1) 10Ryan Lane: Add Greg Grossmeier as a mortal [operations/puppet] - 10https://gerrit.wikimedia.org/r/90678 [00:24:49] (03CR) 10Chad: [C: 031] "I welcome Greg as our new mortal overlord." [operations/puppet] - 10https://gerrit.wikimedia.org/r/90678 (owner: 10Ryan Lane) [00:25:24] (03CR) 10Ryan Lane: [C: 032] Add Greg Grossmeier as a mortal [operations/puppet] - 10https://gerrit.wikimedia.org/r/90678 (owner: 10Ryan Lane) [00:25:45] PROBLEM - Disk space on db1028 is CRITICAL: DISK CRITICAL - free space: / 262 MB (3% inode=79%): [00:26:05] PROBLEM - MySQL disk space on db1028 is CRITICAL: DISK CRITICAL - free space: / 262 MB (3% inode=79%): [00:26:08] greg-g: merged all the way through [00:26:10] it'll take a bit [00:26:22] well, let me force it on tin and bast1001 [00:30:36] weeeee [00:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 00:32:45 UTC 2013 [00:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [00:34:05] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 00:34:00 UTC 2013 [00:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [00:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 00:35:40 UTC 2013 [00:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 00:41:47 UTC 2013 [00:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:55] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 00:42:47 UTC 2013 [00:42:55] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 00:43:52 UTC 2013 [00:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [00:45:15] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 00:45:07 UTC 2013 [00:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [00:51:45] RECOVERY - MySQL disk space on db1024 is OK: DISK OK [00:51:55] RECOVERY - Disk space on db1024 is OK: DISK OK [00:52:15] RECOVERY - Disk space on db1040 is OK: DISK OK [00:52:45] RECOVERY - MySQL disk space on db1002 is OK: DISK OK [00:52:45] RECOVERY - MySQL disk space on db1040 is OK: DISK OK [00:52:45] RECOVERY - Disk space on db1002 is OK: DISK OK [00:54:05] RECOVERY - Disk space on db1009 is OK: DISK OK [00:54:05] RECOVERY - MySQL disk space on db1028 is OK: DISK OK [00:54:25] RECOVERY - MySQL disk space on db1009 is OK: DISK OK [00:54:45] RECOVERY - Disk space on db1043 is OK: DISK OK [00:54:45] RECOVERY - Disk space on db1028 is OK: DISK OK [00:55:05] RECOVERY - MySQL disk space on db1043 is OK: DISK OK [01:13:26] (03PS1) 10Springle: send tcp errors dumps to /dev/null instead of filling up /tmp. pt-query-digest v2.x behaviour change. [operations/puppet] - 10https://gerrit.wikimedia.org/r/90683 [01:14:26] (03CR) 10Springle: [C: 032] send tcp errors dumps to /dev/null instead of filling up /tmp. pt-query-digest v2.x behaviour change. [operations/puppet] - 10https://gerrit.wikimedia.org/r/90683 (owner: 10Springle) [01:20:59] g'morning springle :) [01:21:09] just waving hello before I head home [01:21:34] greg-g: hi, bye :) [01:26:10] (03PS1) 10Diederik: Add dependencies for Wikimetrics to Jenkins to enable CI. [operations/puppet] - 10https://gerrit.wikimedia.org/r/90684 [01:28:05] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [01:29:15] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 27.30 ms [01:32:45] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 01:32:42 UTC 2013 [01:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [01:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 01:33:48 UTC 2013 [01:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [01:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 01:35:44 UTC 2013 [01:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [01:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 01:41:46 UTC 2013 [01:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [01:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 01:42:41 UTC 2013 [01:42:55] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [01:43:45] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 01:43:41 UTC 2013 [01:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [01:43:55] PROBLEM - NTP on mw31 is CRITICAL: NTP CRITICAL: Offset unknown [01:44:55] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 01:44:51 UTC 2013 [01:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [01:47:55] RECOVERY - NTP on mw31 is OK: NTP OK: Offset 0.001633048058 secs [01:52:15] PROBLEM - Disk space on cp1048 is CRITICAL: DISK CRITICAL - free space: /srv/sdb3 12237 MB (3% inode=99%): [02:08:11] !log LocalisationUpdate completed (1.22wmf21) at Sat Oct 19 02:08:11 UTC 2013 [02:08:30] Logged the message, Master [02:15:49] !log LocalisationUpdate completed (1.22wmf22) at Sat Oct 19 02:15:49 UTC 2013 [02:16:02] Logged the message, Master [02:30:42] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Oct 19 02:30:42 UTC 2013 [02:30:55] Logged the message, Master [02:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 02:32:46 UTC 2013 [02:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [02:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 02:33:46 UTC 2013 [02:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [02:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 02:35:42 UTC 2013 [02:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [02:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 02:41:44 UTC 2013 [02:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [02:42:55] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 02:42:44 UTC 2013 [02:42:55] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [02:44:05] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 02:44:00 UTC 2013 [02:44:55] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 02:44:45 UTC 2013 [02:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [02:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [03:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 03:32:47 UTC 2013 [03:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [03:33:45] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 03:33:42 UTC 2013 [03:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [03:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 03:35:43 UTC 2013 [03:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [03:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 03:41:44 UTC 2013 [03:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [03:42:55] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 03:42:45 UTC 2013 [03:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [03:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 03:43:45 UTC 2013 [03:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [03:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 03:44:40 UTC 2013 [03:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [03:55:19] (03PS3) 10Ori.livneh: Remove unused (and broken) labs config from squid manifest [operations/puppet] - 10https://gerrit.wikimedia.org/r/88698 [03:57:04] (03CR) 10Ori.livneh: [C: 032] Remove unused (and broken) labs config from squid manifest [operations/puppet] - 10https://gerrit.wikimedia.org/r/88698 (owner: 10Ori.livneh) [04:07:14] (03PS2) 10Ori.livneh: graphite: upstream doc in storage-aggregation.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/88469 (owner: 10Hashar) [04:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 04:32:45 UTC 2013 [04:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [04:34:05] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 04:34:00 UTC 2013 [04:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [04:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 04:35:35 UTC 2013 [04:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 04:41:42 UTC 2013 [04:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 04:42:42 UTC 2013 [04:42:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [04:43:45] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 04:43:42 UTC 2013 [04:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [04:44:46] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 04:44:43 UTC 2013 [04:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [05:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 05:32:46 UTC 2013 [05:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [05:34:05] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 05:33:56 UTC 2013 [05:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [05:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 05:35:42 UTC 2013 [05:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [05:39:05] PROBLEM - search indices - check lucene status page on search20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 60051 bytes in 0.111 second response time [05:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 05:41:43 UTC 2013 [05:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [05:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 05:42:44 UTC 2013 [05:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [05:43:46] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 05:43:44 UTC 2013 [05:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [05:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 05:44:44 UTC 2013 [05:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [06:23:21] (03CR) 10Ori.livneh: [C: 032] graphite: upstream doc in storage-aggregation.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/88469 (owner: 10Hashar) [06:25:05] PROBLEM - Disk space on cp1064 is CRITICAL: DISK CRITICAL - free space: /srv/sda3 12377 MB (3% inode=99%): /srv/sdb3 18486 MB (5% inode=99%): [06:32:45] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 06:32:43 UTC 2013 [06:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:05] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 06:34:03 UTC 2013 [06:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [06:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 06:35:43 UTC 2013 [06:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [06:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 06:41:45 UTC 2013 [06:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [06:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 06:42:41 UTC 2013 [06:42:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [06:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 06:43:51 UTC 2013 [06:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [06:45:05] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 06:45:01 UTC 2013 [06:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [07:13:16] !log reports of geoiplookup timing out in AU at enwiki VPT: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#geoiplookup.wikimedia.org [07:13:32] Logged the message, Master [07:27:45] PROBLEM - Puppet freshness on db1043 is CRITICAL: No successful Puppet run in the last 10 hours [07:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 07:32:45 UTC 2013 [07:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [07:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 07:33:46 UTC 2013 [07:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [07:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 07:35:41 UTC 2013 [07:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [07:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 07:41:43 UTC 2013 [07:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [07:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 07:42:43 UTC 2013 [07:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [07:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 07:43:48 UTC 2013 [07:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 07:44:43 UTC 2013 [07:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [07:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [08:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 08:32:48 UTC 2013 [08:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [08:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 08:33:53 UTC 2013 [08:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [08:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 08:35:43 UTC 2013 [08:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [08:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 08:41:45 UTC 2013 [08:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [08:42:55] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 08:42:45 UTC 2013 [08:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [08:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 08:43:45 UTC 2013 [08:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 08:44:40 UTC 2013 [08:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [09:32:45] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 09:32:43 UTC 2013 [09:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [09:33:45] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 09:33:44 UTC 2013 [09:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [09:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 09:35:39 UTC 2013 [09:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [09:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 09:41:46 UTC 2013 [09:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [09:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 09:42:41 UTC 2013 [09:42:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [09:43:45] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 09:43:41 UTC 2013 [09:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [09:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 09:44:42 UTC 2013 [09:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [09:45:50] general question for ops: is it technically feasable to have a TOR/I2P/... gateway to wiki? [09:46:08] or has it been done already? [10:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 10:32:46 UTC 2013 [10:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [10:33:45] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 10:33:41 UTC 2013 [10:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [10:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 10:35:37 UTC 2013 [10:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:05] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 10:41:59 UTC 2013 [10:42:55] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 10:42:45 UTC 2013 [10:43:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 10:43:45 UTC 2013 [10:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [10:44:55] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 10:44:45 UTC 2013 [10:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:59] YuviPanda: you always seem to know everything :) ^ [11:30:04] yurik_: see https://en.wikipedia.org/wiki/Wikipedia:Advice_to_users_using_Tor [11:30:26] yurik_: so even if we did have a gateway/hidden node, it needs some way of going around that [11:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 11:32:46 UTC 2013 [11:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [11:34:05] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 11:33:56 UTC 2013 [11:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [11:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 11:35:36 UTC 2013 [11:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [11:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 11:41:42 UTC 2013 [11:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [11:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 11:42:43 UTC 2013 [11:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [11:43:45] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 11:43:43 UTC 2013 [11:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [11:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 11:44:44 UTC 2013 [11:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [12:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 12:32:46 UTC 2013 [12:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [12:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 12:33:46 UTC 2013 [12:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [12:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 12:35:42 UTC 2013 [12:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [12:41:46] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 12:41:44 UTC 2013 [12:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [12:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 12:42:44 UTC 2013 [12:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [12:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 12:43:49 UTC 2013 [12:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [12:44:55] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 12:44:49 UTC 2013 [12:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [12:48:35] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [12:49:15] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 26.58 ms [13:14:27] !log Jenkins: updating Zuul config {{gerrit|90695}} [13:14:42] Logged the message, Master [13:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 13:32:47 UTC 2013 [13:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [13:33:45] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 13:33:42 UTC 2013 [13:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [13:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 13:35:37 UTC 2013 [13:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [13:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 13:41:43 UTC 2013 [13:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [13:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 13:42:44 UTC 2013 [13:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [13:43:45] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 13:43:44 UTC 2013 [13:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [13:44:46] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 13:44:44 UTC 2013 [13:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [14:14:05] RECOVERY - Puppet freshness on db1043 is OK: puppet ran at Sat Oct 19 14:13:57 UTC 2013 [14:24:49] (03Abandoned) 10Hashar: Jenkins #2 (DO NOT SUBMIT) [operations/debs/pybal] - 10https://gerrit.wikimedia.org/r/90549 (owner: 10Hashar) [14:24:53] (03Abandoned) 10Hashar: Jenkins #3 do not submit [operations/debs/pybal] - 10https://gerrit.wikimedia.org/r/90550 (owner: 10Hashar) [14:24:58] (03Abandoned) 10Hashar: Jenkins #1 (please ignore) [operations/debs/pybal] - 10https://gerrit.wikimedia.org/r/84932 (owner: 10Hashar) [14:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 14:32:48 UTC 2013 [14:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [14:33:42] (03PS1) 10Vogone: Enabled WikiLove extension on kowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90699 [14:33:45] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 14:33:43 UTC 2013 [14:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [14:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 14:35:38 UTC 2013 [14:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [14:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 14:41:40 UTC 2013 [14:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [14:42:55] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 14:42:45 UTC 2013 [14:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [14:44:05] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 14:43:55 UTC 2013 [14:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 14:44:40 UTC 2013 [14:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [14:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [15:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 15:32:46 UTC 2013 [15:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [15:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 15:33:46 UTC 2013 [15:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [15:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 15:35:41 UTC 2013 [15:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [15:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 15:41:47 UTC 2013 [15:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [15:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 15:42:43 UTC 2013 [15:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [15:43:45] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 15:43:43 UTC 2013 [15:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [15:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 15:44:43 UTC 2013 [15:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:01:50] (03CR) 10Revi: [C: 031] Enabled WikiLove extension on kowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90699 (owner: 10Vogone) [16:11:52] (03CR) 10QChris: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90665 (owner: 10Yurik) [16:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 16:32:46 UTC 2013 [16:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 16:33:46 UTC 2013 [16:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 16:35:41 UTC 2013 [16:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 16:41:48 UTC 2013 [16:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 16:42:44 UTC 2013 [16:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 16:43:49 UTC 2013 [16:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 16:44:44 UTC 2013 [16:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:56] (03PS1) 10Reedy: Move a lot of the miscellaneous wikis out of their own specific docroots [operations/apache-config] - 10https://gerrit.wikimedia.org/r/90703 [16:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:49:38] (03PS2) 10Reedy: Move a lot of the miscellaneous wikis out of their own specific docroots [operations/apache-config] - 10https://gerrit.wikimedia.org/r/90703 [16:52:02] (03PS1) 10Reedy: Remove all superfluous docroot folders [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90704 [16:54:18] (03PS3) 10Reedy: Move a lot of the miscellaneous wikis out of their own specific docroots [operations/apache-config] - 10https://gerrit.wikimedia.org/r/90703 [16:55:41] (03PS4) 10Reedy: Move a lot of the miscellaneous wikis out of their own specific docroots [operations/apache-config] - 10https://gerrit.wikimedia.org/r/90703 [17:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 17:32:50 UTC 2013 [17:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [17:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 17:33:45 UTC 2013 [17:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [17:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 17:35:41 UTC 2013 [17:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [17:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 17:41:47 UTC 2013 [17:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [17:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 17:42:42 UTC 2013 [17:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [17:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 17:43:47 UTC 2013 [17:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 17:44:42 UTC 2013 [17:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [17:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [17:50:22] (03CR) 10Matanya: "Weird, I didn't get that issue, or missed it :/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90098 (owner: 10Matanya) [17:55:52] (03PS1) 10Odder: (bug 55909) Add an alias for NS_USER on kowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90708 [17:57:41] (03CR) 10Matanya: "mailed you." [operations/puppet] - 10https://gerrit.wikimedia.org/r/90561 (owner: 10Andrew Bogott) [18:32:45] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 18:32:41 UTC 2013 [18:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [18:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 18:33:47 UTC 2013 [18:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [18:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 18:35:42 UTC 2013 [18:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [18:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 18:41:44 UTC 2013 [18:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [18:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 18:42:39 UTC 2013 [18:42:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [18:43:45] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 18:43:44 UTC 2013 [18:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [18:45:05] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 18:44:54 UTC 2013 [18:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [18:47:57] (03PS1) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [operations/debs/gerrit] - 10https://gerrit.wikimedia.org/r/90715 [19:02:16] (03Draft1) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/90716 [19:05:44] (03PS2) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [operations/debs/latexml] - 10https://gerrit.wikimedia.org/r/90717 [19:32:45] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 19:32:43 UTC 2013 [19:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [19:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 19:33:49 UTC 2013 [19:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [19:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 19:35:39 UTC 2013 [19:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [19:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 19:41:45 UTC 2013 [19:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [19:42:55] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 19:42:50 UTC 2013 [19:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [19:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 19:43:46 UTC 2013 [19:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [19:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 19:44:41 UTC 2013 [19:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [19:51:51] (03PS2) 10Hashar: gbp: do not set export-dir [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/90716 [20:06:26] (03CR) 10Hashar: "Andrew, I am trying to get Jenkins to build the kafka package for us :-] The export-dir send it to the labs GlusterFS share which is not " [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/90716 (owner: 10Hashar) [20:12:58] (03PS2) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [operations/debs/varnish] - 10https://gerrit.wikimedia.org/r/90718 [20:23:21] (03PS3) 10Hashar: gbp: point upstream to testing/3.0.3plus-rc1 branch [operations/debs/varnish] - 10https://gerrit.wikimedia.org/r/90718 [20:27:05] RECOVERY - search indices - check lucene status page on search20 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.115 second response time [20:32:02] (03PS4) 10Hashar: gbp: point upstream to testing/3.0.3plus-rc1 branch [operations/debs/varnish] - 10https://gerrit.wikimedia.org/r/90718 [20:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 20:32:49 UTC 2013 [20:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [20:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 20:33:49 UTC 2013 [20:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [20:35:13] (03PS3) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [operations/debs/latexml] - 10https://gerrit.wikimedia.org/r/90717 [20:35:33] (03PS4) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [operations/debs/latexml] - 10https://gerrit.wikimedia.org/r/90717 [20:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 20:35:39 UTC 2013 [20:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [20:40:15] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:41:15] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 1 logical drive(s), 4 physical drive(s) [20:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 20:41:46 UTC 2013 [20:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 20:42:41 UTC 2013 [20:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 20:43:46 UTC 2013 [20:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 20:44:41 UTC 2013 [20:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [20:54:47] <^d> !log gerrit: installed 2.7-rc2-507-g1e7090b, service back up [20:55:01] Logged the message, Master [20:55:17] (03CR) 10Chad: "We're now live. This should be ok to go in now." [operations/puppet] - 10https://gerrit.wikimedia.org/r/84743 (owner: 10QChris) [20:56:15] (03PS2) 10QChris: Switch to single Json object for gerrit's reviewer count query [operations/puppet] - 10https://gerrit.wikimedia.org/r/84743 [20:56:29] ^d \o/ [20:57:08] <^d> :) [21:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 21:32:49 UTC 2013 [21:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [21:33:45] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 21:33:44 UTC 2013 [21:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [21:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 21:35:40 UTC 2013 [21:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [21:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 21:41:46 UTC 2013 [21:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [21:42:55] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 21:42:51 UTC 2013 [21:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [21:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 21:43:47 UTC 2013 [21:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 21:44:42 UTC 2013 [21:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [21:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [21:54:00] (03CR) 10Yurik: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90665 (owner: 10Yurik) [21:55:03] yurik: did you get my messages about Tor? :) [21:56:48] (03PS3) 10Yurik: Fixed incorrect domain matching for ZERO [operations/puppet] - 10https://gerrit.wikimedia.org/r/90665 [21:58:50] YuviPanda: if it was sent over Tor, it is probably still being routed through the network... [21:59:02] burn! :) [22:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 22:32:52 UTC 2013 [22:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [22:33:45] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 22:33:42 UTC 2013 [22:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [22:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 22:35:43 UTC 2013 [22:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [22:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 22:41:40 UTC 2013 [22:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [22:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 22:42:40 UTC 2013 [22:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [22:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 22:43:45 UTC 2013 [22:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [22:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 22:44:40 UTC 2013 [22:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [23:03:31] YuviPanda|zzz: thanks for the info - i passed it on :) [23:03:43] the connection here is a bit spotty :) [23:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sat Oct 19 23:32:45 UTC 2013 [23:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [23:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sat Oct 19 23:33:45 UTC 2013 [23:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [23:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sat Oct 19 23:35:41 UTC 2013 [23:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [23:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sat Oct 19 23:41:42 UTC 2013 [23:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [23:42:55] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sat Oct 19 23:42:47 UTC 2013 [23:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [23:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sat Oct 19 23:43:47 UTC 2013 [23:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sat Oct 19 23:44:37 UTC 2013 [23:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [23:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours