[00:14:59] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [00:52:59] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [01:53:43] !log catrope synchronized php-1.22wmf12/extensions/VisualEditor 'Update VE to master' [01:53:56] Logged the message, Master [02:15:31] !log LocalisationUpdate completed (1.22wmf12) at Thu Aug 1 02:15:31 UTC 2013 [02:15:45] Logged the message, Master [02:27:59] PROBLEM - Puppet freshness on sq41 is CRITICAL: No successful Puppet run in the last 10 hours [02:34:58] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 1 02:34:57 UTC 2013 [02:35:08] Logged the message, Master [03:43:59] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: No successful Puppet run in the last 10 hours [04:19:59] PROBLEM - Host mw16 is DOWN: PING CRITICAL - Packet loss = 100% [04:21:19] RECOVERY - Host mw16 is UP: PING OK - Packet loss = 0%, RTA = 26.58 ms [04:40:15] !log olivneh synchronized php-1.22wmf12/extensions/VisualEditor/VisualEditor.php 'Fix for bug 52368: VisualEditor.php attempts to register a hook handler that was removed' [04:40:27] Logged the message, Master [04:41:03] !log Not synced to srv281, mw1089, mw1173 (connection timed out). [04:41:13] Logged the message, Master [04:41:20] battery dieing, just caught up, hope all is well, ori-l [04:41:33] greg-g: other than me being an idiot, sure [04:41:49] ori-l: cool, everything's normal ;) [04:41:57] * greg-g hugs ori-l  [04:41:59] thanks for fixing [04:42:40] ok, with 5 minutes to spare, going to go plug in [04:43:21] thanks greg-g [04:53:59] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: No successful Puppet run in the last 10 hours [05:14:39] !log deleted half-created account melissajan0@enwiki, broken by bug 52368 [05:14:50] Logged the message, Master [05:27:56] !log delaying db56 during OSC bug 49194 [05:28:06] Logged the message, Master [05:35:43] !log delaying db43 during OSC bug 49194 [05:35:54] Logged the message, Master [05:40:00] !log delaying db45 during OSC bug 49194 [05:40:11] Logged the message, Master [05:40:59] PROBLEM - Puppet freshness on mw1106 is CRITICAL: No successful Puppet run in the last 10 hours [05:42:54] !log delaying db51 during OSC bug 49194 [05:43:05] Logged the message, Master [05:45:48] i smell a master switch coming :) [05:47:59] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [05:54:27] !log delaying db39 during OSC bug 49194 [05:54:38] Logged the message, Master [05:54:59] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [05:55:59] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [05:55:59] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [05:55:59] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [05:55:59] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [05:55:59] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [05:56:00] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [06:01:27] What is OSC? [06:01:53] ongoing schema change? [06:02:03] online schema change [06:02:14] All right. [06:02:45] online, as in without blocking anything else [06:02:53] theoretically :) [06:03:16] :-) [06:04:40] !log delaying db52 during OSC bug 49194 [06:04:50] Logged the message, Master [06:05:59] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [06:05:59] PROBLEM - Puppet freshness on analytics1010 is CRITICAL: No successful Puppet run in the last 10 hours [06:05:59] PROBLEM - Puppet freshness on cp1012 is CRITICAL: No successful Puppet run in the last 10 hours [06:05:59] PROBLEM - Puppet freshness on cp1020 is CRITICAL: No successful Puppet run in the last 10 hours [06:05:59] PROBLEM - Puppet freshness on db1019 is CRITICAL: No successful Puppet run in the last 10 hours [06:07:59] PROBLEM - Puppet freshness on cp1006 is CRITICAL: No successful Puppet run in the last 10 hours [06:07:59] PROBLEM - Puppet freshness on cp1046 is CRITICAL: No successful Puppet run in the last 10 hours [06:07:59] PROBLEM - Puppet freshness on mc1005 is CRITICAL: No successful Puppet run in the last 10 hours [06:07:59] PROBLEM - Puppet freshness on mw1044 is CRITICAL: No successful Puppet run in the last 10 hours [06:07:59] PROBLEM - Puppet freshness on db1048 is CRITICAL: No successful Puppet run in the last 10 hours [06:07:59] PROBLEM - Puppet freshness on cp1050 is CRITICAL: No successful Puppet run in the last 10 hours [06:07:59] PROBLEM - Puppet freshness on mw1111 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:00] PROBLEM - Puppet freshness on mw1124 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:00] PROBLEM - Puppet freshness on mw1125 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:01] PROBLEM - Puppet freshness on gadolinium is CRITICAL: No successful Puppet run in the last 10 hours [06:08:01] PROBLEM - Puppet freshness on mw1130 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:02] PROBLEM - Puppet freshness on ms-be4 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:02] PROBLEM - Puppet freshness on mw118 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:03] PROBLEM - Puppet freshness on mw1161 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:03] PROBLEM - Puppet freshness on mw1147 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:04] PROBLEM - Puppet freshness on mw1190 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:04] PROBLEM - Puppet freshness on mw38 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:05] PROBLEM - Puppet freshness on search36 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:05] PROBLEM - Puppet freshness on mw119 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:06] PROBLEM - Puppet freshness on mw83 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:06] PROBLEM - Puppet freshness on sq79 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:07] PROBLEM - Puppet freshness on srv301 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:07] PROBLEM - Puppet freshness on wtp1024 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:08] PROBLEM - Puppet freshness on srv290 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:59] PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:59] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:59] PROBLEM - Puppet freshness on analytics1023 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:59] PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:59] PROBLEM - Puppet freshness on db1004 is CRITICAL: No successful Puppet run in the last 10 hours [06:10:51] !log delaying db32 during OSC bug 49194 [06:10:59] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [06:10:59] PROBLEM - Puppet freshness on cp1002 is CRITICAL: No successful Puppet run in the last 10 hours [06:10:59] PROBLEM - Puppet freshness on cp1005 is CRITICAL: No successful Puppet run in the last 10 hours [06:10:59] PROBLEM - Puppet freshness on cp3009 is CRITICAL: No successful Puppet run in the last 10 hours [06:10:59] PROBLEM - Puppet freshness on cp3012 is CRITICAL: No successful Puppet run in the last 10 hours [06:11:03] Logged the message, Master [06:11:43] hah, bye icinga [06:12:05] jeremyb: hi. hello, how can I configure svg uploads to wiki based on 1.19? [06:12:25] springle: what's delaying exactly? how do i see that in ganglia? [06:12:50] matanya: err, svg uploads or svg thumbnails? anyway, ask #mediawiki [06:12:59] PROBLEM - Puppet freshness on analytics1004 is CRITICAL: No successful Puppet run in the last 10 hours [06:12:59] PROBLEM - Puppet freshness on calcium is CRITICAL: No successful Puppet run in the last 10 hours [06:12:59] PROBLEM - Puppet freshness on cp1037 is CRITICAL: No successful Puppet run in the last 10 hours [06:12:59] PROBLEM - Puppet freshness on db1002 is CRITICAL: No successful Puppet run in the last 10 hours [06:12:59] PROBLEM - Puppet freshness on db1033 is CRITICAL: No successful Puppet run in the last 10 hours [06:13:11] icinga's funn :) [06:13:14] funny* [06:13:23] matanya: && boker tov! [06:13:24] jeremyb, delaying = pausing one slave while schema changes occur on the master and all other slaves. just a safeguard [06:13:30] thanks jeremyb :) [06:13:37] ganglia will show the slave as stopped [06:13:39] springle: so, just stop sql thread? [06:13:59] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours [06:13:59] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [06:13:59] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [06:13:59] PROBLEM - Puppet freshness on analytics1003 is CRITICAL: No successful Puppet run in the last 10 hours [06:13:59] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 10 hours [06:14:48] jeremyb, sure just sql thread would do it. i stop io thread too just to keep it simple :) [06:14:59] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: No successful Puppet run in the last 10 hours [06:14:59] PROBLEM - Puppet freshness on antimony is CRITICAL: No successful Puppet run in the last 10 hours [06:14:59] PROBLEM - Puppet freshness on cp1058 is CRITICAL: No successful Puppet run in the last 10 hours [06:14:59] PROBLEM - Puppet freshness on cp3011 is CRITICAL: No successful Puppet run in the last 10 hours [06:14:59] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: No successful Puppet run in the last 10 hours [06:15:09] k [06:15:35] doesn't sound like OSC as i understood it. i thought the slaves are done and then master switch and then old master is done. all without hitting a binlog [06:15:38] ? [06:15:43] springle [06:15:59] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [06:15:59] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [06:15:59] PROBLEM - Puppet freshness on analytics1002 is CRITICAL: No successful Puppet run in the last 10 hours [06:15:59] PROBLEM - Puppet freshness on analytics1009 is CRITICAL: No successful Puppet run in the last 10 hours [06:15:59] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: No successful Puppet run in the last 10 hours [06:16:10] jeblair, we're using pt-online-schema-change. doesn't need master/slave switching [06:16:15] jeremyb: ^ [06:16:16] oops jeremyb [06:16:25] hah, didn't realize jeblair was here [06:16:48] springle: huh, i'll reread then [06:16:53] icinga-wm: chill pill? [06:16:59] PROBLEM - Puppet freshness on cp1016 is CRITICAL: No successful Puppet run in the last 10 hours [06:16:59] PROBLEM - Puppet freshness on cp1062 is CRITICAL: No successful Puppet run in the last 10 hours [06:16:59] PROBLEM - Puppet freshness on db1005 is CRITICAL: No successful Puppet run in the last 10 hours [06:16:59] PROBLEM - Puppet freshness on db1009 is CRITICAL: No successful Puppet run in the last 10 hours [06:16:59] PROBLEM - Puppet freshness on db32 is CRITICAL: No successful Puppet run in the last 10 hours [06:16:59] PROBLEM - Puppet freshness on db67 is CRITICAL: No successful Puppet run in the last 10 hours [06:16:59] PROBLEM - Puppet freshness on db69 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:00] PROBLEM - Puppet freshness on es9 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:01] PROBLEM - Puppet freshness on mw1020 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:01] PROBLEM - Puppet freshness on labstore2 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:01] PROBLEM - Puppet freshness on mw1030 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:02] PROBLEM - Puppet freshness on mw1075 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:03] PROBLEM - Puppet freshness on mw1078 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:03] PROBLEM - Puppet freshness on mw1165 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:03] PROBLEM - Puppet freshness on mw1179 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:04] PROBLEM - Puppet freshness on mw1181 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:04] PROBLEM - Puppet freshness on mw45 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:05] PROBLEM - Puppet freshness on sanger is CRITICAL: No successful Puppet run in the last 10 hours [06:17:05] PROBLEM - Puppet freshness on search1015 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:06] PROBLEM - Puppet freshness on search31 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:07] PROBLEM - Puppet freshness on solr1001 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:07] PROBLEM - Puppet freshness on sq62 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:08] PROBLEM - Puppet freshness on srv291 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:59] PROBLEM - Puppet freshness on amssq35 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:59] PROBLEM - Puppet freshness on amssq31 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:59] PROBLEM - Puppet freshness on amssq37 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:59] PROBLEM - Puppet freshness on analytics1008 is CRITICAL: No successful Puppet run in the last 10 hours [06:17:59] PROBLEM - Puppet freshness on cp1038 is CRITICAL: No successful Puppet run in the last 10 hours [06:19:59] PROBLEM - Puppet freshness on db1010 is CRITICAL: No successful Puppet run in the last 10 hours [06:19:59] PROBLEM - Puppet freshness on db46 is CRITICAL: No successful Puppet run in the last 10 hours [06:19:59] PROBLEM - Puppet freshness on db55 is CRITICAL: No successful Puppet run in the last 10 hours [06:19:59] PROBLEM - Puppet freshness on labsdb1001 is CRITICAL: No successful Puppet run in the last 10 hours [06:19:59] PROBLEM - Puppet freshness on mw1001 is CRITICAL: No successful Puppet run in the last 10 hours [06:19:59] PROBLEM - Puppet freshness on es1009 is CRITICAL: No successful Puppet run in the last 10 hours [06:19:59] PROBLEM - Puppet freshness on mw1022 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:00] PROBLEM - Puppet freshness on mw1040 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:00] PROBLEM - Puppet freshness on mw1062 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:01] PROBLEM - Puppet freshness on mw109 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:01] PROBLEM - Puppet freshness on mw107 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:02] PROBLEM - Puppet freshness on mw1132 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:02] PROBLEM - Puppet freshness on mw1185 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:03] PROBLEM - Puppet freshness on mw1218 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:03] PROBLEM - Puppet freshness on mw40 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:04] PROBLEM - Puppet freshness on mw134 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:04] PROBLEM - Puppet freshness on mw53 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:05] PROBLEM - Puppet freshness on mw70 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:05] PROBLEM - Puppet freshness on snapshot4 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:06] PROBLEM - Puppet freshness on srv296 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:06] PROBLEM - Puppet freshness on wtp1021 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:07] PROBLEM - Puppet freshness on tarin is CRITICAL: No successful Puppet run in the last 10 hours [06:20:59] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours [06:20:59] PROBLEM - Puppet freshness on amssq40 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:59] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:59] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:59] PROBLEM - Puppet freshness on cp3006 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:59] maybe varnish can teach icinga-wm how to meditate [06:22:59] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours [06:22:59] PROBLEM - Puppet freshness on chromium is CRITICAL: No successful Puppet run in the last 10 hours [06:22:59] PROBLEM - Puppet freshness on cp1039 is CRITICAL: No successful Puppet run in the last 10 hours [06:22:59] PROBLEM - Puppet freshness on cp1053 is CRITICAL: No successful Puppet run in the last 10 hours [06:22:59] PROBLEM - Puppet freshness on cp1068 is CRITICAL: No successful Puppet run in the last 10 hours [06:22:59] PROBLEM - Puppet freshness on cp1054 is CRITICAL: No successful Puppet run in the last 10 hours [06:22:59] PROBLEM - Puppet freshness on db1045 is CRITICAL: No successful Puppet run in the last 10 hours [06:23:59] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [06:23:59] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: No successful Puppet run in the last 10 hours [06:23:59] PROBLEM - Puppet freshness on cp1001 is CRITICAL: No successful Puppet run in the last 10 hours [06:23:59] PROBLEM - Puppet freshness on cp3019 is CRITICAL: No successful Puppet run in the last 10 hours [06:23:59] PROBLEM - Puppet freshness on cp3021 is CRITICAL: No successful Puppet run in the last 10 hours [06:24:59] PROBLEM - Puppet freshness on cp1008 is CRITICAL: No successful Puppet run in the last 10 hours [06:24:59] PROBLEM - Puppet freshness on cp1009 is CRITICAL: No successful Puppet run in the last 10 hours [06:24:59] PROBLEM - Puppet freshness on cp1064 is CRITICAL: No successful Puppet run in the last 10 hours [06:24:59] PROBLEM - Puppet freshness on db1015 is CRITICAL: No successful Puppet run in the last 10 hours [06:24:59] PROBLEM - Puppet freshness on db1018 is CRITICAL: No successful Puppet run in the last 10 hours [06:25:59] PROBLEM - Puppet freshness on amssq36 is CRITICAL: No successful Puppet run in the last 10 hours [06:25:59] PROBLEM - Puppet freshness on amssq41 is CRITICAL: No successful Puppet run in the last 10 hours [06:25:59] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [06:25:59] PROBLEM - Puppet freshness on analytics1006 is CRITICAL: No successful Puppet run in the last 10 hours [06:25:59] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: No successful Puppet run in the last 10 hours [06:26:59] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [06:26:59] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: No successful Puppet run in the last 10 hours [06:26:59] PROBLEM - Puppet freshness on db1003 is CRITICAL: No successful Puppet run in the last 10 hours [06:26:59] PROBLEM - Puppet freshness on cp1019 is CRITICAL: No successful Puppet run in the last 10 hours [06:26:59] PROBLEM - Puppet freshness on db1043 is CRITICAL: No successful Puppet run in the last 10 hours [06:27:59] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: No successful Puppet run in the last 10 hours [06:27:59] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: No successful Puppet run in the last 10 hours [06:27:59] PROBLEM - Puppet freshness on cp3022 is CRITICAL: No successful Puppet run in the last 10 hours [06:27:59] PROBLEM - Puppet freshness on db1007 is CRITICAL: No successful Puppet run in the last 10 hours [06:27:59] PROBLEM - Puppet freshness on db1039 is CRITICAL: No successful Puppet run in the last 10 hours [06:28:59] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours [06:28:59] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run in the last 10 hours [06:28:59] PROBLEM - Puppet freshness on cp1011 is CRITICAL: No successful Puppet run in the last 10 hours [06:28:59] PROBLEM - Puppet freshness on cp1048 is CRITICAL: No successful Puppet run in the last 10 hours [06:28:59] PROBLEM - Puppet freshness on db1016 is CRITICAL: No successful Puppet run in the last 10 hours [06:29:59] PROBLEM - Puppet freshness on amssq46 is CRITICAL: No successful Puppet run in the last 10 hours [06:29:59] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [06:29:59] PROBLEM - Puppet freshness on cp1055 is CRITICAL: No successful Puppet run in the last 10 hours [06:29:59] PROBLEM - Puppet freshness on db1053 is CRITICAL: No successful Puppet run in the last 10 hours [06:29:59] PROBLEM - Puppet freshness on db1059 is CRITICAL: No successful Puppet run in the last 10 hours [06:30:59] PROBLEM - Puppet freshness on amssq38 is CRITICAL: No successful Puppet run in the last 10 hours [06:30:59] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: No successful Puppet run in the last 10 hours [06:30:59] PROBLEM - Puppet freshness on cp1052 is CRITICAL: No successful Puppet run in the last 10 hours [06:30:59] PROBLEM - Puppet freshness on db1006 is CRITICAL: No successful Puppet run in the last 10 hours [06:30:59] PROBLEM - Puppet freshness on cp1059 is CRITICAL: No successful Puppet run in the last 10 hours [06:32:59] PROBLEM - Puppet freshness on amssq39 is CRITICAL: No successful Puppet run in the last 10 hours [06:32:59] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [06:32:59] PROBLEM - Puppet freshness on analytics1005 is CRITICAL: No successful Puppet run in the last 10 hours [06:32:59] PROBLEM - Puppet freshness on analytics1025 is CRITICAL: No successful Puppet run in the last 10 hours [06:32:59] PROBLEM - Puppet freshness on cp1047 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:59] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:59] PROBLEM - Puppet freshness on mc1003 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:59] PROBLEM - Puppet freshness on db52 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:59] PROBLEM - Puppet freshness on maerlant is CRITICAL: No successful Puppet run in the last 10 hours [06:33:59] PROBLEM - Puppet freshness on mc1015 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:59] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:00] PROBLEM - Puppet freshness on mw1088 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:00] PROBLEM - Puppet freshness on mw1018 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:01] PROBLEM - Puppet freshness on mw1099 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:02] PROBLEM - Puppet freshness on mw1100 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:02] PROBLEM - Puppet freshness on mw1117 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:02] PROBLEM - Puppet freshness on mw1103 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:03] PROBLEM - Puppet freshness on mw1176 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:03] PROBLEM - Puppet freshness on mw1164 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:04] PROBLEM - Puppet freshness on mw1217 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:04] PROBLEM - Puppet freshness on mw16 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:05] PROBLEM - Puppet freshness on mw51 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:05] PROBLEM - Puppet freshness on pc2 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:06] PROBLEM - Puppet freshness on search1010 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:06] PROBLEM - Puppet freshness on search13 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:07] PROBLEM - Puppet freshness on sq67 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:07] PROBLEM - Puppet freshness on srv287 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:08] PROBLEM - Puppet freshness on srv300 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:08] PROBLEM - Puppet freshness on stafford is CRITICAL: No successful Puppet run in the last 10 hours [06:34:09] PROBLEM - Puppet freshness on tmh2 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:25] where's ian's hammer? i want to use it on icinga [06:34:59] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:59] PROBLEM - Puppet freshness on cp1007 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:59] PROBLEM - Puppet freshness on cp1056 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:59] PROBLEM - Puppet freshness on cp1061 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:59] PROBLEM - Puppet freshness on cp3007 is CRITICAL: No successful Puppet run in the last 10 hours [06:35:38] Can we use it on you instead? [06:35:43] If you're annoyed by the bot, ignore it. [06:37:18] Elsie: there comes a point where the bot is counterproductive. idk if we've gotten there yet or how long it's been this bad. but it seems worse than usual to me [06:37:34] I've had it ignored for a long time. [06:37:44] Though I still see the voicing by ChanServ. [06:37:48] Which is kind of irksome. [06:37:54] ian's hammer for the unitiated: https://secure.flickr.com/photos/raindrift/sets/72157629492908038/ [06:38:02] hah, can't miss it all [06:59:59] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours [07:54:59] PROBLEM - Puppet freshness on mchenry is CRITICAL: No successful Puppet run in the last 10 hours [08:09:14] hello [08:29:52] (03PS1) 10Hashar: rake task to generate puppet documentation [operations/puppet] - 10https://gerrit.wikimedia.org/r/77090 [08:37:31] hashar's getting into the CA business [08:38:25] (03PS2) 10Hashar: rake task to generate puppet documentation [operations/puppet] - 10https://gerrit.wikimedia.org/r/77090 [08:38:36] jeremyb: some non profit could definitely regroup and create their own cert authority :-] [08:39:05] (03CR) 10Hashar: "I have dropped the dirname, it is not needed by puppet doc" [operations/puppet] - 10https://gerrit.wikimedia.org/r/77090 (owner: 10Hashar) [08:40:14] hashar: see my mail [08:40:20] jeremyb: yeah seen that [08:40:37] jeremyb: the new foundation would be the only able to sign :) [08:40:47] you don't want wikipedia to sign cert for mozilla domains :] [08:41:28] anyway, to become a cert authority which is published in all browser, you need a whole bunch of money to respect the process and be actually trusted. That surely cost a lot more money than all the certs we get [08:49:07] hashar: i don't think you understand... you'd still have all the other CAs still in the store. and that would still leave you vulnerable [08:50:47] jeremyb: my point was to avoid the "ssl mafia" by creating a new org [08:55:09] PROBLEM - udp2log log age for nginx on gadolinium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:55:59] RECOVERY - udp2log log age for nginx on gadolinium is OK: OK: all log files active [09:06:59] PROBLEM - Puppet freshness on holmium is CRITICAL: No successful Puppet run in the last 10 hours [10:00:36] (03PS11) 10Hashar: contint: publish Zuul git repositories [operations/puppet] - 10https://gerrit.wikimedia.org/r/71968 [10:00:47] (03CR) 10jenkins-bot: [V: 04-1] contint: publish Zuul git repositories [operations/puppet] - 10https://gerrit.wikimedia.org/r/71968 (owner: 10Hashar) [10:01:47] hashar: changing commit message makes it fail? [10:01:57] * AzaToth scratches heads [10:02:34] ah, reebase [10:10:10] (03PS12) 10Hashar: contint: publish Zuul git repositories [operations/puppet] - 10https://gerrit.wikimedia.org/r/71968 [10:10:34] (03CR) 10jenkins-bot: [V: 04-1] contint: publish Zuul git repositories [operations/puppet] - 10https://gerrit.wikimedia.org/r/71968 (owner: 10Hashar) [10:11:33] (03CR) 10Hashar: "Now with a role::zuul::configuration class which let us pass the zuul_git_dir to the website template." [operations/puppet] - 10https://gerrit.wikimedia.org/r/71968 (owner: 10Hashar) [10:13:51] (03PS13) 10Hashar: contint: publish Zuul git repositories [operations/puppet] - 10https://gerrit.wikimedia.org/r/71968 [10:15:59] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [10:35:57] (03CR) 10Hashar: [V: 031] "Works on a labs instance :-]" [operations/puppet] - 10https://gerrit.wikimedia.org/r/71968 (owner: 10Hashar) [10:53:59] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [11:14:18] did someone break ze puppetz? [11:15:23] what's wrong? [11:20:29] RECOVERY - Puppet freshness on sq55 is OK: puppet ran at Thu Aug 1 11:20:27 UTC 2013 [11:20:29] RECOVERY - Puppet freshness on sq64 is OK: puppet ran at Thu Aug 1 11:20:27 UTC 2013 [11:20:39] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Thu Aug 1 11:20:37 UTC 2013 [11:20:40] RECOVERY - Puppet freshness on sq81 is OK: puppet ran at Thu Aug 1 11:20:37 UTC 2013 [11:20:40] RECOVERY - Puppet freshness on aluminium is OK: puppet ran at Thu Aug 1 11:20:37 UTC 2013 [11:24:57] apache on stafford needed a restart [11:25:29] weird [11:25:39] RECOVERY - Puppet freshness on cp1052 is OK: puppet ran at Thu Aug 1 11:25:32 UTC 2013 [11:25:39] I'm so happy I ignored puppet freshness [11:25:39] RECOVERY - Puppet freshness on sq56 is OK: puppet ran at Thu Aug 1 11:25:32 UTC 2013 [11:25:45] the backlog is oh so clean [11:25:48] why didn't I do this before!? [11:25:49] RECOVERY - Puppet freshness on amssq36 is OK: puppet ran at Thu Aug 1 11:25:47 UTC 2013 [11:25:59] RECOVERY - Puppet freshness on gallium is OK: puppet ran at Thu Aug 1 11:25:52 UTC 2013 [11:26:09] RECOVERY - Puppet freshness on amssq41 is OK: puppet ran at Thu Aug 1 11:26:03 UTC 2013 [11:26:09] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Thu Aug 1 11:26:03 UTC 2013 [11:26:59] RECOVERY - Puppet freshness on ersch is OK: puppet ran at Thu Aug 1 11:26:53 UTC 2013 [11:26:59] RECOVERY - Puppet freshness on mw120 is OK: puppet ran at Thu Aug 1 11:26:53 UTC 2013 [11:26:59] RECOVERY - Puppet freshness on wtp1014 is OK: puppet ran at Thu Aug 1 11:26:53 UTC 2013 [11:26:59] RECOVERY - Puppet freshness on sq57 is OK: puppet ran at Thu Aug 1 11:26:53 UTC 2013 [11:26:59] RECOVERY - Puppet freshness on hooper is OK: puppet ran at Thu Aug 1 11:26:53 UTC 2013 [11:28:09] RECOVERY - Puppet freshness on analytics1016 is OK: puppet ran at Thu Aug 1 11:28:03 UTC 2013 [11:28:09] RECOVERY - Puppet freshness on cp3022 is OK: puppet ran at Thu Aug 1 11:28:03 UTC 2013 [11:28:09] RECOVERY - Puppet freshness on db37 is OK: puppet ran at Thu Aug 1 11:28:03 UTC 2013 [11:28:19] RECOVERY - Puppet freshness on search26 is OK: puppet ran at Thu Aug 1 11:28:08 UTC 2013 [11:28:19] RECOVERY - Puppet freshness on mw1167 is OK: puppet ran at Thu Aug 1 11:28:08 UTC 2013 [11:28:19] RECOVERY - Puppet freshness on mw17 is OK: puppet ran at Thu Aug 1 11:28:18 UTC 2013 [11:28:19] RECOVERY - Puppet freshness on mw1013 is OK: puppet ran at Thu Aug 1 11:28:18 UTC 2013 [11:28:20] RECOVERY - Puppet freshness on cp1019 is OK: puppet ran at Thu Aug 1 11:28:18 UTC 2013 [11:28:20] RECOVERY - Puppet freshness on mw1036 is OK: puppet ran at Thu Aug 1 11:28:18 UTC 2013 [11:28:29] RECOVERY - Puppet freshness on mw84 is OK: puppet ran at Thu Aug 1 11:28:18 UTC 2013 [11:28:29] RECOVERY - Puppet freshness on mw71 is OK: puppet ran at Thu Aug 1 11:28:18 UTC 2013 [11:28:29] RECOVERY - Puppet freshness on mw1076 is OK: puppet ran at Thu Aug 1 11:28:23 UTC 2013 [11:28:29] RECOVERY - Puppet freshness on es1006 is OK: puppet ran at Thu Aug 1 11:28:23 UTC 2013 [11:28:29] RECOVERY - Puppet freshness on analytics1026 is OK: puppet ran at Thu Aug 1 11:28:23 UTC 2013 [11:28:39] RECOVERY - Puppet freshness on lvs3 is OK: puppet ran at Thu Aug 1 11:28:29 UTC 2013 [11:28:39] RECOVERY - Puppet freshness on ms-fe4 is OK: puppet ran at Thu Aug 1 11:28:29 UTC 2013 [11:28:39] RECOVERY - Puppet freshness on analytics1013 is OK: puppet ran at Thu Aug 1 11:28:35 UTC 2013 [11:28:39] RECOVERY - Puppet freshness on mw18 is OK: puppet ran at Thu Aug 1 11:28:35 UTC 2013 [11:28:40] RECOVERY - Puppet freshness on mw1133 is OK: puppet ran at Thu Aug 1 11:28:35 UTC 2013 [11:28:40] RECOVERY - Puppet freshness on lvs4 is OK: puppet ran at Thu Aug 1 11:28:35 UTC 2013 [11:28:40] RECOVERY - Puppet freshness on mw1109 is OK: puppet ran at Thu Aug 1 11:28:35 UTC 2013 [11:28:47] hehe, welcome to sanity ;) [11:28:49] RECOVERY - Puppet freshness on mw1014 is OK: puppet ran at Thu Aug 1 11:28:40 UTC 2013 [11:28:49] RECOVERY - Puppet freshness on search24 is OK: puppet ran at Thu Aug 1 11:28:40 UTC 2013 [11:28:49] RECOVERY - Puppet freshness on mw1168 is OK: puppet ran at Thu Aug 1 11:28:45 UTC 2013 [11:28:49] RECOVERY - Puppet freshness on db1043 is OK: puppet ran at Thu Aug 1 11:28:45 UTC 2013 [11:28:59] RECOVERY - Puppet freshness on db49 is OK: puppet ran at Thu Aug 1 11:28:50 UTC 2013 [11:28:59] RECOVERY - Puppet freshness on mw1202 is OK: puppet ran at Thu Aug 1 11:28:50 UTC 2013 [11:28:59] RECOVERY - Puppet freshness on cp1004 is OK: puppet ran at Thu Aug 1 11:28:50 UTC 2013 [11:28:59] RECOVERY - Puppet freshness on es1007 is OK: puppet ran at Thu Aug 1 11:28:55 UTC 2013 [11:28:59] RECOVERY - Puppet freshness on mw1098 is OK: puppet ran at Thu Aug 1 11:28:55 UTC 2013 [11:28:59] RECOVERY - Puppet freshness on nfs1 is OK: puppet ran at Thu Aug 1 11:28:55 UTC 2013 [11:29:00] RECOVERY - Puppet freshness on mw1079 is OK: puppet ran at Thu Aug 1 11:28:55 UTC 2013 [11:29:00] RECOVERY - Puppet freshness on cp1011 is OK: puppet ran at Thu Aug 1 11:28:55 UTC 2013 [11:29:01] RECOVERY - Puppet freshness on formey is OK: puppet ran at Thu Aug 1 11:28:55 UTC 2013 [11:29:09] RECOVERY - Puppet freshness on db74 is OK: puppet ran at Thu Aug 1 11:29:00 UTC 2013 [11:29:09] RECOVERY - Puppet freshness on rdb1001 is OK: puppet ran at Thu Aug 1 11:29:00 UTC 2013 [11:29:09] RECOVERY - Puppet freshness on db1039 is OK: puppet ran at Thu Aug 1 11:29:00 UTC 2013 [11:29:09] RECOVERY - Puppet freshness on mc1010 is OK: puppet ran at Thu Aug 1 11:29:00 UTC 2013 [11:29:09] RECOVERY - Puppet freshness on iron is OK: puppet ran at Thu Aug 1 11:29:00 UTC 2013 [11:29:09] RECOVERY - Puppet freshness on virt1007 is OK: puppet ran at Thu Aug 1 11:29:05 UTC 2013 [11:29:10] RECOVERY - Puppet freshness on db72 is OK: puppet ran at Thu Aug 1 11:29:05 UTC 2013 [11:29:11] RECOVERY - Puppet freshness on ms-be1008 is OK: puppet ran at Thu Aug 1 11:29:05 UTC 2013 [11:29:19] RECOVERY - Puppet freshness on sq70 is OK: puppet ran at Thu Aug 1 11:29:10 UTC 2013 [11:29:19] RECOVERY - Puppet freshness on amslvs3 is OK: puppet ran at Thu Aug 1 11:29:10 UTC 2013 [11:29:19] RECOVERY - Puppet freshness on db1016 is OK: puppet ran at Thu Aug 1 11:29:10 UTC 2013 [11:29:19] RECOVERY - Puppet freshness on mc1004 is OK: puppet ran at Thu Aug 1 11:29:10 UTC 2013 [11:29:19] RECOVERY - Puppet freshness on virt7 is OK: puppet ran at Thu Aug 1 11:29:10 UTC 2013 [11:29:20] RECOVERY - Puppet freshness on db1030 is OK: puppet ran at Thu Aug 1 11:29:15 UTC 2013 [11:29:20] RECOVERY - Puppet freshness on mw69 is OK: puppet ran at Thu Aug 1 11:29:15 UTC 2013 [11:29:20] RECOVERY - Puppet freshness on cp1048 is OK: puppet ran at Thu Aug 1 11:29:15 UTC 2013 [11:29:21] RECOVERY - Puppet freshness on oxygen is OK: puppet ran at Thu Aug 1 11:29:15 UTC 2013 [11:29:21] RECOVERY - Puppet freshness on mw1149 is OK: puppet ran at Thu Aug 1 11:29:15 UTC 2013 [11:29:22] RECOVERY - Puppet freshness on mw63 is OK: puppet ran at Thu Aug 1 11:29:15 UTC 2013 [11:29:29] RECOVERY - Puppet freshness on mw1035 is OK: puppet ran at Thu Aug 1 11:29:21 UTC 2013 [11:29:29] RECOVERY - Puppet freshness on mw97 is OK: puppet ran at Thu Aug 1 11:29:21 UTC 2013 [11:29:29] RECOVERY - Puppet freshness on mw86 is OK: puppet ran at Thu Aug 1 11:29:21 UTC 2013 [11:29:29] RECOVERY - Puppet freshness on snapshot1001 is OK: puppet ran at Thu Aug 1 11:29:21 UTC 2013 [11:29:29] RECOVERY - Puppet freshness on ssl3003 is OK: puppet ran at Thu Aug 1 11:29:21 UTC 2013 [11:29:29] RECOVERY - Puppet freshness on mw72 is OK: puppet ran at Thu Aug 1 11:29:26 UTC 2013 [11:29:30] RECOVERY - Puppet freshness on mw85 is OK: puppet ran at Thu Aug 1 11:29:26 UTC 2013 [11:29:30] RECOVERY - Puppet freshness on mw1055 is OK: puppet ran at Thu Aug 1 11:29:26 UTC 2013 [11:29:31] RECOVERY - Puppet freshness on srv263 is OK: puppet ran at Thu Aug 1 11:29:26 UTC 2013 [11:29:31] RECOVERY - Puppet freshness on mw80 is OK: puppet ran at Thu Aug 1 11:29:26 UTC 2013 [11:29:32] RECOVERY - Puppet freshness on mw88 is OK: puppet ran at Thu Aug 1 11:29:26 UTC 2013 [11:29:32] RECOVERY - Puppet freshness on search32 is OK: puppet ran at Thu Aug 1 11:29:26 UTC 2013 [11:29:39] RECOVERY - Puppet freshness on mw1183 is OK: puppet ran at Thu Aug 1 11:29:31 UTC 2013 [11:29:39] RECOVERY - Puppet freshness on pdf2 is OK: puppet ran at Thu Aug 1 11:29:31 UTC 2013 [11:29:40] RECOVERY - Puppet freshness on hooft is OK: puppet ran at Thu Aug 1 11:29:36 UTC 2013 [11:29:40] RECOVERY - Puppet freshness on mw1156 is OK: puppet ran at Thu Aug 1 11:29:36 UTC 2013 [11:29:40] RECOVERY - Puppet freshness on mw1151 is OK: puppet ran at Thu Aug 1 11:29:36 UTC 2013 [11:29:49] RECOVERY - Puppet freshness on amssq46 is OK: puppet ran at Thu Aug 1 11:29:41 UTC 2013 [11:29:49] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Thu Aug 1 11:29:41 UTC 2013 [11:29:49] RECOVERY - Puppet freshness on mw1051 is OK: puppet ran at Thu Aug 1 11:29:46 UTC 2013 [11:29:49] RECOVERY - Puppet freshness on search1002 is OK: puppet ran at Thu Aug 1 11:29:46 UTC 2013 [11:29:49] RECOVERY - Puppet freshness on mw32 is OK: puppet ran at Thu Aug 1 11:29:46 UTC 2013 [11:29:49] RECOVERY - Puppet freshness on mw1216 is OK: puppet ran at Thu Aug 1 11:29:46 UTC 2013 [11:29:59] RECOVERY - Puppet freshness on es8 is OK: puppet ran at Thu Aug 1 11:29:51 UTC 2013 [11:29:59] RECOVERY - Puppet freshness on db1053 is OK: puppet ran at Thu Aug 1 11:29:51 UTC 2013 [11:29:59] RECOVERY - Puppet freshness on ssl3002 is OK: puppet ran at Thu Aug 1 11:29:56 UTC 2013 [11:30:09] RECOVERY - Puppet freshness on db1059 is OK: puppet ran at Thu Aug 1 11:30:01 UTC 2013 [11:30:09] RECOVERY - Puppet freshness on wtp1023 is OK: puppet ran at Thu Aug 1 11:30:01 UTC 2013 [11:30:09] RECOVERY - Puppet freshness on virt5 is OK: puppet ran at Thu Aug 1 11:30:01 UTC 2013 [11:30:09] RECOVERY - Puppet freshness on wtp1022 is OK: puppet ran at Thu Aug 1 11:30:01 UTC 2013 [11:30:09] RECOVERY - Puppet freshness on ssl3 is OK: puppet ran at Thu Aug 1 11:30:06 UTC 2013 [11:30:10] RECOVERY - Puppet freshness on mw102 is OK: puppet ran at Thu Aug 1 11:30:06 UTC 2013 [11:30:19] RECOVERY - Puppet freshness on cp1055 is OK: puppet ran at Thu Aug 1 11:30:11 UTC 2013 [11:30:19] RECOVERY - Puppet freshness on mw6 is OK: puppet ran at Thu Aug 1 11:30:11 UTC 2013 [11:30:19] RECOVERY - Puppet freshness on srv279 is OK: puppet ran at Thu Aug 1 11:30:11 UTC 2013 [11:30:19] RECOVERY - Puppet freshness on srv241 is OK: puppet ran at Thu Aug 1 11:30:11 UTC 2013 [11:30:19] RECOVERY - Puppet freshness on snapshot1004 is OK: puppet ran at Thu Aug 1 11:30:11 UTC 2013 [11:30:20] RECOVERY - Puppet freshness on mw4 is OK: puppet ran at Thu Aug 1 11:30:11 UTC 2013 [11:30:20] RECOVERY - Puppet freshness on srv288 is OK: puppet ran at Thu Aug 1 11:30:16 UTC 2013 [11:30:20] (03CR) 10Mark Bergsma: [C: 032] "We'll cross that bridge when we'll get there." [operations/puppet] - 10https://gerrit.wikimedia.org/r/76873 (owner: 10Yurik) [11:30:20] RECOVERY - Puppet freshness on mw27 is OK: puppet ran at Thu Aug 1 11:30:16 UTC 2013 [11:30:21] RECOVERY - Puppet freshness on mw44 is OK: puppet ran at Thu Aug 1 11:30:16 UTC 2013 [11:30:29] RECOVERY - Puppet freshness on mw1096 is OK: puppet ran at Thu Aug 1 11:30:21 UTC 2013 [11:30:29] RECOVERY - Puppet freshness on mw1050 is OK: puppet ran at Thu Aug 1 11:30:21 UTC 2013 [11:30:29] RECOVERY - Puppet freshness on mw1163 is OK: puppet ran at Thu Aug 1 11:30:21 UTC 2013 [11:30:29] RECOVERY - Puppet freshness on mw1097 is OK: puppet ran at Thu Aug 1 11:30:21 UTC 2013 [11:30:29] RECOVERY - Puppet freshness on mw1023 is OK: puppet ran at Thu Aug 1 11:30:21 UTC 2013 [11:30:30] RECOVERY - Puppet freshness on mw1034 is OK: puppet ran at Thu Aug 1 11:30:21 UTC 2013 [11:30:30] RECOVERY - Puppet freshness on search1016 is OK: puppet ran at Thu Aug 1 11:30:21 UTC 2013 [11:30:30] RECOVERY - Puppet freshness on mw1002 is OK: puppet ran at Thu Aug 1 11:30:21 UTC 2013 [11:30:31] RECOVERY - Puppet freshness on mw1212 is OK: puppet ran at Thu Aug 1 11:30:26 UTC 2013 [11:30:31] RECOVERY - Puppet freshness on search1017 is OK: puppet ran at Thu Aug 1 11:30:26 UTC 2013 [11:30:32] RECOVERY - Puppet freshness on mw1029 is OK: puppet ran at Thu Aug 1 11:30:26 UTC 2013 [11:30:32] RECOVERY - Puppet freshness on mw1077 is OK: puppet ran at Thu Aug 1 11:30:26 UTC 2013 [11:30:39] RECOVERY - Puppet freshness on sq80 is OK: puppet ran at Thu Aug 1 11:30:36 UTC 2013 [11:30:49] RECOVERY - Puppet freshness on ms1001 is OK: puppet ran at Thu Aug 1 11:30:42 UTC 2013 [11:30:49] RECOVERY - Puppet freshness on amssq38 is OK: puppet ran at Thu Aug 1 11:30:42 UTC 2013 [11:30:49] RECOVERY - Puppet freshness on db47 is OK: puppet ran at Thu Aug 1 11:30:42 UTC 2013 [11:30:49] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Thu Aug 1 11:30:47 UTC 2013 [11:30:49] RECOVERY - Puppet freshness on pc3 is OK: puppet ran at Thu Aug 1 11:30:47 UTC 2013 [11:30:49] RECOVERY - Puppet freshness on db1038 is OK: puppet ran at Thu Aug 1 11:30:47 UTC 2013 [11:30:50] RECOVERY - Puppet freshness on db62 is OK: puppet ran at Thu Aug 1 11:30:47 UTC 2013 [11:30:59] RECOVERY - Puppet freshness on ms-be1001 is OK: puppet ran at Thu Aug 1 11:30:52 UTC 2013 [11:30:59] RECOVERY - Puppet freshness on db35 is OK: puppet ran at Thu Aug 1 11:30:52 UTC 2013 [11:30:59] RECOVERY - Puppet freshness on ms-be1011 is OK: puppet ran at Thu Aug 1 11:30:57 UTC 2013 [11:30:59] RECOVERY - Puppet freshness on locke is OK: puppet ran at Thu Aug 1 11:30:57 UTC 2013 [11:30:59] RECOVERY - Puppet freshness on rdb1004 is OK: puppet ran at Thu Aug 1 11:30:57 UTC 2013 [11:30:59] RECOVERY - Puppet freshness on pc1001 is OK: puppet ran at Thu Aug 1 11:30:57 UTC 2013 [11:31:00] RECOVERY - Puppet freshness on wtp1019 is OK: puppet ran at Thu Aug 1 11:30:57 UTC 2013 [11:31:00] RECOVERY - Puppet freshness on db1047 is OK: puppet ran at Thu Aug 1 11:30:57 UTC 2013 [11:31:01] RECOVERY - Puppet freshness on es1010 is OK: puppet ran at Thu Aug 1 11:30:57 UTC 2013 [11:31:09] RECOVERY - Puppet freshness on cp1059 is OK: puppet ran at Thu Aug 1 11:31:02 UTC 2013 [11:31:09] RECOVERY - Puppet freshness on wtp1017 is OK: puppet ran at Thu Aug 1 11:31:02 UTC 2013 [11:31:09] RECOVERY - Puppet freshness on mw89 is OK: puppet ran at Thu Aug 1 11:31:02 UTC 2013 [11:31:09] RECOVERY - Puppet freshness on mw21 is OK: puppet ran at Thu Aug 1 11:31:08 UTC 2013 [11:31:10] RECOVERY - Puppet freshness on ms-be6 is OK: puppet ran at Thu Aug 1 11:31:08 UTC 2013 [11:31:10] RECOVERY - Puppet freshness on analytics1027 is OK: puppet ran at Thu Aug 1 11:31:08 UTC 2013 [11:31:37] haha :) [11:32:19] RECOVERY - Puppet freshness on mw1209 is OK: puppet ran at Thu Aug 1 11:32:13 UTC 2013 [11:32:19] RECOVERY - Puppet freshness on mw1028 is OK: puppet ran at Thu Aug 1 11:32:18 UTC 2013 [11:32:20] RECOVERY - Puppet freshness on mw1152 is OK: puppet ran at Thu Aug 1 11:32:18 UTC 2013 [11:32:20] RECOVERY - Puppet freshness on mw1005 is OK: puppet ran at Thu Aug 1 11:32:18 UTC 2013 [11:32:29] RECOVERY - Puppet freshness on mw1016 is OK: puppet ran at Thu Aug 1 11:32:23 UTC 2013 [11:32:29] RECOVERY - Puppet freshness on mw1067 is OK: puppet ran at Thu Aug 1 11:32:28 UTC 2013 [11:32:29] RECOVERY - Puppet freshness on search1020 is OK: puppet ran at Thu Aug 1 11:32:28 UTC 2013 [11:32:29] RECOVERY - Puppet freshness on es1001 is OK: puppet ran at Thu Aug 1 11:32:28 UTC 2013 [11:32:29] RECOVERY - Puppet freshness on sq60 is OK: puppet ran at Thu Aug 1 11:32:28 UTC 2013 [11:32:39] RECOVERY - Puppet freshness on mw1192 is OK: puppet ran at Thu Aug 1 11:32:33 UTC 2013 [11:32:39] RECOVERY - Puppet freshness on cp1014 is OK: puppet ran at Thu Aug 1 11:32:33 UTC 2013 [11:32:39] RECOVERY - Puppet freshness on carbon is OK: puppet ran at Thu Aug 1 11:32:33 UTC 2013 [11:32:39] RECOVERY - Puppet freshness on mw123 is OK: puppet ran at Thu Aug 1 11:32:33 UTC 2013 [11:32:39] RECOVERY - Puppet freshness on amssq42 is OK: puppet ran at Thu Aug 1 11:32:33 UTC 2013 [11:32:39] RECOVERY - Puppet freshness on srv247 is OK: puppet ran at Thu Aug 1 11:32:33 UTC 2013 [11:32:40] RECOVERY - Puppet freshness on ms-fe1 is OK: puppet ran at Thu Aug 1 11:32:33 UTC 2013 [11:32:40] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Thu Aug 1 11:32:33 UTC 2013 [11:32:41] RECOVERY - Puppet freshness on zhen is OK: puppet ran at Thu Aug 1 11:32:33 UTC 2013 [11:32:41] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Thu Aug 1 11:32:33 UTC 2013 [11:32:42] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Thu Aug 1 11:32:33 UTC 2013 [11:32:42] RECOVERY - Puppet freshness on amssq33 is OK: puppet ran at Thu Aug 1 11:32:33 UTC 2013 [11:32:43] RECOVERY - Puppet freshness on db71 is OK: puppet ran at Thu Aug 1 11:32:33 UTC 2013 [11:32:43] RECOVERY - Puppet freshness on amssq45 is OK: puppet ran at Thu Aug 1 11:32:33 UTC 2013 [11:32:44] RECOVERY - Puppet freshness on db1017 is OK: puppet ran at Thu Aug 1 11:32:33 UTC 2013 [11:32:44] RECOVERY - Puppet freshness on wtp1009 is OK: puppet ran at Thu Aug 1 11:32:38 UTC 2013 [11:32:45] RECOVERY - Puppet freshness on cp1067 is OK: puppet ran at Thu Aug 1 11:32:38 UTC 2013 [11:32:45] RECOVERY - Puppet freshness on snapshot2 is OK: puppet ran at Thu Aug 1 11:32:38 UTC 2013 [11:32:46] RECOVERY - Puppet freshness on srv265 is OK: puppet ran at Thu Aug 1 11:32:38 UTC 2013 [11:32:49] RECOVERY - Puppet freshness on cp1070 is OK: puppet ran at Thu Aug 1 11:32:43 UTC 2013 [11:32:49] RECOVERY - Puppet freshness on srv277 is OK: puppet ran at Thu Aug 1 11:32:43 UTC 2013 [11:32:49] RECOVERY - Puppet freshness on srv256 is OK: puppet ran at Thu Aug 1 11:32:43 UTC 2013 [11:32:49] RECOVERY - Puppet freshness on cp1040 is OK: puppet ran at Thu Aug 1 11:32:43 UTC 2013 [11:32:49] RECOVERY - Puppet freshness on mw73 is OK: puppet ran at Thu Aug 1 11:32:43 UTC 2013 [11:32:50] RECOVERY - Puppet freshness on mw31 is OK: puppet ran at Thu Aug 1 11:32:43 UTC 2013 [11:32:50] RECOVERY - Puppet freshness on srv236 is OK: puppet ran at Thu Aug 1 11:32:43 UTC 2013 [11:32:50] RECOVERY - Puppet freshness on mw19 is OK: puppet ran at Thu Aug 1 11:32:48 UTC 2013 [11:32:51] RECOVERY - Puppet freshness on mw22 is OK: puppet ran at Thu Aug 1 11:32:48 UTC 2013 [11:32:51] RECOVERY - Puppet freshness on search15 is OK: puppet ran at Thu Aug 1 11:32:48 UTC 2013 [11:32:59] RECOVERY - Puppet freshness on srv248 is OK: puppet ran at Thu Aug 1 11:32:53 UTC 2013 [11:32:59] RECOVERY - Puppet freshness on yvon is OK: puppet ran at Thu Aug 1 11:32:53 UTC 2013 [11:32:59] RECOVERY - Puppet freshness on mw77 is OK: puppet ran at Thu Aug 1 11:32:53 UTC 2013 [11:32:59] RECOVERY - Puppet freshness on mw1193 is OK: puppet ran at Thu Aug 1 11:32:53 UTC 2013 [11:32:59] RECOVERY - Puppet freshness on mw1012 is OK: puppet ran at Thu Aug 1 11:32:58 UTC 2013 [11:32:59] RECOVERY - Puppet freshness on amssq39 is OK: puppet ran at Thu Aug 1 11:32:58 UTC 2013 [11:33:00] RECOVERY - Puppet freshness on mw1071 is OK: puppet ran at Thu Aug 1 11:32:58 UTC 2013 [11:33:09] RECOVERY - Puppet freshness on cp3004 is OK: puppet ran at Thu Aug 1 11:33:03 UTC 2013 [11:33:09] RECOVERY - Puppet freshness on lvs1003 is OK: puppet ran at Thu Aug 1 11:33:08 UTC 2013 [11:33:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:33:59] RECOVERY - Puppet freshness on mw1207 is OK: puppet ran at Thu Aug 1 11:33:58 UTC 2013 [11:33:59] RECOVERY - Puppet freshness on cp1045 is OK: puppet ran at Thu Aug 1 11:33:58 UTC 2013 [11:33:59] RECOVERY - Puppet freshness on mw1187 is OK: puppet ran at Thu Aug 1 11:33:58 UTC 2013 [11:33:59] RECOVERY - Puppet freshness on mc1015 is OK: puppet ran at Thu Aug 1 11:33:58 UTC 2013 [11:33:59] RECOVERY - Puppet freshness on analytics1005 is OK: puppet ran at Thu Aug 1 11:33:58 UTC 2013 [11:34:00] RECOVERY - Puppet freshness on mw24 is OK: puppet ran at Thu Aug 1 11:33:58 UTC 2013 [11:34:00] RECOVERY - Puppet freshness on mw105 is OK: puppet ran at Thu Aug 1 11:33:58 UTC 2013 [11:34:00] RECOVERY - Puppet freshness on mw103 is OK: puppet ran at Thu Aug 1 11:33:58 UTC 2013 [11:34:01] RECOVERY - Puppet freshness on cp3020 is OK: puppet ran at Thu Aug 1 11:33:58 UTC 2013 [11:34:01] RECOVERY - Puppet freshness on mw54 is OK: puppet ran at Thu Aug 1 11:33:58 UTC 2013 [11:34:02] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Thu Aug 1 11:33:58 UTC 2013 [11:34:02] RECOVERY - Puppet freshness on mw101 is OK: puppet ran at Thu Aug 1 11:33:58 UTC 2013 [11:34:09] RECOVERY - Puppet freshness on mw29 is OK: puppet ran at Thu Aug 1 11:34:03 UTC 2013 [11:34:09] RECOVERY - Puppet freshness on db1035 is OK: puppet ran at Thu Aug 1 11:34:03 UTC 2013 [11:34:09] RECOVERY - Puppet freshness on cp3005 is OK: puppet ran at Thu Aug 1 11:34:03 UTC 2013 [11:34:09] RECOVERY - Puppet freshness on cp3008 is OK: puppet ran at Thu Aug 1 11:34:03 UTC 2013 [11:34:09] RECOVERY - Puppet freshness on srv295 is OK: puppet ran at Thu Aug 1 11:34:03 UTC 2013 [11:34:09] RECOVERY - Puppet freshness on mw94 is OK: puppet ran at Thu Aug 1 11:34:03 UTC 2013 [11:34:10] RECOVERY - Puppet freshness on tridge is OK: puppet ran at Thu Aug 1 11:34:03 UTC 2013 [11:34:10] RECOVERY - Puppet freshness on virt6 is OK: puppet ran at Thu Aug 1 11:34:03 UTC 2013 [11:34:11] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Aug 1 11:34:03 UTC 2013 [11:34:11] RECOVERY - Puppet freshness on virt9 is OK: puppet ran at Thu Aug 1 11:34:03 UTC 2013 [11:34:12] RECOVERY - Puppet freshness on mw1006 is OK: puppet ran at Thu Aug 1 11:34:03 UTC 2013 [11:34:12] RECOVERY - Puppet freshness on mw1113 is OK: puppet ran at Thu Aug 1 11:34:03 UTC 2013 [11:34:13] RECOVERY - Puppet freshness on db1037 is OK: puppet ran at Thu Aug 1 11:34:03 UTC 2013 [11:34:13] RECOVERY - Puppet freshness on maerlant is OK: puppet ran at Thu Aug 1 11:34:03 UTC 2013 [11:34:14] RECOVERY - Puppet freshness on mw1037 is OK: puppet ran at Thu Aug 1 11:34:08 UTC 2013 [11:34:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 7.704 second response time [11:34:29] RECOVERY - Puppet freshness on pc2 is OK: puppet ran at Thu Aug 1 11:34:23 UTC 2013 [11:34:30] RECOVERY - Puppet freshness on srv287 is OK: puppet ran at Thu Aug 1 11:34:23 UTC 2013 [11:34:30] RECOVERY - Puppet freshness on sq67 is OK: puppet ran at Thu Aug 1 11:34:23 UTC 2013 [11:34:30] RECOVERY - Puppet freshness on mw1176 is OK: puppet ran at Thu Aug 1 11:34:23 UTC 2013 [11:34:30] RECOVERY - Puppet freshness on mw108 is OK: puppet ran at Thu Aug 1 11:34:23 UTC 2013 [11:34:30] RECOVERY - Puppet freshness on srv300 is OK: puppet ran at Thu Aug 1 11:34:23 UTC 2013 [11:34:30] RECOVERY - Puppet freshness on cp1046 is OK: puppet ran at Thu Aug 1 11:34:23 UTC 2013 [11:34:31] RECOVERY - Puppet freshness on cp1047 is OK: puppet ran at Thu Aug 1 11:34:23 UTC 2013 [11:34:31] RECOVERY - Puppet freshness on search34 is OK: puppet ran at Thu Aug 1 11:34:23 UTC 2013 [11:34:32] RECOVERY - Puppet freshness on snapshot1003 is OK: puppet ran at Thu Aug 1 11:34:24 UTC 2013 [11:34:32] RECOVERY - Puppet freshness on mw1164 is OK: puppet ran at Thu Aug 1 11:34:24 UTC 2013 [11:34:33] RECOVERY - Puppet freshness on mw11 is OK: puppet ran at Thu Aug 1 11:34:24 UTC 2013 [11:34:39] RECOVERY - Puppet freshness on analytics1025 is OK: puppet ran at Thu Aug 1 11:34:29 UTC 2013 [11:34:39] RECOVERY - Puppet freshness on mw1217 is OK: puppet ran at Thu Aug 1 11:34:29 UTC 2013 [11:34:39] RECOVERY - Puppet freshness on mw1064 is OK: puppet ran at Thu Aug 1 11:34:29 UTC 2013 [11:34:49] RECOVERY - Puppet freshness on search1010 is OK: puppet ran at Thu Aug 1 11:34:44 UTC 2013 [11:34:49] RECOVERY - Puppet freshness on db52 is OK: puppet ran at Thu Aug 1 11:34:44 UTC 2013 [11:34:49] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Thu Aug 1 11:34:44 UTC 2013 [11:34:49] RECOVERY - Puppet freshness on search28 is OK: puppet ran at Thu Aug 1 11:34:44 UTC 2013 [11:34:49] RECOVERY - Puppet freshness on sq71 is OK: puppet ran at Thu Aug 1 11:34:44 UTC 2013 [11:34:49] RECOVERY - Puppet freshness on mc1003 is OK: puppet ran at Thu Aug 1 11:34:44 UTC 2013 [11:34:59] RECOVERY - Puppet freshness on db40 is OK: puppet ran at Thu Aug 1 11:34:54 UTC 2013 [11:34:59] RECOVERY - Puppet freshness on cp1007 is OK: puppet ran at Thu Aug 1 11:34:54 UTC 2013 [11:35:09] RECOVERY - Puppet freshness on stafford is OK: puppet ran at Thu Aug 1 11:34:59 UTC 2013 [11:35:09] RECOVERY - Puppet freshness on tmh2 is OK: puppet ran at Thu Aug 1 11:34:59 UTC 2013 [11:35:19] RECOVERY - Puppet freshness on labsdb1003 is OK: puppet ran at Thu Aug 1 11:35:14 UTC 2013 [11:35:19] RECOVERY - Puppet freshness on db1054 is OK: puppet ran at Thu Aug 1 11:35:14 UTC 2013 [11:35:19] RECOVERY - Puppet freshness on mw1088 is OK: puppet ran at Thu Aug 1 11:35:14 UTC 2013 [11:35:19] RECOVERY - Puppet freshness on mw51 is OK: puppet ran at Thu Aug 1 11:35:14 UTC 2013 [11:35:19] RECOVERY - Puppet freshness on cp1061 is OK: puppet ran at Thu Aug 1 11:35:14 UTC 2013 [11:35:29] RECOVERY - Puppet freshness on cp3007 is OK: puppet ran at Thu Aug 1 11:35:19 UTC 2013 [11:35:29] RECOVERY - Puppet freshness on mw20 is OK: puppet ran at Thu Aug 1 11:35:24 UTC 2013 [11:35:29] RECOVERY - Puppet freshness on mw1103 is OK: puppet ran at Thu Aug 1 11:35:24 UTC 2013 [11:35:29] RECOVERY - Puppet freshness on search13 is OK: puppet ran at Thu Aug 1 11:35:24 UTC 2013 [11:35:29] RECOVERY - Puppet freshness on mw122 is OK: puppet ran at Thu Aug 1 11:35:24 UTC 2013 [11:35:31] (03PS1) 10Mark Bergsma: Fix variable name [operations/puppet] - 10https://gerrit.wikimedia.org/r/77095 [11:35:39] RECOVERY - Puppet freshness on mw1099 is OK: puppet ran at Thu Aug 1 11:35:29 UTC 2013 [11:35:39] RECOVERY - Puppet freshness on ms-be8 is OK: puppet ran at Thu Aug 1 11:35:29 UTC 2013 [11:35:39] RECOVERY - Puppet freshness on mw1052 is OK: puppet ran at Thu Aug 1 11:35:34 UTC 2013 [11:35:39] RECOVERY - Puppet freshness on cp3003 is OK: puppet ran at Thu Aug 1 11:35:34 UTC 2013 [11:35:39] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Thu Aug 1 11:35:34 UTC 2013 [11:35:39] RECOVERY - Puppet freshness on mw1123 is OK: puppet ran at Thu Aug 1 11:35:34 UTC 2013 [11:35:40] RECOVERY - Puppet freshness on williams is OK: puppet ran at Thu Aug 1 11:35:34 UTC 2013 [11:35:40] RECOVERY - Puppet freshness on mw1018 is OK: puppet ran at Thu Aug 1 11:35:34 UTC 2013 [11:35:49] RECOVERY - Puppet freshness on mw46 is OK: puppet ran at Thu Aug 1 11:35:39 UTC 2013 [11:35:49] RECOVERY - Puppet freshness on search17 is OK: puppet ran at Thu Aug 1 11:35:39 UTC 2013 [11:35:49] RECOVERY - Puppet freshness on mw1015 is OK: puppet ran at Thu Aug 1 11:35:44 UTC 2013 [11:35:49] RECOVERY - Puppet freshness on mw1117 is OK: puppet ran at Thu Aug 1 11:35:44 UTC 2013 [11:35:49] RECOVERY - Puppet freshness on mw1042 is OK: puppet ran at Thu Aug 1 11:35:44 UTC 2013 [11:35:49] RECOVERY - Puppet freshness on es1 is OK: puppet ran at Thu Aug 1 11:35:44 UTC 2013 [11:35:59] RECOVERY - Puppet freshness on sq83 is OK: puppet ran at Thu Aug 1 11:35:49 UTC 2013 [11:35:59] RECOVERY - Puppet freshness on sq65 is OK: puppet ran at Thu Aug 1 11:35:49 UTC 2013 [11:35:59] RECOVERY - Puppet freshness on mw34 is OK: puppet ran at Thu Aug 1 11:35:54 UTC 2013 [11:35:59] RECOVERY - Puppet freshness on es1003 is OK: puppet ran at Thu Aug 1 11:35:54 UTC 2013 [11:35:59] RECOVERY - Puppet freshness on cp1020 is OK: puppet ran at Thu Aug 1 11:35:54 UTC 2013 [11:35:59] RECOVERY - Puppet freshness on mw1017 is OK: puppet ran at Thu Aug 1 11:35:54 UTC 2013 [11:36:00] RECOVERY - Puppet freshness on search1019 is OK: puppet ran at Thu Aug 1 11:35:54 UTC 2013 [11:36:09] RECOVERY - Puppet freshness on db1023 is OK: puppet ran at Thu Aug 1 11:35:59 UTC 2013 [11:36:09] RECOVERY - Puppet freshness on es1005 is OK: puppet ran at Thu Aug 1 11:36:04 UTC 2013 [11:36:09] RECOVERY - Puppet freshness on cp1056 is OK: puppet ran at Thu Aug 1 11:36:04 UTC 2013 [11:36:23] (03CR) 10Mark Bergsma: [C: 032] Fix variable name [operations/puppet] - 10https://gerrit.wikimedia.org/r/77095 (owner: 10Mark Bergsma) [11:36:29] RECOVERY - Puppet freshness on search1008 is OK: puppet ran at Thu Aug 1 11:36:19 UTC 2013 [11:36:39] RECOVERY - Puppet freshness on analytics1012 is OK: puppet ran at Thu Aug 1 11:36:34 UTC 2013 [11:36:40] RECOVERY - Puppet freshness on dobson is OK: puppet ran at Thu Aug 1 11:36:34 UTC 2013 [11:36:49] RECOVERY - Puppet freshness on srv238 is OK: puppet ran at Thu Aug 1 11:36:39 UTC 2013 [11:36:49] RECOVERY - Puppet freshness on mw49 is OK: puppet ran at Thu Aug 1 11:36:39 UTC 2013 [11:36:49] RECOVERY - Puppet freshness on mw82 is OK: puppet ran at Thu Aug 1 11:36:39 UTC 2013 [11:36:49] RECOVERY - Puppet freshness on labsdb1002 is OK: puppet ran at Thu Aug 1 11:36:39 UTC 2013 [11:36:49] RECOVERY - Puppet freshness on srv284 is OK: puppet ran at Thu Aug 1 11:36:39 UTC 2013 [11:37:59] RECOVERY - Puppet freshness on mw1196 is OK: puppet ran at Thu Aug 1 11:37:50 UTC 2013 [11:37:59] RECOVERY - Puppet freshness on cp3010 is OK: puppet ran at Thu Aug 1 11:37:55 UTC 2013 [11:38:09] RECOVERY - Puppet freshness on cp1069 is OK: puppet ran at Thu Aug 1 11:38:05 UTC 2013 [11:39:09] RECOVERY - Puppet freshness on sq79 is OK: puppet ran at Thu Aug 1 11:39:00 UTC 2013 [11:39:09] RECOVERY - Puppet freshness on wtp1011 is OK: puppet ran at Thu Aug 1 11:39:00 UTC 2013 [11:39:09] RECOVERY - Puppet freshness on wtp1024 is OK: puppet ran at Thu Aug 1 11:39:00 UTC 2013 [11:39:09] RECOVERY - Puppet freshness on lvs5 is OK: puppet ran at Thu Aug 1 11:39:00 UTC 2013 [11:39:09] RECOVERY - Puppet freshness on db1036 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:09] RECOVERY - Puppet freshness on ms-be4 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:09] RECOVERY - Puppet freshness on mc1005 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:10] RECOVERY - Puppet freshness on db1048 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:10] RECOVERY - Puppet freshness on cp1006 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:11] RECOVERY - Puppet freshness on fenari is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:11] RECOVERY - Puppet freshness on mw1044 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:12] RECOVERY - Puppet freshness on mw38 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:12] RECOVERY - Puppet freshness on srv290 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:13] RECOVERY - Puppet freshness on mw118 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:13] RECOVERY - Puppet freshness on mw83 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:14] RECOVERY - Puppet freshness on ssl2 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:14] RECOVERY - Puppet freshness on analytics1015 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:15] RECOVERY - Puppet freshness on search36 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:15] RECOVERY - Puppet freshness on srv301 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:16] RECOVERY - Puppet freshness on cp1050 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:16] RECOVERY - Puppet freshness on kaulen is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:17] RECOVERY - Puppet freshness on mw1190 is OK: puppet ran at Thu Aug 1 11:39:05 UTC 2013 [11:39:29] RECOVERY - Puppet freshness on rubidium is OK: puppet ran at Thu Aug 1 11:39:25 UTC 2013 [11:39:29] RECOVERY - Puppet freshness on mw1161 is OK: puppet ran at Thu Aug 1 11:39:25 UTC 2013 [11:39:29] RECOVERY - Puppet freshness on wtp1003 is OK: puppet ran at Thu Aug 1 11:39:25 UTC 2013 [11:39:29] RECOVERY - Puppet freshness on mw1130 is OK: puppet ran at Thu Aug 1 11:39:25 UTC 2013 [11:39:29] RECOVERY - Puppet freshness on mw1147 is OK: puppet ran at Thu Aug 1 11:39:25 UTC 2013 [11:39:29] RECOVERY - Puppet freshness on mw1125 is OK: puppet ran at Thu Aug 1 11:39:25 UTC 2013 [11:39:30] RECOVERY - Puppet freshness on mw1111 is OK: puppet ran at Thu Aug 1 11:39:25 UTC 2013 [11:39:30] RECOVERY - Puppet freshness on mw1124 is OK: puppet ran at Thu Aug 1 11:39:25 UTC 2013 [11:39:31] RECOVERY - Puppet freshness on mw119 is OK: puppet ran at Thu Aug 1 11:39:25 UTC 2013 [11:39:31] RECOVERY - Puppet freshness on sq86 is OK: puppet ran at Thu Aug 1 11:39:25 UTC 2013 [11:39:39] RECOVERY - Puppet freshness on gadolinium is OK: puppet ran at Thu Aug 1 11:39:30 UTC 2013 [11:39:39] RECOVERY - Puppet freshness on ms-fe3001 is OK: puppet ran at Thu Aug 1 11:39:35 UTC 2013 [11:39:49] RECOVERY - Puppet freshness on vanadium is OK: puppet ran at Thu Aug 1 11:39:40 UTC 2013 [11:39:49] RECOVERY - Puppet freshness on mc1009 is OK: puppet ran at Thu Aug 1 11:39:40 UTC 2013 [11:39:49] RECOVERY - Puppet freshness on analytics1023 is OK: puppet ran at Thu Aug 1 11:39:45 UTC 2013 [11:39:49] RECOVERY - Puppet freshness on srv244 is OK: puppet ran at Thu Aug 1 11:39:45 UTC 2013 [11:39:49] RECOVERY - Puppet freshness on wtp1018 is OK: puppet ran at Thu Aug 1 11:39:45 UTC 2013 [11:39:59] RECOVERY - Puppet freshness on mw1160 is OK: puppet ran at Thu Aug 1 11:39:55 UTC 2013 [11:39:59] RECOVERY - Puppet freshness on ms-fe3 is OK: puppet ran at Thu Aug 1 11:39:55 UTC 2013 [11:39:59] RECOVERY - Puppet freshness on sq43 is OK: puppet ran at Thu Aug 1 11:39:55 UTC 2013 [11:39:59] RECOVERY - Puppet freshness on sq74 is OK: puppet ran at Thu Aug 1 11:39:55 UTC 2013 [11:39:59] RECOVERY - Puppet freshness on db45 is OK: puppet ran at Thu Aug 1 11:39:55 UTC 2013 [11:39:59] RECOVERY - Puppet freshness on srv246 is OK: puppet ran at Thu Aug 1 11:39:55 UTC 2013 [11:39:59] RECOVERY - Puppet freshness on solr3 is OK: puppet ran at Thu Aug 1 11:39:55 UTC 2013 [11:40:00] RECOVERY - Puppet freshness on mw1159 is OK: puppet ran at Thu Aug 1 11:39:55 UTC 2013 [11:40:00] RECOVERY - Puppet freshness on srv249 is OK: puppet ran at Thu Aug 1 11:39:55 UTC 2013 [11:40:01] RECOVERY - Puppet freshness on srv245 is OK: puppet ran at Thu Aug 1 11:39:55 UTC 2013 [11:40:19] RECOVERY - Puppet freshness on db1029 is OK: puppet ran at Thu Aug 1 11:40:10 UTC 2013 [11:40:19] RECOVERY - Puppet freshness on srv289 is OK: puppet ran at Thu Aug 1 11:40:10 UTC 2013 [11:40:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:40:40] RECOVERY - Puppet freshness on manganese is OK: puppet ran at Thu Aug 1 11:40:30 UTC 2013 [11:40:40] RECOVERY - Puppet freshness on mc1001 is OK: puppet ran at Thu Aug 1 11:40:30 UTC 2013 [11:40:40] RECOVERY - Puppet freshness on db43 is OK: puppet ran at Thu Aug 1 11:40:35 UTC 2013 [11:40:40] RECOVERY - Puppet freshness on cp1063 is OK: puppet ran at Thu Aug 1 11:40:35 UTC 2013 [11:40:41] RECOVERY - Puppet freshness on mw1081 is OK: puppet ran at Thu Aug 1 11:40:35 UTC 2013 [11:40:49] RECOVERY - Puppet freshness on mw1056 is OK: puppet ran at Thu Aug 1 11:40:40 UTC 2013 [11:40:49] RECOVERY - Puppet freshness on solr1003 is OK: puppet ran at Thu Aug 1 11:40:40 UTC 2013 [11:40:49] RECOVERY - Puppet freshness on amssq44 is OK: puppet ran at Thu Aug 1 11:40:40 UTC 2013 [11:40:49] RECOVERY - Puppet freshness on es6 is OK: puppet ran at Thu Aug 1 11:40:45 UTC 2013 [11:40:49] RECOVERY - Puppet freshness on mw1048 is OK: puppet ran at Thu Aug 1 11:40:45 UTC 2013 [11:40:49] RECOVERY - Puppet freshness on snapshot1 is OK: puppet ran at Thu Aug 1 11:40:45 UTC 2013 [11:40:49] RECOVERY - Puppet freshness on sq48 is OK: puppet ran at Thu Aug 1 11:40:45 UTC 2013 [11:40:50] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Thu Aug 1 11:40:45 UTC 2013 [11:40:50] RECOVERY - Puppet freshness on ms-be3 is OK: puppet ran at Thu Aug 1 11:40:45 UTC 2013 [11:41:09] RECOVERY - Puppet freshness on lvs1 is OK: puppet ran at Thu Aug 1 11:41:00 UTC 2013 [11:41:09] RECOVERY - Puppet freshness on titanium is OK: puppet ran at Thu Aug 1 11:41:00 UTC 2013 [11:41:09] RECOVERY - Puppet freshness on pc1 is OK: puppet ran at Thu Aug 1 11:41:00 UTC 2013 [11:41:09] RECOVERY - Puppet freshness on neon is OK: puppet ran at Thu Aug 1 11:41:00 UTC 2013 [11:41:09] RECOVERY - Puppet freshness on mw1197 is OK: puppet ran at Thu Aug 1 11:41:00 UTC 2013 [11:41:09] RECOVERY - Puppet freshness on mw58 is OK: puppet ran at Thu Aug 1 11:41:00 UTC 2013 [11:41:09] RECOVERY - Puppet freshness on sq85 is OK: puppet ran at Thu Aug 1 11:41:00 UTC 2013 [11:41:10] RECOVERY - Puppet freshness on mw1186 is OK: puppet ran at Thu Aug 1 11:41:00 UTC 2013 [11:41:10] RECOVERY - Puppet freshness on mw1087 is OK: puppet ran at Thu Aug 1 11:41:00 UTC 2013 [11:41:11] RECOVERY - Puppet freshness on db1004 is OK: puppet ran at Thu Aug 1 11:41:00 UTC 2013 [11:41:11] RECOVERY - Puppet freshness on mw30 is OK: puppet ran at Thu Aug 1 11:41:00 UTC 2013 [11:41:12] RECOVERY - Puppet freshness on mw1038 is OK: puppet ran at Thu Aug 1 11:41:00 UTC 2013 [11:41:12] RECOVERY - Puppet freshness on cp1049 is OK: puppet ran at Thu Aug 1 11:41:00 UTC 2013 [11:41:13] RECOVERY - Puppet freshness on mw8 is OK: puppet ran at Thu Aug 1 11:41:05 UTC 2013 [11:41:13] RECOVERY - Puppet freshness on srv292 is OK: puppet ran at Thu Aug 1 11:41:05 UTC 2013 [11:41:14] RECOVERY - Puppet freshness on solr2 is OK: puppet ran at Thu Aug 1 11:41:06 UTC 2013 [11:41:14] RECOVERY - Puppet freshness on lanthanum is OK: puppet ran at Thu Aug 1 11:41:06 UTC 2013 [11:41:15] RECOVERY - Puppet freshness on rdb1002 is OK: puppet ran at Thu Aug 1 11:41:06 UTC 2013 [11:41:15] RECOVERY - Puppet freshness on srv276 is OK: puppet ran at Thu Aug 1 11:41:06 UTC 2013 [11:41:16] RECOVERY - Puppet freshness on mw1007 is OK: puppet ran at Thu Aug 1 11:41:06 UTC 2013 [11:41:16] RECOVERY - Puppet freshness on ms-be2 is OK: puppet ran at Thu Aug 1 11:41:06 UTC 2013 [11:41:17] RECOVERY - Puppet freshness on srv294 is OK: puppet ran at Thu Aug 1 11:41:06 UTC 2013 [11:41:17] RECOVERY - Puppet freshness on sq76 is OK: puppet ran at Thu Aug 1 11:41:06 UTC 2013 [11:41:18] RECOVERY - Puppet freshness on db1020 is OK: puppet ran at Thu Aug 1 11:41:06 UTC 2013 [11:41:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.996 second response time [11:41:29] RECOVERY - Puppet freshness on db58 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:29] RECOVERY - Puppet freshness on cp3009 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:29] RECOVERY - Puppet freshness on db39 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:29] RECOVERY - Puppet freshness on cp3012 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:29] RECOVERY - Puppet freshness on stat1 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:29] RECOVERY - Puppet freshness on mw1063 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:30] RECOVERY - Puppet freshness on helium is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:30] RECOVERY - Puppet freshness on wtp1015 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:31] RECOVERY - Puppet freshness on mw1043 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:31] RECOVERY - Puppet freshness on mc1007 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:32] RECOVERY - Puppet freshness on mw66 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:32] RECOVERY - Puppet freshness on praseodymium is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:33] RECOVERY - Puppet freshness on db1031 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:33] RECOVERY - Puppet freshness on cp1002 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:34] RECOVERY - Puppet freshness on mw1065 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:34] RECOVERY - Puppet freshness on search18 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:35] RECOVERY - Puppet freshness on srv255 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:35] RECOVERY - Puppet freshness on mw121 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:36] RECOVERY - Puppet freshness on ms-be9 is OK: puppet ran at Thu Aug 1 11:41:26 UTC 2013 [11:41:39] RECOVERY - Puppet freshness on mw1004 is OK: puppet ran at Thu Aug 1 11:41:31 UTC 2013 [11:41:39] RECOVERY - Puppet freshness on potassium is OK: puppet ran at Thu Aug 1 11:41:31 UTC 2013 [11:41:39] RECOVERY - Puppet freshness on mw1140 is OK: puppet ran at Thu Aug 1 11:41:31 UTC 2013 [11:41:39] RECOVERY - Puppet freshness on ekrem is OK: puppet ran at Thu Aug 1 11:41:31 UTC 2013 [11:41:39] RECOVERY - Puppet freshness on wtp1004 is OK: puppet ran at Thu Aug 1 11:41:31 UTC 2013 [11:41:39] RECOVERY - Puppet freshness on mw1041 is OK: puppet ran at Thu Aug 1 11:41:31 UTC 2013 [11:41:39] RECOVERY - Puppet freshness on ms-be1006 is OK: puppet ran at Thu Aug 1 11:41:31 UTC 2013 [11:41:40] RECOVERY - Puppet freshness on sq54 is OK: puppet ran at Thu Aug 1 11:41:31 UTC 2013 [11:41:40] RECOVERY - Puppet freshness on db1001 is OK: puppet ran at Thu Aug 1 11:41:31 UTC 2013 [11:41:41] RECOVERY - Puppet freshness on ms10 is OK: puppet ran at Thu Aug 1 11:41:31 UTC 2013 [11:41:41] RECOVERY - Puppet freshness on search19 is OK: puppet ran at Thu Aug 1 11:41:31 UTC 2013 [11:41:42] RECOVERY - Puppet freshness on sq58 is OK: puppet ran at Thu Aug 1 11:41:31 UTC 2013 [11:41:42] RECOVERY - Puppet freshness on db1044 is OK: puppet ran at Thu Aug 1 11:41:31 UTC 2013 [11:41:43] RECOVERY - Puppet freshness on mw76 is OK: puppet ran at Thu Aug 1 11:41:36 UTC 2013 [11:41:43] RECOVERY - Puppet freshness on mw1146 is OK: puppet ran at Thu Aug 1 11:41:36 UTC 2013 [11:41:44] RECOVERY - Puppet freshness on mw1115 is OK: puppet ran at Thu Aug 1 11:41:36 UTC 2013 [11:41:44] RECOVERY - Puppet freshness on srv273 is OK: puppet ran at Thu Aug 1 11:41:36 UTC 2013 [11:41:49] RECOVERY - Puppet freshness on mw1057 is OK: puppet ran at Thu Aug 1 11:41:41 UTC 2013 [11:41:49] RECOVERY - Puppet freshness on srv253 is OK: puppet ran at Thu Aug 1 11:41:41 UTC 2013 [11:41:49] RECOVERY - Puppet freshness on mw1188 is OK: puppet ran at Thu Aug 1 11:41:41 UTC 2013 [11:41:49] RECOVERY - Puppet freshness on mw1210 is OK: puppet ran at Thu Aug 1 11:41:41 UTC 2013 [11:41:49] RECOVERY - Puppet freshness on db34 is OK: puppet ran at Thu Aug 1 11:41:41 UTC 2013 [11:41:49] RECOVERY - Puppet freshness on ms-be1012 is OK: puppet ran at Thu Aug 1 11:41:46 UTC 2013 [11:41:49] RECOVERY - Puppet freshness on mw57 is OK: puppet ran at Thu Aug 1 11:41:46 UTC 2013 [11:41:50] RECOVERY - Puppet freshness on cp1005 is OK: puppet ran at Thu Aug 1 11:41:46 UTC 2013 [11:41:59] RECOVERY - Puppet freshness on rdb1003 is OK: puppet ran at Thu Aug 1 11:41:51 UTC 2013 [11:41:59] RECOVERY - Puppet freshness on db63 is OK: puppet ran at Thu Aug 1 11:41:51 UTC 2013 [11:41:59] RECOVERY - Puppet freshness on search1024 is OK: puppet ran at Thu Aug 1 11:41:51 UTC 2013 [11:41:59] RECOVERY - Puppet freshness on mw1032 is OK: puppet ran at Thu Aug 1 11:41:56 UTC 2013 [11:41:59] RECOVERY - Puppet freshness on wtp1007 is OK: puppet ran at Thu Aug 1 11:41:56 UTC 2013 [11:41:59] RECOVERY - Puppet freshness on mw62 is OK: puppet ran at Thu Aug 1 11:41:56 UTC 2013 [11:41:59] RECOVERY - Puppet freshness on mw1106 is OK: puppet ran at Thu Aug 1 11:41:56 UTC 2013 [11:42:09] RECOVERY - Puppet freshness on pdf1 is OK: puppet ran at Thu Aug 1 11:42:01 UTC 2013 [11:42:09] RECOVERY - Puppet freshness on tin is OK: puppet ran at Thu Aug 1 11:42:06 UTC 2013 [11:42:09] RECOVERY - Puppet freshness on ms-fe1001 is OK: puppet ran at Thu Aug 1 11:42:06 UTC 2013 [11:42:19] RECOVERY - Puppet freshness on mw43 is OK: puppet ran at Thu Aug 1 11:42:11 UTC 2013 [11:42:19] RECOVERY - Puppet freshness on srv193 is OK: puppet ran at Thu Aug 1 11:42:11 UTC 2013 [11:42:19] RECOVERY - Puppet freshness on mw35 is OK: puppet ran at Thu Aug 1 11:42:11 UTC 2013 [11:42:19] RECOVERY - Puppet freshness on analytics1014 is OK: puppet ran at Thu Aug 1 11:42:11 UTC 2013 [11:42:19] RECOVERY - Puppet freshness on mw106 is OK: puppet ran at Thu Aug 1 11:42:16 UTC 2013 [11:42:19] RECOVERY - Puppet freshness on db1022 is OK: puppet ran at Thu Aug 1 11:42:16 UTC 2013 [11:42:19] RECOVERY - Puppet freshness on cp1010 is OK: puppet ran at Thu Aug 1 11:42:16 UTC 2013 [11:42:20] RECOVERY - Puppet freshness on mw2 is OK: puppet ran at Thu Aug 1 11:42:16 UTC 2013 [11:42:40] RECOVERY - Puppet freshness on mw1150 is OK: puppet ran at Thu Aug 1 11:42:36 UTC 2013 [11:42:52] RECOVERY - Puppet freshness on calcium is OK: puppet ran at Thu Aug 1 11:42:41 UTC 2013 [11:42:52] RECOVERY - Puppet freshness on mw1046 is OK: puppet ran at Thu Aug 1 11:42:41 UTC 2013 [11:42:52] RECOVERY - Puppet freshness on mw1003 is OK: puppet ran at Thu Aug 1 11:42:41 UTC 2013 [11:42:52] RECOVERY - Puppet freshness on mw79 is OK: puppet ran at Thu Aug 1 11:42:41 UTC 2013 [11:42:52] RECOVERY - Puppet freshness on mw1205 is OK: puppet ran at Thu Aug 1 11:42:41 UTC 2013 [11:42:52] RECOVERY - Puppet freshness on sq63 is OK: puppet ran at Thu Aug 1 11:42:41 UTC 2013 [11:42:52] RECOVERY - Puppet freshness on mw1201 is OK: puppet ran at Thu Aug 1 11:42:41 UTC 2013 [11:42:53] RECOVERY - Puppet freshness on mw124 is OK: puppet ran at Thu Aug 1 11:42:41 UTC 2013 [11:42:53] RECOVERY - Puppet freshness on mw1069 is OK: puppet ran at Thu Aug 1 11:42:41 UTC 2013 [11:42:54] RECOVERY - Puppet freshness on mw1068 is OK: puppet ran at Thu Aug 1 11:42:41 UTC 2013 [11:42:54] RECOVERY - Puppet freshness on mw1122 is OK: puppet ran at Thu Aug 1 11:42:41 UTC 2013 [11:42:55] RECOVERY - Puppet freshness on db1040 is OK: puppet ran at Thu Aug 1 11:42:46 UTC 2013 [11:42:55] RECOVERY - Puppet freshness on mw1033 is OK: puppet ran at Thu Aug 1 11:42:46 UTC 2013 [11:42:56] RECOVERY - Puppet freshness on db1002 is OK: puppet ran at Thu Aug 1 11:42:46 UTC 2013 [11:42:56] RECOVERY - Puppet freshness on mw1153 is OK: puppet ran at Thu Aug 1 11:42:46 UTC 2013 [11:42:57] RECOVERY - Puppet freshness on db48 is OK: puppet ran at Thu Aug 1 11:42:46 UTC 2013 [11:42:57] RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Thu Aug 1 11:42:46 UTC 2013 [11:42:58] RECOVERY - Puppet freshness on search33 is OK: puppet ran at Thu Aug 1 11:42:46 UTC 2013 [11:42:58] RECOVERY - Puppet freshness on mw1189 is OK: puppet ran at Thu Aug 1 11:42:46 UTC 2013 [11:42:59] RECOVERY - Puppet freshness on db1033 is OK: puppet ran at Thu Aug 1 11:42:46 UTC 2013 [11:43:02] RECOVERY - Puppet freshness on mw98 is OK: puppet ran at Thu Aug 1 11:42:56 UTC 2013 [11:43:02] RECOVERY - Puppet freshness on ms-be11 is OK: puppet ran at Thu Aug 1 11:42:56 UTC 2013 [11:43:02] RECOVERY - Puppet freshness on cp1037 is OK: puppet ran at Thu Aug 1 11:42:56 UTC 2013 [11:43:02] RECOVERY - Puppet freshness on ssl1001 is OK: puppet ran at Thu Aug 1 11:43:01 UTC 2013 [11:43:12] RECOVERY - Puppet freshness on analytics1004 is OK: puppet ran at Thu Aug 1 11:43:06 UTC 2013 [11:43:12] RECOVERY - Puppet freshness on srv262 is OK: puppet ran at Thu Aug 1 11:43:06 UTC 2013 [11:43:12] RECOVERY - Puppet freshness on mw68 is OK: puppet ran at Thu Aug 1 11:43:06 UTC 2013 [11:43:23] RECOVERY - Puppet freshness on srv269 is OK: puppet ran at Thu Aug 1 11:43:11 UTC 2013 [11:43:23] RECOVERY - Puppet freshness on srv260 is OK: puppet ran at Thu Aug 1 11:43:11 UTC 2013 [11:43:23] RECOVERY - Puppet freshness on mw1166 is OK: puppet ran at Thu Aug 1 11:43:12 UTC 2013 [11:43:23] RECOVERY - Puppet freshness on srv267 is OK: puppet ran at Thu Aug 1 11:43:12 UTC 2013 [11:43:23] RECOVERY - Puppet freshness on mw104 is OK: puppet ran at Thu Aug 1 11:43:17 UTC 2013 [11:43:42] RECOVERY - Puppet freshness on search25 is OK: puppet ran at Thu Aug 1 11:43:37 UTC 2013 [11:43:42] RECOVERY - Puppet freshness on search16 is OK: puppet ran at Thu Aug 1 11:43:37 UTC 2013 [11:43:42] RECOVERY - Puppet freshness on virt11 is OK: puppet ran at Thu Aug 1 11:43:37 UTC 2013 [11:43:52] RECOVERY - Puppet freshness on mw37 is OK: puppet ran at Thu Aug 1 11:43:42 UTC 2013 [11:43:52] RECOVERY - Puppet freshness on mw1010 is OK: puppet ran at Thu Aug 1 11:43:47 UTC 2013 [11:43:52] RECOVERY - Puppet freshness on mw1142 is OK: puppet ran at Thu Aug 1 11:43:47 UTC 2013 [11:43:52] RECOVERY - Puppet freshness on mw1118 is OK: puppet ran at Thu Aug 1 11:43:47 UTC 2013 [11:43:52] RECOVERY - Puppet freshness on search1022 is OK: puppet ran at Thu Aug 1 11:43:47 UTC 2013 [11:44:02] RECOVERY - Puppet freshness on solr1002 is OK: puppet ran at Thu Aug 1 11:43:52 UTC 2013 [11:44:02] RECOVERY - Puppet freshness on fluorine is OK: puppet ran at Thu Aug 1 11:43:52 UTC 2013 [11:44:02] RECOVERY - Puppet freshness on sq51 is OK: puppet ran at Thu Aug 1 11:43:57 UTC 2013 [11:44:02] RECOVERY - Puppet freshness on mw133 is OK: puppet ran at Thu Aug 1 11:43:57 UTC 2013 [11:44:02] RECOVERY - Puppet freshness on mw1092 is OK: puppet ran at Thu Aug 1 11:43:57 UTC 2013 [11:44:02] RECOVERY - Puppet freshness on mw1025 is OK: puppet ran at Thu Aug 1 11:43:57 UTC 2013 [11:44:02] RECOVERY - Puppet freshness on search1001 is OK: puppet ran at Thu Aug 1 11:43:57 UTC 2013 [11:44:03] RECOVERY - Puppet freshness on db54 is OK: puppet ran at Thu Aug 1 11:43:57 UTC 2013 [11:44:03] RECOVERY - Puppet freshness on db1034 is OK: puppet ran at Thu Aug 1 11:43:57 UTC 2013 [11:44:04] RECOVERY - Puppet freshness on mw1091 is OK: puppet ran at Thu Aug 1 11:43:57 UTC 2013 [11:44:04] RECOVERY - Puppet freshness on db1051 is OK: puppet ran at Thu Aug 1 11:43:57 UTC 2013 [11:44:05] RECOVERY - Puppet freshness on db1042 is OK: puppet ran at Thu Aug 1 11:43:57 UTC 2013 [11:44:05] RECOVERY - Puppet freshness on db66 is OK: puppet ran at Thu Aug 1 11:43:57 UTC 2013 [11:44:06] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Thu Aug 1 11:43:57 UTC 2013 [11:44:06] RECOVERY - Puppet freshness on mw116 is OK: puppet ran at Thu Aug 1 11:43:57 UTC 2013 [11:44:07] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Thu Aug 1 11:43:57 UTC 2013 [11:44:07] RECOVERY - Puppet freshness on sockpuppet is OK: puppet ran at Thu Aug 1 11:43:57 UTC 2013 [11:44:12] RECOVERY - Puppet freshness on mw131 is OK: puppet ran at Thu Aug 1 11:44:02 UTC 2013 [11:44:12] RECOVERY - Puppet freshness on mw10 is OK: puppet ran at Thu Aug 1 11:44:02 UTC 2013 [11:44:12] RECOVERY - Puppet freshness on search29 is OK: puppet ran at Thu Aug 1 11:44:02 UTC 2013 [11:44:12] RECOVERY - Puppet freshness on sq42 is OK: puppet ran at Thu Aug 1 11:44:02 UTC 2013 [11:44:12] RECOVERY - Puppet freshness on analytics1003 is OK: puppet ran at Thu Aug 1 11:44:07 UTC 2013 [11:44:12] RECOVERY - Puppet freshness on mw1211 is OK: puppet ran at Thu Aug 1 11:44:07 UTC 2013 [11:44:12] RECOVERY - Puppet freshness on mw1011 is OK: puppet ran at Thu Aug 1 11:44:07 UTC 2013 [11:44:13] RECOVERY - Puppet freshness on mw1213 is OK: puppet ran at Thu Aug 1 11:44:07 UTC 2013 [11:44:13] RECOVERY - Puppet freshness on mw1204 is OK: puppet ran at Thu Aug 1 11:44:07 UTC 2013 [11:44:22] RECOVERY - Puppet freshness on mw1143 is OK: puppet ran at Thu Aug 1 11:44:12 UTC 2013 [11:44:22] RECOVERY - Puppet freshness on search1007 is OK: puppet ran at Thu Aug 1 11:44:12 UTC 2013 [11:44:22] RECOVERY - Puppet freshness on mw1129 is OK: puppet ran at Thu Aug 1 11:44:12 UTC 2013 [11:44:22] RECOVERY - Puppet freshness on mw60 is OK: puppet ran at Thu Aug 1 11:44:12 UTC 2013 [11:44:22] RECOVERY - Puppet freshness on mw1027 is OK: puppet ran at Thu Aug 1 11:44:17 UTC 2013 [11:44:22] RECOVERY - Puppet freshness on mw23 is OK: puppet ran at Thu Aug 1 11:44:17 UTC 2013 [11:44:22] RECOVERY - Puppet freshness on mw28 is OK: puppet ran at Thu Aug 1 11:44:17 UTC 2013 [11:44:32] RECOVERY - Puppet freshness on search30 is OK: puppet ran at Thu Aug 1 11:44:22 UTC 2013 [11:44:32] RECOVERY - Puppet freshness on sq52 is OK: puppet ran at Thu Aug 1 11:44:27 UTC 2013 [11:44:32] RECOVERY - Puppet freshness on mw1107 is OK: puppet ran at Thu Aug 1 11:44:27 UTC 2013 [11:44:32] RECOVERY - Puppet freshness on search1012 is OK: puppet ran at Thu Aug 1 11:44:27 UTC 2013 [11:44:42] RECOVERY - Puppet freshness on mw1066 is OK: puppet ran at Thu Aug 1 11:44:32 UTC 2013 [11:44:42] RECOVERY - Puppet freshness on labstore1001 is OK: puppet ran at Thu Aug 1 11:44:32 UTC 2013 [11:44:42] RECOVERY - Puppet freshness on labstore3 is OK: puppet ran at Thu Aug 1 11:44:37 UTC 2013 [11:44:42] RECOVERY - Puppet freshness on labstore1 is OK: puppet ran at Thu Aug 1 11:44:37 UTC 2013 [11:44:52] RECOVERY - Puppet freshness on ms-fe1002 is OK: puppet ran at Thu Aug 1 11:44:47 UTC 2013 [11:44:52] RECOVERY - Puppet freshness on dataset1001 is OK: puppet ran at Thu Aug 1 11:44:47 UTC 2013 [11:44:52] RECOVERY - Puppet freshness on amslvs1 is OK: puppet ran at Thu Aug 1 11:44:47 UTC 2013 [11:45:02] RECOVERY - Puppet freshness on cp1058 is OK: puppet ran at Thu Aug 1 11:44:52 UTC 2013 [11:45:02] RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Thu Aug 1 11:44:57 UTC 2013 [11:45:02] RECOVERY - Puppet freshness on srv242 is OK: puppet ran at Thu Aug 1 11:44:57 UTC 2013 [11:45:12] RECOVERY - Puppet freshness on cp3011 is OK: puppet ran at Thu Aug 1 11:45:02 UTC 2013 [11:45:12] RECOVERY - Puppet freshness on amslvs4 is OK: puppet ran at Thu Aug 1 11:45:02 UTC 2013 [11:45:12] RECOVERY - Puppet freshness on ms-be12 is OK: puppet ran at Thu Aug 1 11:45:02 UTC 2013 [11:45:12] RECOVERY - Puppet freshness on mw1021 is OK: puppet ran at Thu Aug 1 11:45:07 UTC 2013 [11:45:12] RECOVERY - Puppet freshness on solr1 is OK: puppet ran at Thu Aug 1 11:45:07 UTC 2013 [11:45:12] RECOVERY - Puppet freshness on mc1012 is OK: puppet ran at Thu Aug 1 11:45:07 UTC 2013 [11:45:12] RECOVERY - Puppet freshness on mw1154 is OK: puppet ran at Thu Aug 1 11:45:07 UTC 2013 [11:45:22] RECOVERY - Puppet freshness on db1014 is OK: puppet ran at Thu Aug 1 11:45:12 UTC 2013 [11:45:22] RECOVERY - Puppet freshness on ssl1 is OK: puppet ran at Thu Aug 1 11:45:12 UTC 2013 [11:45:52] RECOVERY - Puppet freshness on wtp1001 is OK: puppet ran at Thu Aug 1 11:45:42 UTC 2013 [11:46:02] RECOVERY - Puppet freshness on antimony is OK: puppet ran at Thu Aug 1 11:45:52 UTC 2013 [11:46:02] RECOVERY - Puppet freshness on mw42 is OK: puppet ran at Thu Aug 1 11:45:52 UTC 2013 [11:46:12] RECOVERY - Puppet freshness on mw55 is OK: puppet ran at Thu Aug 1 11:46:02 UTC 2013 [11:46:12] RECOVERY - Puppet freshness on mw75 is OK: puppet ran at Thu Aug 1 11:46:07 UTC 2013 [11:46:22] RECOVERY - Puppet freshness on mw1171 is OK: puppet ran at Thu Aug 1 11:46:12 UTC 2013 [11:46:22] RECOVERY - Puppet freshness on mw1206 is OK: puppet ran at Thu Aug 1 11:46:17 UTC 2013 [11:46:22] RECOVERY - Puppet freshness on db9 is OK: puppet ran at Thu Aug 1 11:46:17 UTC 2013 [11:46:32] RECOVERY - Puppet freshness on mw1208 is OK: puppet ran at Thu Aug 1 11:46:22 UTC 2013 [11:46:32] RECOVERY - Puppet freshness on mw1104 is OK: puppet ran at Thu Aug 1 11:46:22 UTC 2013 [11:46:32] RECOVERY - Puppet freshness on mw1024 is OK: puppet ran at Thu Aug 1 11:46:27 UTC 2013 [11:46:32] RECOVERY - Puppet freshness on mw112 is OK: puppet ran at Thu Aug 1 11:46:27 UTC 2013 [11:46:32] RECOVERY - Puppet freshness on cp1017 is OK: puppet ran at Thu Aug 1 11:46:27 UTC 2013 [11:46:32] RECOVERY - Puppet freshness on bast1001 is OK: puppet ran at Thu Aug 1 11:46:27 UTC 2013 [11:46:32] RECOVERY - Puppet freshness on sq84 is OK: puppet ran at Thu Aug 1 11:46:27 UTC 2013 [11:46:33] RECOVERY - Puppet freshness on mw93 is OK: puppet ran at Thu Aug 1 11:46:27 UTC 2013 [11:46:33] RECOVERY - Puppet freshness on mw1131 is OK: puppet ran at Thu Aug 1 11:46:27 UTC 2013 [11:46:34] RECOVERY - Puppet freshness on mw1155 is OK: puppet ran at Thu Aug 1 11:46:27 UTC 2013 [11:46:34] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Thu Aug 1 11:46:28 UTC 2013 [11:46:35] RECOVERY - Puppet freshness on capella is OK: puppet ran at Thu Aug 1 11:46:28 UTC 2013 [11:46:35] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Thu Aug 1 11:46:28 UTC 2013 [11:46:36] RECOVERY - Puppet freshness on ms6 is OK: puppet ran at Thu Aug 1 11:46:28 UTC 2013 [11:46:36] RECOVERY - Puppet freshness on sodium is OK: puppet ran at Thu Aug 1 11:46:28 UTC 2013 [11:46:37] RECOVERY - Puppet freshness on cp1018 is OK: puppet ran at Thu Aug 1 11:46:28 UTC 2013 [11:46:42] RECOVERY - Puppet freshness on es1004 is OK: puppet ran at Thu Aug 1 11:46:38 UTC 2013 [11:46:42] RECOVERY - Puppet freshness on ms-be5 is OK: puppet ran at Thu Aug 1 11:46:38 UTC 2013 [11:46:52] RECOVERY - Puppet freshness on db1026 is OK: puppet ran at Thu Aug 1 11:46:43 UTC 2013 [11:46:52] RECOVERY - Puppet freshness on cp1066 is OK: puppet ran at Thu Aug 1 11:46:43 UTC 2013 [11:46:52] RECOVERY - Puppet freshness on db1027 is OK: puppet ran at Thu Aug 1 11:46:43 UTC 2013 [11:46:52] RECOVERY - Puppet freshness on srv283 is OK: puppet ran at Thu Aug 1 11:46:48 UTC 2013 [11:47:02] RECOVERY - Puppet freshness on ssl1005 is OK: puppet ran at Thu Aug 1 11:46:58 UTC 2013 [11:47:02] RECOVERY - Puppet freshness on analytics1002 is OK: puppet ran at Thu Aug 1 11:46:58 UTC 2013 [11:47:02] RECOVERY - Puppet freshness on cp1060 is OK: puppet ran at Thu Aug 1 11:46:58 UTC 2013 [11:47:12] RECOVERY - Puppet freshness on mw74 is OK: puppet ran at Thu Aug 1 11:47:03 UTC 2013 [11:47:12] RECOVERY - Puppet freshness on sq62 is OK: puppet ran at Thu Aug 1 11:47:03 UTC 2013 [11:47:12] RECOVERY - Puppet freshness on ssl4 is OK: puppet ran at Thu Aug 1 11:47:03 UTC 2013 [11:47:12] RECOVERY - Puppet freshness on ssl1003 is OK: puppet ran at Thu Aug 1 11:47:08 UTC 2013 [11:47:12] RECOVERY - Puppet freshness on mw61 is OK: puppet ran at Thu Aug 1 11:47:08 UTC 2013 [11:47:12] RECOVERY - Puppet freshness on analytics1022 is OK: puppet ran at Thu Aug 1 11:47:08 UTC 2013 [11:47:22] RECOVERY - Puppet freshness on search1005 is OK: puppet ran at Thu Aug 1 11:47:13 UTC 2013 [11:47:22] RECOVERY - Puppet freshness on snapshot1002 is OK: puppet ran at Thu Aug 1 11:47:13 UTC 2013 [11:47:22] RECOVERY - Puppet freshness on es9 is OK: puppet ran at Thu Aug 1 11:47:13 UTC 2013 [11:47:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:42] RECOVERY - Puppet freshness on mw13 is OK: puppet ran at Thu Aug 1 11:47:38 UTC 2013 [11:47:52] RECOVERY - Puppet freshness on mw1180 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:52] RECOVERY - Puppet freshness on mw1128 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:52] RECOVERY - Puppet freshness on mw1049 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:52] RECOVERY - Puppet freshness on mw1194 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:52] RECOVERY - Puppet freshness on search1003 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:52] RECOVERY - Puppet freshness on nickel is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:52] RECOVERY - Puppet freshness on srv299 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:53] RECOVERY - Puppet freshness on srv291 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:53] RECOVERY - Puppet freshness on mw1084 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:54] RECOVERY - Puppet freshness on mw1047 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:54] RECOVERY - Puppet freshness on tmh1001 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:55] RECOVERY - Puppet freshness on mw1105 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:55] RECOVERY - Puppet freshness on analytics1009 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:56] RECOVERY - Puppet freshness on hume is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:56] RECOVERY - Puppet freshness on search1015 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:57] RECOVERY - Puppet freshness on amssq35 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:57] RECOVERY - Puppet freshness on db77 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:58] RECOVERY - Puppet freshness on srv261 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:47:58] RECOVERY - Puppet freshness on mw1137 is OK: puppet ran at Thu Aug 1 11:47:48 UTC 2013 [11:48:02] RECOVERY - Puppet freshness on db69 is OK: puppet ran at Thu Aug 1 11:47:53 UTC 2013 [11:48:02] RECOVERY - Puppet freshness on search31 is OK: puppet ran at Thu Aug 1 11:47:53 UTC 2013 [11:48:02] RECOVERY - Puppet freshness on cp1016 is OK: puppet ran at Thu Aug 1 11:47:53 UTC 2013 [11:48:02] RECOVERY - Puppet freshness on mw1054 is OK: puppet ran at Thu Aug 1 11:47:58 UTC 2013 [11:48:02] RECOVERY - Puppet freshness on sanger is OK: puppet ran at Thu Aug 1 11:47:58 UTC 2013 [11:48:12] RECOVERY - Puppet freshness on ms-be1009 is OK: puppet ran at Thu Aug 1 11:48:03 UTC 2013 [11:48:12] RECOVERY - Puppet freshness on ms-fe2 is OK: puppet ran at Thu Aug 1 11:48:03 UTC 2013 [11:48:12] RECOVERY - Puppet freshness on solr1001 is OK: puppet ran at Thu Aug 1 11:48:03 UTC 2013 [11:48:12] RECOVERY - Puppet freshness on labstore2 is OK: puppet ran at Thu Aug 1 11:48:03 UTC 2013 [11:48:12] RECOVERY - Puppet freshness on db65 is OK: puppet ran at Thu Aug 1 11:48:03 UTC 2013 [11:48:12] RECOVERY - Puppet freshness on mw1030 is OK: puppet ran at Thu Aug 1 11:48:03 UTC 2013 [11:48:12] RECOVERY - Puppet freshness on ms-be1007 is OK: puppet ran at Thu Aug 1 11:48:03 UTC 2013 [11:48:13] RECOVERY - Puppet freshness on mw1020 is OK: puppet ran at Thu Aug 1 11:48:03 UTC 2013 [11:48:13] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Thu Aug 1 11:48:03 UTC 2013 [11:48:14] RECOVERY - Puppet freshness on db32 is OK: puppet ran at Thu Aug 1 11:48:03 UTC 2013 [11:48:14] RECOVERY - Puppet freshness on db67 is OK: puppet ran at Thu Aug 1 11:48:09 UTC 2013 [11:48:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.139 second response time [11:48:22] RECOVERY - Puppet freshness on wtp1002 is OK: puppet ran at Thu Aug 1 11:48:14 UTC 2013 [11:48:22] RECOVERY - Puppet freshness on cp1062 is OK: puppet ran at Thu Aug 1 11:48:14 UTC 2013 [11:48:22] RECOVERY - Puppet freshness on cp1038 is OK: puppet ran at Thu Aug 1 11:48:14 UTC 2013 [11:48:32] RECOVERY - Puppet freshness on mw95 is OK: puppet ran at Thu Aug 1 11:48:24 UTC 2013 [11:48:32] RECOVERY - Puppet freshness on srv264 is OK: puppet ran at Thu Aug 1 11:48:24 UTC 2013 [11:48:32] RECOVERY - Puppet freshness on srv272 is OK: puppet ran at Thu Aug 1 11:48:24 UTC 2013 [11:48:32] RECOVERY - Puppet freshness on amssq31 is OK: puppet ran at Thu Aug 1 11:48:24 UTC 2013 [11:48:32] RECOVERY - Puppet freshness on mw99 is OK: puppet ran at Thu Aug 1 11:48:29 UTC 2013 [11:48:32] RECOVERY - Puppet freshness on amslvs2 is OK: puppet ran at Thu Aug 1 11:48:29 UTC 2013 [11:48:32] RECOVERY - Puppet freshness on amssq37 is OK: puppet ran at Thu Aug 1 11:48:29 UTC 2013 [11:48:33] RECOVERY - Puppet freshness on mw1165 is OK: puppet ran at Thu Aug 1 11:48:29 UTC 2013 [11:48:33] RECOVERY - Puppet freshness on mw45 is OK: puppet ran at Thu Aug 1 11:48:29 UTC 2013 [11:48:42] RECOVERY - Puppet freshness on analytics1008 is OK: puppet ran at Thu Aug 1 11:48:34 UTC 2013 [11:48:42] RECOVERY - Puppet freshness on erbium is OK: puppet ran at Thu Aug 1 11:48:34 UTC 2013 [11:48:42] RECOVERY - Puppet freshness on mw39 is OK: puppet ran at Thu Aug 1 11:48:34 UTC 2013 [11:48:42] RECOVERY - Puppet freshness on mw1198 is OK: puppet ran at Thu Aug 1 11:48:34 UTC 2013 [11:48:42] RECOVERY - Puppet freshness on db57 is OK: puppet ran at Thu Aug 1 11:48:34 UTC 2013 [11:48:42] RECOVERY - Puppet freshness on mw1169 is OK: puppet ran at Thu Aug 1 11:48:34 UTC 2013 [11:48:42] RECOVERY - Puppet freshness on mw1094 is OK: puppet ran at Thu Aug 1 11:48:34 UTC 2013 [11:48:43] RECOVERY - Puppet freshness on sq50 is OK: puppet ran at Thu Aug 1 11:48:34 UTC 2013 [11:48:43] RECOVERY - Puppet freshness on db1024 is OK: puppet ran at Thu Aug 1 11:48:34 UTC 2013 [11:48:44] RECOVERY - Puppet freshness on mw1053 is OK: puppet ran at Thu Aug 1 11:48:39 UTC 2013 [11:48:44] RECOVERY - Puppet freshness on mw1179 is OK: puppet ran at Thu Aug 1 11:48:39 UTC 2013 [11:48:45] RECOVERY - Puppet freshness on mw1075 is OK: puppet ran at Thu Aug 1 11:48:39 UTC 2013 [11:48:45] RECOVERY - Puppet freshness on mw1083 is OK: puppet ran at Thu Aug 1 11:48:39 UTC 2013 [11:48:46] RECOVERY - Puppet freshness on search1021 is OK: puppet ran at Thu Aug 1 11:48:39 UTC 2013 [11:48:52] RECOVERY - Puppet freshness on dataset2 is OK: puppet ran at Thu Aug 1 11:48:44 UTC 2013 [11:48:52] RECOVERY - Puppet freshness on db51 is OK: puppet ran at Thu Aug 1 11:48:44 UTC 2013 [11:48:52] RECOVERY - Puppet freshness on search1023 is OK: puppet ran at Thu Aug 1 11:48:44 UTC 2013 [11:48:52] RECOVERY - Puppet freshness on mw1078 is OK: puppet ran at Thu Aug 1 11:48:44 UTC 2013 [11:48:52] RECOVERY - Puppet freshness on mc1008 is OK: puppet ran at Thu Aug 1 11:48:49 UTC 2013 [11:48:52] RECOVERY - Puppet freshness on wtp1013 is OK: puppet ran at Thu Aug 1 11:48:49 UTC 2013 [11:48:52] RECOVERY - Puppet freshness on db1058 is OK: puppet ran at Thu Aug 1 11:48:49 UTC 2013 [11:49:02] RECOVERY - Puppet freshness on brewster is OK: puppet ran at Thu Aug 1 11:48:54 UTC 2013 [11:49:02] RECOVERY - Puppet freshness on mw117 is OK: puppet ran at Thu Aug 1 11:48:54 UTC 2013 [11:49:02] RECOVERY - Puppet freshness on mw26 is OK: puppet ran at Thu Aug 1 11:49:00 UTC 2013 [11:49:02] RECOVERY - Puppet freshness on cp1065 is OK: puppet ran at Thu Aug 1 11:49:00 UTC 2013 [11:49:02] RECOVERY - Puppet freshness on mw1116 is OK: puppet ran at Thu Aug 1 11:49:00 UTC 2013 [11:49:02] RECOVERY - Puppet freshness on mw33 is OK: puppet ran at Thu Aug 1 11:49:00 UTC 2013 [11:49:02] RECOVERY - Puppet freshness on lvs1002 is OK: puppet ran at Thu Aug 1 11:49:00 UTC 2013 [11:49:03] RECOVERY - Puppet freshness on es5 is OK: puppet ran at Thu Aug 1 11:49:00 UTC 2013 [11:49:03] RECOVERY - Puppet freshness on mw36 is OK: puppet ran at Thu Aug 1 11:49:00 UTC 2013 [11:49:12] RECOVERY - Puppet freshness on mw64 is OK: puppet ran at Thu Aug 1 11:49:05 UTC 2013 [11:49:12] RECOVERY - Puppet freshness on mw1136 is OK: puppet ran at Thu Aug 1 11:49:05 UTC 2013 [11:49:12] (03PS1) 10Mark Bergsma: Rename ::interface to ::interface::ip [operations/puppet] - 10https://gerrit.wikimedia.org/r/77097 [11:49:12] RECOVERY - Puppet freshness on analytics1011 is OK: puppet ran at Thu Aug 1 11:49:10 UTC 2013 [11:49:12] RECOVERY - Puppet freshness on mw1074 is OK: puppet ran at Thu Aug 1 11:49:10 UTC 2013 [11:49:12] RECOVERY - Puppet freshness on analytics1019 is OK: puppet ran at Thu Aug 1 11:49:10 UTC 2013 [11:49:22] RECOVERY - Puppet freshness on mw1138 is OK: puppet ran at Thu Aug 1 11:49:20 UTC 2013 [11:49:32] RECOVERY - Puppet freshness on tarin is OK: puppet ran at Thu Aug 1 11:49:30 UTC 2013 [11:49:42] RECOVERY - Puppet freshness on db46 is OK: puppet ran at Thu Aug 1 11:49:35 UTC 2013 [11:49:42] RECOVERY - Puppet freshness on mw134 is OK: puppet ran at Thu Aug 1 11:49:35 UTC 2013 [11:49:42] RECOVERY - Puppet freshness on db55 is OK: puppet ran at Thu Aug 1 11:49:40 UTC 2013 [11:49:52] RECOVERY - Puppet freshness on labsdb1001 is OK: puppet ran at Thu Aug 1 11:49:45 UTC 2013 [11:49:52] RECOVERY - Puppet freshness on snapshot4 is OK: puppet ran at Thu Aug 1 11:49:50 UTC 2013 [11:49:52] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Thu Aug 1 11:49:50 UTC 2013 [11:49:52] RECOVERY - Puppet freshness on es1009 is OK: puppet ran at Thu Aug 1 11:49:50 UTC 2013 [11:49:52] RECOVERY - Puppet freshness on wtp1021 is OK: puppet ran at Thu Aug 1 11:49:50 UTC 2013 [11:49:58] (03CR) 10Mark Bergsma: [C: 032] Rename ::interface to ::interface::ip [operations/puppet] - 10https://gerrit.wikimedia.org/r/77097 (owner: 10Mark Bergsma) [11:50:02] RECOVERY - Puppet freshness on mw70 is OK: puppet ran at Thu Aug 1 11:49:55 UTC 2013 [11:50:02] RECOVERY - Puppet freshness on mw107 is OK: puppet ran at Thu Aug 1 11:49:55 UTC 2013 [11:50:02] RECOVERY - Puppet freshness on mw109 is OK: puppet ran at Thu Aug 1 11:49:55 UTC 2013 [11:50:02] RECOVERY - Puppet freshness on mw53 is OK: puppet ran at Thu Aug 1 11:49:55 UTC 2013 [11:50:03] RECOVERY - Puppet freshness on srv296 is OK: puppet ran at Thu Aug 1 11:50:00 UTC 2013 [11:50:12] RECOVERY - Puppet freshness on mw40 is OK: puppet ran at Thu Aug 1 11:50:02 UTC 2013 [11:50:12] RECOVERY - Puppet freshness on stat1002 is OK: puppet ran at Thu Aug 1 11:50:07 UTC 2013 [11:50:12] RECOVERY - Puppet freshness on mw1001 is OK: puppet ran at Thu Aug 1 11:50:07 UTC 2013 [11:50:12] RECOVERY - Puppet freshness on mw1132 is OK: puppet ran at Thu Aug 1 11:50:07 UTC 2013 [11:50:12] RECOVERY - Puppet freshness on mw1218 is OK: puppet ran at Thu Aug 1 11:50:07 UTC 2013 [11:50:12] RECOVERY - Puppet freshness on mw1022 is OK: puppet ran at Thu Aug 1 11:50:07 UTC 2013 [11:50:12] RECOVERY - Puppet freshness on pdf3 is OK: puppet ran at Thu Aug 1 11:50:07 UTC 2013 [11:50:13] RECOVERY - Puppet freshness on mw1185 is OK: puppet ran at Thu Aug 1 11:50:07 UTC 2013 [11:50:42] RECOVERY - Puppet freshness on sq68 is OK: puppet ran at Thu Aug 1 11:50:38 UTC 2013 [11:50:42] RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Thu Aug 1 11:50:38 UTC 2013 [11:50:42] RECOVERY - Puppet freshness on amssq40 is OK: puppet ran at Thu Aug 1 11:50:38 UTC 2013 [11:50:42] RECOVERY - Puppet freshness on mw1040 is OK: puppet ran at Thu Aug 1 11:50:37 UTC 2013 [11:50:42] RECOVERY - Puppet freshness on mw1062 is OK: puppet ran at Thu Aug 1 11:50:38 UTC 2013 [11:50:52] RECOVERY - Puppet freshness on mc1011 is OK: puppet ran at Thu Aug 1 11:50:43 UTC 2013 [11:50:52] RECOVERY - Puppet freshness on db1005 is OK: puppet ran at Thu Aug 1 11:50:43 UTC 2013 [11:50:52] RECOVERY - Puppet freshness on mw1181 is OK: puppet ran at Thu Aug 1 11:50:48 UTC 2013 [11:50:52] RECOVERY - Puppet freshness on ms-fe1003 is OK: puppet ran at Thu Aug 1 11:50:48 UTC 2013 [11:50:52] RECOVERY - Puppet freshness on ssl1006 is OK: puppet ran at Thu Aug 1 11:50:48 UTC 2013 [11:50:52] RECOVERY - Puppet freshness on analytics1001 is OK: puppet ran at Thu Aug 1 11:50:48 UTC 2013 [11:50:52] RECOVERY - Puppet freshness on mw87 is OK: puppet ran at Thu Aug 1 11:50:48 UTC 2013 [11:51:02] RECOVERY - Puppet freshness on mw111 is OK: puppet ran at Thu Aug 1 11:50:53 UTC 2013 [11:51:02] RECOVERY - Puppet freshness on srv285 is OK: puppet ran at Thu Aug 1 11:50:53 UTC 2013 [11:51:02] RECOVERY - Puppet freshness on cp3006 is OK: puppet ran at Thu Aug 1 11:50:53 UTC 2013 [11:51:02] RECOVERY - Puppet freshness on search27 is OK: puppet ran at Thu Aug 1 11:50:53 UTC 2013 [11:51:02] RECOVERY - Puppet freshness on professor is OK: puppet ran at Thu Aug 1 11:50:53 UTC 2013 [11:51:02] RECOVERY - Puppet freshness on ms-be10 is OK: puppet ran at Thu Aug 1 11:50:53 UTC 2013 [11:51:02] RECOVERY - Puppet freshness on mw1219 is OK: puppet ran at Thu Aug 1 11:50:58 UTC 2013 [11:51:12] RECOVERY - Puppet freshness on mw1178 is OK: puppet ran at Thu Aug 1 11:51:03 UTC 2013 [11:51:12] RECOVERY - Puppet freshness on search1006 is OK: puppet ran at Thu Aug 1 11:51:03 UTC 2013 [11:51:12] RECOVERY - Puppet freshness on mw1080 is OK: puppet ran at Thu Aug 1 11:51:03 UTC 2013 [11:51:12] RECOVERY - Puppet freshness on mw1072 is OK: puppet ran at Thu Aug 1 11:51:03 UTC 2013 [11:51:12] RECOVERY - Puppet freshness on search1011 is OK: puppet ran at Thu Aug 1 11:51:03 UTC 2013 [11:51:12] RECOVERY - Puppet freshness on db31 is OK: puppet ran at Thu Aug 1 11:51:03 UTC 2013 [11:51:12] RECOVERY - Puppet freshness on mw1134 is OK: puppet ran at Thu Aug 1 11:51:08 UTC 2013 [11:51:42] RECOVERY - Puppet freshness on lvs6 is OK: puppet ran at Thu Aug 1 11:51:39 UTC 2013 [11:53:13] RECOVERY - Puppet freshness on stat1001 is OK: puppet ran at Thu Aug 1 11:52:44 UTC 2013 [11:53:37] RECOVERY - Puppet freshness on search1004 is OK: puppet ran at Thu Aug 1 11:52:59 UTC 2013 [11:53:39] RECOVERY - Puppet freshness on mw1215 is OK: puppet ran at Thu Aug 1 11:53:04 UTC 2013 [11:53:39] RECOVERY - Puppet freshness on mw1200 is OK: puppet ran at Thu Aug 1 11:53:04 UTC 2013 [11:53:39] RECOVERY - Puppet freshness on linne is OK: puppet ran at Thu Aug 1 11:53:09 UTC 2013 [11:53:48] RECOVERY - Puppet freshness on mw1220 is OK: puppet ran at Thu Aug 1 11:53:19 UTC 2013 [11:54:09] RECOVERY - Puppet freshness on chromium is OK: puppet ran at Thu Aug 1 11:53:24 UTC 2013 [11:54:09] RECOVERY - Puppet freshness on db1057 is OK: puppet ran at Thu Aug 1 11:53:24 UTC 2013 [11:54:09] RECOVERY - Puppet freshness on ms-be1003 is OK: puppet ran at Thu Aug 1 11:53:24 UTC 2013 [11:54:09] RECOVERY - Puppet freshness on db44 is OK: puppet ran at Thu Aug 1 11:53:24 UTC 2013 [11:54:09] RECOVERY - Puppet freshness on wtp1020 is OK: puppet ran at Thu Aug 1 11:53:24 UTC 2013 [11:54:09] RECOVERY - Puppet freshness on db56 is OK: puppet ran at Thu Aug 1 11:53:24 UTC 2013 [11:54:09] RECOVERY - Puppet freshness on db36 is OK: puppet ran at Thu Aug 1 11:53:24 UTC 2013 [11:54:09] RECOVERY - Puppet freshness on mw1090 is OK: puppet ran at Thu Aug 1 11:53:24 UTC 2013 [11:54:09] RECOVERY - Puppet freshness on sq59 is OK: puppet ran at Thu Aug 1 11:53:24 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on sq37 is OK: puppet ran at Thu Aug 1 11:53:24 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on cp1054 is OK: puppet ran at Thu Aug 1 11:53:24 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on srv271 is OK: puppet ran at Thu Aug 1 11:53:29 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on mw91 is OK: puppet ran at Thu Aug 1 11:53:34 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on ssl3001 is OK: puppet ran at Thu Aug 1 11:53:34 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on mw1 is OK: puppet ran at Thu Aug 1 11:53:34 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on srv298 is OK: puppet ran at Thu Aug 1 11:53:34 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on search35 is OK: puppet ran at Thu Aug 1 11:53:44 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on mw113 is OK: puppet ran at Thu Aug 1 11:53:44 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on emery is OK: puppet ran at Thu Aug 1 11:53:44 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on mw1135 is OK: puppet ran at Thu Aug 1 11:53:44 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on mw1145 is OK: puppet ran at Thu Aug 1 11:53:49 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on mw59 is OK: puppet ran at Thu Aug 1 11:53:49 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Thu Aug 1 11:53:49 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on ms-be1004 is OK: puppet ran at Thu Aug 1 11:53:54 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on mw1174 is OK: puppet ran at Thu Aug 1 11:53:59 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on analytics1018 is OK: puppet ran at Thu Aug 1 11:53:59 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on sq77 is OK: puppet ran at Thu Aug 1 11:54:04 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on es2 is OK: puppet ran at Thu Aug 1 11:54:04 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on mc1006 is OK: puppet ran at Thu Aug 1 11:54:04 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on sq49 is OK: puppet ran at Thu Aug 1 11:54:05 UTC 2013 [11:54:14] RECOVERY - Puppet freshness on cp1001 is OK: puppet ran at Thu Aug 1 11:54:05 UTC 2013 [11:54:22] RECOVERY - Puppet freshness on lvs1001 is OK: puppet ran at Thu Aug 1 11:54:15 UTC 2013 [11:54:22] RECOVERY - Puppet freshness on wtp1008 is OK: puppet ran at Thu Aug 1 11:54:15 UTC 2013 [11:54:22] RECOVERY - Puppet freshness on mw1121 is OK: puppet ran at Thu Aug 1 11:54:15 UTC 2013 [11:54:22] RECOVERY - Puppet freshness on ms-be1010 is OK: puppet ran at Thu Aug 1 11:54:16 UTC 2013 [11:54:22] RECOVERY - Puppet freshness on cp3021 is OK: puppet ran at Thu Aug 1 11:54:16 UTC 2013 [11:54:22] RECOVERY - Puppet freshness on cp1053 is OK: puppet ran at Thu Aug 1 11:54:16 UTC 2013 [11:54:22] RECOVERY - Puppet freshness on mw1141 is OK: puppet ran at Thu Aug 1 11:54:16 UTC 2013 [11:54:23] RECOVERY - Puppet freshness on wtp1006 is OK: puppet ran at Thu Aug 1 11:54:16 UTC 2013 [11:54:23] RECOVERY - Puppet freshness on virt1005 is OK: puppet ran at Thu Aug 1 11:54:16 UTC 2013 [11:54:24] RECOVERY - Puppet freshness on db1056 is OK: puppet ran at Thu Aug 1 11:54:16 UTC 2013 [11:54:24] RECOVERY - Puppet freshness on db1045 is OK: puppet ran at Thu Aug 1 11:54:21 UTC 2013 [11:54:25] RECOVERY - Puppet freshness on cp1068 is OK: puppet ran at Thu Aug 1 11:54:21 UTC 2013 [11:54:32] RECOVERY - Puppet freshness on wtp1016 is OK: puppet ran at Thu Aug 1 11:54:26 UTC 2013 [11:54:32] RECOVERY - Puppet freshness on lvs2 is OK: puppet ran at Thu Aug 1 11:54:26 UTC 2013 [11:54:32] RECOVERY - Puppet freshness on ssl1002 is OK: puppet ran at Thu Aug 1 11:54:26 UTC 2013 [11:54:32] RECOVERY - Puppet freshness on cp1039 is OK: puppet ran at Thu Aug 1 11:54:26 UTC 2013 [11:54:42] RECOVERY - Puppet freshness on mw92 is OK: puppet ran at Thu Aug 1 11:54:36 UTC 2013 [11:54:42] RECOVERY - Puppet freshness on srv251 is OK: puppet ran at Thu Aug 1 11:54:36 UTC 2013 [11:54:42] RECOVERY - Puppet freshness on sq75 is OK: puppet ran at Thu Aug 1 11:54:41 UTC 2013 [11:54:42] RECOVERY - Puppet freshness on srv243 is OK: puppet ran at Thu Aug 1 11:54:41 UTC 2013 [11:54:42] RECOVERY - Puppet freshness on mw15 is OK: puppet ran at Thu Aug 1 11:54:41 UTC 2013 [11:54:42] RECOVERY - Puppet freshness on mw1026 is OK: puppet ran at Thu Aug 1 11:54:41 UTC 2013 [11:54:43] RECOVERY - Puppet freshness on analytics1017 is OK: puppet ran at Thu Aug 1 11:54:41 UTC 2013 [11:54:52] RECOVERY - Puppet freshness on sq66 is OK: puppet ran at Thu Aug 1 11:54:46 UTC 2013 [11:54:52] RECOVERY - Puppet freshness on analytics1020 is OK: puppet ran at Thu Aug 1 11:54:46 UTC 2013 [11:54:52] RECOVERY - Puppet freshness on mw1082 is OK: puppet ran at Thu Aug 1 11:54:46 UTC 2013 [11:54:52] RECOVERY - Puppet freshness on nitrogen is OK: puppet ran at Thu Aug 1 11:54:46 UTC 2013 [11:54:52] RECOVERY - Puppet freshness on mw25 is OK: puppet ran at Thu Aug 1 11:54:46 UTC 2013 [11:54:52] RECOVERY - Puppet freshness on mw7 is OK: puppet ran at Thu Aug 1 11:54:51 UTC 2013 [11:54:52] RECOVERY - Puppet freshness on db1046 is OK: puppet ran at Thu Aug 1 11:54:51 UTC 2013 [11:54:53] RECOVERY - Puppet freshness on cp1009 is OK: puppet ran at Thu Aug 1 11:54:51 UTC 2013 [11:54:53] RECOVERY - Puppet freshness on mw1110 is OK: puppet ran at Thu Aug 1 11:54:51 UTC 2013 [11:54:54] RECOVERY - Puppet freshness on cp1008 is OK: puppet ran at Thu Aug 1 11:54:51 UTC 2013 [11:54:54] RECOVERY - Puppet freshness on db1018 is OK: puppet ran at Thu Aug 1 11:54:51 UTC 2013 [11:54:55] RECOVERY - Puppet freshness on ms1004 is OK: puppet ran at Thu Aug 1 11:54:51 UTC 2013 [11:54:55] RECOVERY - Puppet freshness on mw1009 is OK: puppet ran at Thu Aug 1 11:54:51 UTC 2013 [11:54:56] RECOVERY - Puppet freshness on hydrogen is OK: puppet ran at Thu Aug 1 11:54:51 UTC 2013 [11:55:02] RECOVERY - Puppet freshness on mw1031 is OK: puppet ran at Thu Aug 1 11:54:56 UTC 2013 [11:55:02] RECOVERY - Puppet freshness on db64 is OK: puppet ran at Thu Aug 1 11:54:56 UTC 2013 [11:55:02] RECOVERY - Puppet freshness on mw1158 is OK: puppet ran at Thu Aug 1 11:55:01 UTC 2013 [11:55:02] RECOVERY - Puppet freshness on mw1086 is OK: puppet ran at Thu Aug 1 11:55:01 UTC 2013 [11:55:02] RECOVERY - Puppet freshness on cp3019 is OK: puppet ran at Thu Aug 1 11:55:01 UTC 2013 [11:55:02] RECOVERY - Puppet freshness on mw1199 is OK: puppet ran at Thu Aug 1 11:55:01 UTC 2013 [11:55:02] RECOVERY - Puppet freshness on db1049 is OK: puppet ran at Thu Aug 1 11:55:01 UTC 2013 [11:55:12] RECOVERY - Puppet freshness on virt8 is OK: puppet ran at Thu Aug 1 11:55:06 UTC 2013 [11:55:12] RECOVERY - Puppet freshness on snapshot3 is OK: puppet ran at Thu Aug 1 11:55:06 UTC 2013 [11:55:12] RECOVERY - Puppet freshness on ms-fe1004 is OK: puppet ran at Thu Aug 1 11:55:06 UTC 2013 [11:55:12] RECOVERY - Puppet freshness on mw1060 is OK: puppet ran at Thu Aug 1 11:55:06 UTC 2013 [11:55:12] RECOVERY - Puppet freshness on mc1002 is OK: puppet ran at Thu Aug 1 11:55:06 UTC 2013 [11:55:12] RECOVERY - Puppet freshness on srv268 is OK: puppet ran at Thu Aug 1 11:55:06 UTC 2013 [11:55:12] RECOVERY - Puppet freshness on search23 is OK: puppet ran at Thu Aug 1 11:55:06 UTC 2013 [11:55:13] RECOVERY - Puppet freshness on mw1108 is OK: puppet ran at Thu Aug 1 11:55:06 UTC 2013 [11:55:13] RECOVERY - Puppet freshness on cp1064 is OK: puppet ran at Thu Aug 1 11:55:06 UTC 2013 [11:55:14] RECOVERY - Puppet freshness on harmon is OK: puppet ran at Thu Aug 1 11:55:11 UTC 2013 [11:55:14] RECOVERY - Puppet freshness on mw1148 is OK: puppet ran at Thu Aug 1 11:55:11 UTC 2013 [11:55:22] RECOVERY - Puppet freshness on sq36 is OK: puppet ran at Thu Aug 1 11:55:16 UTC 2013 [11:55:22] RECOVERY - Puppet freshness on srv257 is OK: puppet ran at Thu Aug 1 11:55:16 UTC 2013 [11:55:22] RECOVERY - Puppet freshness on srv235 is OK: puppet ran at Thu Aug 1 11:55:16 UTC 2013 [11:55:22] RECOVERY - Puppet freshness on mw12 is OK: puppet ran at Thu Aug 1 11:55:16 UTC 2013 [11:55:22] RECOVERY - Puppet freshness on mw56 is OK: puppet ran at Thu Aug 1 11:55:21 UTC 2013 [11:55:23] RECOVERY - Puppet freshness on mw48 is OK: puppet ran at Thu Aug 1 11:55:21 UTC 2013 [11:55:23] RECOVERY - Puppet freshness on mw3 is OK: puppet ran at Thu Aug 1 11:55:21 UTC 2013 [11:55:23] RECOVERY - Puppet freshness on mw1170 is OK: puppet ran at Thu Aug 1 11:55:21 UTC 2013 [11:55:23] RECOVERY - Puppet freshness on srv293 is OK: puppet ran at Thu Aug 1 11:55:21 UTC 2013 [11:55:32] RECOVERY - Puppet freshness on mw1058 is OK: puppet ran at Thu Aug 1 11:55:26 UTC 2013 [11:55:32] RECOVERY - Puppet freshness on mw1119 is OK: puppet ran at Thu Aug 1 11:55:26 UTC 2013 [11:55:32] RECOVERY - Puppet freshness on mw1102 is OK: puppet ran at Thu Aug 1 11:55:26 UTC 2013 [11:55:32] RECOVERY - Puppet freshness on mw1085 is OK: puppet ran at Thu Aug 1 11:55:26 UTC 2013 [11:55:32] RECOVERY - Puppet freshness on analytics1021 is OK: puppet ran at Thu Aug 1 11:55:26 UTC 2013 [11:55:42] RECOVERY - Puppet freshness on mw1073 is OK: puppet ran at Thu Aug 1 11:55:31 UTC 2013 [11:55:42] RECOVERY - Puppet freshness on mw1120 is OK: puppet ran at Thu Aug 1 11:55:31 UTC 2013 [11:55:42] RECOVERY - Puppet freshness on mw1070 is OK: puppet ran at Thu Aug 1 11:55:31 UTC 2013 [11:55:42] RECOVERY - Puppet freshness on mw1008 is OK: puppet ran at Thu Aug 1 11:55:31 UTC 2013 [11:55:42] RECOVERY - Puppet freshness on db1015 is OK: puppet ran at Thu Aug 1 11:55:32 UTC 2013 [11:55:42] RECOVERY - Puppet freshness on virt0 is OK: puppet ran at Thu Aug 1 11:55:37 UTC 2013 [11:55:54] (03PS1) 10Mark Bergsma: Migrate all uses to interface::add_ip6_mapped, remove old definition [operations/puppet] - 10https://gerrit.wikimedia.org/r/77098 [11:56:02] RECOVERY - Puppet freshness on mw1157 is OK: puppet ran at Thu Aug 1 11:55:52 UTC 2013 [11:56:02] RECOVERY - Puppet freshness on mw1019 is OK: puppet ran at Thu Aug 1 11:55:57 UTC 2013 [11:56:02] RECOVERY - Puppet freshness on mw1061 is OK: puppet ran at Thu Aug 1 11:55:57 UTC 2013 [11:56:12] RECOVERY - Puppet freshness on es1008 is OK: puppet ran at Thu Aug 1 11:56:07 UTC 2013 [11:57:06] (03CR) 10Mark Bergsma: [C: 032] Migrate all uses to interface::add_ip6_mapped, remove old definition [operations/puppet] - 10https://gerrit.wikimedia.org/r/77098 (owner: 10Mark Bergsma) [11:57:22] RECOVERY - Puppet freshness on srv274 is OK: puppet ran at Thu Aug 1 11:57:13 UTC 2013 [11:57:22] RECOVERY - Puppet freshness on mw1059 is OK: puppet ran at Thu Aug 1 11:57:18 UTC 2013 [11:57:32] RECOVERY - Puppet freshness on terbium is OK: puppet ran at Thu Aug 1 11:57:23 UTC 2013 [12:01:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [12:04:31] (03PS1) 10Mark Bergsma: Test interface::offload [operations/puppet] - 10https://gerrit.wikimedia.org/r/77100 [12:04:33] (03CR) 10Mark Bergsma: [C: 032] Test interface::offload [operations/puppet] - 10https://gerrit.wikimedia.org/r/77100 (owner: 10Mark Bergsma) [12:05:12] ceph rc3 [12:05:24] mumble mumble [12:10:08] (03PS1) 10Faidon: reprepro updates: GetInRelease no for cdh4 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77102 [12:10:11] (03PS1) 10Mark Bergsma: Migrate all uses to interface::offload, remove old definitions [operations/puppet] - 10https://gerrit.wikimedia.org/r/77103 [12:10:12] (03CR) 10Faidon: [C: 032 V: 032] reprepro updates: GetInRelease no for cdh4 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77102 (owner: 10Faidon) [12:10:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:35] (03PS2) 10Mark Bergsma: Migrate all uses to interface::offload, remove old definitions [operations/puppet] - 10https://gerrit.wikimedia.org/r/77103 [12:11:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [12:13:33] agile agile? [12:13:37] (03CR) 10Mark Bergsma: [C: 032] Migrate all uses to interface::offload, remove old definitions [operations/puppet] - 10https://gerrit.wikimedia.org/r/77103 (owner: 10Mark Bergsma) [12:14:18] ? [12:14:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:16:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [12:18:03] (03PS1) 10Mark Bergsma: Migrate interface_manual to interface::manual [operations/puppet] - 10https://gerrit.wikimedia.org/r/77104 [12:19:17] (03CR) 10Mark Bergsma: [C: 032] Migrate interface_manual to interface::manual [operations/puppet] - 10https://gerrit.wikimedia.org/r/77104 (owner: 10Mark Bergsma) [12:21:00] soon, the entirety of wikidata will be cached in varnish ;) [12:23:22] better than yesterday's 0.003? [12:24:33] (03PS1) 10Mark Bergsma: Test interface::ip [operations/puppet] - 10https://gerrit.wikimedia.org/r/77105 [12:24:35] hardly [12:24:46] but even at a very slow pace, all pages will end up in there ;-) [12:24:50] although there's expiry hehe [12:25:25] (03CR) 10Mark Bergsma: [C: 032] Test interface::ip [operations/puppet] - 10https://gerrit.wikimedia.org/r/77105 (owner: 10Mark Bergsma) [12:28:06] PROBLEM - Puppet freshness on sq41 is CRITICAL: No successful Puppet run in the last 10 hours [12:33:15] (03PS1) 10Mark Bergsma: Migrate interface_ip to interface::ip [operations/puppet] - 10https://gerrit.wikimedia.org/r/77106 [12:33:17] (03CR) 10Mark Bergsma: [C: 032] Migrate interface_ip to interface::ip [operations/puppet] - 10https://gerrit.wikimedia.org/r/77106 (owner: 10Mark Bergsma) [12:36:15] (03PS1) 10Mark Bergsma: Test interface::tagged [operations/puppet] - 10https://gerrit.wikimedia.org/r/77107 [12:37:11] (03CR) 10Mark Bergsma: [C: 032] Test interface::tagged [operations/puppet] - 10https://gerrit.wikimedia.org/r/77107 (owner: 10Mark Bergsma) [12:40:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.802 second response time [12:42:22] (03PS1) 10Mark Bergsma: Migrate interface_tagged to interface::tagged [operations/puppet] - 10https://gerrit.wikimedia.org/r/77109 [12:43:28] (03CR) 10Mark Bergsma: [C: 032] Migrate interface_tagged to interface::tagged [operations/puppet] - 10https://gerrit.wikimedia.org/r/77109 (owner: 10Mark Bergsma) [12:48:01] (03PS1) 10Mark Bergsma: Test interface::aggregate [operations/puppet] - 10https://gerrit.wikimedia.org/r/77110 [12:49:12] (03PS2) 10Mark Bergsma: Test interface::aggregate [operations/puppet] - 10https://gerrit.wikimedia.org/r/77110 [12:50:01] (03CR) 10Mark Bergsma: [C: 032] Test interface::aggregate [operations/puppet] - 10https://gerrit.wikimedia.org/r/77110 (owner: 10Mark Bergsma) [12:52:29] (03PS1) 10Mark Bergsma: Correct interface::aggregate definition name [operations/puppet] - 10https://gerrit.wikimedia.org/r/77111 [12:53:21] (03CR) 10Mark Bergsma: [C: 032] Correct interface::aggregate definition name [operations/puppet] - 10https://gerrit.wikimedia.org/r/77111 (owner: 10Mark Bergsma) [13:00:59] aude: no new reports/problems with varnish? [13:02:33] RECOVERY - Puppet freshness on mw1112 is OK: puppet ran at Thu Aug 1 13:02:28 UTC 2013 [13:03:31] (03PS1) 10Mark Bergsma: Migrate interface_aggregate to interface::aggregate [operations/puppet] - 10https://gerrit.wikimedia.org/r/77113 [13:03:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:16] (03CR) 10Mark Bergsma: [C: 032] Migrate interface_aggregate to interface::aggregate [operations/puppet] - 10https://gerrit.wikimedia.org/r/77113 (owner: 10Mark Bergsma) [13:06:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 9.079 second response time [13:09:13] (03PS1) 10Mark Bergsma: Migrate interface_tun6to4 to interface::tun6to4 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77114 [13:09:21] cleanup mode? [13:09:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:15] (03CR) 10Mark Bergsma: [C: 032] Migrate interface_tun6to4 to interface::tun6to4 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77114 (owner: 10Mark Bergsma) [13:11:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 1.772 second response time [13:14:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:14:56] (03PS1) 10Mark Bergsma: Cleanup, migrate the remaining interface_* stuff out of generic-definitions [operations/puppet] - 10https://gerrit.wikimedia.org/r/77115 [13:15:51] (03CR) 10Mark Bergsma: [C: 032] Cleanup, migrate the remaining interface_* stuff out of generic-definitions [operations/puppet] - 10https://gerrit.wikimedia.org/r/77115 (owner: 10Mark Bergsma) [13:16:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 2.132 second response time [13:16:41] andrewbogott_afk: interface module migration all done :) [13:17:01] i think it cut generic_definitions.pp in half [13:18:51] (03PS1) 10Mark Bergsma: Remove obsolete ubuntu dist version check [operations/puppet] - 10https://gerrit.wikimedia.org/r/77116 [13:19:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:19:41] (03CR) 10Mark Bergsma: [C: 032] Remove obsolete ubuntu dist version check [operations/puppet] - 10https://gerrit.wikimedia.org/r/77116 (owner: 10Mark Bergsma) [13:20:01] I envy your getting shit done attitude [13:21:46] the org is slowly eating from it :( [13:25:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.173 second response time [13:27:07] so, i'm converting tabs -> spaces in admins.pp [13:27:18] anyone mind if I make the ordering of account classes there consistent? [13:27:21] going to order by uid [13:27:23] not alpha [13:29:32] (03CR) 10Mark Bergsma: "I appreciate the effort, but I don't like these big commits crossing all kinds of different services that I can't easily test one by one. " [operations/puppet] - 10https://gerrit.wikimedia.org/r/76948 (owner: 10Hashar) [13:30:06] mark: was afraid to split them all as tiny one liners :-D [13:30:09] not in the same commit [13:30:24] will split it [13:30:25] hashar: that's the only way you'll get me to merge that at least [13:30:29] (but YMMV with others) [13:30:36] I understand [13:30:39] i won't merge something that might break stuff all over the place which I can't see [13:30:48] I thought they were trivial enough to avoid spamming in Gerrit hehe [13:31:01] i seriously wonder how puppet even compiles some of those errors now [13:31:09] anyway, I spam gerrit all the time ;) [13:33:54] (03Abandoned) 10Hashar: exec[] -> Exec[] [operations/puppet] - 10https://gerrit.wikimedia.org/r/76948 (owner: 10Hashar) [13:34:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [13:39:41] (03PS1) 10Hashar: rt: package[] -> Package[] [operations/puppet] - 10https://gerrit.wikimedia.org/r/77119 [13:40:09] (03PS1) 10Ottomata: admins.pp - tabs -> spaces & alignment. [operations/puppet] - 10https://gerrit.wikimedia.org/r/77120 [13:40:58] (03PS2) 10Hashar: rt: package[] -> Package[] [operations/puppet] - 10https://gerrit.wikimedia.org/r/77119 [13:41:11] (03PS1) 10Hashar: role::eqiad-proxy: file[] -> File[] [operations/puppet] - 10https://gerrit.wikimedia.org/r/77121 [13:41:36] (03PS1) 10Hashar: role::pmtpa-proxy: file[] -> File[] [operations/puppet] - 10https://gerrit.wikimedia.org/r/77122 [13:42:02] (03PS1) 10Hashar: generic::mysql::server: exec[] -> Exec[] [operations/puppet] - 10https://gerrit.wikimedia.org/r/77123 [13:42:43] (03PS1) 10Hashar: role::nova: package[] -> Package[] [operations/puppet] - 10https://gerrit.wikimedia.org/r/77124 [13:42:57] (03PS1) 10Hashar: mediawiki_singlenode: exec[] -> Exec[] [operations/puppet] - 10https://gerrit.wikimedia.org/r/77125 [13:43:31] mark: done as independent change, so one could merge and check the result :D [13:43:37] though one is hitting the mysql servers [13:43:44] damn you [13:43:50] the plan was not for you to do that right now [13:43:51] ;-) [13:45:15] ottomata: you should stick a vim modeling at top of manifests/admins.pp : # vim: set ts=4 sw=4 et : [13:45:31] oh yeah? [13:45:47] hmm [13:45:54] anyway to make that global for the repo? [13:46:15] I do it locally based on the local path [13:46:20] (03PS2) 10Ottomata: admins.pp - tabs -> spaces & alignment. [operations/puppet] - 10https://gerrit.wikimedia.org/r/77120 [13:46:22] (03CR) 10Ottomata: [C: 032 V: 032] admins.pp - tabs -> spaces & alignment. [operations/puppet] - 10https://gerrit.wikimedia.org/r/77120 (owner: 10Ottomata) [13:46:37] but ops/puppet files are either tabs or spaces indented :( [13:47:22] hmm, didn't seem to work on my local [13:47:28] i added that line to the top of the file [13:47:36] opened in with vim and tabbed. [13:47:39] it put a literal tab in. [13:48:10] ah maybe you don't have modeling enabled: [13:48:12] set modeline [13:48:14] set modelines=5 [13:48:33] instruct vim to look at the first 5 lines of the file for potential instructions [13:49:12] ah ok, well, i don't use vim as my regular editor [13:49:15] i use it on the servers [13:53:40] mark: I have a slightly different take than you on base I think [13:54:37] mark: I think *all* servers should include the main base class; base classes or other classes (ntp) that aren't appropriate for all servers should then be in standard instead of base [13:55:08] and lvs, mail, ntp servers and such are not "standard" but "base + (...)" [13:55:10] that's how I started it off actually [13:55:17] I agree with that [13:55:38] okay, maybe I misinterpreted your mail [13:55:45] no I don't think you did [13:55:57] that is exactly how I originally intended base vs standard [13:56:04] and my mail is kind of how I'd like standard to be [13:57:01] why not just do that in role classes? [13:57:17] i.e. leave base granular, and let role classes bloat? [13:57:46] i have no idea what you mean jeff [13:58:05] it's essentially what I did with role/fundraising.pp [13:58:28] mark: I'm a bit confused; you mean that this is how you started it but now you've changed your mind and are proposing something else? [13:58:35] Jeff_Green: i think that's flawed [13:58:42] the entire idea of base is that EVERY box does all that [13:58:54] and if there has to be an exception, that exception should be clearly defined [13:59:01] as opposed to just a missing class somewhere [13:59:11] it's way way too easy to forget to include stuff in your model [13:59:24] (/me likes that) [13:59:29] they're both flawed in different ways :-) [13:59:37] paravoid: no, it's just diluted over time with more team members and more services ;) [13:59:45] ideally we could even have standard-nontp or standard-lvs or something [13:59:47] i like your enable/disable idea better than what we have now though [14:00:13] we shouldn't have to run around and adjust all role classes everywhere whenever we make a change to base [14:00:21] we have standard-noexim already [14:00:29] I hate standard-noexim [14:00:38] that's not flexible [14:04:14] mark--at some point the exceptions have to be made. what if, for example, we make a change to base and we forget to add an exclusion to some role that shouldn't have it [14:04:48] yes, but base exceptions are far less likely & frequent than base inclusions [14:04:54] it's already pretty weird for a base exception to exist [14:05:38] but if we really need them we should support that somehow [14:05:55] maybe we just need to define a line after which things don't belong in base [14:06:26] feel free to point at some examples on the list [14:06:34] central logging imo [14:06:57] I think it's an excellent thing to have, but not as currently implemented [14:07:42] salt::minions I'm not sure about [14:10:35] every server should be a salt minion shouldn't it [14:12:27] I haven't had a chance to really review the implications. Does it belong on a hardened bastion or a fundraising box? [14:12:44] yes [14:12:47] puppet runs there too doesn't it? ;) [14:12:54] if not it's in a different realm [14:13:45] puppet probably shouldn't run on a hardened bastion [14:13:58] at least not of the style Tim described [14:14:30] anyway, like I said--I don't know about salt. but my inclination was to leave it off of fundraising servers until I understood it better [14:15:15] nrpe doesn't belong anywhere :-P [14:15:38] ok that's all I have to say on the matter [14:16:42] if puppet doesn't run there anyway it can be in base ;) [14:16:48] then it's an entirely different setup altogether [14:17:35] and your non-pci compliant production realm fundraising servers really should track the rest as much as possible [14:17:51] or you won't have any guarantees or help from the rest of the team whatsoever of course [14:18:06] (03PS1) 10Ottomata: Ordering admins user classes by uid. [operations/puppet] - 10https://gerrit.wikimedia.org/r/77128 [14:18:08] if we expect salt to be deployed everywhere it's really bad if those servers don't have it [14:19:05] nice if we do a root password change using salt and your servers don't get the update :) [14:19:39] (03CR) 10Ottomata: [C: 032 V: 032] Ordering admins user classes by uid. [operations/puppet] - 10https://gerrit.wikimedia.org/r/77128 (owner: 10Ottomata) [14:19:41] mark everyone is afraid of them anyway [14:19:48] zomg fundraising box don't touch it! [14:19:50] because you do everything differently [14:20:09] this base diversion is an excellent example ;) [14:20:25] less different from labs for example [14:20:29] i can understand that for whatever is under PCI compliance, but not the rest [14:20:47] I think the differences to labs are better understood [14:20:52] mark: no issues with varnish that i am aware of [14:21:00] bot is behaving :) [14:21:03] and we're also actively working on reducing those differences where possible [14:21:08] that's not happening with fundraising... [14:21:11] aude: excellent :) [14:21:32] thanks for getting it all working :) [14:21:33] mark if you're talking about aluminium, there's really not that's all that different [14:21:33] aude: bot is basically not getting any cached pages, so it's hard for it not to work ;-) [14:21:40] right [14:22:05] (salt doesn't work in esams) [14:22:07] mark and what's there is pretty well of necessity. do we really want a bunch of donor data landing on our production central loggers? [14:22:17] paravoid: i know, it's an example though [14:22:48] I know, that's why I used ( ) :) [14:23:12] Jeff_Green: right, but if you're afraid of that it would be better to have an explicit "disable the logger for reason X" than to just not have it in a big list of base includes of course ;) [14:23:50] mark so the alternative is that someone adds something to the base class, I don't notice, and they were never thinking about the security implications on fundraising servers [14:24:06] I think we need a separate base per realm [14:24:13] labs is also quite different [14:24:14] paravoid: this is not a separaterealm though [14:24:16] but yes [14:24:22] fundraising isn't? [14:24:29] paravoid: not these [14:24:40] the part of fundraising that is a different realm has a different puppet repo anyway [14:24:51] fundraising is really just for illustration, they're all moving to that other repo [14:25:03] Jeff_Green: if something is secure enough to be put in base for all production servers, it would be pretty weird for it not to be secure enough on those non-pci compliant fundraising servers [14:25:04] aluminium is the last straggler iirc [14:26:07] ok, if they're all in your nice little island anyway, why are we even having this discussion? :) [14:26:10] mark central logging is the only obvious example I can think of, and people who aren't thinking about fundraising aren't going to think about the fact that sensitive data is sometimes syslogged. [14:26:11] I'm really hoping that with this whole modularization effort we could perhaps converge back parts of fundraising's puppet [14:26:16] but that's geting a bit besides the point [14:26:25] Jeff_Green: sensitive data gets syslogged everywhere, not just in fundraising [14:26:32] paravoid: yup, I can see importing modules as they develop [14:26:50] ...or even splitting them into submodules and reusing the same git repos :) [14:27:01] mark people who see the fundraising data are supposed to have signed special donor related NDAs [14:27:10] paravoid: sure [14:27:37] ...why is donor data syslogged anyway? [14:28:07] so back to the point [14:28:21] one reason is that the logs can be used to reconstruct transactions that didn't process properly [14:28:27] I don't mind the parameterized classes idea at all [14:28:29] i don't think the current modularisation will help in any way with getting the fundraising realm back [14:28:45] because essentially they're no different from what we have already [14:28:49] (a few exceptions here and there) [14:28:50] but it might be worth it to collect all those cases where standard isn't included [14:28:57] and jeff thinks in a completely different way about config management [14:29:18] mark they may make it easier to import and maintain [14:29:20] i'd call it monolithic instead of modular actually ;) [14:29:31] and maybe have separate "standard" classes for these [14:29:41] i.e. I don't mind standard-noexim much [14:30:09] it's not such a huge difference from the parameterization idea though [14:30:16] so maybe this is getting a bit academic :) [14:30:20] just cleaner and more flexible ;) [14:31:32] so, we were having the base::firewall discussion in amsterdam [14:31:53] you raised the point that if it's not to be included in all servers (thinking of high performance servers etc.) it shouldn't be under base:: [14:32:03] yes [14:32:21] (and I found that reasonable too) [14:33:13] would we keep it under the base module in this new scheme and just have a parameter about it in the main base/init.pp class? [14:33:53] it can go under standard [14:34:08] (03PS1) 10Ottomata: Adding account class for Toby Negrin. RT 5391 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77129 [14:34:20] base everywhere, standard templates for most servers [14:34:26] nod [14:34:32] so under what? a standard module? [14:34:37] or a base::standard class? :) [14:34:40] or a different name perhaps [14:34:45] base::standard++ [14:34:49] base::minimal [14:34:58] base::base :-P [14:35:05] what's minimal? [14:35:18] i'd say central logging is even minimal, but then we already have that problem ;) [14:35:25] hehe [14:35:42] the problem with central logging is that the granularity of per-datacenter isn't good [14:35:45] (03PS2) 10Ottomata: Adding account class for Toby Negrin. RT 5391 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77129 [14:36:00] ottomata: want to add #5512 while at it? [14:36:28] paravoid: sure! [14:37:36] paravoid, that RT isn't clear where I need to give him shell acounts [14:37:39] other than admins::restricted [14:37:41] is that it? [14:38:19] also, do I need to call him to to verify the ssh key? [14:38:25] admins::parsoid I think [14:38:34] ah ok, perfect [14:41:18] probably... [14:41:24] I'll ask on the ticket [14:42:03] hm, it's admins::parsoid [14:42:12] it's just that he'll get access to parsoid varnishes this way :) [14:42:25] that's fine [14:42:40] * mark creates a new ssh key [14:43:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [14:48:18] ottomata: is gerrit fucking up the diff, or did you reorder all those accounts along with tabs/spaces? [14:50:02] that was a second commit, but ja [14:50:17] brb [14:50:52] ...how are we supposed to review this? [14:51:51] it's a huge 2700 line commit which changes different things [14:51:55] and produces no clear diff [14:52:06] and that's concerning all access and ssh keys [14:57:20] (03PS1) 10Mark Bergsma: Revert "Ordering admins user classes by uid." [operations/puppet] - 10https://gerrit.wikimedia.org/r/77130 [14:58:24] I don't like not reviewing massive changes to admin.pp either [14:58:25] (03PS1) 10Mark Bergsma: Revert "admins.pp - tabs -> spaces & alignment." [operations/puppet] - 10https://gerrit.wikimedia.org/r/77131 [14:58:36] (03CR) 10jenkins-bot: [V: 04-1] Revert "admins.pp - tabs -> spaces & alignment." [operations/puppet] - 10https://gerrit.wikimedia.org/r/77131 (owner: 10Mark Bergsma) [14:58:46] (03CR) 10Mark Bergsma: [C: 032] Revert "Ordering admins user classes by uid." [operations/puppet] - 10https://gerrit.wikimedia.org/r/77130 (owner: 10Mark Bergsma) [14:58:53] i'd suggest instead writing a script that does those changes (for the whitespace it should be easy) and put in the commit message [14:59:09] let others review the script, fetch locally, run the script and compare the results [14:59:13] (03PS2) 10Mark Bergsma: Revert "admins.pp - tabs -> spaces & alignment." [operations/puppet] - 10https://gerrit.wikimedia.org/r/77131 [15:00:34] (03CR) 10Mark Bergsma: [C: 032] Revert "admins.pp - tabs -> spaces & alignment." [operations/puppet] - 10https://gerrit.wikimedia.org/r/77131 (owner: 10Mark Bergsma) [15:04:30] Jeff_Green, I don't know how decomissioning.pp works… won't adding the payment systems there cause them to be wiped or shut off or something? [15:05:07] ottomata: i'm sorry, I don't want to block your work and these improvements, but you'll have to do it in a way where it can be properly reviewed by others [15:05:27] andrewbogott: afaik the only thing I know it does is rip out icinga config [15:05:38] ok. [15:06:02] and a few other things [15:06:09] basically, after being put in there it needs a reinstall [15:06:11] it won't shut off [15:06:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:52] mark: OK, but in this case they aren't actually decomissioned they're just moved to a different puppet server. [15:06:59] So, maybe that's not appropriate [15:07:03] mark he's talking about removing payments[1-4] from production puppet [15:07:04] (03PS2) 10Andrew Bogott: Remove payment nodes from site.pp, and add to decomissioning. [operations/puppet] - 10https://gerrit.wikimedia.org/r/77069 [15:07:09] then I wouldn't do that no [15:07:18] 'k [15:07:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 2.790 second response time [15:07:24] (03CR) 10jenkins-bot: [V: 04-1] Remove payment nodes from site.pp, and add to decomissioning. [operations/puppet] - 10https://gerrit.wikimedia.org/r/77069 (owner: 10Andrew Bogott) [15:07:35] mark why not? this is what people have been suggesting I do as I move hosts into frack. [15:07:48] maybe it's not too harmful [15:07:53] but I won't guarantee it ;) [15:07:55] ha [15:08:15] It seems moot since they aren't affected by that puppet code anyway. But it seems generally wrong as they aren't technically decomissioned, just moved. [15:08:18] if you're just doing it to remove icinga config, there are better ways [15:09:10] unrelated: why does iodine's mgmt interface drop me when I try to use it to powercycle the server? [15:09:22] no clue [15:09:31] shares ethernet port maybe? [15:09:33] oh goddamnit. i think I just powercycled iron by accident. [15:09:41] that would explain it [15:09:47] :) [15:09:49] i must have botched my .ssh/config [15:09:55] fail. [15:10:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:47] !log accidentally powercycled iron, trying to do iodine. fail. [15:10:59] Logged the message, Master [15:11:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [15:11:47] sigh. [15:11:50] (03PS3) 10Andrew Bogott: Remove payment nodes from site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/77069 [15:12:02] ok, Jeff_Green, mark, ^ [15:12:43] yeah [15:12:47] and perhaps manually run the script on stafford [15:12:53] so they get purged in the db [15:13:14] Which script is 'the script'? [15:13:38] andrewbogott: cool. i was thinking, if you might as well just add add aluminium as a host-based exclusion in base::remote-syslog for now, the rest doesn't matter [15:14:07] puppetstoredconfigclean.rb [15:14:20] see puppetmaster.pp for details [15:14:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:24] andrewbogott: that almost solves your problem of nonstandard calls to base [15:14:27] ok [15:14:35] Jeff_Green, yeah, sounds good. [15:14:37] although I don't know if naggen removes them then [15:14:45] it might not [15:14:53] it does [15:15:02] naggen just dumps exported resources in one go [15:15:06] ok [15:15:38] i don't think they're even showing up in icinga any longer, i converted them to nsca two years ago [15:15:56] well, showing up in the puppet-built non-nsca part of the config I mean [15:16:07] mark, Jeff_Green, so, can I regard that patch as +1'd? [15:16:16] sure [15:16:20] yes [15:18:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [15:18:54] holy shit, puppet-merge has the biggest diff I've ever seen [15:19:02] Am I stepping on some massive reformatting change? [15:19:03] (03PS4) 10Andrew Bogott: Remove payment nodes from site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/77069 [15:19:06] (03CR) 10Andrew Bogott: [C: 032] "Purge!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/77069 (owner: 10Andrew Bogott) [15:21:08] oh, I guess this is mark's revert. [15:21:16] * andrewbogott crosses fingers and merges [15:21:28] huh [15:21:28] (03PS1) 10Jgreen: add iodine to dhcp conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/77132 [15:21:46] oh I accidently ran it on stafford instead of sockpuppet [15:21:48] sorry [15:21:57] I wanted to run a diff there before the merge ;) [15:22:02] no problem, i should really fix that script so it pulls on 'the other one' rather than just on stafford [15:22:22] (03CR) 10Jgreen: [C: 032 V: 032] add iodine to dhcp conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/77132 (owner: 10Jgreen) [15:22:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.053 second response time [15:23:46] mark, I don't see much in the way of explanation in puppetmaster.pp. Is it just a matter of running "ruby /usr/local/sbin/puppetstoredconfigclean.rb" ? [15:23:59] yeah that's a script that's included with puppet [15:24:05] iirc it takes a hostname argument [15:24:06] 'k [15:24:15] oh, so in that case... [15:24:20] run once per removed node? [15:24:24] yes [15:24:36] that's what the other script does, decom_servers.sh [15:24:45] but that takes the hostnames from the decommissioning.pp list [15:25:27] It says 'Can't find host payments1.wikimedia.org.' [15:26:23] is brewster the one and only dhcp server? [15:27:01] Jeff_Green: right now yes...carbon is the dhcp server in eqiad but it's borked at the moment [15:27:32] k, thx [15:29:51] mark, is that because I needed to run the script before removing the hosts from site.pp? [15:30:10] no [15:30:18] maybe it didn't have any exported resources anymore [15:30:36] ok, fair enough. [15:31:35] !log Prepending AS 14907 once to AS 30217 [15:31:39] * Jeff_Green seeks opinions. I'm building iodine to replace OTRS web host williams. williams has a single partition software RAID1 config. is there any reason we hate that config for the new box? [15:31:47] Logged the message, Master [15:31:59] otrs doesn't have local storage does it [15:33:33] mark correct. it's all mysql [15:33:50] 5.7G used overall on the existing system [15:33:50] then I don't see why a single partition raid1 config is a problem [15:33:54] k [15:34:07] like most other misc servers that don't have significant storage, right [15:34:43] (03PS1) 10Jgreen: add netboot.cfg entry for iodine [operations/puppet] - 10https://gerrit.wikimedia.org/r/77133 [15:35:16] (03CR) 10Jgreen: [C: 032 V: 032] add netboot.cfg entry for iodine [operations/puppet] - 10https://gerrit.wikimedia.org/r/77133 (owner: 10Jgreen) [15:37:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 9.110 second response time [15:44:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 9.189 second response time [15:45:24] (03PS1) 10Andrew Bogott: Reconcile aluminium with standard-noexim. [operations/puppet] - 10https://gerrit.wikimedia.org/r/77135 [15:45:44] (03CR) 10jenkins-bot: [V: 04-1] Reconcile aluminium with standard-noexim. [operations/puppet] - 10https://gerrit.wikimedia.org/r/77135 (owner: 10Andrew Bogott) [15:48:03] (03PS2) 10Andrew Bogott: Reconcile aluminium with standard-noexim. [operations/puppet] - 10https://gerrit.wikimedia.org/r/77135 [15:49:13] hey paravoid, you mentioned once that you review code in the terminal rather than use the gerrit ui. what do you do? just grab the patch and git log -p or something more sophisticated? [15:49:39] Jeff_Green, https://gerrit.wikimedia.org/r/#/c/77135/ [15:50:06] (03CR) 10Ottomata: [C: 032 V: 032] Adding account class for Toby Negrin. RT 5391 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77129 (owner: 10Ottomata) [15:50:06] no I usually don't [15:50:10] I usually use the gerrit ui [15:50:47] ah well. i'm fishing for good alternative workflows. [15:51:52] mark, this fact is indeed dead, right? https://gerrit.wikimedia.org/r/#/c/76120/ [15:52:18] checking [15:52:51] yeah I don't think that's used [15:52:57] i assume you grepped the repo? [15:53:12] go ahead and remove it, if it breaks something we should know soon enough [15:53:53] (03CR) 10Jgreen: [C: 031 V: 031] Reconcile aluminium with standard-noexim. [operations/puppet] - 10https://gerrit.wikimedia.org/r/77135 (owner: 10Andrew Bogott) [15:54:08] andrewbogott: +1'd [15:54:13] hm [15:54:18] actually [15:54:20] mark, yep, I grepped. [15:54:30] andrewbogott: I think that may be used by leslie's tcp optimization [15:54:49] or is it not default_gateway? [15:55:19] andrewbogott: check https://rt.wikimedia.org/Ticket/Display.html?id=2816 [15:55:25] There's alos a default_interface fact... [15:56:03] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [15:56:03] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [15:56:03] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [15:56:03] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [15:56:03] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [15:56:03] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [15:56:19] well, it's the tcptweaks stuff that uses it [15:57:23] (03PS3) 10Ottomata: Adding account class for Toby Negrin. RT 5391 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77129 [15:57:24] Ah, it's used in a template. [15:57:30] OK, I'll leave it alone then! [15:57:40] it should not be installed by puppet though [15:57:43] leslie never fixed that RT [15:57:52] that's broken, since facts are loaded before puppet runs [15:57:54] (03Abandoned) 10Andrew Bogott: Remove default_interface fact [operations/puppet] - 10https://gerrit.wikimedia.org/r/76120 (owner: 10Andrew Bogott) [15:57:55] but the fact gets installed by puppet... [15:58:04] chicken and egg problem [15:58:18] I hope to move the facts to pluginsync soon. That may fix the problem... [15:58:22] yes [15:58:34] but we have other facts that sync already don't we? [15:58:48] is that different from pluginsync? [15:59:03] maybe those are just in modules [15:59:13] Ahhh sorry [15:59:16] mark i missed your revert [15:59:18] As far as I know facts are currently only installed 'by hand' as puppet files. [15:59:18] was very confused there [15:59:34] andrewbogott: ugh, ok [16:00:00] ottomata: if you do a tabs/spaces change, *only change that* in that commit [16:00:06] ok, mark, sorry about the self merge, there weren't any functional changes [16:00:09] i did, didnt' I? [16:00:13] no [16:00:16] the diff was all messed up [16:00:34] there were 2 commits [16:00:53] for one thing, the additions and line removal counts weren't even nearly equal ;0 [16:01:08] yeah i realigned [16:01:13] oh i see [16:01:16] you want tabs/spaces only [16:01:25] i realigned as I did tabs/spaces [16:01:26] Jeff_Green, I'll merge that change if you're able to stand by and verify that aluminium survives [16:01:26] sigh [16:01:28] yargh [16:01:29] my point is, there's no way we can review that from the diffs in gerrit [16:01:31] that took so long to do! [16:01:34] you need to do that n small steps [16:01:39] yeah I'm sorry, but that file is way too sensitive [16:01:42] yeah [16:01:43] fair [16:01:56] you've basically done a few thousand line changes in two commits that noone can review properly :) [16:01:58] andrewbogott: ok [16:02:09] concerning all access across the cluster, and even self merged [16:02:10] that is not ok [16:02:16] k, i can do the tabs/spaces thing in one commit with no alignment changes [16:02:18] (03CR) 10Andrew Bogott: [C: 032] Reconcile aluminium with standard-noexim. [operations/puppet] - 10https://gerrit.wikimedia.org/r/77135 (owner: 10Andrew Bogott) [16:02:22] not sure how I can do the reordering [16:02:23] yes please [16:02:26] and let someone review it [16:02:26] that woludn't be readable in a diff [16:02:30] sure, sorry about that [16:02:34] reordering is difficult indeed [16:02:44] i didn't think it would matter since I didn't make any functional changes, but i understant [16:02:56] yeah I know, but we can't even check [16:02:59] Jeff_Green, merged [16:03:21] i am basically trusting that you didn't know, if not I'd have to review the file anyway as the changes went out by puppet :) [16:03:29] er [16:03:35] i'm trusting that you didn't do any functional changes, I mean [16:03:44] aye, right [16:03:53] !log added my ssh public key to root user of nas1-{a,b}, nas1001-{a,b} [16:03:54] andrewbogott: running puppetd -tv and holding breath [16:04:04] Logged the message, Master [16:05:24] ok, dinner's ready [16:05:37] andrewbogott: ganglia multicast and bind addresses changed [16:05:56] I think that happens on /every/ puppet run :) [16:06:05] it got bumped from "Fundraising Eqiad" to "Miscellaneous eqiad" [16:06:11] Oh, hm. [16:06:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:07] and it got its salt::minion stuff as expected [16:07:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [16:07:16] that's it, so that's not too bad! [16:07:41] Jeff_Green, OK, I think I see why... [16:11:09] ottomata: I made a suggestion above, not sure if you saw that [16:11:37] (partially) script your changes, e.g. sed 's/\t/ /g' and put that in the commit message [16:11:50] that way we can fetch, run the script and look at a more meaningful diff [16:12:06] k, i could do that for tabs, but not reordering [16:12:31] probably not going to do it now though, i'm all annoyed with that file right now [16:12:42] hehe [16:13:22] i understand the revert and am not complaining, but aagggggh am frustrated! that took me like an hour or more [16:13:32] :( [16:14:34] (03PS1) 10Andrew Bogott: Include standard-noexim in role::fundraising::civicrm [operations/puppet] - 10https://gerrit.wikimedia.org/r/77140 [16:14:41] Jeff_Green, I think this will work ^ [16:17:17] andrewbogott: ah, makes sense. ya. +1-ing [16:17:31] (03CR) 10Jgreen: [C: 031 V: 031] Include standard-noexim in role::fundraising::civicrm [operations/puppet] - 10https://gerrit.wikimedia.org/r/77140 (owner: 10Andrew Bogott) [16:17:59] (03CR) 10Andrew Bogott: [C: 032] Include standard-noexim in role::fundraising::civicrm [operations/puppet] - 10https://gerrit.wikimedia.org/r/77140 (owner: 10Andrew Bogott) [16:18:19] Jeff_Green, ok, try it now? [16:20:22] andrewbogott: operating puppet . . . [16:20:58] better. it adjusted the salt config too [16:21:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.153 second response time [16:25:19] Jeff_Green, great. Thanks for indulging my desire for order :) [16:25:26] ha, np [16:25:44] back soon [16:29:26] recursor0 is not my friend. [16:31:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:32] what's the issue Jeff_Green? [16:31:56] mark, around? [16:32:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [16:32:25] mark, i'm not sure if you saw my comments or not before merging https://gerrit.wikimedia.org/r/#/c/76873/ [16:33:26] he replied when he merged [16:33:29] so I think he did [16:38:22] (03Abandoned) 10Ottomata: Adding account class for Toby Negrin. RT 5391 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77129 (owner: 10Ottomata) [16:43:41] (03PS1) 10Ottomata: Adding admins.pp class for Toby Negrin. RT 5391 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77142 [16:43:52] !log bsitu synchronized php-1.22wmf12/extensions/Echo/Echo.php 'Update Echo to master' [16:44:03] Logged the message, Master [16:44:09] (03CR) 10jenkins-bot: [V: 04-1] Adding admins.pp class for Toby Negrin. RT 5391 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77142 (owner: 10Ottomata) [16:48:06] !log set options disk.auto_assign off on nas1001-{a,b}. Then performed disk remove_ownership for 20 disks on nas1001-b and disk assign all on nas1001-a [16:48:16] Logged the message, Master [16:50:41] paravoid: ah just blocked on a negative-cached DNS entry [16:51:11] (03PS2) 10Ottomata: Adding admins.pp class for Toby Negrin. RT 5391 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77142 [16:51:15] on the upside it motivated me to go get lunch and now it's better :-) [16:53:29] "02:20 < jeremyb> maybe varnish can teach icinga-wm how to meditate" :) [16:53:46] (03PS1) 10Tim Landscheidt: Tools: Allow bastions to access other hosts with HBA [operations/puppet] - 10https://gerrit.wikimedia.org/r/77144 [16:56:23] (03CR) 10Ottomata: [C: 032 V: 032] Adding admins.pp class for Toby Negrin. RT 5391 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77142 (owner: 10Ottomata) [17:00:55] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours [17:05:31] notpeter, https://plus.google.com/hangouts/_/b469ae2ef119188621076aad4a20bdf3cfe22a97 [17:09:32] (03CR) 10Yuvipanda: [C: 031] role::pmtpa-proxy: file[] -> File[] [operations/puppet] - 10https://gerrit.wikimedia.org/r/77122 (owner: 10Hashar) [17:11:39] greg-g: Is Echo done with their deploy? [17:13:52] csteipp: looks like it, just double checking he isn't still testing/ [17:14:03] (03CR) 10Tim Landscheidt: [C: 04-1] ""err: /Stage[main]/Toollabs::Bastion/File[/etc/ssh/ssh_config]: Could not evaluate: Could not retrieve information from environment produc" [operations/puppet] - 10https://gerrit.wikimedia.org/r/77144 (owner: 10Tim Landscheidt) [17:14:57] csteipp: go forth [17:15:17] (03PS1) 10CSteipp: Enable use of EventLogging for SUL stats [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/77147 [17:17:16] (03PS2) 10Tim Landscheidt: Tools: Allow bastions to access other hosts with HBA [operations/puppet] - 10https://gerrit.wikimedia.org/r/77144 [17:17:57] (03CR) 10CSteipp: [C: 032] Enable use of EventLogging for SUL stats [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/77147 (owner: 10CSteipp) [17:18:15] (03Merged) 10jenkins-bot: Enable use of EventLogging for SUL stats [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/77147 (owner: 10CSteipp) [17:21:19] (03PS1) 10Jgreen: remove dhcp entry for old hostname europium [operations/puppet] - 10https://gerrit.wikimedia.org/r/77148 [17:22:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:21] (03CR) 10Jgreen: [C: 032 V: 032] remove dhcp entry for old hostname europium [operations/puppet] - 10https://gerrit.wikimedia.org/r/77148 (owner: 10Jgreen) [17:24:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.122 second response time [17:25:35] mark: what was up with 30217 ? [17:26:05] that originated from me [17:26:15] so traffic from here had two paths [17:26:30] level3 -> hostway -> pmtpa [17:26:43] the other one was cogent -> tinet -> eqiad [17:27:06] same as path length, my isp's carrier preferred the first [17:27:20] so I was going to eqiad via pmtpa, at 200ms [17:27:29] ah [17:27:31] !log csteipp synchronized wmf-config/InitialiseSettings.php 'EventLogging for CentralAuth' [17:27:34] and since we don't serve much traffic out of pmtpa these days [17:27:42] Logged the message, Master [17:27:44] nope [17:27:47] cool [17:27:49] it made sense to prepend the hostway link [17:28:09] !log csteipp synchronized wmf-config/CommonSettings.php 'EventLogging for CentralAuth' [17:28:20] Logged the message, Master [17:28:43] cool [17:31:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [17:34:14] greg-g: Done with my deploy, and I don't think I broke the cluster... yet :) [17:35:30] eventlogging is so fragile, just give it time [17:47:11] (03PS1) 10Yuvipanda: Make labs redis be limited to 7G, not 1G [operations/puppet] - 10https://gerrit.wikimedia.org/r/77152 [17:52:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [17:55:56] PROBLEM - Puppet freshness on mchenry is CRITICAL: No successful Puppet run in the last 10 hours [17:57:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.141 second response time [18:03:50] (03PS2) 10Yuvipanda: Make labs redis memory configurable [operations/puppet] - 10https://gerrit.wikimedia.org/r/77152 [18:04:01] andrewbogott: can you take a look at that? [18:04:01] ^ [18:04:19] andrewbogott: I am following the wikitech document for adding an inputbox to the wikitech interface [18:05:06] YuviPanda, we don't really support parameters in the GUI right now... [18:05:33] the best thing to do for configuration is to add a role that uses a global to configure a subclass [18:05:34] andrewbogott: really? https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster#FAQ says otherwise... [18:05:39] specifically How do I add input fields in labsconsole and hand over parameters to my puppet manifests ? [18:06:35] I'm pretty sure that doc is using 'paramter' differently, it doesn't mean with parameterized classes. [18:07:18] Misleading… I think Ottomata wrote that, you might check in with him about what he meant. [18:08:58] hey andrewbogott, e3 & analytics agreed to retire e3-logbot and to share analytics-logbot by having it present in both #wikimedia-analytics and #wikimedia-e3. are you the right person to poke about that? should I file an RT ticket or a bug in bugzilla? [18:09:34] andrewbogott: I... see [18:09:46] andrewbogott: so I should just have 'redis::small' and 'redis::large'? [18:10:11] YuviPanda: those sound like roles [18:10:32] yeah, but I see input boxes in the wikitech interface [18:10:41] would've been nice to just have the amount of memory be specifiable [18:13:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:14:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [18:16:43] YuviPanda, you can create a role that uses a global, and then configure the global via the wikitech interface... [18:16:49] that's what that doc section is talking about [18:16:55] I'm in a meeting, sorry, distracted... [18:18:29] andrewbogott: hmm, I see. I'm just going to create redis::small and redis [18:18:37] and apply redis:: small to tools-mc [18:22:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [18:23:24] (03PS3) 10Yuvipanda: Add roles for a small and large redis instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/77152 [18:24:05] Coren: andrewbogott Ryan_Lane thoughts? ^ [18:24:30] have roles for small instance (tools-mc) and large (tools-redis) [18:24:33] small should go away at some point [18:24:43] YuviPanda: why not just a role that uses a global variable? [18:25:28] I've not done too much puppet, but global icky? also I've no testing environment so not too comfortable doing that... [18:25:45] plus tools-mc should go away in a month. when tools-redis comes online, I'll post to labs-l setting a date for tools-mc to die. [18:29:30] YuviPanda: it's not icky [18:29:51] you'd use the global variable only in the role [18:29:58] and would pass the value into the param [18:30:38] YuviPanda: Can't you test it on toolsbeta? [18:30:50] scfc_de: needs to get merged for me to test, no? [18:30:54] since this is in the wikitech thing [18:31:06] YuviPanda: Uh. Hmmm. Never tried. [18:31:10] Ryan_Lane: can you tell me some other class that uses this approach? [18:31:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:31:29] Ryan_Lane: *Any* global can be set up as a wikitech input field, can't it? [18:31:35] scfc_de: yep [18:31:56] YuviPanda: So you should be able to pull that off. [18:31:59] it's testable using puppetmaster::self [18:32:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.137 second response time [18:32:38] I wonder if we can add a host before wikimania [18:32:59] I think we've used all of them, though [18:33:53] YuviPanda: are you asking about labs puppetmasters + global vars + role classes? [18:33:59] I use them a bunch, if you need some tips [18:34:33] ottomata: yes! [18:35:09] ottomata: I've a toollabs::redis class, and a labs::tools::redis role. I want to be able to specify a variable (maxmemory) for it. [18:35:14] ottomata: currently I have https://gerrit.wikimedia.org/r/#/c/77152/ which just adds new roles [18:35:20] trying to figure out how to make that into a variable [18:36:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:45] oh yeah [18:37:10] YuviPanda: [18:37:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [18:37:28] all these classes are only supposed to be used in labs? [18:37:30] is that correct? [18:37:34] yeah [18:37:34] oh, ha, it is labs.pp [18:37:35] so yeah [18:37:37] ok yeah [18:37:48] ok just have the one class redis inherits [18:37:50] and do [18:38:01] maxmemory => $::redis_maxmemory [18:38:11] (or whatever you want to call the global var) [18:38:22] then, in your labs project [18:38:32] go to [18:38:34] Manage Puppet Groups [18:38:53] you can add a class [18:38:56] and a varaible [18:39:02] your class is whatever your class name is [18:39:03] labs::redis [18:39:04] or wahtever [18:39:12] and then the variable would be the name of that global [18:39:15] redis_maxmemory [18:39:22] then, go to the Instance configuration page [18:39:29] check the box next to labs::redis to include the class [18:39:39] and fill in the redis_maxmemory field with whatever [18:39:42] value you want [18:39:48] super! [18:39:54] but the patch will have to be first merged for that, right? [18:39:58] yes [18:40:00] or [18:40:05] you could set up role::puppet::self [18:40:07] to test puppet changes [18:40:08] ottomata: also, I refer to the global redis_maxmemory from the role, right? (in labs.pp) [18:40:12] yes [18:40:26] øk yeah, so I guess your classname would be role::labs::redis [18:40:29] so something like [18:40:59] role::labs::redis inherits … { [18:41:00] class { 'toollabs::redis': [18:41:00] maxmemory => $::redis_maxmemory [18:41:00] } [18:41:00] } [18:41:06] !log authdns update [18:41:16] Logged the message, Master [18:41:27] ottomata: yeah, patch coming up [18:41:41] grrr git review. /me stabs it [18:42:15] (03PS4) 10Yuvipanda: Add a global redis_maxmemory to configure maxmemory of redis instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/77152 [18:42:21] ottomata: ^ [18:43:47] cool, one sec, phone [18:44:21] ok [18:46:41] ottomata: Just to clarify: The wikitech variables can have arbitrary names (for example, used in a pending patch) and do not need to correlate to the HEAD of the production branch? [18:47:10] yes, that's true [18:47:25] you won't cause any errors by setting random variables [18:47:47] you will cause errors if you try to include a class that hasn't been merged yet [18:48:52] (03PS1) 10Ottomata: Adding account for Subramanya Sastry and adding to admins::parsoid. RT 5512 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77161 [18:49:17] (03CR) 10jenkins-bot: [V: 04-1] Adding account for Subramanya Sastry and adding to admins::parsoid. RT 5512 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77161 (owner: 10Ottomata) [18:49:35] (03PS2) 10Ottomata: Adding account for Subramanya Sastry and adding to admins::parsoid. RT 5512 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77161 [18:50:12] (03CR) 10jenkins-bot: [V: 04-1] Adding account for Subramanya Sastry and adding to admins::parsoid. RT 5512 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77161 (owner: 10Ottomata) [18:50:35] ottomata: can you +1 / +2 that patch? :) [18:52:10] (03CR) 10Ottomata: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/77152 (owner: 10Yuvipanda) [18:52:20] one comment YuviPanda, about class name [18:52:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:44] !log created aggregates backups on nas1001-a and nas1-a. Created flexible volumes baculasd1 on each and setup a snapmirror relationship between them (from eqiad to pmtpa) [18:52:55] Logged the message, Master [18:52:58] ottomata: I was just doing what every other thing there does, no? [18:53:10] mailrelay is toollabs specific, but has its name to be just 'mailerlay' [18:53:15] system_role seems to specify full name [18:53:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [18:53:27] (03PS3) 10Ottomata: Adding account for Subramanya Sastry and adding to admins::parsoid. RT 5512 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77161 [18:53:43] OH! [18:53:49] sorry, YuviPanda, you are right [18:53:52] I didn't look at the whole file [18:53:55] these are nested class defs [18:53:59] which in general are bad [18:54:02] but you shouldn't change that for this commit [18:54:06] yeah [18:54:12] that's cool [18:54:17] ok i can merge for you [18:54:19] \o/ [18:54:33] scfc_de: once that gets merged, I suppose you should add the variable in wikitech config [18:56:34] (waiting for jenkins bot to say ok) [18:57:24] ottomata: I see a +2 [18:57:25] from it [18:57:44] ottomata: jenkins +2s aren't reported in channel :) only -1 and -2 are [18:58:34] mm, redis! [18:58:48] (03CR) 10Ottomata: [C: 032 V: 032] Add a global redis_maxmemory to configure maxmemory of redis instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/77152 (owner: 10Yuvipanda) [18:58:52] ty! [18:59:51] yup! [19:00:28] scfc_de: can you add the 'variable' in the interface? [19:00:33] scfc_de: ottomata explained it in scrollback [19:02:08] (03PS1) 10Ryan Lane: Add hack for using qcow2 ephemeral base images [operations/puppet] - 10https://gerrit.wikimedia.org/r/77162 [19:04:14] (03CR) 10jenkins-bot: [V: 04-1] Add hack for using qcow2 ephemeral base images [operations/puppet] - 10https://gerrit.wikimedia.org/r/77162 (owner: 10Ryan Lane) [19:05:16] (03PS1) 10Ottomata: Forgot to add Toby to admins::restricted. RT 5391 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77163 [19:06:12] hah, our pep8 check is more stringent than openstack's [19:06:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:06:27] pep8 no longer says 79 chars!!!11one [19:06:48] (03PS2) 10Ryan Lane: Add hack for using qcow2 ephemeral base images [operations/puppet] - 10https://gerrit.wikimedia.org/r/77162 [19:07:06] * YuviPanda pokes scfc_de [19:07:13] well, now I'm hacking around that [19:07:17] PROBLEM - Puppet freshness on holmium is CRITICAL: No successful Puppet run in the last 10 hours [19:07:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.149 second response time [19:07:20] no pep8 check on files that aren't .py :D [19:07:37] (03CR) 10Ottomata: [C: 032 V: 032] Forgot to add Toby to admins::restricted. RT 5391 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77163 (owner: 10Ottomata) [19:11:41] (03PS1) 10Hashar: powerdns: pass pep8 on selective-answer.py [operations/puppet] - 10https://gerrit.wikimedia.org/r/77164 [19:12:29] (03PS2) 10Hashar: powerdns: pass pep8 on selective-answer.py [operations/puppet] - 10https://gerrit.wikimedia.org/r/77164 [19:13:34] YuviPanda: Sorry, was AFK. [19:13:49] (03CR) 10Ryan Lane: [C: 032] Add hack for using qcow2 ephemeral base images [operations/puppet] - 10https://gerrit.wikimedia.org/r/77162 (owner: 10Ryan Lane) [19:13:51] (Never done that, so ...) [19:13:59] scfc_de: :D [19:15:03] scfc_de: I guess ottomata can help with issues/questions [19:16:00] YuviPanda: "Added Puppet variable". [19:16:24] YuviPanda: Hmmm. That's all, isn't it? [19:16:42] scfc_de: yeah, you should see a textbox under the redis role in the manage instances pacge [19:16:57] YuviPanda: Shows up in tools-redis Configure. "7G" or what's the syntax? [19:17:03] 7G yeah [19:17:39] scfc_de: you need to also put 1G for tools-mc [19:17:43] YuviPanda: "Successfully modified instance i-00000876 (tools-redis)." Let's trigger a manual Puppet run. [19:17:48] scfc_de: \o/ [19:17:59] YuviPanda: But 1G is default anyway?! [19:18:18] not sure how that behaves when the global is not set [19:18:23] ottomata: what happens when the global is not set? [19:18:27] No default. [19:18:37] 1G is default, yeah [19:18:40] ja, probably just undef [19:18:41] because that's the default in production [19:18:43] buuuuut naw i think [19:18:51] it will be undef, because you are passing the value as the param [19:18:51] so [19:18:59] you should probably explicitly set the default in your role class too [19:19:01] maybe [19:19:34] $maxmemory => $::redis_maxmemory ? { [19:19:34] undef => "1GB", [19:19:34] default => $::redis_maxmemory, [19:19:34] } [19:19:34] class … { [19:19:35] maxmemory => $maxmemory, [19:19:35] } [19:19:35] YuviPanda: "-maxmemory 1GB", "+maxmemory 7G". Sure it's "7G" and not "7GB"? [19:19:40] !log authdns update [19:19:43] scfc_de: let me check [19:19:49] scfc_de: ou, yeah, 1GB [19:19:51] sorry [19:20:24] YuviPanda: Should I run Puppet on tools-mc to see what happens? [19:20:55] scfc_de: perhaps. it shouldn't restart the redis service, so it should be fine [19:21:35] tool-redis: "-maxmemory 7G", "+maxmemory 7GB". [19:21:44] !log morebots are you working? [19:21:46] scfc_de: hmm? [19:21:54] Logged the message, Master [19:22:39] YuviPanda: I changed the configuration to "7G*B*", and Puppet changed /etc/redis/redis.conf accordingly. [19:22:59] ah, that sounds okay [19:23:04] scfc_de: this is on tools-redis, right? [19:23:09] tools-mc being set to 7GB is a bad idea :D [19:23:46] YuviPanda: Yes. [19:23:51] ah, okay :) [19:25:05] YuviPanda: tools-mc puppeted itself before I logged in, and /var/log/puppet.log doesn't refer to /etc/redis/.... Hmmm. [19:25:23] scfc_de: yeah, I guess no change? [19:26:49] scfc_de: hmm, so tools-mc still works, and tools-redis is up too? [19:27:12] YuviPanda: Yes. Re tools-mc: [19:27:14] "notice: /Stage[main]/Redis/File[/etc/redis/redis.conf]: Dependency Exec[/usr/bin/apt-get update] has failures: true" [19:27:21] "warning: /Stage[main]/Redis/File[/etc/redis/redis.conf]: Skipping because of failed dependencies" [19:27:26] what failed?! [19:27:29] "notice: /Stage[main]/Redis/Service[redis]: Dependency Exec[/usr/bin/apt-get update] has failures: true" [19:27:33] perhaps the lack of the global? [19:27:36] "warning: /Stage[main]/Redis/Service[redis]: Skipping because of failed dependencies" [19:27:41] network failure, perhap? [19:27:47] idk [19:28:09] scfc_de: do a manual run? [19:32:29] YuviPanda: "puppetd -tv": "notice: /Stage[first]/Apt::Update/Exec[/usr/bin/apt-get update]/returns: executed successfully", no errors. [19:32:59] anything changed for redis? [19:34:59] !log Slightly described the Zuul pipelines at https://integration.wikimedia.org/zuul/ . Descriptions are in the zuul-config layout.yaml file. [19:35:09] Logged the message, Master [19:35:28] YuviPanda: No. http://pastebin.com/b06Qz518 [19:35:56] scfc_de: meh, looks fine to me :) [19:39:27] !log dns update [19:39:37] Logged the message, Master [19:43:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:44:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [19:56:35] Ryan_Lane, so, questions about the puppet loader. Is everything in /manifests loaded because that's what puppet does? Or because we enumerate all those files someplace? [19:56:50] oops, not here [19:59:03] andrewbogott: puppetjust load site.pp [19:59:26] andrewbogott: and our include the manifests and some wildcard such as include "role/*.pp" [19:59:54] so whenever you run puppet, even to have a simple file {} definition applied, it ends up parsing ALL the /manifests/ .pp :( [20:00:05] but modules will saves us [20:01:50] OK, I was wondering why base.pp explicitly included some things. [20:02:08] Of course, now I'm moving base to a module and still need to include things from manifests, which might be… wrong? impossible? [20:04:07] Ryan_Lane, do you know if it's possible to import things from manifests/ inside a module? [20:04:19] Clearly it's a bad thing, but might be needed for transitional purposes. [20:09:29] it is [20:09:37] it's just not great as you say [20:11:16] paravoid, what's the syntax? [20:11:32] you don't need to import it, it's imported from site.pp [20:11:34] and just works [20:11:48] Oh, well… in this case it's for things which aren't imported by site.pp. [20:11:51] But I guess I can add them there. [20:12:03] (Some things are imported in site.pp thanks to base.pp) [20:15:55] Ah, dammit, it's dependant on the order that files are imported. [20:16:43] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [20:18:16] !log authdnsupdate [20:18:28] Logged the message, RobH [20:21:26] (03PS1) 10RobH: mexia to public ip [operations/puppet] - 10https://gerrit.wikimedia.org/r/77211 [20:26:44] andrewbogott: good luck :( [20:27:11] Does anyone know where 'realm' is first defined? [20:31:11] (03CR) 10RobH: [C: 032] mexia to public ip [operations/puppet] - 10https://gerrit.wikimedia.org/r/77211 (owner: 10RobH) [20:31:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:32:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [20:33:23] andrewbogott: realm.pp ? [20:33:40] if !$::realm { [20:41:24] (03CR) 10BryanDavis: [C: 031] "I've only got +1 to give but I will gladly give it." [operations/puppet] - 10https://gerrit.wikimedia.org/r/77123 (owner: 10Hashar) [20:41:43] i love this bot [20:41:46] YuviPanda: ^^^:) [20:41:55] :D ty, hashar :) [20:42:11] YuviPanda: knowing when someone review a change is really nice [20:42:22] hashar: that was there in the previous one too, no? [20:42:24] YuviPanda: i also noticed you restored the notification on merge. [20:42:26] ah, it pings *you* now [20:42:29] yeah [20:42:31] hashar: hmm? no, I haven't [20:42:33] as the owner [20:42:40] it still only notifies when jenkins-bot merges [20:42:42] since those are async [20:42:44] will work as long as the owner name is hashar which is my nick :-] [20:42:47] any other merges are not notified [20:43:09] hashar: yeah, would be nice if we can find some way to associate gerrit names to irc nicks :D [20:44:20] YuviPanda: LDAP/wiki preferences? [20:44:22] YuviPanda: +1 for nick to gerrit name mapping [20:44:33] scfc_de: can they be 'trivially' added? [20:44:49] it can't really go into LDAP [20:45:01] we should have user profiles on wikitech [20:45:11] isn't that something that qgil was working on? [20:45:12] where the profile attributes are semantic [20:45:14] yep [20:45:22] !hss [20:45:22] Ryan_Lane: stop trying to make wikitech facebook. [20:45:25] * RobH runs away [20:45:27] ;] [20:45:28] hahaha [20:45:42] greg-g: alternatively wikidata [20:45:50] when wikidata can do queries [20:45:52] Ryan_Lane: yeah, I like that idea [20:46:02] wikidata+lua = glorious future [20:46:06] once wikidata can do queries I'm getting rid of SMW [20:46:10] totally agree [20:46:19] SMW makes wikitech less secure, too [20:46:24] :( [20:46:40] any idea when OAuth is supposed to turn up? [20:46:51] all these things can be hacked into the tools themselves once we have OAuth [20:47:16] from [[:wikitech:Deployments]] "August 19th - 9am Pacific, OAuth deploy to testwikis (Chris S) " [20:47:28] so, after that :) [20:47:30] nice! [20:47:33] soon enough, then [20:47:42] hashar: btw, the 'owner:' field was marktraceur's idea :) [20:47:51] * marktraceur bows [20:48:49] let's use RT articles instead of wikitech [20:49:00] * mutante runs away too [20:49:35] * YuviPanda makes mutante pass RT through the design team [20:49:39] Gerrit is sloooow [20:49:40] Ryan_Lane: just curious, why can't irc nicks go in ldap? [20:49:41] marktraceur: owe you a beer [20:49:56] well I owe everyone so many beer that I will end up broken soon [20:50:01] bd808: what attribute would that use? [20:50:06] Krinkle: what does dbbot-wm do? [20:50:20] hashar: Me and YuviPanda both :) I think we'll count it even for everything I owe you re: Jenkins, and we can just each buy our own beers [20:50:24] mutante, except that always sets it to production if unset… presumably there are other possible values (e.g. 'labs') that are set someplace else? [20:50:28] YuviPanda: OATH is targeted within a month I believe. csteipp ? [20:50:31] Ryan_Lane: any arbitrary one added to the schema for that purpose? [20:50:36] marktraceur: sounds like a good plan [20:50:43] Ryan_Lane: OATH or OAuth? OATH is the 2factor auth thing, right? [20:50:45] OATH or OAuth? [20:50:57] whoops [20:50:59] OAuth is this month [20:50:59] OAuth [20:51:04] * hashar wonders why OATH is not named 2factor [20:51:10] bd808: using random attrs is evil [20:51:12] not confusing enough, clearly :P [20:51:26] bd808: also, it's only useful to put stuff into LDAP that will be re-used by other ldap enabled applications [20:51:33] Hey Ryan_Lane / Coren: Can one or both of you provide a Labs / Toolserver update for https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/July#Technical_Operations ? Or was this traditionally done by someone else? [20:51:39] can tools on toollabs become OAuth enabled applications? [20:51:42] err [20:51:45] s/OAuth/LDAP/ [20:51:49] if we can put it into wikitech and query it via SMW, that should be good enough [20:51:58] ksnider: It's usually both of us giving some bits. [20:51:59] ewww [20:52:02] let's please not [20:52:10] @docs [20:52:10] Krinkle: https://www.mediawiki.org/wiki/dbbot-wm [20:52:11] I'd rather them do MW queries [20:52:12] @info s1 [20:52:13] Krinkle: [s1] db1056: 10.64.32.26, db1043: 10.64.16.32, db1049: 10.64.16.144, db1050: 10.64.16.145, db1051: 10.64.32.21, db1052: 10.64.32.22 [20:52:16] @info centralauth [20:52:16] Krinkle: [centralauth: s7] db1041: 10.64.16.30, db1007: 10.64.0.11, db1024: 10.64.16.13, db1028: 10.64.16.17 [20:52:18] @replag all [20:52:19] Krinkle: [s1] db1056: 0s, db1043: 0s, db1049: 0s, db1050: 0s, db1051: 0s, db1052: 0s; [s2] db1034: 0s, db1002: 0s, db1009: 0s, db1018: 0s; [s3] db1019: 0s, db1003: 0s, db1010: 0s, db1035: 3s [20:52:19] Coren: Awesome. Thanks in advance :) [20:52:20] Krinkle: [s4] db1059: 0s, db1004: 0s, db1011: 0s; [s5] db1058: 0s, db1005: 0s, db1026: 0s, db1021: 0s; [s6] db1006: 0s, db1022: 0s, db1027: 0s, db1040: 0s; [s7] db1041: 0s, db1007: 0s, db1024: 0s, db1028: 0s [20:52:20] YuviPanda: [20:52:27] ah, alrighrt. ty Krinkle [20:52:30] YuviPanda: I'm exposing most important LDAP info via wikitech apis anyway [20:52:30] @replag db1040 [20:52:30] Krinkle: [db1040: s6] db1040: 0s [20:52:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:52:34] mutante, specifically, I rearranged the order of some file imports (but still leaving realm.pp at the top) and suddenly my labs system thinks it's production. [20:52:48] (also https://www.mediawiki.org/wiki/Wikimedia_Labs#Status) [20:53:09] ksnider: generally coren, andrewbogott and I update that [20:53:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [20:53:25] hashar: "OATH" == "Open AuTHentication" http://www.nongnu.org/oath-toolkit/ [20:53:35] hashar: yucky codename [20:53:46] hehe, not similar to OAuth at all! [20:53:52] I'd rename OATH to Two Factor Authentication, but someone forked my code, used an incompatible license and used that name [20:53:57] bd808: I am a complete noob regarding all those subjects. I entirely rely on csteipp when it comes to authentication / login / sul whatever :( [20:54:20] Ryan_Lane: btw, not sure if we saw - we've a proper and large tools-redis now :) [20:54:20] Ryan_Lane: http://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&h=labstore3.pmtpa.wmnet&m=cpu_report&s=descending&mc=2&g=cpu_report&c=Labs+NFS+cluster+pmtpa [20:54:29] Ryan_Lane: Is it just me or are things getting worse? [20:54:30] YuviPanda: yep :) [20:54:32] well I am off [20:54:39] see you tomorrow or later :-] [20:54:46] Coren: likely with increased use [20:54:51] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [20:54:57] Coren: murphy's law states that the NFS will go completely dead when you/someone is presenting about toollabs at wikimania :) [20:55:24] * Coren considers some workarounds in the meantime. [20:55:56] Ryan_Lane: my ldap fu is weak sauce. I think that mozilla.org has irc nick in their schema but only because google told me that. [20:56:30] Ryan_Lane, do you recall where/when $::realm gets set to 'labs' in labs? [20:56:43] andrewbogott: directly in the nodes [20:56:57] it's a global variable force set by wikitech [20:57:05] really? Then why does my labs machine suddenly think it's in production? [20:57:27] it's grown sentient! [20:57:36] uhoh, I've been mistreating it [20:57:39] :D [20:57:54] feed it a cookie [20:58:02] Ryan_Lane, instance puppet-testing-3 [20:58:14] "it's got the cookie monster virus!" "Well, feed it a cookie" [20:58:39] "but that will kill our cache!" [20:58:50] ooohhhh hackers [20:59:00] I loved putting that tsr on lab machines in undergrad [20:59:12] oh, that's a movie reference? [20:59:28] hearing a 286 croak "feed me a cookie" from pc speaker was awesome [20:59:35] andrewbogott: puppetVar: realm=labs [20:59:41] it's set in LDAP [20:59:48] andrewbogott: it's using puppetmaster::self? [20:59:55] * YuviPanda hasn't done anything more fun than kill -9 other people's vim instances just before they do :w [20:59:55] yep [20:59:56] are you defining a node in site.pp? [20:59:59] nope [21:00:11] oh [21:00:22] is it handled anywhere in base.pp? [21:00:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:36] if !$::realm { [21:00:36] $realm = "production" [21:00:36] } [21:00:45] though $::realm should exist [21:00:54] hrm, gerrit does not want to accept my submission [21:00:58] internal server error [21:01:01] maybe that doesn't work the same way in modules? [21:01:04] I'm testing a patch that moves base into a module. So it changes the order of operations. [21:01:13] Oh, like, it isn't reading the global from ldap? [21:01:32] maybe it does it before it reads them from ldap? [21:02:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [21:02:50] overage load time for gerrit: 12 seconds per http request [21:03:14] Will I get an email when my request to join ops@ list is approved by moderator? [21:03:43] depends on the moderator [21:03:48] but you should, yes [21:03:59] * YuviPanda didn't when he got on ops@ [21:04:05] Krinkle: thanks [21:04:08] I take it gerrit is down? [21:04:20] yep [21:04:22] ugh [21:04:23] dead for me too [21:04:38] it was dog slow a few minutes ago [21:04:40] !log Gerrit has been slow (~ 10 second per request) for about 30 minutes. Seems down entirely now. [21:04:51] Logged the message, Master [21:05:01] ^d's at an appointment, just fyi [21:05:19] Ryan_Lane: paravoid: [21:06:30] Ryan_Lane, actually, I'm still including realm.pp at the very top of site.pp. [21:06:41] hm [21:06:42] weird [21:06:58] Yeah, unless… are other classes defined in ldap that get loaded before site.pp? [21:08:04] !log dns update [21:09:04] https://en.wikipedia.org/wiki/Cookie_Monster_(computer_program) http://www.imdb.com/title/tt0113243/ featuring young Jonny Lee Miller ("Elementary" TV Show) and young Angelina Jolie [21:09:31] andrewbogott: hm. it's a global variable directly in the node definition, it should definitely be getting set first [21:09:34] (03PS1) 10Lcarr: fixing lvs1001's public-c-eqiad vlan ip [operations/puppet] - 10https://gerrit.wikimedia.org/r/77224 [21:09:41] I hate puppet :( [21:09:44] yeah [21:09:58] maybe I'm not checking its emptyness properly? [21:10:03] !log restarting gerrit on manganese [21:10:14] Logged the message, Mistress of the network gear. [21:10:52] (03CR) 10Lcarr: [C: 032 V: 032] fixing lvs1001's public-c-eqiad vlan ip [operations/puppet] - 10https://gerrit.wikimedia.org/r/77224 (owner: 10Lcarr) [21:11:37] (03PS1) 10Ottomata: Replicating operations/puppet/kafka to github wikimedia/puppet-kafka [operations/puppet] - 10https://gerrit.wikimedia.org/r/77225 [21:11:47] Krinkle: I bet I have the dos trs version on a 5 1/4" floppy somewhere [21:11:56] *tsr [21:12:18] * YuviPanda gets off bd808's lawn [21:12:50] grrrit-wm is stable across gerrit restarts! \o/ [21:12:58] Yay, Gerrit works. Much obliged, LeslieCarr. [21:13:07] :) [21:14:38] (03PS2) 10Ottomata: Replicating operations/puppet/kafka to github wikimedia/puppet-kafka [operations/puppet] - 10https://gerrit.wikimedia.org/r/77225 [21:14:41] * bd808 shakes fist and head at those darn kids [21:14:46] (03CR) 10Ottomata: [C: 032 V: 032] Replicating operations/puppet/kafka to github wikimedia/puppet-kafka [operations/puppet] - 10https://gerrit.wikimedia.org/r/77225 (owner: 10Ottomata) [21:16:24] i need "npm" installed on Debian to be able to build etherpad-lite, but npm : Depends: nodejs (>= 0.6.19~dfsg1-3) but it is not installable, because it's just in sid .. sigh [21:16:27] (03PS1) 10Jgreen: start puppetizing new OTRS host iodine [operations/puppet] - 10https://gerrit.wikimedia.org/r/77227 [21:18:34] so a package manager that is there to "manage dependecy conflicts intelligently" can't be installed due to dependency conflicts in the package manager you already have, yay [21:18:47] (03PS1) 10Cmjohnson: adding ssl1007-9 to node [operations/puppet] - 10https://gerrit.wikimedia.org/r/77228 [21:19:28] * YuviPanda adds that to quips [21:19:38] (03CR) 10Lcarr: [C: 031] adding ssl1007-9 to node [operations/puppet] - 10https://gerrit.wikimedia.org/r/77228 (owner: 10Cmjohnson) [21:19:46] (03CR) 10Cmjohnson: [C: 032 V: 032] adding ssl1007-9 to node [operations/puppet] - 10https://gerrit.wikimedia.org/r/77228 (owner: 10Cmjohnson) [21:22:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:23:14] lesliecarr: i can't merge that change [21:23:18] or did you do it ? [21:23:22] !log mflaschen synchronized php-1.22wmf12/extensions/GuidedTour/ 'Sync GuidedTour for E3 deployment' [21:23:33] Logged the message, Master [21:23:54] cmjohnson1: i didn't [21:23:57] looks like it needs rebase [21:24:13] (03PS2) 10Dzahn: adding ssl1007-9 to node [operations/puppet] - 10https://gerrit.wikimedia.org/r/77228 (owner: 10Cmjohnson) [21:24:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [21:24:24] (03CR) 10Dzahn: [C: 032] adding ssl1007-9 to node [operations/puppet] - 10https://gerrit.wikimedia.org/r/77228 (owner: 10Cmjohnson) [21:24:35] (03CR) 10Dzahn: [V: 032] adding ssl1007-9 to node [operations/puppet] - 10https://gerrit.wikimedia.org/r/77228 (owner: 10Cmjohnson) [21:24:43] cmjohnson1: done:) [21:24:51] thx [21:27:01] (03PS1) 10Ottomata: Updating link in README.md [operations/puppet/kafka] - 10https://gerrit.wikimedia.org/r/77230 [21:27:11] (03CR) 10Dzahn: [C: 032] rt: package[] -> Package[] [operations/puppet] - 10https://gerrit.wikimedia.org/r/77119 (owner: 10Hashar) [21:27:12] (03CR) 10Ottomata: [C: 032 V: 032] Updating link in README.md [operations/puppet/kafka] - 10https://gerrit.wikimedia.org/r/77230 (owner: 10Ottomata) [21:31:20] Hey can someone here give me access to rt.wikimedia.org? [21:32:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:32:47] robla told me to get access to RT as well [21:33:03] * bd808 needs more things to remember [21:33:55] There were a few failed servers on the sync-dir: [21:34:12] srv281, mw1089, and mw1173 [21:34:20] You might want to take them out of rotation. [21:35:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.995 second response time [21:35:44] kenanwang, you're a mobile product manager? I'm not familiar with WMF's internal structure but maybe ask someone like Erik about that [21:39:27] (03PS1) 10Tim Landscheidt: Tools: Manage obsolete files in /usr/local/bin [operations/puppet] - 10https://gerrit.wikimedia.org/r/77234 [21:40:20] Krenair: yes I'm the mobile pm [21:40:29] (03CR) 10Tim Landscheidt: [C: 04-1] "Needs to be tested first on toolsbeta." [operations/puppet] - 10https://gerrit.wikimedia.org/r/77234 (owner: 10Tim Landscheidt) [21:41:00] Krenair: the office wiki tells me to go onto operations irc or wikitech irc to get access [21:41:10] oh, ok [21:41:32] kenanwang: access to .. ? [21:41:36] Hey can someone here give me access to rt.wikimedia.org? [21:41:42] Krenair: the request tracking system [21:41:48] Yes I know what RT is [21:42:00] sorry I meant that for mutante [21:42:08] mutante didn't see your original request [21:42:26] kenanwang: oh, it should actually tell you to send a mail to RT itself, but lemme create that for you [21:42:42] kenanwang: do you already have email @wikimedia? [21:43:21] mutante: thanks, yes it's kwang@wikimedia.org [21:43:49] mutante: here is the wikitech page that gives me the instructions in case you want to change it https://wikitech.wikimedia.org/wiki/Requesting_shell_access [21:44:10] kenanwang: bd808: it would be great if you can just send a quick mail to the "requests" mail address in the topic and i'll set you up [21:44:42] kenanwang: oh, that is for shell access , it's a different thing [21:45:05] kenanwang: do you think you're going to need actual shell access to deploy? [21:45:06] mutante: there's a section that says "The process is handed within our request tracking (RT) system. The ticket should be filed under access-requests. If you do not have access to this system, you should discuss your access request in freenode IRC in #wikimedia-tech or #wikimedia-operations." [21:45:25] ah, gotcha, appending to that [21:45:27] mutante: i need the rt access to request shell access so that I can get access to state one [21:45:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:45:31] stat1 [21:46:39] i see, hold on [21:46:56] mutante: email sent. thanks for the clarification [21:47:12] cool, np [21:47:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 7.929 second response time [21:49:39] mutante: so just for clarify do you want me to send an email? [22:05:47] There seems to be some mistake at https://wikitech.wikimedia.org/wiki/Main_Page, in the left 'production' box [22:05:57] OSs: should probably be its own subheading, not uner 'backups' [22:06:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:07:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [22:08:43] YuviPanda, {{sofixit}}:) [22:09:03] oh wait I do have perms to edit main page! [22:09:42] down the rabbit hole, three templates followed so far [22:09:52] 4! [22:11:04] MaxSem: fixed [22:11:27] :) [22:17:55] BTW, re Puppet, does someone know if you can reference the title of a file if you use "file { [ 'a', 'b', 'c': ... ] }"? Cf. http://serverfault.com/questions/528119/reference-each-title-in-array-of-titles [22:18:22] !log authdns update [22:29:01] PROBLEM - Puppet freshness on sq41 is CRITICAL: No successful Puppet run in the last 10 hours [22:31:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:32:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [22:50:04] (03PS1) 10Cmjohnson: fixing iodine entry adding ssl1007-9 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77251 [22:50:49] (03CR) 10Cmjohnson: [C: 032 V: 032] fixing iodine entry adding ssl1007-9 [operations/puppet] - 10https://gerrit.wikimedia.org/r/77251 (owner: 10Cmjohnson) [23:08:26] What the heck, it is loading realm.pp before site.pp