[00:01:43] RECOVERY - puppet last run on ms1001 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [00:02:24] (03PS3) 10Ori.livneh: mediawiki::web: compatability fixes for apache2.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/147994 [00:02:32] PROBLEM - puppet last run on aluminium is CRITICAL: CRITICAL: Complete puppet failure [00:06:43] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: Complete puppet failure [00:14:22] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sun 20 Jul 2014 22:13:21 UTC [00:17:22] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Mon Jul 21 00:17:16 UTC 2014 [00:18:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [00:18:22] PROBLEM - Puppet freshness on nickel is CRITICAL: Last successful Puppet run was Sun 20 Jul 2014 22:17:50 UTC [00:19:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.059 second response time [00:21:32] RECOVERY - puppet last run on aluminium is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [00:28:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [00:29:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.050 second response time [00:29:42] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: Complete puppet failure [00:35:41] (03PS4) 10Ori.livneh: mediawiki::web: compatability fixes for apache2.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/147994 [00:38:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [00:38:22] RECOVERY - Puppet freshness on nickel is OK: puppet ran at Mon Jul 21 00:38:19 UTC 2014 [00:39:02] RECOVERY - puppet last run on nickel is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [00:43:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.025 second response time [00:45:13] PROBLEM - puppet last run on tridge is CRITICAL: CRITICAL: Complete puppet failure [00:48:43] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [00:52:53] RECOVERY - puppet last run on sanger is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [00:56:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [01:01:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.019 second response time [01:04:36] (03PS1) 10Ori.livneh: beta: add deployment-mediawiki{01,02} to scap targets [operations/puppet] - 10https://gerrit.wikimedia.org/r/148001 [01:04:43] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [01:05:17] (03CR) 10Ori.livneh: "cherry-picked in beta" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148001 (owner: 10Ori.livneh) [01:05:52] (03CR) 10Ori.livneh: "cherry-picked in beta" [operations/puppet] - 10https://gerrit.wikimedia.org/r/147992 (owner: 10Ori.livneh) [01:06:15] (03CR) 10Ori.livneh: "cherry-picked in beta" [operations/puppet] - 10https://gerrit.wikimedia.org/r/147994 (owner: 10Ori.livneh) [01:09:13] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [01:09:22] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Last successful Puppet run was Sun 20 Jul 2014 23:08:26 UTC [01:11:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.049 second response time [01:15:52] RECOVERY - puppet last run on linne is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [01:16:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [01:16:34] (03PS6) 10Ori.livneh: jobrunner: create hhvm-only jobrunners [operations/puppet] - 10https://gerrit.wikimedia.org/r/147086 (owner: 10Giuseppe Lavagetto) [01:17:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.015 second response time [01:18:29] (03CR) 10BryanDavis: beta: add deployment-mediawiki{01,02} to scap targets (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148001 (owner: 10Ori.livneh) [01:21:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [01:21:52] PROBLEM - puppet last run on es4 is CRITICAL: CRITICAL: Complete puppet failure [01:22:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.024 second response time [01:23:32] PROBLEM - puppet last run on aluminium is CRITICAL: CRITICAL: Complete puppet failure [01:24:12] RECOVERY - puppet last run on tridge is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [01:27:22] PROBLEM - Puppet freshness on nescio is CRITICAL: Last successful Puppet run was Sun 20 Jul 2014 23:26:30 UTC [01:27:32] RECOVERY - Puppet freshness on nescio is OK: puppet ran at Mon Jul 21 01:27:22 UTC 2014 [01:28:12] RECOVERY - Puppet freshness on nfs1 is OK: puppet ran at Mon Jul 21 01:28:02 UTC 2014 [01:28:52] RECOVERY - puppet last run on nfs1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [01:29:12] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [01:33:42] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Mon Jul 21 01:33:37 UTC 2014 [01:45:22] PROBLEM - puppetmaster https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:46:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.017 second response time [01:54:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [01:55:13] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 4.118 second response time [02:00:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [02:02:32] RECOVERY - puppet last run on aluminium is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [02:04:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.020 second response time [02:07:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [02:09:12] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: Complete puppet failure [02:10:03] (03CR) 10Ori.livneh: [C: 04-1] "I see you're persistent in pushing for a more hierarchical module layout. I thought it over and I think you're right -- it's a good move," [operations/puppet] - 10https://gerrit.wikimedia.org/r/147086 (owner: 10Giuseppe Lavagetto) [02:12:13] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.020 second response time [02:14:45] !log LocalisationUpdate completed (1.24wmf13) at 2014-07-21 02:13:42+00:00 [02:14:58] Logged the message, Master [02:16:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [02:19:52] RECOVERY - puppet last run on es4 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [02:21:14] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.016 second response time [02:25:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [02:26:11] !log LocalisationUpdate completed (1.24wmf14) at 2014-07-21 02:25:08+00:00 [02:26:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.021 second response time [02:26:17] Logged the message, Master [02:29:13] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [02:29:43] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: Complete puppet failure [02:36:02] PROBLEM - puppet last run on ms1002 is CRITICAL: CRITICAL: Complete puppet failure [02:36:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [02:37:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.028 second response time [02:40:02] PROBLEM - puppet last run on nickel is CRITICAL: CRITICAL: Complete puppet failure [02:45:49] (03CR) 10Krinkle: "fixme" (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/139279 (https://bugzilla.wikimedia.org/66370) (owner: 10Withoutaname) [02:48:30] (03PS1) 10Krinkle: Remove wmgRC2UDPPrefix exception for wikimania2015wiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148003 [02:49:02] (03CR) 10Krinkle: [C: 032] "Deploying as m:CVN is currently tripping on this and I don't want to add another exception there." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148003 (owner: 10Krinkle) [02:49:08] (03Merged) 10jenkins-bot: Remove wmgRC2UDPPrefix exception for wikimania2015wiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148003 (owner: 10Krinkle) [02:49:43] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [02:50:24] !log krinkle Synchronized wmf-config/InitialiseSettings.php: I27c6f82af5e9b (duration: 00m 06s) [02:50:29] Logged the message, Master [02:54:52] PROBLEM - puppet last run on sanger is CRITICAL: CRITICAL: Complete puppet failure [02:56:53] (03PS1) 10TTO: Add notice so people don't add RC2UDP prefixes anymore [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148005 [02:58:10] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 21 02:57:03 UTC 2014 (duration 57m 2s) [02:58:15] Logged the message, Master [03:02:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [03:03:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.093 second response time [03:03:42] PROBLEM - puppet last run on ms1001 is CRITICAL: CRITICAL: Complete puppet failure [03:04:16] (03CR) 10Withoutaname: [C: 031] Add notice so people don't add RC2UDP prefixes anymore [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148005 (owner: 10TTO) [03:11:12] RECOVERY - puppetmaster backend https on strontium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.879 second response time [03:11:38] !log restarted apache on strontium [03:11:42] RECOVERY - puppet last run on titanium is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:11:43] RECOVERY - puppet last run on wtp1011 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [03:11:44] Logged the message, Master [03:11:52] RECOVERY - puppet last run on analytics1014 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [03:11:52] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [03:11:53] RECOVERY - puppet last run on rcs1001 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [03:12:13] RECOVERY - puppet last run on db1070 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [03:12:13] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [03:12:13] RECOVERY - puppet last run on search1013 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [03:12:13] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [03:12:13] RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:12:13] RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [03:12:22] RECOVERY - puppet last run on analytics1011 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [03:12:22] RECOVERY - puppet last run on elastic1002 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [03:12:32] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [03:12:33] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [03:12:43] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [03:12:43] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [03:12:43] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [03:12:43] RECOVERY - puppet last run on es1002 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:12:52] RECOVERY - puppet last run on nitrogen is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:12:52] RECOVERY - puppet last run on wtp1015 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:12:52] RECOVERY - puppet last run on amssq38 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:12:52] RECOVERY - puppet last run on mw1148 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [03:12:53] RECOVERY - puppet last run on sanger is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [03:13:02] RECOVERY - puppet last run on mw1032 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [03:13:02] RECOVERY - puppet last run on db1001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:13:12] RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [03:13:12] RECOVERY - puppet last run on analytics1004 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:13:12] RECOVERY - puppet last run on rdb1003 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [03:13:12] RECOVERY - puppet last run on pc1003 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [03:13:13] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [03:13:13] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [03:13:32] RECOVERY - puppet last run on analytics1027 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [03:13:42] RECOVERY - puppet last run on ssl1007 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [03:13:42] RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:13:42] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:13:43] RECOVERY - puppet last run on mw1167 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:13:43] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [03:13:43] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:13:43] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:13:43] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [03:13:44] RECOVERY - puppet last run on analytics1028 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:13:52] RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [03:14:12] RECOVERY - puppet last run on search1022 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:14:12] RECOVERY - puppet last run on mw1105 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:14:12] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:14:12] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [03:14:12] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [03:14:13] RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:14:32] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:14:33] RECOVERY - puppet last run on mw1077 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [03:14:43] RECOVERY - puppet last run on es1010 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:14:43] RECOVERY - puppet last run on mw1108 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:15:42] RECOVERY - puppet last run on db1033 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:15:42] RECOVERY - puppet last run on copper is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:15:43] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:15:43] RECOVERY - puppet last run on iodine is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:15:52] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:15:53] RECOVERY - puppet last run on mercury is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:16:02] RECOVERY - puppet last run on dbstore1002 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [03:16:02] RECOVERY - puppet last run on db1011 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [03:16:02] RECOVERY - puppet last run on ms-be1011 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:16:02] RECOVERY - puppet last run on ms1002 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:16:12] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:16:12] RECOVERY - puppet last run on analytics1003 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:16:12] RECOVERY - puppet last run on calcium is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:16:12] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [03:16:12] RECOVERY - puppet last run on cp1037 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [03:16:13] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:16:13] RECOVERY - puppet last run on strontium is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:16:13] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [03:16:14] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:16:22] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [03:16:32] RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:16:32] RECOVERY - puppet last run on cp1044 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:16:33] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:16:42] RECOVERY - puppet last run on db1006 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [03:16:42] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [03:16:42] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [03:16:42] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [03:16:43] RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:16:43] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [03:16:43] RECOVERY - puppet last run on ssl1006 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:16:43] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [03:16:44] RECOVERY - puppet last run on mw1016 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:16:44] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [03:16:45] RECOVERY - puppet last run on ms-be1002 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:16:52] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:16:52] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:17:02] RECOVERY - puppet last run on ssl1001 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [03:17:02] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:17:12] RECOVERY - puppet last run on cp1052 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:17:13] RECOVERY - puppet last run on search1006 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [03:17:13] RECOVERY - puppet last run on cp1040 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:17:13] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:17:13] RECOVERY - puppet last run on cp1053 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:17:13] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [03:17:13] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:17:22] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [03:17:33] RECOVERY - puppet last run on mw1139 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [03:17:42] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [03:17:43] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [03:17:43] RECOVERY - puppet last run on search1011 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:17:52] RECOVERY - puppet last run on db60 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:17:52] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:17:52] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:17:52] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:17:52] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:18:02] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:18:02] RECOVERY - puppet last run on db71 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:18:02] RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [03:18:12] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:18:12] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [03:18:13] RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:18:13] RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:18:32] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [03:18:42] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [03:18:42] RECOVERY - puppet last run on search1004 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:18:42] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [03:18:42] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:18:42] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:18:43] RECOVERY - puppet last run on mw1093 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [03:18:43] RECOVERY - puppet last run on search1012 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [03:18:44] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:18:52] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [03:18:52] RECOVERY - puppet last run on mexia is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [03:18:52] RECOVERY - puppet last run on mw1107 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [03:18:52] RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [03:18:52] RECOVERY - puppet last run on cp4009 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:18:53] RECOVERY - puppet last run on amssq33 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:18:53] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [03:19:02] RECOVERY - puppet last run on analytics1021 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:19:02] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [03:19:12] RECOVERY - puppet last run on zinc is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [03:19:12] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [03:19:13] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:19:13] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [03:19:13] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [03:19:13] RECOVERY - puppet last run on ssl1003 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [03:19:13] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [03:19:13] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:19:14] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [03:19:22] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:19:33] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:19:33] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:19:33] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:19:33] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:19:33] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:19:42] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [03:19:42] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:19:42] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [03:19:42] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [03:19:43] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [03:19:43] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [03:19:52] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:19:52] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:19:52] RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:19:52] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [03:19:52] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:20:02] RECOVERY - puppet last run on search1003 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:20:02] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:20:12] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [03:20:12] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [03:20:12] RECOVERY - puppet last run on mw1154 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [03:20:12] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [03:20:12] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [03:20:13] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:20:13] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:20:14] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [03:20:14] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:20:14] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [03:20:15] RECOVERY - puppet last run on ssl3001 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [03:20:15] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [03:20:22] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [03:20:22] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:20:32] RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:20:33] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [03:20:33] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [03:20:33] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [03:20:33] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:20:42] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:20:42] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [03:20:42] RECOVERY - puppet last run on search1008 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:20:42] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [03:20:43] RECOVERY - puppet last run on ssl1004 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [03:20:43] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [03:20:43] RECOVERY - puppet last run on search1019 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [03:20:43] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:20:44] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [03:20:45] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:20:45] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:20:45] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:20:46] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:20:47] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:20:47] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [03:20:48] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [03:20:48] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:20:49] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [03:20:49] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:20:52] RECOVERY - puppet last run on tantalum is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:20:52] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [03:20:52] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:20:52] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [03:20:52] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:20:53] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:20:53] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:20:54] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:20:54] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:21:02] RECOVERY - puppet last run on nickel is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:21:12] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:21:12] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:21:12] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:21:12] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:21:12] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:21:13] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [03:21:13] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [03:21:14] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:21:14] RECOVERY - puppet last run on search1009 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:21:14] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:21:15] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:21:16] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [03:21:16] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:21:17] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [03:21:17] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:21:18] RECOVERY - puppet last run on es10 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [03:21:18] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [03:21:18] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:21:19] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [03:21:22] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [03:21:22] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:21:22] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:21:22] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [03:21:32] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:21:33] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:21:42] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [03:21:42] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [03:21:42] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [03:21:42] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [03:21:43] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [03:21:43] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [03:21:43] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [03:21:44] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:21:44] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [03:21:45] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:21:45] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:21:45] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:21:46] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [03:21:47] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:21:47] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [03:21:47] RECOVERY - puppet last run on search1014 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [03:21:48] RECOVERY - puppet last run on search1021 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:21:48] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:21:49] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:21:50] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:21:52] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [03:21:52] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:21:52] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:21:52] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:21:52] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [03:21:53] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [03:21:53] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [03:21:54] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:21:54] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [03:21:54] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [03:21:55] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:21:55] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [03:22:02] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [03:22:02] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:22:02] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [03:22:02] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [03:22:02] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:22:12] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [03:22:12] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:22:12] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:22:12] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:22:12] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [03:22:13] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:22:13] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:22:13] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:22:14] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [03:22:14] RECOVERY - puppet last run on es7 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:22:15] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [03:22:16] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [03:22:16] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:22:17] RECOVERY - puppet last run on vanadium is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [03:22:17] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [03:22:22] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:22:33] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:22:33] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [03:22:33] RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:22:33] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:22:33] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [03:22:42] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:22:42] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:22:42] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:22:42] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:22:43] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:22:43] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:22:43] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:22:43] RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [03:22:44] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:22:45] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [03:22:45] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:22:45] RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:22:46] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:22:47] RECOVERY - puppet last run on ms1001 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:22:47] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:22:48] RECOVERY - puppet last run on search1020 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:22:48] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [03:22:52] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [03:22:52] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [03:22:52] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [03:22:52] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:22:52] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [03:22:53] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:22:53] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [03:22:54] RECOVERY - puppet last run on fenari is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [03:22:54] RECOVERY - puppet last run on tarin is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:23:02] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:23:02] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:23:02] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [03:23:02] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:23:12] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [03:23:12] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:23:12] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:23:12] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [03:23:12] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:23:13] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [03:23:13] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:23:13] RECOVERY - puppet last run on virt0 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [03:23:14] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:23:15] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:23:15] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [03:23:22] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:23:32] RECOVERY - puppet last run on analytics1025 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [03:23:33] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [03:23:33] RECOVERY - puppet last run on ssl1002 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [03:23:33] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [03:23:33] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [03:23:33] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:23:42] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:23:42] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [03:23:43] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:23:43] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [03:23:43] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [03:23:43] RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [03:23:43] RECOVERY - puppet last run on db1017 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [03:23:43] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [03:23:44] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:23:45] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [03:23:45] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [03:23:46] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:23:46] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [03:23:46] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [03:23:47] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [03:23:52] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [03:23:53] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:23:53] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:23:53] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [03:23:53] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:23:53] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [03:23:53] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:23:53] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [03:23:54] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [03:23:55] RECOVERY - puppet last run on elastic1001 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:23:55] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [03:23:55] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:23:56] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [03:24:12] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [03:24:12] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:24:13] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:24:13] RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:24:13] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [03:24:13] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [03:24:13] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:24:13] RECOVERY - puppet last run on db73 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [03:24:14] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:24:14] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:24:15] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [03:24:15] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:24:22] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [03:24:22] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:24:22] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:24:22] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [03:24:22] RECOVERY - puppet last run on analytics1020 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:24:22] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [03:24:32] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [03:24:32] RECOVERY - puppet last run on francium is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [03:24:32] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [03:24:33] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:24:42] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [03:24:42] RECOVERY - puppet last run on platinum is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [03:24:42] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [03:24:42] RECOVERY - puppet last run on search1016 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [03:24:42] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [03:24:43] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [03:24:43] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:24:43] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [03:24:44] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:24:44] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:24:45] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [03:24:52] RECOVERY - puppet last run on db74 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:24:52] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:24:52] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [03:24:53] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [03:24:53] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [03:24:53] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [03:24:53] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [03:24:54] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [03:24:54] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:24:54] RECOVERY - puppet last run on gold is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [03:25:02] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [03:25:02] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [03:25:02] RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [03:25:02] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [03:25:12] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:25:12] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:25:12] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [03:25:12] RECOVERY - puppet last run on search1010 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [03:25:12] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:25:13] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [03:25:13] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [03:25:14] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:25:14] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:25:15] RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [03:25:15] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:25:15] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:25:16] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [03:25:17] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:25:22] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:25:22] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:25:22] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [03:25:22] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [03:25:32] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [03:25:42] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [03:25:42] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:25:43] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [03:25:43] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:25:43] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:25:43] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [03:25:43] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:25:43] RECOVERY - puppet last run on holmium is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:25:44] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:25:52] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:25:52] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:25:53] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:25:53] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:25:53] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [03:25:53] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:25:53] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:25:54] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:25:54] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [03:26:02] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:26:12] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [03:26:12] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [03:26:12] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:26:13] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:26:22] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [03:26:32] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:26:32] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:26:32] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:26:33] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:26:33] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [03:26:33] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [03:26:33] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [03:26:33] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:26:42] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [03:26:42] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [03:26:42] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [03:26:42] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [03:26:43] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:26:43] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:26:43] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:26:44] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:26:44] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:26:52] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:26:52] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [03:26:52] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [03:26:52] RECOVERY - puppet last run on search1007 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [03:26:53] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:26:53] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:26:53] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [03:27:02] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:27:02] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [03:27:12] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [03:27:12] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [03:27:12] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [03:27:12] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [03:27:12] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:27:12] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:27:13] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:27:14] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:27:14] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [03:27:15] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [03:27:15] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:27:16] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [03:27:16] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [03:27:17] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [03:27:22] RECOVERY - puppet last run on cp1058 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:27:22] RECOVERY - puppet last run on ruthenium is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [03:27:22] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:27:22] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:27:22] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:27:33] RECOVERY - puppet last run on analytics1038 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [03:27:42] RECOVERY - puppet last run on es1007 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [03:27:42] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [03:27:42] RECOVERY - puppet last run on antimony is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:27:42] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:27:43] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [03:27:43] RECOVERY - puppet last run on db1052 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:27:43] RECOVERY - puppet last run on mw1044 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:27:44] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [03:27:44] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [03:27:45] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:27:45] RECOVERY - puppet last run on polonium is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:27:45] RECOVERY - puppet last run on analytics1016 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [03:27:46] RECOVERY - puppet last run on labstore1001 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [03:27:52] RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:27:53] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [03:27:53] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:27:53] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:27:53] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [03:27:53] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [03:27:53] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [03:27:54] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [03:27:54] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [03:27:55] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:28:02] RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:28:02] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:28:12] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:28:12] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [03:28:12] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:28:12] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:28:12] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [03:28:13] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [03:28:13] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:28:14] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [03:28:14] RECOVERY - puppet last run on db1043 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [03:28:15] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [03:28:15] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [03:28:16] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:28:16] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [03:28:17] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:28:22] RECOVERY - puppet last run on pc1002 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:28:32] RECOVERY - puppet last run on db1060 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [03:28:42] RECOVERY - puppet last run on elastic1019 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [03:28:42] RECOVERY - puppet last run on virt1003 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:28:43] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [03:28:43] RECOVERY - puppet last run on silver is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [03:28:43] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:28:43] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:28:43] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [03:28:52] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:28:52] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [03:28:53] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [03:28:53] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [03:28:53] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:28:53] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:29:12] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [03:29:12] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [03:29:12] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:29:13] RECOVERY - puppet last run on search1002 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:29:13] RECOVERY - puppet last run on ssl1005 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [03:29:13] RECOVERY - puppet last run on argon is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [03:29:13] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:29:13] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [03:29:14] RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [03:29:15] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:29:15] RECOVERY - puppet last run on osmium is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:29:16] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:29:16] RECOVERY - puppet last run on search1005 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:29:17] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:29:22] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [03:29:22] RECOVERY - puppet last run on elastic1006 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:29:22] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [03:29:32] RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [03:29:33] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [03:29:33] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:29:33] RECOVERY - puppet last run on gadolinium is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [03:29:42] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:29:42] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [03:29:42] RECOVERY - puppet last run on wtp1018 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:29:42] RECOVERY - puppet last run on analytics1023 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:29:42] RECOVERY - puppet last run on db1069 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [03:29:43] RECOVERY - puppet last run on ssl1009 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:29:43] RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:29:43] RECOVERY - puppet last run on mw1125 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:29:44] RECOVERY - puppet last run on mw1049 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [03:29:44] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:29:45] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:29:45] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:29:46] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [03:29:47] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:29:47] RECOVERY - puppet last run on analytics1013 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:29:48] RECOVERY - puppet last run on thallium is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [03:29:52] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [03:29:52] RECOVERY - puppet last run on rubidium is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:29:52] RECOVERY - puppet last run on ms-be3001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:29:52] RECOVERY - puppet last run on amssq56 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [03:29:53] RECOVERY - puppet last run on mw1056 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [03:29:53] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [03:29:53] RECOVERY - puppet last run on db69 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [03:30:02] RECOVERY - puppet last run on db72 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [03:30:02] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:30:02] RECOVERY - puppet last run on cp4005 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [03:30:12] RECOVERY - puppet last run on wtp1022 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [03:30:12] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [03:30:12] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [03:30:12] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [03:30:12] RECOVERY - puppet last run on mw1014 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [03:30:13] RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:30:13] RECOVERY - puppet last run on mw1030 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [03:30:14] RECOVERY - puppet last run on ms-be1008 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [03:30:14] RECOVERY - puppet last run on mw1133 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:30:15] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:30:15] RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [03:30:16] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:30:16] RECOVERY - puppet last run on amssq42 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [03:30:17] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [03:30:17] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:30:22] RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [03:30:22] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [03:30:22] RECOVERY - puppet last run on ms-be1012 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:30:22] RECOVERY - puppet last run on amssq41 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:30:32] RECOVERY - puppet last run on amssq40 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [03:30:33] RECOVERY - puppet last run on mw1057 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [03:30:33] RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [03:30:42] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [03:30:42] RECOVERY - puppet last run on elastic1015 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:30:42] RECOVERY - puppet last run on oxygen is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [03:30:43] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:30:43] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:30:43] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [03:30:43] RECOVERY - puppet last run on elastic1011 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:30:44] RECOVERY - puppet last run on mc1001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:30:44] RECOVERY - puppet last run on mw1146 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:30:45] RECOVERY - puppet last run on db1055 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:31:02] RECOVERY - puppet last run on db1004 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [03:31:02] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [03:31:12] RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [03:31:12] RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:31:12] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [03:31:12] RECOVERY - puppet last run on wtp1002 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:31:12] RECOVERY - puppet last run on mw1004 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:31:13] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [03:31:13] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [03:31:22] RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [03:31:22] RECOVERY - puppet last run on ms-be1007 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [03:31:32] RECOVERY - puppet last run on ssl1008 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:31:32] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [03:31:42] RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [03:31:42] RECOVERY - puppet last run on search1023 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [03:31:42] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [03:31:42] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [03:31:42] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:31:43] RECOVERY - puppet last run on mw1053 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:31:43] RECOVERY - puppet last run on mw1087 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [03:31:44] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [03:31:44] RECOVERY - puppet last run on mw1156 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:31:45] RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:31:45] RECOVERY - puppet last run on search1024 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [03:31:52] RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [03:31:52] RECOVERY - puppet last run on analytics1032 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [03:31:52] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [03:31:52] RECOVERY - puppet last run on ssl3002 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:31:53] RECOVERY - puppet last run on elastic1014 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [03:31:53] RECOVERY - puppet last run on mw1034 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [03:31:53] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:32:02] RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [03:32:12] RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:32:12] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [03:32:12] RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:32:12] RECOVERY - puppet last run on search1017 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [03:32:12] RECOVERY - puppet last run on search1015 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:32:13] RECOVERY - puppet last run on mc1007 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [03:32:13] RECOVERY - puppet last run on rcs1002 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [03:32:14] RECOVERY - puppet last run on amslvs3 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [03:32:22] RECOVERY - puppet last run on mc1013 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:32:22] RECOVERY - puppet last run on mw1097 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:32:32] RECOVERY - puppet last run on lvs1001 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [03:32:43] RECOVERY - puppet last run on elastic1005 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [03:32:52] RECOVERY - puppet last run on hooft is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [03:32:55] (03CR) 10Santhosh: Open Special:ContentTranslation in the target wiki (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/147918 (owner: 10Amire80) [03:33:41] (03PS1) 10Ori.livneh: wmflib: add ensure_directory() and ensure_link() [operations/puppet] - 10https://gerrit.wikimedia.org/r/148007 [03:37:22] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Mon 21 Jul 2014 01:37:04 UTC [03:39:20] (03PS2) 10Ori.livneh: beta: add deployment-mediawiki{01,02} to scap targets [operations/puppet] - 10https://gerrit.wikimedia.org/r/148001 [03:40:13] (03CR) 10Ori.livneh: "cherry-picked in beta" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148001 (owner: 10Ori.livneh) [03:42:11] (03CR) 1020after4: [C: 031] "Cool, I like this! Good idea Ori." [operations/puppet] - 10https://gerrit.wikimedia.org/r/148007 (owner: 10Ori.livneh) [03:55:42] (03PS1) 10Ori.livneh: mediawiki: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/148010 [03:56:40] (03CR) 10jenkins-bot: [V: 04-1] mediawiki: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/148010 (owner: 10Ori.livneh) [03:56:52] (03PS2) 10Ori.livneh: mediawiki: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/148010 [04:26:35] (03CR) 10Krinkle: [C: 032] Add notice so people don't add RC2UDP prefixes anymore [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148005 (owner: 10TTO) [04:26:42] (03Merged) 10jenkins-bot: Add notice so people don't add RC2UDP prefixes anymore [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148005 (owner: 10TTO) [04:35:52] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There is one unmerged change in mediawiki_config (dir /a/common/). [04:37:12] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Mon Jul 21 04:37:07 UTC 2014 [05:01:22] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Mon 21 Jul 2014 03:00:18 UTC [05:45:29] (03PS1) 10Hoo man: Don't use sudo -u apache for dumpwikidatajson.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/148017 [06:27:22] PROBLEM - puppetmaster backend https on strontium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:12] RECOVERY - puppetmaster backend https on strontium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.030 second response time [06:28:42] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:42] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:43] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:43] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:43] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:43] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:43] PROBLEM - puppet last run on holmium is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:52] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:52] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:52] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:02] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:02] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:12] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:12] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:13] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:22] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:42] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:42] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:42] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:43] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:43] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:43] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:43] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:43] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:44] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:53] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:12] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:12] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:13] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:02] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:40:23] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Mon Jul 21 06:40:17 UTC 2014 [06:44:23] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:44:33] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:44:33] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:45:22] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:45:23] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:45:23] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:45:23] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:45:23] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:45:23] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:45:23] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:45:23] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:45:32] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:45:32] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:45:32] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:45:42] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:45:43] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:45:52] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:46:22] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:46:22] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:46:23] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:46:23] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:46:23] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:46:42] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:46:43] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:47:23] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:47:23] RECOVERY - puppet last run on holmium is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:47:32] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:47:33] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:47:43] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:47:43] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:49:33] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Puppet has 2 failures [06:59:04] apergos: around? [06:59:04] * hoo is on a train [06:59:11] morning [06:59:34] So might loose connection [06:59:50] ok [07:00:00] Did you see my gerrit patch? [07:00:34] I think that's why it broke tonight :/ [07:00:34] no, you sent it over the weekend I guess? [07:01:12] no, the dataset user should not sudo [07:01:32] the dump should run as the dataset user, the directory is set up with write permissions for that already [07:01:46] I didn't notice that bit [07:01:47] Ok [07:02:07] Usually we only want to run MW as apachr [07:02:14] But might be ok here [07:02:25] (03CR) 10ArielGlenn: [C: 032] Don't use sudo -u apache for dumpwikidatajson.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/148017 (owner: 10Hoo man) [07:02:52] all dumps run as the dataset user and all cron jobs as well [07:03:07] Could you maybe run it per hand today? [07:04:12] I don't think there's more, but we don't want to wait for another week just to fix another thing [07:05:14] let me look at the timings of the other cron jobs [07:05:26] Ok [07:05:34] if I can fit it in between nicely, then yep [07:05:58] Would be awesome [07:06:32] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:10:03] I'll soon enter a mobile network blackhole, so will go [07:10:33] ok, have a good trip [07:12:41] Thanks :) [07:20:29] <_joe_> morning [07:20:37] yes it is [07:37:52] (03CR) 10Giuseppe Lavagetto: "@ori: not really: at the moment you have one class - mediawiki::hhvm that takes care of making the runtime for hhvm to work. I would've us" [operations/puppet] - 10https://gerrit.wikimedia.org/r/147086 (owner: 10Giuseppe Lavagetto) [07:49:09] hi _joe_ , apergos :) [07:49:16] morning [07:49:33] have one of you looked at the puppet storm ? [07:55:44] no, it's the 6:30 am restart again [08:52:34] good morning! [08:55:19] greetings! [08:55:59] <_joe_> hey hashar [08:56:42] hashar: welcome back! [08:57:06] lack of internet was great [08:57:23] having two thousands mails to process isn't [08:59:05] welcome back, hashar [08:59:47] _joe_: don't we monitor number of available inodes on partitions? [09:00:06] <_joe_> hashar: we don't. [09:00:10] <_joe_> :/ [09:00:31] (03CR) 10Filippo Giunchedi: [C: 031] Load nutcracker server list from realm-specific yaml file [operations/puppet] - 10https://gerrit.wikimedia.org/r/147830 (owner: 10Ori.livneh) [09:01:29] _joe_: luckily the icinga check_disk plugin has support for inodes monitoring :-D [09:02:05] <_joe_> hashar: not sure it works well [09:04:37] ah there is a RT about it nice [09:04:37] https://rt.wikimedia.org/Ticket/Display.html?id=7879 [09:04:45] left a comment with the check_disk parameters [09:13:17] (03CR) 10Filippo Giunchedi: [C: 031] "IIRC we don't build hhvm-fastcgi as a separate package? that'd need to get removed from reprepro after this is merged" [operations/puppet] - 10https://gerrit.wikimedia.org/r/147836 (owner: 10Ori.livneh) [09:13:49] (03CR) 10Giuseppe Lavagetto: "While I love and support how you're doing this, we should really start using hiera for this. We're not the first ones to do this btw, see" [operations/puppet] - 10https://gerrit.wikimedia.org/r/147830 (owner: 10Ori.livneh) [09:15:00] (03PS1) 10Odder: Enable FlaggedRevs on Module namespace on ukwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148029 (https://bugzilla.wikimedia.org/67748) [09:16:30] _joe_: btw, I've been working on something called... puppetception :) https://gerrit.wikimedia.org/r/#/c/147759/ [09:17:42] <_joe_> YuviPanda: whenever I have time to work on hiera, I'll take a deeper look, promised [09:17:51] :D [09:17:52] ok [09:18:16] <_joe_> I'm 100% HAT [09:18:18] HAT? [09:18:23] <_joe_> eheh [09:18:23] (03PS3) 10Filippo Giunchedi: Add 1 more runner per job runner server [operations/puppet] - 10https://gerrit.wikimedia.org/r/147691 (owner: 10Aaron Schulz) [09:18:25] Head over Tails? [09:18:28] <_joe_> it's SEECRET [09:18:29] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] Add 1 more runner per job runner server [operations/puppet] - 10https://gerrit.wikimedia.org/r/147691 (owner: 10Aaron Schulz) [09:18:32] aah [09:18:43] <_joe_> H stands for HHVM [09:18:48] aaah [09:18:49] nice [09:19:03] <_joe_> A for apache 2.4 and T for trusty I guess [09:19:10] heh [09:19:26] * YuviPanda should remember to put trusty on the new graphite box [09:19:32] when it arrives [09:19:38] <_joe_> or 'do 3 fundamental migrations at once, what could possibly go wrong" [09:19:47] heh [09:20:35] hashar: i works a lame quick and dirty script to check inodes, want to adopt it ? [09:20:52] matanya: check_disk from icinga should just work [09:21:07] else hmm SNMP would do but I don't think we do any SNMP monitoring :-] [09:21:26] *should* sounds like a nice title of a bug [09:21:41] (03PS1) 10Odder: Set $wgUploadNavigationUrl for plwikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148030 (https://bugzilla.wikimedia.org/68191) [09:22:24] YuviPanda: trusty is the default for now labs boxs [09:22:30] yeah [09:22:37] matanya: the new graphite box will be a physical one, though [09:22:44] ah [09:29:16] godog: _joe_: are you familiar with memcached / twemproxy / nutcracker migration? [09:29:37] we used to have a proxy on each apache listening on some port, it apparently changed with a migration to nutcracker [09:31:18] hashar: I was involved yes, but _joe_ did most of the work, what's up? [09:31:48] godog: configuration is messed up on the beta cluster :-] [09:31:51] <_joe_> hashar: yes, 11211 to 11212 [09:31:53] I will drop an email instead [09:32:02] since puppet is disabled on both beta cluster apache servers [09:33:28] even the apache conf is broken bah [09:33:50] (03CR) 10Filippo Giunchedi: [C: 031] Get rid of deprecated HHVM module [operations/puppet] - 10https://gerrit.wikimedia.org/r/147833 (owner: 10Ori.livneh) [09:38:42] (03PS1) 10Odder: Set $wgForceUIMsgAsContentMsg for zhwikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148032 (https://bugzilla.wikimedia.org/64816) [09:40:32] (03CR) 10Giuseppe Lavagetto: [C: 031] "If we're sure it's not being used in labs (checking) we can replace it with the new module that is partially in mediawiki::hhvm now." [operations/puppet] - 10https://gerrit.wikimedia.org/r/147833 (owner: 10Ori.livneh) [09:45:21] (03PS1) 10QChris: Use dfs.blocksize instead of deprecated dfs.block.size [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/148033 [09:47:45] (03PS3) 10Filippo Giunchedi: Increase the number of parsoid job runners [operations/puppet] - 10https://gerrit.wikimedia.org/r/147850 (owner: 10Aaron Schulz) [09:49:04] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] Increase the number of parsoid job runners [operations/puppet] - 10https://gerrit.wikimedia.org/r/147850 (owner: 10Aaron Schulz) [09:52:13] (03PS1) 10Matanya: ldap: qualify vars [operations/puppet] - 10https://gerrit.wikimedia.org/r/148035 [09:52:58] (03PS1) 10Hashar: beta: rename memcached object cache [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148036 [09:53:09] (03CR) 10Hashar: [C: 032] beta: rename memcached object cache [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148036 (owner: 10Hashar) [09:53:12] ahhh [09:53:15] (03Merged) 10jenkins-bot: beta: rename memcached object cache [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148036 (owner: 10Hashar) [09:56:05] (03PS1) 10Alexandros Kosiaris: Fix typo with scalac/scala in b918cc9e2 [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/148038 [09:56:16] hashar: any chance you bring back puppet-compiler ? [09:56:31] (03PS1) 10Hashar: beta: fix sessions memcache [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148039 [09:56:48] <_joe_> matanya: ? [09:56:48] matanya: ask _joe_ about puppet-compiler :-] I guess /tmp ended up being filled again [09:57:03] (03CR) 10Hashar: [C: 032] beta: fix sessions memcache [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148039 (owner: 10Hashar) [09:57:05] yes, it did [09:57:09] (03Merged) 10jenkins-bot: beta: fix sessions memcache [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148039 (owner: 10Hashar) [09:57:15] <_joe_> matanya: when exactly? [09:57:20] https://integration.wikimedia.org/ci/computer/puppet-compiler02.eqiad.wmflabs/ [09:57:30] two days ago [09:57:40] didn't bother poking on the weekend [09:57:42] <_joe_> we test too many things [09:57:52] <_joe_> ok time to change where we store the sourc [10:03:15] (03CR) 10Alexandros Kosiaris: "No, that was an error. Fixed in" [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/147493 (owner: 10Alexandros Kosiaris) [10:03:27] (03CR) 10Alexandros Kosiaris: "Thanks for spotting it out!" [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/147493 (owner: 10Alexandros Kosiaris) [10:03:35] what a mess [10:03:55] see hashar, you go on vacation and this is what happens [10:03:59] beta cluster bastion has nutcracker on port 11211 [10:04:07] no more vacation for you [10:04:24] oh I will take a couple weeks end of august/ beginning of september [10:04:24] :D [10:04:37] lot of things got fixed anywa [10:04:38] y [10:04:54] (03CR) 10Alexandros Kosiaris: [C: 032] icinga - update SSL cipher list [operations/puppet] - 10https://gerrit.wikimedia.org/r/147207 (https://bugzilla.wikimedia.org/53259) (owner: 10Dzahn) [10:06:06] pff [10:06:20] _joe_: i'll just add reviews until the host is back without running on it. please let me know when it is back, thanks! [10:06:36] <_joe_> matanya: in a while [10:06:49] no rush [10:07:31] (03PS2) 10Alexandros Kosiaris: Add package building instruction in the README [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/147492 [10:08:46] godog: a first draft on .deb building procedure we were discussing about. In the form of an user README but it is a start ^ [10:09:13] * YuviPanda should learn to build debs soon [10:09:35] * _joe_ hands YuviPanda the debian policy [10:09:44] yeah, am reading the python policy [10:09:56] * _joe_ watches YuviPanda feel miserable [10:10:00] heh [10:10:01] moar policies!!! [10:10:06] provider => 'pip' :P [10:10:35] <_joe_> provider => 'pip' if forbidden [10:10:46] think I am almost out of this kafka new version thing. I just need ottomata to have a look and we should be able to wrap it up [10:10:57] <_joe_> you're lucky our debian praetorians didn't see you :) [10:11:08] let's waste time looking at another java build tool just for the heck of it [10:11:24] we got praetorians ? [10:11:37] do they got a centurion ? [10:11:59] I know, I know :) [10:12:47] _joe_: my first introduction to ops ever was me attempting to do 'sudo pip install ' on stat1, and getting a stern email from LeslieCarr a while later. [10:12:55] (I didn't have sudo there) [10:13:00] that was almost 1.5 years ago, though [10:13:03] so yeah, no pip. [10:15:16] (03PS1) 10Hashar: mediawiki: extract nutcracker to its own manifest [operations/puppet] - 10https://gerrit.wikimedia.org/r/148041 [10:15:18] (03PS1) 10Hashar: beta: configure nutcracker on bastion [operations/puppet] - 10https://gerrit.wikimedia.org/r/148042 [10:16:04] and no sudo YuviPanda ... :) [10:16:13] akosiaris: excellent! I'll take a look there [10:17:10] matanya: heh :) [10:18:16] E: Unable to locate package nutcracker [10:18:18] seriously [10:19:24] <_joe_> YuviPanda: I'm on record as saying pip system-wide is bad, but I do prefer virtualenvs + pip to run python services in production [10:21:43] _joe_: +1 as well, but that seems to be out of favor in general with ops too [10:21:51] <_joe_> I know [10:21:57] which is sad [10:22:19] since nodejs gets by by just bundling node_modules, and python can't, so makes using python for stuff harder. [10:22:30] if there aren't debs [10:22:46] <_joe_> YuviPanda: that's not even the point for me [10:22:48] <_joe_> :) [10:23:05] <_joe_> building debs for python packages is not *that* hard [10:23:44] still... [10:23:48] <_joe_> it's more about being able to use different versions of libraries for different softwares [10:23:48] plus isolation, etc [10:23:51] indeed [10:24:24] (03PS1) 10Hashar: nutcracker: set service dependency [operations/puppet] - 10https://gerrit.wikimedia.org/r/148043 [10:24:35] <_joe_> but hey, there is consensus this is not a good idea :) [10:25:18] * YuviPanda mumbles to self :) [10:25:36] (03CR) 10Giuseppe Lavagetto: [C: 032] nutcracker: set service dependency [operations/puppet] - 10https://gerrit.wikimedia.org/r/148043 (owner: 10Hashar) [10:26:02] <_joe_> hashar: we should NOT notify nutcracker [10:26:09] <_joe_> it takes time to restart [10:26:16] ahh [10:26:22] so I should amend that commit message hehe [10:26:54] bang my instance has two nutcrackers running now :-]  [10:27:48] <_joe_> hashar: we did that on purpose in prod while upgrading [10:29:44] _joe_: I have two nutcracker instances listening each on two ports http://paste.debian.net/plain/110755 [10:29:49] does that looks correct? :D [10:30:03] apergos: Any luck with the timings? [10:30:30] <_joe_> hashar: it is [10:30:46] <_joe_> hashar: just uninstall the twemproxy package and you're GTG [10:30:53] <_joe_> I guess [10:32:15] _joe_: beta is happy with memcached now :-]  Thanks! [10:32:23] lunch time bbl [10:33:11] (03CR) 10Hashar: [C: 031] "Cherry picked on beta cluster local puppetmaster." [operations/puppet] - 10https://gerrit.wikimedia.org/r/148041 (owner: 10Hashar) [10:33:36] (03CR) 10Hashar: [C: 031] "Cherry picked on beta cluster local puppetmaster. That ensure proper configuration of nutcracker on deployment-bastion.eqiad.wmflabs whic" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148042 (owner: 10Hashar) [10:37:11] hoo, yes I'll be kicking it off in about an hour [10:37:24] Nice :) [10:37:31] I really hope taht it works now [10:45:05] we'll see! [10:45:07] (03PS7) 10Giuseppe Lavagetto: jobrunner: create hhvm-only jobrunners [operations/puppet] - 10https://gerrit.wikimedia.org/r/147086 [10:49:01] (03PS2) 10Giuseppe Lavagetto: Get rid of deprecated HHVM module [operations/puppet] - 10https://gerrit.wikimedia.org/r/147833 (owner: 10Ori.livneh) [10:52:58] (03CR) 10Giuseppe Lavagetto: [C: 032] Get rid of deprecated HHVM module [operations/puppet] - 10https://gerrit.wikimedia.org/r/147833 (owner: 10Ori.livneh) [11:14:20] (03PS2) 10Giuseppe Lavagetto: reprepro: stop importing Facebook's HHVM packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/147836 (owner: 10Ori.livneh) [11:14:27] (03CR) 10Giuseppe Lavagetto: [C: 032] reprepro: stop importing Facebook's HHVM packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/147836 (owner: 10Ori.livneh) [11:14:39] (03CR) 10Giuseppe Lavagetto: [V: 032] reprepro: stop importing Facebook's HHVM packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/147836 (owner: 10Ori.livneh) [12:01:09] !log started /usr/local/bin/dumpwikidatajson.sh in root screen session on snapshot1003 [12:01:13] Logged the message, Master [12:29:18] (03CR) 10Gilles: [C: 031] Remove MediaViewer survey-related settings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146951 (owner: 10Gergő Tisza) [12:30:05] (03CR) 10Gilles: "Can't find it on the deployment calendar?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146951 (owner: 10Gergő Tisza) [12:38:22] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Mon 21 Jul 2014 10:37:14 UTC [12:46:33] PROBLEM - Ubuntu mirror in sync with upstream on carbon is CRITICAL: /srv/ubuntu/project/trace/carbon.wikimedia.org is over 12 hours old. [12:48:33] RECOVERY - Ubuntu mirror in sync with upstream on carbon is OK: /srv/ubuntu/project/trace/carbon.wikimedia.org is over 0 hours old. [12:57:22] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Mon Jul 21 12:57:13 UTC 2014 [13:06:41] (03CR) 10Hashar: [C: 031] beta: add deployment-mediawiki{01,02} to scap targets [operations/puppet] - 10https://gerrit.wikimedia.org/r/148001 (owner: 10Ori.livneh) [13:10:50] The beta cluster git repositories updater could use a merge https://gerrit.wikimedia.org/r/#/c/137463/ :D [13:11:02] it has already been cherry picked on the puppetmaster so it is straightforward [13:16:51] (03PS6) 10Giuseppe Lavagetto: beta: Add mediawiki/core/vendor to beta [operations/puppet] - 10https://gerrit.wikimedia.org/r/137463 (owner: 10BryanDavis) [13:19:16] <_joe_> hashar: I would've merged, if only jenkins worked... [13:19:22] * _joe_ evil grin [13:19:33] :-ad [13:19:53] _joe_: https://integration.wikimedia.org/zuul/ [13:19:58] it is still being tested apparently [13:20:19] <_joe_> it should be instantaneous automagical FTW [13:20:27] <_joe_> or I'll complain :P [13:20:33] * _joe_ typical customer [13:20:56] I am fed up with jenkins [13:21:07] (typical customer service representative) [13:21:23] * hoo eyes apergos and runs [13:21:25] _joe_: jenkins voted! [13:21:56] you can eye me, or eye the log, your choice (log is nicer looking :-P) [13:22:27] which log? [13:22:43] oh it's running :P [13:23:01] Clearly ps -U apache was stupid given the change I did this morning [13:23:03] * hoo hides [13:23:46] heh [13:24:13] (03CR) 10Giuseppe Lavagetto: [C: 032] beta: Add mediawiki/core/vendor to beta [operations/puppet] - 10https://gerrit.wikimedia.org/r/137463 (owner: 10BryanDavis) [13:31:17] (03PS2) 10Giuseppe Lavagetto: mediawiki::multimedia: on trusty, install libvips37 rather than libvips15 [operations/puppet] - 10https://gerrit.wikimedia.org/r/147992 (owner: 10Ori.livneh) [13:37:54] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki::multimedia: on trusty, install libvips37 rather than libvips15 [operations/puppet] - 10https://gerrit.wikimedia.org/r/147992 (owner: 10Ori.livneh) [13:38:53] PROBLEM - puppetmaster backend https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8141: HTTP/1.1 500 Internal Server Error [13:39:26] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Puppet has 18 failures [13:39:26] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Puppet has 21 failures [13:39:27] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: Puppet has 24 failures [13:39:27] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: Puppet has 33 failures [13:39:27] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Puppet has 18 failures [13:39:27] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet has 29 failures [13:39:27] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Puppet has 41 failures [13:39:28] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Puppet has 55 failures [13:39:28] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: Puppet has 57 failures [13:39:29] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Puppet has 30 failures [13:39:29] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: Puppet has 24 failures [13:39:30] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Puppet has 54 failures [13:39:36] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: Puppet has 17 failures [13:39:37] PROBLEM - puppet last run on fenari is CRITICAL: CRITICAL: Puppet has 11 failures [13:39:37] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Puppet has 57 failures [13:39:37] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: Puppet has 11 failures [13:39:37] PROBLEM - puppet last run on search1003 is CRITICAL: CRITICAL: Puppet has 15 failures [13:39:46] PROBLEM - puppet last run on ssl1003 is CRITICAL: CRITICAL: Puppet has 24 failures [13:39:47] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Puppet has 6 failures [13:39:47] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Puppet has 20 failures [13:39:47] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: Puppet has 16 failures [13:39:47] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: Puppet has 17 failures [13:39:47] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: Puppet has 48 failures [13:39:47] PROBLEM - puppet last run on analytics1021 is CRITICAL: CRITICAL: Puppet has 16 failures [13:39:48] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Puppet has 58 failures [13:39:48] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 20 failures [13:39:56] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 21 failures [13:39:57] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: Puppet has 22 failures [13:39:57] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 17 failures [13:39:57] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 23 failures [13:39:57] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 21 failures [13:40:26] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: Last successful Puppet run was Mon 21 Jul 2014 13:38:17 UTC [13:40:26] PROBLEM - puppet last run on db1058 is CRITICAL: CRITICAL: Puppet has 17 failures [13:40:26] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Puppet has 58 failures [13:40:26] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: Puppet has 50 failures [13:40:26] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: Puppet has 16 failures [13:40:27] PROBLEM - puppet last run on search1008 is CRITICAL: CRITICAL: Puppet has 42 failures [13:40:27] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: Puppet has 49 failures [13:40:28] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Puppet has 60 failures [13:40:28] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: Puppet has 51 failures [13:40:29] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: Puppet has 64 failures [13:40:29] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet has 71 failures [13:40:30] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: Puppet has 59 failures [13:40:30] PROBLEM - puppet last run on db1005 is CRITICAL: CRITICAL: Puppet has 21 failures [13:40:31] PROBLEM - puppet last run on search1019 is CRITICAL: CRITICAL: Puppet has 46 failures [13:40:31] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: Puppet has 22 failures [13:40:36] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Puppet has 56 failures [13:40:36] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: Puppet has 55 failures [13:40:36] PROBLEM - puppet last run on es1005 is CRITICAL: CRITICAL: Puppet has 22 failures [13:40:36] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: Puppet has 19 failures [13:40:36] PROBLEM - puppet last run on tantalum is CRITICAL: CRITICAL: Puppet has 27 failures [13:40:37] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Puppet has 57 failures [13:40:37] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Puppet has 59 failures [13:40:46] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: Puppet has 17 failures [13:40:46] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Puppet has 47 failures [13:40:46] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Puppet has 25 failures [13:40:46] PROBLEM - puppet last run on zinc is CRITICAL: CRITICAL: Puppet has 21 failures [13:40:46] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: Puppet has 18 failures [13:40:47] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Puppet has 19 failures [13:40:47] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: Puppet has 30 failures [13:40:48] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Puppet has 59 failures [13:40:48] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: Puppet has 47 failures [13:40:49] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: Puppet has 65 failures [13:40:49] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: Puppet has 52 failures [13:40:49] PROBLEM - puppet last run on search1009 is CRITICAL: CRITICAL: Puppet has 45 failures [13:40:50] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Puppet has 21 failures [13:40:51] PROBLEM - puppet last run on analytics1024 is CRITICAL: CRITICAL: Puppet has 20 failures [13:40:51] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: Puppet has 22 failures [13:40:52] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: Puppet has 27 failures [13:40:52] PROBLEM - puppet last run on virt0 is CRITICAL: CRITICAL: Puppet has 49 failures [13:40:56] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Puppet has 20 failures [13:40:56] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: Puppet has 21 failures [13:40:56] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: Puppet has 23 failures [13:40:57] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 28 failures [13:40:57] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Puppet has 20 failures [13:41:06] <_joe_> !log restarted apache on palladium [13:41:12] Logged the message, Master [13:41:26] PROBLEM - puppet last run on ssl1004 is CRITICAL: CRITICAL: Puppet has 21 failures [13:41:27] PROBLEM - puppet last run on erbium is CRITICAL: CRITICAL: Puppet has 37 failures [13:41:27] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Puppet has 60 failures [13:41:27] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: Puppet has 58 failures [13:41:27] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: Puppet has 20 failures [13:41:27] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: Puppet has 18 failures [13:41:27] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: Puppet has 19 failures [13:41:28] PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: Puppet has 23 failures [13:41:28] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: Puppet has 22 failures [13:41:29] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: Puppet has 48 failures [13:41:29] PROBLEM - puppet last run on search1021 is CRITICAL: CRITICAL: Puppet has 42 failures [13:41:36] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Puppet has 22 failures [13:41:36] PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Puppet has 62 failures [13:41:37] PROBLEM - puppet last run on tmh1002 is CRITICAL: CRITICAL: Puppet has 28 failures [13:41:37] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: Puppet has 53 failures [13:41:46] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Puppet has 57 failures [13:41:46] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: Puppet has 71 failures [13:41:46] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: Puppet has 57 failures [13:41:46] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: Puppet has 20 failures [13:41:46] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Puppet has 16 failures [13:41:47] PROBLEM - puppet last run on analytics1019 is CRITICAL: CRITICAL: Puppet has 14 failures [13:41:47] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: Puppet has 17 failures [13:41:48] PROBLEM - puppet last run on es10 is CRITICAL: CRITICAL: Puppet has 19 failures [13:41:48] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: Puppet has 20 failures [13:41:49] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: Puppet has 19 failures [13:41:49] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: Puppet has 24 failures [13:41:56] RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.013 second response time [13:41:57] PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: Puppet has 21 failures [13:41:57] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: Puppet has 59 failures [13:41:57] PROBLEM - puppet last run on zirconium is CRITICAL: CRITICAL: Puppet has 49 failures [13:41:57] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: Puppet has 27 failures [13:42:26] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: Last successful Puppet run was Mon 21 Jul 2014 13:38:17 UTC [13:42:26] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: Puppet has 26 failures [13:42:26] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: Puppet has 22 failures [13:42:26] PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: Puppet has 21 failures [13:42:26] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Puppet has 57 failures [13:42:27] PROBLEM - puppet last run on search1014 is CRITICAL: CRITICAL: Puppet has 40 failures [13:42:27] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: Puppet has 66 failures [13:42:28] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Complete puppet failure [13:42:28] PROBLEM - puppet last run on ytterbium is CRITICAL: CRITICAL: Puppet has 26 failures [13:42:29] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: Puppet has 25 failures [13:42:36] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: Puppet has 60 failures [13:42:36] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: Puppet has 23 failures [13:42:37] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: Puppet has 55 failures [13:42:47] PROBLEM - puppet last run on mw1124 is CRITICAL: CRITICAL: Puppet has 57 failures [13:42:47] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: Puppet has 62 failures [13:42:47] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: Puppet has 56 failures [13:42:47] PROBLEM - puppet last run on es7 is CRITICAL: CRITICAL: Puppet has 20 failures [13:42:56] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Puppet has 16 failures [13:42:57] PROBLEM - puppet last run on es1009 is CRITICAL: CRITICAL: Puppet has 21 failures [13:42:57] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: Puppet has 19 failures [13:42:57] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 23 failures [13:42:57] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: Puppet has 26 failures [13:43:26] PROBLEM - puppet last run on es1006 is CRITICAL: CRITICAL: Puppet has 11 failures [13:43:26] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Puppet has 15 failures [13:43:26] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Puppet has 21 failures [13:43:26] PROBLEM - puppet last run on mw1192 is CRITICAL: CRITICAL: Puppet has 57 failures [13:43:26] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: Puppet has 59 failures [13:43:27] PROBLEM - puppet last run on rdb1004 is CRITICAL: CRITICAL: Puppet has 10 failures [13:43:27] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: Puppet has 72 failures [13:43:28] PROBLEM - puppet last run on mw1130 is CRITICAL: CRITICAL: Puppet has 51 failures [13:43:28] PROBLEM - puppet last run on mw1147 is CRITICAL: CRITICAL: Puppet has 58 failures [13:43:29] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [13:43:36] PROBLEM - puppet last run on virt1002 is CRITICAL: CRITICAL: Puppet has 24 failures [13:43:36] PROBLEM - puppet last run on tarin is CRITICAL: CRITICAL: Puppet has 15 failures [13:43:37] PROBLEM - puppet last run on cp4007 is CRITICAL: CRITICAL: Puppet has 18 failures [13:43:37] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: Puppet has 9 failures [13:43:46] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: Puppet has 12 failures [13:43:46] PROBLEM - puppet last run on db1007 is CRITICAL: CRITICAL: Puppet has 13 failures [13:43:46] PROBLEM - puppet last run on mw1062 is CRITICAL: CRITICAL: Puppet has 8 failures [13:43:57] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: Puppet has 20 failures [13:44:11] <_joe_> recoveries will arrive [13:44:26] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: Last successful Puppet run was Mon 21 Jul 2014 13:38:17 UTC [13:44:26] PROBLEM - puppet last run on mw1038 is CRITICAL: CRITICAL: Puppet has 1 failures [13:44:56] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: Puppet has 8 failures [13:45:09] so first log line is http://p.defau.lt/?O3_5ckm64srO_21Lr_62fQ [13:45:13] the next are identical [13:45:22] so ... wtf happened ? [13:45:44] <_joe_> akosiaris: passenger crashes and apache is not able to relive it [13:46:00] yeah but why ? [13:46:10] <_joe_> why passenger crashes? [13:46:18] yup [13:46:24] <_joe_> oh we haven't really tried to reproduce [13:46:26] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: Last successful Puppet run was Mon 21 Jul 2014 13:38:17 UTC [13:48:26] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: Last successful Puppet run was Mon 21 Jul 2014 13:38:17 UTC [13:50:26] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: Last successful Puppet run was Mon 21 Jul 2014 13:38:17 UTC [13:52:26] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: Last successful Puppet run was Mon 21 Jul 2014 13:38:17 UTC [13:54:26] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: Last successful Puppet run was Mon 21 Jul 2014 13:38:17 UTC [13:56:26] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: Last successful Puppet run was Mon 21 Jul 2014 13:38:17 UTC [13:56:36] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [13:56:37] RECOVERY - puppet last run on search1003 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [13:56:46] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [13:56:47] RECOVERY - puppet last run on analytics1021 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:56:57] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:56:57] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:57:26] RECOVERY - puppet last run on search1008 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [13:57:26] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [13:57:26] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [13:57:26] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [13:57:26] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [13:57:27] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [13:57:27] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [13:57:28] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [13:57:28] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:57:36] RECOVERY - puppet last run on fenari is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [13:57:37] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [13:57:37] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:57:46] RECOVERY - puppet last run on ssl1003 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [13:57:46] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [13:57:47] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [13:57:47] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [13:57:57] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [13:57:57] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [13:58:26] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: Last successful Puppet run was Mon 21 Jul 2014 13:38:17 UTC [13:58:26] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:58:26] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [13:58:26] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [13:58:26] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [13:58:27] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:58:27] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [13:58:28] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [13:58:28] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [13:58:29] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:58:29] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:58:30] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [13:58:30] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:58:31] RECOVERY - puppet last run on search1019 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:58:36] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [13:58:36] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [13:58:36] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:58:36] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [13:58:36] RECOVERY - puppet last run on tantalum is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:58:37] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:58:37] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:58:46] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:58:46] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [13:58:46] RECOVERY - puppet last run on zinc is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [13:58:46] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [13:58:46] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [13:58:47] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [13:58:47] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [13:58:48] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [13:58:48] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:58:49] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:58:49] RECOVERY - Puppet freshness on ssl1004 is OK: puppet ran at Mon Jul 21 13:58:40 UTC 2014 [13:58:50] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [13:58:50] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [13:58:51] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [13:58:57] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [13:58:57] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [13:58:57] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [13:58:57] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [13:58:57] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:59:18] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "Although the work is precious, there is a lot of work to be done here." (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/147994 (owner: 10Ori.livneh) [13:59:32] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [13:59:32] RECOVERY - puppet last run on ssl1004 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [13:59:32] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:59:32] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:59:32] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:59:33] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:59:33] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:59:34] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [13:59:34] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [13:59:35] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [13:59:35] RECOVERY - puppet last run on search1014 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [13:59:36] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [13:59:36] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [13:59:37] RECOVERY - puppet last run on search1021 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [13:59:37] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:59:38] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [13:59:38] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [13:59:43] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [13:59:43] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [13:59:43] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [13:59:43] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [13:59:43] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [13:59:44] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [13:59:44] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:59:45] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [13:59:45] RECOVERY - puppet last run on search1009 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [13:59:45] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:59:46] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:59:47] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [13:59:47] RECOVERY - puppet last run on es10 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [13:59:48] RECOVERY - puppet last run on virt0 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [13:59:52] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [13:59:52] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:59:52] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [13:59:52] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [13:59:53] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [13:59:53] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [14:00:02] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [14:00:02] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [14:00:02] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:00:12] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:00:22] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [14:00:22] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [14:00:32] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [14:00:32] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [14:00:32] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [14:00:33] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [14:00:33] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [14:00:42] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [14:00:43] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [14:00:43] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [14:00:43] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [14:00:43] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [14:00:43] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [14:00:44] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [14:00:44] RECOVERY - puppet last run on es7 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [14:00:45] at least puppetfails don't page :) [14:00:53] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:00:53] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [14:00:53] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [14:01:02] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [14:01:02] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [14:01:02] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [14:01:12] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [14:01:12] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:01:22] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [14:01:22] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [14:01:22] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [14:01:32] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [14:01:32] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [14:01:42] RECOVERY - puppet last run on tarin is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [14:01:43] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [14:01:43] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [14:02:02] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [14:02:02] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [14:02:02] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [14:02:02] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [14:02:42] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [14:02:42] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:02:42] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [14:02:43] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [14:03:02] RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [14:10:27] (03CR) 10CSteipp: [C: 031] "Yeah, as I said on irc, there are a couple of libraries that check /etc/passwd. It's likely things will still work without being able to r" [operations/puppet] - 10https://gerrit.wikimedia.org/r/147666 (owner: 10Mwalker) [14:13:43] (03PS2) 10Giuseppe Lavagetto: Load nutcracker server list from realm-specific yaml file [operations/puppet] - 10https://gerrit.wikimedia.org/r/147830 (owner: 10Ori.livneh) [14:18:16] (03PS2) 10Giuseppe Lavagetto: wmflib: add ensure_directory() and ensure_link() [operations/puppet] - 10https://gerrit.wikimedia.org/r/148007 (owner: 10Ori.livneh) [14:18:57] (03CR) 10Giuseppe Lavagetto: [C: 032] wmflib: add ensure_directory() and ensure_link() [operations/puppet] - 10https://gerrit.wikimedia.org/r/148007 (owner: 10Ori.livneh) [14:19:31] (03CR) 10Giuseppe Lavagetto: [V: 032] "Verified with the puppet compiler." [operations/puppet] - 10https://gerrit.wikimedia.org/r/148007 (owner: 10Ori.livneh) [14:21:42] RECOVERY - Disk space on analytics1022 is OK: DISK OK [14:25:01] (03CR) 10Alexandros Kosiaris: "Me likes this. Just have to keep in mind to use it more consistently. Perhaps document it at https://wikitech.wikimedia.org/wiki/Puppet_us" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148007 (owner: 10Ori.livneh) [14:26:46] (03CR) 10Ottomata: "Cool, sounds good. We can wait for the mirror maker patch before merging." [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/147338 (owner: 10Kmosher) [14:43:23] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "This patch needs fixing:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/147830 (owner: 10Ori.livneh) [14:49:54] (03CR) 10Ottomata: [C: 032 V: 032] "CooOol" [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/147494 (owner: 10Alexandros Kosiaris) [14:54:37] (03CR) 10Ottomata: [C: 032 V: 032] Use dfs.blocksize instead of deprecated dfs.block.size [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/148033 (owner: 10QChris) [15:00:04] Sir, Please deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140721T1500), the time has come. At your service [15:00:13] * James_F waves. [15:13:52] PROBLEM - Kafka Broker Messages In on analytics1022 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 976.442348897 [15:14:04] <^d> I'll take swat since I've got something on the list and nobody's stepping up. [15:14:31] g'morning [15:14:34] and thanks ^d :) [15:14:41] brad's on vacation, so yeah [15:14:41] <^d> morning. [15:14:44] <^d> you're welcome :) [15:14:57] (03CR) 10Chad: [C: 032] Enable TemplateData GUI tool on the Finnish Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/147632 (https://bugzilla.wikimedia.org/68184) (owner: 10Jforrester) [15:14:58] all on nik's shoulders this week, probably not the best idea [15:15:07] (03Merged) 10jenkins-bot: Enable TemplateData GUI tool on the Finnish Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/147632 (https://bugzilla.wikimedia.org/68184) (owner: 10Jforrester) [15:15:20] <^d> greg-g: Did I have vacation days wrong? [15:15:25] <^d> I thought Nik was out this week. [15:15:52] Thanks ^d! :-) [15:15:54] Also the bot should, you know, ping people again. :-) [15:16:06] ^d: oh, well then they're both out... crap [15:16:22] That'd explain. :-) [15:16:35] * James_F nominates marktraceur for SWAT duties some of the days this week. [15:16:37] * greg-g contemplates [15:16:43] Damn you [15:16:44] oh marky-boy [15:16:47] :) [15:16:52] * James_F grins. [15:17:02] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge. [15:17:03] haha [15:17:08] We should have a swat roulette wheel [15:17:16] domas! [15:17:17] !log demon Synchronized wmf-config/InitialiseSettings.php: TemplateData for fiwiki (duration: 00m 06s) [15:17:22] Logged the message, Master [15:18:03] Reedy: are you available to do some SWAT windows this week? :) [15:18:17] Yup, I should be [15:18:34] ^d: Confirmed working great. Thanks! [15:18:53] Reedy: I'll just add your name to all of the 15 UTC windows this week [15:19:03] haha [15:19:03] ok [15:19:05] (03CR) 10Alexandros Kosiaris: [C: 032] generic_vhost - outdated variable syntax [operations/puppet] - 10https://gerrit.wikimedia.org/r/147210 (owner: 10Dzahn) [15:19:05] :) [15:19:12] greg-g: You mean, each of the four 15:00 UTC windows? :-) [15:19:20] (03PS3) 10Ottomata: Add package building instruction in the README [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/147492 (owner: 10Alexandros Kosiaris) [15:19:20] <^d> James_F: yw [15:19:26] (03CR) 10Ottomata: [C: 032 V: 032] Add package building instruction in the README [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/147492 (owner: 10Alexandros Kosiaris) [15:21:02] James_F: yep, all 4 of them :P [15:21:08] (03CR) 10Chad: [C: 032] Set $wgForceUIMsgAsContentMsg for zhwikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148032 (https://bugzilla.wikimedia.org/64816) (owner: 10Odder) [15:22:10] (03Merged) 10jenkins-bot: Set $wgForceUIMsgAsContentMsg for zhwikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148032 (https://bugzilla.wikimedia.org/64816) (owner: 10Odder) [15:22:40] !log demon Synchronized wmf-config/InitialiseSettings.php: Set $wgForceUIMsgAsContentMsg for zhwikivoyage (duration: 00m 05s) [15:22:45] Logged the message, Master [15:23:50] (03CR) 10Chad: [C: 032] Set $wgUploadNavigationUrl for plwikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148030 (https://bugzilla.wikimedia.org/68191) (owner: 10Odder) [15:25:06] (03CR) 10MarkTraceur: [C: 04-1] "Don't merge until dependency reaches enwiki." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146951 (owner: 10Gergő Tisza) [15:25:08] (03Merged) 10jenkins-bot: Set $wgUploadNavigationUrl for plwikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148030 (https://bugzilla.wikimedia.org/68191) (owner: 10Odder) [15:25:24] !log demon Synchronized php-1.24wmf14/extensions/CirrusSearch: CirrusSearch to master for 1.24wmf14 (duration: 00m 07s) [15:25:29] Logged the message, Master [15:25:59] !log demon Synchronized wmf-config/InitialiseSettings.php: Set $wgUploadNavigationUrl for plwikisource (duration: 00m 04s) [15:26:04] Logged the message, Master [15:27:29] Hi ^d [15:27:47] <^d> hi hi [15:28:15] * odder confirms the plwikisource patch didn't break the site [15:28:35] (03CR) 10Chad: [C: 032] Enable FlaggedRevs on Module namespace on ukwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148029 (https://bugzilla.wikimedia.org/67748) (owner: 10Odder) [15:28:44] (03Merged) 10jenkins-bot: Enable FlaggedRevs on Module namespace on ukwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148029 (https://bugzilla.wikimedia.org/67748) (owner: 10Odder) [15:30:03] !log demon Synchronized wmf-config/flaggedrevs.php: ukwiki gets FR for NS_MODULE (duration: 00m 04s) [15:30:08] Logged the message, Master [15:30:17] James_F: welcome to the /topic ^ [15:30:29] Gosh. [15:30:45] I'm open to better short names :) [15:30:47] * James_F semi-fixes. [15:31:06] heh [15:31:08] We should invent a name and send an e-mail around. [15:31:14] yeah [15:31:18] * greg-g ponders [15:31:36] Weee! The zhwikivoyage bit works, James_F [15:32:01] It's magic. You need to check your mail for admission letters to Hogwarts, James_F [15:32:19] "Product Sage"? "Product Oracle"? "Ask the PM"? :) [15:33:02] * ^d creates #wikimedia-product for greg-g and James_F [15:33:32] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Mon 21 Jul 2014 13:33:16 UTC [15:34:16] ^d: :) [15:34:58] odder: :-) [15:35:16] ^d: I'm already in 47 channels; could I /please/ not need to join another? :-) [15:35:37] greg-g: Product Oracle sounds like it should be "Product Oracle®". [15:36:42] :) [15:37:17] greg-g: Maybe just "On Product duty:" [15:38:22] "Productifying you today: " [15:39:55] (03PS1) 10Hoo man: Make use of new lines more consistent within wikidata json dumps [operations/puppet] - 10https://gerrit.wikimedia.org/r/148094 [15:40:35] James_F: sure [15:40:56] * James_F shrugs. [15:41:01] aude: https://gerrit.wikimedia.org/r/148094 would be best if we could make that go out with the branch update for wikidata [15:41:04] Feel free to send out the e-mail with a plea for a name. :-) [15:41:10] but could also be merged earlier [15:41:21] only merging it later is really ugly [15:41:37] James_F: /me is typing [15:42:01] greg-g: If only you were in the office for me to hear your clicketty-clacking. [15:42:39] (03CR) 10Alexandros Kosiaris: beta: New script to restart apaches (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/125888 (https://bugzilla.wikimedia.org/36422) (owner: 10BryanDavis) [15:44:32] (03PS3) 10Ori.livneh: beta: add deployment-mediawiki{01,02} to scap targets [operations/puppet] - 10https://gerrit.wikimedia.org/r/148001 [15:44:38] (03CR) 10Ori.livneh: [C: 032 V: 032] beta: add deployment-mediawiki{01,02} to scap targets [operations/puppet] - 10https://gerrit.wikimedia.org/r/148001 (owner: 10Ori.livneh) [15:45:52] James_F: :P [15:47:06] Woo. [15:47:11] Status: ???? [15:47:22] Status: Trout [15:47:28] (03CR) 10Chad: [C: 032] Undeploy CommunityVoice/ClientSide extensions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/147012 (owner: 10Chad) [15:47:28] product duty? :p [15:47:37] (03Merged) 10jenkins-bot: Undeploy CommunityVoice/ClientSide extensions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/147012 (owner: 10Chad) [15:47:47] greg-g: Will anyone else ever be on product duty? :) [15:48:16] !log demon Synchronized wmf-config: Undeploying CommunityVoice/ClientSide extensions (duration: 00m 08s) [15:48:21] Logged the message, Master [15:48:27] <^d> less extensions \o/ [15:48:32] Fewer [15:48:36] <^d> lesser. [15:49:39] ^d: Lesser extensions, like what, Extension:StringFunctions? [15:50:16] <^d> I was thinking ParserFunctions but sure. [15:50:20] Heh [15:50:50] marktraceur: we'll see.... :P [15:51:43] JohnLewis: short version: some times deployers aren't familiar with the context of eg a "simple shell bug that's just a config change" and want a second pair of eyes [15:53:17] <^d> If you have a second set of eyes you should see a doctor. [15:53:19] <^d> That's not right. [15:53:31] greg-g: ah :p [15:53:52] James_F: good luck on the first ppp duty! [15:54:04] which will likely last forever :D [15:54:25] we're working on that issue (redundancy for James) :) [15:54:38] might just totally re-align the SWAT windows :) [15:54:55] where do you find a second james? [15:54:55] James will be on product duty until we find a replacement. [15:54:57] Never >:D [15:55:04] greg-g: how is this ppp as a name? [15:55:06] mark: In LCA ;) [15:55:36] matanya: just "PPP: James"? [15:55:42] maybe too cryptic :) [15:55:49] ppp duty [15:58:20] (03CR) 10Andrew Bogott: [C: 032] Set default sql server for mysql [operations/puppet] - 10https://gerrit.wikimedia.org/r/102721 (owner: 10Merlijn van Deen) [15:58:33] matanya: Ha. [16:00:04] Sir, Please deploy Wikitech (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140721T1600), the time has come. At your service [16:00:29] hmm, yeah, lost the ircnick [16:00:43] jouncebot: you're proper speaking, but forgetful now [16:01:24] <^d> Deploy wikitech? [16:01:28] 'please deploy Wikitech' wait so wikitech is not deployed yet :o [16:01:33] (03CR) 10Andrew Bogott: "Hm, rebase is ugly, can one of you please rebase by hand?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102721 (owner: 10Merlijn van Deen) [16:01:35] ^d: Wikitech upgrade. But yes. [16:01:46] <^d> Yes I know, I'm being snarky with the bot :p [16:01:49] yeah, "deploy" is hard-coded in jounce [16:02:10] deploy -> so some ops stuff with [16:02:14] *do [16:02:18] * James_F re-words the deploy slot to "Wikitech update". [16:02:21] !log updated OpenStackManager on wikitech [16:02:26] Logged the message, Master [16:02:33] "Please deploy Wikitech update" isn't too bad. [16:02:47] <^d> Won't make a difference now. [16:02:50] <^d> Bot's already done. [16:03:04] True. [16:03:09] But it makes me happy. [16:03:28] And it sets precedent for the next time we add such an entry. :-) [16:03:52] (03CR) 10Alexandros Kosiaris: [C: 032] Fix typo with scalac/scala in b918cc9e2 [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/148038 (owner: 10Alexandros Kosiaris) [16:03:57] James_F: I'll forget [16:04:01] <^d> I'm going to call my next window "me" [16:04:09] <^d> "Sir, please deploy me" [16:04:16] <^d> Or "it" [16:04:18] <^d> Deploy it! [16:04:22] ^d: That sounds vaguely inappropriate. :-) [16:04:24] Do I need to tell jouncebot that I'm done? [16:04:32] andrewbogott: No, it'd ignore you anyway. [16:04:33] James_F: add a note 'Please name your component in a way so when jouncebot says it; it doesn't make it sound like a lunitic' :p [16:04:55] JohnLewis: I worry that that would invite unhelpful naming. ;-) [16:05:39] hey - if it encourages humorous names; who cares? [16:06:13] greg-g: God King: James_F [16:06:18] bd808: your echo patches are now actually installed on wikitech. [16:06:41] * bd808 runs off to make some new instances to test [16:06:46] <^d> marktraceur: Product Potentate [16:07:02] Ooh. [16:07:07] Product Supremo: [16:07:28] bd808: hm… I now see 99+ echos that weren't there before. I hope that doesn't mean that we just send out a billion emails [16:07:37] *sent [16:07:37] <^d> James_F: There's even a costume for you. https://newlondonregalia.com/shop/products/Potentate-Costume-.html [16:07:41] Err. [16:07:48] OK. [16:08:47] Control the product, control the universe. [16:09:12] (03CR) 10Andrew Bogott: [C: 032] labs_vagrant: Ensure proper ownership of vagrant homedir [operations/puppet] - 10https://gerrit.wikimedia.org/r/147982 (owner: 10Yuvipanda) [16:09:16] and the award for best slogan goes to marktraceur [16:09:51] :D [16:10:02] Product Responsive Office Doing Unhelpful Change Traductions, or PRODUCT. [16:10:22] UK diplomatic full uniform is https://en.wikipedia.org/wiki/Court_uniform_and_dress_in_the_United_Kingdom#mediaviewer/File:James_Bryce_ca_1907.jpg [16:10:36] Acronym: PRODUCT. First word; Product :p [16:10:48] James_F: itym "PRODUCT: Responsive Office Doing Unhelpful Change Traductions" [16:10:50] JohnLewis: I'm nothing if not a techie geek. :-) [16:11:26] I had a recursive acronym for TECH [16:11:27] :p [16:11:36] Technical Enhancements Causing Havoc [16:11:44] That works. [16:12:16] Surely Technical Enhancements Causing Havoc, Nightly Embarrassments Wince-inducingly Severe? [16:12:32] <^d> I had an acronym for an application in a previous life that spelt RETARDS. [16:13:12] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Mon Jul 21 16:13:08 UTC 2014 [16:15:42] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.002 second response time [16:16:22] poor graphite [16:16:33] yeah it needs love [16:16:37] taking a look [16:19:03] (03PS1) 10Giuseppe Lavagetto: mediawiki/apache: use ports.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/148098 [16:19:04] (03PS1) 10Giuseppe Lavagetto: mediawiki: use mods-enabled, prepare for HAT [operations/puppet] - 10https://gerrit.wikimedia.org/r/148099 [16:19:12] usual deal btw, a lot of data requested which hooks all the uwsgi workers [16:19:30] <_joe_> see you at the ops meeting [16:19:42] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.003 second response time [16:19:55] !log restarted uwsgi on tungsten [16:19:59] Logged the message, Master [16:22:52] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:43] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 6.273 second response time [16:28:16] (03PS1) 10Rush: user cleanup script match system uid range [operations/puppet] - 10https://gerrit.wikimedia.org/r/148102 [16:30:44] (03CR) 10Jgreen: [C: 032 V: 031] Some additional AppArmor paths for OCG [operations/puppet] - 10https://gerrit.wikimedia.org/r/147666 (owner: 10Mwalker) [16:32:51] (03PS2) 10Rush: user cleanup script match system uid range [operations/puppet] - 10https://gerrit.wikimedia.org/r/148102 [16:32:56] (03CR) 10Rush: [C: 032] user cleanup script match system uid range [operations/puppet] - 10https://gerrit.wikimedia.org/r/148102 (owner: 10Rush) [16:33:10] (03CR) 10Rush: [V: 032] user cleanup script match system uid range [operations/puppet] - 10https://gerrit.wikimedia.org/r/148102 (owner: 10Rush) [16:36:11] James_F: there ain't no comma in there, so no :) [16:36:29] odder: ? [16:36:38] 18:12 James_F: Surely Technical Enhancements Causing Havoc, Nightly Embarrassments Wince-inducingly Severe? [16:36:48] ^^ no comma [16:36:53] odder: There's a "/" instead… [16:40:47] Not Exactly Worthy of Solicitude fits better, James_F [16:44:10] odder: Perhaps. [16:47:57] (03CR) 10Waldir: "For the record, I updated the page at https://www.mediawiki.org/wiki/Extension:FlaggedRevs/WMF_Wikis pointing here and to the server admin" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134935 (owner: 10Chad) [17:01:28] (03CR) 10Waldir: "Update: that page was deprecated; info can be found at https://meta.wikimedia.org/wiki/Flagged_Revisions#Rejected instead." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134935 (owner: 10Chad) [17:10:04] (03PS1) 10Rush: Matthew Flaschen key revoke (lost device) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148107 [17:10:26] RobH, Jeff_Green and I are profiling the pdf renderer right now [17:10:41] we think we need 3 boxes with 24 cores and lots of RAM [17:10:45] HDD doesn't really matter [17:10:54] do we have anything in the store room like that? [17:11:03] chasemp: you made it a sec before me [17:11:19] please add RT $7941 to commit message [17:11:23] *# [17:11:35] mwalker: its not 24 cores on any of our machiens [17:11:39] i think its dual 6 core [17:11:43] with hyperthreading turned on [17:11:58] so, thats a bit of a confusing thing there [17:11:59] (03CR) 10Greg Grossmeier: [C: 031] Matthew Flaschen key revoke (lost device) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148107 (owner: 10Rush) [17:12:10] RobH, um... well, whatever tantalum has [17:12:15] ok, lemme see [17:12:26] (03CR) 10Rush: [C: 032 V: 032] Matthew Flaschen key revoke (lost device) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148107 (owner: 10Rush) [17:12:54] yep, dual intel xeon X5650 [17:13:00] so its a dual cpu 6 core per cpu system [17:13:07] with hyperthreading turned on... not sure thats a good thing [17:13:26] mwalker: has anyone actually flipped hyperthreading on intentionally, or was it a mistake [17:13:30] cuz it can affect performance [17:13:33] a lot. [17:13:40] I just took it was it was given to me [17:13:41] chasemp: and when you have time, please explain to me the difference between ssh keys: [] and ssh keys: [absent] [17:13:46] (both in a positive and negative way, depdnign on the service) [17:13:46] I can test to see what happens when we turn it off though [17:13:50] yea, i think it was on by mistake then [17:13:59] we should try to turn off to get you a comparison [17:14:07] and see which is best, cuz it can matter [17:14:17] that being said, i can still see if we have more spare like it before you do so [17:14:19] lets see... [17:14:43] <^d> Who's on duty as puppetmaster this week? Still jgage? [17:15:06] chris takes over after the ops meeting [17:15:10] which is the traditional handoff time it seems [17:15:16] but, uhh [17:15:21] cmjohnson1 is now i guess [17:15:53] ^d tag I'm it [17:15:56] mwalker: So, we don't have any systems that are identical to that spec in spare. Everything with that amount of memory is single cpu. [17:16:11] we do have newer dual cpu systems that have 64gb mem [17:16:14] and dual 500GB disks [17:16:18] !log installing package upgrades on iron [17:16:19] <^d> cmjohnson1: I'm trying to find a merge for https://gerrit.wikimedia.org/r/#/c/147651/ [17:16:20] which is the closest to tantalum i have [17:16:21] <^d> yay metrics. [17:16:24] Logged the message, Master [17:16:29] matanya: that array is a literal value so empty means no keys, 'absent' would mean absent string [17:16:42] but it used to be empty value errored, I amended for another use case awhile ago [17:16:55] RobH, so what I can tell you is that I'm processing 2 req/s with tantalum -- and that I need capacity for 6 req/s -- and we're looking to also put math rendering on these boxes too [17:17:00] mwalker: So, for those I do have some spares. If you'd need some, you can request. Just make sure you include why and such in the procurement ticket, because mark will need to know for budget mapping [17:17:44] ok, so does the quick math of throwing 2 more at that make sense, or will it scale differently across multiple hosts? [17:17:58] should scale linearly [17:18:08] Does anybody know if the ops list accepts mail from non-list members? [17:18:13] (03CR) 10Cmjohnson: "Spacing issues on s4.graph that needs to be corrected first." [operations/puppet] - 10https://gerrit.wikimedia.org/r/147651 (owner: 10Chad) [17:18:18] we should turn off ht and see if it ups your req/s [17:18:21] ^d ^ [17:18:26] but can project based on that for the procurement ticket [17:18:31] NM [17:18:50] Thanks for quick response chasemp [17:18:54] hyperthreading is very strange and i dont feel qualified to predict how disabling it will change how the pdf systems work [17:19:03] StevenW: np [17:19:06] it's unlikely to affect it much [17:19:20] ok, so good to put in a procurment ticket based off the current specifications [17:19:23] ? [17:19:24] mwalker: what's the 6 req/s based on? [17:19:33] peak per hour requests over a week of data [17:19:45] <^d> cmjohnson1: Trailing w/s at end of file? On it. [17:19:46] that was the highest load we saw [17:19:54] hm ok [17:20:05] so 3 boxes is still a bit tight? [17:20:31] We presently have 6 spare Dell PowerEdge R420, dual Intel Xeon E5-2450 v2 2.50GHz, 64GB Memory, (2) 500GB Disks (which is what I would allocate to match what tantalum, which is older gen of this) [17:20:38] well; some number of those are going to be cached (I dont know how many); and I'm going to reduce some IO load which will give us ~5% [17:20:40] https://wikitech.wikimedia.org/wiki/Server_Spares [17:20:48] but yes; probably a little tight if we also do math rendering on the same boxes [17:20:57] but I dont know what that load is going to look like [17:21:17] whether we'll want to do math rendering on these boxes we can look at later [17:21:32] PROBLEM - LVS HTTP IPv4 on ocg.svc.eqiad.wmnet is CRITICAL: Connection refused [17:21:45] thats still the testing stuff right? [17:21:46] RobH: go ahead procuring 3 of those boxes for them [17:21:49] (it just generated a page) [17:21:51] it's entering production soonish [17:21:51] the ocg. [17:22:13] (03PS2) 10Chad: Add average/90th pct search latency for 1 hour window [operations/puppet] - 10https://gerrit.wikimedia.org/r/147651 [17:22:17] I have approval from greg-g to make it a render option this week if we can get the hardware [17:22:33] so 3 additional ocg boxes for eqiad right? [17:22:36] mwalker: ^? [17:22:54] I would say give me 3 and take back tantalum? [17:23:02] Oh, even nicer [17:23:04] that makes sense to me [17:23:09] Well, I assume take it back once these are live right? [17:23:13] *nods* [17:23:23] which... depends on how fast we can get them set up [17:23:25] cool, I'll create the related tickets for this and add you as a requestor for them [17:23:32] RECOVERY - LVS HTTP IPv4 on ocg.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 238 bytes in 0.010 second response time [17:23:34] well, I'll get them allocated today [17:23:37] RobH: yeah, afaik still configured to QA host tantalum and in testing (ocg) [17:23:39] mwalker: who's helping you with that? [17:23:45] brandon? [17:23:47] I'll put it back in downtime for a while so we don't get spammed [17:23:50] yes [17:23:53] ok [17:24:08] (03CR) 10Chad: "PS2 fixes trailing w/s in s-2 and s-4." [operations/puppet] - 10https://gerrit.wikimedia.org/r/147651 (owner: 10Chad) [17:24:08] !log elastic1009,analytics1004,silver, various misc. boxes - upgrading libssl [17:24:13] Logged the message, Master [17:24:24] so I'll allocate the three new boxes and get them with OS and handed off to brandon.... by end of day tomorrow? (I wanna give myself some time if something goes badly) [17:24:40] i already have enough stuff today to keep me busy playing catchup ;] [17:24:41] would be better t try for sooner [17:24:51] if there are no problems on os install then today yes [17:24:59] thanks [17:25:00] RobH, *thumbs up* [17:25:05] but everytime i promise a monday server soemthing is badly wrong [17:25:08] hehe [17:25:11] understood [17:25:13] in the installer, so if no problems, then end of day today [17:25:18] :) [17:25:19] it needs to be ubuntu 14.04 btw [17:25:28] gtk [17:27:08] when that kinda thing happens it makes me think we're a few steps away from wmf run bouncer =P [17:27:19] RobH: I can do the OS if you don't have time, if mgmt is set up in dns for them so I know where they are. [17:27:37] I'll give you OS if they have an issue in the isntaller, otherwise its no problem. [17:27:39] heh [17:27:51] ok [17:28:17] so these are ocg servers right? [17:28:21] right [17:28:23] so i should justname them ocg1001-1003 [17:28:24] cool. [17:28:31] that makes it even easier [17:28:32] that seems reasonable [17:28:42] .eqiad.wmnet, in private IP subnet [17:28:46] its already matching https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions =] [17:29:18] other things to know; externally these talk to parsoid, the api cluster, and redis -- and they are independent (e.g. they basically just run jobs) [17:29:24] I dont know if that changes where you put them in the racks [17:29:29] sounds good. 3 of the dual cpu 64gb dual 500gb disk systems ocg1001-1003, trusty os, internal vlan [17:29:46] mwalker: nah, these are just our normal internal vlans which can all chat with one another [17:29:54] firewallign may matter on individual hsots, dunno. [17:30:06] I already have ferm rules in the puppet config [17:30:24] but if there's an external firewall I can provide the rulesets I'll need [17:30:36] there isn't [17:30:55] mwalker: base::firewall on the node and some rules in the role? [17:31:05] *nods* [17:31:11] cool [17:31:55] he has a ferm rule on the ocg hosts themselves to limit to internal inbound traffic [17:32:20] I don't think we actually need that, since they're in private space, but I guess it doesn't hurt. [17:34:58] (03CR) 10Cmjohnson: [C: 032] Add average/90th pct search latency for 1 hour window [operations/puppet] - 10https://gerrit.wikimedia.org/r/147651 (owner: 10Chad) [17:36:32] (03PS1) 10Jgreen: change ocg temp dir to /mnt/tmpfs [operations/puppet] - 10https://gerrit.wikimedia.org/r/148116 [17:40:24] (03PS4) 10Tim Landscheidt: Set default sql server for mysql [operations/puppet] - 10https://gerrit.wikimedia.org/r/102721 (owner: 10Merlijn van Deen) [17:41:02] (03PS1) 10Gage: Erik Zachte: key revoke (client compromised) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148118 [17:41:38] (03CR) 10Rush: [C: 031] Erik Zachte: key revoke (client compromised) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148118 (owner: 10Gage) [17:42:09] (03CR) 10Dzahn: [C: 031] Erik Zachte: key revoke (client compromised) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148118 (owner: 10Gage) [17:43:11] (03CR) 10BryanDavis: [C: 031] Describe Math related packages in a class [operations/puppet] - 10https://gerrit.wikimedia.org/r/115133 (https://bugzilla.wikimedia.org/61090) (owner: 10Hashar) [17:43:20] (03CR) 10Gage: [C: 032] Erik Zachte: key revoke (client compromised) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148118 (owner: 10Gage) [17:45:09] (03CR) 10Jgreen: [C: 032 V: 031] change ocg temp dir to /mnt/tmpfs [operations/puppet] - 10https://gerrit.wikimedia.org/r/148116 (owner: 10Jgreen) [17:48:09] (03CR) 10BryanDavis: [C: 031] filter changes to support messages from Hadoop [operations/puppet] - 10https://gerrit.wikimedia.org/r/140623 (owner: 10Gage) [17:49:35] (03CR) 10Andrew Bogott: [C: 032] Set default sql server for mysql [operations/puppet] - 10https://gerrit.wikimedia.org/r/102721 (owner: 10Merlijn van Deen) [17:50:51] greg-g, regarding the petition extension; we have to wait till thurs to get that on the cluster? [17:51:15] (given that I deployed to beta last thurs) -- are there any other steps? [17:52:07] (03CR) 10Rush: phab-login screen HTML-replace deprecated HTML (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/147640 (owner: 10Dzahn) [17:52:33] (03CR) 10BryanDavis: [C: 031] phab-login screen HTML-replace deprecated HTML [operations/puppet] - 10https://gerrit.wikimedia.org/r/147640 (owner: 10Dzahn) [17:54:23] (03CR) 10Dzahn: phab-login screen HTML-replace deprecated HTML (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/147640 (owner: 10Dzahn) [17:55:45] (03CR) 10Rush: phab-login screen HTML-replace deprecated HTML (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/147640 (owner: 10Dzahn) [17:59:36] <^d> cmjohnson1: New stats look great, thx! [17:59:50] great news! YW [18:06:21] mwalker: I'm not always a stickler for 168 hours on beta cluster, anything crazy found on the beta cluster? [18:06:38] mwalker: the security review and perf reveiws are done, right? [18:06:49] (well, perf can be "partly" in this case, probably) [18:09:07] greg-g, looks like peter addressed all of the comments in the perf review bug: https://bugzilla.wikimedia.org/show_bug.cgi?id=65851 [18:09:14] so I'd say yes; the reviews are completed [18:09:22] and the extension works as expected on beta [18:09:39] (03CR) 10Hashar: [C: 04-1] "We need them :-) That is used by browser tests triggered when one send a patch in Gerrit. The underlying puppet error has to be fixed in" [operations/puppet] - 10https://gerrit.wikimedia.org/r/147713 (https://bugzilla.wikimedia.org/68260) (owner: 10Krinkle) [18:10:09] mwalker: cool, then whenever ya'll need it starting tomorrow [18:10:59] greg-g, may I take the 10 - 11 window tomorrow? [18:11:17] mwalker: yeppers [18:12:13] (03PS1) 10Ori.livneh: wmflib: add ubuntu_is() function [operations/puppet] - 10https://gerrit.wikimedia.org/r/148123 [18:13:04] (03PS1) 10Matanya: access: add new ezachte key [operations/puppet] - 10https://gerrit.wikimedia.org/r/148124 [18:17:54] (03Abandoned) 10Hashar: contint: Remove browsertests class from role::ci::slave::labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/147713 (https://bugzilla.wikimedia.org/68260) (owner: 10Krinkle) [18:18:23] (03CR) 10Erik Zachte: [C: 031] access: add new ezachte key [operations/puppet] - 10https://gerrit.wikimedia.org/r/148124 (owner: 10Matanya) [18:28:42] greg-g, on wed I'd like to take the 1400-1600 deploy window to push out something like 2 months of centralnotice changes [18:29:26] I'm going to add that to the calendar but we can talk about it if you want [18:29:55] mwalker: whatevs [18:29:57] :) [18:30:18] heh; so long as I dont break the cluster; otherwise ! thumbscrews! [18:30:28] indeed [18:32:35] its going to be a busy deploy week for me [18:34:49] :) [18:48:16] (03CR) 1020after4: [C: 031] wmflib: add ubuntu_is() function [operations/puppet] - 10https://gerrit.wikimedia.org/r/148123 (owner: 10Ori.livneh) [18:50:58] (03CR) 10Ori.livneh: [C: 031] jobrunner: create hhvm-only jobrunners [operations/puppet] - 10https://gerrit.wikimedia.org/r/147086 (owner: 10Giuseppe Lavagetto) [18:57:35] (03CR) 10Ori.livneh: [C: 031] mediawiki/apache: use ports.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/148098 (owner: 10Giuseppe Lavagetto) [18:58:14] (03CR) 10Rush: [C: 04-1] "don't want to be mr. negative but has anyone verified this is all clear? We voided based on the assumption there was a compromise, was th" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148124 (owner: 10Matanya) [18:58:46] (03PS1) 10RobH: adding ocg1001-1003 to dns [operations/dns] - 10https://gerrit.wikimedia.org/r/148131 [19:00:24] So who wants to +1 my dns change to prove im not nuts? [19:00:33] (03PS1) 10Yuvipanda: dynamicproxy: Block requests with no UA specified [operations/puppet] - 10https://gerrit.wikimedia.org/r/148133 [19:00:34] cmjohnson1: yer poc [19:00:36] so ping! [19:00:40] Betacommand: ^ will block requests with no UA [19:00:56] added you as reviewer, just +1 if sane i can merge. [19:04:18] (03CR) 10Erik Zachte: "no the one device that was compromised is iphone" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148124 (owner: 10Matanya) [19:05:29] (03CR) 10Erik Zachte: "and iphone was reinstalled and upgraded from iOS 6 to iOS7 today (last time I did jailbreak)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148124 (owner: 10Matanya) [19:08:01] (03PS1) 10Ottomata: Remove WMF specific 'standard' and 'adhoc' queues from default fair-scheduler.xml.erb [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/148137 [19:09:39] (03PS2) 10Ottomata: Remove WMF specific 'standard' and 'adhoc' queues from default fair-scheduler.xml.erb [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/148137 [19:09:50] (03PS1) 10JanZerebecki: Remove blog vhosts that do not exist in DNS anymore. [operations/puppet] - 10https://gerrit.wikimedia.org/r/148138 [19:10:53] (03PS3) 10Ottomata: Remove WMF specific 'standard' and 'adhoc' queues from default fair-scheduler.xml.erb [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/148137 [19:11:04] (03CR) 10Ottomata: [C: 032 V: 032] Remove WMF specific 'standard' and 'adhoc' queues from default fair-scheduler.xml.erb [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/148137 (owner: 10Ottomata) [19:11:50] robh: are we continuing to add to the mess that is wmnet file?(specifically mgmt) [19:12:34] cmjohnson1: what do you mean? [19:12:38] (03CR) 10Cmjohnson: [C: 031] adding ocg1001-1003 to dns [operations/dns] - 10https://gerrit.wikimedia.org/r/148131 (owner: 10RobH) [19:12:43] I've added the new entries in the correct areas [19:12:49] the only messy part is assettag area [19:12:56] but im not willing to redo it all today, heh [19:13:27] don't know why not :-P [19:14:15] (03PS1) 10RobH: setup new ocg1001-1003 servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/148140 [19:14:31] i dont htink the folks im blocking wanna wait for me to do a cleanup of the file ;D [19:16:00] (03CR) 10RobH: [C: 032] setup new ocg1001-1003 servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/148140 (owner: 10RobH) [19:16:06] (03PS1) 10Ottomata: Update for cdh module, move fair-schedule.xml.erb queue allocation to puppet repo [operations/puppet] - 10https://gerrit.wikimedia.org/r/148143 [19:16:14] (03CR) 10RobH: [C: 032] adding ocg1001-1003 to dns [operations/dns] - 10https://gerrit.wikimedia.org/r/148131 (owner: 10RobH) [19:17:44] (03PS2) 10Ottomata: Update for cdh module, move fair-schedule.xml.erb queue allocation to puppet repo [operations/puppet] - 10https://gerrit.wikimedia.org/r/148143 [19:19:32] PROBLEM - DPKG on db1017 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:19:47] db1017 that's me, sec [19:19:52] (03CR) 10Ottomata: [C: 032 V: 032] Update for cdh module, move fair-schedule.xml.erb queue allocation to puppet repo [operations/puppet] - 10https://gerrit.wikimedia.org/r/148143 (owner: 10Ottomata) [19:20:22] ACKNOWLEDGEMENT - DPKG on db1017 is CRITICAL: DPKG CRITICAL dpkg reports broken packages Filippo Giunchedi monitoring/opentsdb tests [19:20:34] (03PS2) 10Ori.livneh: wmflib: add ubuntu_version() function [operations/puppet] - 10https://gerrit.wikimedia.org/r/148123 [19:24:37] (03CR) 10BBlack: wmflib: add ubuntu_version() function (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148123 (owner: 10Ori.livneh) [19:25:12] PROBLEM - Unmerged changes on repository puppet on virt0 is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [19:25:23] (03CR) 10Ori.livneh: wmflib: add ubuntu_version() function (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148123 (owner: 10Ori.livneh) [19:25:38] (03PS1) 10JanZerebecki: Correct blog redirects to not go from HTTPS to HTTP [operations/puppet] - 10https://gerrit.wikimedia.org/r/148159 [19:26:25] jzerebecki: anything related to blog is likely a waste of time at this point.. it is supposed to move.. today [19:26:35] (03PS3) 10Ori.livneh: wmflib: add ubuntu_version() function [operations/puppet] - 10https://gerrit.wikimedia.org/r/148123 [19:27:09] (03CR) 10Dzahn: "blog is scheduled to move away from our servers today or very soon" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148159 (owner: 10JanZerebecki) [19:27:41] (03CR) 10Dzahn: "blog is scheduled to move away from our servers today or very soon" [operations/puppet] - 10https://gerrit.wikimedia.org/r/147739 (https://bugzilla.wikimedia.org/53259) (owner: 10Chmarkine) [19:27:56] (03CR) 10Dzahn: "blog is scheduled to move away from our servers today or very soon" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148138 (owner: 10JanZerebecki) [19:29:59] mutante: any ticket id for that move? [19:31:24] (03PS1) 10Matanya: etherpad: monitor web access [operations/puppet] - 10https://gerrit.wikimedia.org/r/148164 [19:31:30] jzerebecki: RT-528 [19:31:31] jzerebecki: ask RobH [19:31:35] jzerebecki: RT-5288 [19:32:37] (03CR) 10Rush: [C: 031] "godspeed" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148123 (owner: 10Ori.livneh) [19:33:01] (03PS1) 10JanZerebecki: Add ssl cipher and protocol setting for techblog.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/148170 [19:33:23] (03CR) 10BBlack: [C: 031] wmflib: add ubuntu_version() function [operations/puppet] - 10https://gerrit.wikimedia.org/r/148123 (owner: 10Ori.livneh) [19:33:50] (03PS4) 10Ori.livneh: wmflib: add ubuntu_version() function [operations/puppet] - 10https://gerrit.wikimedia.org/r/148123 [19:33:55] (03CR) 10Ori.livneh: [C: 032 V: 032] wmflib: add ubuntu_version() function [operations/puppet] - 10https://gerrit.wikimedia.org/r/148123 (owner: 10Ori.livneh) [19:34:28] jgage: https://gerrit.wikimedia.org/r/#/c/148164/ <-- etherpad outage actionable from me :) [19:35:12] RECOVERY - Unmerged changes on repository puppet on virt0 is OK: No changes to merge. [19:35:27] the blog is slated to migrate today but no updated email yet [19:35:38] matanya: you are awesome, thanks! [19:35:40] nor request for the final export [19:36:12] :) [19:36:30] (03PS1) 10Tim Landscheidt: Tools: Deploy root web automatically [operations/puppet] - 10https://gerrit.wikimedia.org/r/148172 [19:37:10] (03Abandoned) 10JanZerebecki: Correct blog redirects to not go from HTTPS to HTTP [operations/puppet] - 10https://gerrit.wikimedia.org/r/148159 (owner: 10JanZerebecki) [19:37:40] (03CR) 10Gage: [C: 032] etherpad: monitor web access [operations/puppet] - 10https://gerrit.wikimedia.org/r/148164 (owner: 10Matanya) [19:38:16] (03CR) 10JanZerebecki: "According to RT: 5288 blog will be external soon." [operations/puppet] - 10https://gerrit.wikimedia.org/r/148170 (owner: 10JanZerebecki) [19:38:31] (03Abandoned) 10JanZerebecki: Add ssl cipher and protocol setting for techblog.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/148170 (owner: 10JanZerebecki) [19:38:54] (03PS1) 10RobH: forgot to include trusty for ocg1001-1033 [operations/puppet] - 10https://gerrit.wikimedia.org/r/148173 [19:39:17] (03Abandoned) 10JanZerebecki: Remove blog vhosts that do not exist in DNS anymore. [operations/puppet] - 10https://gerrit.wikimedia.org/r/148138 (owner: 10JanZerebecki) [19:40:55] (03CR) 10RobH: [C: 032] forgot to include trusty for ocg1001-1033 [operations/puppet] - 10https://gerrit.wikimedia.org/r/148173 (owner: 10RobH) [19:41:21] (03PS4) 10Matanya: mailman: monitor queue size [operations/puppet] - 10https://gerrit.wikimedia.org/r/146756 [19:43:35] matanya: let's check a string on HTTPS ? [19:43:57] mutante: etherpad related ? [19:44:02] matanya: yea [19:44:11] pushing a patch [19:44:14] check_https_url_for_string [19:44:21] Etherpad or something [19:44:21] <mutante> cool [19:50:12] <icinga-wm> PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [19:50:12] <icinga-wm> PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [19:51:09] <matanya> RobH / jgage ^ ? [19:51:23] <mutante> cmjohnson1: i cant read what you said ... [19:51:38] <grrrit-wm> (03PS1) 10Matanya: etherpad: monitor https access [operations/puppet] - 10https://gerrit.wikimedia.org/r/148211 [19:51:50] <matanya> mutante: not sure if this ^ is what you meant [19:53:00] <RobH> fixed [19:53:12] <icinga-wm> RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge. [19:53:13] <icinga-wm> RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [19:53:26] <mutante> matanya: almost, but let me amend something [19:53:30] <matanya> sure [19:54:08] <grrrit-wm> (03CR) 10Gage: "This key appears to be 1024 bits. Please generate a key of size 2048 bits." [operations/puppet] - 10https://gerrit.wikimedia.org/r/148124 (owner: 10Matanya) [19:54:13] <James_F> hashar: Can you tell why Beta Labs (deployment-prep) is in read-only "sync with slaves" mode? Did some make a DB schema change? [20:00:32] <MatmaRex> James_F: https://gerrit.wikimedia.org/r/117373 ? [20:00:33] <grrrit-wm> (03PS5) 10BryanDavis: beta: New script to restart apaches [operations/puppet] - 10https://gerrit.wikimedia.org/r/125888 (https://bugzilla.wikimedia.org/36422) [20:01:30] <subbu> time to do a parsoid deploy [20:02:50] <grrrit-wm> (03PS2) 10Dzahn: etherpad: monitor https access [operations/puppet] - 10https://gerrit.wikimedia.org/r/148211 (owner: 10Matanya) [20:03:05] <mutante> aaarrrrr [20:04:15] <grrrit-wm> (03CR) 10BryanDavis: beta: New script to restart apaches (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/125888 (https://bugzilla.wikimedia.org/36422) (owner: 10BryanDavis) [20:07:08] <grrrit-wm> (03PS3) 10Dzahn: etherpad: monitor https access [operations/puppet] - 10https://gerrit.wikimedia.org/r/148211 (owner: 10Matanya) [20:07:57] <grrrit-wm> (03CR) 10Dzahn: [C: 031] "root@neon:~# /usr/lib/nagios/plugins/check_http -H etherpad.wikimedia.org -I 208.80.154.41 -u /p/Etherpad -s '<title>Etherpad'" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148211 (owner: 10Matanya) [20:08:14] <bd808|LUNCH> James_F: Db update script is actively running in beta -- https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/ [20:08:48] <James_F> bd808: Aha, thanks. Clearly we need fewer Beta Labs wikis. :-) [20:08:50] <grrrit-wm> (03CR) 10Dzahn: "eh, with -S of course:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148211 (owner: 10Matanya) [20:09:10] <grrrit-wm> (03PS2) 10Ori.livneh: mediawiki::jobrunner: deduplicate job groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/147871 [20:09:11] <subbu> !log deployed parsoid version 1c9277d6 [20:09:16] <morebots> Logged the message, Master [20:09:24] <grrrit-wm> (03CR) 10Dzahn: [C: 032] etherpad: monitor https access [operations/puppet] - 10https://gerrit.wikimedia.org/r/148211 (owner: 10Matanya) [20:10:54] <icinga-wm> PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:11:38] <bd808> James_F: There is some slow as heck migration in there too -- "Updating *_from_namespace fields in links tables." -- https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/label=deployment-bastion-eqiad,wikidb=commonswiki/2739/console [20:12:05] <James_F> bd808: Oh dear. Is that the Flow fix? [20:12:08] <grrrit-wm> (03PS1) 10Plucas: Check KAFKA_START before JAVA_HOME (kafka-mirror) [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/148217 [20:12:11] <James_F> bd808: That… will take a while. [20:12:54] <icinga-wm> RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [20:13:59] <grrrit-wm> (03CR) 10Plucas: "Submitted in a second review: https://gerrit.wikimedia.org/r/#/c/148217/" [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/147338 (owner: 10Kmosher) [20:14:11] <grrrit-wm> (03PS2) 10Matanya: access: add new ezachte key [operations/puppet] - 10https://gerrit.wikimedia.org/r/148124 [20:14:52] <grrrit-wm> (03CR) 10Dzahn: [C: 032] icinga - update SSL cipher list [operations/puppet] - 10https://gerrit.wikimedia.org/r/147207 (https://bugzilla.wikimedia.org/53259) (owner: 10Dzahn) [20:15:14] <grrrit-wm> (03PS3) 10Dzahn: icinga - update SSL cipher list [operations/puppet] - 10https://gerrit.wikimedia.org/r/147207 (https://bugzilla.wikimedia.org/53259) [20:15:45] <grrrit-wm> (03CR) 10Erik Zachte: [C: 031] access: add new ezachte key [operations/puppet] - 10https://gerrit.wikimedia.org/r/148124 (owner: 10Matanya) [20:16:22] <grrrit-wm> (03PS2) 10Ottomata: Check KAFKA_START before JAVA_HOME [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/147338 (owner: 10Kmosher) [20:16:27] <grrrit-wm> (03CR) 10Ottomata: [C: 032 V: 032] Check KAFKA_START before JAVA_HOME [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/147338 (owner: 10Kmosher) [20:16:31] <grrrit-wm> (03PS2) 10Ottomata: Check KAFKA_START before JAVA_HOME (kafka-mirror) [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/148217 (owner: 10Plucas) [20:16:35] <grrrit-wm> (03CR) 10Ottomata: [C: 032 V: 032] Check KAFKA_START before JAVA_HOME (kafka-mirror) [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/148217 (owner: 10Plucas) [20:16:50] <grrrit-wm> (03CR) 10Ottomata: "Ok! Both merged. Thank you!" [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/147338 (owner: 10Kmosher) [20:20:21] <grrrit-wm> (03CR) 10Andrew Bogott: puppetception: Initial commit (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/147759 (owner: 10Yuvipanda) [20:21:02] <grrrit-wm> (03PS1) 10Jgreen: unpuppetize deprecated aluminium config [operations/puppet] - 10https://gerrit.wikimedia.org/r/148218 [20:21:17] <grrrit-wm> (03CR) 10Yuvipanda: [C: 031] puppetception: Initial commit (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/147759 (owner: 10Yuvipanda) [20:21:22] <kaldari> beta labs is broken again [20:21:35] <kaldari> "The Wikipedia database is temporarily in read-only mode." [20:21:45] <matanya> kaldari: thanks for the fix in mobile [20:21:57] <kaldari> NP [20:22:05] <legoktm> kaldari: it's doing a schema update [20:22:09] <kaldari> ah [20:22:14] <legoktm> https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/ [20:23:03] <kaldari> legoktm: any idea how long that might take? [20:23:08] <legoktm> nope [20:23:36] <grrrit-wm> (03CR) 10Jgreen: [C: 032 V: 031] unpuppetize deprecated aluminium config [operations/puppet] - 10https://gerrit.wikimedia.org/r/148218 (owner: 10Jgreen) [20:23:58] <grrrit-wm> (03CR) 10Krinkle: "@Hashar: You misunderstood. This isn't removing it from any instance. This removes it from the default role::ci::slave::labs class so that" [operations/puppet] - 10https://gerrit.wikimedia.org/r/147713 (https://bugzilla.wikimedia.org/68260) (owner: 10Krinkle) [20:24:03] <Krinkle> hashar: ^ [20:25:00] <hashar> role::ci::slave::labs is the class to be applied on all jenkins slaves and they should be able to run browsertests [20:25:03] <icinga-wm> PROBLEM - https.etherpad.wikimedia.org on zirconium is CRITICAL: (null) [20:25:04] <grrrit-wm> (03PS1) 10Plucas: Use log4j-1.2 instead of log4j-1.2.16 [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/148219 [20:25:13] <mutante> icinga-wm: bah, new monitoring , no worries [20:25:14] <matanya> mutante: ^^ [20:25:17] <hashar> Krinkle: I am pretty sure it is a dependency issue [20:25:17] <mutante> YES [20:25:25] <hashar> Krinkle: though it is supposed to be fixed [20:25:55] <hashar> Krinkle: wanna debug it together over hangout ? I have half an hour ahead :-] [20:26:22] <grrrit-wm> (03CR) 10Andrew Bogott: [C: 032] puppetception: Initial commit [operations/puppet] - 10https://gerrit.wikimedia.org/r/147759 (owner: 10Yuvipanda) [20:26:46] <Krinkle> hashar: I spend 2 days debugging it already, I know every detail of it. [20:26:53] <Krinkle> The packages are not available on trusty yet. [20:26:56] <Krinkle> that's all. [20:27:06] <Krinkle> However I think these roles should be separated regardless [20:27:13] <icinga-wm> PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There are 5 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [20:27:14] <Krinkle> To keep the code more modular. [20:27:14] <icinga-wm> PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There are 5 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [20:27:20] <Krinkle> After all, they are also separate labels in jenkins/zuul. [20:27:37] <Krinkle> Or we can make them separate labels. [20:28:21] <hashar> Krinkle: it uses to a be a different class running on a dedicated slave [20:28:32] <hashar> I have merged it back with role::ci::slave::labs [20:28:49] <hashar> that will eventually be used to create a generic openstack instance we will use for all kind of jobs [20:29:08] <icinga-wm> RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge. [20:29:08] <icinga-wm> RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [20:29:24] <hashar> Krinkle: is that on integration-slave1005.eqiad.wmflabs ? [20:29:53] <Krinkle> hashar: integration-slave1005 is precise, set up according to the new manual for setting up instances. [20:30:03] <Krinkle> with node 0.10 [20:30:05] <Krinkle> not pooled right now [20:30:08] <Krinkle> just experimental [20:30:11] <Krinkle> I ran a few jobs on it to test [20:30:22] <grrrit-wm> (03PS1) 10Ori.livneh: role::mediawiki: include mediawiki::web::sites in role class [operations/puppet] - 10https://gerrit.wikimedia.org/r/148221 [20:30:22] <hashar> ok [20:30:27] <hashar> that one has an unrelated error [20:30:28] <hashar> npm : Depends: nodejs-dev (>= 0.4) [20:30:28] <hashar> E: Unable to correct problems, you have held broken packages. [20:31:07] <Krinkle> nodejs-dev and other such old packages are from ubuntu, not chris's ppa. [20:31:16] <Krinkle> the nodejs package there includes npm and build tools. [20:31:25] <Krinkle> (which npm needs to install a package) [20:31:27] <hashar> where is the trusty instance ? :D [20:31:27] <Krinkle> so that one should be unaccessible [20:31:34] <Krinkle> there is none because nothing works on trusty [20:31:36] <Krinkle> because of missing packages [20:31:40] <Krinkle> I filed various bugs about this [20:31:54] <Krinkle> they are blocking the tracker bug "Jenkins: .* trusty" [20:32:10] <hashar> yeah I noticed those bugs today will triaging my emails [20:32:31] <Krinkle> I created it as 1004 but it failed to provide anythign [20:32:42] <Krinkle> then it turned out 1001-1003 were not providing since May 2014 either [20:32:47] <hashar> doh [20:32:53] <Krinkle> because the classes broke due to conflict with mediawiki manifest [20:33:08] <icinga-wm> PROBLEM - DPKG on platinum is CRITICAL: DPKG CRITICAL dpkg reports broken packages [20:33:12] <hashar> ah yeah I remember puppet broke when Ori started refactoring mediawiki classes or something along those lines [20:33:22] <Krinkle> so I spend almost 2 days chasing down different random manifests with ops to unbreak it [20:33:29] <Krinkle> hashar: when did you know about this? [20:34:07] <icinga-wm> RECOVERY - https.etherpad.wikimedia.org on zirconium is OK: HTTP OK: HTTP/1.1 200 OK - 28846 bytes in 0.037 second response time [20:34:07] <icinga-wm> RECOVERY - DPKG on platinum is OK: All packages OK [20:34:34] <grrrit-wm> (03CR) 10Ori.livneh: [C: 032] "OK'd by Giuseppe" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148221 (owner: 10Ori.livneh) [20:34:56] <icinga-wm> PROBLEM - HTTP on aluminium is CRITICAL: Connection refused [20:35:33] <grrrit-wm> (03PS1) 10Dzahn: fix escaping in etherpad https icinga check [operations/puppet] - 10https://gerrit.wikimedia.org/r/148223 [20:37:00] <hashar> Krinkle: at some point during my burn out beginning of july ? [20:37:29] <hashar> Krinkle: puppet has been breaking every single week on either integration or beta since day 1 [20:37:30] <grrrit-wm> (03CR) 10Aaron Schulz: jobrunner: create hhvm-only jobrunners (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/147086 (owner: 10Giuseppe Lavagetto) [20:37:42] <Krinkle> hashar: Aye.. we need to talk more often. This is good information. [20:37:42] <icinga-wm> ACKNOWLEDGEMENT - HTTP on aluminium is CRITICAL: Connection refused daniel_zahn RT-4667 (- Jeff) [20:37:48] <hashar> Krinkle: so sometime I am fed up figuring out what is going wrong, let the issues pill up and eventually fix them after a few weeks [20:37:57] <hashar> Krinkle: yeah we need to resurrect the CI checkin :D [20:38:11] <Krinkle> Yeah, that's fine to let it pile up, but have the pile be visible to me :D [20:38:20] <Krinkle> so I don't build the same pile [20:38:23] <hashar> we need puppet monitoring as well [20:38:24] <grrrit-wm> (03CR) 10Dzahn: [C: 032] fix escaping in etherpad https icinga check [operations/puppet] - 10https://gerrit.wikimedia.org/r/148223 (owner: 10Dzahn) [20:38:28] <Krinkle> Yep [20:38:31] <Krinkle> Labs needs monitoring. [20:38:35] <Krinkle> like, at all. [20:38:35] <hashar> beta cluster has the same issue [20:38:48] <hashar> and I think Bryan / Yuvi worked on something to report puppet errors [20:39:21] <Krinkle> basic icinga/ganglia would be enough. Not even custom monitors, though that'd be cool of course. [20:39:31] <hashar> the monitoring, there is now diamond (a daemon that collect a bunch of metrics and send them to graphite). So we can reuse the graphite data to replace ganglia and add icinga checks :- D [20:39:39] <Krinkle> Right [20:39:46] <Krinkle> No alerts though, right? [20:40:01] <hashar> meanwhile, can you create a Trusty instance so we get puppet fixed on it ? :D [20:40:11] <hashar> integration-slave1004-trusty would do I guess :D [20:40:12] <mutante> how crazy is it to suggest we apply changes first on beta [20:40:31] <hashar> mutante: that is what most devs team are doing nowadays. [20:40:34] <ori> it depends: how crazy is beta? [20:40:40] <hashar> mutante: and some ops as well depending on the project / prod impact [20:40:49] <Krinkle> hashar: I did that last weekend, I reported what I found. [20:40:56] <hashar> mutante: I know mobile is playing with varnish VCL on beta :D [20:40:57] <greg-g> mutante: if you can convince ops :) [20:41:03] <Krinkle> It's a fatal error each time, so it doesn't go further until those are fixed. [20:41:08] <icinga-wm> PROBLEM - https.etherpad.wikimedia.org on zirconium is CRITICAL: (null) [20:41:13] <hashar> Krinkle: recreate it and lets iterate together until puppet pass [20:41:24] <Krinkle> hashar: are those packages published for Trusty? [20:41:28] <icinga-wm> PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Puppet has 1 failures [20:41:36] <hashar> Krinkle: nop we need ops to build the packages [20:41:44] <hashar> Krinkle: parsekit got imported from pecl iirc [20:42:13] <Krinkle> you can create it from wikitech in a minute if you want. m1.large with trusty and apply the classes https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Setup [20:42:22] <matanya> mutante: what is up with this check ? [20:42:59] <Krinkle> hashar: since when does integration have a puppetmaster btw? And the first steps I wrote on https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Setup (to apply the class, then force a puppet run, ca cert etc.) were those documented elsewhere already? [20:43:07] <Krinkle> I couldn't find it, took me a long time to figure it out why it wans't working etc. [20:43:11] <hashar> Krinkle: since I got fed up waiting for ops :-D [20:43:37] <hashar> I did it with Bryan when he switched beta cluster to its own puppetmaster [20:44:10] <icinga-wm> ACKNOWLEDGEMENT - https.etherpad.wikimedia.org on zirconium is CRITICAL: (null) daniel_zahn false alarm, daniel on it [20:44:30] <Krinkle> sounds like a romantic way to end a hacking weekend ^_^ [20:44:42] <grrrit-wm> (03PS1) 10Plucas: Use JAVA_HOME from environment if present [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/148227 [20:46:09] <hashar> Krinkle: yeah Bryan and I have a platonic relation ship with puppet :D [20:46:35] <hashar> so anyway, setting up slave with Trusty is going to be a pain in the a** [20:46:45] <hashar> I am not sure how to keep the jobs on precise instances [20:47:04] <Krinkle> hashar: Keep it outside jenkins pool? [20:47:06] <Krinkle> offline slave [20:47:16] <grrrit-wm> (03PS1) 10Jgreen: fix temp dir for ocg production role too [operations/puppet] - 10https://gerrit.wikimedia.org/r/148228 [20:47:32] <Krinkle> I never even got to the jenkins part because puppet never completed a single successful run [20:47:51] <Krinkle> I think you won't have to worry about how to keep the jenkins slave managed, getting the instance to run properly at all will be 80% of the work. [20:47:54] <hashar> Krinkle: yeah fixing puppet is the prerequisite [20:48:28] <hashar> Krinkle: then we are going to have different version of packages on Precise and Trusty, so we probably don't want the jobs to roam around both types [20:49:49] <grrrit-wm> (03CR) 10Jgreen: [C: 032 V: 031] fix temp dir for ocg production role too [operations/puppet] - 10https://gerrit.wikimedia.org/r/148228 (owner: 10Jgreen) [20:50:38] <Krinkle> hashar: indeed [20:50:46] <bd808> hashar, Krinkle: We have a lot more jobs than I've done this for before, but in a past life I would make matrix builds out of all jobs to run on current and planned OS for upgrades like this. [20:50:48] <grrrit-wm> (03PS1) 10Ori.livneh: apache: provision a dummy site so 'Include sites-enabled/*' always works [operations/puppet] - 10https://gerrit.wikimedia.org/r/148230 [20:50:49] <Krinkle> separate label, and apply to certain jobs [20:50:55] <hashar> Krinkle: we could apply a label on each slaves. Something like: ubuntu-precise [20:51:08] <Krinkle> gradually migrate all jobs [20:51:11] <Krinkle> Yep [20:51:14] <hashar> Krinkle: then tie all jobs to ubuntu-precise labels by editing the jjb defaults and all occurrences of nodes [20:51:27] <hashar> Krinkle: then slowly switch them to ubuntu-trusty label [20:51:51] <Krinkle> Could also re-use the existing labsSlave label and create a new labsSlaveTrusty. A bit uglier but allows not having to edit all existing jobs [20:52:03] <Krinkle> and makes sure you won't miss any [20:52:15] <hashar> bd808: we don't really need job to run on both on Ubuntu versions though. Just need to make sure they still work on Trusty [20:53:14] <Krinkle> Except for php jobs we may want to run both php53 and hhvm (we do already, no?) [20:53:31] <hashar> i don't mind updating all jobs. I can handle that [20:53:42] <hashar> for php jobs we only run Zend version [20:53:50] <hashar> need a wrapper that let us easily switch between both versions [20:53:56] <hashar> damn I need to fill a ton of bugs [20:56:46] <hashar> https://bugzilla.wikimedia.org/show_bug.cgi?id=68340 Jenkins: label slaves with their ubuntu version [20:57:35] <Krinkle> hashar: there used to be a regression job runing hhvm? [20:57:45] <Krinkle> I don't know if that's gone, but we can re-create it with proper hhvm on trusty [20:57:56] <hashar> it is gone i think [20:58:21] <hashar> Krinkle: can we get some hangout this week to talk about it ? [20:58:29] <Krinkle> hashar: Sure! [20:58:31] <hashar> Krinkle: and find a better time/day to have regular hangout? [20:58:53] <grrrit-wm> (03PS1) 10Ori.livneh: beta: load site configs from /usr/local/apache/conf/all.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/148231 [20:58:54] <Krinkle> You pick a time (anywhere between 15:00 and 00:00 your time) [20:59:04] <Krinkle> I'm going to be in London from tomorrow onwards until Aug 14 [20:59:33] <icinga-wm> RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [20:59:50] <grrrit-wm> (03CR) 10Ori.livneh: "Cherry-picked on beta" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148231 (owner: 10Ori.livneh) [21:01:11] <hashar> Krinkle: can you edit the existing CI checkin we have on monday at 5pm and move it to 3 or 4pm ? [21:01:15] <hashar> Krinkle: monday sounds fine [21:01:42] <icinga-wm> PROBLEM - puppet last run on ocg1003 is CRITICAL: CRITICAL: Puppet has 1 failures [21:01:53] <icinga-wm> PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: Puppet has 1 failures [21:02:02] <icinga-wm> RECOVERY - https.etherpad.wikimedia.org on zirconium is OK: HTTP OK: HTTP/1.1 200 OK - 28852 bytes in 0.032 second response time [21:03:17] <grrrit-wm> (03CR) 10Dzahn: "https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=etherpad" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148211 (owner: 10Matanya) [21:03:22] <grrrit-wm> (03CR) 10Dzahn: "https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=etherpad" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148164 (owner: 10Matanya) [21:04:20] <Krinkle> hashar: done, 15:30 AMS time [21:05:11] <icinga-wm> ACKNOWLEDGEMENT - NTP on ocg1001 is CRITICAL: NTP CRITICAL: Offset unknown daniel_zahn freshly setup (Jeff,Rob) [21:05:11] <icinga-wm> ACKNOWLEDGEMENT - NTP on ocg1002 is CRITICAL: NTP CRITICAL: Offset unknown daniel_zahn freshly setup (Jeff,Rob) [21:05:11] <icinga-wm> ACKNOWLEDGEMENT - puppet last run on ocg1002 is CRITICAL: CRITICAL: Puppet has 1 failures daniel_zahn freshly setup (Jeff,Rob) [21:05:11] <icinga-wm> ACKNOWLEDGEMENT - NTP on ocg1003 is CRITICAL: NTP CRITICAL: Offset unknown daniel_zahn freshly setup (Jeff,Rob) [21:05:11] <icinga-wm> ACKNOWLEDGEMENT - puppet last run on ocg1003 is CRITICAL: CRITICAL: Puppet has 1 failures daniel_zahn freshly setup (Jeff,Rob) [21:06:29] <hashar> Krinkle: wonderful thanks. If you can attend while in London it is not a big deal [21:07:17] <Krinkle> yeah [21:07:20] <hashar> Krinkle: or just ping me during day time and we can do the hangout when it is calm [21:07:37] <Krinkle> I'm there to work, and check out places to live in the future on some days, but before WIkimania I'll just be working from "home" [21:07:51] <Krinkle> but from there instead of here [21:07:53] <icinga-wm> PROBLEM - Host tantalum is DOWN: PING CRITICAL - Packet loss = 100% [21:09:05] <icinga-wm> ACKNOWLEDGEMENT - Host tantalum is DOWN: PING CRITICAL - Packet loss = 100% daniel_zahn RT-7942 [21:11:16] <James_F> bd808: So the 'Updating *_from_namespace fields in links tables.' step in updatedb.php is timing out… Fun. [21:11:38] <bd808> blerg. Is the job still running? [21:11:56] <James_F> bd808: It auto-kills after 30 minutes. [21:12:02] <James_F> bd808: So now two of them have failed. [21:12:12] <Krinkle> Is James_F running maintenance scripts on the cluster #omgomg [21:12:13] <Krinkle> :P [21:12:17] <James_F> bd808: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/label=deployment-bastion-eqiad,wikidb=commonswiki/2739/console and then https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/label=deployment-bastion-eqiad,wikidb=commonswiki/2740/console [21:12:30] <James_F> Krinkle: Beta Labs. Hush, you. :-) [21:12:39] <James_F> Krinkle: And technically cron is running them. :-) [21:12:42] <hashar> ohhh [21:12:52] * bd808 thinks someone should be running migrations like that manually [21:13:01] <hashar> I guess so [21:13:03] <Krinkle> James_F: No bad word from me. I'd encourage it. I was just wondering whether this was that moment. [21:13:07] <James_F> bd808: Any idea what might be causing that? [21:13:14] <James_F> Krinkle: Ha. No, not today. :-) [21:13:18] <hashar> unless it is good enough to recover and start again where it left [21:13:42] <icinga-wm> RECOVERY - puppet last run on ocg1003 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [21:14:04] <James_F> hashar: It isn't. [21:14:14] <hashar> yeah [21:14:19] <hashar> I guess running in a screen: sudo -u mwdeploy mwscript update.php --wiki=commonswiki --quick [21:14:21] <hashar> will do [21:14:41] <hashar> cawiki I have no clue [21:14:44] <James_F> Looks like https://gerrit.wikimedia.org/r/#/c/117373/ by Aaron is the culprit. [21:14:47] <hashar> it is supposed to be a small wiki [21:15:21] <James_F> hashar: It's running on the Beta Labs version, so it's the size of the Beta Labs wiki, not the real wiki – maybe it's been expanded? [21:16:02] <icinga-wm> RECOVERY - Host tantalum is UP: PING OK - Packet loss = 0%, RTA = 1.43 ms [21:16:10] <James_F> Maybe change the "Abort the build if it's stuck" flag for a bit? [21:16:16] <grrrit-wm> (03PS1) 10Andrew Bogott: Remove openstack::project-storage and related [operations/puppet] - 10https://gerrit.wikimedia.org/r/148234 [21:16:18] <grrrit-wm> (03PS1) 10Andrew Bogott: Remove unused gluster module [operations/puppet] - 10https://gerrit.wikimedia.org/r/148235 [21:16:20] <James_F> bd808, hashar: Or is that a bad idea? [21:16:26] <hashar> na just run it manually [21:16:49] <James_F> OK… kill the job for now/ [21:16:51] <James_F> ? [21:18:38] * bd808 starts `sudo -u mwdeploy mwscript update.php --wiki=commonswiki --quick` on deployment-bastion [21:18:44] <grrrit-wm> (03CR) 10Dzahn: [C: 031] Remove openstack::project-storage and related [operations/puppet] - 10https://gerrit.wikimedia.org/r/148234 (owner: 10Andrew Bogott) [21:19:14] <hashar> bd808: can you do the same for cawiki please? :-] [21:19:37] <bd808> hashar: yup [21:19:41] <hashar> my hero [21:19:53] <icinga-wm> RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [21:20:17] <hashar> bd808: ori: also memcached was broken on beta since nutcracker migration. Some conf was still pointing to the old twemproxy port and at least bastion had no nutcracker configuration [21:20:18] <grrrit-wm> (03CR) 10Andrew Bogott: [C: 032] Remove openstack::project-storage and related [operations/puppet] - 10https://gerrit.wikimedia.org/r/148234 (owner: 10Andrew Bogott) [21:20:27] <James_F> bd808: And enwiki? :-) [21:21:10] <grrrit-wm> (03CR) 10Andrew Bogott: [C: 032] Remove unused gluster module [operations/puppet] - 10https://gerrit.wikimedia.org/r/148235 (owner: 10Andrew Bogott) [21:21:35] <grrrit-wm> (03CR) 10Dzahn: [C: 032] generic_vhost (webserver) - update SSL ciphers [operations/puppet] - 10https://gerrit.wikimedia.org/r/147208 (https://bugzilla.wikimedia.org/53259) (owner: 10Dzahn) [21:21:37] <ori> hashar: i know; i fixed it [21:21:47] <grrrit-wm> (03PS3) 10Dzahn: generic_vhost (webserver) - update SSL ciphers [operations/puppet] - 10https://gerrit.wikimedia.org/r/147208 (https://bugzilla.wikimedia.org/53259) [21:22:38] <hashar> ori: I factored out nutcracker out of ::mediawiki to ::mediawiki::nutcracker so I can include it on the bastion host (which does not have ::mediawiki applied) https://gerrit.wikimedia.org/r/#/c/148041/ and https://gerrit.wikimedia.org/r/#/c/148042/ [21:22:52] <hashar> ori: both cherry picked on puppetmaster and I have added you as a reviewer :] [21:27:55] <hashar> !log beta: removed build timeout from beta-update-databases-eqiad Jenkins jobs. There is a huge schema change being processed by update.php [21:28:00] <morebots> Logged the message, Master [21:30:03] <hashar> James_F: ok being dealt by Jenkins now [21:30:12] <James_F> hashar: Excellent, thanks. [21:30:13] <hashar> James_F: someone suggested to drop the timeout [21:30:18] <James_F> hashar: I did. :-) [21:30:37] <hashar> James_F: so i just hacked the job manually and removed it. [21:30:49] <hashar> tricking bryan in manually run in screen was not a good idea [21:30:49] * James_F nods. [21:30:53] <hashar> :D [21:30:59] <James_F> Ah, OK. [21:32:21] <matanya> hashar: did you find a nicer way to manage +2 on puppet than layout.yaml ? [21:32:28] <mwalker> RobH, I'll put this in the ticket too; but we do actually want the OCG servers to have HT on -- the dequeue rate basically halves with it off [21:32:52] <hashar> matanya: what do you mean ? [21:33:10] <RobH> mwalker: ahh, then hrmm... i didnt check [21:33:15] <matanya> email filter is hacky, isn't it ? [21:33:15] <RobH> lets see if its on [21:33:20] <RobH> cuz i just finished with them, heh [21:33:49] <RobH> it is NOT on [21:33:50] <RobH> so i [21:33:53] <James_F> matanya: We could do an LDAP group instead but it'd be just as much a hack. [21:33:57] <RobH> so i'll do a rolling reboot and enable now [21:34:03] <RobH> while its a nonissue [21:34:07] <mwalker> awesome; thanks :) [21:34:50] <matanya> James_F: i see. ldap causes me a lot of pain atm, so please no :) [21:34:58] <James_F> matanya: :-D [21:35:49] <RobH> welcome [21:35:51] <matanya> unless someone here wants to accidentally add me to wmf group, that would ease my life a lot [21:35:57] <James_F> matanya: :-) [21:35:58] <marktraceur> Whoooops [21:36:13] <icinga-wm> PROBLEM - Host ocg1001 is DOWN: PING CRITICAL - Packet loss = 100% [21:36:14] <mutante> just wrap email_filter into more than one line [21:36:19] <mutante> that would alreayd make it so much better [21:36:22] <icinga-wm> PROBLEM - Host ocg1002 is DOWN: PING CRITICAL - Packet loss = 100% [21:36:31] <ottomata> grrr, jgage, we had a zookeeper timeout again today [21:36:42] <ottomata> org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 35887ms [21:36:42] <icinga-wm> PROBLEM - Host ocg1003 is DOWN: PING CRITICAL - Packet loss = 100% [21:36:48] <ottomata> an22 was removed from ISR for a bit [21:37:03] <ottomata> thougth we had gotten around that issue with the longer timeout [21:37:08] <ottomata> 35 seconds is way too long [21:37:24] <mutante> please ack those servers if working on them [21:37:35] <mutante> already did a couple times [21:37:37] <ottomata> !log running kafka preferred-replica-election to rebalance topics [21:37:43] <morebots> Logged the message, Master [21:37:51] <ottomata> mutante: it'llgo away in a second [21:37:56] <ottomata> but won't solve the source of the problem [21:38:17] <ottomata> mutante: if you want to do what I just did in the future [21:38:18] <ottomata> https://wikitech.wikimedia.org/wiki/Analytics/Kraken/Kafka/Administration#Replica_Elections [21:38:21] <mutante> ottomata: yea, thanks for fixing that, i meant the other checks [21:38:23] <ottomata> just log into a kafka broker and run [21:38:25] <ottomata> kafka preferred-replica-election [21:38:42] <RobH> mwalker: enabled, they are coming back up now [21:39:22] <icinga-wm> RECOVERY - Kafka Broker Messages In on analytics1022 is OK: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate OKAY: 2473.82440151 [21:39:22] <icinga-wm> RECOVERY - Host ocg1001 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [21:39:31] <mwalker> Jeff_Green, do you have time today to enable the ocg role on the new boxen; and change over the LVS entry to point to them? [21:39:52] <icinga-wm> RECOVERY - Host ocg1002 is UP: PING OK - Packet loss = 0%, RTA = 0.45 ms [21:40:52] <icinga-wm> RECOVERY - Host ocg1003 is UP: PING OK - Packet loss = 0%, RTA = 0.57 ms [21:44:01] <grrrit-wm> (03PS2) 10Tim Landscheidt: Tools: Deploy root web automatically [operations/puppet] - 10https://gerrit.wikimedia.org/r/148172 [21:47:28] <grrrit-wm> (03PS1) 10Plucas: Depend on any java7 jre/jdk [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/148240 [21:47:36] <grrrit-wm> (03PS2) 10Dzahn: generic_vhost - retab Apache template [operations/puppet] - 10https://gerrit.wikimedia.org/r/147209 [21:49:46] <grrrit-wm> (03CR) 10Dzahn: [C: 032] generic_vhost - retab Apache template [operations/puppet] - 10https://gerrit.wikimedia.org/r/147209 (owner: 10Dzahn) [21:50:24] <RobH> !log shutting down ms1002 for reclaim into labstore1003 [21:50:29] <morebots> Logged the message, RobH [21:51:43] <grrrit-wm> (03CR) 10Tim Landscheidt: "Tested on Toolsbeta." [operations/puppet] - 10https://gerrit.wikimedia.org/r/148172 (owner: 10Tim Landscheidt) [21:52:30] <grrrit-wm> (03PS4) 10Dzahn: generic_vhost - outdated variable syntax [operations/puppet] - 10https://gerrit.wikimedia.org/r/147210 [21:58:31] <grrrit-wm> (03CR) 10Dzahn: [C: 032] generic_vhost - outdated variable syntax [operations/puppet] - 10https://gerrit.wikimedia.org/r/147210 (owner: 10Dzahn) [22:01:09] <andrewbogott> Can someone with a native-linux laptop do a test for Matanya? [22:01:16] <andrewbogott> He's looking at alternative videoconf options [22:01:30] <matanya> windows works too [22:01:32] <mwalker> matanya, send me a link [22:01:47] <mwalker> we'll see if it works today :) [22:01:55] <matanya> mwalker: https://wiki.tox.im/UTox [22:04:09] <mwalker> matanya, here's my toxid: DBBD3C117674AFA47C9E63F2B245A4C9047CE4964563485D0C1434395FF3721E50990AB895A9 [22:04:33] <matanya> requested [22:08:17] <mwalker> greg-g, do you have a moment to download a binary and see if you can talk with matanya and I? [22:08:35] <greg-g> heh, uhhhh [22:09:05] <mwalker> it's totally not sketchy :p [22:11:45] <jgage> ottomata: hm, not analytics1021! [22:13:17] <greg-g> mwalker: I have "./utox" sitting on my terminal, scared to press enter [22:13:46] <mwalker> greg-g, I think we just proved that this isn't going to work [22:13:51] <mwalker> so your computer is safe [22:13:54] <greg-g> whew [22:14:00] * greg-g rm -rfs [22:14:01] <greg-g> :) [22:14:43] <greg-g> "Let's make a secure chat client to keep our conversations secret from the gobt, but, uh, signing releases and have a trust chain? nahhhh" [22:15:17] <mwalker> "it's in early stages" [22:15:25] <grrrit-wm> (03PS1) 10Ori.livneh: mediawiki: add a trusty check to apache envvars [operations/puppet] - 10https://gerrit.wikimedia.org/r/148248 [22:15:41] <mwalker> but it looks like it doesn't yet support group video chat [22:15:47] <matanya> mwalker: greg-g: quote from dev: matanya, group chat video isn't possible in core yet. [22:15:49] <greg-g> code for "we'll definitely pwn the l33t hackers then!" [22:15:55] <greg-g> ah [22:16:09] <matanya> i'm close to give up [22:16:30] <ottomata> yeah, jgage it was an22 [22:16:44] <matanya> I think i have tested above 20 solutions for video group chat [22:17:20] <matanya> all of them are from the almost category to the useless category [22:17:32] <ori> Reedy: yt? [22:18:19] <matanya> one last resort, will be tested tomorrow. wish me luck [22:18:26] <mwalker> I commend you for doing the work! [22:18:27] <mwalker> good luck! [22:18:37] <mwalker> ping me if you need a test person [22:18:59] <matanya> thanks mwalker and greg-g for suffering my failed tests! [22:19:23] <matanya> at least you don't go through those which fail to compile :) [22:20:04] <mwalker> I've got this totally fantastic thing! [22:20:07] <mwalker> which everyone should use! [22:20:14] <mwalker> and it totally works! [22:20:19] <mwalker> (if everyone helps debug it) [22:21:18] <mutante> !log installing package upgrades on bast1001 [22:21:24] <morebots> Logged the message, Master [22:24:38] <Reedy> ori: Ya [22:24:49] <ori> change I2cddc62d421952e478cbc02834adb816a66c4bad [22:24:51] <ori> it ends with "bad" [22:24:55] <ori> short for "bad regex" [22:25:10] <Reedy> hahah [22:25:32] <ori> Reedy: https://gerrit.wikimedia.org/r/148250 [22:26:09] <Reedy> Does that work? [22:26:25] <ori> we don't let that stop us from merging [22:26:31] <ori> hang on, i'm testing [22:26:32] <Reedy> Should it be an extra one added on the other end? [22:26:58] <Reedy> there's the | in the middle as it's zh and sr [22:27:16] <ori> Reedy: it works [22:27:30] <ori> staged on deployment-mediawiki02 [22:32:18] <grrrit-wm> (03CR) 10Gage: [C: 032] access: add new ezachte key [operations/puppet] - 10https://gerrit.wikimedia.org/r/148124 (owner: 10Matanya) [22:40:18] <grrrit-wm> (03PS2) 10Ori.livneh: beta: load site configs from /usr/local/apache/conf/all.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/148231 [22:54:25] <grrrit-wm> (03PS1) 10Ori.livneh: Revert various HHVM workarounds [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148260 [22:55:29] <grrrit-wm> (03CR) 10Ori.livneh: [C: 032] "beta-only" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148260 (owner: 10Ori.livneh) [22:55:35] <grrrit-wm> (03Merged) 10jenkins-bot: Revert various HHVM workarounds [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148260 (owner: 10Ori.livneh) [23:00:04] <jouncebot> The time is nigh to deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140721T2300) [23:02:26] <icinga-wm> RECOVERY - DPKG on db1017 is OK: All packages OK [23:02:36] <mwalker> interesting; wonder why it has stopped giving names... [23:02:51] <mwalker> ori, MaxSem; I'm a bit busy right now but I can do it if either of you cannot [23:02:53] <Reedy> I think it did after the last code update [23:03:13] <grrrit-wm> (03PS1) 10Ori.livneh: role::cache on beta: switch over appservers to HHVM [operations/puppet] - 10https://gerrit.wikimedia.org/r/148263 [23:03:18] <MaxSem> mwalker, I'll do it [23:03:21] <greg-g> yeah, we gave it manners, then it lost it's ability to remember names [23:03:21] <ori> MaxSem: thank you [23:04:04] <mwalker> greg-g, I know; and then I reverted the change; but it still has amnesia [23:04:11] * mwalker will have to debug more tonight [23:04:24] <greg-g> huh [23:07:08] <James_F> mwalker: Was the change deployed? [23:07:46] <greg-g> James_F: Max is on it [23:07:56] <greg-g> or are you referring to jouncebot ? [23:08:06] <James_F> greg-g: I meant jouncebot. :-) [23:08:13] <mwalker> James_F, ya; the text is back to "the time is nigh" instead of "the time has come" [23:08:40] <James_F> Cool. [23:27:20] <logmsgbot> !log maxsem Synchronized php-1.24wmf14/resources/: https://gerrit.wikimedia.org/r/#/c/147854/ (duration: 00m 05s) [23:27:26] <morebots> Logged the message, Master [23:27:38] <MaxSem> James_F, ^^ please verify [23:28:03] * James_F checks. [23:29:12] <logmsgbot> !log maxsem Synchronized php-1.24wmf14/extensions/AbuseFilter/: https://gerrit.wikimedia.org/r/#/c/148027/ (duration: 00m 06s) [23:29:17] <morebots> Logged the message, Master [23:30:17] <logmsgbot> !log maxsem Synchronized php-1.24wmf14/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/148128 (duration: 00m 06s) [23:30:22] <morebots> Logged the message, Master [23:30:58] <MaxSem> legoktm, ^^ [23:31:09] <legoktm> thanks /me tests [23:32:04] <James_F> MaxSem: All working great. Thanks! [23:32:25] <logmsgbot> !log maxsem Synchronized php-1.24wmf14/extensions/Flow/: https://gerrit.wikimedia.org/r/#/c/148249/ (duration: 00m 04s) [23:32:29] <morebots> Logged the message, Master [23:34:44] <grrrit-wm> (03CR) 10Dzahn: [C: 032] rt -- update cipher suite list to support PFS [operations/puppet] - 10https://gerrit.wikimedia.org/r/147715 (https://bugzilla.wikimedia.org/53259) (owner: 10Chmarkine) [23:35:34] <grrrit-wm> (03CR) 10Dzahn: [C: 031] nginx: qualify vars [operations/puppet] - 10https://gerrit.wikimedia.org/r/147902 (owner: 10Matanya) [23:36:52] <grrrit-wm> (03CR) 10Dzahn: [C: 031] puppetmaster: qualify var [operations/puppet] - 10https://gerrit.wikimedia.org/r/147900 (owner: 10Matanya) [23:36:58] <logmsgbot> !log maxsem Synchronized php-1.24wmf14/extensions/TimedMediaHandler/: https://gerrit.wikimedia.org/r/#/c/148241/ (duration: 00m 04s) [23:37:04] <morebots> Logged the message, Master [23:37:22] <grrrit-wm> (03PS2) 10Dzahn: rt -- update cipher suite list to support PFS [operations/puppet] - 10https://gerrit.wikimedia.org/r/147715 (https://bugzilla.wikimedia.org/53259) (owner: 10Chmarkine) [23:38:35] <legoktm> MaxSem: confirmed working, thanks! [23:38:36] <grrrit-wm> (03PS2) 10Dzahn: ishmael -- update cipher suite list to support PFS [operations/puppet] - 10https://gerrit.wikimedia.org/r/147740 (https://bugzilla.wikimedia.org/53259) (owner: 10Chmarkine) [23:39:47] <logmsgbot> !log maxsem Synchronized php-1.24wmf14/extensions/Flow/: https://gerrit.wikimedia.org/r/#/c/148270/ - revert previous change (duration: 00m 04s) [23:39:52] <morebots> Logged the message, Master [23:39:55] <bawolff> I was wondering, how come there's no entry in the wmflabs ganglia for deployment-db2? There's an entry for deployment-db1 [23:41:25] <JohnLewis> good question actually; I didn't notice that earlier [23:41:42] <MaxSem> ok, I think that's all [23:42:10] <legoktm> !log cleaned up stalled global rename of Felipegaspars --> L'editeur [23:42:14] <morebots> Logged the message, Master [23:42:35] <bawolff> JohnLewis: I'm wondering if there's something wrong with it's configuration, based on the slave-lag bug on beta (68349) [23:44:16] <mutante> !log graceful apache on magnesium [23:44:22] <morebots> Logged the message, Master [23:48:16] <grrrit-wm> (03CR) 10Dzahn: [C: 032] ishmael -- update cipher suite list to support PFS [operations/puppet] - 10https://gerrit.wikimedia.org/r/147740 (https://bugzilla.wikimedia.org/53259) (owner: 10Chmarkine) [23:48:36] <icinga-wm> PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There are 4 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [23:49:27] <icinga-wm> RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [23:50:08] <mutante> if that was just 1 pending change.. but 4 ? [23:51:51] <grrrit-wm> (03CR) 10Ori.livneh: "cherry-picked on beta" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148263 (owner: 10Ori.livneh) [23:59:28] <grrrit-wm> (03CR) 10Dzahn: "+0.5 - lgtm besides joining file resources into one - we have been doing the opposite a lot, also for style/lint reasons. +matanya" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148010 (owner: 10Ori.livneh)