[00:02:22] PROBLEM - MySQL Processlist on db1030 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 89 statistics [00:02:31] (03PS1) 10Ori.livneh: rcstream: customize location of error/access log [operations/puppet] - 10https://gerrit.wikimedia.org/r/145996 [00:02:33] (03PS1) 10Ori.livneh: rcstream: make lvs health check fetch /nginx_status [operations/puppet] - 10https://gerrit.wikimedia.org/r/145997 (https://bugzilla.wikimedia.org/67957) [00:03:22] RECOVERY - MySQL Processlist on db1030 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 0 statistics [00:04:16] bd808: how should rcstream log to logstash? [00:04:21] it's a python wsgi app [00:04:46] well… [00:05:03] "We" need to pick a stable input [00:05:22] what are "our" choices? [00:06:05] Right now we have udp2log in use in prod. [00:06:37] The redis input looked promising but we have not tested it [00:06:59] I have puppet sending to a tcp port in beta [00:07:09] rcstream already uses redis pubsub to get changes from mediawiki [00:07:14] so that would work especially well here [00:07:35] Oh cool. Maybe we can use that to figure out how to make it work then [00:07:52] (03CR) 10Ori.livneh: [C: 032] "trivial" [operations/puppet] - 10https://gerrit.wikimedia.org/r/145996 (owner: 10Ori.livneh) [00:07:57] Logstash needs badly to be updated too [00:08:38] We are running 1.2.2 and 1.4.2 is the latest upstream [00:08:50] RT? [00:09:06] Yeah. I need to test in beta first I think. [00:09:18] * ori nods. [00:09:19] And I need to fix a dumb thing I did in the puppet module to test in beta [00:09:28] I pinned the version :( [00:09:58] Prod needs elasticsearch and kibana updated too [00:10:08] Beta is running newer versions of both [00:10:22] Anyway, lets try to get the redis input working [00:10:42] We added it originally with the idea that mediawiki could use it too [00:11:49] The input is setup in beta, but I think there is something wrong with the redis server on logstash01 that it uses [00:12:22] I haven't bothered to debug because the input isn't in use, but I remember seeing startup errors from logstash about not being able to connect [00:18:51] (03PS1) 10Ori.livneh: rcstream: whitespace lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/145999 [00:19:11] (03CR) 10Ori.livneh: [C: 032 V: 032] "trivial" [operations/puppet] - 10https://gerrit.wikimedia.org/r/145999 (owner: 10Ori.livneh) [00:26:52] (03PS1) 10Tim Landscheidt: Tools: Add IP mapping for tools.wmflabs.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/146000 (https://bugzilla.wikimedia.org/54052) [00:28:20] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: Fetching origin [00:29:10] (03PS1) 10Ori.livneh: rcstream: add a 5s post-stop sleep delay [operations/puppet] - 10https://gerrit.wikimedia.org/r/146001 [01:03:24] RECOVERY - Unmerged changes on repository puppet on strontium is OK: Fetching origin [01:08:06] (03Abandoned) 10Ori.livneh: [HAT] Add configuration guards [operations/apache-config] - 10https://gerrit.wikimedia.org/r/141062 (owner: 10Ori.livneh) [02:13:58] !log LocalisationUpdate completed (1.24wmf12) at 2014-07-14 02:12:54+00:00 [02:14:06] Logged the message, Master [02:24:42] !log LocalisationUpdate completed (1.24wmf13) at 2014-07-14 02:23:39+00:00 [02:24:47] Logged the message, Master [02:57:28] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 14 02:56:22 UTC 2014 (duration 56m 21s) [02:57:34] Logged the message, Master [03:25:28] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Puppet has 1 failures [03:33:48] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Mon 14 Jul 2014 01:33:33 UTC [03:40:46] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Mon 14 Jul 2014 01:40:14 UTC [03:43:26] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [03:53:26] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Mon Jul 14 03:53:23 UTC 2014 [04:40:28] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Mon Jul 14 04:40:25 UTC 2014 [04:55:06] (03PS3) 10Ori.livneh: mediawiki: use apache module [operations/puppet] - 10https://gerrit.wikimedia.org/r/145620 [04:57:58] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: Puppet has 1 failures [04:57:58] PROBLEM - puppet last run on rdb1003 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:08] PROBLEM - puppet last run on nitrogen is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:09] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:09] PROBLEM - puppet last run on mw1108 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:18] PROBLEM - puppet last run on pc1003 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:28] PROBLEM - puppet last run on mw1105 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:28] PROBLEM - puppet last run on mw1122 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:28] PROBLEM - puppet last run on ssl1007 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:29] PROBLEM - puppet last run on mw1185 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:38] PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:38] PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:38] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:48] PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:48] PROBLEM - puppet last run on mw1121 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:48] PROBLEM - puppet last run on wtp1007 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:58] PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:58] PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:58] PROBLEM - puppet last run on wtp1003 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:58] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:58] PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:08] PROBLEM - puppet last run on db1064 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:08] PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:09] PROBLEM - puppet last run on mw1016 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:09] PROBLEM - puppet last run on search1013 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:09] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:09] PROBLEM - puppet last run on analytics1028 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:09] PROBLEM - puppet last run on mw1033 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:18] PROBLEM - puppet last run on search1011 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:18] PROBLEM - puppet last run on mw1093 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:19] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:19] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:19] PROBLEM - puppet last run on mw1010 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:19] PROBLEM - puppet last run on calcium is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:28] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:28] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:28] PROBLEM - puppet last run on mw1139 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:28] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:28] PROBLEM - puppet last run on ssl1006 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:29] PROBLEM - puppet last run on ssl1001 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:29] PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:38] PROBLEM - puppet last run on search1006 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:38] PROBLEM - puppet last run on search1022 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:38] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:38] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:58] PROBLEM - puppet last run on mercury is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:59] PROBLEM - puppet last run on copper is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:08] PROBLEM - puppet last run on db1033 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:09] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:18] PROBLEM - puppet last run on wtp1010 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:18] PROBLEM - puppet last run on cp1044 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:18] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:18] PROBLEM - puppet last run on analytics1018 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:18] PROBLEM - puppet last run on search1012 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:19] PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:19] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:20] PROBLEM - puppet last run on mw1071 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:20] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:21] PROBLEM - puppet last run on stat1001 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:28] PROBLEM - puppet last run on mw1066 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:38] PROBLEM - puppet last run on strontium is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:38] PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:48] PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:58] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:58] PROBLEM - puppet last run on mw1107 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:58] PROBLEM - puppet last run on search1004 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:58] PROBLEM - puppet last run on virt1008 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:58] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:59] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:59] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:00] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:00] PROBLEM - puppet last run on osm-db1002 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:01] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:08] PROBLEM - puppet last run on db60 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:08] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:08] PROBLEM - puppet last run on mw1204 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:09] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:09] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:09] PROBLEM - puppet last run on fenari is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:09] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:18] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:18] PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:19] PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:19] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:28] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:28] PROBLEM - puppet last run on ssl3001 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:28] PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:28] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:28] PROBLEM - puppet last run on bast1001 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:38] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:38] PROBLEM - puppet last run on mw1155 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:38] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:38] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:38] PROBLEM - puppet last run on osm-cp1001 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:48] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Puppet has 1 failures [05:01:59] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:08] PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:08] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:08] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:08] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:08] PROBLEM - puppet last run on ms-fe1002 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:09] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:09] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:10] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:10] PROBLEM - puppet last run on linne is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:11] PROBLEM - puppet last run on mexia is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:11] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:12] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:18] PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:18] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:18] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:18] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:18] PROBLEM - puppet last run on analytics1021 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:28] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:28] PROBLEM - puppet last run on es1004 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:28] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:28] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:28] PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:29] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:38] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:38] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:48] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:48] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:07] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:07] PROBLEM - puppet last run on mw1037 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:07] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:07] PROBLEM - puppet last run on ssl1003 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:07] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:07] RECOVERY - puppet last run on db1033 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [05:03:07] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:08] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:08] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:09] PROBLEM - puppet last run on db1005 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:09] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:10] PROBLEM - puppet last run on search1003 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:10] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:16] PROBLEM - puppet last run on zinc is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:16] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:16] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:17] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:17] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:17] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:17] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:18] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:18] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:19] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:26] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:26] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:36] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:36] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Puppet has 1 failures [05:03:49] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 1 failures [05:04:06] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: Puppet has 1 failures [05:04:07] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Puppet has 1 failures [05:04:07] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: Puppet has 1 failures [05:04:07] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Puppet has 1 failures [05:04:07] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: Puppet has 1 failures [05:04:07] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 1 failures [05:04:16] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: Puppet has 1 failures [05:04:16] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: Puppet has 1 failures [05:04:16] PROBLEM - puppet last run on virt0 is CRITICAL: CRITICAL: Puppet has 1 failures [05:04:26] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Puppet has 1 failures [05:05:06] PROBLEM - puppet last run on search1019 is CRITICAL: CRITICAL: Puppet has 1 failures [05:05:06] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Puppet has 1 failures [05:05:07] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: Puppet has 1 failures [05:05:16] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Puppet has 1 failures [05:05:16] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Puppet has 1 failures [05:05:16] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Puppet has 1 failures [05:05:17] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: Puppet has 1 failures [05:05:26] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Puppet has 1 failures [05:05:26] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Puppet has 1 failures [05:05:36] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: Puppet has 1 failures [05:05:46] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet has 1 failures [05:05:46] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: Puppet has 1 failures [05:12:06] RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [05:12:17] RECOVERY - puppet last run on pc1003 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [05:12:26] RECOVERY - puppet last run on ssl1007 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [05:12:36] RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [05:12:36] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [05:12:46] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [05:12:46] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [05:12:56] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [05:13:06] RECOVERY - puppet last run on nitrogen is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [05:13:06] RECOVERY - puppet last run on search1013 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [05:13:07] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [05:13:16] RECOVERY - puppet last run on rdb1003 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [05:13:16] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [05:13:16] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [05:13:26] RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [05:13:26] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [05:13:26] RECOVERY - puppet last run on mw1105 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [05:13:36] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [05:13:36] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [05:13:46] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [05:13:46] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [05:13:56] RECOVERY - puppet last run on mercury is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [05:13:56] RECOVERY - puppet last run on cp1052 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [05:14:07] RECOVERY - puppet last run on mw1016 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [05:14:07] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [05:14:07] RECOVERY - puppet last run on analytics1028 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [05:14:16] RECOVERY - puppet last run on mw1108 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [05:14:16] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [05:14:17] RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [05:14:17] RECOVERY - puppet last run on search1011 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [05:14:17] RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [05:14:17] RECOVERY - puppet last run on mw1093 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [05:14:17] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [05:14:18] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [05:14:18] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [05:14:26] RECOVERY - puppet last run on calcium is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [05:14:26] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [05:14:26] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [05:14:26] RECOVERY - puppet last run on mw1139 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [05:14:26] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [05:14:27] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [05:14:27] RECOVERY - puppet last run on ssl1006 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [05:14:28] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [05:14:28] RECOVERY - puppet last run on ssl1001 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [05:14:36] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [05:14:36] RECOVERY - puppet last run on ms-be1011 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [05:14:36] RECOVERY - puppet last run on search1022 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [05:14:36] RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [05:14:36] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [05:14:46] RECOVERY - puppet last run on search1004 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [05:14:56] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [05:15:06] RECOVERY - puppet last run on copper is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [05:15:06] RECOVERY - puppet last run on db60 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [05:15:06] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [05:15:16] RECOVERY - puppet last run on cp1044 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [05:15:16] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [05:15:16] RECOVERY - puppet last run on search1012 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [05:15:17] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [05:15:26] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [05:15:36] RECOVERY - puppet last run on strontium is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [05:15:36] RECOVERY - puppet last run on search1006 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [05:15:36] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [05:15:36] RECOVERY - puppet last run on mw1107 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [05:15:37] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [05:15:46] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [05:15:46] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [05:15:56] RECOVERY - puppet last run on osm-db1002 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [05:16:06] RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [05:16:06] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [05:16:07] RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [05:16:07] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [05:16:07] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [05:16:07] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [05:16:07] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [05:16:08] RECOVERY - puppet last run on linne is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [05:16:08] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [05:16:16] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [05:16:16] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [05:16:16] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [05:16:17] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [05:16:17] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [05:16:17] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [05:16:17] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [05:16:18] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [05:16:18] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [05:16:19] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [05:16:19] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [05:16:20] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [05:16:26] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [05:16:26] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [05:16:26] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [05:16:26] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [05:16:26] RECOVERY - puppet last run on ssl3001 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [05:16:27] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [05:16:27] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [05:16:28] RECOVERY - puppet last run on cp1053 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [05:16:28] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [05:16:36] RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [05:16:36] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [05:16:36] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [05:16:36] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [05:16:46] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [05:16:46] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [05:16:56] RECOVERY - puppet last run on mw1154 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [05:16:56] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [05:17:06] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [05:17:06] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [05:17:06] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [05:17:06] RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [05:17:07] RECOVERY - puppet last run on mexia is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [05:17:16] RECOVERY - puppet last run on zinc is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [05:17:16] RECOVERY - puppet last run on ssl1003 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [05:17:16] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [05:17:16] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [05:17:16] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [05:17:17] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [05:17:17] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [05:17:18] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [05:17:18] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [05:17:19] RECOVERY - puppet last run on analytics1021 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [05:17:19] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [05:17:26] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [05:17:26] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [05:17:26] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [05:17:36] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [05:17:36] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [05:17:36] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [05:17:36] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [05:17:36] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [05:17:37] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [05:17:37] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [05:17:38] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [05:17:46] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [05:17:46] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [05:17:46] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [05:18:06] RECOVERY - puppet last run on search1019 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [05:18:06] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [05:18:06] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [05:18:06] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [05:18:06] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [05:18:07] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [05:18:07] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [05:18:08] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [05:18:08] RECOVERY - puppet last run on search1003 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [05:18:09] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [05:18:09] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [05:18:10] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [05:18:10] RECOVERY - puppet last run on fenari is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [05:18:11] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [05:18:16] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [05:18:16] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [05:18:17] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [05:18:17] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [05:18:17] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [05:18:17] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [05:18:26] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [05:18:26] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [05:18:26] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [05:18:26] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [05:18:36] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [05:18:36] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [05:18:46] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [05:19:16] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [05:19:16] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [05:19:16] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [05:19:16] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [05:19:16] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [05:19:16] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [05:19:45] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 74 seconds ago with 0 failures [05:19:45] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [05:20:16] RECOVERY - puppet last run on virt0 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [05:54:31] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Mon 14 Jul 2014 03:53:23 UTC [06:28:21] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 6 below the confidence bounds [06:28:50] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:10] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:30] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:50] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:00] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:21] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:21] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:30] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 2 failures [06:44:34] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:34] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:45:44] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:45:44] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:46:04] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:46:04] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:46:24] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:46:24] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [06:53:34] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:54:34] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: Puppet has 1 failures [07:00:40] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [07:11:40] RECOVERY - puppet last run on hooft is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [07:12:15] (03PS1) 10Legoktm: Remove unused wmgUseMarkAsHelpful from InitialiseSettings.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146031 [07:19:27] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [07:53:45] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Mon Jul 14 07:53:36 UTC 2014 [08:01:06] (03CR) 10Adamw: [C: 031] "Beta testing is complete." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145703 (owner: 10Awight) [08:03:06] (03CR) 10Awight: Enable FundraisingTranslateWorkflow on metawiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145703 (owner: 10Awight) [08:33:58] !log depool ms-fe1001 for swift icehouse upgrade [08:34:04] Logged the message, Master [08:44:12] (03PS1) 10Filippo Giunchedi: add swift icehouse repo to ms-fe* [operations/puppet] - 10https://gerrit.wikimedia.org/r/146038 [08:44:44] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] add swift icehouse repo to ms-fe* [operations/puppet] - 10https://gerrit.wikimedia.org/r/146038 (owner: 10Filippo Giunchedi) [08:47:52] (03PS7) 10Giuseppe Lavagetto: Add and apply twemproxy::decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/144350 (owner: 10Ori.livneh) [08:48:02] (03CR) 10Giuseppe Lavagetto: [C: 032] Add and apply twemproxy::decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/144350 (owner: 10Ori.livneh) [08:55:07] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: Puppet has 1 failures [08:55:17] PROBLEM - puppet last run on mw1167 is CRITICAL: CRITICAL: Puppet has 1 failures [08:55:38] PROBLEM - puppet last run on mw1105 is CRITICAL: CRITICAL: Puppet has 1 failures [08:55:38] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: Puppet has 1 failures [08:55:57] PROBLEM - puppet last run on mw1033 is CRITICAL: CRITICAL: Puppet has 1 failures [08:55:57] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: Puppet has 1 failures [08:55:57] PROBLEM - puppet last run on mw1122 is CRITICAL: CRITICAL: Puppet has 1 failures [08:55:57] PROBLEM - puppet last run on mw1077 is CRITICAL: CRITICAL: Puppet has 1 failures [08:56:07] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: Puppet has 1 failures [08:57:05] <_joe_> mmmh [08:59:07] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [08:59:12] <_joe_> that is nice, I managed to merge a change that was removing a file while puppet was already running on those servers [08:59:31] <_joe_> the file was not included in the new catalog, but it was in the old one [08:59:43] <_joe_> causing puppet to temporarily fail on said machines [09:02:29] !log repool ms-fe1001 after upgrade, basic testing successful [09:02:35] Logged the message, Master [09:07:19] (03PS2) 10Giuseppe Lavagetto: apache: on service refresh, do a graceful reload instead of start/stop [operations/puppet] - 10https://gerrit.wikimedia.org/r/145908 (owner: 10Ori.livneh) [09:12:52] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [09:12:52] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [09:13:02] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [09:13:42] RECOVERY - puppet last run on mw1105 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [09:13:42] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [09:14:02] RECOVERY - puppet last run on mw1077 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [09:14:12] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [09:14:13] RECOVERY - puppet last run on mw1167 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [09:21:33] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Mon 14 Jul 2014 07:20:40 UTC [09:32:13] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: Puppet has 1 failures [09:33:43] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: Puppet has 1 failures [09:34:13] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: Puppet has 1 failures [09:35:17] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "Sorry but I don't like this approach. There are a few things apache _won't_ change upon an apache reload and that do need a restart instea" [operations/puppet] - 10https://gerrit.wikimedia.org/r/145908 (owner: 10Ori.livneh) [09:35:58] !log ran initSiteStats.php on all wikivoyages for bug 64370 [09:36:04] Logged the message, Master [09:55:27] _joe_: at what level is the duplication happening here? http://paste.debian.net/109609/ [09:55:49] I used to think it was tiscali (they also have weird delays in delivery sometimes), but look the first two excerpts [09:58:43] <_joe_> Nemo_bis: no idea :) [09:59:05] Nemo_bis: I'm not sure what you mean? [09:59:33] <_joe_> godog: he's receiving emails from bugzilla twice, probably once for a DL and once directly [10:00:20] Those are excerpts of two emails I received at 1 minute distance. They're the same in everything except the two top "Received" headers [10:00:37] <_joe_> which were usually de-duplicated internally which is not the case anymore for some reason [10:01:01] <_joe_> that is my best guess just looking at it [10:01:25] <_joe_> but I have my head deep in ruby code right now [10:02:06] <_joe_> I can't believe puppet does not set the 'hasrestart' flag to true by default when you choose provider => debian in a service type definition [10:02:15] Nemo_bis: can you paste full headers for both? [10:04:46] godog: http://paste.debian.net/hidden/20d1edd6/ [10:05:47] _joe_: btw, unsure if I told you, but submitted pull requests for diamond to do configurable whitelists/blacklist for metrics to collect on a per collector level. Now to wait for it to be merged... [10:06:10] <_joe_> YuviPanda: hi :) [10:06:19] <_joe_> YuviPanda: I am in the reviewers list? [10:06:31] <_joe_> I was really AFK for one weekend [10:06:49] <_joe_> and that was healthy but now I have a ton of catch up to do [10:06:53] _joe_: no, it is to *diamond* itself https://github.com/BrightcoveOS/Diamond/pulls so... [10:07:04] <_joe_> oh GOOD [10:07:12] yeah, upstream all the things! [10:07:56] <_joe_> well we do have some local fixes... [10:08:00] oh? [10:08:08] <_joe_> we should send them upstream as well [10:08:08] bah :( [10:08:10] yeah [10:08:22] where's the repo at? I could extract and send 'em upstream [10:08:34] <_joe_> YuviPanda: I think fixes chasemp did, but maybe upstream wasn't unrsesponsive? [10:08:41] <_joe_> we should speak with him [10:08:58] _joe_: yeah, my patches have been there for 4 days or so now, no response from upstream [10:09:00] _joe_: will do! [10:09:41] <_joe_> YuviPanda: we may maintain a github repo with wikimedia's patches and sync it with upstream [10:09:50] <_joe_> I hate gh, btw [10:10:02] <_joe_> on ethical grounds, technically it's amaizing [10:10:09] FOSS reasons or the 'pull request model' reason? [10:10:10] aah [10:12:51] <_joe_> FOSS [10:13:00] <_joe_> the bitkeeper exception [10:13:26] right [10:13:36] <_joe_> I don't like developing free software on commercial, closed services [10:13:41] <_joe_> like SF or GH [10:13:41] history repeating itself, etc. [10:14:17] right. the popcorntime DMCA ones are perhaps the start [10:15:16] <_joe_> eheh [10:15:33] <_joe_> popcorntime, I downloaded it out of rage against GH [10:15:37] <_joe_> it's a neat system [10:15:42] :D [10:15:48] yeah. I wonder where they'll move to [10:16:06] self hosted code review systems still have some way to go. I've to check where gitlab is at. [10:16:21] Gerrit isn't the most userfriendly or hacking-friendly system,. [10:22:55] <_joe_> YuviPanda: they should find someone willing to host controversial projects without fear of DMCA (thus, non-US), and someone doing that on a FLOSS platform IMO [10:23:05] +1 [10:23:25] Pirate gitlab/phab? [10:24:32] <_joe_> YuviPanda: we did have some free code repositories around the world, but they have basically died because devs want new and shiny and not free and auto-organized [10:25:06] <_joe_> YuviPanda: that's the same reason why I bought a mac a few years ago, so I'm not innocent either [10:25:54] _joe_: same here, yeah. [10:26:15] I'm also young enough to have worked on only one big project before the coming of git [11:00:23] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Mon Jul 14 11:00:19 UTC 2014 [11:29:28] (03PS1) 10Yuvipanda: tools: Actually branch on distro name for packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/146050 [11:29:46] scfc_de: ^ [11:33:37] (03CR) 10Tim Landscheidt: [C: 031] tools: Actually branch on distro name for packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/146050 (owner: 10Yuvipanda) [11:34:21] scfc_de: I still don't know where adminbot comes from, and there seems to be no apparent gdal replacement [11:35:01] I don't even know why adminbot has to be a package, but I suppose that's because it runs from prod as well [11:36:05] IIRC that's from apt.wikimedia.org. [11:36:52] Now how can I make aptitude say which repo a package is from? [11:37:47] "aptitude show -v adminbot" works: "Archiv: precise-wikimedia, now". So someone needs to build the adminbot for Trusty and upload it to trusty-wikimedia (or whatever that's called). [11:38:04] YuviPanda: Could you file an RT for that? [11:38:07] right [11:38:10] yeah, will do [11:38:37] the vips package is still fucked, of course :) [11:40:11] scfc_de: I also wonder how labsdebrepo will work with precise and trusty mixed hosts [11:40:49] YuviPanda: huh? how can one machine have two distributions installed? [11:41:00] Trminator: as in, a project with both hosts I mean [11:41:02] (this is labs) [11:43:39] YuviPanda: You probably have already tested that, as for example python-flask was installed from there. But there are certainly dragons out there ... :-) [11:43:47] *on tools-exec-12 [11:43:51] ah [11:43:52] yeah [11:43:53] true [11:58:39] (03CR) 10Prtksxna: "These changes aren't specific to a user's preference. They strip out parts that aren't exactly part of the article's text." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/126226 (https://bugzilla.wikimedia.org/63164) (owner: 10Prtksxna) [12:53:18] (03PS2) 10Yuvipanda: tools: Actually branch on distro name for packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/146050 [13:01:26] (03CR) 10Filippo Giunchedi: [C: 031] mediawiki: move SSHD nice override from web.pp to init.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/145510 (owner: 10Ori.livneh) [13:02:08] ori: let me know when you are around, I'm happy to deploy https://gerrit.wikimedia.org/r/#/c/145510/ and https://gerrit.wikimedia.org/r/#/c/145620 [13:20:11] (03PS3) 10Giuseppe Lavagetto: apache: on service refresh, do a graceful reload instead of start/stop [operations/puppet] - 10https://gerrit.wikimedia.org/r/145908 (owner: 10Ori.livneh) [13:21:10] <_joe_> godog: on 145620... please let me check it [13:21:23] <_joe_> I think we can bring it online this evening (our time [13:21:34] <_joe_> when I perform the jobrunner change with ori [13:21:39] <_joe_> or maybe earlier [13:21:49] _joe_: yep no problem [13:22:44] <_joe_> godog: including ::apache should be harmless on mediawikis, where apache.conf is completely custom [13:23:22] (03CR) 10Giuseppe Lavagetto: [C: 031] mediawiki: use apache module [operations/puppet] - 10https://gerrit.wikimedia.org/r/145620 (owner: 10Ori.livneh) [13:24:55] (03CR) 10Dzahn: "historically, there has been the "apache sanity check" that was called via apache-graceful-all script, has that script been replaced alrea" [operations/puppet] - 10https://gerrit.wikimedia.org/r/145908 (owner: 10Ori.livneh) [13:25:05] (03CR) 10Giuseppe Lavagetto: "In the end, I looked at all our code and the only relevant point where your change could've meant harm is in the apache2::def - where I ma" [operations/puppet] - 10https://gerrit.wikimedia.org/r/145908 (owner: 10Ori.livneh) [13:25:26] <_joe_> mutante: the sanity check gets done by the debian init scripts [13:25:34] <_joe_> both for reload and for restart [13:25:44] (03CR) 10Dzahn: "..because that did exactly this, check config before restarting.." [operations/puppet] - 10https://gerrit.wikimedia.org/r/145908 (owner: 10Ori.livneh) [13:26:20] _joe_: did we replace the scripts already? [13:26:27] <_joe_> mutante: apache2ctl -t [13:26:34] <_joe_> mutante: I don't think so [13:26:38] no, the sanity check was in the appserver package? [13:26:57] looks [13:27:09] <_joe_> mutante: no idea, but it's useless [13:27:10] what are we refactoring? [13:27:19] useless? [13:27:34] <_joe_> mutante: for what we're doing with this patch, yes [13:27:35] <_joe_> :) [13:28:04] <_joe_> mutante: this patch is against puppet, not the apache init scripts. It's puppet that is doing everything wrong [13:28:36] <_joe_> when we notify service 'apache2', puppet executes by default 'invoke-rc.d stop; invoke-rc.d start) [13:28:50] <_joe_> s/\)/\'/ [13:29:10] since when did puppet restart apaches on cluster? [13:29:28] <_joe_> mutante: it does in every place but mediawiki [13:29:33] <_joe_> that will change soon btw [13:29:34] <_joe_> :) [13:29:48] <_joe_> s/mediawiki/appservers/ [13:30:03] ... that's why i ask what we are refactoring [13:30:13] and if the goal is to change it all [13:30:15] seems it is [13:31:02] <_joe_> it is - there is an initial change (WiP) by me that shows how it may become if you're interested... https://gerrit.wikimedia.org/r/#/c/144453/ [13:31:31] <_joe_> it will be redone basically from scratch soon btw [13:33:27] modules/mediawiki/files/apache ? ehm,.. ...o ...k... [13:33:43] didnt think it was part of mediawiki [13:33:55] <_joe_> ? [13:34:52] <_joe_> what is part of mediawiki, sorry [13:34:58] apache-config used to be a separate repo, separate from mediawiki-config [13:35:09] apache-config was mostly done by ops, mw by platform [13:35:10] <_joe_> mutante: I wrote a mail about it last week :) [13:35:23] ok [13:35:25] <_joe_> apache-config is still a repo [13:35:40] <_joe_> it is imported in puppet via a submodule at the moment [13:35:49] <_joe_> something I want to change quickly :) [13:36:05] it wasnt imported in puppet at all.. [13:36:09] like .. last week ? [13:36:19] <_joe_> yes this change is last week's [13:36:21] <_joe_> :) [13:36:50] should this have a warning that it's oudated ? https://wikitech.wikimedia.org/wiki/Apache#Deploying_config [13:37:31] <_joe_> yes, this is supposed to be my homework for today to change that :) [13:37:44] <_joe_> I wanted to get rid of the submodule and update that [13:38:00] <_joe_> I thought that a mail to ops@ while we're in transition was enough [13:41:10] hey, two people that can merge! /me pokes mutante and _joe_ [13:41:58] * Nemo_bis merges a delicious plum to matanya's face [13:42:56] _joe_: alright, i had not read that part yet " I'm going to make puppet manage the reload of apache upon a [13:43:00] config change" etc [13:43:16] <_joe_> mutante: :) [13:44:55] today's riddle: there's clearly a drop in requests at the top of the hour, legit? https://graphite.wikimedia.org/render/?title=Wiki%20Pageviews/sec%20-8hours&from=-8hours&width=1024&height=500&until=now&areaMode=none&hideLegend=false&lineWidth=1&lineMode=connected&target=color(cactiStyle(alias(scale(reqstats.pageviews,%220.01666%22),%20%22pageviews/sec%22)),%22blue%22) [13:45:56] (or not, it seems like it) [13:47:24] godog: i just see "no data" ? [13:50:34] gah [13:50:48] mh no I can see that link [13:51:12] mutante: wrong hilight/"open url" perhaps? [13:51:35] yes, i think so [13:52:37] <_joe_> godog: that's most probably an artifact, either of some bot/whatever, or of our collection system [13:53:27] _joe_: ye could be [13:56:39] godog: is swift work done? [13:58:03] PROBLEM - DPKG on stat1002 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:58:46] matanya: "ish", I'm finishing the upgrades this week (on wed) :)) [13:59:30] godog: so i'll schedule lint merge for monday, if that works for you [14:00:00] matanya: yep sounds good, might have to rebase I have added a line this morning [14:00:07] i saw [14:02:53] !log stat1002 - "Could not find declared class ::oozie" [14:02:58] Logged the message, Master [14:04:15] i'm on stat1002 righ tnow [14:04:17] agr [14:04:25] yeah, cdh modules are weird right now, especially there righ tnow [14:04:28] sorry mutante, fixing now [14:04:46] ottomata: heh, alright, just saying because of icinga-wm :) [14:05:03] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Complete puppet failure [14:05:06] removing submodules? [14:10:03] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [14:11:03] RECOVERY - DPKG on stat1002 is OK: All packages OK [14:12:58] (03PS1) 10Ottomata: Don't include role::analytics::clients on stat1002 while CDH5 upgrade is in progress [operations/puppet] - 10https://gerrit.wikimedia.org/r/146065 [14:13:12] (03CR) 10Ottomata: [C: 032 V: 032] Don't include role::analytics::clients on stat1002 while CDH5 upgrade is in progress [operations/puppet] - 10https://gerrit.wikimedia.org/r/146065 (owner: 10Ottomata) [14:17:32] (03PS4) 10Giuseppe Lavagetto: remove SSL cipher DHE-RSA-AES128-GCM-SHA256 [operations/puppet] - 10https://gerrit.wikimedia.org/r/145688 (owner: 10Dzahn) [14:18:03] (03PS5) 10Giuseppe Lavagetto: remove SSL cipher DHE-RSA-AES128-GCM-SHA256 [operations/puppet] - 10https://gerrit.wikimedia.org/r/145688 (owner: 10Dzahn) [14:18:10] (03PS6) 10Giuseppe Lavagetto: remove SSL cipher DHE-RSA-AES128-GCM-SHA256 [operations/puppet] - 10https://gerrit.wikimedia.org/r/145688 (owner: 10Dzahn) [14:18:40] (03CR) 10Giuseppe Lavagetto: [C: 032] remove SSL cipher DHE-RSA-AES128-GCM-SHA256 [operations/puppet] - 10https://gerrit.wikimedia.org/r/145688 (owner: 10Dzahn) [14:23:10] (03CR) 10Rush: "phabricator will be on trusty. even trusty only can work out for us." [operations/debs/php-mailparse] (review) - 10https://gerrit.wikimedia.org/r/142751 (owner: 1020after4) [14:23:22] (03CR) 10Manybubbles: [C: 031] Increase weighting of title field for Commons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145588 (owner: 10Chad) [14:23:30] (03CR) 10Dzahn: [C: 032] platform: simplify hardware specific configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/144033 (owner: 10Matanya) [14:23:32] (03PS4) 10Giuseppe Lavagetto: apache: on service refresh, do a graceful reload instead of start/stop [operations/puppet] - 10https://gerrit.wikimedia.org/r/145908 (owner: 10Ori.livneh) [14:24:16] _joe_: thanks [14:25:49] <_joe_> mutante: btw, we don't really need to reload nginx for this patch [14:25:58] <_joe_> I may do that on one frontend only if really needed [14:26:02] Base::Platform/Generic::Upstart_job[undef]/File[/etc/init.d/]: Not removing directory; use 'force' to override [14:26:06] hrmrmm [14:26:12] no :) [14:26:18] not removing init.d [14:26:23] <_joe_> who has done that? [14:26:26] me [14:26:28] <_joe_> Upstart_job[undef]/ [14:26:44] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 1 failures [14:26:53] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:03] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:03] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:03] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:09] :/ [14:27:13] <_joe_> wtf? [14:27:15] but why [14:27:23] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:28] _joe_: https://gerrit.wikimedia.org/r/144033 [14:27:33] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:33] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:33] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:33] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:33] PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:40] (03PS1) 10Dzahn: Revert "platform: simplify hardware specific configuration" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146068 [14:27:43] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:43] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:44] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:44] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:44] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:44] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:44] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:45] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:53] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:53] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:53] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:53] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:54] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:54] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:54] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:55] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:55] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 1 failures [14:27:56] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:03] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:03] <_joe_> Notice: /Stage[main]/Base::Platform/File[/etc/init/.conf]/ensure: defined content as '{md5}e42e0d6e0c19a49127d4ec21931eff58' [14:28:03] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:03] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:04] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:04] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:08] <_joe_> oh shit [14:28:10] _joe_: it even had a puppet compiler link.. [14:28:13] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:13] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:13] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:15] where it was tested [14:28:23] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:23] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:23] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:33] PROBLEM - puppet last run on search1007 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:43] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:43] PROBLEM - puppet last run on holmium is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:43] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:44] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:44] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:44] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:44] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:45] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:45] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:46] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:46] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:49] <_joe_> REVERTING [14:28:53] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:53] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:54] PROBLEM - Host nescio is DOWN: PING CRITICAL - Packet loss = 100% [14:28:54] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:54] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: Puppet has 1 failures [14:28:57] (03PS1) 10Giuseppe Lavagetto: Revert "platform: simplify hardware specific configuration" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146069 [14:29:01] (03CR) 10Dzahn: [C: 032] Revert "platform: simplify hardware specific configuration" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146068 (owner: 10Dzahn) [14:29:03] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:03] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:09] _joe_: already reverted [14:29:13] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:16] <_joe_> ok [14:29:23] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:23] PROBLEM - puppet last run on analytics1038 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:23] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:23] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:23] PROBLEM - puppet last run on labstore1001 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:33] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:33] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:33] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:33] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:33] PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:34] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:36] now i need to figure out why it broke it all [14:29:39] I love the smell of icinga-spam in the morning [14:29:43] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:43] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:43] PROBLEM - Host 91.198.174.6 is DOWN: PING CRITICAL - Packet loss = 100% [14:29:44] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:44] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:44] PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:44] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:45] PROBLEM - puppet last run on silver is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:45] PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:46] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:46] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:47] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:47] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:48] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:53] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:53] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:54] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:54] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:54] PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:54] PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:54] PROBLEM - puppet last run on polonium is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:55] PROBLEM - puppet last run on antimony is CRITICAL: CRITICAL: Puppet has 1 failures [14:29:58] <_joe_> matanya: http://puppet-compiler.wmflabs.org/128/change/144033/html/holmium.wikimedia.org.html [14:30:01] <_joe_> for example [14:30:03] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: Puppet has 1 failures [14:30:03] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: Puppet has 1 failures [14:30:03] PROBLEM - puppet last run on mw1044 is CRITICAL: CRITICAL: Puppet has 1 failures [14:30:05] <_joe_> your change was there [14:30:13] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: Puppet has 1 failures [14:30:13] PROBLEM - puppet last run on analytics1016 is CRITICAL: CRITICAL: Puppet has 1 failures [14:30:23] PROBLEM - Host sodium is DOWN: CRITICAL - Host Unreachable (208.80.154.61) [14:30:40] stopped icinga-wm temp [14:30:43] will get it back asap [14:31:10] <_joe_> Host sodium is DOWN? [14:31:55] <_joe_> hey mutante nescio and sodium are down [14:32:05] (03PS1) 10Rush: removing admins.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/146070 [14:32:30] going to sodium.mgmt [14:32:44] confirmed sodium is unreachable for me via ssh [14:32:48] oh yes kill admin.pp ! [14:32:48] <_joe_> mutante: looks like some power issue honestly [14:32:54] mgmt output = �������� [14:33:01] seems unrelated, but nice timing?? [14:33:04] this is my change [14:33:14] i fear [14:33:24] <_joe_> seriously guys? [14:33:35] !log powercycling sodium [14:33:41] Logged the message, Master [14:33:42] <_joe_> http://puppet-compiler.wmflabs.org/128/change/144033/html/sodium.wikimedia.org.html I don't think so [14:35:38] sodium is back [14:36:13] not a great start for the week, is it? [14:36:53] (03CR) 10coren: [C: 032] tools: Actually branch on distro name for packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/146050 (owner: 10Yuvipanda) [14:39:50] <_joe_> !log rebooted nescio, stuck and with console showing just a truncated log (timestamp only) [14:39:54] PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: Puppet has 1 failures [14:39:55] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: Puppet has 1 failures [14:39:55] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: Puppet has 1 failures [14:39:55] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: Puppet has 1 failures [14:39:55] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: Puppet has 1 failures [14:39:56] Logged the message, Master [14:40:05] Coren: merge https://gerrit.wikimedia.org/r/#/c/146000/ as well? :) [14:40:34] PROBLEM - puppet last run on amssq42 is CRITICAL: CRITICAL: Puppet has 1 failures [14:40:44] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Puppet has 1 failures [14:40:55] PROBLEM - puppet last run on amssq41 is CRITICAL: CRITICAL: Puppet has 1 failures [14:42:14] RECOVERY - Host nescio is UP: PING OK - Packet loss = 0%, RTA = 96.35 ms [14:43:04] RECOVERY - Host 91.198.174.6 is UP: PING OK - Packet loss = 0%, RTA = 95.55 ms [14:43:55] PROBLEM - puppet last run on ssl3002 is CRITICAL: CRITICAL: Puppet has 1 failures [14:44:22] awight: Note I removed your SWAT entry for this morning. Deployment of new extensions is not part of SWAT's remit. You'll want to talk to greg-g to schedule a separate window. [14:44:34] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [14:44:34] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:44:44] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [14:45:04] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:45:04] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [14:45:24] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [14:45:34] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [14:45:34] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [14:45:34] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [14:45:42] waits for that to settle [14:46:09] anomie: aha, no worries. Thanks for the note! [14:56:03] ottomata: the analytics servers in d2 are okay for install now [14:56:42] brought icinga-wm back [14:56:44] ok tahnks [14:57:13] vlan wasn't set correctly [14:57:32] ah ok [14:57:32] (03PS1) 10Mark Bergsma: Remove DNS recursor on sodium [operations/puppet] - 10https://gerrit.wikimedia.org/r/146072 [15:00:05] manybubbles, anomie, ^d: The time is nigh to deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140714T1500) [15:00:34] (03CR) 10Manybubbles: [C: 032] Disable incoming link counts on commons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145430 (owner: 10Chad) [15:00:37] (03CR) 10Manybubbles: [C: 032] Increase weighting of title field for Commons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145588 (owner: 10Chad) [15:00:46] (03Merged) 10jenkins-bot: Disable incoming link counts on commons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145430 (owner: 10Chad) [15:00:48] (03Merged) 10jenkins-bot: Increase weighting of title field for Commons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145588 (owner: 10Chad) [15:01:52] <^demon|away> manybubbles: I'll do the deploy in 2hrs then :) [15:01:58] !log manybubbles Synchronized wmf-config: SWAT update cirrus settings for commons (duration: 00m 05s) [15:02:03] Logged the message, Master [15:02:12] (03CR) 10coren: [C: 032] "That's useful." [operations/puppet] - 10https://gerrit.wikimedia.org/r/146000 (https://bugzilla.wikimedia.org/54052) (owner: 10Tim Landscheidt) [15:03:30] (03PS1) 10Manybubbles: Revert "Increase weighting of title field for Commons" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146074 [15:03:36] (03CR) 10Manybubbles: [C: 032] Revert "Increase weighting of title field for Commons" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146074 (owner: 10Manybubbles) [15:03:45] (03Merged) 10jenkins-bot: Revert "Increase weighting of title field for Commons" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146074 (owner: 10Manybubbles) [15:04:13] !log manybubbles Synchronized wmf-config: SWAT update cirrus settings for commons (duration: 00m 04s) [15:04:19] Logged the message, Master [15:04:24] ^demon|away: https://logstash.wikimedia.org/#/dashboard/elasticsearch/fatalmonitor [15:04:25] (03CR) 10Filippo Giunchedi: "ACK, if that's the case there's my comment re: gbp.conf and we should be good to go!" [operations/debs/php-mailparse] (review) - 10https://gerrit.wikimedia.org/r/142751 (owner: 1020after4) [15:05:22] <^demon|away> manybubbles: aw boo, ok [15:06:06] ^demon|away: needs a default? [15:07:29] * ^demon|away git pull'd locally but not seeing the second change [15:07:58] <^demon|away> Oh because revert. [15:10:24] (03PS1) 10Chad: Revert "Revert "Increase weighting of title field for Commons"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146075 [15:10:27] <^demon|away> manybubbles: fixed ^ [15:10:53] (03CR) 10Manybubbles: [C: 032] Revert "Revert "Increase weighting of title field for Commons"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146075 (owner: 10Chad) [15:10:54] (03Merged) 10jenkins-bot: Revert "Revert "Increase weighting of title field for Commons"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146075 (owner: 10Chad) [15:11:27] !log manybubbles Synchronized wmf-config: SWAT update cirrus settings for commons (duration: 00m 04s) [15:11:42] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: Fetching origin [15:11:52] PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: Fetching origin [15:12:12] PROBLEM - Unmerged changes on repository puppet on virt0 is CRITICAL: Fetching origin [15:13:00] ^demon|away: wasn't really good enough - might want to crank the number to 40 or something. Or just wait for Tuesday for the link counts fix. (I think it is Tuesday, right?) [15:13:49] <^demon|away> manybubbles: Well only the weighting went out. [15:13:55] ^demon|away: yeah [15:14:05] <^demon|away> The links de-boosting needs newest wmf branch. [15:14:08] <^demon|away> I guess tomorrow? [15:14:17] its something, but lets wait to be happy until the links stuff goes too [15:14:19] yeah [15:14:21] I think [15:14:35] (03CR) 10Dzahn: [C: 031] "yes please, this will get rid of all the warnings related to admins.pp that we still see in jenkins checks" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146070 (owner: 10Rush) [15:14:45] <^demon|away> Yeah, it was designed to work together. I didn't want to throw the title weighting *too* far without the link de-boosting live. [15:14:52] <^demon|away> If we're still not happy after that we can go for 40. [15:16:06] k [15:17:02] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Mon Jul 14 15:16:55 UTC 2014 [15:18:08] (03PS5) 10Giuseppe Lavagetto: apache: on service refresh, do a graceful reload instead of start/stop [operations/puppet] - 10https://gerrit.wikimedia.org/r/145908 (owner: 10Ori.livneh) [15:18:16] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] apache: on service refresh, do a graceful reload instead of start/stop [operations/puppet] - 10https://gerrit.wikimedia.org/r/145908 (owner: 10Ori.livneh) [15:19:21] (03CR) 10Giuseppe Lavagetto: [C: 031] naggen2: only pick up resources older than 1 hour by default [operations/puppet] - 10https://gerrit.wikimedia.org/r/145315 (owner: 10BBlack) [15:23:51] (03CR) 10Dzahn: [C: 032] Need oxygen access to get at lsearchd logs [operations/puppet] - 10https://gerrit.wikimedia.org/r/145054 (owner: 10Chad) [15:24:35] RECOVERY - Unmerged changes on repository puppet on strontium is OK: Fetching origin [15:24:45] RECOVERY - Unmerged changes on repository puppet on palladium is OK: Fetching origin [15:26:12] (03CR) 10Dzahn: "^d: root@oxygen:~# id demon" [operations/puppet] - 10https://gerrit.wikimedia.org/r/145054 (owner: 10Chad) [15:30:19] (03PS1) 10Giuseppe Lavagetto: monitoring-git: fix icinga message [operations/puppet] - 10https://gerrit.wikimedia.org/r/146078 [15:31:27] (03CR) 10Dzahn: [C: 031] "ah :) yes we get that all the time "fetching origin.."" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146078 (owner: 10Giuseppe Lavagetto) [15:37:37] (03Abandoned) 10Giuseppe Lavagetto: Revert "platform: simplify hardware specific configuration" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146069 (owner: 10Giuseppe Lavagetto) [15:41:57] (03CR) 10Andrew Bogott: [C: 031] "This seems right to me. I'd like to get a +1 from Yuvi before I merge" [operations/puppet] - 10https://gerrit.wikimedia.org/r/145974 (owner: 10BryanDavis) [15:42:58] (03CR) 10Andrew Bogott: [C: 032] labs_vagrant: cleanup sudoers config [operations/puppet] - 10https://gerrit.wikimedia.org/r/145975 (owner: 10BryanDavis) [15:45:04] (03CR) 10Andrew Bogott: [C: 032] Tools: Unify Tools and Toolsbeta configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/102385 (owner: 10Tim Landscheidt) [15:45:27] (03CR) 10Filippo Giunchedi: [C: 031] gitblit: fully qualify vars [operations/puppet] - 10https://gerrit.wikimedia.org/r/145894 (owner: 10Matanya) [15:56:50] (03PS1) 10Giuseppe Lavagetto: mediawiki: manage single configs via apache::site (WiP) [operations/puppet] - 10https://gerrit.wikimedia.org/r/146082 [15:58:16] (03CR) 10Giuseppe Lavagetto: "At the moment this is a noop change, I want your feedback before moving forward with this." [operations/puppet] - 10https://gerrit.wikimedia.org/r/146082 (owner: 10Giuseppe Lavagetto) [16:00:17] (03PS1) 10Dzahn: retab role/gerrit.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/146087 [16:01:05] (03PS2) 10Rush: removing admins.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/146070 [16:03:31] (03CR) 10Rush: [C: 032 V: 032] "3...2...1...explosions? :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146070 (owner: 10Rush) [16:03:33] (03PS1) 10Reedy: Apache config for foundationwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146088 [16:04:33] (03CR) 10Tim Landscheidt: "Ran Puppet manually on tools-mail and tools-login and checked the logs of the automatic runs; the existing exim4 configuration wasn't touc" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102385 (owner: 10Tim Landscheidt) [16:06:01] <_joe_> chasemp: \o/ [16:06:01] (03PS1) 10Reedy: Apache config for Wikimania wikis using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146089 [16:06:36] <_joe_> Reedy: arg, now? [16:06:57] _joe_: I don't think so... ori just asked me to do it [16:07:16] <_joe_> Reedy: uhm ok [16:07:49] <_joe_> Reedy: deployment for apache has changed btw [16:08:01] <_joe_> but, I'll update wikitech [16:11:34] (03PS1) 10BBlack: beta::natfix removal step 1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/146090 [16:11:36] (03PS1) 10BBlack: beta::natfix removal step 2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/146091 [16:13:13] (03PS1) 10Reedy: Apache config for metawiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146092 [16:15:38] (03PS1) 10Reedy: Apache config for sourceswiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146093 [16:17:31] (03PS1) 10Reedy: Apache config for commonswiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146095 [16:19:18] (03PS2) 10Giuseppe Lavagetto: mediawiki: manage single configs via apache::site (WiP) [operations/puppet] - 10https://gerrit.wikimedia.org/r/146082 [16:28:03] (03PS1) 10Reedy: Apache config for grantswiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146096 [16:29:09] RECOVERY - Unmerged changes on repository puppet on virt0 is OK: Fetching origin [16:32:02] (03PS1) 10Reedy: Apache config for fdcwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146097 [16:32:39] Reedy: !!!!!!!!!!!! [16:32:41] yes! [16:32:42] sweet! [16:33:07] (03PS1) 10Ori.livneh: Mark instance vars with @ in erb template [operations/puppet] - 10https://gerrit.wikimedia.org/r/146098 [16:33:56] (03PS1) 10Reedy: Apache config for internalwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146099 [16:34:14] (03CR) 10Ori.livneh: [C: 032] "trivial" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146098 (owner: 10Ori.livneh) [16:34:50] <_joe_> ori: of course Reedy did this the moment I copied the files from apache-config into puppet for my latest patch :P [16:35:13] <_joe_> ok now I'm really off [16:35:35] (03PS1) 10Reedy: Apache config for boardwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146100 [16:36:57] (03PS1) 10Reedy: Apache config for boardgovcomwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146101 [16:38:29] (03PS1) 10Reedy: Apache config for spcomwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146102 [16:39:37] (03PS1) 10Reedy: Apache config for chapcomwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146103 [16:40:34] (03PS1) 10Reedy: Apache config for incubatorwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146104 [16:42:01] (03PS1) 10Reedy: Apache config for specieswiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146105 [16:43:42] (03PS1) 10Reedy: Apache config for searchcomwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146106 [16:44:35] (03PS1) 10Reedy: Apache config for usabilitywiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146107 [16:45:15] (03PS4) 10Andrew Bogott: Modify nova role to better support labs uses. [operations/puppet] - 10https://gerrit.wikimedia.org/r/141836 [16:46:25] (03PS1) 10Reedy: Apache config for strategywiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146108 [16:46:40] (03CR) 10Katie Horn: [C: 032] Add new CentralNotice cookie config var [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145736 (owner: 10Ejegg) [16:46:50] (03PS5) 10Andrew Bogott: Modify nova role to better support labs uses. [operations/puppet] - 10https://gerrit.wikimedia.org/r/141836 [16:47:10] (03PS1) 10Chad: nlwiki gets Cirrus as primary search [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146110 [16:47:12] (03PS1) 10Chad: jawiki gets Cirrus as primary search [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146111 [16:47:17] (03PS1) 10Dzahn: replace literal tabs in role/cache.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/146112 [16:47:42] (03PS1) 10Reedy: Apache config for officewiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146113 [16:48:39] (03PS1) 10Aaron Schulz: Paramerize # runners per loop and add a transcode one [operations/puppet] - 10https://gerrit.wikimedia.org/r/146114 [16:48:56] (03PS1) 10Reedy: Apache config for chairwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146116 [16:49:04] ori: ^ [16:49:36] (03CR) 10jenkins-bot: [V: 04-1] Paramerize # runners per loop and add a transcode one [operations/puppet] - 10https://gerrit.wikimedia.org/r/146114 (owner: 10Aaron Schulz) [16:49:38] (03PS1) 10Ori.livneh: rename: Exec['apache2-restart'] => Exec['apache2_hard_restart'] [operations/puppet] - 10https://gerrit.wikimedia.org/r/146117 [16:49:48] (03PS1) 10Reedy: Apache config for advisorywiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146118 [16:49:52] AaronSchulz: Jenkins says no [16:51:06] (03PS1) 10Reedy: Apache config for auditcomwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146119 [16:51:20] missing , [16:51:58] (03PS1) 10Reedy: Apache config for qualitywiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146121 [16:52:01] (03PS2) 10Aaron Schulz: Paramerize # runners per loop and add a transcode one [operations/puppet] - 10https://gerrit.wikimedia.org/r/146114 [16:52:48] (03CR) 10Ori.livneh: [C: 04-1] "I don't love it. The name and number of parameters that this class accepts should remain constant even if new job types are added or old o" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146114 (owner: 10Aaron Schulz) [16:52:50] (03PS1) 10Reedy: Apache config for otrswiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146122 [16:54:09] (03PS1) 10Reedy: Apache config for collabwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146123 [16:54:57] (03PS1) 10Reedy: Apache config for outreachwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146124 [16:56:02] (03PS1) 10Reedy: Apache config for movementroleswiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146125 [16:56:16] ori: so do you want to ignore TMH jobs for now? [16:56:45] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: Fetching readonly [16:56:53] oh, I'm a dork. I see what you're saying now [16:57:00] we need to parametrize it different for that class of machines [16:57:03] (03PS1) 10Ottomata: Fix unused variable in oozie.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/146126 [16:57:05] (03PS1) 10Reedy: Apache config for checkuserwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146127 [16:57:05] oof, hang on. [16:57:35] (03CR) 10Ottomata: [C: 032 V: 032] Fix unused variable in oozie.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/146126 (owner: 10Ottomata) [16:58:05] (03PS1) 10Reedy: Apache config for stewardwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146128 [16:58:52] (03PS1) 10Reedy: Apache config for ombudsmenwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146129 [16:59:31] (03PS6) 10Andrew Bogott: Modify nova role to better support labs uses. [operations/puppet] - 10https://gerrit.wikimedia.org/r/141836 [17:00:04] manybubbles, ^d: The time is nigh to deploy Search (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140714T1700) [17:01:26] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Mon 14 Jul 2014 15:00:27 UTC [17:02:56] <^demon|away> manybubbles: You ready to do this shiz? [17:03:13] ^demon|away: do it [17:03:22] (03PS7) 10Andrew Bogott: Modify nova role to better support labs uses. [operations/puppet] - 10https://gerrit.wikimedia.org/r/141836 [17:03:26] (03CR) 10Manybubbles: [C: 031] jawiki gets Cirrus as primary search [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146111 (owner: 10Chad) [17:03:30] (03CR) 10Manybubbles: [C: 031] nlwiki gets Cirrus as primary search [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146110 (owner: 10Chad) [17:03:36] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [500.0] [17:03:48] (03CR) 10Chad: [C: 032] nlwiki gets Cirrus as primary search [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146110 (owner: 10Chad) [17:03:55] (03Merged) 10jenkins-bot: nlwiki gets Cirrus as primary search [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146110 (owner: 10Chad) [17:04:34] !log demon Synchronized wmf-config/InitialiseSettings.php: nlwiki getting cirrus as primary (duration: 00m 04s) [17:04:39] Logged the message, Master [17:04:46] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: Fetching readonly [17:05:15] !log started mailman on sodium post-reboot [17:05:20] Logged the message, Master [17:05:40] AaronSchulz: what about having the class accept a runners param that is a hash that looks like this: https://dpaste.de/OWo0/raw [17:06:10] (03CR) 10Andrew Bogott: [C: 032] Modify nova role to better support labs uses. [operations/puppet] - 10https://gerrit.wikimedia.org/r/141836 (owner: 10Andrew Bogott) [17:06:28] AaronSchulz: (with additional keys for other groups) [17:06:52] i can implement that quickly if you like [17:07:17] (03CR) 10Ori.livneh: [C: 032] rename: Exec['apache2-restart'] => Exec['apache2_hard_restart'] [operations/puppet] - 10https://gerrit.wikimedia.org/r/146117 (owner: 10Ori.livneh) [17:07:41] ^demon|away: looks like something is up with the DB? [17:07:45] Error connecting to 10.64.16.26: :real_connect(): (HY000/1040): Too many connections [17:07:50] not us I think [17:07:58] <^demon|away> Where you see that? [17:08:06] logstash [17:08:26] (03PS3) 10Ori.livneh: mediawiki: move SSHD nice override from web.pp to init.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/145510 [17:08:34] (03CR) 10Ori.livneh: [C: 032 V: 032] mediawiki: move SSHD nice override from web.pp to init.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/145510 (owner: 10Ori.livneh) [17:09:24] <^demon|away> manybubbles: looks like mostly dewiki? [17:09:29] <^demon|away> there's a dashboard for that error. [17:09:52] ^demon|away: hmmm - all I know off hand is that dewiki and wikidata share dbs [17:09:59] beyond that, I don't know anything about the error [17:10:04] and it isn't likely to by us [17:10:09] hopefully transient? [17:10:14] is it an eexternal storage? [17:10:16] ori: that would be nice [17:10:19] !log purging old local-* service group entries from labs ldap (via purgeOldServiceGroups.php) [17:10:20] * hoo to lazy to look up [17:10:25] Logged the message, Master [17:10:26] <^demon|away> springle: There seems to be a spike in "too many connections" errors in mw. s5 I think? [17:10:56] <^demon|away> 1371 in the last ~hour. [17:11:41] <^demon|away> Hmm, all at 17:02. Anyway, moving on. [17:11:44] <^demon|away> manybubbles: Not us. [17:12:32] ~~ [17:15:23] (03PS2) 10Ori.livneh: twemproxy: remove leftovers post-decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/144757 [17:16:03] (03CR) 10jenkins-bot: [V: 04-1] twemproxy: remove leftovers post-decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/144757 (owner: 10Ori.livneh) [17:16:18] <^demon|away> manybubbles: elastic seems to be all dandy as far as I can see. Ready for jawiki? [17:16:22] (03PS1) 10Ori.livneh: add 'puppet-run' bash alias to my .bash_profile [operations/puppet] - 10https://gerrit.wikimedia.org/r/146132 [17:16:31] ^demon|away: sure - do it [17:17:03] (03CR) 10Chad: [C: 032] jawiki gets Cirrus as primary search [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146111 (owner: 10Chad) [17:17:05] (03CR) 10jenkins-bot: [V: 04-1] add 'puppet-run' bash alias to my .bash_profile [operations/puppet] - 10https://gerrit.wikimedia.org/r/146132 (owner: 10Ori.livneh) [17:17:14] (03Merged) 10jenkins-bot: jawiki gets Cirrus as primary search [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146111 (owner: 10Chad) [17:18:06] !log demon Synchronized wmf-config/InitialiseSettings.php: jawiki getting cirrus as primary (duration: 00m 04s) [17:18:36] 17:17:11 stderr: fatal: unable to connect to zuul.eqiad.wmnet: [17:18:36] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [17:18:37] 17:17:11 zuul.eqiad.wmnet: Temporary failure in name resolution [17:18:37] 17:17:11 [17:18:39] (from labs) [17:18:42] Eh... [17:20:03] ^demon|away: looks good. I see requests coming in and load is a-ok-awesoem [17:20:56] !log mailman down on sodium (i.e. mailing lists), seven lists with zero-ed config and hence unavailable [17:21:20] Krinkle: being looked at in -labs [17:21:48] <^demon|away> manybubbles: Yay :) [17:22:11] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Puppet has 12 failures [17:22:20] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Puppet has 12 failures [17:22:20] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: Puppet has 12 failures [17:22:30] PROBLEM - Hadoop DataNode on analytics1037 is CRITICAL: NRPE: Command check_hadoop-hdfs-datanode not defined [17:22:40] PROBLEM - Hadoop DataNode on analytics1036 is CRITICAL: NRPE: Command check_hadoop-hdfs-datanode not defined [17:22:40] PROBLEM - Hadoop NodeManager on analytics1037 is CRITICAL: NRPE: Command check_hadoop-yarn-nodemanager not defined [17:22:40] PROBLEM - Hadoop DataNode on analytics1035 is CRITICAL: NRPE: Command check_hadoop-hdfs-datanode not defined [17:22:41] PROBLEM - Hadoop NodeManager on analytics1036 is CRITICAL: NRPE: Command check_hadoop-yarn-nodemanager not defined [17:22:50] PROBLEM - Hadoop NodeManager on analytics1035 is CRITICAL: NRPE: Command check_hadoop-yarn-nodemanager not defined [17:24:20] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:24:30] RECOVERY - Hadoop DataNode on analytics1037 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode [17:24:40] RECOVERY - Hadoop DataNode on analytics1036 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode [17:24:40] RECOVERY - Hadoop NodeManager on analytics1037 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [17:24:40] RECOVERY - Hadoop NodeManager on analytics1036 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [17:24:41] RECOVERY - Hadoop DataNode on analytics1035 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode [17:24:45] (03PS1) 10Ottomata: Install CDH5 clients on stat1002 [operations/puppet] - 10https://gerrit.wikimedia.org/r/146135 [17:24:50] RECOVERY - Hadoop NodeManager on analytics1035 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [17:24:54] !log installing CDH5 on analytics1035-1037 [17:25:04] (03PS2) 10Ottomata: Install CDH5 clients on stat1002 [operations/puppet] - 10https://gerrit.wikimedia.org/r/146135 [17:25:10] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [17:25:20] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [17:25:45] (03CR) 10jenkins-bot: [V: 04-1] Install CDH5 clients on stat1002 [operations/puppet] - 10https://gerrit.wikimedia.org/r/146135 (owner: 10Ottomata) [17:26:42] (03CR) 10Ottomata: "recheck" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146135 (owner: 10Ottomata) [17:30:08] (03CR) 10Ottomata: [C: 032 V: 032] Install CDH5 clients on stat1002 [operations/puppet] - 10https://gerrit.wikimedia.org/r/146135 (owner: 10Ottomata) [17:32:32] (03PS4) 10Ori.livneh: mediawiki: use apache module [operations/puppet] - 10https://gerrit.wikimedia.org/r/145620 [17:33:35] (03PS2) 10Ori.livneh: add 'puppet-run' bash alias to my .bash_profile [operations/puppet] - 10https://gerrit.wikimedia.org/r/146132 [17:34:50] PROBLEM - NTP on analytics1036 is CRITICAL: NTP CRITICAL: Offset unknown [17:35:34] !log Jenkins slaves in labs are unable to reach zuul.eqiad.wmnet [17:35:39] Logged the message, Master [17:35:50] RECOVERY - NTP on analytics1036 is OK: NTP OK: Offset -0.004885435104 secs [17:38:53] (03PS5) 10Ori.livneh: mediawiki: use apache module [operations/puppet] - 10https://gerrit.wikimedia.org/r/145620 [17:40:17] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Mon Jul 14 17:40:13 UTC 2014 [17:43:18] !log npm-cache for integration slaves got corrupted again. Depooling/Repooling integration-slave100{1,2,3} onoe by one to clear cache and let it warm up again. [17:43:23] Logged the message, Master [17:46:01] (03CR) 10Ori.livneh: [C: 032] mediawiki: use apache module [operations/puppet] - 10https://gerrit.wikimedia.org/r/145620 (owner: 10Ori.livneh) [17:55:27] (03PS9) 10Ori.livneh: Move a lot of the miscellaneous wikis out of their own specific docroots [operations/apache-config] - 10https://gerrit.wikimedia.org/r/90703 (owner: 10Reedy) [17:55:38] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: Fetching origin [17:55:47] PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: Fetching origin [17:56:38] RECOVERY - Unmerged changes on repository puppet on strontium is OK: Fetching origin [17:56:47] RECOVERY - Unmerged changes on repository puppet on palladium is OK: Fetching origin [17:58:54] (03PS1) 10Ori.livneh: mediawiki: move File['/usr/local/apache'] from web.pp -> sync.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/146142 [17:59:17] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: Complete puppet failure [17:59:17] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Complete puppet failure [17:59:29] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Complete puppet failure [17:59:29] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Complete puppet failure [17:59:29] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Complete puppet failure [17:59:29] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Complete puppet failure [17:59:39] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: Complete puppet failure [17:59:39] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Complete puppet failure [17:59:40] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: Complete puppet failure [17:59:49] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Complete puppet failure [17:59:49] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: Complete puppet failure [18:00:04] awight: The time is nigh to deploy Translate (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140714T1800) [18:00:09] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Complete puppet failure [18:00:10] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: Complete puppet failure [18:00:10] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: Complete puppet failure [18:00:10] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: Complete puppet failure [18:00:19] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: Complete puppet failure [18:00:19] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: Complete puppet failure [18:00:24] (03CR) 10Awight: [C: 032] Enable FundraisingTranslateWorkflow on metawiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145703 (owner: 10Awight) [18:00:29] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: Complete puppet failure [18:00:29] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: Complete puppet failure [18:00:36] (03Merged) 10jenkins-bot: Enable FundraisingTranslateWorkflow on metawiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145703 (owner: 10Awight) [18:00:39] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Complete puppet failure [18:00:39] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: Complete puppet failure [18:00:49] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Complete puppet failure [18:01:29] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: Complete puppet failure [18:01:39] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: Complete puppet failure [18:01:39] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: Complete puppet failure [18:01:49] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: Complete puppet failure [18:01:49] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: Complete puppet failure [18:01:55] <_joe_> ori: Error: Failed to apply catalog: Could not find dependent Service[apache] for File[/etc/apache2/sites-enabled/99-monitoring.conf] at /etc/puppet/modules/mediawiki/manifests/monitoring/webserver.pp:42 [18:01:59] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Complete puppet failure [18:02:09] PROBLEM - puppet last run on mw1062 is CRITICAL: CRITICAL: Complete puppet failure [18:02:09] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: Complete puppet failure [18:02:10] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: Complete puppet failure [18:02:10] PROBLEM - puppet last run on mw1147 is CRITICAL: CRITICAL: Complete puppet failure [18:02:10] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Complete puppet failure [18:02:39] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: Complete puppet failure [18:02:39] PROBLEM - puppet last run on mw1124 is CRITICAL: CRITICAL: Complete puppet failure [18:02:40] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: Complete puppet failure [18:02:47] (03CR) 1020after4: "gbp.conf is recommended by the Debian php maintainers so I was just following best practices. How is this wrong?" (031 comment) [operations/debs/php-mailparse] (review) - 10https://gerrit.wikimedia.org/r/142751 (owner: 1020after4) [18:02:49] PROBLEM - puppet last run on mw1130 is CRITICAL: CRITICAL: Complete puppet failure [18:03:09] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: Complete puppet failure [18:03:10] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: Complete puppet failure [18:03:10] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: Complete puppet failure [18:03:10] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: Complete puppet failure [18:03:10] PROBLEM - puppet last run on mw1192 is CRITICAL: CRITICAL: Complete puppet failure [18:03:10] PROBLEM - puppet last run on mw1038 is CRITICAL: CRITICAL: Complete puppet failure [18:03:14] (03PS1) 10Andrew Bogott: Remove /etc/init.d/dnsmasq on labs network nodes. [operations/puppet] - 10https://gerrit.wikimedia.org/r/146144 [18:03:20] !log awight updated /a/common to {{Gerrit|Ie7599fb6e}}: jawiki gets Cirrus as primary search [18:03:24] Logged the message, Master [18:03:39] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: Complete puppet failure [18:03:39] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: Complete puppet failure [18:03:39] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Complete puppet failure [18:03:49] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: Complete puppet failure [18:03:50] !log awight Synchronized wmf-config: Deploying FundraisingTranslateWorkflow on metawiki (duration: 00m 05s) [18:03:54] Logged the message, Master [18:03:59] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: Complete puppet failure [18:04:09] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Complete puppet failure [18:04:09] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: Complete puppet failure [18:04:09] PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: Complete puppet failure [18:04:10] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: Complete puppet failure [18:04:10] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Complete puppet failure [18:04:10] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Complete puppet failure [18:04:29] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: Complete puppet failure [18:04:29] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: Complete puppet failure [18:04:39] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: Complete puppet failure [18:04:49] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: Complete puppet failure [18:04:59] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Complete puppet failure [18:05:09] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: Complete puppet failure [18:05:09] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: Complete puppet failure [18:05:10] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: Complete puppet failure [18:05:49] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: Complete puppet failure [18:05:49] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: Complete puppet failure [18:05:49] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: Complete puppet failure [18:05:59] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Complete puppet failure [18:05:59] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Complete puppet failure [18:06:09] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: Complete puppet failure [18:06:09] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Complete puppet failure [18:06:19] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Complete puppet failure [18:06:39] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: Complete puppet failure [18:06:39] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: Complete puppet failure [18:06:49] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Complete puppet failure [18:06:49] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: Complete puppet failure [18:06:49] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Complete puppet failure [18:06:49] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: Complete puppet failure [18:06:49] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Complete puppet failure [18:06:49] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: Complete puppet failure [18:06:49] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Complete puppet failure [18:06:50] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: Complete puppet failure [18:07:09] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Complete puppet failure [18:07:10] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Complete puppet failure [18:07:10] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Complete puppet failure [18:07:10] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: Complete puppet failure [18:07:10] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: Complete puppet failure [18:07:19] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Complete puppet failure [18:07:40] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Complete puppet failure [18:07:49] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Complete puppet failure [18:07:49] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Complete puppet failure [18:07:49] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Complete puppet failure [18:08:09] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Complete puppet failure [18:08:09] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Complete puppet failure [18:08:10] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: Complete puppet failure [18:08:10] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Complete puppet failure [18:08:10] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: Complete puppet failure [18:08:10] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Complete puppet failure [18:08:10] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Complete puppet failure [18:08:29] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Complete puppet failure [18:08:39] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: Complete puppet failure [18:08:39] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: Complete puppet failure [18:08:39] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: Complete puppet failure [18:08:49] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: Complete puppet failure [18:08:49] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: Complete puppet failure [18:09:09] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: Complete puppet failure [18:09:10] PROBLEM - puppet last run on mw1044 is CRITICAL: CRITICAL: Complete puppet failure [18:09:28] (03PS1) 10Hoo man: webserver.pp: Service is name apache2 not apache [operations/puppet] - 10https://gerrit.wikimedia.org/r/146146 [18:09:29] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Complete puppet failure [18:09:39] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: Complete puppet failure [18:09:39] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: Complete puppet failure [18:09:39] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Complete puppet failure [18:09:39] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: Complete puppet failure [18:09:43] fix for puppet failures: https://gerrit.wikimedia.org/r/146146 [18:09:44] godog: ^ [18:09:44] <_joe_> hoo: thanks, we're in meeting atm [18:09:49] PROBLEM - puppet last run on mw1111 is CRITICAL: CRITICAL: Complete puppet failure [18:09:52] <_joe_> hoo: thanks [18:10:09] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Complete puppet failure [18:10:09] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: Complete puppet failure [18:10:10] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: Complete puppet failure [18:10:10] PROBLEM - puppet last run on mw1076 is CRITICAL: CRITICAL: Complete puppet failure [18:10:10] PROBLEM - puppet last run on mw1049 is CRITICAL: CRITICAL: Complete puppet failure [18:10:14] (03CR) 10Giuseppe Lavagetto: [C: 032] webserver.pp: Service is name apache2 not apache [operations/puppet] - 10https://gerrit.wikimedia.org/r/146146 (owner: 10Hoo man) [18:10:39] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Complete puppet failure [18:10:39] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: Complete puppet failure [18:10:49] PROBLEM - puppet last run on mw1125 is CRITICAL: CRITICAL: Complete puppet failure [18:10:49] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: Complete puppet failure [18:10:49] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Complete puppet failure [18:10:58] (03CR) 10Giuseppe Lavagetto: [V: 032] webserver.pp: Service is name apache2 not apache [operations/puppet] - 10https://gerrit.wikimedia.org/r/146146 (owner: 10Hoo man) [18:11:09] PROBLEM - puppet last run on mw1057 is CRITICAL: CRITICAL: Complete puppet failure [18:11:09] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: Complete puppet failure [18:11:09] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Complete puppet failure [18:11:09] PROBLEM - puppet last run on mw1030 is CRITICAL: CRITICAL: Complete puppet failure [18:11:10] PROBLEM - puppet last run on mw1056 is CRITICAL: CRITICAL: Complete puppet failure [18:11:10] PROBLEM - puppet last run on mw1159 is CRITICAL: CRITICAL: Complete puppet failure [18:11:10] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: Complete puppet failure [18:11:10] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: Complete puppet failure [18:11:10] PROBLEM - puppet last run on mw1156 is CRITICAL: CRITICAL: Complete puppet failure [18:11:19] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: Complete puppet failure [18:11:32] I was here to report something but topic says all :) [18:11:39] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: Complete puppet failure [18:11:39] PROBLEM - puppet last run on mw1146 is CRITICAL: CRITICAL: Complete puppet failure [18:11:39] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: Complete puppet failure [18:11:49] <_joe_> JohnLewis: what? [18:11:49] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: Complete puppet failure [18:11:49] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: Complete puppet failure [18:11:55] greg-g: Remedial deployment question. I've merged a config change and sync-dir'ed wmf-config, but still do not see the change on metawiki. Ideas? [18:11:58] _joe_: lists thing :) [18:12:05] <_joe_> JohnLewis: oh ok :) [18:12:09] PROBLEM - puppet last run on mw1023 is CRITICAL: CRITICAL: Complete puppet failure [18:12:09] PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: Complete puppet failure [18:12:10] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: Complete puppet failure [18:12:10] PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: Complete puppet failure [18:12:10] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: Complete puppet failure [18:12:10] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Complete puppet failure [18:12:19] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: Complete puppet failure [18:12:22] awight: What did you change? Which files (eg. a .dblist?) [18:12:29] PROBLEM - puppet last run on mw1171 is CRITICAL: CRITICAL: Complete puppet failure [18:12:39] PROBLEM - puppet last run on mw1116 is CRITICAL: CRITICAL: Complete puppet failure [18:12:39] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: Complete puppet failure [18:13:10] PROBLEM - puppet last run on mw1087 is CRITICAL: CRITICAL: Complete puppet failure [18:13:13] hoo: CommonSettings.php InitialiseSettings.php extension-list [18:13:17] what happened with the lists? [18:13:20] (03PS1) 10Giuseppe Lavagetto: Revert "webserver.pp: Service is name apache2 not apache" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146147 [18:13:21] looks lik we're having some trouble with Fundraising extension? [18:13:39] PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: Complete puppet failure [18:13:45] manybubbles: is this FundraisingTranslateWorkflow ? [18:13:48] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Revert "webserver.pp: Service is name apache2 not apache" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146147 (owner: 10Giuseppe Lavagetto) [18:13:49] PROBLEM - puppet last run on mw1105 is CRITICAL: CRITICAL: Complete puppet failure [18:13:49] PROBLEM - puppet last run on mw1032 is CRITICAL: CRITICAL: Complete puppet failure [18:13:56] 10,000 fatals from it in the past few minutes [18:13:57] awight: yeah [18:13:59] PROBLEM - puppet last run on mw1148 is CRITICAL: CRITICAL: Complete puppet failure [18:14:05] manybubbles: okay, rolling back... [18:14:09] fail to open stream [18:14:10] PROBLEM - puppet last run on mw1185 is CRITICAL: CRITICAL: Complete puppet failure [18:14:10] PROBLEM - puppet last run on mw1108 is CRITICAL: CRITICAL: Complete puppet failure [18:14:14] like it isn't actually deployed [18:14:21] but we're trying to any way [18:14:23] godog: ? [18:14:29] <_joe_> hoo: your patch was goot [18:14:30] _joe_: argh i missed the ping i'll fix [18:14:53] (03PS1) 10Awight: Revert "Enable FundraisingTranslateWorkflow on metawiki" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146148 [18:15:00] (03CR) 10Awight: [C: 032] Revert "Enable FundraisingTranslateWorkflow on metawiki" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146148 (owner: 10Awight) [18:15:09] PROBLEM - Apache HTTP on mw1077 is CRITICAL: Connection refused [18:15:10] PROBLEM - Apache HTTP on mw1024 is CRITICAL: Connection refused [18:15:10] PROBLEM - Apache HTTP on mw1022 is CRITICAL: Connection refused [18:15:10] PROBLEM - Apache HTTP on mw1043 is CRITICAL: Connection refused [18:15:10] PROBLEM - Apache HTTP on mw1167 is CRITICAL: Connection refused [18:15:12] <_joe_> ori: stop [18:15:20] (03Merged) 10jenkins-bot: Revert "Enable FundraisingTranslateWorkflow on metawiki" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146148 (owner: 10Awight) [18:15:24] <_joe_> ori: your change just killed apache [18:15:29] (03PS1) 10Legoktm: Add debug log group for "CentralAuth-Bug39996" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146149 [18:15:29] PROBLEM - Apache HTTP on mw1121 is CRITICAL: Connection refused [18:15:30] <_joe_> please revert it now [18:15:39] PROBLEM - Apache HTTP on mw1209 is CRITICAL: Connection refused [18:15:39] PROBLEM - Apache HTTP on mw1122 is CRITICAL: Connection refused [18:15:49] !log awight Synchronized wmf-config: Revert: Deploying FundraisingTranslateWorkflow on metawiki (duration: 00m 04s) [18:15:49] PROBLEM - Apache HTTP on mw1186 is CRITICAL: Connection refused [18:15:54] Logged the message, Master [18:15:59] PROBLEM - Apache HTTP on mw1033 is CRITICAL: Connection refused [18:15:59] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: Puppet has 1 failures [18:15:59] manybubbles: okay, rolled back. Can you point me to the error? [18:16:10] PROBLEM - Apache HTTP on mw1139 is CRITICAL: Connection refused [18:16:10] PROBLEM - Apache HTTP on mw1219 is CRITICAL: Connection refused [18:16:10] PROBLEM - Apache HTTP on mw1142 is CRITICAL: Connection refused [18:16:10] PROBLEM - Apache HTTP on mw1091 is CRITICAL: Connection refused [18:16:10] awight: https://logstash.wikimedia.org/#/dashboard/elasticsearch/fatalmonitor [18:16:10] PROBLEM - puppet last run on mw1033 is CRITICAL: CRITICAL: Puppet has 1 failures [18:16:10] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: Puppet has 1 failures [18:16:17] <_joe_> ori: this is your change [18:16:19] PROBLEM - Apache HTTP on mw1152 is CRITICAL: Connection refused [18:16:19] PROBLEM - puppet last run on mw1167 is CRITICAL: CRITICAL: Puppet has 1 failures [18:16:19] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: Puppet has 1 failures [18:16:21] <_joe_> the apache failures [18:16:23] <_joe_> I [18:16:27] <_joe_> m reverting it [18:16:29] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: Puppet has 1 failures [18:16:39] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: Puppet has 1 failures [18:16:39] PROBLEM - puppet last run on mw1121 is CRITICAL: CRITICAL: Puppet has 1 failures [18:16:40] PROBLEM - Apache HTTP on mw1093 is CRITICAL: Connection refused [18:16:40] PROBLEM - puppet last run on mw1077 is CRITICAL: CRITICAL: Puppet has 1 failures [18:16:46] ok thanks [18:16:48] fuck [18:16:49] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: Puppet has 1 failures [18:16:49] PROBLEM - Apache HTTP on mw1215 is CRITICAL: Connection refused [18:16:59] PROBLEM - Apache HTTP on mw1193 is CRITICAL: Connection refused [18:16:59] PROBLEM - Apache HTTP on mw1064 is CRITICAL: Connection refused [18:16:59] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: Puppet has 1 failures [18:17:01] <_joe_> ori: can you submit the revert now? [18:17:02] <_joe_> please [18:17:09] PROBLEM - puppet last run on mw1122 is CRITICAL: CRITICAL: Puppet has 1 failures [18:17:09] PROBLEM - puppet last run on mw1139 is CRITICAL: CRITICAL: Puppet has 1 failures [18:17:10] PROBLEM - puppet last run on mw1093 is CRITICAL: CRITICAL: Puppet has 1 failures [18:17:10] PROBLEM - Apache HTTP on mw1112 is CRITICAL: Connection refused [18:17:10] PROBLEM - Apache HTTP on mw1071 is CRITICAL: Connection refused [18:17:19] (03PS1) 10Ori.livneh: Revert "mediawiki: use apache module" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146152 [18:17:28] (03CR) 10Ori.livneh: [C: 032 V: 032] Revert "mediawiki: use apache module" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146152 (owner: 10Ori.livneh) [18:17:29] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: Puppet has 1 failures [18:17:29] PROBLEM - Apache HTTP on mw1143 is CRITICAL: Connection refused [18:17:29] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: Puppet has 1 failures [18:17:37] <_joe_> ori: thanks [18:17:39] PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: Puppet has 1 failures [18:17:39] PROBLEM - Apache HTTP on mw1037 is CRITICAL: Connection refused [18:17:39] PROBLEM - Apache HTTP on mw1027 is CRITICAL: Connection refused [18:17:49] PROBLEM - Apache HTTP on mw1066 is CRITICAL: Connection refused [18:17:49] PROBLEM - Apache HTTP on mw1220 is CRITICAL: Connection refused [18:17:59] PROBLEM - Apache HTTP on mw1204 is CRITICAL: Connection refused [18:18:10] PROBLEM - puppet last run on mw1204 is CRITICAL: CRITICAL: Puppet has 1 failures [18:18:10] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Complete puppet failure [18:18:10] PROBLEM - Apache HTTP on mw1090 is CRITICAL: Connection refused [18:18:10] PROBLEM - Apache HTTP on mw1131 is CRITICAL: Connection refused [18:18:10] PROBLEM - puppet last run on mw1071 is CRITICAL: CRITICAL: Puppet has 1 failures [18:18:10] PROBLEM - Apache HTTP on mw1203 is CRITICAL: Connection refused [18:18:10] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: Puppet has 1 failures [18:18:20] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: Puppet has 1 failures [18:18:29] PROBLEM - Apache HTTP on mw1113 is CRITICAL: Connection refused [18:18:29] PROBLEM - Apache HTTP on mw1086 is CRITICAL: Connection refused [18:18:29] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Puppet has 1 failures [18:18:32] manybubbles: I cannot get into logstash, but I do have fluoring access. I don't see any Fundraising errors in the fatal log?? [18:18:39] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Complete puppet failure [18:18:39] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Complete puppet failure [18:18:39] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: Puppet has 1 failures [18:18:39] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Complete puppet failure [18:18:40] ^ fluorine [18:18:50] PROBLEM - puppet last run on mw1037 is CRITICAL: CRITICAL: Puppet has 1 failures [18:18:50] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: Puppet has 1 failures [18:18:50] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Complete puppet failure [18:18:50] PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: Puppet has 1 failures [18:18:50] PROBLEM - Apache HTTP on mw1021 is CRITICAL: Connection refused [18:18:50] PROBLEM - Apache HTTP on mw1135 is CRITICAL: Connection refused [18:18:51] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Complete puppet failure [18:18:59] PROBLEM - Apache HTTP on mw1104 is CRITICAL: Connection refused [18:18:59] PROBLEM - puppet last run on mw1066 is CRITICAL: CRITICAL: Puppet has 1 failures [18:18:59] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: Puppet has 1 failures [18:19:09] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Complete puppet failure [18:19:09] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Complete puppet failure [18:19:10] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: Puppet has 1 failures [18:19:10] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Puppet has 1 failures [18:19:10] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:19:10] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: Puppet has 1 failures [18:19:13] try fenari and look ata fatalmonitor - I'm surprised you can't do logstash though if you can deploy [18:19:19] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: Puppet has 1 failures [18:19:29] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:19:32] Ino. [18:19:35] where are the logstash logs [18:19:41] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:19:41] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [18:19:51] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Puppet has 1 failures [18:19:51] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [18:19:51] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [18:20:01] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: Puppet has 1 failures [18:20:11] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: Puppet has 1 failures [18:20:21] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [18:20:41] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [18:21:01] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [18:21:01] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:21:02] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [18:21:11] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:21:11] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:21:11] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:21:12] RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [18:21:31] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [18:21:41] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [18:21:41] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:21:41] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [18:21:49] manybubbles: okay thanks, I see what happened. [18:21:51] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [18:21:51] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:22:01] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [18:22:01] (I failed to deploy the new extension) [18:22:11] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:22:11] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:22:12] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [18:22:21] RECOVERY - Apache HTTP on mw1152 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.171 second response time [18:22:21] RECOVERY - Apache HTTP on mw1113 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.443 second response time [18:22:31] RECOVERY - Apache HTTP on mw1086 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.073 second response time [18:22:31] RECOVERY - Apache HTTP on mw1143 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.174 second response time [18:22:31] RECOVERY - Apache HTTP on mw1121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.148 second response time [18:22:41] RECOVERY - Apache HTTP on mw1037 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.106 second response time [18:22:41] RECOVERY - Apache HTTP on mw1209 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.112 second response time [18:22:41] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:22:41] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:22:41] RECOVERY - Apache HTTP on mw1122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.282 second response time [18:22:41] RECOVERY - Apache HTTP on mw1027 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.065 second response time [18:22:42] RECOVERY - Apache HTTP on mw1093 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.092 second response time [18:22:42] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [18:22:51] RECOVERY - Apache HTTP on mw1021 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.093 second response time [18:22:51] RECOVERY - Apache HTTP on mw1066 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.094 second response time [18:22:51] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [18:22:51] RECOVERY - Apache HTTP on mw1186 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.082 second response time [18:22:51] RECOVERY - Apache HTTP on mw1135 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.160 second response time [18:22:52] RECOVERY - Apache HTTP on mw1215 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.081 second response time [18:22:52] RECOVERY - Apache HTTP on mw1220 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.088 second response time [18:22:52] RECOVERY - Apache HTTP on mw1204 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.083 second response time [18:23:01] RECOVERY - Apache HTTP on mw1033 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.068 second response time [18:23:01] RECOVERY - Apache HTTP on mw1064 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.109 second response time [18:23:01] RECOVERY - Apache HTTP on mw1104 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.102 second response time [18:23:01] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:23:01] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:23:01] RECOVERY - Apache HTTP on mw1193 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.179 second response time [18:23:02] RECOVERY - Apache HTTP on mw1077 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.100 second response time [18:23:02] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:23:11] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:23:12] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [18:23:12] RECOVERY - Apache HTTP on mw1022 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.095 second response time [18:23:12] RECOVERY - Apache HTTP on mw1024 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.105 second response time [18:23:12] RECOVERY - Apache HTTP on mw1043 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.107 second response time [18:23:12] RECOVERY - Apache HTTP on mw1112 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.097 second response time [18:23:12] RECOVERY - Apache HTTP on mw1091 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.110 second response time [18:23:12] RECOVERY - Apache HTTP on mw1090 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.078 second response time [18:23:13] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [18:23:13] RECOVERY - Apache HTTP on mw1131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.138 second response time [18:23:14] RECOVERY - Apache HTTP on mw1142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.159 second response time [18:23:14] RECOVERY - Apache HTTP on mw1139 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.163 second response time [18:23:14] RECOVERY - Apache HTTP on mw1219 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.086 second response time [18:23:15] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [18:23:16] RECOVERY - Apache HTTP on mw1071 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.084 second response time [18:23:16] RECOVERY - Apache HTTP on mw1167 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.089 second response time [18:23:17] RECOVERY - Apache HTTP on mw1203 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.090 second response time [18:23:18] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [18:23:21] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:23:31] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [18:23:51] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [18:24:01] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:24:02] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:24:11] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:24:11] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:24:41] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [18:24:41] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:24:51] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:24:51] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:24:51] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:24:51] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:25:01] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:25:11] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:25:11] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:25:12] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [18:25:41] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:25:51] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [18:25:51] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:25:51] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:25:51] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:26:01] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:26:11] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:26:11] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [18:26:12] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [18:26:12] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:26:12] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:26:12] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [18:26:12] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:26:21] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:26:21] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [18:26:31] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:26:41] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [18:26:41] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:26:42] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [18:26:51] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [18:26:51] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [18:26:51] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [18:27:12] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:27:12] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:27:12] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [18:27:21] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:27:41] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:27:41] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:27:41] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:27:51] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:27:51] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [18:28:01] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:28:11] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [18:28:12] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [18:28:12] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:28:12] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:28:12] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [18:28:41] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [18:28:41] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:28:42] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:28:51] RECOVERY - puppet last run on mw1125 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:28:51] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [18:29:02] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:29:11] RECOVERY - puppet last run on mw1044 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [18:29:11] RECOVERY - puppet last run on mw1049 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:29:41] RECOVERY - puppet last run on mw1133 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [18:29:51] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:29:51] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:29:51] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:30:01] RECOVERY - puppet last run on mw1057 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [18:30:02] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:30:11] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:30:11] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [18:30:11] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [18:30:11] RECOVERY - puppet last run on mw1030 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [18:30:12] RECOVERY - puppet last run on mw1056 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [18:30:12] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:30:12] RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [18:30:12] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:30:13] RECOVERY - puppet last run on mw1156 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [18:30:21] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:30:21] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:30:31] RECOVERY - puppet last run on mw1034 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:30:41] RECOVERY - puppet last run on mw1146 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:30:41] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:30:51] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:31:11] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:31:11] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [18:31:31] RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [18:31:41] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:31:41] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:31:51] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [18:32:01] RECOVERY - puppet last run on mw1148 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [18:32:01] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [18:32:11] RECOVERY - puppet last run on mw1097 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [18:32:12] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:32:12] RECOVERY - puppet last run on mw1087 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [18:32:21] RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [18:32:51] RECOVERY - puppet last run on mw1032 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [18:33:11] RECOVERY - puppet last run on mw1108 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [18:33:12] RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [18:33:12] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:33:21] RECOVERY - puppet last run on mw1167 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:33:31] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:33:41] ^d: my @tin ssh pubkey is not working with gerrit. Any ideas? [18:33:41] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:33:41] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [18:33:41] RECOVERY - puppet last run on mw1077 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:33:51] RECOVERY - puppet last run on mw1105 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [18:34:01] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [18:34:01] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [18:34:11] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [18:34:11] RECOVERY - puppet last run on mw1093 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:34:12] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [18:34:21] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [18:34:21] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [18:34:37] awight: Is it working with tin? [18:34:51] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [18:35:01] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [18:35:06] Krinkle: not sure I understand the question, but no it is not working. [18:35:08] awight: You can change public key for Gerrit and labs (separate from production) in wikitech [18:35:11] RECOVERY - puppet last run on mw1139 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [18:35:12] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:35:12] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [18:35:21] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:35:21] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:35:24] Krinkle: I did just regenerate and update my pubkeys in the gerrit UI [18:35:31] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:35:31] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:35:33] * awight looks at wikitech interface [18:35:41] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [18:35:41] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:35:51] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [18:35:51] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [18:35:53] awight: I asked because you called it your "@tin pubkey" assuming you were referring to the wmf production server "tin" [18:36:01] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [18:36:05] you shouldn't use the same keys though (I think). [18:36:11] Krinkle: yes, exactly. I have a separate pubkey in ~/.ssh on tin [18:36:11] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [18:36:11] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:36:11] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [18:36:12] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:36:12] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [18:36:18] so I shouldn't have to ssh -A [18:36:21] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [18:36:41] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [18:36:41] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [18:36:42] You can load as many keys in your ssh config locally as you like (e.g. one for gerrit/labs, one for wmf, one for github, one for amazon etc.) [18:36:48] Krinkle: I don't see any ssh interface on wikitech. [18:36:51] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:37:11] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [18:37:11] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:37:21] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:37:28] awight: https://wikitech.wikimedia.org/wiki/Special:NovaKey [18:37:40] I'm not sure if that's still used though [18:37:41] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:37:51] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:37:51] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [18:37:51] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [18:37:51] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:37:58] I think gerrit has its own one now, maybe imported once from wikitech, but not used live. [18:38:04] Krinkle: thx, I added there but it did not change anything [18:38:11] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [18:38:20] awight: And the thing you're trying to do is pushing to gerrit? [18:38:21] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [18:38:21] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [18:38:21] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [18:38:24] (a git commit) [18:38:31] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:38:31] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [18:38:36] Krinkle: no, I'm trying to git fetch in /a/common/php-1.24wmf12 [18:38:41] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [18:38:41] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:38:41] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:38:56] awight: Are you connected to tin with -A ? [18:39:00] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:39:00] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [18:39:05] Krinkle: I have also tried that, with no success [18:39:10] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [18:39:10] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [18:39:11] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [18:39:11] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [18:39:14] awight: You have to connect with -A from both ends [18:39:21] both your localhost to bastion/fenari and from there to tin [18:39:30] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [18:39:36] ok, /me inspects .ssh/config [18:39:39] and same username of course [18:39:48] what's the error? [18:40:04] actually, the git repo on tin afaik doesn't use credentials [18:40:35] it does, h.. but yeah, that just works if you have -A [18:40:46] make sure your .ssh/config loads both keys [18:41:04] awight: https://github.com/Krinkle/dotfiles/blob/master/hosts/KrinkleMac/templates/sshconfig#L3-L5 [18:41:05] example [18:41:56] yuck. I added a "readonly" remote, that seems to do the nasty trick. [18:43:57] !log awight Synchronized php-1.24wmf12/extensions/FundraisingTranslateWorkflow: Update FundraisingTranslateWorkflow extension (duration: 00m 05s) [18:44:03] Logged the message, Master [18:45:00] ^d: you should have oxygen access [18:45:29] <^d> Indeed. Thank you sir! [18:45:54] !log awight Synchronized php-1.24wmf13/extensions/FundraisingTranslateWorkflow: Update FundraisingTranslateWorkflow extension (wmf13) (duration: 00m 05s) [18:46:00] Logged the message, Master [18:47:03] (03PS1) 10Awight: Enable FundraisingTranslateWorkflow on metawiki (take 2) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146160 [18:47:52] (03CR) 10Awight: [C: 032] Enable FundraisingTranslateWorkflow on metawiki (take 2) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146160 (owner: 10Awight) [18:48:10] (03Merged) 10jenkins-bot: Enable FundraisingTranslateWorkflow on metawiki (take 2) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146160 (owner: 10Awight) [18:49:16] !log awight Synchronized wmf-config: Deploying FundraisingTranslateWorkflow on metawiki (t [18:49:21] Logged the message, Master [18:49:47] (03PS1) 10Giuseppe Lavagetto: mediawiki: use apache module [operations/puppet] - 10https://gerrit.wikimedia.org/r/146162 [18:51:11] PROBLEM - Disk space on sodium is CRITICAL: DISK CRITICAL - free space: / 226 MB (1% inode=52%): /var/lib/ureadahead/debugfs 226 MB (1% inode=52%): [18:51:38] godog: ^ yay, does that mean restore worked?:) [18:53:18] haha no :( it is mailman stuff queueing up [18:54:14] :/ [18:56:01] oh no nevermind I think you are right [18:56:10] RECOVERY - Disk space on sodium is OK: DISK OK [19:00:04] ori, _joe_: The time is nigh to deploy Jobrunner (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140714T1900) [19:00:21] <_joe_> yeah thanks :) [19:00:42] <_joe_> ori: so... wanna try mw again first? [19:00:49] yes, looking at yr change [19:01:06] <_joe_> it misses the fix to monitoring.pp btw [19:01:48] i need to submit an update to aaron's patch first, hang on [19:02:11] <_joe_> ok [19:02:32] <_joe_> take your time [19:03:58] !log re-enabling mailman on sodium, missing list config restored [19:04:03] Logged the message, Master [19:04:07] godog: awesome [19:04:16] that works :) [19:04:27] godog: i told -staff .. updating [19:04:38] mutante: thanks! watching mailman now [19:04:57] what happened to sodium? [19:06:02] greg-g: it froze completely and we had to powercycle, after that some mailman configs were corrupted [19:06:08] so mailman service would not start [19:06:11] do we know why it froze? [19:06:16] greg-g: why it froze.. maybe it was this [19:06:55] greg-g: https://gerrit.wikimedia.org/r/#/c/144033/ https://gerrit.wikimedia.org/r/#/c/146068/ [19:07:18] that create a file /etc/init/.conf [19:07:25] on older hosts with puppet 2.7 [19:07:37] should have been /etc/init/something.conf [19:08:53] bbiaw [19:11:52] ok [19:12:00] i think we should go with aaron's change as-is right now [19:12:11] this is https://gerrit.wikimedia.org/r/#/c/146114/ [19:12:22] <_joe_> ori: let me take another look please [19:12:29] yeah, flagging it for that reason [19:12:36] i'll look over your patch once again [19:13:02] (03PS2) 10Giuseppe Lavagetto: mediawiki: use apache module [operations/puppet] - 10https://gerrit.wikimedia.org/r/146162 [19:15:08] <_joe_> ori: grinding Aaron's change through the compiler just to be sure [19:15:22] <_joe_> my willingness to risk has been drastically reduced right now [19:15:54] <_joe_> one near-outage with my +1 is enough for today :( [19:19:30] mmm, I'm voiced? :P [19:20:10] <_joe_> MaxSem: you are [19:20:29] (03PS3) 10Giuseppe Lavagetto: Paramerize # runners per loop and add a transcode one [operations/puppet] - 10https://gerrit.wikimedia.org/r/146114 (owner: 10Aaron Schulz) [19:20:54] <_joe_> ori: ok this change is a noop at the moment given the mediawiki::jobrunner class isn't included in production... [19:21:03] yep [19:21:22] (03CR) 10Giuseppe Lavagetto: [C: 032] Paramerize # runners per loop and add a transcode one [operations/puppet] - 10https://gerrit.wikimedia.org/r/146114 (owner: 10Aaron Schulz) [19:21:35] <_joe_> oh god jenkins is slooowwww [19:22:12] <_joe_> ori: merged [19:23:32] AaronSchulz: do you have a followup patch parametrizing it for prod? [19:24:13] no, I can make one though [19:24:24] * AaronSchulz leaves php-src land [19:24:46] <_joe_> we should enable it only for one host first [19:27:15] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 7 below the confidence bounds [19:27:24] PROBLEM - Apache HTTP on mw1152 is CRITICAL: Connection timed out [19:27:31] <_joe_> mmmh [19:27:34] PROBLEM - Apache HTTP on mw1064 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:44] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [19:30:42] _joe_: one host? Via running it on puppet there first or the actual repo change? [19:32:23] <_joe_> AaronSchulz: no, like setting a guard if hostname == 'mw1001' {} else {} [19:32:24] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: Puppet has 1 failures [19:33:11] <_joe_> which we can remove after we're sure this is working [19:33:15] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: Puppet has 1 failures [19:34:31] <_joe_> http://ganglia.wikimedia.org/latest/?c=Application%20servers%20eqiad&h=mw1152.eqiad.wmnet&m=cpu_report&r=hour&s=descending&hc=4&mc=2 we should probably reboot mw1152 [19:35:02] yeah [19:35:22] should i depool it? [19:35:26] <_joe_> no [19:35:35] <_joe_> pybal already did that [19:35:37] <_joe_> apache is dead [19:35:43] <_joe_> rebooting it [19:36:01] <_joe_> ori, AaronSchulz I'll deal with this machine and the other one [19:39:24] (03PS2) 10Legoktm: Add two new CentralAuth debug log groups [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146149 [19:40:19] RECOVERY - Apache HTTP on mw1152 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.256 second response time [19:40:40] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [19:44:32] (03PS1) 10Ori.livneh: jobrunner: provision on mw1001; fix template [operations/puppet] - 10https://gerrit.wikimedia.org/r/146177 [19:44:33] <_joe_> !log killed a lone apache2 child on mw1152, stuck in a futex, after a segfault of another apache process. Restarted apache, now working correctly [19:44:39] Logged the message, Master [19:45:11] <_joe_> !log doing the same on mw1064, segfaulted for the same reason [19:45:16] Logged the message, Master [19:45:31] (03CR) 10jenkins-bot: [V: 04-1] jobrunner: provision on mw1001; fix template [operations/puppet] - 10https://gerrit.wikimedia.org/r/146177 (owner: 10Ori.livneh) [19:45:42] <_joe_> ori: bbiab [19:46:19] RECOVERY - Apache HTTP on mw1064 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.071 second response time [19:46:39] (03PS2) 10Ori.livneh: jobrunner: provision on mw1001; fix template [operations/puppet] - 10https://gerrit.wikimedia.org/r/146177 [19:48:49] (03CR) 10Matanya: "dup of https://gerrit.wikimedia.org/r/140678" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146112 (owner: 10Dzahn) [19:49:32] ori: I thought ~= was funny, hehe [19:49:50] (03PS1) 10Andrew Bogott: Refactor the project-loading logic. [operations/debs/adminbot] - 10https://gerrit.wikimedia.org/r/146180 [19:49:52] (03PS1) 10Andrew Bogott: Update the get_projects so that it also enumerates service groups. [operations/debs/adminbot] - 10https://gerrit.wikimedia.org/r/146181 [19:49:54] (03CR) 10Ori.livneh: [C: 04-1] mediawiki: use apache module (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/146162 (owner: 10Giuseppe Lavagetto) [19:50:19] (03PS2) 10Andrew Bogott: Update get_projects so that it also enumerates service groups. [operations/debs/adminbot] - 10https://gerrit.wikimedia.org/r/146181 [19:50:43] (03CR) 10Aaron Schulz: [C: 031] jobrunner: provision on mw1001; fix template [operations/puppet] - 10https://gerrit.wikimedia.org/r/146177 (owner: 10Ori.livneh) [19:51:20] <_joe_> ori: i did not understand your comment... [19:51:50] the apache module already declares the 'apache2-mpm-prefork' package [19:51:56] so you'll get a duplicate def'n with that [19:52:04] <_joe_> ugh? [19:52:12] <_joe_> so why did that fuckup happen? [19:52:26] because a minority of app servers had a different mpm installed [19:52:38] <_joe_> which one? [19:52:42] <_joe_> a minority? [19:53:07] the ones that died, presumably [19:53:08] <_joe_> I'd say what I saw happening was mpm-prefork being uninstalled [19:53:23] <_joe_> no ori something else happened I think [19:53:34] <_joe_> let me recheck then [19:54:22] <_joe_> one question about your change: shouldn't we de-commission jobqueue on mw1001? [19:54:29] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:54:42] _joe_: nope [19:54:43] mutante: know etherpad is down? [19:55:06] _joe_: we want to gradually have jobrunner manage more children than the current jobs-loop [19:55:07] (03PS3) 10Andrew Bogott: Update get_projects so that it also enumerates service groups. [operations/debs/adminbot] - 10https://gerrit.wikimedia.org/r/146181 [19:55:15] <_joe_> ori: oh ok [19:55:19] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [19:56:20] jgage: oh, you're on RT duty. etherpad is down [19:57:14] !log postponing jobrunner deployment to tomorrow; ran over time [19:57:18] Logged the message, Master [19:57:23] ^ AaronSchulz :/ [19:57:28] hm how is that related to RT duty? [19:57:58] jgage: RT duty isn't just about RT triage, it's also quasi on-point/first level tech support :P [19:58:01] (03CR) 10Merlijn van Deen: [C: 031] Update get_projects so that it also enumerates service groups. [operations/debs/adminbot] - 10https://gerrit.wikimedia.org/r/146181 (owner: 10Andrew Bogott) [19:58:10] jgage: eg: take the issue and triage/assign to whomever appropriate [19:58:19] PROBLEM - puppet last run on ssl1003 is CRITICAL: CRITICAL: Puppet has 1 failures [19:58:19] (is how I understand it) [19:58:39] ok [19:58:52] <_joe_> can someone in a more comfortable TZ look at ssl1003? [19:59:01] <_joe_> I'm off for a bit [19:59:41] _joe_: I will look [19:59:58] * jgage looks at etherpad [20:00:03] (03CR) 10Merlijn van Deen: "Note that this removes the '-local' quick-name-option. Maybe this can be re-added by allowing 'X' to be resolved to 'project.X' if that's " [operations/debs/adminbot] - 10https://gerrit.wikimedia.org/r/146180 (owner: 10Andrew Bogott) [20:00:04] gwicke, subbu, cscott: The time is nigh to deploy Parsoid (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140714T2000) [20:00:21] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [20:01:02] greg-g, etherpad.wikimedia.org is working for me. *searches scrollback for problem description* [20:01:21] RECOVERY - puppet last run on ssl1003 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [20:02:22] (03CR) 10Andrew Bogott: "One of the purposes of moving away from the local-* naming scheme was to ensure that tool names are fully qualified and unique across all " [operations/debs/adminbot] - 10https://gerrit.wikimedia.org/r/146180 (owner: 10Andrew Bogott) [20:02:23] ori: :( [20:02:24] jgage: you're right, it's back, it definitely wasn't for 5 minutes [20:02:58] it uses ssl, so maybe you were hitting the ssl1003 problem that _joe_ mentioned? [20:03:08] welp, no sign of an actual puppet failure on ssl1003 [20:03:08] icinga doesn't show any recent outages [20:03:15] <_joe_> jgage: nope [20:03:19] hmm [20:03:21] jgage: maybe? it was a Service Unavailable message :/ [20:03:33] ok, i'll take a look at logs [20:04:16] legoktm says I wasn't crazy [20:04:29] hmmm [20:04:42] yeah, I saw it too [20:04:56] (we're both west-coasters, if that matters) [20:06:55] thanks. did you see an HTTP 5xx error code? [20:07:14] no, just Service Temporarily Unavailable [20:07:39] ok. could you tell whether that was coming from zirconium or the load balancer? [20:08:07] no idea [20:08:22] k [20:08:42] i'll keep looking, anybody who has further problems with etherpad please speak up [20:10:23] * greg-g nods [20:10:38] i see many http requests from an IP in france, coulda been a scan/attack [20:10:43] huh [20:10:46] antoine! [20:10:47] invalid requsts, taht is [20:13:19] about 2000 http errors from one source IP between 19:48 and 19:59, looks like a php vuln scanner [20:15:42] (03CR) 10Andrew Bogott: [C: 032] Refactor the project-loading logic. [operations/debs/adminbot] - 10https://gerrit.wikimedia.org/r/146180 (owner: 10Andrew Bogott) [20:15:54] (03CR) 10Andrew Bogott: [C: 032] Update get_projects so that it also enumerates service groups. [operations/debs/adminbot] - 10https://gerrit.wikimedia.org/r/146181 (owner: 10Andrew Bogott) [20:22:42] !log updated Parsoid to version d51e64097bb1b18e356584d4f3ddcfd90a6071ba [20:22:45] Logged the message, Master [20:26:55] (03PS1) 10Andrew Bogott: Bump debian version [operations/debs/adminbot] - 10https://gerrit.wikimedia.org/r/146254 [20:27:30] (03CR) 10Andrew Bogott: [C: 032] Bump debian version [operations/debs/adminbot] - 10https://gerrit.wikimedia.org/r/146254 (owner: 10Andrew Bogott) [20:37:16] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: Puppet has 1 failures [20:43:19] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [20:45:58] (03PS1) 10Yuvipanda: tools: remove libvips from exec_environ [operations/puppet] - 10https://gerrit.wikimedia.org/r/146259 [20:45:59] andrewbogott: ^ [20:46:38] (03CR) 10Andrew Bogott: [C: 032] "Yep, this isn't helping." [operations/puppet] - 10https://gerrit.wikimedia.org/r/146259 (owner: 10Yuvipanda) [21:09:23] bd808, ori; idea for logstash -- we should write a handler so that we can have structured logs that push data to graphite (e.g. have a node like perf.. = ) [21:09:34] that way I dont have to have a log line and a stats line [21:10:42] mwalker: file a bug! And then implement! :) [21:13:12] (03PS1) 10Reedy: Apache config for wikimedia chapters using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146265 [21:14:04] (03PS1) 10Reedy: Apache config for loginwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146266 [21:15:06] (03PS1) 10Reedy: Apache config for legalteamwiki sing mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146267 [21:15:25] * YuviPanda sings mod_proxy_fcgi to Reedy [21:16:03] (03PS1) 10Reedy: Apache config for zerowiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146268 [21:17:15] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Mon 14 Jul 2014 19:16:33 UTC [21:17:25] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Mon Jul 14 21:17:19 UTC 2014 [21:20:09] (03PS1) 10Reedy: Apache config for transitionteamwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146270 [21:22:27] (03PS1) 10Reedy: Apache config for iegcomwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146271 [21:27:54] (03PS1) 10Reedy: Apache config for Wikiversity using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146275 [21:29:45] (03PS1) 10Reedy: Apache config for Wikinews using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146276 [21:30:48] (03PS1) 10Reedy: Apache config for Wikisource using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146277 [21:31:53] (03PS1) 10Reedy: Apache config for Wikibooks using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146278 [21:33:38] (03PS1) 10Reedy: Apache config for Wikipedia using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146280 [21:35:00] (03PS1) 10Reedy: Apache config for Wikiquote using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146283 [21:35:48] who from Ops should I rope into debugging a memcached issue on tin? jgage ? [21:36:46] (03PS1) 10Reedy: Apache config for Wiktionary using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146284 [21:38:34] (03PS1) 10Reedy: Apache config for mediawikiwiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146285 [21:40:36] (03PS1) 10Reedy: Apache config for testwikidatawiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146286 [21:41:17] <_joe_> Reedy: you're cheating! committing while in the meeting [21:41:18] <_joe_> :) [21:42:55] (03PS1) 10Reedy: Apache config for wikidatawiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146287 [21:44:57] (03PS1) 10Ori.livneh: tin: include ::mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/146288 [21:45:01] ^ _joe_ [21:45:06] (03PS1) 10Reedy: Apache config for donatewiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146289 [21:47:26] (03PS1) 10Reedy: Apache config for votewiki using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146292 [22:04:41] morebots, did you come back up properly? [22:04:41] I am a logbot running on tools-exec-11. [22:04:41] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [22:04:41] To log a message, type !log . [22:16:59] TimStarling: (offtopic) is there a reason to prefer apache2-mpm-itk to apache2-mpm-prefork? [22:17:16] some of the app servers had the former [22:17:21] is there a principled reason for that? [22:17:40] I don't even know what itk is [22:18:07] The ITK Multi-Processing Module (MPM) works in about the same way as the [22:18:07] classical "prefork" module (that is, without threads), except that it allows [22:18:07] you to constrain each individual vhost to a particular system user. This [22:18:09] allows you to run several different web sites on a single server without [22:18:11] worrying that they will be able to read each others' files. [22:18:25] we don't do that [22:18:38] per-vhost security settings i mean [22:18:39] this is in Apache 2.2? [22:18:42] yeah [22:24:06] RobH: ping ;) [22:28:36] ori: I don't see anything in the SAL or in puppet [22:28:54] and I'm pretty sure I didn't do it because I've never heard of it [22:29:01] it's hard to imagine it happening by accident... [22:35:03] (03CR) 10Yuvipanda: Tools: Use toollabs::hba in toollabs::webnode (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/145388 (owner: 10Tim Landscheidt) [22:38:26] (03CR) 10Yuvipanda: "Needs manual rebase :(" [operations/puppet] - 10https://gerrit.wikimedia.org/r/124001 (owner: 10Tim Landscheidt) [22:40:52] (03CR) 10Tim Landscheidt: Tools: Use toollabs::hba in toollabs::webnode (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/145388 (owner: 10Tim Landscheidt) [22:44:59] HaeB: Sorry, I was away. Whats up? [22:48:18] (03CR) 10Yuvipanda: [C: 031] Tools: Use toollabs::hba in toollabs::webnode (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/145388 (owner: 10Tim Landscheidt) [22:48:25] scfc_de: I should've slept :) [22:51:36] YuviPanda: Never too late for that :-). [22:51:41] true, true [22:56:17] (03PS1) 10Reedy: Redirect wikimedia.org/research to survey [operations/apache-config] - 10https://gerrit.wikimedia.org/r/146334 [22:57:55] is anyone available to help me with a dns change? [22:58:32] cajoel: by help, do you mean deploy? [22:58:46] a dns deploy -- not puppet [22:58:54] so, sorta [22:59:16] Have you made a patchset? [22:59:50] no, it's a one liner [22:59:57] I can, but it's a one liner.. [23:00:04] mwalker, ori, MaxSem: The time is nigh to deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140714T2300) [23:00:33] cajoel: It still needs to go via version control... So either you need to make it, or the opens deploying it will [23:01:11] MaxSem, mwalker: any chance one of you could take it? [23:01:19] aye; I can do it [23:01:22] thanks [23:02:36] mwalker: thanks! [23:02:51] (03CR) 10Mwalker: [C: 032] Add two new CentralAuth debug log groups [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146149 (owner: 10Legoktm) [23:03:47] James_F, I cannot deploy https://gerrit.wikimedia.org/r/#/c/146166/ [23:03:47] it failed its tests [23:04:23] (03Merged) 10jenkins-bot: Add two new CentralAuth debug log groups [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146149 (owner: 10Legoktm) [23:04:48] (03PS1) 10Jkrauska: Change corp ns servers - step 1 [operations/dns] - 10https://gerrit.wikimedia.org/r/146335 [23:04:54] RoanKattouw, Krinkle, Krenair; as james is currently in the kitchen not paying attention to irc -- do any of you have ideas? [23:05:05] e.g. quickly want to correct the issue whatever it is [23:05:23] RoanKattouw is not here :) [23:05:53] Ideas on what [23:06:00] https://gerrit.wikimedia.org/r/#/c/146166/ [23:06:02] Why the test is broken [23:06:04] /failing [23:06:20] Reedy: dns change ready [23:07:03] mwalker: Reedy: Unrelated test that was added to core recently to catch tests leaking jQuery animations. [23:07:25] mwalker: Reedy: No idea what's causing it to fail. I'm quite sure it fails in that branch regardless of this new commit. [23:07:43] I'd say it's safe to go ahead [23:08:05] This is exactly why Jenkins should test against the appropiate branch of mw-core, not 'master'. [23:08:18] This is a backport to wmf/1.24wmf13 and jenkins tests it with mw-core master [23:08:28] Ignore :) [23:14:25] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: Fetching readonly [23:14:59] (03CR) 10Cmjohnson: [C: 032] Change corp ns servers - step 1 [operations/dns] - 10https://gerrit.wikimedia.org/r/146335 (owner: 10Jkrauska) [23:17:25] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: Fetching readonly [23:17:55] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Mon 14 Jul 2014 21:17:19 UTC [23:20:47] Krinkle, speaking of jenkins; there are a large number of what look like free executors that are not being used to execute jobs and a number of jobs waiting to execute [23:21:03] but we aren't at a complete stall [23:21:23] hah :) [23:24:35] awight, you have an undeployed patch to donation interface in the 1.24wmf13 branch; but you are not part of the swat [23:24:37] what's the story there? [23:24:55] the patch is https://gerrit.wikimedia.org/r/#/c/146156/ [23:25:27] K4-713, ^ [23:27:10] mwalker: thanks. I ran out of time this morning, so triaged only deploying to the wmf12 branch, where this was enabled. [23:29:02] mwalker: Regarding unused executors, this is a known problem because everything related to Jenkins is shit. SNAFU, ask Antoine for details when he's back. [23:29:46] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: Puppet last ran 14448 seconds ago, expected 14400 [23:29:58] Main problem is that the job queue (gearman) and its manager (zuul) are often perfoming blocking execution on some tangent activity and thus unable to distribute jobs until that silly computation finishes. [23:30:01] mwalker: ^ [23:32:18] !log mwalker Started scap: Updating for SWAT {{gerrit|146304}}, {{gerrit|146306}}, {{gerrit|146149}}, {{gerrit|146165}}, {{gerrit|146166}}, {{gerrit|146282}}, and {{gerrit|146281}}. Also finishing awight's deploy of FundraisingTranslateWorkflow. [23:32:24] Logged the message, Master [23:33:17] "19:08 < Krinkle> This is exactly why Jenkins should test against the appropiate branch of mw-core, not 'master'." uhhhh, wtf, why isn't that the case for unit tests? [23:35:08] greg-g: It sounds obvious the way I put it because obviously want us to do this. However implementing that is quite the non-trivial endeavour because branches don't always have a matching one in each repo (e.g. feature branches), and because of all the abstraction layers this will probably need a dozen different angles adjusted. Things tend to not work and with 2 part time resources on it, doing [23:35:08] that isn't a priority. [23:35:47] Things tend to just work against master, and with only 2 part time resources on it, implementing that isn't a priority cmpared to the other issues. [23:35:59] Krinkle: yeah :/ [23:39:15] (03CR) 10Tim Landscheidt: "This isn't true, as libvips15 is not only installable, but in fact perfectly installed on all exec nodes." [operations/puppet] - 10https://gerrit.wikimedia.org/r/146259 (owner: 10Yuvipanda) [23:41:43] Krinkle: Yeah. :-( [23:42:40] (03PS1) 10Tim Landscheidt: Revert "tools: remove libvips from exec_environ" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146348 (https://bugzilla.wikimedia.org/52717) [23:46:06] legoktm, James_F -- please check your changes; scap has pushed the changes we're only waiting on the cdb rebuild [23:47:16] (03PS2) 10Tim Landscheidt: Revert "tools: remove libvips from exec_environ" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146348 (https://bugzilla.wikimedia.org/52717) [23:47:17] awight, your change is now on wmf13; but the only wiki that has it enabled is meta and that's on wmf12, so no change [23:48:17] mwalker: yup, looks good [23:48:17] bbl [23:48:22] mwalker: awesome, thanks for cleaning up after me [23:48:26] mwalker: Looks good from this end. [23:49:00] mwalker: Thank you! [23:52:01] !log mwalker Finished scap: Updating for SWAT {{gerrit|146304}}, {{gerrit|146306}}, {{gerrit|146149}}, {{gerrit|146165}}, {{gerrit|146166}}, {{gerrit|146282}}, and {{gerrit|146281}}. Also finishing awight's deploy of FundraisingTranslateWorkflow. (duration: 19m 42s) [23:52:06] Logged the message, Master [23:54:49] mwalker: eep, that's a lot. thanks very much. [23:55:15] np; someone had to do it :) [23:56:10] (03PS6) 1020after4: Packaging for debian using pkg-php-tools/dh_php5. [operations/debs/php-mailparse] (review) - 10https://gerrit.wikimedia.org/r/142751 [23:57:10] (03CR) 1020after4: "ok I removed gbp.conf just so this can move forward, chase really needs this it's already holding him up." [operations/debs/php-mailparse] (review) - 10https://gerrit.wikimedia.org/r/142751 (owner: 1020after4) [23:58:43] mmww gbp