[00:15:28] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:16:09] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:16:28] PROBLEM - puppet last run on mw2143 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:16:38] PROBLEM - puppet last run on wtp2010 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:16:39] PROBLEM - puppet last run on mw1253 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:16:39] PROBLEM - puppet last run on elastic1027 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:16:40] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:16:40] PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:16:48] PROBLEM - puppet last run on mw2003 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:16:48] PROBLEM - puppet last run on mw2176 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:16:49] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:16:49] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:16:49] PROBLEM - puppet last run on mw2113 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:00] PROBLEM - puppet last run on ms-fe2003 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:10] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:11] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:11] PROBLEM - puppet last run on db2054 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:19] PROBLEM - puppet last run on mw2110 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:20] PROBLEM - puppet last run on db2038 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:29] PROBLEM - puppet last run on mw2070 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:29] PROBLEM - puppet last run on es2010 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:29] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:30] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:33] eh? [00:17:38] PROBLEM - puppet last run on mw2134 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:38] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:39] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:39] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:40] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:48] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:48] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:48] PROBLEM - puppet last run on mw2127 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:49] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:49] PROBLEM - puppet last run on mw2184 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:50] PROBLEM - puppet last run on mw2142 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:50] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:58] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:58] PROBLEM - puppet last run on mw2196 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:58] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:59] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:59] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:17:59] PROBLEM - puppet last run on wtp2016 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:00] PROBLEM - puppet last run on mw1155 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:08] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:18] PROBLEM - puppet last run on mw2093 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:18] PROBLEM - puppet last run on mw2079 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:18] PROBLEM - puppet last run on mw2049 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:19] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:19] PROBLEM - puppet last run on mw2123 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:20] PROBLEM - puppet last run on mw2067 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:20] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:28] PROBLEM - puppet last run on mw2182 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:28] PROBLEM - puppet last run on mw2039 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:29] PROBLEM - puppet last run on mw2131 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:29] PROBLEM - puppet last run on mw2056 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:29] PROBLEM - puppet last run on db2047 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:29] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:30] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:38] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:38] PROBLEM - puppet last run on mw2090 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:39] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:39] PROBLEM - puppet last run on mw2212 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:40] PROBLEM - puppet last run on mw2096 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:40] PROBLEM - puppet last run on mw2092 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:48] PROBLEM - puppet last run on db2029 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:48] PROBLEM - puppet last run on mw1230 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:49] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:49] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:49] PROBLEM - puppet last run on analytics1002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:50] PROBLEM - puppet last run on mw2084 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:59] PROBLEM - puppet last run on install2001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:18:59] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:00] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:00] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:08] PROBLEM - puppet last run on ms-be2009 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:10] PROBLEM - puppet last run on analytics1056 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:18] PROBLEM - puppet last run on mw2172 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:19] PROBLEM - puppet last run on mw2168 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:19] PROBLEM - puppet last run on mw2030 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:19] PROBLEM - puppet last run on ms-be1021 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:20] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:20] PROBLEM - puppet last run on mw2085 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:28] PROBLEM - puppet last run on mw2174 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:28] PROBLEM - puppet last run on mw2130 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:29] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:29] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:30] PROBLEM - puppet last run on mw2111 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:30] PROBLEM - puppet last run on mw2055 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:39] PROBLEM - puppet last run on db2023 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:39] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:48] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:49] PROBLEM - puppet last run on mw2203 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:49] PROBLEM - puppet last run on mw2047 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:49] PROBLEM - puppet last run on nembus is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:50] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:58] PROBLEM - puppet last run on analytics1052 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:59] PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:19:59] PROBLEM - puppet last run on db2041 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:08] PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:10] PROBLEM - puppet last run on analytics1051 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:19] PROBLEM - puppet last run on ms-be2010 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:19] PROBLEM - puppet last run on ms-be2014 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:19] PROBLEM - puppet last run on mw2101 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:19] PROBLEM - puppet last run on elastic1006 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:20] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:29] PROBLEM - puppet last run on db1055 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:40] PROBLEM - puppet last run on mw2062 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:40] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:40] PROBLEM - puppet last run on mw2200 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:40] PROBLEM - puppet last run on mw1238 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:40] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:40] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:40] PROBLEM - puppet last run on db2037 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:20:59] PROBLEM - puppet last run on mira is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:08] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:08] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:09] PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:09] PROBLEM - puppet last run on mw2107 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:09] PROBLEM - puppet last run on ms-be2008 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:09] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:10] PROBLEM - puppet last run on mw2133 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:10] PROBLEM - puppet last run on analytics1044 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:10] PROBLEM - puppet last run on mw2152 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:11] PROBLEM - puppet last run on mw2094 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:20] PROBLEM - puppet last run on mw2098 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:29] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:29] PROBLEM - puppet last run on wtp2009 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:38] PROBLEM - puppet last run on mw1049 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:39] PROBLEM - puppet last run on mw1240 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:39] PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:40] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:40] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:40] PROBLEM - puppet last run on mw2150 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:49] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:49] PROBLEM - puppet last run on mw1030 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:59] PROBLEM - puppet last run on mw2053 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:21:59] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:08] PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:08] PROBLEM - puppet last run on wtp1002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:08] PROBLEM - puppet last run on mw2048 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:09] PROBLEM - puppet last run on mw2202 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:09] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:09] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:10] PROBLEM - puppet last run on mw2144 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:20] PROBLEM - puppet last run on mw1116 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:28] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:28] PROBLEM - puppet last run on mw1062 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:29] PROBLEM - puppet last run on mw1245 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:29] PROBLEM - puppet last run on mw1256 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:39] PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:39] PROBLEM - puppet last run on ms-be1017 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:40] PROBLEM - puppet last run on ms-be2016 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:48] (03CR) 10Chad: [C: 032] Arcanist configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/254468 (owner: 10Dereckson) [00:22:50] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:50] PROBLEM - puppet last run on mw1233 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:50] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:50] PROBLEM - puppet last run on mw2027 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:58] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:58] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:59] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:22:59] PROBLEM - puppet last run on db1070 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:23:00] PROBLEM - puppet last run on ms-be2017 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:23:12] (03Merged) 10jenkins-bot: Arcanist configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/254468 (owner: 10Dereckson) [00:23:18] PROBLEM - puppet last run on mw2106 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:23:18] PROBLEM - puppet last run on mw2068 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:23:18] PROBLEM - puppet last run on mw2046 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:23:20] PROBLEM - puppet last run on ms-be2013 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:23:28] PROBLEM - puppet last run on wtp1013 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:23:29] PROBLEM - puppet last run on dbstore1002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:23:39] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:23:40] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:23:40] PROBLEM - puppet last run on es2002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:23:49] PROBLEM - puppet last run on es2005 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:23:58] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:23:59] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:23:59] PROBLEM - puppet last run on elastic1023 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:23:59] PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:00] PROBLEM - puppet last run on mw2167 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:08] PROBLEM - puppet last run on mw2193 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:08] PROBLEM - puppet last run on heze is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:09] PROBLEM - puppet last run on mw1185 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:09] PROBLEM - puppet last run on mw2008 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:10] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:11] !log demon@tin Synchronized .arcconfig: no-op, for completeness (duration: 00m 31s) [00:24:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:24:18] PROBLEM - puppet last run on sca1002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:18] PROBLEM - puppet last run on mw2058 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:18] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:19] PROBLEM - puppet last run on mw1001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:19] PROBLEM - puppet last run on mw2186 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:19] PROBLEM - puppet last run on mw2118 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:19] PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:20] PROBLEM - puppet last run on mw2141 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:20] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:21] PROBLEM - puppet last run on mw2026 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:28] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:29] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:30] PROBLEM - puppet last run on db2050 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:40] PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:41] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:41] PROBLEM - puppet last run on analytics1028 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:41] PROBLEM - puppet last run on mw2032 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:41] PROBLEM - puppet last run on elastic1020 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:48] PROBLEM - puppet last run on mw2057 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:49] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:49] PROBLEM - puppet last run on mw1139 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:58] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:24:59] PROBLEM - puppet last run on db2034 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:00] PROBLEM - puppet last run on mw2209 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:00] PROBLEM - puppet last run on mw2157 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:08] PROBLEM - puppet last run on db1011 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:08] PROBLEM - puppet last run on mw2076 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:08] PROBLEM - puppet last run on mw2020 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:09] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:09] PROBLEM - puppet last run on elastic1021 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:10] PROBLEM - puppet last run on elastic1001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:10] PROBLEM - puppet last run on mw1231 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:18] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:19] PROBLEM - puppet last run on ms-be2020 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:19] PROBLEM - puppet last run on ms-be1019 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:28] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:28] PROBLEM - puppet last run on mw2069 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:29] PROBLEM - puppet last run on mw1228 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:29] PROBLEM - puppet last run on mw2091 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:29] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:30] PROBLEM - puppet last run on mw1093 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:39] PROBLEM - puppet last run on analytics1053 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:39] PROBLEM - puppet last run on dbproxy1008 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:39] PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:49] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:49] PROBLEM - puppet last run on mw2081 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:49] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:49] PROBLEM - puppet last run on analytics1047 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:50] PROBLEM - puppet last run on wtp1010 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:50] PROBLEM - puppet last run on es2009 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:50] PROBLEM - puppet last run on mw1260 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:58] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:59] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:25:59] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:08] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:08] PROBLEM - puppet last run on mw2021 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:09] PROBLEM - puppet last run on mw1222 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:09] PROBLEM - puppet last run on mw2033 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:09] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:18] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:19] PROBLEM - puppet last run on mw2023 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:30] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:30] PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:38] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:39] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:39] PROBLEM - puppet last run on mw2045 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:40] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:40] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:48] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:49] PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:49] PROBLEM - puppet last run on wtp2008 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:50] PROBLEM - puppet last run on mw2126 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:50] PROBLEM - puppet last run on mw2036 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:50] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:59] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:59] PROBLEM - puppet last run on mw2207 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:59] PROBLEM - puppet last run on wtp2017 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:26:59] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:00] PROBLEM - puppet last run on db2044 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:00] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:00] PROBLEM - puppet last run on mw2043 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:08] PROBLEM - puppet last run on subra is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:10] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:10] PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:18] PROBLEM - puppet last run on mw2018 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:19] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:19] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:20] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:20] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:20] PROBLEM - puppet last run on wtp2015 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:20] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:20] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:29] PROBLEM - puppet last run on sca1001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:39] PROBLEM - puppet last run on mw2145 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:39] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:48] PROBLEM - puppet last run on ms-be2021 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:49] PROBLEM - puppet last run on mw2016 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:49] PROBLEM - puppet last run on mw2120 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:49] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:50] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:58] PROBLEM - puppet last run on mw2158 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:27:59] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:08] PROBLEM - puppet last run on mw2129 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:08] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:08] PROBLEM - puppet last run on mw2050 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:09] PROBLEM - puppet last run on wtp2019 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:09] PROBLEM - puppet last run on eventlog2001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:10] PROBLEM - puppet last run on mw2095 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:10] PROBLEM - puppet last run on mw2073 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:10] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:10] PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:19] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:19] PROBLEM - puppet last run on mw1249 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:19] PROBLEM - puppet last run on silver is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:19] PROBLEM - puppet last run on mw2146 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:29] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:29] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:38] PROBLEM - puppet last run on mw1251 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:39] PROBLEM - puppet last run on db1060 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:39] PROBLEM - puppet last run on mw2136 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:48] PROBLEM - puppet last run on mw2024 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:49] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:49] PROBLEM - puppet last run on mw2166 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:49] PROBLEM - puppet last run on mw2077 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:50] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:50] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:50] PROBLEM - puppet last run on db2016 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:58] PROBLEM - puppet last run on analytics1026 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:58] PROBLEM - puppet last run on analytics1045 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:28:58] PROBLEM - puppet last run on mw1076 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:00] PROBLEM - puppet last run on mw2198 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:00] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:00] PROBLEM - puppet last run on analytics1038 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:08] PROBLEM - puppet last run on wtp2004 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:08] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:09] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:19] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:28] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:28] PROBLEM - puppet last run on suhail is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:29] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:39] PROBLEM - puppet last run on dbproxy1003 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:39] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:39] PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:40] PROBLEM - puppet last run on mw1259 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:40] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:48] PROBLEM - puppet last run on mw2138 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:48] PROBLEM - puppet last run on mw2052 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:49] PROBLEM - puppet last run on mw2061 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:49] PROBLEM - puppet last run on ms-be2011 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:58] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:59] PROBLEM - puppet last run on db2049 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:59] PROBLEM - puppet last run on mw2035 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:29:59] PROBLEM - puppet last run on ms-be2015 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:09] PROBLEM - puppet last run on francium is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:18] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:20] PROBLEM - puppet last run on elastic1028 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:28] PROBLEM - puppet last run on graphite2001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:30] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:30] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:39] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:39] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:39] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:39] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:48] PROBLEM - puppet last run on wtp2003 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:49] PROBLEM - puppet last run on mw2191 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:50] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 7.14% of data above the critical threshold [500.0] [00:30:50] PROBLEM - puppet last run on wtp1023 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:50] PROBLEM - puppet last run on mw2132 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:50] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:50] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:58] PROBLEM - puppet last run on mw2012 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:58] PROBLEM - puppet last run on mw2054 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:59] PROBLEM - puppet last run on elastic1011 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:30:59] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:08] PROBLEM - puppet last run on mw2177 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:08] PROBLEM - puppet last run on mw2088 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:09] PROBLEM - puppet last run on db2051 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:09] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:09] PROBLEM - puppet last run on wtp1022 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:18] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:18] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:18] PROBLEM - puppet last run on mw1244 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:19] PROBLEM - puppet last run on mw2125 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:19] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:20] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:20] PROBLEM - puppet last run on mw1258 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:20] PROBLEM - puppet last run on elastic1014 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:28] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:28] PROBLEM - puppet last run on mw1192 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:29] PROBLEM - puppet last run on analytics1050 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:39] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:40] PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:48] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:50] PROBLEM - puppet last run on ms-be1020 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:50] PROBLEM - puppet last run on analytics1054 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:59] PROBLEM - puppet last run on mw2162 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:59] PROBLEM - puppet last run on mw2037 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:31:59] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:08] PROBLEM - puppet last run on elastic1005 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:08] PROBLEM - puppet last run on mw1246 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:09] PROBLEM - puppet last run on mw2072 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:09] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:09] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:09] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:09] PROBLEM - puppet last run on pc1001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:10] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:10] PROBLEM - puppet last run on mw1234 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:18] PROBLEM - puppet last run on mw1252 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:18] PROBLEM - puppet last run on wtp1011 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:19] PROBLEM - puppet last run on mw1156 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:19] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:20] PROBLEM - puppet last run on es2007 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:28] PROBLEM - puppet last run on wtp1019 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:29] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:29] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:30] PROBLEM - puppet last run on mw1257 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:38] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:38] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:39] PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:39] PROBLEM - puppet last run on mw2210 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:39] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:40] PROBLEM - puppet last run on wtp1003 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:48] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:49] PROBLEM - puppet last run on mw2181 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:49] PROBLEM - puppet last run on mw2038 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:49] PROBLEM - puppet last run on pc1003 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:49] PROBLEM - puppet last run on rcs1001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:50] PROBLEM - puppet last run on analytics1042 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:58] PROBLEM - puppet last run on db2035 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:58] PROBLEM - puppet last run on mw2187 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:58] PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:59] PROBLEM - puppet last run on elastic1031 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:59] PROBLEM - puppet last run on mw2051 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:32:59] PROBLEM - puppet last run on elastic1002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:09] PROBLEM - puppet last run on mw1005 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:10] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:10] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:18] PROBLEM - puppet last run on mw2214 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:18] PROBLEM - puppet last run on mw1148 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:19] PROBLEM - puppet last run on mw2155 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:19] PROBLEM - puppet last run on mw1229 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:19] PROBLEM - puppet last run on mw2169 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:19] PROBLEM - puppet last run on mw2028 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:19] PROBLEM - puppet last run on mw1239 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:20] PROBLEM - puppet last run on db2053 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:39] PROBLEM - puppet last run on analytics1055 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:39] PROBLEM - puppet last run on mw1108 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:39] PROBLEM - puppet last run on mw2204 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:39] PROBLEM - puppet last run on mw2034 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:48] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:48] PROBLEM - puppet last run on db1065 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:48] PROBLEM - puppet last run on mw2122 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:49] PROBLEM - puppet last run on mw2014 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:50] PROBLEM - puppet last run on mw1248 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:50] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:50] PROBLEM - puppet last run on wtp2006 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:59] PROBLEM - puppet last run on mw1023 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:33:59] PROBLEM - puppet last run on graphite1001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:09] PROBLEM - puppet last run on analytics1027 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:10] PROBLEM - puppet last run on mw1121 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:10] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:18] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:19] PROBLEM - puppet last run on mw1167 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:19] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:29] PROBLEM - puppet last run on mw1077 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:29] PROBLEM - puppet last run on mw2074 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:29] PROBLEM - puppet last run on mw2071 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:29] PROBLEM - puppet last run on neptunium is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:38] PROBLEM - puppet last run on wtp2018 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:39] PROBLEM - puppet last run on mw2121 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:40] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:48] PROBLEM - puppet last run on mw2108 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:48] PROBLEM - puppet last run on mw2185 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:49] PROBLEM - puppet last run on wtp2013 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:59] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:34:59] PROBLEM - puppet last run on mw2124 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:00] PROBLEM - puppet last run on mw1105 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:00] PROBLEM - puppet last run on dbstore2001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:08] PROBLEM - puppet last run on mw1016 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:08] PROBLEM - puppet last run on dbstore2002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:09] PROBLEM - puppet last run on mw2205 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:09] PROBLEM - puppet last run on mw2175 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:09] PROBLEM - puppet last run on mw1250 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:10] PROBLEM - puppet last run on mw1223 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:10] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:18] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:28] PROBLEM - puppet last run on mw1242 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:29] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:30] PROBLEM - puppet last run on mw2148 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:30] PROBLEM - puppet last run on mw2180 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:38] PROBLEM - puppet last run on mw2159 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:39] PROBLEM - puppet last run on mw2213 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:48] PROBLEM - puppet last run on ms-fe2002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:49] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:50] PROBLEM - puppet last run on mw2154 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:50] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:58] PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:58] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:58] PROBLEM - puppet last run on db2052 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:35:59] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:00] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:00] PROBLEM - puppet last run on mw1224 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:08] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:09] PROBLEM - puppet last run on es2001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:10] PROBLEM - puppet last run on mw1236 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:18] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:18] PROBLEM - puppet last run on elastic1029 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:19] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:19] PROBLEM - puppet last run on mw2115 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:19] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:20] PROBLEM - puppet last run on elastic1030 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:28] PROBLEM - puppet last run on mw2195 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:28] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:29] PROBLEM - puppet last run on mw2001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:29] PROBLEM - puppet last run on wtp2011 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:29] PROBLEM - puppet last run on mw2156 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:29] PROBLEM - puppet last run on elastic1022 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:29] PROBLEM - puppet last run on mw2128 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:38] PROBLEM - puppet last run on mw2104 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:38] PROBLEM - puppet last run on mw2173 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:36:39] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [00:37:12] !log disabled puppet on neon for one hour and killed icinga-wm to silence a tide of puppet failure alerts [00:37:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:45:07] The http 5xx spike is real https://grafana.wikimedia.org/dashboard/db/varnish-http-errors [00:45:14] two spikes [00:46:18] !log rampant puppet failures appear to be the result of a Hash Sum mismatch error during apt-get update; attempting to fix by running sudo -u mirror /usr/local/sbin/update-ubuntu-mirror on carbon. [00:46:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:47:58] Krinkle: they're negligible -- peaked at 506 / min [00:48:28] I mean, worth investigating, but not enough to qualify as an outage, not even a small one [00:49:53] doesn't register in the 30 day view: https://graphite.wikimedia.org/render/?title=HTTP%205xx%20Responses%20-30days&from=-30days&width=1024&height=500&until=now&areaMode=none&hideLegend=false&lineWidth=2&lineMode=connected&target=color(cactiStyle(alias(reqstats.500,%22500%20resp/min%22)),%22red%22)&target=color(cactiStyle(alias(reqstats.5xx,%225xx%20resp/min%22)),%22blue%22) [00:50:41] !log update-ubuntu-mirror on carbon completed successfully ; going to re-enable icinga-wm [00:50:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:51:43] enjoy your courtesy botspam-free irc moment [00:52:04] RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:52:04] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:52:04] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:52:04] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [00:52:08] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [00:52:08] RECOVERY - puppet last run on mw2150 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [00:52:08] RECOVERY - puppet last run on mw1030 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [00:52:19] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [00:52:19] RECOVERY - puppet last run on mw2053 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:52:19] RECOVERY - puppet last run on wtp1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:52:24] ok, maybe it needs another minute [00:52:28] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [00:52:28] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [00:52:28] RECOVERY - puppet last run on elastic1023 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [00:52:29] RECOVERY - puppet last run on mw2048 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [00:52:29] RECOVERY - puppet last run on mw2202 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [00:52:29] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:57:46] andrewbogott: sorry for the super-slow reaction re: svg, what needs to be tested? [01:03:29] andrewbogott: or rather, how can I test it? [01:06:39] 6operations, 10SEO: secure.wikimedia.org entries still showing up in Google search results - https://phabricator.wikimedia.org/T93531#1822461 (10Krenair) [01:09:45] Krenair: Can we just remove it or something? [01:10:00] entirely from the dns/apache config etc. [01:10:01] ? [01:10:02] I have access to some webmaster tools stuff.. Can ask ops to give me secure.wm.o too [01:10:08] I'm meaning from Google [01:10:20] yeah [01:10:27] I was expecting them to do something like that [01:10:45] But I opened it 8 months ago [01:10:54] ran into it again just now [01:11:15] heh [01:11:16] we also have interwikis in google search results, like https://meta.wikimedia.org/wiki/w:Heartbleed [01:11:27] but that's an entirely different issue to secure.wm.o. [01:11:39] Google don't tend to be the most helpful [01:11:48] Maybe we can ask Denny nicely for some help [01:12:17] might not even need to go that far, let's find out what can be done by people with webmaster tools access first [01:13:35] mutante: Any chance you could add secure.wm.o to my webmaster tools access please? [01:13:39] Step 1 ^ [01:13:50] (If we've even claimed it...) [01:14:35] Might need to file a ticket for ops [01:16:58] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:16:59] RECOVERY - puppet last run on mw1107 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:17:08] RECOVERY - puppet last run on mw2079 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [01:17:09] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [01:17:09] RECOVERY - puppet last run on mw2019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:17:10] RECOVERY - puppet last run on mw2143 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [01:17:18] RECOVERY - puppet last run on mw1253 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:17:19] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [01:17:19] RECOVERY - puppet last run on wtp2010 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [01:17:20] RECOVERY - puppet last run on mw1235 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:17:20] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [01:17:28] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:17:29] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [01:17:29] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [01:17:30] RECOVERY - puppet last run on mw2212 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [01:17:30] RECOVERY - puppet last run on mw2176 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [01:17:30] RECOVERY - puppet last run on mw2003 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [01:17:38] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [01:17:39] RECOVERY - puppet last run on mw2096 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [01:17:39] RECOVERY - puppet last run on mw2113 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [01:17:39] RECOVERY - puppet last run on db2029 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [01:17:49] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [01:17:49] RECOVERY - puppet last run on ms-fe2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:17:49] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:17:58] RECOVERY - puppet last run on mw2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:17:59] RECOVERY - puppet last run on ms-be2009 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [01:17:59] RECOVERY - puppet last run on mw2087 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:18:09] RECOVERY - puppet last run on db2038 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [01:18:18] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:18:18] RECOVERY - puppet last run on mw2070 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [01:18:19] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [01:18:19] RECOVERY - puppet last run on mw2134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:18:20] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [01:18:20] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:18:29] RECOVERY - puppet last run on mw2111 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [01:18:29] RECOVERY - puppet last run on mw2055 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [01:18:29] RECOVERY - puppet last run on mw2127 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:18:30] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [01:18:38] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [01:18:38] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [01:18:39] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [01:18:39] RECOVERY - puppet last run on mw2142 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [01:18:39] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:18:40] RECOVERY - puppet last run on mw2196 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:18:40] RECOVERY - puppet last run on wtp2016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:18:40] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:18:48] RECOVERY - puppet last run on mw2047 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [01:18:49] RECOVERY - puppet last run on nembus is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:18:49] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [01:18:58] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [01:18:59] RECOVERY - puppet last run on analytics1051 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [01:18:59] RECOVERY - puppet last run on mw2049 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [01:19:09] RECOVERY - puppet last run on mw2123 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:09] RECOVERY - puppet last run on elastic1006 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [01:19:09] RECOVERY - puppet last run on mw2067 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:09] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [01:19:10] RECOVERY - puppet last run on mw2182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:10] RECOVERY - puppet last run on mw2101 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [01:19:10] RECOVERY - puppet last run on mw2131 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [01:19:10] RECOVERY - puppet last run on mw2039 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:18] RECOVERY - puppet last run on mw2056 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [01:19:18] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:18] RECOVERY - puppet last run on db2047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:19] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:20] RECOVERY - puppet last run on mw2090 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:20] RECOVERY - puppet last run on mw1230 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:20] RECOVERY - puppet last run on mw2062 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [01:19:20] RECOVERY - puppet last run on mw1238 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [01:19:20] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [01:19:28] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [01:19:29] RECOVERY - puppet last run on analytics1002 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [01:19:29] RECOVERY - puppet last run on mw2092 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:29] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [01:19:39] RECOVERY - puppet last run on mw2084 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [01:19:39] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:40] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:49] RECOVERY - puppet last run on install2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:49] RECOVERY - puppet last run on analytics1056 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:50] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [01:19:59] RECOVERY - puppet last run on mw2110 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:59] RECOVERY - puppet last run on ms-be1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:59] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:20:08] RECOVERY - puppet last run on mw2172 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [01:20:08] RECOVERY - puppet last run on mw2152 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [01:20:08] RECOVERY - puppet last run on mw2168 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [01:20:09] RECOVERY - puppet last run on mw2030 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:20:09] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:20:09] RECOVERY - puppet last run on mw2085 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:20:10] RECOVERY - puppet last run on mw2174 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [01:20:10] RECOVERY - puppet last run on mw2130 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [01:20:18] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:20:18] RECOVERY - puppet last run on mw2098 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [01:20:19] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [01:20:19] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:20:20] RECOVERY - puppet last run on mw1049 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [01:20:29] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:20:29] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:20:38] RECOVERY - puppet last run on mw2203 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:21:08] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:21:09] RECOVERY - puppet last run on ms-be2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:21:19] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:21:29] RECOVERY - puppet last run on mw2200 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:21:48] RECOVERY - puppet last run on mira is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:39:57] (03PS1) 10Yuvipanda: labs: Move vmbuilder role into module [puppet] - 10https://gerrit.wikimedia.org/r/254621 [01:40:21] my turn to break things now! [01:42:28] (03CR) 10Yuvipanda: [C: 032] labs: Move vmbuilder role into module [puppet] - 10https://gerrit.wikimedia.org/r/254621 (owner: 10Yuvipanda) [01:47:20] RECOVERY - puppet last run on mw2093 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [01:47:27] (03PS4) 10Dzahn: racktables: remove role from magnesium [puppet] - 10https://gerrit.wikimedia.org/r/254588 (https://phabricator.wikimedia.org/T105555) [01:49:13] also one last merge here :p [01:49:57] which grrit-wm did not report yet [01:50:11] (03PS1) 10Yuvipanda: labs: Move dns / dnsrecursor roles to role module [puppet] - 10https://gerrit.wikimedia.org/r/254622 [01:51:44] (03PS2) 10Yuvipanda: labs: Move dns / dnsrecursor roles to role module [puppet] - 10https://gerrit.wikimedia.org/r/254622 [01:53:48] (03CR) 10Yuvipanda: [C: 032] labs: Move dns / dnsrecursor roles to role module [puppet] - 10https://gerrit.wikimedia.org/r/254622 (owner: 10Yuvipanda) [01:55:04] 6operations, 10SEO: secure.wikimedia.org entries still showing up in Google search results - https://phabricator.wikimedia.org/T93531#1822489 (10Dzahn) I gave Reedy (restricted) access to secure.wm.org so he can see stats. (using the noc@ account and delegating from there to sreed@) [02:09:30] PROBLEM - Check correctness of the icinga configuration on neon is CRITICAL: Icinga configuration contains errors [02:12:17] (03PS1) 10Yuvipanda: Move debdeploy role into autolayout compatible names [puppet] - 10https://gerrit.wikimedia.org/r/254625 [02:12:19] (03PS1) 10Yuvipanda: extdist: Make role name autolayout compatible [puppet] - 10https://gerrit.wikimedia.org/r/254626 [02:12:21] (03PS1) 10Yuvipanda: Make redisdb role conform to autolayout [puppet] - 10https://gerrit.wikimedia.org/r/254627 [02:12:28] PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [02:12:48] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [02:20:33] !log l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 18s) [02:20:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:21:26] that was nice and quick [02:27:48] Reedy: Probably means i18n bot didn't run today or something. [02:27:56] lo [02:27:57] l [02:32:45] Looks like it did on core [02:33:01] * James_F shrugs. [02:40:39] Failed to sync-dir 'php-1.27.0-wmf.7/cache/l10n' [02:41:11] Oh, that's a lie [02:41:23] That's from the mira master cosync fail I think [02:42:14] ^[[32m02:15:52 Started rsync master^[[0m [02:42:14] sudo: a password is required [02:42:14] ^[[32m02:15:52 Finished rsync master (duration: 00m 00s)^[[0m [02:43:56] 6operations: ssh connection to wikimedia cluster is slooooooow for me - https://phabricator.wikimedia.org/T119275#1822544 (10Reedy) 3NEW [02:56:18] 6operations: ssh connection to wikimedia cluster is slooooooow for me - https://phabricator.wikimedia.org/T119275#1822552 (10hoo) What bastion are you trying to connect to? Have you tried stracing it to see what exactly blocks? [03:05:30] 6operations: ssh connection to wikimedia cluster is slooooooow for me - https://phabricator.wikimedia.org/T119275#1822554 (10Reedy) I've tried bast1001.wikimedia.org and hooft.esams.wikimedia.org Don't seem to have this problem to other (non WMF) servers [03:19:42] !log SecurePoll arbcomlist.php run for enwiki (requested by Jamesofur) complete. [03:19:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [03:53:16] 6operations, 10MediaWiki-Special-pages: Query pages that use categorylinks tables have not been updated since nov 1 - https://phabricator.wikimedia.org/T119276#1822592 (10Bawolff) 3NEW [04:09:51] PROBLEM - Incoming network saturation on labstore1003 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [100000000.0] [04:32:21] RECOVERY - Incoming network saturation on labstore1003 is OK: OK: Less than 10.00% above the threshold [75000000.0] [04:53:59] RECOVERY - Router interfaces on cr2-codfw is OK: OK: host 208.80.153.193, interfaces up: 120, down: 0, dormant: 0, excluded: 0, unused: 0 [04:54:19] RECOVERY - Router interfaces on cr1-eqord is OK: OK: host 208.80.154.198, interfaces up: 37, down: 0, dormant: 0, excluded: 0, unused: 0 [05:01:11] (03PS3) 10Dereckson: Enable rollbacker and patroller group at maiwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/254443 (https://phabricator.wikimedia.org/T118934) (owner: 10Luke081515) [05:01:47] (03CR) 10Dereckson: [C: 031] Enable rollbacker and patroller group at maiwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/254443 (https://phabricator.wikimedia.org/T118934) (owner: 10Luke081515) [05:03:05] (03PS2) 10Yuvipanda: Make redisdb role conform to autolayout [puppet] - 10https://gerrit.wikimedia.org/r/254627 [05:05:59] (03PS2) 10Yuvipanda: Move debdeploy role into autolayout compatible names [puppet] - 10https://gerrit.wikimedia.org/r/254625 [06:31:29] PROBLEM - puppet last run on mw2207 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:39] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 3 failures [06:31:58] PROBLEM - puppet last run on db2056 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:18] PROBLEM - puppet last run on mw2158 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:19] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:28] PROBLEM - puppet last run on mw2129 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:49] PROBLEM - puppet last run on mw2073 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:19] PROBLEM - puppet last run on mw2126 is CRITICAL: CRITICAL: Puppet has 2 failures [06:34:08] PROBLEM - puppet last run on mw2052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:59] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:59] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:56:18] RECOVERY - puppet last run on db2056 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:56:38] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:56:40] RECOVERY - puppet last run on mw2129 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:57:08] RECOVERY - puppet last run on mw2073 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:57:38] RECOVERY - puppet last run on mw2126 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:57:40] RECOVERY - puppet last run on mw2207 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:58:19] RECOVERY - puppet last run on mw2052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:28] RECOVERY - puppet last run on mw2158 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:08:09] PROBLEM - Hadoop NodeManager on analytics1048 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [07:12:20] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:13:48] RECOVERY - Hadoop NodeManager on analytics1048 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [08:00:00] PROBLEM - puppet last run on mc1017 is CRITICAL: CRITICAL: Puppet has 1 failures [08:26:19] RECOVERY - puppet last run on mc1017 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [12:44:36] 6operations, 6Commons, 10MediaWiki-File-management, 6Multimedia, 10Traffic: Image cache issue when 'over-writing' an image on commons - https://phabricator.wikimedia.org/T119038#1822923 (10Steinsplitter) Moor complains here: https://commons.wikimedia.org/wiki/Commons:Forum#Irgendwas_spinnt_.28Cache.29 (... [13:38:46] Steinsplitter: more* [13:38:47] :P [13:39:48] uhoh, verry ugly typo. thanks Reedy :-D [14:02:20] (03PS1) 10Bene: Use custom generator for mobile search on Wikibase [mediawiki-config] - 10https://gerrit.wikimedia.org/r/254645 (https://phabricator.wikimedia.org/T85368) [14:49:20] 6operations, 6Commons, 10Wikimedia-Media-storage: Cannot delete file: - https://phabricator.wikimedia.org/T119301#1823021 (10Steinsplitter) 3NEW [14:50:08] 6operations, 6Commons, 10Wikimedia-Media-storage: Cannot delete file: File:JGY5201.jpg - https://phabricator.wikimedia.org/T119301#1823032 (10Steinsplitter) [14:57:17] 6operations, 6Commons, 10MediaWiki-File-management, 6Multimedia, 10Traffic: Image cache issue when 'over-writing' an image on commons - https://phabricator.wikimedia.org/T119038#1823034 (10BBlack) >>! In T119038#1820653, @Bawolff wrote: > the vhtcpd stats look weird to me ( http://ganglia.wikimedia.org/l... [15:00:39] PROBLEM - puppet last run on mw1228 is CRITICAL: CRITICAL: Puppet has 1 failures [15:16:49] PROBLEM - High load average on labstore1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [24.0] [15:20:30] RECOVERY - High load average on labstore1001 is OK: OK: Less than 50.00% above the threshold [16.0] [15:27:00] RECOVERY - puppet last run on mw1228 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:42:29] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: puppet fail [16:56:13] (03PS1) 10Addshore: Add addshore.com to planet.wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/254653 [17:09:09] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [17:11:02] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 2 below the confidence bounds [17:19:52] 6operations, 6Commons, 10MediaWiki-File-management, 6Multimedia, 10Traffic: Image cache issue when 'over-writing' an image on commons - https://phabricator.wikimedia.org/T119038#1823125 (10Bawolff) >>! In T119038#1823034, @BBlack wrote: >>>! In T119038#1820653, @Bawolff wrote: >> the vhtcpd stats look we... [17:21:58] PROBLEM - Ensure mysql credential creation for tools users is running on labstore1001 is CRITICAL: CRITICAL - Expecting active but unit create-dbusers is failed [17:36:48] PROBLEM - Disk space on restbase1008 is CRITICAL: DISK CRITICAL - free space: /var 70350 MB (3% inode=99%) [17:48:29] RECOVERY - Ensure mysql credential creation for tools users is running on labstore1001 is OK: OK - create-dbusers is active [17:48:49] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected [18:20:18] PROBLEM - Disk space on restbase1008 is CRITICAL: DISK CRITICAL - free space: /var 70068 MB (3% inode=99%) [18:26:23] 10Ops-Access-Requests, 6operations: Requesting access to stat1002 for ejegg - https://phabricator.wikimedia.org/T118320#1823145 (10Ejegg) a:5Wwes>3RobH Thanks @Wwes! @RobH, my manager is out of town for a while - I hope the approval of her boss is a suitable substitute. [19:05:48] PROBLEM - Disk space on restbase1008 is CRITICAL: DISK CRITICAL - free space: /var 70477 MB (3% inode=99%) [19:05:59] waaat? ^ [19:06:09] gwicke: ^ [19:06:20] it's at 1.6T [19:06:38] and 7-9 only have two ssds [19:07:37] nodetool compactionstats shows one involving 2318295263525 bytes 97.27% done [19:07:52] ah kk [19:07:56] another with 926732473498 bytes at 97.56% [19:08:59] if one of those compactions finish before running out it'll be fine [19:09:35] yeah i think we're good [19:12:05] the bigger of those two compactions takes up 365G on disk right now, and there is enough space for 3 more % of that [19:13:17] looking forward to having those peaks be a lot smaller with multiple instances [19:18:52] you have about 68 minutes of runway before the the disk is full, assuming the rate at which it is writing is stable [19:20:16] 64328488K available, filling up at a rate of 974096K / min [19:23:57] the smaller of the two is at 98.23% [19:24:16] yeah, progressing at 0.03% / min [19:24:20] which puts it at 60 minutes [19:24:43] so i think you're in luck, but it's tight [19:27:42] it's closer to 0.04 on the smaller one [19:28:52] 6operations, 10MediaWiki-Special-pages: Query pages that use categorylinks tables have not been updated since nov 1 - https://phabricator.wikimedia.org/T119276#1823168 (10Umherirrender) Maybe handled under T113721 [19:52:45] 6operations, 6Commons, 10MediaWiki-File-management, 6Multimedia, 10Traffic: Image cache issue when 'over-writing' an image on commons - https://phabricator.wikimedia.org/T119038#1823204 (10Ankry) I observed this effect today on two images from multi-page DjVu file: https://commons.wikimedia.org/wiki/File... [20:08:18] RECOVERY - Disk space on restbase1008 is OK: DISK OK [20:30:00] PROBLEM - puppet last run on cp3048 is CRITICAL: CRITICAL: puppet fail [20:56:39] RECOVERY - puppet last run on cp3048 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [21:40:19] PROBLEM - puppet last run on cp3038 is CRITICAL: CRITICAL: puppet fail [21:40:29] PROBLEM - Ensure legal html en.m.wp on en.m.wikipedia.org is CRITICAL: a href=//wikimediafoundation.org/wiki/Privacy_policy title=wikimedia:Privacy policyPrivacy/a html not found [21:40:49] PROBLEM - Ensure legal html en.wp on en.wikipedia.org is CRITICAL: a href=//wikimediafoundation.org/wiki/Privacy_policy title=wikimedia:Privacy policyPrivacy policy/a html not found [21:43:55] Sorry? [21:44:43] (03PS1) 10Dereckson: Namespace configuration for ur.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/254665 (https://phabricator.wikimedia.org/T119308) [21:45:09] PROBLEM - Ensure mysql credential creation for tools users is running on labstore1001 is CRITICAL: CRITICAL - Expecting active but unit create-dbusers is failed [21:47:08] RECOVERY - Ensure mysql credential creation for tools users is running on labstore1001 is OK: OK - create-dbusers is active [22:06:48] RECOVERY - puppet last run on cp3038 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [22:12:53] (03CR) 10Luke081515: Namespace configuration for ur.wikipedia (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/254665 (https://phabricator.wikimedia.org/T119308) (owner: 10Dereckson) [22:33:00] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: puppet fail [22:58:21] Reedy: weird icinga check indeed [23:03:19] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures