[01:30:55] PROBLEM - restbase endpoints health on restbase-dev1006 is CRITICAL: /en.wikipedia.org/v1/data/citation/{format}/{query} (Get citation for Darth Vader) timed out before a response was received [01:32:01] RECOVERY - restbase endpoints health on restbase-dev1006 is OK: All endpoints are healthy [01:32:17] PROBLEM - citoid endpoints health on scb1001 is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received [01:35:55] RECOVERY - citoid endpoints health on scb1001 is OK: All endpoints are healthy [02:33:26] 10Operations, 10MediaWiki-extensions-CentralAuth, 10Wikimedia-General-or-Unknown: Rename global users with invalid characters on their usernames - https://phabricator.wikimedia.org/T160296 (10Base) Someone has to actually run rename queries, I am not sure if Operations is the right choice but hopefully some... [03:04:49] (03PS1) 10Tulsi Bhagat: Configure $wgNamespaceAliases for yue.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481578 [03:10:55] (03PS2) 10Tulsi Bhagat: Configure $wgNamespaceAliases for yue.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481578 (https://phabricator.wikimedia.org/T212678) [03:13:54] (03CR) 10Tulsi Bhagat: "Requires `namespaceDupes.php --wiki=yuewiktionary --fix` after deployment." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481578 (https://phabricator.wikimedia.org/T212678) (owner: 10Tulsi Bhagat) [03:19:38] (03PS1) 10Tulsi Bhagat: Configure $wgAddGroups, $wgRemoveGroups and $wgImportSources for ur.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481579 [03:26:08] (03PS2) 10Tulsi Bhagat: Configure $wgAddGroups, $wgRemoveGroups and $wgImportSources for ur.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481579 (https://phabricator.wikimedia.org/T212612) [03:35:13] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 959.49 seconds [04:16:39] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 195.28 seconds [10:07:25] 10Operations, 10Continuous-Integration-Infrastructure (shipyard), 10Patch-For-Review: wikimedia-jessie & wikimedia-stretch docker images don't have deb-src set for apt.wikimedia.org - https://phabricator.wikimedia.org/T179354 (10hashar) 05Open→03Declined The purpose of `apt-get build-dep hhvm` was to pro... [12:52:41] (03PS1) 10Alexandros Kosiaris: Revoke ladsgroup access due to lost laptop [puppet] - 10https://gerrit.wikimedia.org/r/481603 [12:53:07] d'oh :( [12:55:47] (03CR) 10Alexandros Kosiaris: [C: 03+2] Revoke ladsgroup access due to lost laptop [puppet] - 10https://gerrit.wikimedia.org/r/481603 (owner: 10Alexandros Kosiaris) [12:56:36] 10Operations, 10Continuous-Integration-Infrastructure, 10Traffic: trafficserver debian-glue builds failing on integration-slave-jessie-1001: No space left on device - https://phabricator.wikimedia.org/T209703 (10hashar) 05Open→03Resolved a:03hashar Indeed, the debian glue jobs run on legacy permanent s... [13:00:57] PROBLEM - puppet last run on snapshot1009 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:02:21] PROBLEM - puppet last run on mw2212 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:02:37] PROBLEM - puppet last run on ores2007 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:03:23] PROBLEM - puppet last run on mw2280 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:03:37] PROBLEM - puppet last run on mw1346 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:03:37] PROBLEM - puppet last run on mw1238 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:03:41] PROBLEM - puppet last run on mw1309 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:03:47] PROBLEM - puppet last run on mw2172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:03:47] PROBLEM - puppet last run on mw2156 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:03:47] PROBLEM - puppet last run on mw2141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:03:51] PROBLEM - puppet last run on mw1311 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:03:51] PROBLEM - puppet last run on mw2277 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:04:47] PROBLEM - puppet last run on mw1322 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:04:47] PROBLEM - puppet last run on mw1282 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:04:59] PROBLEM - puppet last run on ores2006 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:05:03] PROBLEM - puppet last run on mw2225 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:03] PROBLEM - puppet last run on mw2255 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:03] PROBLEM - puppet last run on mw1233 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:19] PROBLEM - puppet last run on mw2265 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:19] PROBLEM - puppet last run on mw2256 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:19] PROBLEM - puppet last run on mw2274 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:19] PROBLEM - puppet last run on mw2288 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:23] PROBLEM - puppet last run on deploy2001 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members],Exec[deploy-service_ensure_members] [13:05:23] PROBLEM - puppet last run on mw2252 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:25] PROBLEM - puppet last run on mw2234 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:33] PROBLEM - puppet last run on mw1232 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:33] PROBLEM - puppet last run on mw2152 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:33] PROBLEM - puppet last run on mw2157 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:33] PROBLEM - puppet last run on mw2137 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:47] PROBLEM - puppet last run on mw1275 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:47] PROBLEM - puppet last run on mw1312 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:47] PROBLEM - puppet last run on mw1262 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:57] PROBLEM - puppet last run on mw2144 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:57] PROBLEM - puppet last run on mw2201 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:05:57] PROBLEM - puppet last run on mw2202 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:06:03] PROBLEM - puppet last run on ores1008 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:06:03] PROBLEM - puppet last run on mw1263 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:06:21] PROBLEM - puppet last run on mw1295 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:06:21] PROBLEM - puppet last run on mw2240 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:06:21] PROBLEM - puppet last run on mw2287 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:06:45] PROBLEM - puppet last run on mw2246 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:06:45] PROBLEM - puppet last run on mw2223 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:06:45] PROBLEM - puppet last run on mw2162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:07:19] PROBLEM - puppet last run on mw1242 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:07:27] PROBLEM - puppet last run on mwlog2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:07:27] PROBLEM - puppet last run on ores1004 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:07:29] PROBLEM - puppet last run on ores2002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:07:29] PROBLEM - puppet last run on mw2176 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:07:39] PROBLEM - puppet last run on mw1294 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:07:43] PROBLEM - puppet last run on mw1310 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:07:43] PROBLEM - puppet last run on mw2169 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:07:49] PROBLEM - puppet last run on mw2266 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:08:03] PROBLEM - puppet last run on an-master1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[analytics-privatedata-users_ensure_members] [13:08:03] PROBLEM - puppet last run on mw1276 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:08:15] PROBLEM - puppet last run on mw2253 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:08:20] 10Operations, 10Traffic, 10Continuous-Integration-Infrastructure (Slipway), 10Patch-For-Review, 10User-ArielGlenn: CI jobs for authdns linting need to run on Stretch - https://phabricator.wikimedia.org/T205439 (10hashar) [13:08:21] PROBLEM - puppet last run on mw1345 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:08:33] PROBLEM - puppet last run on mw1274 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:08:37] PROBLEM - puppet last run on ores2008 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:08:37] PROBLEM - puppet last run on mw2251 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:08:39] PROBLEM - puppet last run on mw2290 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:08:43] PROBLEM - puppet last run on mw2262 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:08:45] PROBLEM - puppet last run on mw2286 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:08:45] PROBLEM - puppet last run on labtestweb2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:08:53] PROBLEM - puppet last run on mw1332 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:08:53] PROBLEM - puppet last run on mw1287 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:08:53] PROBLEM - puppet last run on mw2231 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:08:53] PROBLEM - puppet last run on mw1284 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:08:53] PROBLEM - puppet last run on mw2233 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:08:55] PROBLEM - puppet last run on mw2207 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:09:01] PROBLEM - puppet last run on mw2138 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:09:03] PROBLEM - puppet last run on labweb1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:09:03] PROBLEM - puppet last run on mw1339 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:09:31] PROBLEM - puppet last run on mw2218 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:09:31] PROBLEM - puppet last run on mw2220 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:09:33] PROBLEM - puppet last run on mw2177 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:09:33] PROBLEM - puppet last run on mw2208 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:09:37] PROBLEM - puppet last run on mw2186 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:09:47] PROBLEM - puppet last run on mw1255 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:09:47] PROBLEM - puppet last run on mw2146 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:09:53] PROBLEM - puppet last run on mw1222 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:09:53] PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:09:55] PROBLEM - puppet last run on mw1293 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:10:03] PROBLEM - puppet last run on mw1245 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:10:08] 10Operations, 10DNS, 10Traffic, 10Continuous-Integration-Config, 10User-jijiki: Add jenkins syntax verification on operations/dns - https://phabricator.wikimedia.org/T205579 (10hashar) The verifications done for operations/dns have been overhauled as part of T205439 It is working fine now and there is ev... [13:10:41] PROBLEM - puppet last run on mw2250 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:10:49] PROBLEM - puppet last run on mw2185 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:11:03] PROBLEM - puppet last run on mw1279 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:11:07] PROBLEM - puppet last run on mw2236 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:11:13] PROBLEM - puppet last run on mw2214 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:11:29] PROBLEM - puppet last run on ores2009 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:11:33] PROBLEM - puppet last run on mw1267 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:11:35] PROBLEM - puppet last run on mw1331 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:11:37] PROBLEM - puppet last run on mw2279 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:11:43] PROBLEM - puppet last run on snapshot1007 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:11:53] PROBLEM - puppet last run on ores2003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:11:59] PROBLEM - puppet last run on mw2259 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:11:59] PROBLEM - puppet last run on mw2271 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:12:01] PROBLEM - puppet last run on mw2283 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:12:01] PROBLEM - puppet last run on mw2160 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:12:01] PROBLEM - puppet last run on mw2203 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:12:05] PROBLEM - puppet last run on mw2153 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:12:05] PROBLEM - puppet last run on mw2174 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:12:08] akosiaris: ping ^ [13:12:39] akosiaris: ^could this be related to ladsgroup? [13:12:46] probably [13:12:47] Exec[deployment_ensure_members] [13:12:57] I'm thinking that as well [13:13:00] and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/481603/ [13:13:09] PROBLEM - puppet last run on mw2238 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:13:23] PROBLEM - puppet last run on mw1251 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:13:41] PROBLEM - puppet last run on mw1224 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:13:41] PROBLEM - puppet last run on mw1225 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:13:45] PROBLEM - puppet last run on mw1221 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:13:59] PROBLEM - puppet last run on mw2191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:14:13] PROBLEM - puppet last run on mw2155 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:14:17] PROBLEM - puppet last run on ores1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:14:19] PROBLEM - puppet last run on mw2193 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:14:19] PROBLEM - puppet last run on mw2189 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:14:19] PROBLEM - puppet last run on mw2206 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:14:21] poking him by text message [13:14:31] Ok. Thanks [13:14:33] PROBLEM - puppet last run on mw2226 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:14:59] PROBLEM - puppet last run on mw1223 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:14:59] PROBLEM - puppet last run on mw1231 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:05] PROBLEM - puppet last run on mw1277 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:05] PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:07] PROBLEM - puppet last run on mw1344 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:11] PROBLEM - puppet last run on mw1285 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:15] PROBLEM - puppet last run on mw1315 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:17] PROBLEM - puppet last run on mw1321 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:17] PROBLEM - puppet last run on mw1330 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:19] PROBLEM - puppet last run on ores1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:15:29] PROBLEM - puppet last run on mw2275 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:33] PROBLEM - puppet last run on mw1313 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:33] PROBLEM - puppet last run on mw1234 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:33] PROBLEM - puppet last run on mw1228 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:35] PROBLEM - puppet last run on mw1320 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:37] PROBLEM - puppet last run on mw2273 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:37] PROBLEM - puppet last run on mw2254 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:45] PROBLEM - puppet last run on ores2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:15:45] PROBLEM - puppet last run on ores2005 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:15:45] PROBLEM - puppet last run on mw2278 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:45] PROBLEM - puppet last run on mw2282 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:45] PROBLEM - puppet last run on mw2269 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:49] PROBLEM - puppet last run on mw2230 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:53] PROBLEM - puppet last run on mw2249 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:59] PROBLEM - puppet last run on mw1256 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:15:59] PROBLEM - puppet last run on mw2145 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:16:09] PROBLEM - puppet last run on stat1006 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[statistics-users_ensure_members] [13:16:19] PROBLEM - puppet last run on mw2215 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:16:21] PROBLEM - puppet last run on mw1299 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:16:29] PROBLEM - puppet last run on mw1269 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:16:41] PROBLEM - puppet last run on notebook1004 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[analytics-privatedata-users_ensure_members] [13:16:41] PROBLEM - puppet last run on mw1268 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:16:45] PROBLEM - puppet last run on snapshot1005 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:16:49] PROBLEM - puppet last run on mw2228 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:16:49] PROBLEM - puppet last run on mw2190 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:16:53] PROBLEM - puppet last run on mw1243 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:17:09] PROBLEM - puppet last run on stat1004 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[analytics-privatedata-users_ensure_members] [13:17:11] PROBLEM - puppet last run on mw2235 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:17:11] PROBLEM - puppet last run on mw2178 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:17:15] PROBLEM - puppet last run on mw1301 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:17:19] PROBLEM - puppet last run on ores1006 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:17:19] PROBLEM - puppet last run on mw1273 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:17:45] PROBLEM - puppet last run on mwdebug1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:17:45] PROBLEM - puppet last run on mw1240 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:17:53] PROBLEM - puppet last run on mw1333 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:17:55] PROBLEM - puppet last run on mw2194 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:17:59] PROBLEM - puppet last run on mw2143 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:18:07] PROBLEM - puppet last run on mw1286 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:18:11] PROBLEM - puppet last run on mw2164 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:18:11] PROBLEM - puppet last run on mw2211 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:18:32] hashar: seems we have to remove ladsgroup from deployment if we are making it absent [13:18:43] PROBLEM - puppet last run on mw2243 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:18:51] PROBLEM - puppet last run on stat1007 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[analytics-privatedata-users_ensure_members] [13:18:53] PROBLEM - puppet last run on mw1254 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:18:57] PROBLEM - puppet last run on mw1271 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:19:05] I don't have priv to revert the change. [13:19:11] PROBLEM - puppet last run on mw2224 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:19:11] PROBLEM - puppet last run on mw2159 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:19:15] PROBLEM - puppet last run on mw1241 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:19:21] PROBLEM - puppet last run on snapshot1006 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:19:21] PROBLEM - puppet last run on mw1296 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:19:21] PROBLEM - puppet last run on mw2227 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:19:21] PROBLEM - puppet last run on mw1290 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:19:21] PROBLEM - puppet last run on mw1257 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:19:21] PROBLEM - puppet last run on mw2192 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:19:25] PROBLEM - puppet last run on mw2167 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:19:25] PROBLEM - puppet last run on mw2200 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:19:31] PROBLEM - puppet last run on mw2209 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:19:45] PROBLEM - puppet last run on mw2263 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:19:55] PROBLEM - puppet last run on mw2219 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:20:25] PROBLEM - puppet last run on mw1281 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:20:27] PROBLEM - puppet last run on mw1328 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:20:27] PROBLEM - puppet last run on mw1329 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:20:29] PROBLEM - puppet last run on mw1337 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:20:43] PROBLEM - puppet last run on mw2268 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:20:43] PROBLEM - puppet last run on mw2272 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:20:45] PROBLEM - puppet last run on mw1249 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:20:51] PROBLEM - puppet last run on mw2270 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:20:59] PROBLEM - puppet last run on mw2264 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:21:13] PROBLEM - puppet last run on mw2198 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:21:13] PROBLEM - puppet last run on mw2199 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:21:21] PROBLEM - puppet last run on mw1302 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:21:21] PROBLEM - puppet last run on mw1227 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:21:27] PROBLEM - puppet last run on mw2261 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:21:31] PROBLEM - puppet last run on mw1335 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:21:31] PROBLEM - puppet last run on mw2232 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:21:35] PROBLEM - puppet last run on mw2163 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:21:39] (03PS1) 10Mathew.onipe: Revert "Revoke ladsgroup access due to lost laptop" [puppet] - 10https://gerrit.wikimedia.org/r/481604 [13:21:41] PROBLEM - puppet last run on mw1324 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:21:41] PROBLEM - puppet last run on ores1009 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:21:43] PROBLEM - puppet last run on mw1306 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:22:01] PROBLEM - puppet last run on mw1244 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:22:07] PROBLEM - puppet last run on mw1261 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:22:19] PROBLEM - puppet last run on mw1338 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:22:23] PROBLEM - puppet last run on mw2237 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:22:23] PROBLEM - puppet last run on mw2216 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:22:25] PROBLEM - puppet last run on mw2158 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:22:27] PROBLEM - puppet last run on notebook1003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[analytics-privatedata-users_ensure_members] [13:22:27] PROBLEM - puppet last run on ores1005 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:22:31] PROBLEM - puppet last run on snapshot1008 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:22:33] PROBLEM - puppet last run on mw1304 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:22:39] PROBLEM - puppet last run on mw1266 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:22:42] probably the way to do it is to remove the user from the groups listed higher up, when you set a user to absent. [13:22:49] PROBLEM - puppet last run on mwmaint1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:23:01] I reverted the change already.. [13:23:07] PROBLEM - puppet last run on mw1336 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:23:07] PROBLEM - puppet last run on mw1272 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:23:07] PROBLEM - puppet last run on mw1270 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:23:07] PROBLEM - puppet last run on mw1250 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:23:13] PROBLEM - puppet last run on mw2180 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:23:19] PROBLEM - puppet last run on mw1264 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:23:23] PROBLEM - puppet last run on mw2187 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:23:23] PROBLEM - puppet last run on mw2142 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:23:31] PROBLEM - puppet last run on mw2260 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:23:31] PROBLEM - puppet last run on mw2276 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:23:31] PROBLEM - puppet last run on mwdebug2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:23:38] yeah fixing [13:24:13] PROBLEM - puppet last run on mw1325 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:24:13] PROBLEM - puppet last run on mw1343 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:24:13] PROBLEM - puppet last run on mw1342 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:24:23] PROBLEM - puppet last run on mw2210 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:24:23] PROBLEM - puppet last run on mw2139 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:24:29] PROBLEM - puppet last run on mw1258 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:24:29] PROBLEM - puppet last run on mw1334 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:24:29] PROBLEM - puppet last run on mw1341 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:24:39] PROBLEM - puppet last run on mw2205 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:24:45] PROBLEM - puppet last run on mw2171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:25:15] PROBLEM - puppet last run on mw2184 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:25:23] PROBLEM - puppet last run on mw2175 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:25:23] PROBLEM - puppet last run on mw2204 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:25:41] PROBLEM - puppet last run on mw1317 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:25:41] PROBLEM - puppet last run on mw1347 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:25:47] PROBLEM - puppet last run on labweb1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:25:59] PROBLEM - puppet last run on mwdebug1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:26:01] PROBLEM - puppet last run on mwmaint2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:26:05] PROBLEM - puppet last run on mw2289 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:26:13] PROBLEM - puppet last run on mw2284 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:26:17] PROBLEM - puppet last run on mw2140 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:26:23] PROBLEM - puppet last run on an-master1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[analytics-privatedata-users_ensure_members] [13:26:25] PROBLEM - puppet last run on mw2196 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:26:33] PROBLEM - puppet last run on mw1340 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:26:39] PROBLEM - puppet last run on mw1280 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:26:43] PROBLEM - puppet last run on mw2239 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:26:45] PROBLEM - puppet last run on mw1288 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:26:46] thedj: so that is being handled. Thank you for the ping :) [13:26:49] PROBLEM - puppet last run on mw2170 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:27:13] PROBLEM - puppet last run on mw1246 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:27:39] PROBLEM - puppet last run on mw2136 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:27:39] PROBLEM - puppet last run on mw2181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:28:11] PROBLEM - puppet last run on mw1318 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:28:11] PROBLEM - puppet last run on mw1308 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:28:21] PROBLEM - puppet last run on mw1300 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:28:23] PROBLEM - puppet last run on mw2267 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:28:23] PROBLEM - puppet last run on mw2222 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:28:25] PROBLEM - puppet last run on ores2004 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:28:27] PROBLEM - puppet last run on mw2168 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:28:37] PROBLEM - puppet last run on mw2188 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:28:37] PROBLEM - puppet last run on mw2173 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:28:45] PROBLEM - puppet last run on mw2257 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:28:49] PROBLEM - puppet last run on mw2229 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:28:53] PROBLEM - puppet last run on mw1289 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:28:59] PROBLEM - puppet last run on mw1323 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:29:07] PROBLEM - puppet last run on mwlog1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:29:07] PROBLEM - puppet last run on mw1247 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:29:07] PROBLEM - puppet last run on mw1230 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:29:17] hashar: np. figured during holiday season an extra prod could be useful [13:29:25] PROBLEM - puppet last run on mw1314 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:29:35] PROBLEM - puppet last run on mw2242 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:29:37] PROBLEM - puppet last run on mw2197 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:29:37] PROBLEM - puppet last run on mw2179 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:29:43] PROBLEM - puppet last run on mw1283 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:29:47] PROBLEM - puppet last run on mw1305 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:29:53] PROBLEM - puppet last run on ores1003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:29:57] PROBLEM - puppet last run on mw1253 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:29:59] PROBLEM - puppet last run on mw2161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:29:59] PROBLEM - puppet last run on mw2154 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:29:59] PROBLEM - puppet last run on mw2166 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:30:09] PROBLEM - puppet last run on mw2147 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:30:19] PROBLEM - puppet last run on mw1348 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:30:23] PROBLEM - puppet last run on mw2241 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:30:23] PROBLEM - puppet last run on mw2217 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:30:29] PROBLEM - puppet last run on mw1229 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:30:37] PROBLEM - puppet last run on mw2195 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:30:37] PROBLEM - puppet last run on mw2165 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:30:56] (03PS1) 10Alexandros Kosiaris: Followup for ladsgroup removal [puppet] - 10https://gerrit.wikimedia.org/r/481606 [13:30:57] PROBLEM - puppet last run on mw1327 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:30:57] PROBLEM - puppet last run on mw1326 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:30:57] PROBLEM - puppet last run on mw1319 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:31:08] (03CR) 10Alexandros Kosiaris: [C: 03+2] Followup for ladsgroup removal [puppet] - 10https://gerrit.wikimedia.org/r/481606 (owner: 10Alexandros Kosiaris) [13:31:13] PROBLEM - puppet last run on mw1252 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:31:17] PROBLEM - puppet last run on mw1265 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:31:19] PROBLEM - puppet last run on mwdebug2002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:31:27] PROBLEM - puppet last run on mw2281 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:31:31] PROBLEM - puppet last run on mw2247 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:31:39] PROBLEM - puppet last run on mw2221 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:31:39] PROBLEM - puppet last run on mw2183 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:31:39] PROBLEM - puppet last run on mw2182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:31:49] PROBLEM - puppet last run on mw1278 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:32:25] PROBLEM - puppet last run on ores1007 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ores-admin_ensure_members] [13:32:29] PROBLEM - puppet last run on mw2285 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:32:33] PROBLEM - puppet last run on mw1307 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:32:33] PROBLEM - puppet last run on deploy1001 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members],Exec[deploy-service_ensure_members] [13:32:45] PROBLEM - puppet last run on mw1303 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:32:51] PROBLEM - puppet last run on mw2258 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:32:58] I'm shutting ircecho for a couple of puppet runs at least, just spam now [13:33:01] cc akosiaris [13:33:29] PROBLEM - puppet last run on mw1248 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:33:37] PROBLEM - puppet last run on mw1239 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[deployment_ensure_members] [13:37:10] godog: patch has been merged, should be good to re-enable icinga-wm in ~25 mins or so [13:39:09] (03PS2) 10TheDJ: wmcs: Add postgres maps users for eqiad1-r region [puppet] - 10https://gerrit.wikimedia.org/r/481341 (https://phabricator.wikimedia.org/T212596) (owner: 10BryanDavis) [13:41:21] akosiaris: ack, thanks will do [13:43:11] (03PS1) 10Alexandros Kosiaris: Revert "Revoke ladsgroup access due to lost laptop" [puppet] - 10https://gerrit.wikimedia.org/r/481608 [14:13:09] (03CR) 10Urbanecm: [C: 03+1] "LGTM. Don't forget namespaceDupes.php." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481578 (https://phabricator.wikimedia.org/T212678) (owner: 10Tulsi Bhagat) [14:14:33] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy [14:22:22] (03PS1) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/481612 [14:23:43] (03CR) 10Hashar: "git build package fails due to:" [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/481612 (owner: 10Hashar) [14:25:19] PROBLEM - citoid endpoints health on scb1003 is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received: /api (Scrapes sample page) timed out before a response was received [14:25:37] PROBLEM - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received: /api (Scrapes sample page) timed out before a response was received [14:26:36] (03PS1) 10Hashar: Fix .gitreview to point to proper repo [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/481613 [14:26:43] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy [14:27:39] RECOVERY - citoid endpoints health on scb1003 is OK: All endpoints are healthy [14:34:09] PROBLEM - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is CRITICAL: /api (Scrapes sample page) timed out before a response was received [14:35:48] (03PS2) 10Hashar: Fix .gitreview to point to proper repo [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/481613 [14:35:50] (03PS1) 10Hashar: gbp: use upstream branch master, not tags [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/481615 [14:37:43] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy [14:40:55] (03CR) 10Hashar: "That comes from the forked repo mediawiki/php/excimer :)" [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/481613 (owner: 10Hashar) [14:49:38] (03PS3) 10Hashar: Fix .gitreview to point to proper repo [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/481613 [14:49:40] (03PS2) 10Hashar: gbp: use upstream branch master, not tags [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/481615 [14:55:52] (03CR) 10MarcoAurelio: [C: 03+1] "lgtm" [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/481613 (owner: 10Hashar) [14:55:56] (03CR) 10Hashar: "Pending upstream change https://gerrit.wikimedia.org/r/#/c/mediawiki/php/excimer/+/481621/" (031 comment) [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/481613 (owner: 10Hashar) [14:56:04] (03CR) 10Hashar: [C: 04-1] Fix .gitreview to point to proper repo [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/481613 (owner: 10Hashar) [14:59:56] (03CR) 10Hashar: "I think it is fine. The CI job runs for stretch + wikimedia but ends up failing with:" [debs/php-excimer] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/481615 (owner: 10Hashar) [15:00:31] (03CR) 10MarcoAurelio: Configure $wgNamespaceAliases for yue.wiktionary (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481578 (https://phabricator.wikimedia.org/T212678) (owner: 10Tulsi Bhagat) [15:04:54] 10Operations, 10monitoring, 10Kubernetes: debianize docker-registry 2.7.0-rc0 and upload in stretch-wikimedia - https://phabricator.wikimedia.org/T210071 (10hashar) [15:05:10] 10Operations, 10Prod-Kubernetes, 10serviceops, 10Kubernetes, 10Patch-For-Review: improve docker registry architecture - https://phabricator.wikimedia.org/T209271 (10hashar) [15:09:56] 10Operations, 10Kubernetes: set up a test node with new version, Redis as cache, a new Swift container and export metrics over Fraphana - https://phabricator.wikimedia.org/T210076 (10hashar) [15:10:49] 10Operations, 10Kubernetes: Evaluate VMWare's Harbour as a docker registry - https://phabricator.wikimedia.org/T202504 (10hashar) [15:19:08] 10Operations, 10Continuous-Integration-Infrastructure, 10Release Pipeline, 10Release-Engineering-Team (Kanban): Switch CI Docker Storage Driver to its own partition and to use devicemapper - https://phabricator.wikimedia.org/T178663 (10hashar) [15:32:42] (03PS3) 10Tulsi Bhagat: Configure $wgNamespaceAliases for yue.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481578 (https://phabricator.wikimedia.org/T212678) [15:58:24] (03CR) 10MarcoAurelio: [C: 03+1] "Looks good to me now." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481578 (https://phabricator.wikimedia.org/T212678) (owner: 10Tulsi Bhagat) [16:02:01] PROBLEM - Long running screen/tmux on certcentral1001 is CRITICAL: CRIT: Long running SCREEN process. (user: vgutierrez PID: 17173, 1733298s 1728000s). [16:02:56] that's me [16:11:55] RECOVERY - Long running screen/tmux on certcentral1001 is OK: OK: No SCREEN or tmux processes detected. [16:12:05] happy new year folks :) [16:14:56] always :) [16:15:46] feliz año vgutierrez ! [16:16:08] happy new year vgutierrez [16:16:27] feliz año vgutierrez [16:34:10] feliz año vgutierrez ;) [17:11:09] 10Operations: uwsgi's logsocket_plugin.so causes segfaults during log rotation - https://phabricator.wikimedia.org/T212697 (10Volans) Thanks @elukey, I'll have a look in the next days, at first look it seems that: - the logrotate config is the default one from the `uwsgi` package and doesn't point to where we ac... [18:10:03] 10Operations, 10Beta-Cluster-Infrastructure, 10Traffic, 10monitoring: Monitor Varnish caches on beta cluster have two varnishd process running - https://phabricator.wikimedia.org/T75944 (10hashar) 05Open→03Declined Abandoning this old task. It could be addressed by adding proper monitoring to the whole... [21:25:03] PROBLEM - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is CRITICAL: cluster=cache_text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [21:27:29] RECOVERY - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [23:04:33] PROBLEM - Varnish traffic drop between 30min ago and now at esams on icinga1001 is CRITICAL: 57.17 le 60 https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6fullscreenorgId=1 [23:17:57] RECOVERY - Varnish traffic drop between 30min ago and now at esams on icinga1001 is OK: (C)60 le (W)70 le 71.81 https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6fullscreenorgId=1