[00:03:15] !log video `sudo -H pip install -U youtube-dl` on all encoding hosts [00:03:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [00:04:27] !log video depool v2c worker on encoding01,02; repool encoding01,03 [00:04:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [00:29:58] bd808: Thanks for https://phabricator.wikimedia.org/T109179#3365689 [00:30:16] I kind of separated wmflabs.org from wikimedia.org in my mind conceptually, but maybe that's outdated. [00:33:28] Huh, just learned that 'en' is a secret alias when using `sql`. [00:38:05] Esther: you are not alone in the "labs is for testing" thoughts at all. I work on education with lots of people and event founded a new team on the idea that we need to think about things differently in the modern day. :) [00:38:18] *even founded [00:40:08] I think it was originally meant as that and then it ended up being used for quasi-production services like MapHack [00:40:15] Maybe not truly production, but still getting real-world uses [00:40:25] Well, and now there's like horizon.wikimedia.org. [00:40:28] Which is for Labs, but on wikimedia.org? [00:40:32] And I'm like whaaaaaaat? [00:40:36] Plus there was always wikitech.wikimedia.org. [00:40:43] Everything happens so much. [00:41:37] Its not surprising that what the project was meant for in 2011 and what it really is used for in 2017 differ [00:42:04] and..the controller is in a different domain from the instances it controls on purpose [00:42:09] the wikimedia.org thing is mostly about how er run the wmflabs.org dns [00:42:14] *we run [00:42:27] wmflabs.org is for user content, right? [00:42:32] whereas horizon is an officially run service [00:42:33] pretty much [00:47:32] 10Labs, 10DBA, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3366131 (10MZMcBride) [00:49:38] bd808: Sure, though as someone who started on the Toolserver, it's even more to wrap my head around. :-) [00:50:40] 10Labs, 10DBA, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#2415416 (10MZMcBride) >>! In T138967#3333050, @bd808 wrote: > Cross-post from {T166091}: > > ```SELECT pl_namespace, pl_title > FROM page > JOIN pagelinks ON pl_from = page_id > WHERE page_namespace=0 AND page_tit... [00:51:22] I look into this every six months or so. So I tried to leave myself a status update for the next time I wonder what's going on. [00:52:29] bd808: To be quite honest, I thought the "cloud" thing was just dumb rebranding/marketing/buzzwordery. [00:52:47] But I guess if it'll help understanding, it's a little less grating. [00:53:35] how does one request a new subdomain proxy for https://tools.wmflabs.org/openstack-browser/project/xtools ? [00:53:46] we want xtools.wmflabs.org [00:54:05] Hi samwilson. [00:54:18] hello Esther [00:54:18] :) [00:54:41] sorry, i've just found https://wikitech.wikimedia.org/wiki/Help:Proxy and am reading [00:54:53] samwilson: in horizon i believe [00:55:10] I was skimming https://wikitech.wikimedia.org/wiki/User:MZMcBride/Labs_notes [00:55:13] Which has been moved, apparently! [00:55:20] And looking to see if it had the answer. [00:55:34] samwilson: try https://horizon.wikimedia.org/project/proxy/ [00:55:53] hm seems likely... it's not letting me log in yet.... [00:56:09] Heh, preach. [00:57:26] ah there we go. cool, yes looks like it's the thing. [00:57:28] thanks! [00:58:13] woo, yup is all good :) [03:16:41] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Restrict access to users' edit stats unless opted-in - https://phabricator.wikimedia.org/T165401#3366243 (10Samwilson) In production, it seems to be a lot faster: https://xtools.wmflabs.org/ec/fr.wikipedia.org/Toughpigs In the profiling I've been seeing, the... [05:37:36] 10Labs, 10DBA, 10User-bd808, 10cloud-services-team (Kanban): setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 - https://phabricator.wikimedia.org/T168021#3366391 (10Marostegui) Excellent! Thanks for letting us know and thanks for testing the new labs servers, please let us know if you see wei... [05:50:26] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Planning for Xtools beta - https://phabricator.wikimedia.org/T167217#3366398 (10Samwilson) The new prod site is up and running, and the notice added [[ https://xtools.wmflabs.org/ | there ]]. I don't have access to old XTools, so can't add the notice there (I'... [05:53:47] 10Tool-Labs-tools-Other: Using SVG Translate tool in Arabic language - https://phabricator.wikimedia.org/T168498#3366404 (10alanajjar) [06:42:37] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:43:03] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1428 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:45:53] PROBLEM - Puppet errors on tools-worker-1027 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:22:37] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [07:23:01] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [07:25:53] RECOVERY - Puppet errors on tools-worker-1027 is OK: OK: Less than 1.00% above the threshold [0.0] [07:44:07] 10Tool-Labs-tools-fatameh: URL regexes are too loose - https://phabricator.wikimedia.org/T168363#3366557 (10Tarrow) 05Open>03Resolved a:03Tarrow [07:47:28] 10Labs, 10DBA: enwiki_p logging vs logging_userindex returning dramatically different results - https://phabricator.wikimedia.org/T168349#3366564 (10Marostegui) I tried the original two queries on labsdb-web.eqiad.wmnet without the LIMIT. The first query; ``` 12471 rows in set (10 min 52.70 sec) ``` The secon... [07:48:38] 10Tool-Labs-tools-fatameh: Changes retrieved by date even if the data hasn't changed - https://phabricator.wikimedia.org/T168502#3366565 (10Tarrow) [07:52:42] 10Tool-Labs-tools-fatameh: Clobbering existing references on statements - https://phabricator.wikimedia.org/T168504#3366591 (10Tarrow) [07:53:53] 10Tool-Labs-tools-fatameh: Return details in case of success or failure - https://phabricator.wikimedia.org/T168505#3366605 (10Tarrow) [08:47:56] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:27:53] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [09:56:28] 10Labs, 10Labs-Infrastructure: Labs Jessie images come with puppet 3.7.2, should be 3.8.5 - https://phabricator.wikimedia.org/T168511#3366794 (10hashar) [09:59:40] 10Labs, 10Tool-Labs, 10DBA: labsdb1001 and labsdb1003 short on available space - https://phabricator.wikimedia.org/T132431#3366819 (10Marostegui) [09:59:42] 10Tool-Labs-tools-Other, 10Tracking: merl tools (tracking) - https://phabricator.wikimedia.org/T69556#3366820 (10Marostegui) [09:59:46] 10Labs, 10Tool-Labs, 10DBA: s51127__dewiki_lists (merlbot) database using 13G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133325#3366816 (10Marostegui) 05Open>03Resolved a:03Marostegui And I have dropped this: ``` mysql:root@localhost [s51127__dewiki_lists]> drop database if exists s511... [13:56:05] !log tools depooling tools-exec-1401, 1407, 1408, 1430, 1431, 1432, 1435, 1438, 1439, 1440, 1441, tools-webgrid-lighttpd-1402, tools-webgrid-lighttpd-1407 [13:56:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:03:38] PROBLEM - Host tools-exec-1440 is DOWN: CRITICAL - Host Unreachable (10.68.22.236) [14:04:01] PROBLEM - Host tools-webgrid-lighttpd-1402 is DOWN: PING CRITICAL - Packet loss = 100% [14:04:05] PROBLEM - Host tools-exec-1438 is DOWN: CRITICAL - Host Unreachable (10.68.18.208) [14:04:19] PROBLEM - Host tools-exec-1432 is DOWN: CRITICAL - Host Unreachable (10.68.22.85) [14:04:57] PROBLEM - Host tools-exec-1435 is DOWN: CRITICAL - Host Unreachable (10.68.19.160) [14:05:19] PROBLEM - Host tools-exec-1408 is DOWN: CRITICAL - Host Unreachable (10.68.18.14) [14:05:29] PROBLEM - Host tools-exec-1407 is DOWN: CRITICAL - Host Unreachable (10.68.18.16) [14:05:52] PROBLEM - Host tools-exec-1430 is DOWN: CRITICAL - Host Unreachable (10.68.16.40) [14:06:09] PROBLEM - Host tools-exec-1439 is DOWN: CRITICAL - Host Unreachable (10.68.18.83) [14:06:11] PROBLEM - Host tools-webgrid-lighttpd-1407 is DOWN: CRITICAL - Host Unreachable (10.68.17.251) [14:06:23] PROBLEM - Host tools-exec-1441 is DOWN: CRITICAL - Host Unreachable (10.68.16.126) [14:06:46] PROBLEM - Host tools-exec-1401 is DOWN: CRITICAL - Host Unreachable (10.68.17.202) [14:07:24] PROBLEM - Host tools-exec-1431 is DOWN: CRITICAL - Host Unreachable (10.68.16.151) [14:19:00] RECOVERY - Host tools-exec-1407 is UP: PING OK - Packet loss = 0%, RTA = 2.38 ms [14:19:10] RECOVERY - Host tools-exec-1408 is UP: PING OK - Packet loss = 0%, RTA = 2.83 ms [14:19:18] RECOVERY - Host tools-exec-1401 is UP: PING OK - Packet loss = 0%, RTA = 3.84 ms [14:19:24] RECOVERY - Host tools-exec-1431 is UP: PING OK - Packet loss = 0%, RTA = 9.95 ms [14:19:29] !log tools repooling tools-exec-1401, 1407, 1408, 1430, 1431, 1432, 1435, 1438, 1439, 1440, 1441, tools-webgrid-lighttpd-1402, tools-webgrid-lighttpd-1407 [14:19:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:19:32] RECOVERY - Host tools-exec-1439 is UP: PING OK - Packet loss = 0%, RTA = 1.35 ms [14:19:55] RECOVERY - Host tools-exec-1435 is UP: PING OK - Packet loss = 0%, RTA = 2.21 ms [14:19:57] RECOVERY - Host tools-exec-1441 is UP: PING OK - Packet loss = 0%, RTA = 1.12 ms [14:19:59] RECOVERY - Host tools-webgrid-lighttpd-1407 is UP: PING OK - Packet loss = 0%, RTA = 1.04 ms [14:20:09] RECOVERY - Host tools-exec-1438 is UP: PING OK - Packet loss = 0%, RTA = 1.39 ms [14:20:31] RECOVERY - Host tools-exec-1430 is UP: PING OK - Packet loss = 0%, RTA = 2.37 ms [14:20:44] !log depooling tools-exec-1405, tools-exec-1425, tools-webgrid-generic-1403, tools-webgrid-lighttpd-1401, tools-webgrid-lighttpd-1405 [14:20:44] andrewbogott: Unknown project "depooling" [14:20:55] RECOVERY - Host tools-exec-1440 is UP: PING OK - Packet loss = 0%, RTA = 1.57 ms [14:20:57] !log tools depooling tools-exec-1405, tools-exec-1425, tools-webgrid-generic-1403, tools-webgrid-lighttpd-1401, tools-webgrid-lighttpd-1405 [14:20:59] RECOVERY - Host tools-webgrid-lighttpd-1402 is UP: PING OK - Packet loss = 0%, RTA = 2.00 ms [14:21:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:23:16] PROBLEM - Host tools-webgrid-lighttpd-1401 is DOWN: CRITICAL - Host Unreachable (10.68.16.34) [14:23:34] PROBLEM - SSH on tools-redis-1001 is CRITICAL: Connection refused [14:24:14] PROBLEM - Host tools-exec-1425 is DOWN: CRITICAL - Host Unreachable (10.68.17.151) [14:24:26] PROBLEM - Host tools-webgrid-lighttpd-1405 is DOWN: CRITICAL - Host Unreachable (10.68.17.65) [14:24:46] PROBLEM - Host tools-services-01 is DOWN: CRITICAL - Host Unreachable (10.68.16.29) [14:26:16] PROBLEM - Host tools-exec-1405 is DOWN: CRITICAL - Host Unreachable (10.68.18.3) [14:26:36] PROBLEM - Host tools-k8s-etcd-03 is DOWN: CRITICAL - Host Unreachable (10.68.21.239) [14:26:45] PROBLEM - Host tools-webgrid-generic-1403 is DOWN: CRITICAL - Host Unreachable (10.68.18.52) [14:28:41] PROBLEM - Host tools-redis-1001 is DOWN: CRITICAL - Host Unreachable (10.68.22.56) [14:29:22] PROBLEM - Host tools-checker-02 is DOWN: CRITICAL - Host Unreachable (10.68.16.17) [14:29:30] PROBLEM - Host tools-services-02 is DOWN: CRITICAL - Host Unreachable (10.68.18.36) [14:29:30] PROBLEM - Host tools-proxy-01 is DOWN: CRITICAL - Host Unreachable (10.68.21.49) [14:32:04] RECOVERY - Host tools-exec-1425 is UP: PING OK - Packet loss = 0%, RTA = 2.58 ms [14:32:12] RECOVERY - Host tools-webgrid-lighttpd-1401 is UP: PING OK - Packet loss = 0%, RTA = 5.62 ms [14:32:16] RECOVERY - Host tools-k8s-etcd-03 is UP: PING OK - Packet loss = 0%, RTA = 2.83 ms [14:32:19] !log tools repooling tools-exec-1405, tools-exec-1425, tools-webgrid-generic-1403, tools-webgrid-lighttpd-1401, tools-webgrid-lighttpd-1405 [14:32:20] RECOVERY - Host tools-webgrid-lighttpd-1405 is UP: PING OK - Packet loss = 0%, RTA = 6.33 ms [14:32:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:32:57] RECOVERY - Host tools-redis-1001 is UP: PING OK - Packet loss = 0%, RTA = 2.30 ms [14:33:03] RECOVERY - Host tools-services-01 is UP: PING OK - Packet loss = 0%, RTA = 4.80 ms [14:33:07] RECOVERY - Host tools-exec-1405 is UP: PING OK - Packet loss = 0%, RTA = 8.03 ms [14:33:36] RECOVERY - SSH on tools-redis-1001 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0) [14:35:22] RECOVERY - Host tools-webgrid-generic-1403 is UP: PING OK - Packet loss = 0%, RTA = 3.58 ms [14:37:43] !log tools depooling tools-exec-1403, tools-exec-gift-trusty-01, tools-webgrid-generic-1402, tools-webgrid-lighttpd-1403 [14:37:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:39:30] PROBLEM - Puppet errors on tools-k8s-etcd-03 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [14:40:32] PROBLEM - Host tools-redis-1002 is DOWN: CRITICAL - Host Unreachable (10.68.22.51) [14:40:44] RECOVERY - Host tools-services-02 is UP: PING OK - Packet loss = 0%, RTA = 2.17 ms [14:40:51] PROBLEM - Host tools-k8s-master-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.142) [14:41:03] PROBLEM - Host tools-webgrid-generic-1402 is DOWN: CRITICAL - Host Unreachable (10.68.18.50) [14:41:07] PROBLEM - Host tools-webgrid-lighttpd-1403 is DOWN: CRITICAL - Host Unreachable (10.68.17.191) [14:41:19] RECOVERY - Host tools-checker-02 is UP: PING OK - Packet loss = 0%, RTA = 3.99 ms [14:41:53] PROBLEM - Host tools-exec-gift-trusty-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.131) [14:42:11] PROBLEM - Host tools-exec-1403 is DOWN: CRITICAL - Host Unreachable (10.68.17.239) [14:42:24] PROBLEM - Host tools-paws-master-01 is DOWN: CRITICAL - Host Unreachable (10.68.22.164) [14:42:42] PROBLEM - Host tools-cron-01 is DOWN: CRITICAL - Host Unreachable (10.68.23.89) [14:43:04] RECOVERY - Host tools-proxy-01 is UP: PING OK - Packet loss = 0%, RTA = 1.19 ms [14:44:30] RECOVERY - Puppet errors on tools-k8s-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:51:55] RECOVERY - Host tools-exec-1403 is UP: PING OK - Packet loss = 0%, RTA = 3.26 ms [14:51:57] RECOVERY - Host tools-k8s-master-01 is UP: PING OK - Packet loss = 0%, RTA = 3.85 ms [14:52:09] RECOVERY - Host tools-webgrid-lighttpd-1403 is UP: PING OK - Packet loss = 0%, RTA = 11.60 ms [14:52:32] RECOVERY - Host tools-exec-gift-trusty-01 is UP: PING OK - Packet loss = 0%, RTA = 3.52 ms [14:52:33] RECOVERY - Host tools-paws-master-01 is UP: PING OK - Packet loss = 0%, RTA = 1.68 ms [14:52:37] PROBLEM - SSH on tools-bastion-05 is CRITICAL: Connection refused [14:52:43] RECOVERY - Host tools-webgrid-generic-1402 is UP: PING OK - Packet loss = 0%, RTA = 1.56 ms [14:52:47] RECOVERY - Host tools-redis-1002 is UP: PING OK - Packet loss = 0%, RTA = 4.56 ms [14:52:47] !log tools repooling tools-exec-1403, tools-exec-gift-trusty-01, tools-webgrid-generic-1402, tools-webgrid-lighttpd-1403 [14:52:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:53:10] !log tools depooling tools-exec-1402, 1426, 1429, 1433, tools-webgrid-lighttpd-1408, 1414, 1424 [14:53:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:55:11] PROBLEM - Host tools-docker-registry-01 is DOWN: CRITICAL - Host Unreachable (10.68.23.203) [14:55:18] PROBLEM - SSH on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:58] PROBLEM - Host tools-webgrid-lighttpd-1408 is DOWN: CRITICAL - Host Unreachable (10.68.18.42) [14:56:08] PROBLEM - SSH on tools-grid-shadow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:57:00] PROBLEM - Host tools-exec-1402 is DOWN: CRITICAL - Host Unreachable (10.68.17.205) [14:57:02] PROBLEM - Host tools-exec-1433 is DOWN: CRITICAL - Host Unreachable (10.68.21.151) [14:57:28] PROBLEM - Host tools-webgrid-lighttpd-1414 is DOWN: CRITICAL - Host Unreachable (10.68.20.250) [14:58:00] PROBLEM - Host tools-webgrid-lighttpd-1424 is DOWN: CRITICAL - Host Unreachable (10.68.21.172) [14:58:26] PROBLEM - Host tools-exec-1426 is DOWN: CRITICAL - Host Unreachable (10.68.18.205) [14:58:33] PROBLEM - Host tools-elastic-01 is DOWN: CRITICAL - Host Unreachable (10.68.20.9) [14:59:00] PROBLEM - Host tools-exec-1429 is DOWN: CRITICAL - Host Unreachable (10.68.16.61) [14:59:04] PROBLEM - Host tools-flannel-etcd-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.15) [15:01:08] PROBLEM - Host tools-grid-shadow is DOWN: CRITICAL - Host Unreachable (10.68.18.210) [15:08:34] RECOVERY - Host tools-elastic-01 is UP: PING OK - Packet loss = 0%, RTA = 5.82 ms [15:08:40] RECOVERY - Host tools-exec-1426 is UP: PING OK - Packet loss = 0%, RTA = 4.58 ms [15:09:06] RECOVERY - Host tools-exec-1402 is UP: PING OK - Packet loss = 0%, RTA = 3.99 ms [15:09:10] RECOVERY - Host tools-exec-1433 is UP: PING OK - Packet loss = 0%, RTA = 4.49 ms [15:09:16] RECOVERY - Host tools-docker-registry-01 is UP: PING OK - Packet loss = 0%, RTA = 2.82 ms [15:09:26] RECOVERY - Host tools-flannel-etcd-01 is UP: PING OK - Packet loss = 0%, RTA = 3.57 ms [15:09:28] RECOVERY - Host tools-exec-1429 is UP: PING OK - Packet loss = 0%, RTA = 4.06 ms [15:09:36] RECOVERY - Host tools-webgrid-lighttpd-1424 is UP: PING OK - Packet loss = 0%, RTA = 8.41 ms [15:10:06] RECOVERY - Host tools-grid-shadow is UP: PING OK - Packet loss = 0%, RTA = 1.98 ms [15:10:28] RECOVERY - Host tools-webgrid-lighttpd-1414 is UP: PING OK - Packet loss = 0%, RTA = 1.81 ms [15:10:48] !log tools repooling tools-exec-1402, 1426, 1429, 1433, tools-webgrid-lighttpd-1408, 1414, 1424 [15:10:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:10:59] RECOVERY - SSH on tools-grid-shadow is OK: SSH OK - OpenSSH_6.9p1 Ubuntu-2~trusty1 (protocol 2.0) [15:10:59] RECOVERY - Host tools-webgrid-lighttpd-1408 is UP: PING OK - Packet loss = 0%, RTA = 5.49 ms [15:12:44] !log tools depooling tools-exec-1404, tools-exec-1434, tools-worker-1026 [15:12:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:14:09] PROBLEM - Puppet errors on tools-docker-registry-01 is CRITICAL: CRITICAL: 16.67% of data above the critical threshold [0.0] [15:14:17] PROBLEM - Puppet errors on tools-flannel-etcd-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:14:34] PROBLEM - Host tools-bastion-05 is DOWN: CRITICAL - Host Unreachable (10.68.23.74) [15:15:32] PROBLEM - Host tools-exec-1434 is DOWN: CRITICAL - Host Unreachable (10.68.23.166) [15:17:06] PROBLEM - Host tools-flannel-etcd-02 is DOWN: CRITICAL - Host Unreachable (10.68.17.19) [15:17:18] PROBLEM - Host tools-worker-1026 is DOWN: CRITICAL - Host Unreachable (10.68.22.168) [15:18:04] PROBLEM - Host tools-exec-1404 is DOWN: CRITICAL - Host Unreachable (10.68.18.12) [15:19:09] RECOVERY - Puppet errors on tools-docker-registry-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:23:44] RECOVERY - Host tools-exec-1404 is UP: PING OK - Packet loss = 0%, RTA = 5.44 ms [15:23:52] RECOVERY - Host tools-flannel-etcd-02 is UP: PING OK - Packet loss = 0%, RTA = 3.15 ms [15:24:06] RECOVERY - Host tools-exec-1434 is UP: PING OK - Packet loss = 0%, RTA = 2.62 ms [15:24:16] RECOVERY - Host tools-bastion-05 is UP: PING OK - Packet loss = 0%, RTA = 3.05 ms [15:24:20] RECOVERY - Puppet errors on tools-flannel-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:24:36] RECOVERY - Host tools-worker-1026 is UP: PING OK - Packet loss = 0%, RTA = 3.62 ms [15:26:30] !log tools depooling tools-exec-1422, tools-webgrid-lighttpd-1413 [15:26:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:27:35] RECOVERY - SSH on tools-bastion-05 is OK: SSH OK - OpenSSH_6.9p1 Ubuntu-2~trusty1 (protocol 2.0) [15:31:41] !log tools temporarily pointing the tools-proxy IP to tools-proxy-01