[00:12:51] RECOVERY - Puppet errors on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [00:27:37] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:28:02] You are trusted and identified by the name .*@wikipedia/.* [01:28:02] @whoami [01:28:17] I trust: bd808!.*@wikimedia/BDavis-WMF (2admin), .*@wikimedia/andrew-bogott (2admin), .*@wikimedia/mviswanathan-wmf (2admin), .*@wikimedia/.* (2trusted), .*@mediawiki/.* (2trusted), .*@wikipedia/.* (2trusted), [01:28:17] @trusted [01:28:33] huh. I used to be an admin. [01:28:48] bd808: ^ [01:34:45] probably when the channel was renamed [01:37:37] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [02:03:38] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:33:38] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [03:44:07] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [04:14:07] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [04:22:56] 10Tools, 10I18n: Consider merging "Wikinity-stats-before" and "Wikinity-stats-after"? - https://phabricator.wikimedia.org/T177268#3652846 (10Liuxinyu970226) [05:05:44] 10Cloud-Services, 10Toolforge, 10MediaWiki-extensions-WikimediaMaintenance: Make maintance script for sending annual survey emails - https://phabricator.wikimedia.org/T148783#3652887 (10bd808) @Legoktm would it be crazy to implement the functionality of https://www.mediawiki.org/wiki/Extension:MassMessageEma... [05:51:04] 10Toolforge: Rust support on Toolforge - https://phabricator.wikimedia.org/T177271#3652925 (10Legoktm) [06:09:15] 10cloud-services-team, 10DBA: Provide a new sanitarium host for s8 - https://phabricator.wikimedia.org/T177274#3652960 (10Marostegui) [06:52:58] PROBLEM - Puppet errors on tools-exec-1425 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [07:27:59] RECOVERY - Puppet errors on tools-exec-1425 is OK: OK: Less than 1.00% above the threshold [0.0] [07:29:36] PROBLEM - High iowait on tools-exec-1428 is CRITICAL: CRITICAL: tools.tools-exec-1428.cpu.total.iowait (>12.50%) [07:58:53] PROBLEM - Puppet errors on tools-exec-1428 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:04:38] RECOVERY - High iowait on tools-exec-1428 is OK: OK: All targets OK [08:13:51] RECOVERY - Puppet errors on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [09:21:07] 10Cloud-Services, 10Toolforge, 10DBA: What would be the preferred way to run a series of read queries on DB replica, each on two *_p databases ? - https://phabricator.wikimedia.org/T124805#1966479 (10Marostegui) Hi @Alphos I hope you are well. I was wondering if this task is still needed? [09:25:08] PROBLEM - Puppet errors on tools-exec-1407 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:32:46] 10Cloud-VPS (Quota-requests): Request increased quota for webperf labs project - https://phabricator.wikimedia.org/T177279#3653163 (10Peter) [09:50:38] 10Cloud-VPS (Quota-requests), 10Performance-Team: Request increased quota for webperf labs project - https://phabricator.wikimedia.org/T177279#3653233 (10Peter) [10:05:10] RECOVERY - Puppet errors on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [11:03:14] PROBLEM - Puppet errors on tools-checker-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:03:30] PROBLEM - Puppet errors on tools-checker-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:04:32] PROBLEM - Puppet errors on tools-paws-worker-1005 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:04:34] PROBLEM - Puppet errors on tools-prometheus-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:04:58] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:05:54] PROBLEM - Puppet errors on tools-exec-1414 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:07:00] PROBLEM - Puppet errors on tools-k8s-etcd-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:07:16] PROBLEM - Puppet errors on tools-paws-worker-1007 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:07:29] PROBLEM - Puppet errors on tools-worker-1011 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:07:29] PROBLEM - Puppet errors on tools-bastion-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:07:33] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1419 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:07:33] PROBLEM - Puppet errors on tools-worker-1012 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:07:35] PROBLEM - Puppet errors on tools-docker-registry-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:07:43] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1427 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:07:43] PROBLEM - Puppet errors on tools-exec-1438 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:07:45] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:07:57] PROBLEM - Puppet errors on tools-worker-1025 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:07:57] PROBLEM - Puppet errors on tools-exec-1401 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:08:33] PROBLEM - Puppet errors on tools-exec-1423 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:08:38] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1415 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:08:47] PROBLEM - Puppet errors on tools-exec-1427 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:08:58] PROBLEM - Puppet errors on tools-paws-worker-1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:09:02] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:09:14] PROBLEM - Puppet errors on tools-exec-1418 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:09:28] PROBLEM - Puppet errors on tools-logs-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:09:35] PROBLEM - Puppet errors on tools-exec-1442 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:09:35] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1421 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:09:53] PROBLEM - Puppet errors on tools-grid-master is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:10:06] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [11:10:06] PROBLEM - Puppet errors on tools-exec-1421 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:10:10] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1426 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:10:14] PROBLEM - Puppet errors on tools-exec-1405 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:10:30] PROBLEM - Puppet errors on tools-exec-1435 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:10:58] PROBLEM - Puppet errors on tools-grid-shadow is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:11:00] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1420 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:11:01] PROBLEM - Puppet errors on tools-k8s-etcd-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:11:04] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:11:14] PROBLEM - Puppet errors on tools-paws-worker-1017 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:11:33] 10Cloud-VPS (Quota-requests), 10Performance-Team (Radar): Request increased quota for webperf labs project - https://phabricator.wikimedia.org/T177279#3653332 (10Gilles) [11:11:54] PROBLEM - Puppet errors on tools-puppetmaster-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:11:56] PROBLEM - Puppet errors on tools-exec-1410 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:12:06] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1428 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:12:18] PROBLEM - Puppet errors on tools-exec-1424 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:12:26] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:12:40] PROBLEM - Puppet errors on tools-exec-1415 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:12:47] PROBLEM - Puppet errors on tools-exec-1412 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:13:09] PROBLEM - Puppet errors on tools-services-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:13:23] PROBLEM - Puppet errors on tools-worker-1003 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:13:31] PROBLEM - Puppet errors on tools-paws-master-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:13:41] PROBLEM - Puppet errors on tools-cron-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:13:47] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1405 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:13:58] PROBLEM - Puppet errors on tools-exec-1426 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [11:14:01] PROBLEM - Puppet errors on tools-docker-builder-05 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:14:03] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:14:03] PROBLEM - Puppet errors on tools-exec-1432 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:14:19] PROBLEM - Puppet errors on tools-exec-gift-trusty-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:14:36] PROBLEM - Puppet errors on tools-worker-1026 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:14:48] PROBLEM - Puppet errors on tools-exec-1416 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:14:56] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:15:18] PROBLEM - Puppet errors on tools-static-10 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:15:26] PROBLEM - Puppet errors on tools-redis-1002 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:15:50] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:16:22] PROBLEM - Puppet errors on tools-exec-1419 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:16:27] those failures will be recovering soon [11:17:14] PROBLEM - Puppet errors on tools-paws-worker-1002 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:17:31] PROBLEM - Puppet errors on tools-flannel-etcd-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:17:53] PROBLEM - Puppet errors on tools-elastic-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:18:19] PROBLEM - Puppet errors on tools-paws-worker-1013 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:18:23] PROBLEM - Puppet errors on tools-exec-1430 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:18:59] PROBLEM - Puppet errors on tools-exec-1425 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:19:03] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1417 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:19:05] PROBLEM - Puppet errors on tools-worker-1027 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:19:11] PROBLEM - Puppet errors on tools-worker-1008 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:19:29] PROBLEM - Puppet errors on tools-worker-1005 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:19:37] PROBLEM - Puppet errors on tools-worker-1002 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:20:09] PROBLEM - Puppet errors on tools-paws-worker-1006 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:20:36] PROBLEM - Puppet errors on tools-exec-1429 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:20:44] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:20:52] PROBLEM - Puppet errors on tools-exec-1402 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:21:58] PROBLEM - Puppet errors on tools-prometheus-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:22:00] PROBLEM - Puppet errors on tools-exec-1433 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:22:02] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:22:04] PROBLEM - Puppet errors on tools-worker-1022 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:22:06] PROBLEM - Puppet errors on tools-elastic-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:22:14] PROBLEM - Puppet errors on tools-k8s-master-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:22:18] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:22:21] PROBLEM - Puppet errors on tools-worker-1021 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:22:22] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:22:22] PROBLEM - Puppet errors on tools-worker-1004 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:22:24] PROBLEM - Puppet errors on tools-worker-1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:22:48] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:23:11] PROBLEM - Puppet errors on tools-paws-worker-1010 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:23:21] PROBLEM - Puppet errors on tools-docker-registry-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:24:21] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1424 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:24:25] PROBLEM - Puppet errors on tools-proxy-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:26:09] PROBLEM - Puppet errors on tools-exec-1407 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:41:59] RECOVERY - Puppet errors on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:43:15] RECOVERY - Puppet errors on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:43:29] RECOVERY - Puppet errors on tools-checker-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:44:03] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [11:44:30] RECOVERY - Puppet errors on tools-paws-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [11:44:36] RECOVERY - Puppet errors on tools-exec-1442 is OK: OK: Less than 1.00% above the threshold [0.0] [11:44:36] RECOVERY - Puppet errors on tools-prometheus-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:45:00] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [11:45:08] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1426 is OK: OK: Less than 1.00% above the threshold [0.0] [11:45:12] RECOVERY - Puppet errors on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [11:45:54] RECOVERY - Puppet errors on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [11:46:00] RECOVERY - Puppet errors on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [11:46:00] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [11:46:14] RECOVERY - Puppet errors on tools-paws-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [11:46:58] RECOVERY - Puppet errors on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [11:47:17] RECOVERY - Puppet errors on tools-exec-1424 is OK: OK: Less than 1.00% above the threshold [0.0] [11:47:17] RECOVERY - Puppet errors on tools-paws-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [11:47:29] RECOVERY - Puppet errors on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [11:47:31] RECOVERY - Puppet errors on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:47:35] RECOVERY - Puppet errors on tools-docker-registry-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:47:35] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1419 is OK: OK: Less than 1.00% above the threshold [0.0] [11:47:35] RECOVERY - Puppet errors on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [11:47:44] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1427 is OK: OK: Less than 1.00% above the threshold [0.0] [11:47:46] RECOVERY - Puppet errors on tools-exec-1438 is OK: OK: Less than 1.00% above the threshold [0.0] [11:47:46] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [11:47:54] RECOVERY - Puppet errors on tools-worker-1025 is OK: OK: Less than 1.00% above the threshold [0.0] [11:47:56] RECOVERY - Puppet errors on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [11:48:30] RECOVERY - Puppet errors on tools-paws-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:48:34] RECOVERY - Puppet errors on tools-exec-1423 is OK: OK: Less than 1.00% above the threshold [0.0] [11:48:38] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [11:48:44] RECOVERY - Puppet errors on tools-exec-1427 is OK: OK: Less than 1.00% above the threshold [0.0] [11:48:59] RECOVERY - Puppet errors on tools-paws-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [11:49:05] RECOVERY - Puppet errors on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [11:49:13] RECOVERY - Puppet errors on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [11:49:29] RECOVERY - Puppet errors on tools-logs-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:49:33] RECOVERY - Puppet errors on tools-worker-1026 is OK: OK: Less than 1.00% above the threshold [0.0] [11:49:35] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1421 is OK: OK: Less than 1.00% above the threshold [0.0] [11:49:53] RECOVERY - Puppet errors on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0] [11:50:03] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [11:50:03] RECOVERY - Puppet errors on tools-exec-1421 is OK: OK: Less than 1.00% above the threshold [0.0] [11:50:20] RECOVERY - Puppet errors on tools-static-10 is OK: OK: Less than 1.00% above the threshold [0.0] [11:50:24] RECOVERY - Puppet errors on tools-redis-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [11:50:32] RECOVERY - Puppet errors on tools-exec-1435 is OK: OK: Less than 1.00% above the threshold [0.0] [11:50:50] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [11:51:00] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [11:51:01] RECOVERY - Puppet errors on tools-k8s-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:51:55] RECOVERY - Puppet errors on tools-puppetmaster-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:52:07] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [11:52:13] RECOVERY - Puppet errors on tools-paws-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [11:52:25] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [11:52:41] RECOVERY - Puppet errors on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [11:52:48] RECOVERY - Puppet errors on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [11:53:10] RECOVERY - Puppet errors on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:53:18] RECOVERY - Puppet errors on tools-paws-worker-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [11:53:22] RECOVERY - Puppet errors on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [11:53:39] RECOVERY - Puppet errors on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:53:47] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [11:53:57] RECOVERY - Puppet errors on tools-exec-1426 is OK: OK: Less than 1.00% above the threshold [0.0] [11:54:01] RECOVERY - Puppet errors on tools-docker-builder-05 is OK: OK: Less than 1.00% above the threshold [0.0] [11:54:03] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [11:54:04] RECOVERY - Puppet errors on tools-worker-1027 is OK: OK: Less than 1.00% above the threshold [0.0] [11:54:20] RECOVERY - Puppet errors on tools-exec-gift-trusty-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:54:28] RECOVERY - Puppet errors on tools-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [11:54:38] RECOVERY - Puppet errors on tools-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [11:54:46] RECOVERY - Puppet errors on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [11:54:56] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [11:55:50] RECOVERY - Puppet errors on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [11:56:22] RECOVERY - Puppet errors on tools-exec-1419 is OK: OK: Less than 1.00% above the threshold [0.0] [11:57:07] RECOVERY - Puppet errors on tools-worker-1022 is OK: OK: Less than 1.00% above the threshold [0.0] [11:57:07] RECOVERY - Puppet errors on tools-elastic-03 is OK: OK: Less than 1.00% above the threshold [0.0] [11:57:15] RECOVERY - Puppet errors on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:57:22] RECOVERY - Puppet errors on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [11:57:24] RECOVERY - Puppet errors on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [11:57:30] RECOVERY - Puppet errors on tools-flannel-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:57:52] RECOVERY - Puppet errors on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:58:10] RECOVERY - Puppet errors on tools-paws-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [11:58:22] RECOVERY - Puppet errors on tools-docker-registry-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:58:26] RECOVERY - Puppet errors on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0] [11:59:00] RECOVERY - Puppet errors on tools-exec-1425 is OK: OK: Less than 1.00% above the threshold [0.0] [11:59:05] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [11:59:13] RECOVERY - Puppet errors on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0] [11:59:25] RECOVERY - Puppet errors on tools-proxy-01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:00:09] RECOVERY - Puppet errors on tools-paws-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [12:00:35] RECOVERY - Puppet errors on tools-exec-1429 is OK: OK: Less than 1.00% above the threshold [0.0] [12:00:45] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [12:01:09] RECOVERY - Puppet errors on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [12:01:58] RECOVERY - Puppet errors on tools-prometheus-01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:02:04] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [12:02:04] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [12:02:18] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [12:02:20] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [12:02:24] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [12:02:48] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [12:04:22] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1424 is OK: OK: Less than 1.00% above the threshold [0.0] [12:59:39] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:34:37] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [14:05:59] Good morning, shinken! [14:43:26] Hello! I am using my tool over toolforge and need to access database via python. For that, i need to know the HOST and PORT of running mySQL server. using linux command 'netstat -tln' i could list all the LISTENING PORTS but couldn't conclude which port is assigned to MySql. [14:44:54] * I Need help in figuring out the HOST and PORT for MySql database replicas available at toolforge. [14:44:54] robust: the databases are on separate servers [14:45:19] are you trying to host a database, or trying to connect to the wiki database replicas? [14:45:43] the former is tools-db [14:46:15] zhuyifei1999: I am trying to connect database replica to my tool. the enwiki_p [14:46:17] the latter is *.{web,analytics}.db.svc.eqiad.wmflabs [14:46:35] then enwiki.{web,analytics}.db.svc.eqiad.wmflabs [14:46:47] choose web or analytics depending on your usage [14:47:13] they are same at the moment, but in the future web may have a smaller query time limit [14:47:48] zhuyifei1999: i.e. HOST = 'tools-db' and PORT= 'enwiki.{web,analytics}.db.svc.eqiad.wmflabs' ?? [14:49:02] did i get the host and port right? [14:53:03] no [14:53:34] tools-db is the host if you want to /host/ a database [14:53:43] as in create a database [14:54:21] for {web,analytics} you have to choose one of them [14:54:45] port is the default one (I forgot the number) [14:54:57] 3306 [14:55:14] zhuyifei1999_: thanks for the help- indeed, the idea of web is to have multiple-short lived queries, while analytics is for less concurrency, but longer-lived connections [14:55:35] np [14:55:50] although there is not difference enforced right now [14:56:07] jynus: btw, is my uid 3306? (I suddenly remembered) [14:56:21] 3306 is the default mysql port [14:56:35] sorry if I answered the wrong question [14:56:35] well, it could be a coincidence [14:57:04] But when I run 'netstat -tln' it doesn't show port 3306 [14:57:10] I mean can you check my uid? just curious cuz I’m on mobile atm [14:57:15] is my direction wrong? [14:57:30] robust: netstat scans localhost [14:57:45] the database servers are not your login servers [14:57:56] (aka bastions) [14:59:24] zhuyifei1999_: Oh! Okay! Thanks for the help! [14:59:44] np [15:25:38] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:30:13] 10Cloud-Services, 10Operations, 10User-bd808: Investigate ceasing self-service new Trusty instance creation in Labs - https://phabricator.wikimedia.org/T161899#3653856 (10chasemp) a:03bd808 [15:33:28] 10Cloud-VPS (Project-requests): Request creation of IIAB VPS project - https://phabricator.wikimedia.org/T176926#3641585 (10chasemp) +1 [15:45:02] 10Cloud-Services, 10Huggle: Create a general-purpose labs download server for big files - https://phabricator.wikimedia.org/T177293#3653890 (10Andrew) [16:05:36] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [16:31:03] 10Cloud-Services, 10cloud-services-team (Kanban): Setup wikitech, horizon, and striker on new labweb hardware - https://phabricator.wikimedia.org/T168470#3653968 (10bd808) Dropping FY17/18 Q2 goal status. Still high priority, but not goal level. [16:34:39] 10Cloud-Services, 10Huggle: Huggle development environment - portable virtual box - https://phabricator.wikimedia.org/T177145#3653975 (10Tarawa1943) [16:34:41] 10Cloud-Services, 10Huggle: Create a general-purpose labs download server for big files - https://phabricator.wikimedia.org/T177293#3653972 (10Tarawa1943) 05Open>03Resolved a:03Tarawa1943 Avaliable in 1.31.0-wmf.2 today. [16:36:39] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Operations, 10User-bd808: End self-service new Trusty instance creation in Cloud VPS; standardize on Debian base images - https://phabricator.wikimedia.org/T161899#3653981 (10bd808) [16:44:11] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Operations, 10User-bd808: End self-service new Trusty instance creation in Cloud VPS; standardize on Debian base images - https://phabricator.wikimedia.org/T161899#3653994 (10bd808) Retitled the task yet again. The Foundation production networks are getting cl... [16:58:56] 10cloud-services-team, 10Operations: Upgrade to puppet 4 (4.8 or newer) - https://phabricator.wikimedia.org/T177254#3654046 (10bd808) [17:08:50] 10Cloud-Services, 10Huggle: Huggle development environment - portable virtual box - https://phabricator.wikimedia.org/T177145#3654090 (10Andrew) [17:08:52] 10Cloud-Services, 10Huggle: Create a general-purpose labs download server for big files - https://phabricator.wikimedia.org/T177293#3654088 (10Andrew) 05Resolved>03Open Closed in error, best I can tell [17:22:36] 10Cloud-Services, 10Huggle: Create a general-purpose labs download server for big files - https://phabricator.wikimedia.org/T177293#3654114 (10Zoranzoki21) a:05Tarawa1943>03None I will now remove blocked user from assignee.. Sorry users from Cloud-Services and Huggle projects! [17:25:25] 10cloud-services-team, 10Operations: Upgrade to puppet 4 (4.8 or newer) - https://phabricator.wikimedia.org/T177254#3654131 (10bd808) [17:30:20] 10Toolforge, 10cloud-services-team (Kanban), 10Puppet: Switch Toolforge project hosts to the future parser - https://phabricator.wikimedia.org/T177298#3654146 (10bd808) [17:31:06] legoktm: Yay thanks [17:33:23] PROBLEM - Puppet errors on tools-paws-worker-1016 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:33:45] 10Cloud-Services, 10Toolforge, 10puppet-compiler: toolsbeta: set up puppet-compiler / temporary-apply - https://phabricator.wikimedia.org/T97081#3654172 (10bd808) [17:33:48] 10Toolforge, 10cloud-services-team (Kanban), 10Puppet: Switch Toolforge project hosts to the future parser - https://phabricator.wikimedia.org/T177298#3654171 (10bd808) [17:34:41] 10Cloud-VPS (Quota-requests): Increase floating-ip quota for 'cvn' project (trusty to debian migratino) - https://phabricator.wikimedia.org/T177299#3654178 (10Krinkle) [17:35:26] 10Toolforge, 10cloud-services-team (Kanban), 10puppet-compiler: toolsbeta: set up puppet-compiler / temporary-apply - https://phabricator.wikimedia.org/T97081#1232397 (10bd808) [17:36:08] 10Cloud-VPS (Quota-requests): Increase floating-ip quota for 'cvn' project (trusty to debian migration) - https://phabricator.wikimedia.org/T177299#3654202 (10Krinkle) [17:54:28] jynus: $ getent passwd 3306 [17:54:28] zhuyifei1999:x:3306:500:Zhuyifei1999:/home/zhuyifei1999:/bin/bash [17:54:39] ^ what I was asking ;) [17:55:13] (don't ask why I'm 3306; coincidence ;) ) [17:56:45] 10cloud-services-team, 10Operations: Upgrade to puppet 4 (4.8 or newer) - https://phabricator.wikimedia.org/T177254#3654352 (10herron) [18:03:11] :-) [18:04:27] 10cloud-services-team, 10wikitech.wikimedia.org, 10Patch-For-Review: contentadmin has suddenly less permissions - https://phabricator.wikimedia.org/T171208#3654393 (10EddieGP) >>! In T171208#3649644, @Krenair wrote: > Hmm. I think I can safely say we wouldn't want a contentadmin using tboverride Looking at... [18:08:21] RECOVERY - Puppet errors on tools-paws-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [18:14:49] 10cloud-services-team, 10Operations: Upgrade to puppet 4 (4.8 or newer) - https://phabricator.wikimedia.org/T177254#3654441 (10herron) [18:24:16] 10Cloud-VPS (Project-requests): Request creation of IIAB VPS project - https://phabricator.wikimedia.org/T176926#3654510 (10bd808) p:05Triage>03Normal I have created the [[https://wikitech.wikimedia.org/wiki/Nova_Resource:Iiab|iiab]] project and granted the project a quota of 1 floating ip. @Doc_James, chec... [18:24:25] 10Cloud-VPS (Project-requests), 10User-bd808: Request creation of IIAB VPS project - https://phabricator.wikimedia.org/T176926#3654512 (10bd808) 05Open>03Resolved a:03bd808 [18:41:46] 10Cloud-VPS (Quota-requests), 10Performance-Team (Radar): Request increased quota for webperf labs project - https://phabricator.wikimedia.org/T177279#3654586 (10Andrew) Approved! @chasemp will help with the specifics of performance testing. [18:44:48] 10Cloud-VPS (Quota-requests), 10Performance-Team (Radar): Request increased quota for webperf labs project - https://phabricator.wikimedia.org/T177279#3654608 (10chasemp) I am going to bump this project to handle 1 more xlarge instance, but I'm wondering what Cloud instance(s) are you using for perf? I was ho... [18:46:12] 10Cloud-VPS (Quota-requests), 10Performance-Team (Radar): Request increased quota for webperf labs project - https://phabricator.wikimedia.org/T177279#3654617 (10chasemp) p:05Triage>03Normal [18:51:25] 10Cloud-VPS (Quota-requests), 10Performance-Team (Radar): Request increased quota for webperf labs project - https://phabricator.wikimedia.org/T177279#3654632 (10chasemp) 05Open>03Resolved a:03chasemp [18:53:20] i'm getting persistent 503s after logging in at https://paws.wmflabs.org/ (under both my staff and volunteer accounts) [18:53:37] HaeB: it hates you :/ [18:53:44] let me see if I can find a cause [18:54:09] https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=paws.wmflabs.org says it's OK [18:55:31] that icinga check probably doesnt do the login [18:55:40] so it just gets that it's still up [18:56:11] i could login without seeing a 503 though [18:56:50] hm... i tried several times, with at least 5min or so inbetween [18:57:16] I'm not getting a 503, but I'm not getting a proper page either [18:57:18] trying now w/ staff account [18:57:23] oh... now I get the 503 [18:57:37] seems this person reported the same (on the wrong bug) https://phabricator.wikimedia.org/T175454#3629718 [18:57:53] wasn't paws moved to https://paws.tools.wmflabs.org/ ? I'm getting SSL error though [18:58:36] zhuyifei1999_: backend parts were moved to toolforge, but the "front door" is still https://paws.wmflabs.org/ [18:58:42] hangs on Your server is starting up. but not 5XX yet [18:58:50] bd808: ok [18:59:41] chasemp: I see your pod starting. I'm not seeing that happen for me or HaeB though... [18:59:51] 500 : Internal Server Error [18:59:53] is what I got back [19:00:02] yeah. it crashed [19:00:22] that k8s cluster is sick? [19:00:48] I think so. my k8s admin fu is pretty weak so googling stuff :) [19:04:17] my basic poking doesn't turn up anything [19:22:09] HaeB: I'm trying to ping Yuvi [19:22:19] thanks! [19:30:36] !log tools `kubectl --namespace=prod delete pod --all` on tools-paws-master-01 [19:30:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:37:25] well that didn't fix it :/ [19:37:36] now the hub is in CrashLopBackOff [19:37:50] bd808: try deleting all teh deployments? [19:39:11] if I do that... how do I start them again? [19:39:31] All the user pods are gone. its just the core that is not restarting properly now [19:39:52] Yuvi is out a lunch but should be able to help in 45m or so [19:40:24] bd808: ah, yeah that's probably unrecoverable so maybe wait :) [19:40:31] for user stuff I mean [20:13:40] bd808: I restarted docker a few days ago on one of the paws VMs. Is this possibly fallout from that? [20:14:11] andrewbogott: maybe, but the k8s cluster should route around one node if it is messed up. [20:14:30] ok — that's certainly what I was hoping at the time :) [20:15:08] andrewbogott: was tools-paws-worker-1002 the host you did that on? [20:15:31] I think so — / was full so it was probably already breaking a fair bit before I broke it more [20:16:35] *nod* I noticed that node was marked as being unavailable for scheduling when I ran `kubctl get node` [20:16:48] hm, I depooled and re-pooled it, I think [20:16:55] but maybe I mistyped or something [20:28:55] 10cloud-services-team, 10Community-Tech, 10DBA, 10Security: Create core ip_changes view for replicas - https://phabricator.wikimedia.org/T173891#3655053 (10Bawolff) >>! In T173891#3645059, @MusikAnimal wrote: > The table is now live and populated on all wikis, so I've repurposed this ticket for creating th... [21:34:47] https://paws.wmflabs.org is now 502 - i guess that's an under construction thing? [21:37:40] PROBLEM - Puppet errors on tools-exec-1417 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:39:05] HaeB: it's all kinds of messed up, but we are looking at it [21:39:29] got it, thanks! [22:05:58] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [22:12:39] RECOVERY - Puppet errors on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [22:29:58] PROBLEM - Puppet errors on tools-paws-worker-1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [22:30:30] PROBLEM - Puppet errors on tools-paws-worker-1005 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [22:31:04] 10Tool-Global-user-contributions: Switch GUC to use new Wiki Replicas - https://phabricator.wikimedia.org/T176686#3655384 (10Krinkle) a:05Krinkle>03None [22:31:15] 10Data-Services: Update meta_p database for new service names - https://phabricator.wikimedia.org/T176886#3655386 (10Krinkle) [22:31:24] 10Tool-Global-user-contributions: Switch GUC to use new Wiki Replicas - https://phabricator.wikimedia.org/T176686#3633849 (10Krinkle) 05Open>03stalled [22:34:24] PROBLEM - Puppet errors on tools-paws-worker-1016 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [22:34:32] PROBLEM - Puppet errors on tools-paws-master-01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [22:39:21] RECOVERY - Puppet errors on tools-paws-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [22:39:31] RECOVERY - Puppet errors on tools-paws-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:39:57] RECOVERY - Puppet errors on tools-paws-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [22:40:29] RECOVERY - Puppet errors on tools-paws-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [22:44:47] 10Cloud-Services, 10Huggle: Huggle development environment - portable virtual box - https://phabricator.wikimedia.org/T177145#3655474 (10Andrew) In order to keep big files off of NFS, I've created a static download site for things like this. Your file, for example, is now: http://download.wmflabs.org/huggle.... [22:46:03] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:46:06] 10Cloud-Services, 10Huggle: Huggle development environment - portable virtual box - https://phabricator.wikimedia.org/T177145#3655478 (10Andrew) [22:46:09] 10Cloud-Services, 10Huggle, 10Patch-For-Review: Create a general-purpose labs download server for big files - https://phabricator.wikimedia.org/T177293#3655475 (10Andrew) 05Open>03Resolved a:03Andrew http://download.wmflabs.org/ To add new files, copy them to download-01.download.eqiad.wmlabs:/srv/pub... [23:10:58] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [23:14:12] (03Draft1) 10Paladox: Up time for apt check to 10 mins [labs/icinga2] - 10https://gerrit.wikimedia.org/r/382091 [23:14:14] (03PS2) 10Paladox: Up time for apt check to 10 mins [labs/icinga2] - 10https://gerrit.wikimedia.org/r/382091 [23:14:22] (03CR) 10Paladox: [V: 032 C: 032] Up time for apt check to 10 mins [labs/icinga2] - 10https://gerrit.wikimedia.org/r/382091 (owner: 10Paladox) [23:16:21] PROBLEM - Puppet errors on tools-static-10 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:21:01] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [23:33:40] PROBLEM - Puppet errors on tools-exec-1417 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:34:31] I want to SCP something from a tool account, but I am not sure how to do that since as far as I know there's no way to directly log in to a tool account. [23:51:13] harej: you mean scp (download) a file from a directory owned by your tool account to local? [23:51:20] RECOVERY - Puppet errors on tools-static-10 is OK: OK: Less than 1.00% above the threshold [0.0] [23:51:36] then copy the file to a world readable directory [23:51:47] or set g+r on the directory [23:56:27] be gentle with scp too. it like to eat up all of the NFS resources on the bastion it runs from [23:56:36] *it likes