[00:09:51] PROBLEM - Free space - all mounts on tools-static-01 is CRITICAL: CRITICAL: tools.tools-static-01.diskspace._srv.byte_percentfree (<50.00%) [00:10:08] hmm [00:10:33] PROBLEM - Puppet run on tools-static-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:17:05] PROBLEM - Puppet run on tools-static-02 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [00:20:34] RECOVERY - Puppet run on tools-static-01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:24:52] RECOVERY - Free space - all mounts on tools-static-01 is OK: OK: All targets OK [00:30:51] PROBLEM - Free space - all mounts on tools-static-01 is CRITICAL: CRITICAL: tools.tools-static-01.diskspace._srv.byte_percentfree (<55.56%) [00:35:25] PROBLEM - Free space - all mounts on tools-static-02 is CRITICAL: CRITICAL: tools.tools-static-02.diskspace._srv.byte_percentfree (<44.44%) [00:36:33] PROBLEM - Puppet run on tools-static-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:36:54] sigh [00:39:30] all I wanted to do was bump up worker_connections [00:39:40] here I am, running git repack -a -d -f --depth=250 --window=250 on a cdnjs git repo [00:41:59] doesn't look related at all :) [00:43:16] Platonides yeah. [00:44:01] I have to override that in nginx.conf, and nginx.conf needs to be different for trusty and jessie, and I don't want to branch on that since we wanted to move the static hosts to jessie anyway, and then I set them up and immediately ran into alerts because cdnjs is almost 60G now but only because of history... [00:46:34] RECOVERY - Puppet run on tools-static-01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:57:51] PROBLEM - SSH on tools-static-02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:00:25] RECOVERY - Free space - all mounts on tools-static-02 is OK: OK: tools.tools-static-02.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) tools.tools-static-02.diskspace._srv.byte_percentfree (More than half of the datapoints are undefined) [01:07:42] RECOVERY - SSH on tools-static-02 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0) [01:11:22] PROBLEM - Free space - all mounts on tools-static-02 is CRITICAL: CRITICAL: tools.tools-static-02.diskspace._srv.byte_percentfree (<100.00%) [01:12:14] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:12:34] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:12:43] PROBLEM - Puppet run on tools-k8s-master-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:12:51] PROBLEM - Puppet run on tools-exec-1208 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [01:12:54] uh oh [01:12:59] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:13:09] PROBLEM - Puppet run on tools-exec-1216 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:13:41] PROBLEM - Puppet run on tools-web-static-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [01:13:45] PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:13:51] that isn't me [01:14:05] PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:14:13] PROBLEM - Puppet run on tools-worker-1016 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:14:22] PROBLEM - Puppet run on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:14:50] PROBLEM - Puppet run on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:15:14] PROBLEM - Puppet run on tools-exec-1212 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:15:28] PROBLEM - Puppet run on tools-worker-1017 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:15:32] PROBLEM - Puppet run on tools-k8s-etcd-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:15:50] PROBLEM - Puppet run on tools-mail is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:16:04] PROBLEM - Puppet run on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:16:09] !log tools restarted puppetmaster on tools-puppetmaster-01 [01:16:12] next runs seem fine [01:16:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [01:16:38] PROBLEM - Puppet run on tools-exec-1217 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:17:04] RECOVERY - Puppet run on tools-static-02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:17:14] PROBLEM - Puppet run on tools-exec-gift is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:17:20] PROBLEM - Puppet run on tools-flannel-etcd-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:17:24] PROBLEM - Puppet run on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:17:44] PROBLEM - Puppet run on tools-worker-1018 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:17:52] PROBLEM - Puppet run on tools-elastic-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:18:00] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:18:22] PROBLEM - Puppet run on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:18:36] PROBLEM - Puppet run on tools-worker-1009 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:19:16] PROBLEM - Puppet run on tools-k8s-etcd-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:20:02] PROBLEM - Puppet run on tools-worker-1011 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:20:28] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:27:43] RECOVERY - Puppet run on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:43:46] PROBLEM - Puppet run on tools-puppetmaster-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [01:43:52] PROBLEM - SSH on tools-static-01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:46] RECOVERY - SSH on tools-static-01 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0) [01:55:27] RECOVERY - Puppet run on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [01:55:31] RECOVERY - Puppet run on tools-k8s-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:56:35] RECOVERY - Puppet run on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0] [01:57:13] RECOVERY - Puppet run on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [01:57:15] RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [01:57:19] RECOVERY - Puppet run on tools-flannel-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:57:23] RECOVERY - Puppet run on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [01:57:33] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [01:57:34] PROBLEM - Puppet run on tools-static-01 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [01:57:46] RECOVERY - Puppet run on tools-worker-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [01:57:50] RECOVERY - Puppet run on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:57:50] RECOVERY - Puppet run on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [01:58:00] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [01:58:08] RECOVERY - Puppet run on tools-exec-1216 is OK: OK: Less than 1.00% above the threshold [0.0] [01:58:20] RECOVERY - Puppet run on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [01:58:36] RECOVERY - Puppet run on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0] [01:58:40] RECOVERY - Puppet run on tools-web-static-01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:58:42] RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:59:04] RECOVERY - Puppet run on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [01:59:12] RECOVERY - Puppet run on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [01:59:16] RECOVERY - Puppet run on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:59:22] RECOVERY - Puppet run on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [01:59:54] RECOVERY - Puppet run on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [01:59:59] RECOVERY - Puppet run on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [02:00:15] RECOVERY - Puppet run on tools-exec-1212 is OK: OK: Less than 1.00% above the threshold [0.0] [02:00:27] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [02:00:52] RECOVERY - Puppet run on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [02:01:04] RECOVERY - Puppet run on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [02:02:59] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [02:08:05] PROBLEM - Puppet run on tools-static-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [02:12:33] RECOVERY - Puppet run on tools-static-01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:43:03] RECOVERY - Puppet run on tools-static-02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:00:20] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:31:01] RECOVERY - Puppet run on tools-mail-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:35:17] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [07:36:08] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.66 ms [07:38:30] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [07:39:25] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 1.57 ms [07:42:01] PROBLEM - Puppet run on tools-mail-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:47:18] 06Labs: Build and package traefik - https://phabricator.wikimedia.org/T143294#2581480 (10AlexMonk-WMF) I guess the package should include a file to register traefik as a service [08:03:17] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [08:12:29] 06Labs, 10Beta-Cluster-Infrastructure, 13Patch-For-Review, 07Puppet: /etc/puppet/puppet.conf keeps getting double content - first for labs-wide puppetmaster, then for the correct puppetmaster - https://phabricator.wikimedia.org/T132689#2581564 (10hashar) 05Open>03Resolved a:03mmodell Thanks @Dzahn... [08:25:55] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.52 ms [08:50:54] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [09:23:57] 06Labs, 06Operations, 13Patch-For-Review: grafana-labs.wikimedia.org doesn't reflect grafana-labs-admin.wikimedia.org - https://phabricator.wikimedia.org/T143556#2581695 (10fgiunchedi) cors is fixed, the only bit left to automate is changing `files/grafana/grafana_create_anon_user` to also add `Viewer` right... [09:41:56] 06Labs, 06Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Rename specific account in LDAP, Wikitech, Gerrit and Phabricator - https://phabricator.wikimedia.org/T85913#2581769 (10zeljkofilipin) [09:42:35] PROBLEM - Puppet run on tools-exec-1217 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:43:12] PROBLEM - Puppet run on tools-exec-gift is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:43:22] PROBLEM - Puppet run on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:43:34] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:43:40] PROBLEM - Puppet run on tools-k8s-master-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:43:49] PROBLEM - Puppet run on tools-exec-1208 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:44:07] PROBLEM - Puppet run on tools-exec-1216 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:44:37] PROBLEM - Puppet run on tools-worker-1009 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:44:40] PROBLEM - Puppet run on tools-web-static-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:44:41] PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:45:05] PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [09:45:13] PROBLEM - Puppet run on tools-worker-1016 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:45:15] PROBLEM - Puppet run on tools-k8s-etcd-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:45:23] PROBLEM - Puppet run on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:45:49] PROBLEM - Puppet run on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:46:01] PROBLEM - Puppet run on tools-worker-1011 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:46:13] PROBLEM - Puppet run on tools-exec-1212 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:46:27] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:46:29] PROBLEM - Puppet run on tools-worker-1017 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:46:33] PROBLEM - Puppet run on tools-k8s-etcd-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:46:49] PROBLEM - Puppet run on tools-mail is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:47:20] PROBLEM - Puppet run on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:47:34] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:48:02] PROBLEM - Puppet run on tools-worker-1020 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:48:06] PROBLEM - Puppet run on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:48:18] PROBLEM - Puppet run on tools-flannel-etcd-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:48:20] PROBLEM - Puppet staleness on tools-exec-cyberbot is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [09:48:44] PROBLEM - Puppet run on tools-worker-1018 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:48:48] PROBLEM - Puppet run on tools-worker-1012 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:48:50] PROBLEM - Puppet run on tools-elastic-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:48:54] PROBLEM - Puppet run on tools-worker-1006 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:48:58] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:49:14] PROBLEM - Puppet run on tools-grid-master is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:49:20] PROBLEM - Puppet run on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:49:24] PROBLEM - Puppet run on tools-cron-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:49:38] PROBLEM - Puppet run on tools-web-static-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:50:02] PROBLEM - Puppet run on tools-checker-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:50:40] PROBLEM - Puppet run on tools-exec-1209 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:51:00] PROBLEM - Puppet run on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:51:18] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:52:23] PROBLEM - Puppet run on tools-grid-shadow is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:52:57] PROBLEM - Puppet run on tools-exec-1215 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:52:59] PROBLEM - Puppet run on tools-k8s-etcd-03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:52:59] PROBLEM - Puppet run on tools-exec-1219 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:53:03] PROBLEM - Puppet run on tools-elastic-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:53:11] PROBLEM - Puppet run on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:53:39] PROBLEM - Puppet run on tools-worker-1021 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:53:47] PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:53:49] PROBLEM - Puppet run on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:54:05] PROBLEM - Puppet run on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:54:05] PROBLEM - Puppet run on tools-static-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:54:17] PROBLEM - Puppet run on tools-exec-1203 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:54:31] PROBLEM - Puppet run on tools-webgrid-lighttpd-1407 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:54:49] PROBLEM - Puppet run on tools-exec-1405 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:54:51] PROBLEM - Puppet run on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:54:52] PROBLEM - Puppet run on tools-exec-1205 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:55:11] PROBLEM - Puppet run on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:55:17] PROBLEM - Puppet run on tools-exec-1207 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:55:20] PROBLEM - Puppet run on tools-worker-1008 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:55:23] PROBLEM - Puppet run on tools-logs-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:55:33] PROBLEM - Puppet run on tools-redis-1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:55:39] PROBLEM - Puppet run on tools-exec-1218 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:55:49] PROBLEM - Puppet run on tools-proxy-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:56:05] PROBLEM - Puppet run on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:56:15] PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:56:37] PROBLEM - Puppet run on tools-docker-registry-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:57:55] PROBLEM - Puppet run on tools-worker-1005 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:58:15] PROBLEM - Puppet run on tools-worker-1004 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:58:25] PROBLEM - Puppet run on tools-worker-1022 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:58:29] PROBLEM - Puppet run on tools-exec-1404 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:59:15] PROBLEM - Puppet run on tools-worker-1019 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:59:41] PROBLEM - Puppet run on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:59:49] PROBLEM - Puppet run on tools-exec-1408 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:00:13] PROBLEM - Puppet run on tools-worker-1015 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:00:17] PROBLEM - Puppet run on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:00:40] PROBLEM - Puppet run on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:00:44] PROBLEM - Puppet run on tools-exec-1402 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:02:36] PROBLEM - Puppet run on tools-worker-1003 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:02:38] PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:03:04] PROBLEM - Puppet run on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:03:13] PROBLEM - Puppet run on tools-exec-1221 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:03:23] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:03:23] PROBLEM - Puppet run on tools-exec-1202 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:03:25] PROBLEM - Puppet run on tools-worker-1014 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:03:35] PROBLEM - Puppet run on tools-static-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:03:49] PROBLEM - Puppet run on tools-webgrid-lighttpd-1405 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:03:53] PROBLEM - Puppet run on tools-elastic-03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:04:09] PROBLEM - Puppet run on tools-exec-1206 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:04:31] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:05:23] PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:05:27] PROBLEM - Puppet run on tools-flannel-etcd-03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:07:50] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:08:02] PROBLEM - Puppet run on tools-redis-1002 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:08:08] PROBLEM - Puppet run on tools-worker-1025 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [10:08:52] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:09:16] PROBLEM - Puppet run on tools-k8s-master-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:09:26] PROBLEM - Puppet run on tools-worker-1007 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:09:35] PROBLEM - Puppet run on tools-worker-1013 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:09:43] PROBLEM - Puppet run on tools-proxy-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:09:47] PROBLEM - Puppet run on tools-worker-1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:10:03] PROBLEM - Puppet run on tools-exec-1201 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:10:11] PROBLEM - Puppet run on tools-prometheus-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:10:13] PROBLEM - Puppet run on tools-exec-1407 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:10:13] PROBLEM - Puppet run on tools-flannel-etcd-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:10:25] PROBLEM - Puppet run on tools-exec-1210 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:10:27] PROBLEM - Puppet run on tools-exec-1214 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:10:30] PROBLEM - Puppet run on tools-prometheus-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:10:37] PROBLEM - Puppet run on tools-exec-1406 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:11:35] PROBLEM - Puppet run on tools-worker-1023 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [10:11:38] PROBLEM - Puppet run on tools-checker-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:12:30] PROBLEM - Puppet run on tools-worker-1002 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:12:36] PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [10:13:18] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:13:58] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:36:25] 06Labs, 10Labs-Infrastructure, 10Graphite: Can't use wmflabs graphite datasource in grafana.wikimedia.org - https://phabricator.wikimedia.org/T141891#2581904 (10fgiunchedi) 05Open>03Resolved a:03fgiunchedi given that {T143556} is essentially fixed (grafana-labs available for non-logged in users too) I... [11:40:56] PROBLEM - Puppet staleness on tools-exec-1211 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [11:44:58] PROBLEM - Puppet staleness on tools-exec-1213 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [13:20:29] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Track labs instances hanging - https://phabricator.wikimedia.org/T141673#2582281 (10chasemp) Caught yesterday Candidate: tools-exec-1217 Details: stuck or frozen and can do no IO (amount other things such as SSH inaccessible completely) I used `virsh dom... [13:34:34] PROBLEM - Puppet run on tools-exec-1213 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:38:29] 06Labs, 10Tool-Labs: Linkwatcher spawns many processes without parent - https://phabricator.wikimedia.org/T123121#2582313 (10valhallasw) Resubmitted the continuous jobs. [13:44:59] RECOVERY - Puppet staleness on tools-exec-1213 is OK: OK: Less than 1.00% above the threshold [3600.0] [14:21:25] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Agaherbert was created, changed by Agaherbert link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Agaherbert edit summary: Created page with "{{Tools Access Request |Justification=Good afternoon, my name is Herbert Habermann, I'm a Brazilian student from Federal University of Alfenas. My project is about analysis o..." [14:23:41] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Agaherbert was modified, changed by Agaherbert link https://wikitech.wikimedia.org/w/index.php?diff=818346 edit summary: [14:50:34] 06Labs: Request creation of Mathematical Refresh Rate Policies labs project - https://phabricator.wikimedia.org/T143901#2582592 (10Agaherbert) [14:59:20] hello, I tried the rcstream example (https://wikitech.wikimedia.org/wiki/RCStream) for python. it runs very well, but sometimes I got the following error: "WARNING:socketIO_client:stream.wikimedia.org:443/socket.io/1: [packet error] unhandled namespace path ()". Is there any solution for that problem? [15:02:34] FNDE: I think Krinkle and ori are your best bets for answers on that question. [15:03:16] the shinken backscroll here today is pretty impressive [15:04:55] thank u:) I'll wait until they will be back. [15:05:14] PROBLEM - Puppet staleness on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [15:05:24] PROBLEM - Puppet staleness on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [15:18:17] bd808, and it'll spout loads more when the problem is resolved :D [15:21:38] FNDE: i know that error, and i think i fixed it, but atm i don't know what it is [15:37:36] Reedy: Are you around today? Still have some AWB questions. [16:26:03] Reedy: https://phabricator.wikimedia.org/T100457 was one of them. [16:32:56] RECOVERY - Puppet run on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [0.0] [16:33:10] RECOVERY - Puppet run on tools-webgrid-lighttpd-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [16:33:46] RECOVERY - Puppet run on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [16:34:03] RECOVERY - Puppet run on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [16:34:19] RECOVERY - Puppet run on tools-exec-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [16:34:31] RECOVERY - Puppet run on tools-webgrid-lighttpd-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [16:34:45] RECOVERY - Puppet run on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [16:34:49] RECOVERY - Puppet run on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [16:34:51] RECOVERY - Puppet run on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [16:34:51] RECOVERY - Puppet run on tools-exec-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [16:35:11] RECOVERY - Puppet run on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [16:35:19] RECOVERY - Puppet run on tools-exec-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [16:35:22] RECOVERY - Puppet run on tools-logs-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:35:30] RECOVERY - Puppet run on tools-redis-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [16:35:38] RECOVERY - Puppet run on tools-exec-1218 is OK: OK: Less than 1.00% above the threshold [0.0] [16:35:50] RECOVERY - Puppet run on tools-proxy-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:36:04] RECOVERY - Puppet run on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [16:36:40] RECOVERY - Puppet run on tools-docker-registry-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:37:55] RECOVERY - Puppet run on tools-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [16:38:13] RECOVERY - Puppet run on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [16:38:25] RECOVERY - Puppet run on tools-worker-1022 is OK: OK: Less than 1.00% above the threshold [0.0] [16:38:27] RECOVERY - Puppet run on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [16:38:44] 06Labs, 10Continuous-Integration-Infrastructure, 07Wikimedia-Incident: OpenStack misreports number of instances per project - https://phabricator.wikimedia.org/T143018#2582880 (10hashar) For reference, `nova absolute-limits` seems to show the quota and their usage: ``` $ nova absolute-limits +---------------... [16:39:04] RECOVERY - Puppet run on tools-static-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:39:16] RECOVERY - Puppet run on tools-worker-1019 is OK: OK: Less than 1.00% above the threshold [0.0] [16:39:40] RECOVERY - Puppet run on tools-webgrid-generic-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:14] RECOVERY - Puppet run on tools-worker-1015 is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:18] RECOVERY - Puppet run on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:26] RECOVERY - Puppet run on tools-flannel-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:40] RECOVERY - Puppet run on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:40] RECOVERY - Puppet run on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [16:42:38] RECOVERY - Puppet run on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [16:42:38] RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [16:43:02] RECOVERY - Puppet run on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [16:43:08] RECOVERY - Puppet run on tools-worker-1025 is OK: OK: Less than 1.00% above the threshold [0.0] [16:43:12] RECOVERY - Puppet run on tools-exec-1221 is OK: OK: Less than 1.00% above the threshold [0.0] [16:43:24] RECOVERY - Puppet run on tools-exec-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [16:43:25] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [16:43:26] RECOVERY - Puppet run on tools-worker-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [16:43:50] RECOVERY - Puppet run on tools-webgrid-lighttpd-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [16:43:52] RECOVERY - Puppet run on tools-elastic-03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:44:26] RECOVERY - Puppet run on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [16:44:31] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [16:44:45] RECOVERY - Puppet run on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:45:09] RECOVERY - Puppet run on tools-prometheus-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:45:13] RECOVERY - Puppet run on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [16:45:13] RECOVERY - Puppet run on tools-flannel-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:45:23] RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [16:45:25] RECOVERY - Puppet run on tools-exec-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [16:46:33] RECOVERY - Puppet run on tools-worker-1023 is OK: OK: Less than 1.00% above the threshold [0.0] [16:46:37] RECOVERY - Puppet run on tools-checker-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:48:03] RECOVERY - Puppet run on tools-redis-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [16:48:33] RECOVERY - Puppet run on tools-static-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:49:11] RECOVERY - Puppet run on tools-exec-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [16:49:17] RECOVERY - Puppet run on tools-k8s-master-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:49:39] RECOVERY - Puppet run on tools-worker-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [16:49:45] RECOVERY - Puppet run on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [16:50:01] RECOVERY - Puppet run on tools-exec-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [16:50:28] RECOVERY - Puppet run on tools-exec-1214 is OK: OK: Less than 1.00% above the threshold [0.0] [16:50:30] RECOVERY - Puppet run on tools-prometheus-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:50:38] RECOVERY - Puppet run on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [16:52:28] RECOVERY - Puppet run on tools-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [16:52:34] RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [16:52:50] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:53:16] RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [16:53:34] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [16:53:40] RECOVERY - Puppet run on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:53:50] RECOVERY - Puppet run on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [16:53:52] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:54:00] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [16:54:36] RECOVERY - Puppet run on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0] [16:55:12] RECOVERY - Puppet run on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [16:55:16] RECOVERY - Puppet run on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:55:22] RECOVERY - Puppet run on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [16:55:50] RECOVERY - Puppet run on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [16:56:00] RECOVERY - Puppet run on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [16:56:15] RECOVERY - Puppet run on tools-exec-1212 is OK: OK: Less than 1.00% above the threshold [0.0] [16:56:27] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:56:31] RECOVERY - Puppet run on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [16:56:35] RECOVERY - Puppet run on tools-k8s-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:56:51] RECOVERY - Puppet run on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [16:57:17] RECOVERY - Puppet run on tools-webgrid-lighttpd-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [16:57:25] RECOVERY - Puppet run on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [16:57:31] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [16:57:37] RECOVERY - Puppet run on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0] [16:58:03] RECOVERY - Puppet run on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [16:58:05] RECOVERY - Puppet run on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [16:58:11] RECOVERY - Puppet run on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [16:58:19] RECOVERY - Puppet run on tools-flannel-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:58:23] RECOVERY - Puppet run on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [16:58:42] RECOVERY - Puppet run on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [16:58:42] RECOVERY - Puppet run on tools-worker-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [16:58:48] RECOVERY - Puppet run on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [16:58:50] RECOVERY - Puppet run on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:58:52] RECOVERY - Puppet run on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [16:58:58] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [16:59:08] RECOVERY - Puppet run on tools-exec-1216 is OK: OK: Less than 1.00% above the threshold [0.0] [16:59:10] RECOVERY - Puppet run on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0] [16:59:21] RECOVERY - Puppet run on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [16:59:23] RECOVERY - Puppet run on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:59:37] RECOVERY - Puppet run on tools-web-static-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:59:39] RECOVERY - Puppet run on tools-web-static-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:59:42] RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:00:04] RECOVERY - Puppet run on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:00:06] RECOVERY - Puppet run on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [17:00:20] RECOVERY - Puppet run on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0] [17:00:40] RECOVERY - Puppet run on tools-exec-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [17:01:00] RECOVERY - Puppet run on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [17:01:13] RECOVERY - Puppet run on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [17:01:19] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [17:02:59] RECOVERY - Puppet run on tools-exec-1215 is OK: OK: Less than 1.00% above the threshold [0.0] [17:03:01] RECOVERY - Puppet run on tools-k8s-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:03:03] RECOVERY - Puppet run on tools-elastic-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:03:47] RECOVERY - Puppet run on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:45] !log tools depooled tools-webgrid-1413 [17:41:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [17:57:01] RECOVERY - Puppet run on tools-mail-01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:06:14] PROBLEM - Puppet run on tools-flannel-etcd-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:07:40] !log tools restart puppetmaster on tools-puppetmaster-01 [18:07:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [18:09:34] RECOVERY - Puppet run on tools-exec-1213 is OK: OK: Less than 1.00% above the threshold [0.0] [18:09:48] PROBLEM - Puppet run on tools-webgrid-lighttpd-1405 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:09:52] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:11:12] PROBLEM - Puppet run on tools-exec-1407 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:11:22] PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:13:50] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:44:47] RECOVERY - Puppet run on tools-webgrid-lighttpd-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [18:46:13] RECOVERY - Puppet run on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [18:46:13] RECOVERY - Puppet run on tools-flannel-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:46:25] RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [18:53:50] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:54:52] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:56:43] 06Labs, 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Wikimedia-Incident: Nodepool instance instance creation quota management - https://phabricator.wikimedia.org/T143016#2583542 (10hashar) [18:57:27] 06Labs, 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Wikimedia-Incident: Nodepool instance instance creation quota management - https://phabricator.wikimedia.org/T143016#2554147 (10hashar) @chasemp found a parameter that would cause Nova to refresh the quota on each reservation if the las... [19:01:11] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Striker, and 2 others: Deploy "Striker" Tool Labs console to WMF production - https://phabricator.wikimedia.org/T136256#2583557 (10bd808) On 2016-08-24, @yuvipanda and I tried to get Striker deployed onto californium in the WMF production cluster. While do... [19:11:57] 06Labs, 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Wikimedia-Incident: Nodepool instance instance creation quota management - https://phabricator.wikimedia.org/T143016#2583575 (10hashar) The quota issue has been solved. With 4 instances floating around, `nova absolute-limits` properly... [19:30:16] 06Labs, 10Tool-Labs, 07Wikimedia-Incident: Tune nginx config parameters for tools / labs proxies - https://phabricator.wikimedia.org/T143637#2583591 (10yuvipanda) So I fell into a bit of a hole yesterday around this. nginx is running different versions between tools-static and tools-proxy, because tools-stat... [19:35:12] 06Labs, 10Tool-Labs, 07Wikimedia-Incident: Tune nginx config parameters for tools / labs proxies - https://phabricator.wikimedia.org/T143637#2583608 (10yuvipanda) Remember we can't switch floating IP from horizon since we need to control which IP gets assigned where. Needs to happen on labcontrol1001 [19:36:03] 06Labs, 10Tool-Labs, 07Wikimedia-Incident: Tune nginx config parameters for tools / labs proxies - https://phabricator.wikimedia.org/T143637#2583610 (10yuvipanda) On second thought (1) doesn't need to happen right away, we should still do 2-4 tho. [19:40:39] puppet storm incoming, probably [19:41:49] averted! [19:43:33] PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:43:35] PROBLEM - Puppet run on tools-exec-1217 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:44:15] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:44:33] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:44:34] yuvipanda: /msg chanserv quiet #wikimedia-labs shinken-wm ;-) [19:44:41] PROBLEM - Puppet run on tools-k8s-master-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:44:51] PROBLEM - Puppet run on tools-exec-1208 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:44:52] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:44:54] although that doesn't tell you when you canenable it again [19:44:59] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:45:04] yuvipanda: shouldn't we move shinken to labs-admin? [19:45:14] clearly not averted [19:45:18] we should just kill shinken [19:45:24] no, definitely not to labs-admin :) [19:45:40] PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:45:47] chanserv says i'm not authorized to perform this operation [19:45:54] hrm. [19:46:22] PROBLEM - Puppet run on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:16:05] 06Labs, 10DBA: s2 replag currently 8 hours - https://phabricator.wikimedia.org/T143934#2583734 (10valhallasw) [20:18:44] RECOVERY - Puppet run on tools-puppetmaster-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:23:35] RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [20:24:15] RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [20:24:35] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [20:24:45] RECOVERY - Puppet run on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:24:49] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:26:23] RECOVERY - Puppet run on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [20:26:51] RECOVERY - Puppet run on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [20:28:35] RECOVERY - Puppet run on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0] [20:29:50] RECOVERY - Puppet run on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [20:30:00] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [20:30:40] RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:49:38] 06Labs, 10Tool-Labs: Problem with utf-8 on Grid - https://phabricator.wikimedia.org/T143691#2583941 (10Vladis13) The problem still exists. "Print" works, but don't run other programs. The following script in utf-8, Unix newline format LF: ``` # coding: utf8 import sys print(sys.stdin.encoding, sys.stdout.encod... [20:53:42] 06Labs, 10Tool-Labs: Problem with utf-8 on Grid - https://phabricator.wikimedia.org/T143691#2583956 (10valhallasw) You can try setting LANG=en_US.UTF-8 instead, but I can't guarantee that'll work. In general, you should not depend on Python's magic conversion from text to bytes, and just do that conversion you... [21:01:07] yuvipanda: bastion-03 is too slow [21:02:22] valhallasw`cloud: ^ [21:02:50] Yep, it is [21:04:00] bastion-02 is okay [21:04:08] User "jortola" is using lots of CPU on bastion-03 [21:04:14] *tools-bastion-03 [21:04:22] 20% is not a lot of CPU, but it is unwanted [21:04:32] it's definitely not something that would cause the instance to be super slow [21:04:40] that is typically ocnnected to someone overloading NFS [21:04:49] it seems ok to me? [21:05:03] somethign specific you guys are trying to do? [21:05:04] yeah, was slow when I logged in, but I could start iotop without issues [21:05:07] even nvfs seems fine [21:05:10] nfs even [21:05:35] running a pywikibot from tools-login is pretty lame [21:05:49] I've seen loads of people doing it [21:06:31] There was another user using about 20% of CPU as well as jortola when it was really slow [21:06:32] 23 python processes running right now. people just don't get it [21:06:40] it's generally considered bad manners [21:06:41] it depends a bit on whether it's an interactive job or something that runs continuously [21:06:43] i think thats the same reason noone uses their car blinkers. They say loads of people doing it :P [21:06:50] s/say/saw/ [21:06:56] hey I always do :) [21:07:05] Maybe put a notice in the ssh banner? [21:07:32] Like the one that says "This is a server of the tools project, the home"... [21:07:36] maybe we should write a chaos monkey that just randomly kills procs on -login ;) [21:08:04] it's gone a really really long time w/o me nuking things manually [21:08:07] that's nice [21:08:30] I hate it when people compile stuff on there. [21:08:47] Sometimes the package managers like pip do it automatically [21:08:54] compiling is a bigger issue because it also causes lots of i/o [21:09:11] this pstree line is .. interesting: sshd---sshd---bash---sudo---bash---sudo---bash---sudo---bash---sudo---bash---sudo---bash---sudo---bash---sudo---bash [21:09:46] vladis13 is running stuff on bastion because he's having issues with encodings and the grid I think [21:10:04] My pstree line should look interesting because I use tmux. [21:10:19] I see some detache screens and all kinds of noise [21:11:16] The detaching is a good feature though [21:11:58] If I'm doing a one time operation (e.g. compiling) on tools-bastion-02 say, then I detach the tmux session. [21:12:06] sure but over time it creates a mess because if it's not actively managed [21:13:30] Maybe restart the bastion every so often? [21:13:43] That clears out the tmux/screen sessions [21:14:17] maybe pick a paas and git rid of bastions ;) [21:14:40] I like the bastion :( [21:15:16] And I do ~90% of my dev work on there. [21:15:43] this is where my gut reaction is to say "gross" but I know that's not nice [21:15:59] If I could do it on my computer, then I would. [21:16:09] 06Labs, 10DBA: s2 replag currently 8 hours - https://phabricator.wikimedia.org/T143934#2584028 (10AlexMonk-WMF) [21:16:15] 06Labs, 10Wikimedia-Labs-General, 10DBA, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#2584027 (10AlexMonk-WMF) [21:16:16] bd808: dev work that hits the replicas is difficult without a bastion [21:16:26] it's possible (with a bit of magic tunneling), but definitely not easy [21:16:28] whenver the bastion is restarted we get a bunch of complaints :) but I'm not too worried about necessarily [21:16:33] but hte point is, you can't please everyone [21:17:49] valhallasw`cloud: we should have a vpn that makes getting to the dbs easy. I agree that ssh tunnels aren't simple for everyone [21:18:07] My computer is Windows, so not ideal for developing + my computer doesn't support virtual machines :/ [21:18:36] mmm, vpn with samba [21:18:40] * valhallasw`cloud dreams away [21:18:47] bd808, that'd be very helpful for labs as a whole [21:19:16] it would also make connecting to labs-internal hosts easy suddenly [21:19:59] +1 [21:20:16] That wouldn't be hard to do tbh [21:20:38] Hmm. [21:20:46] But it'd need a public IP. [21:20:50] That's always easy to say when it's other people doing the work [21:21:13] * tom29739 volunteers [21:21:22] it would need to be real infrastructure, not a labs project [21:21:35] it could actually be a labs project (just like bastion is) [21:21:38] and it would need to work easily with many os versions [21:21:40] ^ [21:21:48] OpenVPN does [21:22:00] and it would certainly need to not allow relaying [21:22:12] i.e. you can't bounce out to non_labs ips [21:22:13] but yeah, figuring out how to set up and configure openvpn correctly, and to connect that to ldap/etc, is the main question [21:22:25] there isa package somewhere a guy wrote that mimicks a vpn w/ ssh tunneling silently [21:22:28] I can't find it atm [21:22:30] OpenVPN doesn't allow relaying unless you configure it that way [21:23:37] SSH tunnelling is slower than a VPN [21:23:45] sshuttle is one [21:24:00] tom29739: {{cn}} on slower [21:24:30] an ssh forwared port is really no different than an openswan vpn connection [21:24:52] sshuttle is the one I meant [21:25:23] and yeah it's as fast or slow as a normal VPN from my experience, probably faster and less of a PITA but openvpn is all ssl vpn now iirc [21:25:29] and was pretty performant last I used it [21:25:44] openvpn is pretty good. [21:27:07] bd808, "Any forwarding done over an SSH will be subject to the well known TCP-over-TCP problem. The TCP protocol adds a fair amount of overhead because it is a transactional protocol. Using a UDP tunnel which is the OpenVPN default, will allow you to avoid all the issues with tunneling TCP over TCP." [21:28:30] nice quote. I've seen it before, but unless you are piping GB of data I bet you can't tell the difference [21:28:44] You can. [21:29:07] I do everything I can in a 6in4 tunnel and its *faster* than my raw ipv4 [21:29:30] and 6in4 is tcp-in-tcp as well [21:29:44] udp tunnel is no longer the openvpn default is it? [21:29:55] Hmm. [21:29:57] Not sure. [21:30:04] You can probably configure it [21:30:06] https://openvpn.net/index.php/open-source/339-why-ssl-vpn.html [21:40:07] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584103 (10tom29739) [21:40:43] ^ decided to request to see whether it works reliably, etc [21:42:38] interesting [21:42:59] * Platonides is curious to see how it works out [21:43:08] looks like you can use LDAP to authenticate users: https://openvpn.net/index.php/access-server/docs/admin-guides/190-how-to-authenticate-users-with-active-directory.html [21:44:10] Platonides, thanks :) [21:45:58] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584141 (10tom29739) [21:48:11] tom29739: Is the privpol-captcha system ready? [21:48:51] what does privpol-captcha? [21:49:18] Matthew_, unfortunately not, I need to deploy it, but the core API works so it shouldn't take to long [21:49:26] tom29739: Okay. [21:49:36] Platonides: A replacement for ReCaptcha that is valid to use on labs. [21:49:37] oh [21:49:42] nice [21:49:51] where's the repo? [21:50:01] Uhm... Hold on. [21:50:05] Platonides, github.com/labs-captcha/captcha [21:50:08] https://github.com/labs-captcha/captcha [21:50:11] found it [21:50:15] That one, yes. [21:50:18] That was quick [21:50:22] Sorry, I can never remember that. [21:50:37] what's the goal? [21:50:50] interface parity with recaptcha? [21:51:17] Platonides: To prevent automated form inputs by presenting a captcha. [21:51:18] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584103 (10AlexMonk-WMF) Discussed in... what? Your text cut off there @tom29739 :) [21:51:47] I honestly don't know how we can get on recaptcha's level... with the javascript algorithms and stuff. But as a decent replacement for just labs? Yes. [21:51:49] AFAIK, anyway. [21:52:03] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584166 (10tom29739) [21:52:33] I'd make it a bridge to ConfirmEdit's captchas [21:52:39] 06Labs, 10Tool-Labs, 07Upstream: Add SSHFP dns records to bastions - https://phabricator.wikimedia.org/T132225#2584170 (10AlexMonk-WMF) The patches are merged, we'll probably need to wait for OpenStack Newton to use it (we currently run Liberty with labtesthorizon on Mitaka) unless they backport to Mitaka I... [21:52:51] which should actually be easier to use outside mediawiki [21:52:52] Platonides, the unreadable text ones? [21:53:05] tom29739: ConfirmEdit has multiple backends [21:53:16] amongst them FancyCaptcha [21:53:22] but also others [21:53:44] Matthew_ suggested making something like recaptcha shows if you don't pass the automated checkbox. [21:53:52] With the select images thing [21:54:01] I originally invisioned the image selection idea. But eh. [21:54:02] With commons images of course. [21:54:41] It could have multiple backends. The possibilities are endless. [21:55:10] We call it a "captcha", but the idea is to stop spam and bots, etc [21:55:39] It's an interesting idea to spin it into ConfirmEdit though. I couldn't find a lot of information about the backends but... [21:57:15] 06Labs, 10Tool-Labs, 07Upstream: Add SSHFP dns records to bastions - https://phabricator.wikimedia.org/T132225#2584208 (10AlexMonk-WMF) They might backport it actually - see http://eavesdrop.openstack.org/meetings/designate/2016/designate.2016-08-17-16.59.html [21:59:18] I'm not a fan of reinventing the wheel, so borrowing ConfirmEdit stuff is a good idea [21:59:51] Oh typical: License: GNU General Public License 2.0 or later [22:03:34] is that a problem for you? [22:05:01] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584230 (10AlexMonk-WMF) [22:06:10] I'd be interested in having a more reusable captcha [22:07:08] What do you mean by "more reusable"? [22:08:44] I mean that the mw captchas were more reusable by other projects [22:08:59] And I don't particularly like the GPL because it's "viral". [22:09:40] So all my work would have to be GPL too, and anybody down the line reusing it would have to use GPL, etc etc etc. [22:10:37] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584261 (10AlexMonk-WMF) Logs in http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs/20160825.txt from 21:17:49 to 21:44:10 Key things: * Would be for windows users who can't easily do simple SSH tunnelling? * Would... [22:11:06] maybe you could avoid "linking" to it [22:11:21] there's no need for the js files to be GPL for instance [22:11:25] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584103 (10valhallasw) To recap some of the arguments from #wikimedia-labs: - ProxyCommand is complex for windows users; a VPN would allow a much more user-friendly connection (directly connecting to e.g. tools-exec-1... [22:11:39] (although I'd dual-license with GPL in that case, if there are other GPL files) [22:21:19] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584345 (10tom29739) OpenVPN * Using LDAP to authenticate would be easy: https://openvpn.net/index.php/access-server/docs/admin-guides/190-how-to-authenticate-users-with-active-directory.html * Would only carry traffic... [22:24:30] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Striker, and 2 others: Deploy "Striker" Tool Labs console to WMF production - https://phabricator.wikimedia.org/T136256#2584414 (10bd808) [22:24:56] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Striker, and 2 others: Deploy "Striker" Tool Labs console to WMF production - https://phabricator.wikimedia.org/T136256#2328692 (10bd808) [22:29:35] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584425 (10AlexMonk-WMF) >>! In T143939#2584345, @tom29739 wrote: > * Would only carry traffic to *.wmflabs hosts, not to internet. You'd need labsdb/wmnet for database replica access. I'm not quite sure how DNS would... [22:34:44] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584436 (10tom29739) >>! In T143939#2584425, @AlexMonk-WMF wrote: > You'd need labsdb/wmnet for database replica access. I'm not quite sure how DNS would work actually - the client would need to be able to resolve these... [22:35:25] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584438 (10Platonides) > Sane configuration, not allowing traffic through labs (e.g. user -> labs -> google) Note that it is possible to configure openvpn so it only provides routes to a subrange (ie. labs hosts). It c... [22:38:03] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584440 (10AlexMonk-WMF) >>! In T143939#2584438, @Platonides wrote: >> Sane configuration, not allowing traffic through labs (e.g. user -> labs -> google) > > Note that it is possible to configure openvpn so it only pr... [22:38:44] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584442 (10tom29739) >>! In T143939#2584438, @Platonides wrote: >> Sane configuration, not allowing traffic through labs (e.g. user -> labs -> google) > > Note that it is possible to configure openvpn so it only provid... [22:41:32] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584449 (10tom29739) OpenVPN server can be configured to only provide routes to certain IP ranges/hosts [22:41:34] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584450 (10Platonides) "can" as in "are technically able to" So, if it is hard to misconfigure the vpn, we could simply trust our users to do the right thing™ [22:42:06] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2584452 (10Platonides) >>! In T143939#2584449, @tom29739 wrote: > OpenVPN server can be configured to only provide routes to certain IP ranges/hosts That's the configuration I was describing [22:44:11] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/ԱշոտՏՆՂ was created, changed by ԱշոտՏՆՂ link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/%d4%b1%d5%b7%d5%b8%d5%bf%d5%8f%d5%86%d5%82 edit summary: Created page with "{{Tools Access Request |Justification=I want to run Pywikibot on Tools. |Completed=false |User Name=ԱշոտՏՆՂ }}" [22:47:44] Krenair: thanks for fixing my absent-mindedness on T143939 [22:47:45] T143939: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939 [22:48:03] tom29739, the wmnet/labsdb thing? [22:48:36] The complete cut off of the "Discussed in". [22:48:45] yeah np [22:48:58] figured you just forgot to paste a link or something [22:58:16] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/ԱշոտՏՆՂ was modified, changed by ԱշոտՏՆՂ link https://wikitech.wikimedia.org/w/index.php?diff=818387 edit summary: [22:58:32] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/ԱշոտՏՆՂ was modified, changed by ԱշոտՏՆՂ link https://wikitech.wikimedia.org/w/index.php?diff=818388 edit summary: [23:02:32] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/ԱշոտՏՆՂ was modified, changed by ԱշոտՏՆՂ link https://wikitech.wikimedia.org/w/index.php?diff=818389 edit summary: [23:03:09] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/ԱշոտՏՆՂ was modified, changed by ԱշոտՏՆՂ link https://wikitech.wikimedia.org/w/index.php?diff=818390 edit summary: [23:06:39] (03PS1) 10BryanDavis: Split mysql init command into two parts [labs/striker] - 10https://gerrit.wikimedia.org/r/306838 [23:06:55] (03CR) 10BryanDavis: [C: 032] Split mysql init command into two parts [labs/striker] - 10https://gerrit.wikimedia.org/r/306838 (owner: 10BryanDavis) [23:18:34] (03Merged) 10jenkins-bot: Split mysql init command into two parts [labs/striker] - 10https://gerrit.wikimedia.org/r/306838 (owner: 10BryanDavis) [23:18:54] 06Labs, 10Phlogiston (Interrupt): Create new Phlogiston instance for production - https://phabricator.wikimedia.org/T142277#2584550 (10JAufrecht) [23:19:04] (03PS1) 10BryanDavis: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/306841 [23:19:13] 06Labs, 10Phlogiston (Interrupt): Phlogiston-1 server is unstable and no longer able to run Phlogiston reports - https://phabricator.wikimedia.org/T141796#2584555 (10JAufrecht) [23:19:15] 06Labs, 10Phlogiston (Interrupt): Create new Phlogiston instance for production - https://phabricator.wikimedia.org/T142277#2529844 (10JAufrecht) 05Open>03Resolved a:03JAufrecht server is up, stable, and running Phlogiston reports. [23:19:17] (03CR) 10BryanDavis: [C: 032] Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/306841 (owner: 10BryanDavis) [23:19:23] (03Merged) 10jenkins-bot: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/306841 (owner: 10BryanDavis) [23:19:40] 06Labs, 10Phlogiston (Interrupt): Phlogiston-1 server is unstable and no longer able to run Phlogiston reports - https://phabricator.wikimedia.org/T141796#2512472 (10JAufrecht) 05Open>03Resolved a:03JAufrecht Resolved by bringing up phlogiston-03 as a replacement. [23:21:29] (03PS1) 10BryanDavis: Labs target changed to striker-uwsgi02 [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/306843 [23:21:39] (03CR) 10BryanDavis: [C: 032] Labs target changed to striker-uwsgi02 [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/306843 (owner: 10BryanDavis) [23:21:45] (03Merged) 10jenkins-bot: Labs target changed to striker-uwsgi02 [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/306843 (owner: 10BryanDavis) [23:25:28] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/ԱշոտՏՆՂ was modified, changed by ԱշոտՏՆՂ link https://wikitech.wikimedia.org/w/index.php?diff=818392 edit summary: [23:29:04] 06Labs, 06Editing-Analysis: Replicate editor month table to Labs - https://phabricator.wikimedia.org/T143955#2584567 (10Neil_P._Quinn_WMF) [23:54:56] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Striker, and 2 others: Deploy "Striker" Tool Labs console to WMF production - https://phabricator.wikimedia.org/T136256#2584651 (10bd808)