[01:10:38] !log clouddb-services the sync is done and we have a good copy of the toolsdb data, proceeding with the upgrades and stuff to that hypervisor while configuring replication to work again T266587 [01:10:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [01:10:42] T266587: ToolsDB replication is broken - https://phabricator.wikimedia.org/T266587 [02:14:19] !log clouddb-services toolsdb is back and so is the replica T266587 [02:14:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [02:14:22] T266587: ToolsDB replication is broken - https://phabricator.wikimedia.org/T266587 [02:22:16] !log paws Set PAWS hub back to using mariadb T266587 [02:22:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [02:22:18] T266587: ToolsDB replication is broken - https://phabricator.wikimedia.org/T266587 [04:50:59] hello [04:51:12] hello [04:52:30] hello [06:07:10] !log tools.majavah-bot re-enable tasks that were disabled due to ToolsDB maintenance [06:07:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.majavah-bot/SAL [12:13:31] !log tools created `tools-k8s-etcd` anti-affinity server group [12:13:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:15:23] !log tools created VM `tools-k8s-etcd-7` (T267966) [12:15:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:15:27] T267966: Add more k8s-etcd nodes to the cluster on tools project - https://phabricator.wikimedia.org/T267966 [12:16:05] !log tools created VM `tools-k8s-etcd-8` (T267966) [12:16:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:49:27] !log tools dropping unused hiera keys in the tools-k8s-etcd puppet prefix https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/2b4cb4a41756e602fb0996e7d0210e9102172424 [12:49:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:50:26] !log tools dropping more unused hiera keys in the tools-k8s-etcd puppet prefix https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/e9e66a6787d9b91c08cf4742a27b90b3e6d05aac [12:50:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:52:30] !log tools adding more etcd nodes in the hiera key in tools-k8s-etcd puppet prefix https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/b4f60768078eccdabdfab4cd99c7c57076de51b2 [12:52:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:54:53] !log tools joining new etcd nodes in the k8s etcd cluster (T267966) [12:54:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:54:56] T267966: Add more k8s-etcd nodes to the cluster on tools project - https://phabricator.wikimedia.org/T267966 [13:12:20] !log tools making k8s api server aware of the new etcd nodes via hiera update https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/3761c4c4dab1c3ed0ab0a1133d2ccf3df6c28baf%5E%21/ (T267966) [13:12:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:12:23] T267966: Add more k8s-etcd nodes to the cluster on tools project - https://phabricator.wikimedia.org/T267966 [13:56:44] !log tools adding etcd dns_alt_names hiera keys to the puppet prefix https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/beb27b45a74765a64552f2d4f70a40b217b4f4e9%5E%21/ [13:56:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:12:16] !log tools updated kube-apiserver manifest with new etcd nodes (T267966) [14:12:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:12:21] T267966: Add more k8s-etcd nodes to the cluster on tools project - https://phabricator.wikimedia.org/T267966 [14:15:03] !log tools regenerating puppet cert with proper alt names in tools-k8s-etcd-8 (T267966) [14:15:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:17:09] !log tools regenerating puppet cert with proper alt names in tools-k8s-etcd-7 (T267966) [14:17:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:18:58] !log tools regenerating puppet cert with proper alt names in tools-k8s-etcd-6 (T267966) [14:19:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:19:02] T267966: Add more k8s-etcd nodes to the cluster on tools project - https://phabricator.wikimedia.org/T267966 [14:21:53] !log tools regenerating puppet cert with proper alt names in tools-k8s-etcd-5 (T267966) [14:21:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:23:22] !log tools regenerating puppet cert with proper alt names in tools-k8s-etcd-4 (T267966) [14:23:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:24:00] !log tools.quickcategories removed expected_database_error from config, ToolsDB maintenance is over [17:24:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [18:58:28] !log tools disabling puppet on k8s-etcd servers to alter the timeouts T267966 [18:58:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:58:33] T267966: Add more k8s-etcd nodes to the cluster on tools project - https://phabricator.wikimedia.org/T267966 [19:44:06] !log tools set etcd timeouts seed value to 20 instead of the default 10 (profile::wmcs::kubeadm::etcd_latency_ms) T267966 [19:44:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:44:10] T267966: Add more k8s-etcd nodes to the cluster on tools project - https://phabricator.wikimedia.org/T267966 [19:56:18] !log tools puppet enabled one at a time, letting things catch up. Timeouts are now adjusted to something closer to fsync values T267966 [19:56:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:56:22] T267966: Add more k8s-etcd nodes to the cluster on tools project - https://phabricator.wikimedia.org/T267966 [20:35:20] bd808: newbie question here. How do I find out who maintains tools.wmflabs.org/cssk/scripts/cssk.js ? [20:35:53] Jdlrobson: https://admin.toolforge.org/tool/cssk [20:36:01] * Jdlrobson bookmarks [20:36:02] thanks! [21:42:16] !log tools doing the same procedure to increase the timeouts more T267966 [21:42:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:42:22] T267966: Add more k8s-etcd nodes to the cluster on tools project - https://phabricator.wikimedia.org/T267966 [21:57:43] Neat. Wmflabs.org (and a few other related domains) ended up on a pi-hole blacklist for some reason: https://dbl.oisd.nl (warning - HUGE page). I made a false positive report here: https://oisd.nl/?p=fp [22:10:26] !log admin setting autoscale to 'warn' for both ceph pools (eqiad1-compute and eqiad1-glance-images) [22:10:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:14:45] !log admin setting pg number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) T270305 [22:14:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:14:50] T270305: Ceph performance tuning - https://phabricator.wikimedia.org/T270305 [22:16:07] !log admin setting pgp number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) T270305 (same as pg) [22:16:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:17:24] !log admin correction to above, set the pg and pgp to 1024 for eqiad1-glance-images [22:17:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL