[00:20:13] !log tools.zppixbot restarting for code and config changes [00:20:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [07:35:06] !log tools.zppixbot-test tools.zppixbot-test@tools-sgebastion-07:~/public_html/new-web/public_html/Font-Awesome$ git submodule init && git submodule update [07:35:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [07:35:16] Reception123: ^ beat ya to it [07:39:38] heh [07:46:02] Reception123: I'm going to try and fix it [07:47:51] ok [08:47:32] !log tools.wudele Hacking lighttpd config for removing spurious URL parameters [08:47:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wudele/SAL [09:10:47] !log mwoffliner refresh project quotas: 56 vCPUs (from 44) 96GB RAM (from 88) and 1 floating IP (from 3) (T253836) [09:10:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mwoffliner/SAL [09:10:49] T253836: Update quotas for MWoffliner VPS - https://phabricator.wikimedia.org/T253836 [09:13:51] !log mwoffliner refresh project quotas: 52 vCPUs (typo/wrong math in previous quota) (T253836) [09:13:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mwoffliner/SAL [09:27:38] !log tools.zppixbot deploy a bunch of changes so I can test the new logo on the wiki [09:27:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [09:40:13] !log tools.zppixbot remove logo after test for https://phabricator.wikimedia.org/T254432#6191935 [09:40:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [09:43:59] !log tools.wikibugs Restarting wikibugs, not listening to gerrit [09:44:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [09:57:45] !log tools.wikibugs Restarting wb2-grrrrit, still not parsing gerrit feed [10:00:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [12:02:09] !log paws created 2 medium VM instances: paws-k8s-ingress-1 and paws-k8s-ingress-2 with haproxy anti-affinity (T195217) [12:02:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [12:02:11] T195217: Sane ingress methods for PAWS - https://phabricator.wikimedia.org/T195217 [12:04:50] !log paws created `paws-k8s-ingress` puppet prefix and add the `role::wmcs::paws::k8s::worker` role (T195217) [12:04:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [12:18:45] !log paws bootstrapped paws-k8s-ingress nodes, added them to the k8s cluster (T195217) [12:18:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [12:18:49] T195217: Sane ingress methods for PAWS - https://phabricator.wikimedia.org/T195217 [12:28:05] !log paws added node taints to ingress nodes: `kubectl taint nodes paws-k8s-ingress-1 ingress=true:NoSchedule` (T195217) [12:28:32] !log paws added node taints to ingress nodes: `kubectl taint nodes paws-k8s-ingress-1 ingress=true:NoSchedule` (T195217) [12:28:41] :-/ stashbot is sad? [12:31:59] WTH? [12:32:36] we don't have any [local] commit in labs/private in tools-puppetmaster-02 [12:43:45] arturo: just noticed https://sal.toolforge.org/ is empty, and given stashbot's madness, is it possible there's something happening in the infrastructure side? [12:44:11] Urbanecm: yes, I already identified the issue [12:44:15] okay, cool! [12:44:30] unfortunately, no quick fix. Trying to asses the situation now [13:08:01] !log admin disabling puppet on all instances for /labs/private recovery [13:15:15] !log tools.zppixbot turn logo back on [13:15:38] (/me will log again later) [13:32:07] !log tools Manually restored /etc/haproxy/conf.d/elastic.cfg on tools-elastic-* [13:32:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:00:55] !log tools.zppixbot 14:15:15 !log tools.zppixbot turn logo back on [14:00:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [14:16:17] !log paws added node taints to ingress nodes: `kubectl taint nodes paws-k8s-ingress-1 ingress=true:NoSchedule` (T195217) [14:16:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [14:16:20] T195217: Sane ingress methods for PAWS - https://phabricator.wikimedia.org/T195217 [14:18:24] andrewbogott: note that your message didn't get logged either [14:23:00] !log admin disabling puppet on all instances for /labs/private recovery [14:23:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:23:09] MacFan4000: someone should probably try and baclfill the ops ones as well [14:24:32] !log admin disabling puppet on all instances for /labs/private recovery [14:24:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:25:14] now its there twice ;) [14:49:46] !log tools.zppixbot-test restart for config and code changes [14:49:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [15:02:43] !log tools.zppixbot restart for config and code changes [15:02:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [15:45:31] !log tools.zppixbot restarting web service as php7.3 [15:45:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [15:48:15] !log tools.zppixbot-test restarting web service as php7.3 [15:48:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [15:50:20] !log tools.zppixbot git pull in public_html/wiki [15:50:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [17:41:25] !log clouddb-services Restarting mariadb after hand applying {{gerrit|602433}} (T253738) [17:41:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [17:41:28] T253738: ToolsDB: master crashed, replica having consistency issues - https://phabricator.wikimedia.org/T253738 [20:07:49] !log maps Checked all hosts and found no missing secrets; puppet re-enabled (T254491) [20:07:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Maps/SAL [20:07:51] T254491: Puppet labs/private.git data loss incident affecting some projects - https://phabricator.wikimedia.org/T254491 [20:52:49] !log shinken deleting shinken-buster VM because it was always broken and never did anything useful [20:52:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL [20:57:25] !log shinken Checked all hosts and found no missing secrets; puppet re-enabled (T254491) [20:57:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL [20:57:27] T254491: Puppet labs/private.git data loss incident affecting some projects - https://phabricator.wikimedia.org/T254491 [21:06:29] !log toolsbeta added krenair to toolsbeta.admin group in ldap [21:06:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [22:26:02] !log tools.os-deprecation Forcing run of output builder job to switch to toolforge.org links [22:26:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.os-deprecation/SAL