[00:10:35] <mutante>	 !log wikistats instance -kraken, upgrade php7 and mariadb packages
[00:10:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL
[01:37:24] <andrewbogott>	 !log wikistream rebooting ws-web, enabling puppet, truncating syslog, etc.  Drive is full so I can't make it any worse
[01:37:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistream/SAL
[02:17:33] <andrewbogott>	 !log math rebooting drmf in an attempt to get puppet working
[02:17:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Math/SAL
[07:02:56] <jynus>	 wikidata may get outdated for some minutes while we setup the new s8 group of servers on labs
[13:43:20] <chasemp>	 thanks jynus!
[15:05:23] <andrewbogott>	 milimetric: is the 'dashiki' project still doing anything?  I'm chasing down some puppet issues on dashiki-01 and I'm not clear on if it ever really worked.
[15:09:28] <milimetric>	 andrewbogott: yeah, it hosts a bunch of dashboards
[15:09:51] <andrewbogott>	 ok, but apache is broken on dashiki-01.  Maybe just that instance is stale?
[15:10:07] <milimetric>	 We have some hiera that configures apache sites on there to point to static js/html that we just copy up
[15:10:21] <milimetric>	 hm... lemme check the dashboards
[15:10:33] <andrewbogott>	 maybe I broke the apache by un-breaking puppet
[15:10:40] <milimetric>	 yep, dashboards are broken
[15:11:05] <milimetric>	 They were fine last week, but we don’t check them regularly
[15:11:17] <andrewbogott>	 ok — well, puppet was failing to load on both of those VMs.  Can you (or someone) fix that so I don't have to?
[15:11:44] <milimetric>	 sure, I will try, thanks for the heads up
[15:13:16] <andrewbogott>	 milimetric: the primary issues seems to be passing in [] to role::simplestatic so it doesn't actually configure any hosts
[15:14:05] <milimetric>	 yeah, I have no idea how that was set up, madhu did it, I’ll look around though
[17:12:15] <bd808>	 arturo: ^ topic changed so it's 'official' :)
[17:12:28] <arturo>	 great :-)
[19:21:05] <wikibugs>	 (03PS203) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296
[19:25:03] <wikibugs>	 (03CR) 10Ricordisamoa: [C: 04-2] "PS203 updates rollup from 0.53.0 to 0.53.3" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa)
[19:25:59] <wikibugs>	 (03PS204) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296
[19:30:37] <wikibugs>	 (03CR) 10Ricordisamoa: [C: 04-2] "PS204 renames options in rollup.config.js" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa)
[19:42:05] <chasemp>	 !log tools reboot tools-worker-1010
[19:42:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:50:47] <chasemp>	 !log tools clush -w @all 'sudo puppet agent --test'
[19:50:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:15:50] <chasemp>	 !log tools disable puppet on proxies and k8s workers
[20:15:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:32:05] <mutante>	 !log wikidata-dev fixed puppet runs on wikibase-stretch, wikibase-vue, wikibase with https://gerrit.wikimedia.org/r/#/c/403232/
[20:32:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL
[20:34:43] <mutante>	 !log wikidata-dev wikibase-vue cant start Apache because docker-proxy is already using port 80
[20:34:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL
[20:34:51] <mutante>	 andrewbogott: ^ that issue is unrelated
[20:35:24] <mutante>	 it doesnt fully stop puppet though, just fails to start webserver
[20:51:49] <chasemp>	 !log tools kubectl cordon tools-worker-1001.tools.eqiad.wmflabs
[20:51:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:55:58] <chasemp>	 !log tools for n in `kubectl get nodes | awk '{print $1}' | grep -v -e tools-worker-1001  -e tools-worker-1016`; do kubectl cordon $n; done
[20:56:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[21:10:15] <chasemp>	 !log tools tools-k8s-master-01:~# for n in `kubectl get nodes | awk '{print $1}' | grep -v -e tools-worker-1001  -e tools-worker-1016 -e tools-worker-1028 -e tools-worker-1029 `; do kubectl uncordon $n; done
[21:10:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[21:17:25] <chasemp>	 !log tools tools-clushmaster-01:~$ clush -f 1 -w @k8s-worker "sudo puppet agent --enable --test"
[21:17:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[21:17:54] <chasemp>	 !log tools ...rush@tools-clushmaster-01:~$ clush -f 1 -w @k8s-worker "sudo puppet agent --enable && sudo puppet agent --test"
[21:17:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:19:40] <Hauskatze>	 some reports of PAWS down, yuvipanda told me to let you all know :)
[22:21:14] <chasemp>	 ^ madhuvishy can you take a look and see if anything makes sesne in relation to ferm or such?
[22:22:07] <madhuvishy>	 Hauskatze: hmmm, is up for me, who is reporting as down?
[22:23:05] <madhuvishy>	 chasemp: did you check the iptables stuff in paws after the ferm patch roll out?
[22:23:17] <chasemp>	 madhuvishy: I didn't think of it
[22:23:20] <madhuvishy>	 probably broke paws
[22:23:24] <madhuvishy>	 i'll look no worries
[22:23:34] <chasemp>	 madhuvishy: how though? ferm isn't even on the paws nodes afaict
[22:24:22] <bd808>	 we're not sure, but it definately seemed related on Friday
[22:24:36] <chasemp>	 I'm not pushing back so much as totally confused
[22:24:38] <madhuvishy>	 chasemp: unsure, but I noticed last time that when we do these ferm changes it also makes the paws iptables rules FORWARD chain to change to DROP on all the workers
[22:24:45] <chasemp>	 hm
[22:25:49] <Hauskatze>	 madhuvishy: T184566 & T184500
[22:25:50] <stashbot>	 T184500: PAWS is down again - https://phabricator.wikimedia.org/T184500
[22:25:50] <stashbot>	 T184566: As usual.... PAWS is down - https://phabricator.wikimedia.org/T184566
[22:26:50] <bd808>	 I'm not very motivated to fix bugs with titles like "As usual.... PAWS is down".
[22:27:03] <madhuvishy>	 they made another ticket too
[22:27:20] <madhuvishy>	 I think this may just be tools-paws-worker-1007
[22:27:25] <madhuvishy>	 that I rebooted earlier today
[22:27:36] <madhuvishy>	 i assume it came up with the iptables rule changed
[22:27:47] <madhuvishy>	 so may be not related to the ferm stuff
[22:29:38] <madhuvishy>	 chasemp: ^
[22:32:29] <Zppix>	 Tbh PAWS is broken alot but yes task title is very encouraging :P
[22:32:39] <Zppix>	 Isnt*
[22:35:58] <Hauskatze>	 bd808: I agree wrt title
[22:37:18] <bd808>	 3 months from now we may be ready to spend more time with PAWS. Today we are barely able to keep the rest of the world running. :/
[22:37:45] <bd808>	 it turns out that we have a lot of moving parts to keep moving around here ;)
[22:39:17] <Hauskatze>	 wonder if somebody could have a minute to +2 https://gerrit.wikimedia.org/r/#/c/402580/
[22:40:02] <Zppix>	 bd808: lot of moving parts or not alot of people moving them?
[22:41:37] <harej>	 A lot of software components that move in eccentric and sometimes unpredictable ways.
[22:42:21] <bd808>	 and we were down 1 staff member for ~9 months
[22:46:13] <yuvipanda>	 !log run clush -w tools-paws-worker-10[01-20] 'sudo bash /home/yuvipanda/kubeadm-bootstrap/remove-worker.bash' to completely destroy the paws k8s cluster
[22:46:14] <stashbot>	 yuvipanda: Unknown project "run"
[22:46:35] <madhuvishy>	 !log tools run clush -w tools-paws-worker-10[01-20] 'sudo bash /home/yuvipanda/kubeadm-bootstrap/remove-worker.bash' to completely destroy the paws k8s cluster
[22:46:37] <yuvipanda>	 !log tools run clush -w tools-paws-worker-10[01-20] 'sudo bash /home/yuvipanda/kubeadm-bootstrap/remove-worker.bash' to completely destroy the paws k8s cluster
[22:46:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:46:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:46:47] <madhuvishy>	 :)
[22:46:56] <yuvipanda>	 !log tools reboot all paws-worker nodes
[22:47:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:47:12] <yuvipanda>	 awww, ty madhuvishy :) didn't wanna burden y'all, so I'll come here just to log for a bit and then disappear
[22:47:16] <Hauskatze>	 complicated commands :|
[22:48:30] <yuvipanda>	 !log tools run 'clush -w tools-paws-worker-10[01-20] 'sudo bash /home/yuvipanda/kubeadm-bootstrap/install-kubeadm.bash'' to setup kubeadm on all paws worker nodes
[22:48:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:49:05] <yuvipanda>	 !log tools run  clush -w tools-paws-worker-10[01-20] 'sudo bash /home/yuvipanda/kubeadm-bootstrap/init-worker.bash' to bring paws workers back up again, but as 1.8
[22:49:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:53:20] <yuvipanda>	 !log tools redo tools-paws-worker-1006 manually, since clush seems to have missed it for some reason
[22:53:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:54:05] <yuvipanda>	 !log tools kill all PAWS pods
[22:54:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:54:37] <yuvipanda>	 !log tools kill all kube-system pods in paws cluster
[22:54:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:57:23] <bd808>	 !log tool.replag Deployed be6109b (add s8 slice)
[22:57:23] <stashbot>	 bd808: Unknown project "tool.replag"
[22:57:30] <bd808>	 !log tools.replag Deployed be6109b (add s8 slice)
[22:57:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.replag/SAL
[23:01:26] <yuvipanda>	 !log tools kill paws master and reboot it
[23:01:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:08:23] <yuvipanda>	 !log tools turns out the version of k8s we had wasn't recent enough to support easy upgrades, so destroy entire cluster again and install 1.9.1
[23:08:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:21:27] <yuvipanda>	 !log tools paws new cluster master is up, re-adding nodes by executing same sequence of commands for upgrading
[23:21:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL