[03:05:29] <wikibugs>	 10serviceops, 10Gerrit, 10Operations, 10Release-Engineering-Team-TODO, and 2 others: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster) - https://phabricator.wikimedia.org/T222391 (10Dzahn)
[04:42:29] <wikibugs>	 10serviceops, 10Operations: Confd died on bast3002 - https://phabricator.wikimedia.org/T227592 (10jijiki) 05Open→03Resolved a:03jijiki It has not happened again, Resolving for now.
[05:43:52] <wikibugs>	 10serviceops, 10MediaWiki-extensions-Mailgun, 10Operations, 10cloud-services-team, and 5 others: Switch cronjobs on maintenance hosts to PHP7 - https://phabricator.wikimedia.org/T195392 (10jijiki)
[05:44:17] <wikibugs>	 10serviceops, 10Operations: SRE FY19-20 Q1 goal: complete the transition to PHP7 - https://phabricator.wikimedia.org/T219127 (10jijiki)
[05:44:23] <wikibugs>	 10serviceops, 10MediaWiki-extensions-Mailgun, 10Operations, 10cloud-services-team, and 5 others: Switch cronjobs on maintenance hosts to PHP7 - https://phabricator.wikimedia.org/T195392 (10jijiki) 05Open→03Resolved
[09:18:42] <wikibugs>	 10serviceops, 10Phabricator, 10Release-Engineering-Team-TODO, 10User-MModell: Mukunda to set up a meeting with service ops to discuss operational best practices for phabricator - https://phabricator.wikimedia.org/T232058 (10mmodell) 05Open→03Resolved
[14:21:05] <akosiaris>	 _joe_: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/536593/ I am gonna enable coreDNS for codfw
[14:21:24] <_joe_>	 that should have no impact right?
[14:21:24] <akosiaris>	 I 've got it running with 4 replicas, forcefully set to 1 per rack row
[14:21:28] <_joe_>	 we're not using it
[14:21:40] <akosiaris>	 it will have all pods move through it 
[14:21:45] <akosiaris>	 so yes, it might have impact
[14:21:48] <_joe_>	 oh you mean to have it manage all queries
[14:21:51] <akosiaris>	 all pods DNS traffic
[14:21:55] <_joe_>	 so, on friday evening?
[14:21:56] <_joe_>	 ok!
[14:21:59] <akosiaris>	 lol
[14:22:01] <_joe_>	 lemme clock out first though
[14:22:18] <akosiaris>	 obviously not today, just letting you know
[14:22:37] <akosiaris>	 for now I 'll do the same for eqiad so we are ready for both
[14:23:15] <_joe_>	 cool
[15:44:22] <akosiaris>	 wikifeeds might be deployed on Monday
[18:36:04] <mutante>	 in a meeting with Mukunda on Phabricator plans
[18:54:45] <wikibugs>	 10serviceops, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) 05Open→03Stalled
[18:54:56] <wikibugs>	 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, and 3 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832 (10Dzahn)
[18:56:11] <wikibugs>	 10serviceops, 10Operations, 10Phabricator, 10hardware-requests, 10Release-Engineering-Team (Development services): The server, WMF7426, was given to us temporarily, we would like to make it permanent - https://phabricator.wikimedia.org/T232887 (10mmodell)
[18:56:33] <wikibugs>	 10serviceops, 10Operations, 10Phabricator, 10hardware-requests, 10Release-Engineering-Team (Development services): The phabricator server, WMF7426, was given to us temporarily, we would like to make it permanent - https://phabricator.wikimedia.org/T232887 (10mmodell)
[19:01:55] <wikibugs>	 10serviceops, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Reimage both phab1001 and phab2001 to stretch / buster - https://phabricator.wikimedia.org/T190568 (10Dzahn)
[19:22:32] <wikibugs>	 10serviceops, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Reimage both phab1001 and phab2001 to stretch / buster - https://phabricator.wikimedia.org/T190568 (10Dzahn)
[22:13:44] <mutante>	 paladox: got a sec 
[22:13:52] <paladox>	 yup
[22:14:02] <mutante>	 can you go back to that cloud VPS .. eh.. phab-10 ?
[22:14:16] <mutante>	 the one with the gerrit role despite the name :)
[22:14:45] <mutante>	 and run 'file /var/lib/gerrit2/review_site/lib/mysql-connector-java.jar'
[22:15:16] <paladox>	 yup
[22:15:18] <paladox>	 "ls: cannot access '/var/lib/gerrit2/review_site/lib/mysql-connector-java.jar': No such file or directory"
[22:15:49] <mutante>	 interesting.. shouldnt puppet have created that before i merged the change.. probably because it failed in other ways still
[22:16:01] <mutante>	 ok, otherwise i would have asked you to unlink it now and run puppet again
[22:16:28] <mutante>	 how does the puppet run look now
[22:16:43] <paladox>	 it runs buster, so it would have failed
[22:16:49] <paladox>	 because the package would not install
[22:16:53] <paladox>	 thus symnlink would fail?
[22:17:02] <mutante>	 ah, dependencies. yes
[22:18:13] <mutante>	 do you see new errors now?
[22:19:33] <paladox>	 mutante yup
[22:20:12] <paladox>	 v
[22:20:14] <paladox>	 * https://phabricator.wikimedia.org/P9106
[22:21:47] <mutante>	  Fetch from: http://phab-tin.phabricator.eqiad.wmflabs/g
[22:21:53] <mutante>	 Host key verification failed.
[22:22:09] <mutante>	 rsync error: unexplained error
[22:22:13] <paladox>	 yup
[22:23:14] <mutante>	 wanna run the scap command manually. /usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False
[22:23:26] <mutante>	 did something happen with phab-tin?
[22:23:48] <paladox>	 mutante it uses ssh.
[22:23:57] <paladox>	 I had to do a nasty hack that used my user to do the ssh.
[22:24:07] <paladox>	 (for gerrit-test5 & phabricator)
[22:24:15] <mutante>	 oh, it's the "git fat pull" part?
[22:24:29] <mutante>	 why was there git fat again
[22:25:11] <mutante>	 anyways, Host key verification failed is pretty obvious
[22:25:23] <mutante>	 and re: "nasty hack" eh... 
[22:25:27] <paladox>	 git fat is for the binary
[22:29:27] <mutante>	 the host keys are puppetized
[22:29:31] <mutante>	 wont work in labs with those 
[22:47:45] <wikibugs>	 10serviceops, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, and 2 others: Reimage both phab1001 and phab2001 to stretch / buster - https://phabricator.wikimedia.org/T190568 (10Dzahn) 05Stalled→03Open
[22:47:51] <wikibugs>	 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, and 3 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832 (10Dzahn)
[23:12:56] <wikibugs>	 10serviceops, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Reimage both phab1001 and phab2001 to stretch / buster - https://phabricator.wikimedia.org/T190568 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin10...
[23:13:40] <mutante>	 reinstalling phab1001 (jessie) with buster
[23:59:46] <wikibugs>	 10serviceops, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Reimage both phab1001 and phab2001 to stretch / buster - https://phabricator.wikimedia.org/T190568 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['phab1001.eqiad.wmne...