[05:15:59] 10serviceops, 10Growth-Team, 10Operations, 10Performance-Team, and 4 others: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10elukey) Thanks a lot for the deploy! I checked metrics and nothing seems changed :( [05:51:38] 10serviceops, 10DBA, 10Operations, 10Performance-Team, and 2 others: Increased instability in MediaWiki backends (according to load balancers) - https://phabricator.wikimedia.org/T223952 (10Marostegui) Update from the subtask related to the slow query detected (which might or might not be the cause for thi... [06:11:29] 10serviceops, 10DBA, 10Operations, 10Performance-Team, 10HHVM: Increased instability in MediaWiki backends (according to load balancers) - https://phabricator.wikimedia.org/T223952 (10Joe) [06:20:48] jijiki: hi! wanna switch something else today? [06:21:51] Eeny, meeny, miny moe, which one is the lucky job? [06:22:19] jijiki: wikibase-addUsagesForPage is the highest-traffic job we have.. [06:22:37] should I try and upload a patch ? [06:22:50] oh wait [06:22:55] why not try an easier one? [06:22:58] :) [06:23:14] jijiki: I donno, is that interesting? [06:23:55] ofc it is more interesting migrating bigger jobs [06:24:11] let's hope we wont break stuff [06:24:56] on the other hand I have a dentist appointment today and need to go to city center for this, so I won't be around for some time mid day to help.. [06:25:22] I will be here for another hour and then I will be back later [06:25:31] we can do a smaller one now if you can [06:25:43] ok, then let's reconvien when we are both back [06:25:55] deal! [06:26:17] I will try and upload a patch for wikibase-addUsagesForPage [06:26:21] in the mean time [06:26:31] I'm not sure what our strategy is though.. Are we trying to switch one-by-one, or are we trying to switch all the harder ones paying high attention to them and then just flip a switch on all the rest? [06:27:12] if the strategy is the latter, we only have maybe 5 steps left [06:27:55] we started with one my one [06:28:04] and we got confident along the way [06:28:30] I like the second approach [06:28:56] I will have to run it through my partners in crime [06:29:07] so I will know by the time we are both back [06:29:22] then we have wikibase -> recentChanges -> cirrus -> refreshLinks -> video transcode -> thumbnail render -> all the rest [06:29:40] oh cirrus is very busy yes [06:29:57] cirrus is less busy then wikibase, but it's very weird [06:31:17] we'll throw the ball to search if things go south :p [07:02:24] !phabric [07:02:27] arg [08:56:29] 10serviceops, 10Operations, 10WMDE-QWERTY-Team, 10wikidiff2, 10WMDE-QWERTY-Sprint-2019-05-15: Deploy Wikidiff2 version 1.8.2 with the timeout issue fixed - https://phabricator.wikimedia.org/T223391 (10thiemowmde) @Tobi_WMDE_SW, can you take care of this and possibly assign people who are able to deploy t... [13:13:59] akosiaris: what if anything remains for T220401? [13:14:49] akosiaris: also, I'd be curious about general steps required to replicate what you did with minikube [13:15:06] urandom: I am at kubecon EU currently, I 'll take a look later and let you know. [13:15:21] akosiaris: oh, cool! No rush [13:18:03] urandom: but overall, 1) helm repo add wmf https://releases.wikimedia.org/charts/ 2) helm init 3) helm upgrade --install --set main_app.version=2019-04-08-182317-production testkask wmf/kask after you get minikube running should give you (after a while) a working environment) [13:18:53] akosiaris: great; I'll give that a try [13:42:33] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, and 4 others: Kask integration testing with Cassandra via the Deployment Pipeline - https://phabricator.wikimedia.org/T224041 (10akosiaris) >>! In T224041#5201573, @thcipriani wrote: > It seems that the cassandra subchart already e... [15:04:09] 10serviceops, 10Operations, 10User-jijiki: Ramp up percentage of users on php7.2 to 100% on both API and appserver clusters - https://phabricator.wikimedia.org/T219150 (10Jdforrester-WMF) [15:45:33] 10serviceops, 10Operations, 10Core Platform Team Backlog (Later), 10Services (next): Migrate node-based services in production to node10 - https://phabricator.wikimedia.org/T210704 (10Maintenance_bot) [15:48:34] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Performance-Team (Radar), and 2 others: Apply -R 200 to all the memcached mw object cache instances running in eqiad/codfw - https://phabricator.wikimedia.org/T208844 (10Maintenance_bot) [20:38:59] 10serviceops, 10Gerrit, 10Operations, 10ops-eqiad, 10Release-Engineering-Team (Watching / External): Gerrit Hardware Upgrade - https://phabricator.wikimedia.org/T222391 (10Dzahn) 05Open→03Stalled [21:29:59] is there a buster base Docker image yet? [21:33:15] Nop, is pretty easy to create one but also buster have not been released [21:33:20] Yet [21:34:18] *nod* I knew it was still in freeze upstream. Just day dreaming about a python 3.6 container for Toolforge :) [21:40:22] bd808: 3.7? [21:40:29] stretch 3.5, buster 3.7 [21:40:40] volans: that would be nice, but Buster froze with 3.6 [21:41:08] If you can convince m.oritz to backport a 3.7 though that would be stellar [21:41:35] https://packages.debian.org/buster/python3 [21:42:47] also we have available all py3 packages also for stretch, mostly to be used within a venv [21:44:59] https://tools.wmflabs.org/apt-browser/stretch-wikimedia/component/pyall/ [21:45:06] bd808 ^^ [21:45:57] volans: oh wow! [21:46:36] I'll have to play with that and see if I can get a 3.7 stretch image working [21:47:03] the idea is that those stretch packages are not meant to be used as a replacement of OS shipped Python [21:47:11] because of potential subtle incompatibilities [21:47:32] but makes a perfect fit for venv stuff where your deps are surely compatible [21:47:42] *OS shipped Python packages [21:48:11] so not sure if they fit the container idea [21:48:22] anyway Buster has 3.7 for all I know [21:57:06] TIL. I'd heard before that 3.6 was all that made the freeze but having 3.7 will be much nicer I think. Lots of folks at the hackathon were bugging me for f-string support :) [22:01:37] yes, buster has 3.7 [22:02:15] the py3.4/3.6/3.7 packages for stretch are lagging a bit behind [22:02:25] incl. some CVEs I think [22:02:50] and they wouldn't work with any in-Debian dependencies, they're really only useful with venvs [22:03:50] 10serviceops, 10Operations: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 (10Dzahn) [22:05:12] paravoid: luckily the use case in Toolforge Kubernetes containers is very much venv driven [22:06:18] 10serviceops, 10Operations: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 (10Dzahn) [22:07:16] 10serviceops, 10Operations: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 (10Dzahn) [22:08:35] 10serviceops, 10Operations: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 (10Dzahn) [22:12:09] 10serviceops, 10Operations: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 (10Dzahn) ~~test~~ [22:13:04] 10serviceops, 10Operations: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 (10Dzahn) [22:21:53] 10serviceops, 10Operations: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 (10Dzahn) [22:23:03] 10serviceops, 10Operations: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 (10Dzahn) [22:29:25] 10serviceops, 10Operations: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 (10Dzahn) grafana is not on this host anymore meanwhile. unlinking subtask , not blocking this anymore [22:44:16] 10serviceops, 10Operations, 10PHP 7.2 support: switch webserver_misc_apps to PHP 7.2 - https://phabricator.wikimedia.org/T224194 (10Dzahn)