[00:37:33] 10serviceops, 10MediaWiki-General-or-Unknown, 10Operations, 10Core Platform Team (PHP7 (TEC4)), and 4 others: Some pages will become completely unreachable after PHP7 update due to Unicode changes - https://phabricator.wikimedia.org/T219279 (10Krinkle) [00:38:13] 10serviceops, 10MediaWiki-General-or-Unknown, 10Operations, 10Core Platform Team (PHP7 (TEC4)), and 4 others: Some pages will become completely unreachable after PHP7 update due to Unicode changes - https://phabricator.wikimedia.org/T219279 (10Krinkle) [00:38:37] 10serviceops, 10MediaWiki-General-or-Unknown, 10Operations, 10Core Platform Team (PHP7 (TEC4)), and 4 others: Some pages will become completely unreachable after PHP7 update due to Unicode changes - https://phabricator.wikimedia.org/T219279 (10Krinkle) > for example https://en.wikipedia.org/w/index.php?tit... [02:38:09] 10serviceops, 10Operations, 10hardware-requests: requesting WMF7426 as phabricator system in eqiad - https://phabricator.wikimedia.org/T215335 (10Dzahn) [02:38:12] 10serviceops, 10Operations, 10Patch-For-Review: setup/install WMF7426 as phab1003.eqiad.wmnet - https://phabricator.wikimedia.org/T221389 (10Dzahn) 05Open→03Resolved [02:39:47] 10serviceops, 10Operations, 10Patch-For-Review: setup/install WMF7426 as phab1003.eqiad.wmnet - https://phabricator.wikimedia.org/T221389 (10Dzahn) 
 2019-05-23      02:35 mutante: phabricator - going read-write again     02:24 twentyafterfour: manually started aphlict on phab1003     02:06 dzahn@cumin10...
[02:39:57] 	 10serviceops, 10MediaWiki-General-or-Unknown, 10Operations, 10Core Platform Team (PHP7 (TEC4)), and 4 others: Some pages will become completely unreachable after PHP7 update due to Unicode changes - https://phabricator.wikimedia.org/T219279 (10Anomie) >>! In T219279#5207043, @Krinkle wrote: > This now work...
[04:50:56] 	 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) 05Stalled→03Open a:03Dzahn
[04:51:08] 	 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn)
[04:55:28] 	 10serviceops, 10Operations, 10Patch-For-Review: setup/install WMF7426 as phab1003.eqiad.wmnet - https://phabricator.wikimedia.org/T221389 (10Dzahn)
[04:59:37] 	 10serviceops, 10Operations, 10Patch-For-Review: setup/install WMF7426 as phab1003.eqiad.wmnet - https://phabricator.wikimedia.org/T221389 (10Dzahn)
[05:03:21] 	 and that was the last step for phabricator migration... the SPF records in DNS 
[05:03:32] 	 on PHP 7.2 / stretch / fpm now ... and good night
[05:34:58] 	 congrats!
[06:17:32] <_joe_>	 mutante: great
[07:50:15] 	 10serviceops, 10Operations, 10Performance-Team, 10HHVM, 10User-Marostegui: Increased instability in MediaWiki backends (according to load balancers) - https://phabricator.wikimedia.org/T223952 (10Marostegui) I am removing the DBA tag for now here (but remain subscribed), as we are following up on {T22401...
[11:14:12] 	 10serviceops, 10Operations, 10Performance-Team, 10HHVM, 10User-Marostegui: Increased instability in MediaWiki backends (according to load balancers) - https://phabricator.wikimedia.org/T223952 (10Marostegui) The slow query that is being investigated at T224017 is only reported as slow when `'rev_page'= '...
[13:36:36] <_joe_>	 mark, apergos, mutante it's us 4 today, we're down 3 people. Should we postpone our meeting, or cancel it?
[13:36:50] 	 huh
[13:37:05] 	 if mark has updates from annual plan thta might benice to hear
[13:37:13] 	 I don't have ynthing urgent myself though
[13:37:15] 	 you?
[13:37:28] <_joe_>	 nope
[13:37:47] 	 mutante is not going to be on at this hour
[13:37:50] 	 6:30 am...
[13:38:01] 	 so we and mark should make a command decision
[13:38:13] 	 i don't have much either
[13:38:18] 	 we're being asked to reduce our budget
[13:38:25] 	 which is not a surprise
[13:38:31] 	 this is my 'amazing disbelief' face
[13:38:56] 	 so negotiations continue in other words
[13:38:58] 	 but things will probably keep changing in the next few days
[13:39:00] 	 yeah
[13:39:09] 	 ok, sounds like we might as well cancel for today
[13:39:09] 	 so whatever I can tell you now may be irrelevant tomorrow
[13:39:09] <_joe_>	 we'll do what we can do within the new restrictions I guess. Some projects will be postponed to when we have budget
[13:39:26] 	 indeed
[13:39:33] 	 that's the idea
[13:39:46] 	 which is at least better than "do everything anyway!" ;)
[13:39:55] 	 yep
[13:39:56] <_joe_>	 like, I'm not sure parsoid/php can be done without the appropriate number of application servers added.
[13:39:58] <_joe_>	 yes
[13:40:08] 	 well yeah
[13:40:19] 	 i'm not too worried about capex, although they're asking for reductions there too
[13:40:27] <_joe_>	 oh I see
[13:40:28] 	 some will get cut but probably not that
[13:40:41] 	 that's pretty core and even part of our strategic plan that seems to be making it
[13:40:45] 	 "reduce complexity of mediawiki"... ;)
[13:41:53] 	 so you want to mark today's meeting as cancelled, mark? I think you're the owner, no?
[13:42:05] 	 ok I can do so now
[13:42:15] 	 ty
[15:05:40] 	 10serviceops, 10Operations, 10Release Pipeline, 10Services, and 4 others: Deploy the RESTBase front-end service (RESTRouter) to Kubernetes - https://phabricator.wikimedia.org/T223953 (10mobrovac)
[15:22:40] 	 thanks for the congrats. and yes, i also like we canceled the meeting. replied by email. watching the W3C thing and then be back later
[16:04:56] 	 worker processes leaking memory on phab ... fixed by upgrade as we were hoping for :)
[16:04:59] 	 https://grafana.wikimedia.org/d/000000587/phabricator?orgId=1&panelId=7&fullscreen&from=now-30d&to=now
[16:14:48] 	 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn)
[16:15:31] 	 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) Phabricator has been switched to phab1003 as the prod server now and that meant:  - php 5 to...
[16:16:32] 	 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) I am thinking now we could make the process easier and just keep phab1003 as the prod server...
[16:18:28] 	 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10mmodell) @dzahn: agreed. I don't know who should decide if we keep phab1001 or return it. We've pro...
[16:19:51] 	 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) I will bring this up in my next subteam discussion meeting which should be in a week. Until...
[16:32:02] 	 10serviceops, 10Operations, 10Wikimedia-Site-requests, 10Patch-For-Review, and 2 others: Increase Memory Limit for Scribunto - https://phabricator.wikimedia.org/T223737 (10Anomie) I have no opinion. I don't think the existing 50MB limit was chosen for any specific reason.
[20:12:16] 	 10serviceops, 10Operations, 10PHP 7.2 support, 10Patch-For-Review: switch wikitech to PHP 7.2 - https://phabricator.wikimedia.org/T223393 (10bd808)
[20:12:37] 	 10serviceops, 10Operations, 10wikitech.wikimedia.org, 10PHP 7.2 support, 10Patch-For-Review: switch wikitech to PHP 7.2 - https://phabricator.wikimedia.org/T223393 (10bd808)
[20:49:54] 	 10serviceops, 10Operations: upgrade and rename krypton & create its codfw equivalent - https://phabricator.wikimedia.org/T224247 (10Dzahn)
[20:50:01] 	 10serviceops, 10Operations: upgrade and rename krypton & create its codfw equivalent - https://phabricator.wikimedia.org/T224247 (10Dzahn) a:03Dzahn
[20:50:14] 	 10serviceops, 10Operations: upgrade and rename krypton & create its codfw equivalent - https://phabricator.wikimedia.org/T224247 (10Dzahn) p:05Triage→03Normal
[21:11:54] 	 10serviceops, 10Operations: upgrade and rename krypton & create its codfw equivalent - https://phabricator.wikimedia.org/T224247 (10Dzahn)
[21:41:59] 	 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, and 3 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832 (10Paladox) Updated image for phab1003:  {F29226685}  It looks much much more hea...