[02:28:31] <wikibugs>	 10serviceops, 10MediaWiki-Page-derived-data, 10Performance-Team: Watchlist missing revisions with pages near the size limit - https://phabricator.wikimedia.org/T248564 (10tstarling) MW should have the lowest limit, because it can deliver the nicest error messages. So I would increase the php-fpm CPU limit fr...
[05:21:18] <wikibugs>	 10serviceops, 10Operations, 10Performance-Team, 10Traffic, and 2 others: Remove "Cache-control: no-cache" hack from wmf-config - https://phabricator.wikimedia.org/T247783 (10Joe)
[05:45:54] <wikibugs>	 10serviceops, 10MediaWiki-Page-derived-data, 10Performance-Team: Watchlist missing revisions with pages near the size limit - https://phabricator.wikimedia.org/T248564 (10Joe) >>! In T248564#6021276, @tstarling wrote: > MW should have the lowest limit, because it can deliver the nicest error messages. So I w...
[08:40:32] <hashar>	 hi, I still need a container to build wheels using python2 (that is to migrate Zuul to scap)  https://gerrit.wikimedia.org/r/#/c/operations/docker-images/production-images/+/580128/  ) :)
[08:40:36] <hashar>	 should be straightforward
[08:43:47] <_joe_>	 hashar: hi!
[08:43:56] <_joe_>	 still no one did it?
[08:44:03] <_joe_>	 heh I'll do it today
[08:44:09] <_joe_>	 between meetings
[08:50:34] <hashar>	 _joe_: :]
[08:50:47] <hashar>	 I guess my IRC pings are not bold enough ehhe
[08:51:00] <hashar>	 or I should have some kind of weekly meetings with the team
[08:59:41] <_joe_>	 hashar: or scrum of scrums
[08:59:49] <hashar>	 yeah I did that ;]
[09:00:19] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: CORS errors on commons on debug servers - https://phabricator.wikimedia.org/T249107 (10ema) p:05Triage→03Medium
[09:03:20] <hashar>	 through https://phabricator.wikimedia.org/T249110
[09:17:22] <_joe_>	 moritzm: it looks like installation of docker-pkg failed on deneb
[09:18:01] <_joe_>	 that's because we have no wheels for buster
[09:24:20] <_joe_>	 hashar: so I'm fixing docker-pkg not being on buster for you as well
[09:25:32] <hashar>	 ;]
[09:25:49] <_joe_>	 I don't remember how to deploy with scap3
[09:25:53] <hashar>	 one of my shower thoughts this morning was to get docker-pkg to run on CI when proposing changes to those Dockerfile.template stuff
[09:26:05] <_joe_>	 hashar: yeah that would be nice
[09:26:10] <hashar>	  deploy1001   cd /srv/docker-pkg  , scap deploy  ?
[09:26:35] <_joe_>	 is that enough?
[09:26:37] <_joe_>	 nice
[09:26:44] <hashar>	 I don't know
[09:26:46] <hashar>	 just making things up
[09:26:52] <hashar>	 I am unfortunately not familiar with scap
[09:27:46] <hashar>	 https://doc.wikimedia.org/mw-tools-scap/scap3/deploy_commands.html#scap-deploy  
[09:27:50] <hashar>	 seems to give some basics
[09:28:57] <_joe_>	 09:28:38 connection to deneb.eqiad.wmnet failed and future stages will not be attempted for this target
[09:29:34] <_joe_>	 sigh it's in codfw
[09:29:36] <_joe_>	 :P
[09:29:42] <hashar>	 ahah
[09:29:58] <hashar>	 time to rename it deneb2001.codfw.wmnet
[09:39:13] <volans>	 build2001 or debuild2001 would be better :-P
[09:41:57] <_joe_>	 hashar: pip is getting passive-aggressive about python 2
[09:42:00] <_joe_>	 2020-04-02 09:41:25,421 [docker-pkg-build] INFO - DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
[09:42:11] <hashar>	 yeah
[09:42:36] <hashar>	 and I guess we will kind of be forced to migrate to Zuul v3 regardless of the new CI initiative
[09:43:08] <hashar>	 that zuul v2 is from 3+ years ago
[09:45:28] <_joe_>	 hashar: Successfully published image docker-registry.discovery.wmnet/python2-build-buster
[09:45:58] <wikibugs>	 10serviceops, 10Operations, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Build and publish a python2 based container to build wheels - https://phabricator.wikimedia.org/T249110 (10Joe) 05Open→03Resolved a:03Joe Successfully published i...
[09:46:08] <_joe_>	 hashar: done, and sorry for the time it took
[09:48:28] <hashar>	 Error response from daemon: manifest for docker-registry.wikimedia.org/python2-build-buster:latest not found
[09:48:30] <hashar>	 stupid docker
[09:50:32] <_joe_>	 uh
[09:51:00] <hashar>	 _joe_: I think it is a cache issue somewhere in the chain
[09:51:07] <hashar>	 that might disappear in a few minutes
[09:51:24] <_joe_>	 uhm not sure tbh
[09:51:28] <wikibugs>	 10serviceops, 10Operations, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Build and publish a python2 based container to build wheels - https://phabricator.wikimedia.org/T249110 (10hashar) `counterexample $ docker pull docker-registry.wikimed...
[09:52:51] <_joe_>	 hashar: I think the image wasn't published because we forgot to update the ACLs of our docker registry
[09:53:12] <hashar>	 even though docker-pkg claimed it got published? :/
[09:55:40] <_joe_>	 hashar: yeah but that step is a bit wonky
[09:55:45] <_joe_>	 I blame docker for it
[09:56:07] <_joe_>	 well, docker-py
[10:03:54] <hashar>	 nice :)
[10:04:01] <hashar>	 I was looking at your puppet patch
[10:08:06] <hashar>	 _joe_: I will check again this afternoon, I  have to prepare lunch for the kids
[10:08:24] <hashar>	 + they have been on free wheel for a couple hours, so I guess I have some mess to cleanup
[10:08:27] <_joe_>	 hashar: it's done
[10:08:29] <_joe_>	 if you need
[10:08:45] <_joe_>	 and yes, go properly manage the kids :D
[10:08:53] <hashar>	 you probably need to publish again don't you?
[10:08:59] <_joe_>	 just did
[10:09:07] <hashar>	 it is still unhappy bah
[10:09:10] <_joe_>	 hashar: uh
[10:09:16] <_joe_>	 ok that's strange
[10:10:01] <_joe_>	 definitely very strange
[10:10:18] <_joe_>	 I can assure you docker told me the images were published this time
[10:10:20] <hashar>	 $ curl https://docker-registry.wikimedia.org/v2/python-build-buster/manifests/latest
[10:10:20] <hashar>	 {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"latest"}}]}
[10:10:35] <hashar>	 with headers  x-cache-status: hit-front   | age: 9
[10:10:44] <_joe_>	 yeah must be something with internal caching in the registry
[10:11:06] <_joe_>	 hashar: oh sigh I know what it is
[10:11:11] <_joe_>	 someone repooled eqiad
[10:11:29] <hashar>	 curl --verbose https://docker-registry.wikimedia.org/v2/python-build-buster/manifests/0.0.1  # that one was a miss on varnish
[10:11:43] <hashar>	 still same manifest unknown. So that seems to be on the docker registry side
[10:12:02] <_joe_>	 and nope
[10:12:13] <_joe_>	 yes hashar it's on the registry
[10:12:22] <hashar>	 it worked!
[10:12:27] <hashar>	 docker pull worked for me
[10:13:23] <hashar>	 so maybe my url was all wrong
[10:13:36] <_joe_>	 no I think the two registries are out of sync
[10:13:55] <hashar>	 :\
[10:14:07] <_joe_>	 so yeah I'll investigate later
[10:14:13] <hashar>	 sorry for the rabbit hole :-\
[10:15:06] <wikibugs>	 10serviceops, 10Operations, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Build and publish a python2 based container to build wheels - https://phabricator.wikimedia.org/T249110 (10hashar) There is some issue with the registry, but eventually...
[10:17:08] <hashar>	 _joe_: thank you!
[10:27:57] <_joe_>	 hasharLunch: lol you were using the wrong name
[10:29:56] <_joe_>	 it's python2-build-buster I think
[10:30:09] <_joe_>	 yes that works
[10:56:34] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: upgrade planet.wikimedia.org backends to buster - https://phabricator.wikimedia.org/T247651 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `planet1001.eqiad.wmnet` -  planet1001.eqiad.wmnet (**PASS**)   - Downtime...
[10:57:53] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: upgrade planet.wikimedia.org backends to buster - https://phabricator.wikimedia.org/T247651 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `planet2001.codfw.wmnet` -  planet2001.codfw.wmnet (**PASS**)   - Downtime...
[11:26:00] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: upgrade planet.wikimedia.org backends to buster - https://phabricator.wikimedia.org/T247651 (10Dzahn)
[11:41:24] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: upgrade planet.wikimedia.org backends to buster - https://phabricator.wikimedia.org/T247651 (10Dzahn) 05Open→03Resolved done. planet1002 and planet2002 on buster have replaced 1001/2001 and the old stretch VMs have been decom'ed.
[12:11:35] <hasharLunch>	 _joe_: I used python instead of python2 in my curl command hehe
[12:42:17] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: decom old appservers in eqiad - https://phabricator.wikimedia.org/T247780 (10Dzahn)
[12:48:17] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: decom old appservers in eqiad - https://phabricator.wikimedia.org/T247780 (10Dzahn) @cmjohnson @RobH   36 servers have been decom'ed.  30 in D5 and 6 in D4  But the original procurement ticket https://rt.wikimedia.org/Ticket/Display.html?id=8786  and installati...
[13:01:04] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: decom old appservers in eqiad - https://phabricator.wikimedia.org/T247780 (10Dzahn) 05Open→03Resolved Yes, it's thumbor1003 and thumbor1004, they are from the same procurement RT ticket.    36 of the 38 old servers from RT8786  have been decom'ed and the 2...
[13:01:47] <mutante>	 ^ i checked the status of all mw1* servers in netbox and the old procurement ticket. 
[13:02:06] <mutante>	 There are 187 mw servers in eqiad. 151 are "active" and 36 are "decommissioning".
[13:03:08] <mutante>	 The original old procurement ticket RT 8786 had 38 servers which are matching the decom'ed ones. The 2 missing ones are thumbor1003/thumbor1004 which have once been renamed from mw and are still in production.
[13:03:37] <mutante>	 So that means all the actual old mw servers in eqiad are properly removed from site/conftool etc but still in the rack if we need them.
[13:04:03] <mutante>	 A couple of them were in wrong state "staged" when they were actually active but fixed now.
[13:33:29] <wikibugs>	 10serviceops, 10Operations, 10docker-pkg: Investigate why the apt configuration of the wikimedia-buster docker image doesn't seem to prefer wikimedia packages - https://phabricator.wikimedia.org/T249218 (10Joe)
[13:33:37] <wikibugs>	 10serviceops, 10Operations, 10docker-pkg: Investigate why the apt configuration of the wikimedia-buster docker image doesn't seem to prefer wikimedia packages - https://phabricator.wikimedia.org/T249218 (10Joe) p:05Triage→03Medium
[13:34:02] <_joe_>	 jayme: when you're done with basic access, this could be an interesting first thing to investigate ^^
[13:35:48] <jayme>	 _joe_: ack. Not completely done yet (will continue as soon as moritz is back), so depends on the definition of medium priority I guess
[13:36:34] <_joe_>	 jayme: that's the horizon of weeks :)
[13:38:58] <jayme>	 oh, okay. Totally possible :-)
[14:32:08] <wikibugs>	 10serviceops, 10Core Platform Team, 10MediaWiki-API, 10Operations, 10Patch-For-Review: CORS errors on commons on debug servers - https://phabricator.wikimedia.org/T249107 (10Tgr)
[15:00:42] <wikibugs>	 10serviceops, 10MediaWiki-API, 10Operations, 10Core Platform Team Workboards (External Code Reviews), 10Patch-For-Review: CORS errors on commons on debug servers - https://phabricator.wikimedia.org/T249107 (10Anomie)
[15:30:13] <apergos>	 be right there
[16:07:03] <wikibugs>	 10serviceops, 10MediaWiki-General, 10Operations, 10observability, 10Patch-For-Review: MediaWiki Prometheus support - https://phabricator.wikimedia.org/T240685 (10colewhite) a:03colewhite
[17:03:52] <liw>	 !log testing upcoming scap on beta
[17:06:18] <rlazarus>	 liw: wrong channel fyi :)
[17:07:20] <liw>	 erf,sorry. my irssi channels have moved
[17:41:55] <wikibugs>	 10serviceops, 10Release-Engineering-Team-TODO, 10Scap: Deploy Scap version 3.14.0-1 - https://phabricator.wikimedia.org/T249250 (10LarsWirzenius)
[19:34:05] <wikibugs>	 10serviceops, 10LDAP-Access-Requests, 10Operations, 10Patch-For-Review: Grant Access to Logstash to Peter(peter.ovchyn@speedandfunction.com) - https://phabricator.wikimedia.org/T249037 (10KFrancis) @jcrespo  I checked with legal counsel... Could you please provide more information on what's in Logstash. If...
[19:43:31] <wikibugs>	 10serviceops, 10LDAP-Access-Requests, 10Operations, 10observability, 10Patch-For-Review: Grant Access to Logstash to Peter(peter.ovchyn@speedandfunction.com) - https://phabricator.wikimedia.org/T249037 (10Dzahn)
[20:06:24] <wikibugs>	 10serviceops, 10LDAP-Access-Requests, 10Operations, 10observability, 10Patch-For-Review: Grant Access to Logstash to Peter(peter.ovchyn@speedandfunction.com) - https://phabricator.wikimedia.org/T249037 (10AMooney) @KFrancis to my understanding. Logstash does contain PII.
[20:14:49] <wikibugs>	 10serviceops, 10LDAP-Access-Requests, 10Operations, 10observability, 10Patch-For-Review: Grant Access to Logstash to Peter(peter.ovchyn@speedandfunction.com) - https://phabricator.wikimedia.org/T249037 (10bd808) >>! In T249037#6024232, @KFrancis wrote: > @jcrespo  I checked with legal counsel... Could yo...
[20:30:07] <wikibugs>	 10serviceops, 10LDAP-Access-Requests, 10Operations, 10observability, 10Patch-For-Review: Grant Access to Logstash to Peter(peter.ovchyn@speedandfunction.com) - https://phabricator.wikimedia.org/T249037 (10Nuria) >This would be the same or similar NDA needed for access to the production databases as a sof...