[01:48:40] 10serviceops, 10Cassandra, 10Operations, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), and 4 others: Fix restbase1017's physical rack - https://phabricator.wikimedia.org/T222960 (10Eevans) Current snafu: None of data volumes are mounted (entries are missing from fstab). @... [07:37:19] 10serviceops, 10Operations, 10ops-codfw, 10User-jijiki: Degraded RAID on mw2250 - https://phabricator.wikimedia.org/T226948 (10MoritzMuehlenhoff) We don't use a lot of disk space on mw servers, let's go with option 2. [08:07:41] 10serviceops, 10Cassandra, 10Operations, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), and 4 others: Fix restbase1017's physical rack - https://phabricator.wikimedia.org/T222960 (10fgiunchedi) >>! In T222960#5327124, @Eevans wrote: > Current snafu: None of data volumes are... [08:31:44] 10serviceops, 10Wikidata-Termbox-Iteration-19, 10Patch-For-Review: Create termbox release for test.wikidata.org - https://phabricator.wikimedia.org/T226814 (10Tarrow) This service would just be to serve test.wikidata.org. I'm not clear if you would see this only as a testing service (because it doesn't have... [11:07:25] 10serviceops, 10DC-Ops, 10Operations, 10ops-eqiad: mw1239 memory errors - https://phabricator.wikimedia.org/T227867 (10jijiki) [11:09:17] 10serviceops, 10DC-Ops, 10Operations, 10ops-eqiad: mw1239 memory errors - https://phabricator.wikimedia.org/T227867 (10jijiki) p:05Triage→03Normal [11:59:55] 10serviceops: recreate staging cluster namespaces using helmfile - https://phabricator.wikimedia.org/T227775 (10fsero) [12:00:05] 10serviceops: recreate staging cluster namespaces using helmfile - https://phabricator.wikimedia.org/T227775 (10fsero) [12:00:47] !log recreating termbox staging namespace T227775 [12:34:15] 10serviceops, 10Core Platform Team (Cassandra Operational ), 10Core Platform Team Workboards (Team 2): Management of Cassandra schema and keyspace/table configuration - https://phabricator.wikimedia.org/T220246 (10holger.knust) [13:12:16] 10serviceops, 10Core Platform Team (Session Management Service (CDP2)), 10User-Clarakosi, 10User-Eevans: Package table_properties utility for Debian - https://phabricator.wikimedia.org/T226551 (10jijiki) [14:45:59] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Performance-Team (Radar), 10User-Elukey: Deprecate the usage of nutcracker for memcached - https://phabricator.wikimedia.org/T214275 (10elukey) >>! In T214275#5313635, @Andrew wrote: >>>! In T214275#5307951, @elukey wrote: >> >> The latter should be doabl... [14:46:57] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Performance-Team (Radar), 10User-Elukey: Deprecate the usage of nutcracker for memcached - https://phabricator.wikimedia.org/T214275 (10elukey) [14:48:07] 10serviceops, 10Operations, 10Performance-Team (Radar), 10User-Elukey, 10User-jijiki: Upgrade memcached for Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10elukey) [15:21:05] fsero: did you have a chance to look at the termbox-test stuff? Or are you thinking of holding back until we come to a conclusion about LVS or not? [15:22:14] tarrow: id prefer that you guys reach an agreement [15:22:26] If akosiaris is happy I can make it happen [15:22:39] fsero: cool! That's fine. Could you run me through the alternatives to LVS? [15:23:43] I'm happy to do some leg work once I know what needs to be done because I realise you guys are super busy. I'm just not sure what the right alternative is [15:27:09] We neither for now, we need to consider alternatives sorry [15:31:31] Ah, right! Makes sense; any suggestions for how we could move forward in the short term? It feels unwise for us start at www.wikidata.org without going via test.wikidata.org first. Is there anyway we can reuse the current termbox domain and IPs but run the test deployment on a different port or hint with a header or something? [15:53:52] There are options yes but no-one is a solid one, we can use envoy for that but needs to be discussed. I'll reply on ticket lvs seems the easiest now but I think akosiaris wants to ensure this is not for testing while it seems is a production service for testing if I understood it correctly [15:58:39] 10serviceops, 10Machine vision, 10Operations, 10Service-deployment-requests, 10Services (watching): Internal deployment of open_nsfw-- image scoring service - https://phabricator.wikimedia.org/T225664 (10Mholloway) a:03Mholloway [15:59:00] 10serviceops, 10Machine vision, 10Operations, 10Reading-Infrastructure-Team-Backlog (Kanban), and 2 others: Internal deployment of open_nsfw-- image scoring service - https://phabricator.wikimedia.org/T225664 (10Mholloway) [17:39:12] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Performance-Team (Radar), 10User-Elukey: Deprecate the usage of nutcracker for memcached - https://phabricator.wikimedia.org/T214275 (10aaron) >>! In T214275#5328625, @elukey wrote: >>>! In T214275#5313635, @Andrew wrote: >>>>! In T214275#5307951, @elukey... [17:54:43] 10serviceops, 10Core Platform Team (Session Management Service (CDP2)), 10User-Clarakosi, 10User-Eevans: Package table_properties utility for Debian - https://phabricator.wikimedia.org/T226551 (10Eevans) [18:43:38] 10serviceops, 10DC-Ops, 10Operations, 10ops-eqiad: mw1239 memory errors - https://phabricator.wikimedia.org/T227867 (10Cmjohnson) This server is out of warranty, I can reseat the DIMM but will need the server to taken down. [20:38:37] 10serviceops, 10Machine vision, 10Operations, 10Reading-Infrastructure-Team-Backlog (Kanban), and 2 others: Internal deployment of open_nsfw-- image scoring service - https://phabricator.wikimedia.org/T225664 (10Mholloway) @joe Thanks (belatedly) for the comments. I've got a fork at https://github.com/mdh... [20:47:34] 10serviceops, 10Machine vision, 10Operations, 10Reading-Infrastructure-Team-Backlog (Kanban), and 2 others: Update open_nsfw-- for Wikimedia production deployment - https://phabricator.wikimedia.org/T225664 (10Mholloway) [20:48:25] 10serviceops, 10Machine vision, 10Operations, 10Reading-Infrastructure-Team-Backlog (Kanban), and 2 others: Update open_nsfw-- for Wikimedia production deployment - https://phabricator.wikimedia.org/T225664 (10Mholloway) [20:55:00] 10serviceops, 10Machine vision, 10Operations, 10Reading-Infrastructure-Team-Backlog (Kanban), and 2 others: Update open_nsfw-- for Wikimedia production deployment - https://phabricator.wikimedia.org/T225664 (10Mholloway) On the subject of race conditions and accepting raw image data, I imagine the scoring...