[01:34:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [04:52:30] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10916349 (10Marostegui) [05:34:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [05:36:41] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10916400 (10Marostegui) [06:12:41] 06Data-Engineering, 06Data-Engineering-Radar, 10ConfirmEdit (CAPTCHA extension), 10MediaWiki-extensions-Campaigns, and 3 others: Send hCaptcha API response data to event platform - https://phabricator.wikimedia.org/T379179#10916486 (10kostajh) [07:40:05] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop afl_patrolled_by from abuse_filter_log in production - https://phabricator.wikimedia.org/T391056#10916817 (10Marostegui) s2 codfw master has been switched T396976 [07:47:36] 06Data-Engineering, 06Data-Engineering-Radar, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-user for Anton Kokh (WMDE) - https://phabricator.wikimedia.org/T395917#10916847 (10Anton.Kokh) @KFrancis thank you, I just signed it! [08:36:43] 10Data-Engineering (Q4 2025 April 1st - June 30th): Facilitate automatic artifact cache warming for airflow-dags artifacts - https://phabricator.wikimedia.org/T392244#10917043 (10Gehel) [08:42:51] 10Quarry: quarry: Drop manual frontend build process - https://phabricator.wikimedia.org/T396991 (10taavi) 03NEW [08:53:38] 10Data-Engineering (Q4 2025 April 1st - June 30th): Facilitate automatic artifact cache warming for airflow-dags artifacts - https://phabricator.wikimedia.org/T392244#10917227 (10BTullis) [09:34:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [09:54:20] 06Data-Engineering, 06Data-Engineering-Icebox, 06SRE Observability, 10Data-Platform-SRE (2025.06.13 - 2025.07.04), 13Patch-For-Review: [Data Platform] Install a Prometheus connector for Presto, pointed at thanos-query - https://phabricator.wikimedia.org/T347430#10917569 (10fgiunchedi) >>! In T347430#1091... [11:35:40] 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: Add schema diffing support to jsonschema-tools and run diff in CI - https://phabricator.wikimedia.org/T321850#10917837 (10Ottomata) If we do https://gitlab.wikimedia.org/repos/data-engineering/jsonschema-tools/-/merge_requests/57, latest.yaml files wi... [11:43:53] 10Quarry: [bug] "Internal Server Error" when logging into Quarry - https://phabricator.wikimedia.org/T333043#10917896 (10SD0001) 05Open→03Resolved The root cause was probably T332650, which is not Quarry-related and long since resolved. >>! In T333043#8726691, @Tgr wrote: > Quarry should probably be fix... [12:26:13] !log add an-conf1006 to zookeeper cluster T374922 [12:26:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:26:16] T374922: Bring an-conf100[4-6] into service to replace an-conf100[1-3] - https://phabricator.wikimedia.org/T374922 [12:35:07] !log roll-restart-zookeeper analytics cluster to add an-conf1006 T374922 [12:35:10] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:35:10] T374922: Bring an-conf100[4-6] into service to replace an-conf100[1-3] - https://phabricator.wikimedia.org/T374922 [12:52:13] PROBLEM - Zookeeper Server on an-conf1006 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg https://wikitech.wikimedia.org/wiki/Zookeeper [12:59:31] stevemunene: is that ^ expected? [13:01:08] No it's not but I am looking at it as we speak, and sending a patch asap [13:06:53] brouberol: small patch for the above https://gerrit.wikimedia.org/r/c/operations/puppet/+/1159451 [13:07:20] approved [13:07:38] Thanks! [13:11:54] !log roll-restart-zookeeper analytics cluster to add an-conf1006 T374922 [13:11:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:11:56] T374922: Bring an-conf100[4-6] into service to replace an-conf100[1-3] - https://phabricator.wikimedia.org/T374922 [13:15:13] RECOVERY - Zookeeper Server on an-conf1006 is OK: PROCS OK: 1 process with command name java, args org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg https://wikitech.wikimedia.org/wiki/Zookeeper [13:21:51] \o/ [13:30:44] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE: Grant Access to Product's Superset & Turnilo for SKivlehan - https://phabricator.wikimedia.org/T393626#10918393 (10herron) 05Open→03Stalled [13:34:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [13:35:32] 06Data-Engineering, 06Data-Platform-SRE, 06Java-Scala-Standardization, 10Discovery-Search (2025.06.13 - 2025.07.04), 13Patch-For-Review: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all depen... - https://phabricator.wikimedia.org/T367405#10918479 [13:36:21] 06Data-Engineering, 06Java-Scala-Standardization, 10Discovery-Search (2025.06.13 - 2025.07.04): Create Gitlab CI templates for JVM packages - https://phabricator.wikimedia.org/T386406#10918504 (10Gehel) [13:36:35] 06Data-Engineering, 06Data-Engineering-Radar, 10CirrusSearch, 10Structured Data Engineering, and 3 others: Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#10918508 (10Gehel) [13:41:50] 06Data-Engineering, 06Data-Engineering-Radar, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-user (and LDAP nda, wmde) for Anton Kokh (WMDE) - https://phabricator.wikimedia.org/T395917#10918563 (10herron) [13:43:13] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [13:47:17] 06Data-Engineering, 06Data-Engineering-Radar, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-user (and LDAP nda, wmde) for Anton Kokh (WMDE) - https://phabricator.wikimedia.org/T395917#10918595 (10herron) Hi @Anton.Kokh could you please add a unique SSH key he... [15:07:34] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: DPE Deep Dive on MW Content Tables - https://phabricator.wikimedia.org/T396882#10919043 (10xcollazo) [15:18:28] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:24:56] 14Analytics, 07Analytics-Data-Problem, 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Engineering-Wikistats, 06Movement-Insights: Sharp spike in unique devices for past month on all projects - https://phabricator.wikimedia.org/T395727#10919166 (10Ahoelzl) a:03mforns [15:40:48] 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team, 06serviceops: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10919258 (10jijiki) [19:18:28] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [21:20:43] 06Data-Engineering, 10MediaWiki-DomainEvents, 10ci-test-error (WMF-deployed Build Failure), 10Event-Platform: phpunit\integration\PageChangeEmissionTest::testPageMove with data set "Valid move with redirect" ('SourcePageA', 'DestinationPageA', true, 3) - https://phabricator.wikimedia.org/T397087#10920621 (1... [21:20:54] 06Data-Engineering, 10MediaWiki-DomainEvents, 10ci-test-error (WMF-deployed Build Failure), 10Event-Platform: phpunit\integration\PageChangeEmissionTest::testPageMove with data set "Valid move with redirect" ('SourcePageA', 'DestinationPageA', true, 3) - https://phabricator.wikimedia.org/T397087#10920623 (1... [22:02:39] 06Data-Engineering, 10Data-Platform-SRE (2025.06.13 - 2025.07.04): Request for dedicated Airflow instance for WME - https://phabricator.wikimedia.org/T396672#10920750 (10Ahoelzl) @HShaikh please provide more input on concrete future Airflow needs. As an alternative to an own instance WME could leverage `plat... [23:18:28] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem