[01:13:57] <wikibugs>	 06Data-Engineering, 06DBA, 10MediaWiki-extensions-OATHAuth, 05FY2025-26 WE 4.6 - Account Security (WE 4.6.4 - 2FA improvements and passkey support), 07Schema-change: New database table for tracking WebAuthn userHandle values (oathauth_user_handles) - https://phabricator.wikimedia.org/T416544 (10Catrope)...
[01:15:36] <wikibugs>	 06Data-Engineering, 06DBA, 10MediaWiki-extensions-OATHAuth, 05FY2025-26 WE 4.6 - Account Security (WE 4.6.4 - 2FA improvements and passkey support), and 2 others: New database table for tracking WebAuthn userHandle values (oathauth_user_handles) - https://phabricator.wikimedia.org/T416544#11585748 (10Reedy)
[01:15:49] <wikibugs>	 06Data-Engineering, 06DBA, 10MediaWiki-extensions-OATHAuth, 05FY2025-26 WE 4.6 - Account Security (WE 4.6.4 - 2FA improvements and passkey support), and 2 others: New database table for tracking WebAuthn userHandle values (oathauth_user_handles) - https://phabricator.wikimedia.org/T416544#11585749 (10Catrop...
[01:16:18] <wikibugs>	 06Data-Engineering, 06DBA, 10MediaWiki-extensions-OATHAuth, 05FY2025-26 WE 4.6 - Account Security (WE 4.6.4 - 2FA improvements and passkey support), and 2 others: New database table for tracking WebAuthn userHandle values (oathauth_user_handles) - https://phabricator.wikimedia.org/T416544#11585750 (10Catrop...
[01:17:14] <wikibugs>	 06Data-Engineering, 06DBA, 10MediaWiki-extensions-OATHAuth, 05FY2025-26 WE 4.6 - Account Security (WE 4.6.4 - 2FA improvements and passkey support), and 2 others: New database table for tracking WebAuthn userHandle values (oathauth_user_handles) - https://phabricator.wikimedia.org/T416544#11585751 (10Catrop...
[01:17:52] <jinxer-wm>	 FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[01:17:52] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[01:46:08] <wikibugs>	 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Create a `DbtSkeinOperator` in the Airflow `wmf_airflow_common` library - https://phabricator.wikimedia.org/T415194#11585786 (10Ottomata) @amastilovic apologies for late/drive by review, but I left some comments, especially around the use of hardcoded `/t...
[05:17:51] <jinxer-wm>	 FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[05:17:52] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[06:33:39] <wikibugs>	 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11586027 (10Marostegui)
[06:42:18] <wikibugs>	 06Data-Engineering, 06Data-Persistence, 10MediaWiki-extensions-OATHAuth, 05FY2025-26 WE 4.6 - Account Security (WE 4.6.4 - 2FA improvements and passkey support), and 2 others: New database table for tracking WebAuthn userHandle values (oathauth_user_handle... - https://phabricator.wikimedia.org/T416544#11586030
[07:49:58] <wikibugs>	 06Data-Engineering, 10Data-Engineering-Wikistats, 10Pageviews-Anomaly: Sudden traffic increase on 1 November 2025 - https://phabricator.wikimedia.org/T412655#11586074 (10Rooiratel) Hi @Milimetric and @JAllemandou, would you be able to assist us in getting this ticket assigned to someone. The problem is causi...
[08:19:35] <wikibugs>	 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11586115 (10Marostegui)
[08:56:20] <wikibugs>	 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11586183 (10ops-monitoring-bot) Start pool of db2205 gradually with 4 steps - After schema change - marostegui@cumin1003
[08:56:50] <wikibugs>	 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11586184 (10Marostegui)
[08:59:11] <wikibugs>	 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11586195 (10Marostegui)
[08:59:53] <wikibugs>	 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11586212 (10Marostegui)
[09:10:04] <wikibugs>	 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11586255 (10Marostegui)
[09:17:52] <jinxer-wm>	 FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[09:17:52] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[09:28:50] <wikibugs>	 06Data-Engineering, 10Data-Engineering-Wikistats, 10Pageviews-Anomaly: Sudden traffic increase on 1 November 2025 - https://phabricator.wikimedia.org/T412655#11586320 (10JAllemandou) Ping @Ahoelzl and @GGoncalves-WMF for prioritization.
[09:41:40] <wikibugs>	 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11586370 (10ops-monitoring-bot) Completed pool of db2205 gradually with 4 steps - After schema change - marostegui@cumin1003
[10:02:37] <jinxer-wm>	 FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[10:02:37] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[10:07:36] <jinxer-wm>	 FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[10:07:37] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[10:50:39] <wikibugs>	 06Data-Engineering, 06Machine-Learning-Team, 10Event-Platform: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11586695 (10Samwalton9-WMF)
[10:56:37] <wikibugs>	 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11586737 (10Marostegui)
[12:19:14] <wikibugs>	 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Research, 10Event-Platform, 13Patch-For-Review: Implement stream of HTML content on mw.page_change event - https://phabricator.wikimedia.org/T360794#11587133 (10BTullis) I have created the s3 user for the deployment of `mw-page-html-content-change...
[12:53:02] <wikibugs>	 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Research, 10Event-Platform, 13Patch-For-Review: Implement stream of HTML content on mw.page_change event - https://phabricator.wikimedia.org/T360794#11587279 (10BTullis) I have added these secrets to `hieradata/role/common/deployment_server/kubern...
[12:53:15] <jinxer-wm>	 FIRING: HdfsRpcQueueLength: RPC call queue length on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=54&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLength
[12:56:30] <wikibugs>	 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11587286 (10ops-monitoring-bot) Start pool of db2204 gradually with 4 steps - After schema change - marostegui@cumin1003
[13:02:18] <wikibugs>	 06Data-Engineering, 06Data-Persistence, 10MediaWiki-extensions-OATHAuth, 05FY2025-26 WE 4.6 - Account Security (WE 4.6.4 - 2FA improvements and passkey support), and 2 others: New database table for tracking WebAuthn userHandle values (oathauth_user_handle... - https://phabricator.wikimedia.org/T416544#11587322
[13:03:15] <jinxer-wm>	 RESOLVED: HdfsRpcQueueLength: RPC call queue length on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=54&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLength
[13:41:53] <wikibugs>	 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11587448 (10ops-monitoring-bot) Completed pool of db2204 gradually with 4 steps - After schema change - marostegui@cumin1003
[14:07:37] <jinxer-wm>	 FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[14:07:37] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[14:12:38] <wikibugs>	 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content: Use wmf.mediawiki_history as baseline for slo completeness - https://phabricator.wikimedia.org/T416312#11587560 (10APizzata-WMF) After a fruitful conversation with @JAllemandou, we came to the conclusion that for the different n...
[14:12:41] <wikibugs>	 06Data-Engineering, 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Superset "track job" button leads to broken URL - https://phabricator.wikimedia.org/T410149#11587561 (10Gehel)
[14:21:59] <wikibugs>	 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Blunderbuss: Move Hadoop/HDFS XML configuration into Helm deployment chart - https://phabricator.wikimedia.org/T402323#11587587 (10Gehel) I don't expect #data-platform-sre to do any work...
[14:24:00] <wikibugs>	 06Data-Engineering, 06Data-Platform-SRE, 07Essential-Work: Simplify the datahub ingestion pipelines - https://phabricator.wikimedia.org/T402306#11587597 (10Gehel) This is stalled and seems more complex than initially expected. Let's see what #data-engineering thinks in terms of priority. I'm not sure how muc...
[14:31:09] <wmftkbot>	 !log Test Kitchen edge-unique experiments (poll 94261) - adds: synth-aaa-test-mw-js; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD
[14:31:11] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[14:35:08] <wikibugs>	 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE: [OpsWeek] Testing on airflow-devenvs can generate false alerts such as SLO misses - https://phabricator.wikimedia.org/T416596 (10xcollazo) 03NEW
[14:36:40] <wikibugs>	 06Data-Engineering, 10Data-Engineering-Wikistats, 10Pageviews-Anomaly: Sudden traffic increase on 1 November 2025 - https://phabricator.wikimedia.org/T412655#11587665 (10Milimetric) Thank you so much for pinging.  This is suspicious indeed, and especially interesting is @jhsoby-WMNO's finding about non-latin...
[14:55:59] <wmftkbot>	 !log Test Kitchen edge-unique experiments (poll 94335) - adds: none; removes: synth-aaa-test-mw-js; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD
[14:56:01] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[14:57:37] <wikibugs>	 (03PS1) 10AKhatun: Add hql files for new growthbook pipeline [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1237249 (https://phabricator.wikimedia.org/T414105)
[14:57:40] <wmftkbot>	 !log Test Kitchen edge-unique experiments (poll 94340) - adds: synth-aaa-test-mw-js; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD
[14:57:42] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:03:20] <wikibugs>	 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content: Missing reconciliation for MWCH - https://phabricator.wikimedia.org/T416491#11587846 (10APizzata-WMF)
[15:07:18] <wikibugs>	 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): [Iceberg Migration] Extend Iceberg table maintenance mechanism to support multiple Airflow instances - https://phabricator.wikimedia.org/T373693#11587856 (10Snwachukwu)
[15:12:50] <wikibugs>	 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11587912 (10brouberol) We have deployed the "infrastructure" that will support `rsync`-ing files from airflow-sre task pods to th...
[17:55:05] <wikibugs>	 06Data-Engineering, 06Data-Persistence, 10MediaWiki-extensions-OATHAuth, 06Security-Team, and 3 others: New database table for tracking WebAuthn userHandle values (oathauth_user_handles) - https://phabricator.wikimedia.org/T416544#11588773 (10Catrope)
[18:04:28] <wikibugs>	 06Data-Engineering, 06Data-Persistence, 10MediaWiki-extensions-OATHAuth, 06Security-Team, and 3 others: New database table for tracking WebAuthn userHandle values (oathauth_user_handles) - https://phabricator.wikimedia.org/T416544#11588804 (10Ahoelzl) @JAllemandou can you take a look if DE is impacted?
[18:04:58] <wikibugs>	 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Adapt Sqoop for imagelinks schema changes - https://phabricator.wikimedia.org/T416481#11588806 (10Ahoelzl)
[18:07:52] <jinxer-wm>	 FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[18:07:52] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[18:08:26] <wikibugs>	 06Data-Engineering, 06Machine-Learning-Team, 10Event-Platform: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11588822 (10Milimetric) Andrew - what's the timeline on this for DE?  Do we just track it in radar and advise or d...
[18:15:49] <wikibugs>	 06Data-Engineering, 06Privacy Engineering: The soon-to-be-released pageview datasets should be linked from dumps page - https://phabricator.wikimedia.org/T335958#11588859 (10Milimetric) @Ahoelzl will touch base with Andrew + Guilherme on this - I personally think it's important to de-duplicate datasets and I'd...
[18:16:51] <wikibugs>	 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Wikidata, 10Wikidata Analytics: [Analytics] [Tech Debt] Exploration of initial data quality assessment system for Airflow DAGs - https://phabricator.wikimedia.org/T415783#11588865 (10Ahoelzl)
[18:17:05] <wikibugs>	 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: [Analytics] [Tech Debt] Exploration of initial data quality assessment system for Airflow DAGs - https://phabricator.wikimedia.org/T415783#11588868 (10Milimetric) @Ahoelzl will follow up
[18:18:59] <wikibugs>	 06Data-Engineering, 10ChangeProp, 06MW-Interfaces-Team, 10Observability-Tracing, and 2 others: Implement tracing across changeprop-jobqueue, kafka, eventgate - https://phabricator.wikimedia.org/T395038#11588874 (10Milimetric) Andreas reviewing in weekly sync
[18:20:40] <wikibugs>	 06Data-Engineering, 10Dumps-Generation: wikidatawiki fails dumps of the wbt_* tables, also lagging on XML Dumps - https://phabricator.wikimedia.org/T396125#11588881 (10Milimetric) @xcollazo Is this obsolete?
[18:26:16] <wikibugs>	 06Data-Engineering, 06Product-Analytics: [Request] Sample of refined webrequest table in data lake - https://phabricator.wikimedia.org/T413952#11588898 (10Milimetric) Yeah, we used to have this dataset, it was called webrequest_sampled_128 and I think this was the table create statement for it (the population...
[18:26:51] <wikibugs>	 06Data-Engineering, 10Data-Engineering-Wikistats, 10Pageviews-Anomaly: Sudden traffic increase on 1 November 2025 - https://phabricator.wikimedia.org/T412655#11588903 (10Milimetric)
[18:26:52] <wikibugs>	 06Data-Engineering, 10Data-Engineering-Wikistats, 10Pageviews-Anomaly: Questionably high pageviews on afwiki need to be investigated - https://phabricator.wikimedia.org/T413949#11588905 (10Milimetric) →14Duplicate dup:03T412655
[18:28:12] <wikibugs>	 06Data-Engineering, 10Data-Engineering-Wikistats, 10Pageviews-Anomaly: Sudden traffic increase on 1 November 2025 - https://phabricator.wikimedia.org/T412655#11588908 (10Milimetric) we're working on this as part of normal incident response
[18:44:48] <wikibugs>	 06Data-Engineering, 06Data-Platform-SRE, 10Dumps-Generation, 07Essential-Work: Certain *recombine tasks in dumps_v1 are non-idempotent and can generate corrupt files - https://phabricator.wikimedia.org/T404859#11588964 (10BTullis) Moving back to the main backlog. We will probably end up declining to fix th...
[18:45:21] <wikibugs>	 06Data-Engineering, 06Data-Platform-SRE, 10Dumps-Generation, 07Essential-Work: Certain *recombine tasks in dumps_v1 are non-idempotent and can generate corrupt files - https://phabricator.wikimedia.org/T404859#11588970 (10BTullis) p:05High→03Low
[19:25:28] <wikibugs>	 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services: Add datetime versions of timestamp fields to Wikireplica databases - https://phabricator.wikimedia.org/T414199#11589123 (10Huji) I created https://meta.wikimedia.org/wiki/Community_Wishlist/W509 accordingly
[19:37:06] <jinxer-wm>	 FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[19:53:49] <wikibugs>	 06Data-Engineering, 06Data-Persistence, 10MediaWiki-extensions-OATHAuth, 06Security-Team, and 3 others: New database table for tracking WebAuthn userHandle values (oathauth_user_handles) - https://phabricator.wikimedia.org/T416544#11589201 (10Catrope) a:03Catrope >>! In T416544#11587322, @Ladsgroup wrote...
[20:07:06] <jinxer-wm>	 RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[20:17:06] <jinxer-wm>	 FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[20:28:35] <wikibugs>	 06Data-Engineering, 06Movement-Insights, 10Wmfdata-Python: Revisit the default configuration of Wmfdata Spark sessions - https://phabricator.wikimedia.org/T406417#11589331 (10OSefu-WMF)
[20:37:56] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10Data Pipelines, 06Movement-Insights: Wikistats "Active Editors by Country" does not follow definition for active editors - https://phabricator.wikimedia.org/T360073#11589379 (10OSefu-WMF)
[20:44:26] <wikibugs>	 10Analytics-Canonical-Data, 06Product-Analytics, 06Movement-Insights (FY25-26 H2): Add a column to differentiate various wiki-types on canonical_data.wikis - https://phabricator.wikimedia.org/T361211#11589401 (10OSefu-WMF)
[20:44:28] <wikibugs>	 06Data-Engineering, 10Wmfdata-Python, 06Movement-Insights (FY25-26 H2): spark.run returns SQL dates as Python datetime.date, not Pandas datetime - https://phabricator.wikimedia.org/T384548#11589403 (10OSefu-WMF)
[20:44:30] <wikibugs>	 06Data-Engineering, 10Wmfdata-Python, 06Movement-Insights (FY25-26 H2): mariadb.run returns SQL dates as Python datetime.date, not Pandas datetime - https://phabricator.wikimedia.org/T384546#11589405 (10OSefu-WMF)
[20:44:33] <wikibugs>	 10Analytics-Canonical-Data, 06Movement-Insights (FY25-26 H2): Review and support addition of language and incubating wikis to canonical data - https://phabricator.wikimedia.org/T401149#11589407 (10OSefu-WMF)
[20:44:38] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10Wmfdata-Python, 06Movement-Insights (FY25-26 H2): Support creating a Spark session with a GitLab-built Conda environment - https://phabricator.wikimedia.org/T378253#11589409 (10OSefu-WMF)
[20:45:52] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10Wmfdata-Python, 06Movement-Insights (FY25-26 H2): Fix lint issues in Wmfdata-Python and make the linting CI blocking - https://phabricator.wikimedia.org/T381657#11589419 (10OSefu-WMF)
[20:58:36] <wikibugs>	 10Analytics-Canonical-Data, 06Movement-Insights (FY25-26 H2): Upload canonical datasets to the Data Lake using Wmfdata's spark.upload_pandas - https://phabricator.wikimedia.org/T355847#11589490 (10OSefu-WMF)
[20:59:07] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10Data Pipelines, 06Movement-Insights: New Data Pipeline for New and Returning Active Editor metrics by Geo (Country & Region) and wiki - https://phabricator.wikimedia.org/T359646#11589494 (10OSefu-WMF)
[20:59:11] <wikibugs>	 06Data-Engineering, 10Data Pipelines, 06Movement-Insights: Investigate spikes in unique devices per domain, mobile, US+CA, Jun 2023 - https://phabricator.wikimedia.org/T355143#11589496 (10OSefu-WMF)
[21:02:06] <jinxer-wm>	 RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[21:56:19] <wikibugs>	 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): OpsWeek: wikidatawiki fails to export due to OOM on partitionByXMLFileBoundaries - https://phabricator.wikimedia.org/T416642 (10xcollazo) 03NEW
[22:07:52] <jinxer-wm>	 FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[22:07:52] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[22:09:24] <wikibugs>	 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Dumps-Generation: SQL metadata files problem - https://phabricator.wikimedia.org/T416416#11589752 (10Legoktm) 05Open→03Invalid Based on https://en.wikipedia.org/wiki/Wikipedia:Bots/Noticeboard#c-The_Anome-20260204155000-Legoktm-20260204003800...