[00:28:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 84.26% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:33:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 83.78% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[03:55:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 86.12% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[04:00:56] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 86.3% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[05:01:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 85.89% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[05:26:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 88.84% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[07:27:52] <wikibugs>	 10netops, 06Infrastructure-Foundations, 07sre-alert-triage: Alert in need of triage: PeeringBGPDown (instance cr3-eqsin:9804) - https://phabricator.wikimedia.org/T407833 (10LSobanski) 03NEW
[07:56:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 81.75% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[08:01:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 85.53% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[08:48:20] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Nokia: add new switches in eqiad/codfw to monitoring and make 'active' - https://phabricator.wikimedia.org/T405558#11292788 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=d70417af-8325-49e7-a880-7a0cd37bd2d2) set by cmo...
[10:21:07] <elukey>	 quick change if anybody has time https://gerrit.wikimedia.org/r/c/operations/puppet/+/1197611
[10:21:16] <elukey>	 pcc is broken for the host, sadly
[13:35:01] <wikibugs>	 10SRE-tools, 10Observability-Alerting, 10Spicerack, 06Traffic: Alertmanager triggers an alert on IRC and email after the alert has resolved - https://phabricator.wikimedia.org/T407787#11293662 (10ssingh) > It looks like spicerack should check that alerts for the downtimed host have been resolved (not in fi...
[13:59:56] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 82.94% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[14:04:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 85.39% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[14:06:39] <wikibugs>	 10SRE-tools, 06Infrastructure-Foundations, 10Observability-Alerting, 10Spicerack, 06Traffic: Alertmanager triggers an alert on IRC and email after the alert has resolved - https://phabricator.wikimedia.org/T407787#11293886 (10Volans) For some related historical context on the lack of parity between the I...
[14:14:29] <wikibugs>	 07Puppet, 10MW-on-K8s, 10Observability-Alerting: Clean up "git repo needs merge" checks - https://phabricator.wikimedia.org/T370530#11293938 (10tappof) a:03tappof
[14:15:10] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 86.33% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[14:25:10] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 82.08% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[14:30:10] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 83.1% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[14:32:25] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 83.1% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[14:40:10] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 85.6% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[14:41:25] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 85.6% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[15:42:55] <wikibugs>	 07Puppet, 10MobileFrontend (Tracking): Mobile site does not automatically redirect to desktop version (and not possible to use browser "use desktop view") - https://phabricator.wikimedia.org/T60425#11294353 (10Jdlrobson-WMF) @Jdforrester-WMF I believe this can now be resolved. Visit en.m.wikipedia.org on my de...