[06:37:03] <_joe_> mutante: oh dear, we were supposed to check and remove those packages from production, someone must have dropped the ball [09:45:40] 10serviceops, 10Parsoid-PHP, 10Patch-For-Review: EasyTimeline extension shell error - https://phabricator.wikimedia.org/T237304 (10Joe) 05Open→03Resolved Sorry for the inconvenience. This wasn't spotted earlier as we didn't actively remove the packages from the canaries in production. [09:45:42] 10serviceops, 10Operations: Make the parsoid cluster support parsoid/PHP - https://phabricator.wikimedia.org/T233654 (10Joe) [09:51:00] 10serviceops, 10Operations, 10Puppet, 10User-jbond: Rolling restart of etcd to pick up the renewed CA public certificate. - https://phabricator.wikimedia.org/T237362 (10Joe) [09:52:09] 10serviceops, 10Operations, 10Puppet, 10User-jbond: Rolling restart of etcd to pick up the renewed CA public certificate. - https://phabricator.wikimedia.org/T237362 (10Joe) [10:39:20] 10serviceops, 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Switchover backup director service from helium to backup1001 - https://phabricator.wikimedia.org/T236406 (10jcrespo) [10:39:49] 10serviceops, 10DBA, 10Operations, 10Patch-For-Review: Backups on buster hosts fail to run - https://phabricator.wikimedia.org/T235838 (10jcrespo) 05Open→03Resolved [10:39:54] 10serviceops, 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10jcrespo) [11:21:53] 10serviceops, 10Operations, 10observability, 10Performance-Team (Radar): Messages in Logstash from php-fatal-error.php are missing from type:mediawiki/channel:fatal - https://phabricator.wikimedia.org/T234283 (10fgiunchedi) Looks like this is working now! cc @Krinkle https://logstash.wikimedia.org/app/kib... [14:05:32] 10serviceops, 10Operations, 10observability: basic prometheus monitoring for PoolCounter - https://phabricator.wikimedia.org/T237407 (10CDanis) [14:05:50] I got asked (indirectly) to create a prometheus exporter of bacula, and I would like to have a discussion, if possible, a on how to best organize that- not time sensitive [14:06:04] before I even start to code [14:06:37] ups, sorry, wrong channel [14:25:10] 10serviceops, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar), and 2 others: Upgrade memcached for Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10elukey) Added metrics to http://beta-prometheus.wmflabs.org about memcached: http://beta-prometheus.wmflabs.org/beta/graph?g0... [15:13:51] serviceops folks [15:13:54] it sounds like in https://phabricator.wikimedia.org/T231255 [15:14:06] you're asking to make changes to the quote, to change RAM? [15:14:33] if that is final, can you make that explicit, reassign the task to RobH, and also update https://phabricator.wikimedia.org/T233639 as necessary? [15:18:11] joe ^ [15:19:21] 10serviceops, 10Machine vision, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Patch-For-Review: How should the MachineVision extension interact with external APIs from production? - https://phabricator.wikimedia.org/T236797 (10Mholloway) 05Open→03Resolved a:03Mholloway The replacement of t... [16:48:08] 10serviceops, 10Operations, 10Parsoid-PHP: wt2html: Out of memory crashers - https://phabricator.wikimedia.org/T236833 (10ssastry) @Dzahn: @Joe and I chatted and we decided to bump the memory limit by 100MB for the parsoid cluster. Can you prepare the patches for this? Thanks! [16:50:57] 10serviceops, 10Machine vision, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Patch-For-Review: How should the MachineVision extension interact with external APIs from production? - https://phabricator.wikimedia.org/T236797 (10Mholloway) [16:55:13] <_joe_> paravoid: sorry, I was in meetings until now [16:55:31] <_joe_> yes, I wanted to be sure of all details before doing so, will do now [17:31:41] _joe_: but do we want ploticus back? [17:31:56] <_joe_> mutante: definitely [17:33:52] ah, it's already merged. nice [17:47:20] _joe_: do we want an extra ticket to remove the other packages from all.. or we just let reimaging happen [17:48:47] or i can reopen "Clean up artifacts from LaTeX based math rendering" that shoudld be the original ticket [17:49:44] ah, heh, and the last comment there is that they will be reimaged anyways as part of hhvm removal, so that's why it was closed [18:08:23] <_joe_> mutante: well yeah but it still didn't happen [18:18:59] left a comment on that ticket [20:44:43] 10serviceops, 10Operations: decom cobalt - https://phabricator.wikimedia.org/T236187 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `cobalt.wikimedia.org` - cobalt.wikimedia.org (**PASS**) - Downtimed host on Icinga - Downtimed management interface on Icinga... [20:45:37] yay! [20:52:32] 10serviceops, 10Operations: decom cobalt - https://phabricator.wikimedia.org/T236187 (10Dzahn) [23:49:12] 10serviceops, 10Operations: decom cobalt - https://phabricator.wikimedia.org/T236187 (10Dzahn) a:05Dzahn→03Jclark-ctr [23:49:44] 10serviceops, 10Operations: decom cobalt - https://phabricator.wikimedia.org/T236187 (10Dzahn) 05Stalled→03Open p:05Normal→03Low [23:49:48] 10serviceops, 10Gerrit, 10Operations, 10Release-Engineering-Team-TODO, and 2 others: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster) - https://phabricator.wikimedia.org/T222391 (10Dzahn) [23:50:00] 10serviceops, 10Operations, 10Parsoid-PHP, 10Patch-For-Review: wt2html: Out of memory crashers - https://phabricator.wikimedia.org/T236833 (10Krinkle) To keep things easy to reason about, I think it would be preferred to apply this to all app servers (more in comments at 10serviceops, 10Operations, 10Parsoid-PHP, 10Patch-For-Review: wt2html: Out of memory crashers - https://phabricator.wikimedia.org/T236833 (10Krinkle) a:03Dzahn [23:53:30] 10serviceops, 10Operations: decom cobalt - https://phabricator.wikimedia.org/T236187 (10Dzahn) @wiki_willy Purchase date Dec. 4, 2015 Support contract — Support expiry date Dec. 5, 2018 ^ I guess this means we'll keep it around for another year or so in the spare pool. [23:58:28] 10serviceops, 10Operations, 10Parsoid-PHP, 10Patch-For-Review: wt2html: Out of memory crashers - https://phabricator.wikimedia.org/T236833 (10ssastry) >>! In T236833#5638686, @Krinkle wrote: > To keep things easy to reason about, I think it would be preferred to apply this to all app servers (more in comme... [23:58:30] 10serviceops, 10Operations: decom cobalt - https://phabricator.wikimedia.org/T236187 (10wiki_willy) @Dzahn - sounds good to me.