[00:00:01] (03PS1) 10CRusnov: netbox: Update configuration to support v2.7 [puppet] - 10https://gerrit.wikimedia.org/r/572123 [00:00:04] RoanKattouw, Niharika, and Urbanecm: Dear deployers, time to do the Evening SWAT(Max 6 patches) deploy. Dont look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200214T0000). [00:00:04] Zoranzoki21: A patch you scheduled for Evening SWAT(Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [00:00:22] I'm here :) [00:00:25] And ready for show [00:01:06] (03CR) 10CRusnov: "This change is ready for review." [puppet] - 10https://gerrit.wikimedia.org/r/572123 (owner: 10CRusnov) [00:01:33] 10Operations, 10ops-codfw, 10Patch-For-Review: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet - https://phabricator.wikimedia.org/T241337 (10Papaul) [00:02:10] (03CR) 10CRusnov: "FYI I have tested this change in af-nb-fe, and it works as expected (the stanzas for 2.7 are found and used by 2.7). We can merge this ASA" [puppet] - 10https://gerrit.wikimedia.org/r/572123 (owner: 10CRusnov) [00:02:52] Niharika: Are you online? [00:03:28] Zoranzoki21: I'm not clear why that's being backported [00:04:01] Reedy: To fix showing Times New Roman font in ContentTranslation [00:05:23] So mostly because the train hasn't finished this week... [00:05:52] The question is whether it will be next week. [00:06:12] I've been waiting for two weeks now. [00:06:59] This patch was only merged into master in time for .19... [00:08:24] And other users, who thankfully do not have vision problems, are already angry. I don't know what's going on anymore. Why is problem to this be in wmf.18 also? [00:08:40] No problem should happen. I know the procedure. [00:09:01] Just because there's a bug, and there's a fix, doesn't necessarily mean it should be backported [00:09:07] On other wikis that have wmf.19 everything with ContentTranslation is fine. [00:10:08] What rule does that say? This can't, other patches gets backported. [00:10:24] Just because there is a fix, doesn't mean it has to be backported [00:10:33] Not all bugfixes get backported [00:11:02] Zoranzoki21: backporting bugfixes takes work, usually from a volunteer [00:11:18] Difference between https://gerrit.wikimedia.org/r/c/561169/ and my patch is? [00:11:21] Zoranzoki21: that work could alternatively be spent fixing new bugs, which many choose to do [00:11:46] Difference between https://gerrit.wikimedia.org/r/c/561169/ and my patch is? [00:12:17] It's something that was reported on 21st January, wasn't fixed for 2 weeks later. It wasn't backported into .18 at the time [00:12:34] I'm just not sure why there's a sudden urgency to backport a fix [00:12:57] I'm not against deploying it, but there's an amount of "Why now?" [00:13:00] ? [00:14:01] Because the satisfaction of users who want to contribute is important. Nobody likes the current bugged interface of ContentTranslation [00:14:17] I don't see people complaining on the bug though [00:14:31] I understand it's a visibly annoying bug [00:15:34] Does this interface suit you from the perspective of the average user? [00:15:46] Zoranzoki21: I understand what your previous question was now -- whether or not a patch gets backported is determined in some part by whether or not someone wants to do it [00:15:49] It doesn't look pretty, but it's not broken [00:15:55] It works functionally, right? [00:17:50] So someone can deploy it, someone won't deploy it, someone doesn't have to deploy it... It's crazy.. Yes, it works, but not as expected. [00:18:11] It's also already been broken for 3 weeks [00:18:15] At least three weeks [00:18:31] Likely for months. [00:18:39] So, it should be broken and in next week [00:18:53] If the train had worked correctly, it would've been fixed [00:19:35] Yes, right. But the patch did not cause any problems with it [00:19:59] .18 has been active for 9 days, the fix was merged the day after .18 was cut [00:20:13] If someone thought it worthwhile to backport, why wasn't it done before? [00:20:36] Because they expected the train to go regularly, without blocking and problems [00:20:53] Anyone who expects the train to "go regularly" hasn't been here for long. :-) [00:21:03] So it was fine for the problem to exist for the last 8 days, but 9 days, no we must backport? [00:21:09] That makes absolutely no sense [00:22:04] James_F: I don't know another reason. [00:22:06] Reedy: Exactly, it doesn't make sense. [00:22:57] And it doesn't make sense why there still has to be a problem, when it can be solved. [00:23:19] Shall I also go and deploy all the other bug fixes I can find in the repos that have been committed since .18 was cut? [00:24:44] (no comment)... [00:29:58] Instead of sleeping, I'm waiting for someone to deploy patch (hoping that someone would be available at all, given that rarely when I added patch for this SWAT window in calendar was anyone at all available) [00:30:22] Such is the procedure, what shall I do... [00:30:52] Now I go to sleep really, I see that patch won't be deployed. Good night. [00:39:05] (03CR) 10CRusnov: "I have tested all reports that don't depend on external resources, and the dump. Everything works after porting, however Netbox 2.7 is sub" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/572116 (owner: 10CRusnov) [00:40:04] (03CR) 10CRusnov: [C: 04-1] "This appears to work from a test deploy on af-nb-fe. We should block deploying until the fix to the configuration in puppet is deployed. W" [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/572025 (owner: 10CRusnov) [00:41:13] (03PS1) 10EBernhardson: cirrus: redirect more_like to codfw to rebuild query cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/572126 (https://phabricator.wikimedia.org/T245202) [00:41:32] would this be the channel to ask about help with phab, or would it be in another channel? [00:43:59] b/c I did a bulk job on phab and it still says "exporting data" for me despite daemons marking the job as complete [00:44:24] ebernhardson: Should we deploy 572126 now? [00:44:58] Baylee: Umm. No idea, never seen that before, sorry. How many tasks did you try to batch-change? [00:45:25] (And if it's a lot, did you consider having it be run server-side quietly, rather than spamming a lot of people?) [00:46:33] I didn't try to batch change anything, I'm trying to export a CSV of all the tasks I'm subscribed to so I can see how many tasks I'm subscribed too (phab only shows 100 per page, and I don't feel like clicking forever since I think its more than 500) [00:46:43] James_F: we can, should be fine [00:46:54] OK, here goes everything. [00:46:58] (03CR) 10Jforrester: [C: 03+2] cirrus: redirect more_like to codfw to rebuild query cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/572126 (https://phabricator.wikimedia.org/T245202) (owner: 10EBernhardson) [00:47:56] (03Merged) 10jenkins-bot: cirrus: redirect more_like to codfw to rebuild query cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/572126 (https://phabricator.wikimedia.org/T245202) (owner: 10EBernhardson) [00:48:29] ebernhardson: Sync order matter? PoolCounterSettings then IS? [00:48:36] James_F: doesn't matter [00:48:40] OK. [00:48:50] Worth testing on mwdebug1001? [00:49:01] i can make a request or two, sure [00:49:03] (It's live there. [00:49:06] ) [00:49:53] James_F: seems fine [00:50:01] Okie-dokie. [00:51:22] it may hiccup a bit initially as the disk-caches in codfw are cold, but it's usually not a big problem. I ran a few "all indexes" queries which usually helps it warmup..maybe :) [00:51:39] * James_F crosses fingers and keeps staring at grafana. [00:51:50] !log jforrester@deploy1001 Synchronized wmf-config/PoolCounterSettings.php: T245202 cirrus: Increase the pool counter limits a bit (duration: 01m 05s) [00:51:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:51:54] T245202: RESTBase 500 spike of all /page/related/ hits following 1.35.0-wmf.19 all-wiki deployment - https://phabricator.wikimedia.org/T245202 [00:52:54] traffic is moving in the graphs at least [00:53:35] !log jforrester@deploy1001 Synchronized wmf-config/InitialiseSettings.php: T245202 cirrus: Move all move_like traffic to codfw (duration: 01m 02s) [00:53:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:55:17] wel, by "moving" it reports 12 qps, but i imagine it will go up :) [00:56:26] ebernhardson: Feel like this is stable to leave? [00:57:01] still waiting for graphs to catch up, but we've switched this numerous times i don't think there's ever been a proper problem (cross fingers :) [00:57:13] There's always a first time. :-) [00:57:23] this is usually the response to a problem :) [00:58:51] OK, I'm calling a lid. [00:59:08] Thanks all. Let's try to run the train in the PST morning. [00:59:12] kk [01:27:38] !log dpifke@deploy1001 Started deploy [performance/navtiming@2eec00a]: (no justification provided) [01:27:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:27:43] !log dpifke@deploy1001 Finished deploy [performance/navtiming@2eec00a]: (no justification provided) (duration: 00m 05s) [01:27:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:44:21] 10Operations, 10Analytics, 10Research, 10Traffic: Wikipedia Accessibility, check false positives and false negatives of traffic alarms - https://phabricator.wikimedia.org/T245166 (10Nuria) [01:44:22] (03PS1) 10Papaul: DHCP: Add NAC address for elastic2055 to elastic2060 [puppet] - 10https://gerrit.wikimedia.org/r/572128 (https://phabricator.wikimedia.org/T241337) [01:47:44] (03PS2) 10Papaul: DHCP: Add MAC address for elastic2055 to elastic2060 [puppet] - 10https://gerrit.wikimedia.org/r/572128 (https://phabricator.wikimedia.org/T241337) [01:51:34] (03CR) 10Papaul: [C: 03+2] DHCP: Add MAC address for elastic2055 to elastic2060 [puppet] - 10https://gerrit.wikimedia.org/r/572128 (https://phabricator.wikimedia.org/T241337) (owner: 10Papaul) [01:51:45] 10Operations, 10ops-eqiad, 10hardware-requests, 10Patch-For-Review: Decommission labsdb1002 - https://phabricator.wikimedia.org/T146455 (10wiki_willy) Chatted John a bit on this earlier, who was also talking to Rob about it last week. I think we're all good now, and this should be progressing along soon.... [01:56:25] (03PS1) 10Dave Pifke: Add Swift user for ArcLamp [puppet] - 10https://gerrit.wikimedia.org/r/572129 (https://phabricator.wikimedia.org/T244776) [02:05:05] 10Operations, 10ops-codfw, 10Patch-For-Review: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet - https://phabricator.wikimedia.org/T241337 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` elastic2055.cod... [02:07:51] 10Operations, 10ops-codfw, 10Patch-For-Review: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet - https://phabricator.wikimedia.org/T241337 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` elastic2056.cod... [02:21:19] (03PS3) 10C. Scott Ananian: Load Parsoid from the vendor repo, not from an ad-hoc deploy dir [mediawiki-config] - 10https://gerrit.wikimedia.org/r/572051 [03:00:23] 10Operations, 10ops-codfw: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet - https://phabricator.wikimedia.org/T241337 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['elastic2055.codfw.wmnet'] ` Of which those **FAILED**: ` ['elastic2055.codfw.wmnet'] ` [03:00:25] 10Operations, 10ops-codfw: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet - https://phabricator.wikimedia.org/T241337 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['elastic2056.codfw.wmnet'] ` Of which those **FAILED**: ` ['elastic2056.codfw.wmnet'] ` [03:36:16] James_F: do I you think I should open a phab task about it? (b/c phab is still exporting it despite it being marked as complete like 5 hours ago) [04:55:11] (03PS1) 10Dave Pifke: Scrape webperf Prometheus metrics [puppet] - 10https://gerrit.wikimedia.org/r/572141 (https://phabricator.wikimedia.org/T175087) [05:29:17] (03PS1) 10Marostegui: Revert "db2086: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/572144 [05:38:30] (03PS3) 10KartikMistry: Add config for OpusMT [deployment-charts] - 10https://gerrit.wikimedia.org/r/563110 (https://phabricator.wikimedia.org/T234194) [05:38:56] (03CR) 10KartikMistry: Add config for OpusMT (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/563110 (https://phabricator.wikimedia.org/T234194) (owner: 10KartikMistry) [05:40:23] (03CR) 10KartikMistry: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/416973 (owner: 10KartikMistry) [05:44:53] still "exporting data" so I'm going to go ahead and create a phab task [05:48:56] (https://phabricator.wikimedia.org/T245236) [06:12:14] 10Operations, 10Analytics: Remove references to m4-master - https://phabricator.wikimedia.org/T245238 (10Marostegui) [06:13:42] (03PS1) 10Marostegui: wmnet: Remove m4-master, m4-slave [dns] - 10https://gerrit.wikimedia.org/r/572148 (https://phabricator.wikimedia.org/T245238) [06:14:15] (03CR) 10Marostegui: "elukey, let me know if this can go or if we should point m4-master to db1108 and just remove m5-slave instead." [dns] - 10https://gerrit.wikimedia.org/r/572148 (https://phabricator.wikimedia.org/T245238) (owner: 10Marostegui) [06:16:09] (03CR) 10Marostegui: [C: 03+2] Revert "db2086: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/572144 (owner: 10Marostegui) [06:28:46] (03PS1) 10Vgutierrez: Revert "ATS: Disable allow_half_open in cp4031" [puppet] - 10https://gerrit.wikimedia.org/r/572149 (https://phabricator.wikimedia.org/T236458) [06:30:46] (03CR) 10Vgutierrez: [C: 03+2] Revert "ATS: Disable allow_half_open in cp4031" [puppet] - 10https://gerrit.wikimedia.org/r/572149 (https://phabricator.wikimedia.org/T236458) (owner: 10Vgutierrez) [06:44:38] (03CR) 10Vgutierrez: [C: 03+2] ssl_ciphersuite: Fix TLSv1.3 ciphersuites names [puppet] - 10https://gerrit.wikimedia.org/r/571978 (https://phabricator.wikimedia.org/T170567) (owner: 10Vgutierrez) [06:46:46] (03PS4) 10Vgutierrez: ssl_ciphersuite: Enable TLSv1.3 where available [puppet] - 10https://gerrit.wikimedia.org/r/571985 (https://phabricator.wikimedia.org/T170567) [06:50:44] (03PS5) 10Vgutierrez: ssl_ciphersuite: Enable TLSv1.3 where available [puppet] - 10https://gerrit.wikimedia.org/r/571985 (https://phabricator.wikimedia.org/T170567) [06:51:40] !log updating puppet compiler facts [06:51:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:58:09] (03CR) 10Elukey: [C: 03+1] wmnet: Remove m4-master, m4-slave [dns] - 10https://gerrit.wikimedia.org/r/572148 (https://phabricator.wikimedia.org/T245238) (owner: 10Marostegui) [06:58:29] (03CR) 10Marostegui: [C: 03+2] wmnet: Remove m4-master, m4-slave [dns] - 10https://gerrit.wikimedia.org/r/572148 (https://phabricator.wikimedia.org/T245238) (owner: 10Marostegui) [07:15:47] (03CR) 10Giuseppe Lavagetto: [C: 03+1] "one nitpick, but overall lgtm" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/571985 (https://phabricator.wikimedia.org/T170567) (owner: 10Vgutierrez) [07:25:10] (03PS6) 10Vgutierrez: ssl_ciphersuite: Enable TLSv1.3 where available [puppet] - 10https://gerrit.wikimedia.org/r/571985 (https://phabricator.wikimedia.org/T170567) [07:25:42] (03CR) 10Vgutierrez: ssl_ciphersuite: Enable TLSv1.3 where available (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/571985 (https://phabricator.wikimedia.org/T170567) (owner: 10Vgutierrez) [07:30:19] (03CR) 10Vgutierrez: [C: 03+2] ssl_ciphersuite: Enable TLSv1.3 where available [puppet] - 10https://gerrit.wikimedia.org/r/571985 (https://phabricator.wikimedia.org/T170567) (owner: 10Vgutierrez) [07:50:52] 10Operations, 10Traffic, 10Goal, 10Patch-For-Review, 10Performance-Team (Radar): Support TLSv1.3 - https://phabricator.wikimedia.org/T170567 (10Vgutierrez) Sites running nginx or apache outside the caching cluster that have been upgraded to buster are now offering TLSv1.3: a few examples: ` $ /usr/local... [07:52:21] (03PS1) 10Elukey: Add new role to irc2001.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/572191 (https://phabricator.wikimedia.org/T244719) [07:53:54] (03PS2) 10Elukey: Add new role to irc2001.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/572191 (https://phabricator.wikimedia.org/T244719) [07:56:01] (03PS5) 10Vgutierrez: ATS: Enable TLSv1.3 for ats-be <--> applayer communication [puppet] - 10https://gerrit.wikimedia.org/r/571976 (https://phabricator.wikimedia.org/T170567) [07:57:14] (03CR) 10Vgutierrez: [C: 03+1] Switch dns* to standard Partman recipes [puppet] - 10https://gerrit.wikimedia.org/r/571928 (https://phabricator.wikimedia.org/T156955) (owner: 10Muehlenhoff) [08:00:08] (03PS3) 10Muehlenhoff: Switch dns* to standard Partman recipes [puppet] - 10https://gerrit.wikimedia.org/r/571928 (https://phabricator.wikimedia.org/T156955) [08:02:39] (03CR) 10Muehlenhoff: [C: 03+2] Switch dns* to standard Partman recipes [puppet] - 10https://gerrit.wikimedia.org/r/571928 (https://phabricator.wikimedia.org/T156955) (owner: 10Muehlenhoff) [08:06:00] !log marostegui@cumin1001 dbctl commit (dc=all): 'Repool db1107 with weight 100 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10408 and previous config saved to /var/cache/conftool/dbconfig/20200214-080600-marostegui.json [08:06:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:06:05] T242702: Test MariaDB 10.4 in production - https://phabricator.wikimedia.org/T242702 [08:09:50] (03CR) 10Ema: [C: 03+1] ATS: Enable TLSv1.3 for ats-be <--> applayer communication [puppet] - 10https://gerrit.wikimedia.org/r/571976 (https://phabricator.wikimedia.org/T170567) (owner: 10Vgutierrez) [08:10:28] (03PS3) 10Elukey: Add new role to irc2001.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/572191 (https://phabricator.wikimedia.org/T244719) [08:14:22] (03CR) 10Alexandros Kosiaris: [C: 03+2] apt: remove unused package_updates custom fact [puppet] - 10https://gerrit.wikimedia.org/r/572101 (owner: 10Volans) [08:14:40] (03CR) 10Alexandros Kosiaris: [C: 03+2] "Thanks. one less piece of tech debt!" [puppet] - 10https://gerrit.wikimedia.org/r/572101 (owner: 10Volans) [08:16:06] (03PS1) 10Muehlenhoff: Switch more WMCS systems to standard Partman recipes [puppet] - 10https://gerrit.wikimedia.org/r/572196 (https://phabricator.wikimedia.org/T156955) [08:18:45] (03CR) 10Muehlenhoff: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/572191 (https://phabricator.wikimedia.org/T244719) (owner: 10Elukey) [08:36:00] (03PS1) 10Muehlenhoff: Remove obsolete Partman recipe [puppet] - 10https://gerrit.wikimedia.org/r/572197 (https://phabricator.wikimedia.org/T156955) [08:46:10] !log installing 4.19.98 kernel update on Buster systems [08:46:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:49:51] (03CR) 10Gehel: "Not really blocked on anything. I'm just not sure who should be pushing this forward since it is Wikidata related and neither David nor me" [puppet] - 10https://gerrit.wikimedia.org/r/526757 (https://phabricator.wikimedia.org/T222321) (owner: 10Smalyshev) [10:35:02] !log revert ats 8.0.6-rc0 experiment on cp40[26,32] [10:35:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:41:11] (03CR) 10Elukey: [C: 03+2] Add new role to irc2001.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/572191 (https://phabricator.wikimedia.org/T244719) (owner: 10Elukey) [10:44:48] 10Operations, 10Analytics, 10serviceops, 10vm-requests, and 2 others: Create a replacement for kraz.wikimedia.org - https://phabricator.wikimedia.org/T244719 (10elukey) 05Open→03Resolved a:03elukey [10:46:08] (03PS2) 10Muehlenhoff: Remove obsolete Partman recipe [puppet] - 10https://gerrit.wikimedia.org/r/572197 (https://phabricator.wikimedia.org/T156955) [11:08:31] (03CR) 10Muehlenhoff: [C: 03+2] Remove obsolete Partman recipe [puppet] - 10https://gerrit.wikimedia.org/r/572197 (https://phabricator.wikimedia.org/T156955) (owner: 10Muehlenhoff) [11:10:13] 10Operations, 10netops: RRDP status alert - https://phabricator.wikimedia.org/T245121 (10jbond) >https://rrdp.ripe.net/notification.xml=-1 it works with out `=-1` (i.e. https://rrdp.ripe.net/notification.xml) is that a typo? [11:23:52] (03PS1) 10Arturo Borrero Gonzalez: eqiad1.wikimedia.cloud: introduce delegation to the new Designate FQDNs [dns] - 10https://gerrit.wikimedia.org/r/572209 (https://phabricator.wikimedia.org/T243766) [11:24:06] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good to me" [puppet] - 10https://gerrit.wikimedia.org/r/571957 (owner: 10Jbond) [11:24:47] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] eqiad1.wikimedia.cloud: introduce delegation to the new Designate FQDNs [dns] - 10https://gerrit.wikimedia.org/r/572209 (https://phabricator.wikimedia.org/T243766) (owner: 10Arturo Borrero Gonzalez) [11:27:42] (03CR) 10Muehlenhoff: admin: add fr-tech-admins and allow dns-admin and gitpuppt privileges (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/572052 (owner: 10Jbond) [11:30:28] (03PS5) 10Jbond: admin: add fr-tech-admins and allow dns-admin and gitpuppt privileges [puppet] - 10https://gerrit.wikimedia.org/r/572052 [11:30:30] (03CR) 10Jbond: "thanks updated" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/572052 (owner: 10Jbond) [11:31:10] 10Operations, 10Chinese-Sites, 10Wikimedia-Incident: Tracking task: 2020-02-07 MW API server outage(s) - https://phabricator.wikimedia.org/T244578 (10Shizhao) [11:34:54] (03PS2) 10Filippo Giunchedi: Switch bast*/cumin*/scandium to standard Partman recipes [puppet] - 10https://gerrit.wikimedia.org/r/572014 (https://phabricator.wikimedia.org/T156955) [11:34:56] (03PS1) 10Filippo Giunchedi: Switch mw/mwdebug to standard Partman recipes [puppet] - 10https://gerrit.wikimedia.org/r/572211 (https://phabricator.wikimedia.org/T156955) [11:40:40] (03PS1) 10Arturo Borrero Gonzalez: cloud: refresh names for DNS servers in eqiad1/codfw1dev [puppet] - 10https://gerrit.wikimedia.org/r/572213 (https://phabricator.wikimedia.org/T243766) [11:47:20] (03PS2) 10Giuseppe Lavagetto: profile::configmaster: use wmflib::service functions [puppet] - 10https://gerrit.wikimedia.org/r/570070 [11:47:22] (03PS2) 10Giuseppe Lavagetto: profile::lvs::realserver: use wmflib::service::fetch [puppet] - 10https://gerrit.wikimedia.org/r/570071 [11:47:24] (03PS2) 10Giuseppe Lavagetto: profile::cache::base: remove the useless inclusion of lvs::configuration [puppet] - 10https://gerrit.wikimedia.org/r/570072 [11:47:26] (03PS1) 10Giuseppe Lavagetto: discoverydns: integrate into servicecatalog [puppet] - 10https://gerrit.wikimedia.org/r/572214 [11:47:28] (03PS1) 10Giuseppe Lavagetto: profile::lvs: use wmflib::fetch [puppet] - 10https://gerrit.wikimedia.org/r/572215 [11:51:02] 10Operations, 10LDAP-Access-Requests: Add Amooney to wmf LDAP Group - https://phabricator.wikimedia.org/T245215 (10Aklapper) Not sure if needed but https://phabricator.wikimedia.org/tag/ldap-access-requests/ also asks to define a purpose [11:53:28] (03CR) 10jerkins-bot: [V: 04-1] discoverydns: integrate into servicecatalog [puppet] - 10https://gerrit.wikimedia.org/r/572214 (owner: 10Giuseppe Lavagetto) [11:53:52] (03CR) 10jerkins-bot: [V: 04-1] profile::lvs: use wmflib::fetch [puppet] - 10https://gerrit.wikimedia.org/r/572215 (owner: 10Giuseppe Lavagetto) [11:57:53] (03CR) 10Lucas Werkmeister (WMDE): "If this and the next change (I862cd4628c) are deployed anywhere (Beta?), I’d be happy to test them and let you know if they seem to work c" [puppet] - 10https://gerrit.wikimedia.org/r/526757 (https://phabricator.wikimedia.org/T222321) (owner: 10Smalyshev) [12:12:05] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/572052 (owner: 10Jbond) [12:18:14] (03PS2) 10Giuseppe Lavagetto: discoverydns: integrate into servicecatalog [puppet] - 10https://gerrit.wikimedia.org/r/572214 [12:21:05] (03PS3) 10Giuseppe Lavagetto: discoverydns: integrate into servicecatalog [puppet] - 10https://gerrit.wikimedia.org/r/572214 [12:26:21] (03PS4) 10Giuseppe Lavagetto: discoverydns: integrate into servicecatalog [puppet] - 10https://gerrit.wikimedia.org/r/572214 [12:26:29] (03PS6) 10WMDE-leszek: Wikibase: added config variables to configure entity sources [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569031 (https://phabricator.wikimedia.org/T242087) [12:26:40] (03PS4) 10WMDE-leszek: Beta wikidata: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569204 (https://phabricator.wikimedia.org/T242087) [12:26:54] (03PS3) 10WMDE-leszek: Beta commons: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569205 (https://phabricator.wikimedia.org/T242087) [12:29:06] (03CR) 10Muehlenhoff: "Unfortunately we have some hosts for which even the new 80G / is not enough: I checked the current / size consumption of mw* hosts and the" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/572211 (https://phabricator.wikimedia.org/T156955) (owner: 10Filippo Giunchedi) [12:29:35] (03PS3) 10WMDE-leszek: Beta cluster: use entity source Wikibase setting for all wikibase-enabled wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569206 (https://phabricator.wikimedia.org/T242087) [12:29:44] (03PS5) 10Giuseppe Lavagetto: discoverydns: integrate into servicecatalog [puppet] - 10https://gerrit.wikimedia.org/r/572214 [12:29:46] (03PS4) 10WMDE-leszek: Beta commons: Remove custom wmgWikibaseRepoForeignRepositories setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569207 (https://phabricator.wikimedia.org/T242087) [12:34:28] (03CR) 10WMDE-leszek: "This change is ready for review." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569207 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:34:33] (03CR) 10WMDE-leszek: "This change is ready for review." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569206 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:34:43] (03PS4) 10WMDE-leszek: Beta cluster: remove custom wmgWikibaseClientRepositories settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569208 (https://phabricator.wikimedia.org/T242087) [12:34:52] (03PS4) 10WMDE-leszek: Test wikidata: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569209 (https://phabricator.wikimedia.org/T242087) [12:35:02] (03PS3) 10WMDE-leszek: Test wikibase clients: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569256 (https://phabricator.wikimedia.org/T242087) [12:35:11] (03PS3) 10WMDE-leszek: Test commons: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569257 (https://phabricator.wikimedia.org/T242087) [12:35:26] (03PS3) 10WMDE-leszek: Wikidata: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569258 (https://phabricator.wikimedia.org/T242087) [12:37:09] (03PS3) 10WMDE-leszek: Wikidata client wikis: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569259 (https://phabricator.wikimedia.org/T242087) [12:37:12] (03PS6) 10Giuseppe Lavagetto: discoverydns: integrate into servicecatalog [puppet] - 10https://gerrit.wikimedia.org/r/572214 [12:37:32] (03PS3) 10WMDE-leszek: Commons: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569260 (https://phabricator.wikimedia.org/T242087) [12:37:54] (03PS4) 10WMDE-leszek: Wikidata/Wikibase: use entity source Wikibase setting for all wikibase-enabled wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569261 (https://phabricator.wikimedia.org/T242087) [12:38:04] (03PS4) 10WMDE-leszek: Wikibase: Removed config option wmgUseEntitySourceBasedFederation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569263 (https://phabricator.wikimedia.org/T241975) [12:38:08] (03CR) 10WMDE-leszek: "This change is ready for review." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569263 (https://phabricator.wikimedia.org/T241975) (owner: 10WMDE-leszek) [12:38:13] (03CR) 10WMDE-leszek: "This change is ready for review." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569208 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:38:17] (03CR) 10WMDE-leszek: "This change is ready for review." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569209 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:38:22] (03CR) 10WMDE-leszek: "This change is ready for review." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569256 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:38:31] (03CR) 10WMDE-leszek: "This change is ready for review." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569257 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:38:36] (03CR) 10WMDE-leszek: "This change is ready for review." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569258 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:38:43] (03CR) 10jerkins-bot: [V: 04-1] Test commons: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569257 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:38:45] (03CR) 10WMDE-leszek: "This change is ready for review." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569259 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:38:47] (03CR) 10WMDE-leszek: "This change is ready for review." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569260 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:38:51] (03CR) 10WMDE-leszek: "This change is ready for review." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569261 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:39:23] (03CR) 10jerkins-bot: [V: 04-1] Wikibase: Removed config option wmgUseEntitySourceBasedFederation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569263 (https://phabricator.wikimedia.org/T241975) (owner: 10WMDE-leszek) [12:40:53] (03CR) 10WMDE-leszek: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569257 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:45:06] (03CR) 10Giuseppe Lavagetto: "https://puppet-compiler.wmflabs.org/compiler1001/20807/authdns1001.wikimedia.org/" [puppet] - 10https://gerrit.wikimedia.org/r/572214 (owner: 10Giuseppe Lavagetto) [12:45:10] (03CR) 10Muehlenhoff: "Actually just one host, data for the rest:" [puppet] - 10https://gerrit.wikimedia.org/r/572211 (https://phabricator.wikimedia.org/T156955) (owner: 10Filippo Giunchedi) [12:45:32] (03PS2) 10Vgutierrez: ATS: Add @reboot and @swap to the list of restricted syscalls groups [puppet] - 10https://gerrit.wikimedia.org/r/532348 [12:53:59] 10Operations, 10serviceops, 10Wikimedia-production-error: PHP7 corruption: Method call executed on unrelated object (also: Call to undefined method) - https://phabricator.wikimedia.org/T245183 (10Ladsgroup) I looked at the code base and couldn't find any instance that EntityLinkFormatterFactory object would... [13:01:37] (03CR) 10Vgutierrez: [C: 03+1] "LGTM (on Monday)" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/572214 (owner: 10Giuseppe Lavagetto) [13:04:45] (03CR) 10Muehlenhoff: [C: 03+1] "Sound good, those are now available in Buster!" [puppet] - 10https://gerrit.wikimedia.org/r/532348 (owner: 10Vgutierrez) [13:07:13] 10Operations, 10Analytics: Remove references to m4-master - https://phabricator.wikimedia.org/T245238 (10jbond) p:05Triage→03Medium [13:14:00] 10Operations, 10LDAP-Access-Requests: Add Amooney to wmf LDAP Group - https://phabricator.wikimedia.org/T245215 (10jbond) @AMooney have you created a developer account i don't seem to be able to find your account in ldap [13:14:15] 10Operations, 10LDAP-Access-Requests: Add Amooney to wmf LDAP Group - https://phabricator.wikimedia.org/T245215 (10jbond) p:05Triage→03Medium [13:14:48] 10Operations, 10Analytics, 10Research, 10Traffic: Wikipedia Accessibility, check false positives and false negatives of traffic alarms - https://phabricator.wikimedia.org/T245166 (10jbond) p:05Triage→03Medium [13:40:31] 10Operations, 10Traffic, 10Patch-For-Review, 10Performance-Team (Radar): Improve ATS backend connection reuse against origin servers - https://phabricator.wikimedia.org/T241145 (10ema) It turns out that cache_text nodes send quite a lot of RSTs to `appservers-rw`. See for example cp3054: ` 11:13:59 ema@c... [13:41:16] 10Operations, 10Discovery, 10Traffic, 10Wikidata, and 3 others: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service) - https://phabricator.wikimedia.org/T243701 (10Ladsgroup) >>! In T243701#5880988, @Bugreporter wrote: > The only way to resolve the issue is increase... [13:50:31] 10Operations, 10LDAP-Access-Requests: Add Amooney to wmf LDAP Group - https://phabricator.wikimedia.org/T245215 (10AMooney) @jbond I have added an account as AMooney just now. [13:50:41] !log restart relforge for JVM upgrade - T245120 [13:50:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:50:48] T245120: Restart Elasticsearch clusters for JVM upgrade - https://phabricator.wikimedia.org/T245120 [13:54:49] (03PS2) 10Filippo Giunchedi: Switch mw/mwmaint to standard Partman recipes [puppet] - 10https://gerrit.wikimedia.org/r/572211 (https://phabricator.wikimedia.org/T156955) [13:55:15] (03CR) 10Filippo Giunchedi: Switch mw/mwmaint to standard Partman recipes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/572211 (https://phabricator.wikimedia.org/T156955) (owner: 10Filippo Giunchedi) [13:55:26] 10Operations, 10Discovery, 10Traffic, 10Wikidata, and 3 others: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service) - https://phabricator.wikimedia.org/T243701 (10Bugreporter) Currently Widar use a policy of sleeping 3x+1 second if the lag (x second) is higher than... [13:58:24] (03PS2) 10Vgutierrez: ATS: Disable DNS resolution on ats-tls [puppet] - 10https://gerrit.wikimedia.org/r/571270 (https://phabricator.wikimedia.org/T244538) [14:02:19] 10Operations, 10Discovery, 10Traffic, 10Wikidata, and 3 others: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service) - https://phabricator.wikimedia.org/T243701 (10Strainu) >>! In T243701#5884801, @Ladsgroup wrote: > I have been thinking about this and I think I have... [14:05:41] 10Operations, 10Discovery, 10Traffic, 10Wikidata, and 3 others: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service) - https://phabricator.wikimedia.org/T243701 (10Bugreporter) By default pywikibot will do one edit every 10 seconds (or longer if you use parallel proc... [14:18:34] (03PS1) 10Jbond: puppetmaster: notify apache2 when the hiera.yaml file is updated [puppet] - 10https://gerrit.wikimedia.org/r/572230 (https://phabricator.wikimedia.org/T244222) [14:25:10] (03PS1) 10Jbond: admin: add amooney to ldap only group [puppet] - 10https://gerrit.wikimedia.org/r/572232 (https://phabricator.wikimedia.org/T245215) [14:30:31] 10Operations, 10User-jbond: CAS SSO: failed u2f registration - https://phabricator.wikimedia.org/T242438 (10jbond) 05Open→03Resolved a:03jbond cas reset his key and registration worked, re-open if a simlar issues is observed [14:31:21] 10Operations, 10Patch-For-Review, 10User-jbond: Systemd hardening of CAS service unit - https://phabricator.wikimedia.org/T233951 (10jbond) 05Open→03Resolved a:03jbond [14:31:23] 10Operations, 10Security-Team, 10User-jbond: Further steps for CAS/web SSO - https://phabricator.wikimedia.org/T233921 (10jbond) [14:31:49] 10Operations, 10Puppet, 10Packaging, 10User-jbond: Create a resources for installing components - https://phabricator.wikimedia.org/T240324 (10jbond) 05Open→03Resolved [14:34:46] (03PS1) 10Elukey: Remove some broken disks from Hadoop Test's hosts [puppet] - 10https://gerrit.wikimedia.org/r/572241 [14:35:36] (03CR) 10Andrew Bogott: [C: 03+2] puppetmaster: notify apache2 when the hiera.yaml file is updated [puppet] - 10https://gerrit.wikimedia.org/r/572230 (https://phabricator.wikimedia.org/T244222) (owner: 10Jbond) [14:35:59] (03CR) 10Elukey: [C: 03+2] Remove some broken disks from Hadoop Test's hosts [puppet] - 10https://gerrit.wikimedia.org/r/572241 (owner: 10Elukey) [14:40:41] (03PS1) 10Andrew Bogott: cloud: hiera: puppetmaster: refactor hiera (for VM instances) [puppet] - 10https://gerrit.wikimedia.org/r/572243 (https://phabricator.wikimedia.org/T229441) [15:09:25] (03PS1) 10Muehlenhoff: Add script to track OS migrations status (WIP) [puppet] - 10https://gerrit.wikimedia.org/r/572251 [15:11:47] (03CR) 10Andrew Bogott: [C: 03+2] cloud: hiera: puppetmaster: refactor hiera (for VM instances) [puppet] - 10https://gerrit.wikimedia.org/r/572243 (https://phabricator.wikimedia.org/T229441) (owner: 10Andrew Bogott) [15:12:25] (03CR) 10jerkins-bot: [V: 04-1] Add script to track OS migrations status (WIP) [puppet] - 10https://gerrit.wikimedia.org/r/572251 (owner: 10Muehlenhoff) [15:13:50] (03PS1) 10Alexandros Kosiaris: httpd: Add a new LogFormat, wmfjson [puppet] - 10https://gerrit.wikimedia.org/r/572252 [15:13:52] (03PS1) 10Alexandros Kosiaris: otrs: Add wmfjson logs to OTRS [puppet] - 10https://gerrit.wikimedia.org/r/572253 [15:17:23] (03CR) 10Alexandros Kosiaris: "After spending tons of time transforming apache logs from https://wikitech.wikimedia.org/wiki/Apache_log_format to json I realized that it" [puppet] - 10https://gerrit.wikimedia.org/r/572252 (owner: 10Alexandros Kosiaris) [15:18:10] (03CR) 10Alexandros Kosiaris: "And here's the first experiment" [puppet] - 10https://gerrit.wikimedia.org/r/572253 (owner: 10Alexandros Kosiaris) [15:21:27] (03PS1) 10Ayounsi: Add cookbook to control CF BGP advertisements [cookbooks] - 10https://gerrit.wikimedia.org/r/572262 [15:23:26] (03PS2) 10Muehlenhoff: Add script to track OS migrations status (WIP) [puppet] - 10https://gerrit.wikimedia.org/r/572251 [15:31:17] (03PS2) 10Ayounsi: Add cookbook to control CF BGP advertisements [cookbooks] - 10https://gerrit.wikimedia.org/r/572262 [15:31:56] (03PS3) 10Muehlenhoff: Add script to track OS migrations status [puppet] - 10https://gerrit.wikimedia.org/r/572251 [15:32:57] !log restart mc-gp* for updates [15:32:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:40:16] (03PS2) 10Bstorm: toolforge-k8s: add kube-bench config files for use on the cluster [puppet] - 10https://gerrit.wikimedia.org/r/572117 (https://phabricator.wikimedia.org/T240009) [15:40:42] (03PS1) 10Andrew Bogott: Cloud puppet hiera: rename all our 'host' trees to 'hosts' [puppet] - 10https://gerrit.wikimedia.org/r/572268 (https://phabricator.wikimedia.org/T229441) [15:41:23] (03PS1) 10CDanis: Manitoba: better served by codfw [dns] - 10https://gerrit.wikimedia.org/r/572269 [15:41:25] (03PS1) 10CDanis: Saskatchewan: ulsfo >> codfw > eqiad [dns] - 10https://gerrit.wikimedia.org/r/572270 [15:41:32] (03CR) 10Jbond: [C: 03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/572268 (https://phabricator.wikimedia.org/T229441) (owner: 10Andrew Bogott) [15:42:26] (03CR) 10Andrew Bogott: [C: 03+2] Cloud puppet hiera: rename all our 'host' trees to 'hosts' [puppet] - 10https://gerrit.wikimedia.org/r/572268 (https://phabricator.wikimedia.org/T229441) (owner: 10Andrew Bogott) [15:42:35] !log jmm@cumin2001 START - Cookbook sre.hosts.downtime [15:42:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:42:37] !log jmm@cumin2001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [15:42:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:44:00] (03CR) 10Bstorm: [C: 03+2] toolforge-k8s: add kube-bench config files for use on the cluster [puppet] - 10https://gerrit.wikimedia.org/r/572117 (https://phabricator.wikimedia.org/T240009) (owner: 10Bstorm) [15:46:19] !log ebernhardson@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Resync initialisesetting to try and pick up previoiusly deployed cirrus query routing changes (duration: 01m 05s) [15:46:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:52:08] !log uploaded pypuppetdb 0.3.3-2~wmf+deb9u1 to apt.wikimedia.org [15:52:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:54:43] !log marostegui@cumin1001 dbctl commit (dc=all): 'Repool db2086:3318 T239453', diff saved to https://phabricator.wikimedia.org/P10414 and previous config saved to /var/cache/conftool/dbconfig/20200214-155443-marostegui.json [15:54:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:54:48] T239453: Remove partitions from revision table - https://phabricator.wikimedia.org/T239453 [15:55:55] !log uploaded pypuppetdb 0.3.3-2~wmf+deb10u1 to apt.wikimedia.org [15:55:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:59:10] (03PS1) 10CDanis: maps.wm.o: reduce TTL from 1D to 10m [dns] - 10https://gerrit.wikimedia.org/r/572274 [15:59:32] 10Operations, 10Puppet, 10User-jijiki: Add require_package() variant with repository component to wmflib - https://phabricator.wikimedia.org/T178575 (10MoritzMuehlenhoff) 05Open→03Declined With package_from_component() I don't think we need this any longer, it serves a similar purpose and proper dependen... [16:01:03] (03CR) 10Muehlenhoff: [C: 03+1] admin: add amooney to ldap only group [puppet] - 10https://gerrit.wikimedia.org/r/572232 (https://phabricator.wikimedia.org/T245215) (owner: 10Jbond) [16:01:50] (03CR) 10Jbond: [C: 03+2] admin: add amooney to ldap only group [puppet] - 10https://gerrit.wikimedia.org/r/572232 (https://phabricator.wikimedia.org/T245215) (owner: 10Jbond) [16:10:56] 10Operations: Integrate Buster 10.3 point update - https://phabricator.wikimedia.org/T244693 (10MoritzMuehlenhoff) [16:11:56] !log installing git-lfs updates from Buster 10.3 point update [16:11:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:13:54] (03PS7) 10Holger Knust: Migrate changeprop & cpjobqueue to kubernetes [deployment-charts] - 10https://gerrit.wikimedia.org/r/554576 [16:14:50] 10Operations: Integrate Buster 10.3 point update - https://phabricator.wikimedia.org/T244693 (10MoritzMuehlenhoff) [16:15:55] (03PS8) 10Holger Knust: Migrate changeprop & cpjobqueue to kubernetes [deployment-charts] - 10https://gerrit.wikimedia.org/r/554576 [16:19:20] 10Operations, 10Analytics, 10serviceops, 10vm-requests, 10User-Elukey: decom kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10Dzahn) [16:20:01] (03PS1) 10Muehlenhoff: Remove obsolete Partman recipe [puppet] - 10https://gerrit.wikimedia.org/r/572275 (https://phabricator.wikimedia.org/T156955) [16:20:43] 10Operations: Integrate Buster 10.3 point update - https://phabricator.wikimedia.org/T244693 (10MoritzMuehlenhoff) [16:20:49] 10Operations, 10Analytics, 10serviceops: decommission kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10Dzahn) [16:21:12] 10Operations, 10Analytics, 10serviceops: decommission kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10Dzahn) a:05elukey→03Dzahn [16:22:47] 10Operations, 10Analytics, 10serviceops: decommission kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10elukey) As reminder for everybody (to avoid issues): please don't do it until irc2001 is fully set up and working with the new tool. We'll keep kraz working side to side with irc2001 for a... [16:23:05] 10Operations, 10Analytics, 10serviceops, 10vm-requests, 10User-Elukey: Create a replacement for kraz.wikimedia.org - https://phabricator.wikimedia.org/T244719 (10Dzahn) >@MoritzMuehlenhoff wrote: > Given that Luca also had an error during initial setup related to name resolution, this sounds like some er... [16:24:15] (03PS9) 10Holger Knust: Migrate changeprop & cpjobqueue to kubernetes [deployment-charts] - 10https://gerrit.wikimedia.org/r/554576 [16:28:12] 10Operations, 10Analytics, 10serviceops: decommission kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10Dzahn) 05Open→03Stalled Setting to stalled to reflect that. We should change status to Open when it's ready to go. [16:28:17] 10Operations, 10Analytics, 10serviceops, 10vm-requests, 10User-Elukey: Create a replacement for kraz.wikimedia.org - https://phabricator.wikimedia.org/T244719 (10Dzahn) [16:28:44] 10Operations, 10Analytics, 10serviceops: decommission kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10Dzahn) a:05Dzahn→03None [16:39:03] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "We need around 150 GB for /var/log, now that we also keep the forensic log." [puppet] - 10https://gerrit.wikimedia.org/r/572211 (https://phabricator.wikimedia.org/T156955) (owner: 10Filippo Giunchedi) [16:40:42] 10Operations, 10ops-eqiad: Degraded RAID on cloudvirt1024 - https://phabricator.wikimedia.org/T245282 (10ops-monitoring-bot) [16:44:09] 10Operations, 10ops-eqiad: Degraded RAID on cloudvirt1024 - https://phabricator.wikimedia.org/T245282 (10JHedden) [16:44:15] 10Operations, 10ops-eqiad, 10Patch-For-Review, 10cloud-services-team (Hardware): Degraded RAID on cloudvirt1024 - https://phabricator.wikimedia.org/T241884 (10JHedden) [16:48:39] 10Operations, 10LDAP-Access-Requests: Add Amooney to wmf LDAP Group - https://phabricator.wikimedia.org/T245215 (10jbond) @AMooney you should have wmf access now please reopen if you still have issues authentication [16:51:02] 10Operations, 10ops-eqiad, 10Patch-For-Review, 10cloud-services-team (Hardware): Degraded RAID on cloudvirt1024 - https://phabricator.wikimedia.org/T241884 (10JHedden) We're still having the same issue after the firmware and BIOS upgrades. We've been trying to identify this issue since last August T230289#... [16:51:18] (03CR) 10BBlack: [C: 03+1] maps.wm.o: reduce TTL from 1D to 10m [dns] - 10https://gerrit.wikimedia.org/r/572274 (owner: 10CDanis) [16:53:46] (03CR) 10BBlack: [C: 03+1] add wikiźródła.pl and wikisłownik.pl, link to parking [dns] - 10https://gerrit.wikimedia.org/r/567163 (https://phabricator.wikimedia.org/T240446) (owner: 10Dzahn) [16:59:34] 10Operations, 10ops-eqiad, 10cloud-services-team, 10Patch-For-Review: Degraded RAID on cloudvirt1024 -- Filesystem mounted read-only - https://phabricator.wikimedia.org/T230289 (10Bstorm) [17:01:55] (03CR) 10Dzahn: [C: 03+2] add wikiźródła.pl and wikisłownik.pl, link to parking [dns] - 10https://gerrit.wikimedia.org/r/567163 (https://phabricator.wikimedia.org/T240446) (owner: 10Dzahn) [17:01:59] (03PS2) 10Dzahn: add wikiźródła.pl and wikisłownik.pl, link to parking [dns] - 10https://gerrit.wikimedia.org/r/567163 (https://phabricator.wikimedia.org/T240446) [17:02:11] (03CR) 10Effie Mouzeli: [C: 03+1] "The idea is great! We have a few possible things to consider if we later on decide to switch to this format (eg mtail), but I'd like to t" [puppet] - 10https://gerrit.wikimedia.org/r/572252 (owner: 10Alexandros Kosiaris) [17:03:28] 10Operations, 10Puppet, 10User-jbond: Add check for changes applied at all runs - https://phabricator.wikimedia.org/T242910 (10jbond) I had a quick look at this, i dont think anything can be don on the node its self but as you say we can query puppet db. The following script is a quick and dirty example of... [17:07:10] 10Operations, 10netops: RRDP status alert - https://phabricator.wikimedia.org/T245121 (10ayounsi) It's because Grafana reports Routinator pulling data from https://rrdp.ripe.net/notification.xml as a -1 on its graph. Where I think -1 means timeout. So the correct URL is https://rrdp.ripe.net/notification.xml... [17:08:10] marxarelli: Heads-up; I'm going to try out the Wikibase fix for T245062 on wmf.19. [17:08:11] T245062: Double ?uselang= passed to a file results in "Cache key contains characters that are not allowed: `P180_1101514477_fr?uselang=fr_label`" - https://phabricator.wikimedia.org/T245062 [17:18:48] (03CR) 10RLazarus: [C: 03+1] "%i escapes quotation marks, but there's nothing else that a header could do to trip up a JSON parser, right?" [puppet] - 10https://gerrit.wikimedia.org/r/572252 (owner: 10Alexandros Kosiaris) [17:20:32] 10Operations, 10Beta-Cluster-Infrastructure: Upgrade puppet in deployment-prep (Puppet agent broken in Beta Cluster) - https://phabricator.wikimedia.org/T243226 (10herron) Turning off debug logging on puppetmaster04 (I set `logdest = /dev/null` in `/etc/puppet/puppet.conf`) has helped with the disk usage, and... [17:22:30] 10Operations, 10observability, 10serviceops, 10Patch-For-Review: Stream a subset of mediawiki apache logs to logstash - https://phabricator.wikimedia.org/T244472 (10herron) Learned today that T243226 is tracking the current beta cluster puppetmaster issues [17:22:35] (03PS1) 10Jbond: tlsproxy::localssl: change duplicate definition detection [puppet] - 10https://gerrit.wikimedia.org/r/572282 (https://phabricator.wikimedia.org/T242910) [17:25:34] (03CR) 10jerkins-bot: [V: 04-1] tlsproxy::localssl: change duplicate definition detection [puppet] - 10https://gerrit.wikimedia.org/r/572282 (https://phabricator.wikimedia.org/T242910) (owner: 10Jbond) [17:31:53] (03PS6) 10Dzahn: wmflib: replace install1002 with bast1002 in ipresolve_spec [puppet] - 10https://gerrit.wikimedia.org/r/569688 (https://phabricator.wikimedia.org/T224576) [17:32:11] (03PS2) 10Jbond: tlsproxy::localssl: change duplicate definition detection [puppet] - 10https://gerrit.wikimedia.org/r/572282 (https://phabricator.wikimedia.org/T242910) [17:32:13] (03CR) 10Ppchelko: [C: 04-1] Migrate changeprop & cpjobqueue to kubernetes (0317 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/554576 (owner: 10Holger Knust) [17:32:42] (03CR) 10jerkins-bot: [V: 04-1] wmflib: replace install1002 with bast1002 in ipresolve_spec [puppet] - 10https://gerrit.wikimedia.org/r/569688 (https://phabricator.wikimedia.org/T224576) (owner: 10Dzahn) [17:33:14] I'm live on mwdebug1001. [17:33:54] !log jforrester@deploy1001 Locking from deployment [ALL REPOSITORIES]: Testing T245062 fix on mwdebug1001 (planned duration: 60m 00s) [17:33:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:33:58] T245062: Double ?uselang= passed to a file results in "Cache key contains characters that are not allowed: `P180_1101514477_fr?uselang=fr_label`" - https://phabricator.wikimedia.org/T245062 [17:34:32] James_F: why Friday deploys and the term "try out" ? [17:34:44] Monday is holiday and the deployment calendar is empty [17:35:25] 10Operations, 10ops-esams: Terminate OE10,11,12,13 Racks - https://phabricator.wikimedia.org/T237055 (10wiki_willy) Latest update from our Iron Mountain account rep - he's still chasing down his Legal team to get some guidance. His proposal to them was to cancel the 4 cabinets and renew OE14 with a term of 6... [17:35:33] mutante: UBN testing. If it does indeed fix it, I'll talk to SRE about whether it's OK to deploy. [17:35:38] (03PS7) 10Effie Mouzeli: WIP mediawiki: send apache logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/571239 (https://phabricator.wikimedia.org/T244472) [17:36:05] James_F: i refer to the talk in -releng about not deploying today [17:36:09] Which, indeed, it seems to do. [17:36:54] mutante: Hmm. That's the totally wrong channel to have that kind of discussion in; prod issues should always be discussed in here. [17:36:55] * James_F sighs. [17:37:00] !log jforrester@deploy1001 Unlocked for deployment [ALL REPOSITORIES]: Testing T245062 fix on mwdebug1001 (duration: 03m 05s) [17:37:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:37:47] Anyway. Time to talk to SRE. [17:37:52] nevertheless the calendar is completely empty and the mail said specifically that the train would run on Monday [17:38:14] mutante: This isn't the train? [17:38:32] mutante: That's a different UBN. We had eight this week. [17:38:43] (03PS8) 10Effie Mouzeli: WIP mediawiki: send apache logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/571239 (https://phabricator.wikimedia.org/T244472) [17:38:56] The calendar is *always* empty on Fridays. [17:39:11] (03CR) 10Effie Mouzeli: WIP mediawiki: send apache logs to logstash (036 comments) [puppet] - 10https://gerrit.wikimedia.org/r/571239 (https://phabricator.wikimedia.org/T244472) (owner: 10Effie Mouzeli) [17:47:53] 10Operations, 10Beta-Cluster-Infrastructure: Upgrade puppet in deployment-prep (Puppet agent broken in Beta Cluster) - https://phabricator.wikimedia.org/T243226 (10Andrew) That last issue (the resolution failure) was a side-effect of work I was doing for T229441. That issue is resolved, but now the failure us... [17:51:18] 10Operations, 10LDAP-Access-Requests: LDAP access to the wmf group for CherRaye Glenn - https://phabricator.wikimedia.org/T244410 (10CGlenn) Thank you! @Dzahn [17:51:28] 10Operations, 10observability, 10serviceops, 10Patch-For-Review: Stream a subset of mediawiki apache logs to logstash - https://phabricator.wikimedia.org/T244472 (10jijiki) Oh great, thanks! From https://logstash-beta.wmflabs.org/, it looks that logs are flowing, so maybe we can try the change there after all [17:52:24] 10Operations, 10LDAP-Access-Requests: Add Amooney to wmf LDAP Group - https://phabricator.wikimedia.org/T245215 (10jbond) 05Open→03Resolved a:03jbond [17:55:44] 10Operations, 10Puppet, 10Patch-For-Review, 10User-jbond: puppet: Custom type providers - https://phabricator.wikimedia.org/T241160 (10jbond) p:05Triage→03Medium [18:05:28] 10Operations, 10LDAP-Access-Requests: Request LDAP access to the WMF group for Edna M - https://phabricator.wikimedia.org/T244176 (10Edna) Hi! Apologies for the late reply! I was participating in an event post All Hands. I´ve verfied this information: - shell user name: emedina - wikitech user name: Ed... [18:06:25] (03PS5) 10VolkerE: Fix latin Wikipedia (VICIPÆDIA) wordmark [mediawiki-config] - 10https://gerrit.wikimedia.org/r/571838 (https://phabricator.wikimedia.org/T240728) [18:06:51] (03PS3) 10Jbond: tlsproxy::localssl: change duplicate definition detection [puppet] - 10https://gerrit.wikimedia.org/r/572282 (https://phabricator.wikimedia.org/T242910) [18:14:48] (03CR) 10Dzahn: "seems a good idea to try it on OTRS first. Where do i find the definition of "wmfjson" though?" [puppet] - 10https://gerrit.wikimedia.org/r/572253 (owner: 10Alexandros Kosiaris) [18:14:58] 10Operations, 10LDAP-Access-Requests: Request LDAP access to the WMF group for Edna M - https://phabricator.wikimedia.org/T244176 (10Iflorez) Hi All! I work with Edna on the Partnerships team. I am a Partnerships data analyst focusing on analyzing the data coming out of the GLOW project. Edna and I are on th... [18:17:46] (03PS1) 10Elukey: Set minimum memory/vcpu for Yarn Fair schduler in Hadoop Test [puppet] - 10https://gerrit.wikimedia.org/r/572295 (https://phabricator.wikimedia.org/T244499) [18:18:10] (03CR) 10Dzahn: "asking because if "wmfjson" is just a nickname then shouldn't i find it in a separate LogFormat directive elsewhere?" [puppet] - 10https://gerrit.wikimedia.org/r/572253 (owner: 10Alexandros Kosiaris) [18:23:06] (03PS7) 10Dzahn: wmflib: replace install1002 with bast1002 in ipresolve_spec [puppet] - 10https://gerrit.wikimedia.org/r/569688 (https://phabricator.wikimedia.org/T224576) [18:24:17] (03CR) 10Elukey: [C: 03+2] Set minimum memory/vcpu for Yarn Fair schduler in Hadoop Test [puppet] - 10https://gerrit.wikimedia.org/r/572295 (https://phabricator.wikimedia.org/T244499) (owner: 10Elukey) [18:29:43] (03CR) 10Jdlrobson: [C: 03+1] "This is much better." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/571838 (https://phabricator.wikimedia.org/T240728) (owner: 10VolkerE) [18:29:45] (03CR) 10Dzahn: [C: 03+2] wmflib: replace install1002 with bast1002 in ipresolve_spec [puppet] - 10https://gerrit.wikimedia.org/r/569688 (https://phabricator.wikimedia.org/T224576) (owner: 10Dzahn) [18:30:17] (03CR) 10Dzahn: "using a bastion host for the DNS test because new install servers will move to private IPs" [puppet] - 10https://gerrit.wikimedia.org/r/569688 (https://phabricator.wikimedia.org/T224576) (owner: 10Dzahn) [18:31:46] (03CR) 10Dzahn: "duh, ignore my comments above, i now see unmerged https://gerrit.wikimedia.org/r/c/operations/puppet/+/572253 as well and had just search" [puppet] - 10https://gerrit.wikimedia.org/r/572252 (owner: 10Alexandros Kosiaris) [18:32:17] (03CR) 10Dzahn: [C: 03+1] "duh, ignore my comments above, i now see unmerged https://gerrit.wikimedia.org/r/c/operations/puppet/+/572253 as well and had just search" [puppet] - 10https://gerrit.wikimedia.org/r/572253 (owner: 10Alexandros Kosiaris) [18:38:49] (03CR) 10Dzahn: "looks good. and since we will be trying it on OTRS first i'd give it a go. Only thing that stood out to me was how protocol after url is h" [puppet] - 10https://gerrit.wikimedia.org/r/572252 (owner: 10Alexandros Kosiaris) [18:39:37] (03CR) 10Dzahn: [C: 03+1] "but then we wouldn't use the custom LogFormat in those configs, so +1" [puppet] - 10https://gerrit.wikimedia.org/r/572252 (owner: 10Alexandros Kosiaris) [18:41:26] (03PS1) 10Reedy: Don't use 'ip' as a logging parameter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/572300 (https://phabricator.wikimedia.org/T245280) [18:58:25] (03CR) 10Cwhite: [C: 03+1] "+1 idempotency!" [puppet] - 10https://gerrit.wikimedia.org/r/572282 (https://phabricator.wikimedia.org/T242910) (owner: 10Jbond) [19:00:22] (03PS2) 10CRusnov: netbox: Update configuration to support v2.7 [puppet] - 10https://gerrit.wikimedia.org/r/572123 (https://phabricator.wikimedia.org/T244291) [19:01:12] (03CR) 10CRusnov: "> Patch Set 1: Code-Review-1" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/572123 (https://phabricator.wikimedia.org/T244291) (owner: 10CRusnov) [19:01:55] (03CR) 10jerkins-bot: [V: 04-1] netbox: Update configuration to support v2.7 [puppet] - 10https://gerrit.wikimedia.org/r/572123 (https://phabricator.wikimedia.org/T244291) (owner: 10CRusnov) [19:01:59] (03CR) 10CRusnov: "Latest Patchset also adds systemd unit for the RQ worker as specified in the RQ docs." [puppet] - 10https://gerrit.wikimedia.org/r/572123 (https://phabricator.wikimedia.org/T244291) (owner: 10CRusnov) [19:04:35] 10Operations, 10Beta-Cluster-Infrastructure: Upgrade puppet in deployment-prep (Puppet agent broken in Beta Cluster) - https://phabricator.wikimedia.org/T243226 (10Krenair) a:03Krenair [19:05:44] (03PS3) 10CRusnov: netbox: Update configuration to support v2.7 [puppet] - 10https://gerrit.wikimedia.org/r/572123 (https://phabricator.wikimedia.org/T244291) [19:09:59] !log jforrester@deploy1001 Synchronized php-1.35.0-wmf.19/extensions/Wikibase: T245062 Prevent invalid term languages from cached PrefetchingTermLookup (duration: 01m 09s) [19:10:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:10:04] T245062: Double ?uselang= passed to a file results in "Cache key contains characters that are not allowed: `P180_1101514477_fr?uselang=fr_label`" - https://phabricator.wikimedia.org/T245062 [19:10:12] 10Operations, 10serviceops-radar, 10vm-requests: vm requests for APT repo / webserver - https://phabricator.wikimedia.org/T244626 (10Dzahn) [19:21:46] (03PS1) 10Dzahn: add apt1001.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/572311 (https://phabricator.wikimedia.org/T244626) [19:34:41] (03PS2) 10Dzahn: add apt1001.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/572311 (https://phabricator.wikimedia.org/T244626) [19:39:28] (03CR) 10Dzahn: [C: 04-2] "merging this is too early/blocked i guess (ticket is stalled) but comments on the related ticket would be appreciated nevertheless" [dns] - 10https://gerrit.wikimedia.org/r/567558 (https://phabricator.wikimedia.org/T241944) (owner: 10Dzahn) [19:46:33] (03PS2) 10CRusnov: Bump Netbox revision to v2.7.4 [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/572025 (https://phabricator.wikimedia.org/T244291) [19:47:34] (03PS1) 10Dzahn: site: add apt[12]001.wikimedia.org with role::apt_repo [puppet] - 10https://gerrit.wikimedia.org/r/572312 (https://phabricator.wikimedia.org/T224576) [19:48:49] (03PS1) 10Andrew Bogott: Fix a few invocations of osm::planet_sync [puppet] - 10https://gerrit.wikimedia.org/r/572313 (https://phabricator.wikimedia.org/T239728) [19:49:47] (03CR) 10Andrew Bogott: "This change left several inclusions of this class broken... that is fixed in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/572313" [puppet] - 10https://gerrit.wikimedia.org/r/559581 (https://phabricator.wikimedia.org/T239728) (owner: 10Mathew.onipe) [19:49:50] (03CR) 10CRusnov: [C: 04-1] "> Patch Set 1: Code-Review-1" (031 comment) [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/572025 (https://phabricator.wikimedia.org/T244291) (owner: 10CRusnov) [19:54:54] (03CR) 10Andrew Bogott: [C: 03+2] Fix a few invocations of osm::planet_sync [puppet] - 10https://gerrit.wikimedia.org/r/572313 (https://phabricator.wikimedia.org/T239728) (owner: 10Andrew Bogott) [19:55:30] (03PS1) 10Cmjohnson: Adding mgmt dns for snapshot1010 [dns] - 10https://gerrit.wikimedia.org/r/572314 (https://phabricator.wikimedia.org/T241794) [19:57:38] (03CR) 10Cmjohnson: [C: 03+2] Adding mgmt dns for snapshot1010 [dns] - 10https://gerrit.wikimedia.org/r/572314 (https://phabricator.wikimedia.org/T241794) (owner: 10Cmjohnson) [20:01:12] (03PS1) 10Cmjohnson: Adding production dns for snapshot1010 [dns] - 10https://gerrit.wikimedia.org/r/572315 (https://phabricator.wikimedia.org/T241794) [20:01:48] (03PS1) 10Andrew Bogott: osm::planet_sync:hours is a list of ints, not of strings [puppet] - 10https://gerrit.wikimedia.org/r/572316 (https://phabricator.wikimedia.org/T239728) [20:02:39] (03PS2) 10Cmjohnson: Adding production dns for snapshot1010 [dns] - 10https://gerrit.wikimedia.org/r/572315 (https://phabricator.wikimedia.org/T241794) [20:04:01] (03CR) 10Cmjohnson: [C: 03+2] Adding production dns for snapshot1010 [dns] - 10https://gerrit.wikimedia.org/r/572315 (https://phabricator.wikimedia.org/T241794) (owner: 10Cmjohnson) [20:05:50] 10Operations, 10ops-eqiad, 10Dumps-Generation, 10Patch-For-Review: (Need By Jan 25) rack/setup/install snapshot1010.eqiad.wmnet - https://phabricator.wikimedia.org/T241794 (10Cmjohnson) [20:05:54] (03CR) 10Andrew Bogott: [C: 03+2] osm::planet_sync:hours is a list of ints, not of strings [puppet] - 10https://gerrit.wikimedia.org/r/572316 (https://phabricator.wikimedia.org/T239728) (owner: 10Andrew Bogott) [20:07:02] 10Operations, 10ops-eqiad, 10Dumps-Generation, 10Patch-For-Review: (Need By Jan 25) rack/setup/install snapshot1010.eqiad.wmnet - https://phabricator.wikimedia.org/T241794 (10Cmjohnson) @Jclark-ctr 10.65.2.44 is mgmt ip [20:13:31] (03CR) 10Brennen Bearnes: "I'm low on context for this, but I guess it makes sense. Do we need to document the process for upgrading composer somewhere?" [puppet] - 10https://gerrit.wikimedia.org/r/561684 (https://phabricator.wikimedia.org/T218900) (owner: 10Dzahn) [20:19:09] (03CR) 10Reedy: [C: 03+2] Don't use 'ip' as a logging parameter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/572300 (https://phabricator.wikimedia.org/T245280) (owner: 10Reedy) [20:20:06] (03Merged) 10jenkins-bot: Don't use 'ip' as a logging parameter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/572300 (https://phabricator.wikimedia.org/T245280) (owner: 10Reedy) [20:20:35] (03CR) 10Dzahn: "we can probably do this for a bunch of other things now that have port 80 firewall holes they don't actually use anymore." [puppet] - 10https://gerrit.wikimedia.org/r/569100 (owner: 10Dzahn) [20:21:56] !log reedy@deploy1001 Synchronized wmf-config/CommonSettings.php: Prevent some logspam T245280 (duration: 01m 05s) [20:21:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:22:00] T245280: logstash_formatter_key_conflict in mediawiki logs - https://phabricator.wikimedia.org/T245280 [20:32:17] (03PS3) 10CRusnov: Bump Netbox revision to v2.7.4 [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/572025 (https://phabricator.wikimedia.org/T244291) [20:38:55] (03PS1) 10Cmjohnson: Adding production dns for new es servers [dns] - 10https://gerrit.wikimedia.org/r/572327 (https://phabricator.wikimedia.org/T241359) [20:41:11] 10Operations, 10ops-eqiad, 10DBA, 10Patch-For-Review: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Cmjohnson) dns has been updated, I need to verify the network ports. I have a conflict with es1025...that port shows utilized (@Jcl... [20:50:19] (03PS1) 10Dzahn: noc::site: remove port 80 firewall hole [puppet] - 10https://gerrit.wikimedia.org/r/572337 [20:53:34] (03PS1) 10Gergő Tisza: Increase Commons purge rate limit for patrollers [mediawiki-config] - 10https://gerrit.wikimedia.org/r/572339 (https://phabricator.wikimedia.org/T245214) [20:53:51] 10Operations, 10ops-eqiad, 10DBA, 10Patch-For-Review: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Cmjohnson) [20:58:53] (03PS1) 10Dzahn: role::noc::site: refactor role/profile, stop duplicate include [puppet] - 10https://gerrit.wikimedia.org/r/572343 [20:59:48] (03CR) 10jerkins-bot: [V: 04-1] role::noc::site: refactor role/profile, stop duplicate include [puppet] - 10https://gerrit.wikimedia.org/r/572343 (owner: 10Dzahn) [21:01:11] !log volker-e@deploy1001 Started deploy [design/style-guide@1928c00]: Deploy design/style-guide: [21:01:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:01:19] !log volker-e@deploy1001 Finished deploy [design/style-guide@1928c00]: Deploy design/style-guide: (duration: 00m 09s) [21:01:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:02:07] 10Operations, 10ops-eqiad, 10serviceops: (Need By Feb 28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10wiki_willy) [21:08:54] (03PS1) 10Dzahn: wikimania_scholarships: remove port 80 firewall hole [puppet] - 10https://gerrit.wikimedia.org/r/572350 [21:09:31] 10Operations, 10Analytics, 10decommission, 10serviceops: decommission kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10RobH) [21:10:59] (03PS1) 10Dzahn: microsites::httpd: remove port 80 firewall hole [puppet] - 10https://gerrit.wikimedia.org/r/572352 [21:12:21] (03PS1) 10Dzahn: releases: remove port 80 firewall hole [puppet] - 10https://gerrit.wikimedia.org/r/572353 [21:14:14] (03PS1) 10Dzahn: add apt[12]00[12] to partman recipes, flat, standard for VMs [puppet] - 10https://gerrit.wikimedia.org/r/572354 [21:19:01] (03CR) 10Cmjohnson: [C: 03+2] Adding production dns for new es servers [dns] - 10https://gerrit.wikimedia.org/r/572327 (https://phabricator.wikimedia.org/T241359) (owner: 10Cmjohnson) [21:25:54] (03PS1) 10Jhedden: icinga: add cloudvps ceph alerts [puppet] - 10https://gerrit.wikimedia.org/r/572355 (https://phabricator.wikimedia.org/T240715) [21:28:56] (03CR) 10jerkins-bot: [V: 04-1] icinga: add cloudvps ceph alerts [puppet] - 10https://gerrit.wikimedia.org/r/572355 (https://phabricator.wikimedia.org/T240715) (owner: 10Jhedden) [21:31:15] (03CR) 10Dzahn: icinga: add cloudvps ceph alerts (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/572355 (https://phabricator.wikimedia.org/T240715) (owner: 10Jhedden) [21:33:01] (03PS2) 10Jhedden: icinga: add cloudvps ceph alerts [puppet] - 10https://gerrit.wikimedia.org/r/572355 (https://phabricator.wikimedia.org/T240715) [21:36:13] (03CR) 10Jhedden: "> Patch Set 1:" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/572355 (https://phabricator.wikimedia.org/T240715) (owner: 10Jhedden) [21:36:30] (03CR) 10Jhedden: "PCC results https://puppet-compiler.wmflabs.org/compiler1002/20812/" [puppet] - 10https://gerrit.wikimedia.org/r/572355 (https://phabricator.wikimedia.org/T240715) (owner: 10Jhedden) [21:41:03] (03CR) 10Masumrezarock100: [C: 03+1] Increase Commons purge rate limit for patrollers [mediawiki-config] - 10https://gerrit.wikimedia.org/r/572339 (https://phabricator.wikimedia.org/T245214) (owner: 10Gergő Tisza) [21:46:41] (03CR) 10Dzahn: [C: 03+2] icinga: add cloudvps ceph alerts [puppet] - 10https://gerrit.wikimedia.org/r/572355 (https://phabricator.wikimedia.org/T240715) (owner: 10Jhedden) [21:55:16] (03CR) 10Dzahn: "3 new checks added to Icinga and all are green now" [puppet] - 10https://gerrit.wikimedia.org/r/572355 (https://phabricator.wikimedia.org/T240715) (owner: 10Jhedden) [21:57:17] 10Operations, 10LDAP-Access-Requests: LDAP access to the wmf group for CherRaye Glenn - https://phabricator.wikimedia.org/T244410 (10Dzahn) @CGlenn You can but there is an extra step involved where @ottomata or @elukey need to sync users (per https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hue#Access) [21:58:20] 10Operations, 10LDAP-Access-Requests: LDAP access to the wmf group for CherRaye Glenn - https://phabricator.wikimedia.org/T244410 (10Dzahn) 05Invalid→03Open @ottomata or @elukey Please sync users for Hue access ^. [21:58:41] 10Operations, 10LDAP-Access-Requests: LDAP access to the wmf group for CherRaye Glenn (superset, turnilo, hue) - https://phabricator.wikimedia.org/T244410 (10Dzahn) [21:58:59] 10Operations, 10LDAP-Access-Requests: LDAP access to the wmf group for CherRaye Glenn (superset, turnilo, hue) - https://phabricator.wikimedia.org/T244410 (10Dzahn) a:05jijiki→03elukey [21:59:25] 10Operations, 10Analytics, 10LDAP-Access-Requests: LDAP access to the wmf group for CherRaye Glenn (superset, turnilo, hue) - https://phabricator.wikimedia.org/T244410 (10Dzahn) [22:13:13] (03CR) 10Brian Wolff: [C: 03+1] Increase Commons purge rate limit for patrollers [mediawiki-config] - 10https://gerrit.wikimedia.org/r/572339 (https://phabricator.wikimedia.org/T245214) (owner: 10Gergő Tisza) [22:16:53] (03PS2) 10Herron: logstash: add ES 7 compatible logstash template [puppet] - 10https://gerrit.wikimedia.org/r/571622 (https://phabricator.wikimedia.org/T234854) [22:28:39] (03PS2) 10Effie Mouzeli: WIP hieradata: test streaming apache logs to logstash from mwdebug1001 [puppet] - 10https://gerrit.wikimedia.org/r/572057 (https://phabricator.wikimedia.org/T244472) [22:30:41] (03CR) 10Brian Wolff: "Meh, now I think maybe this should just be reverted." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477005 (owner: 10Brian Wolff) [22:31:16] 10Operations, 10ops-eqiad, 10Dumps-Generation: (Need By Jan 25) rack/setup/install snapshot1010.eqiad.wmnet - https://phabricator.wikimedia.org/T241794 (10Jclark-ctr) [22:36:36] 10Operations, 10ops-eqiad, 10Dumps-Generation: (Need By Jan 25) rack/setup/install snapshot1010.eqiad.wmnet - https://phabricator.wikimedia.org/T241794 (10Jclark-ctr) configured bios and idrac. [22:44:14] (03CR) 10Brian Wolff: [C: 04-1] "actually i think this should be linkpurge" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/572339 (https://phabricator.wikimedia.org/T245214) (owner: 10Gergő Tisza) [22:56:21] 10Operations, 10MediaWiki-Revision-backend: Compress data at external storage - https://phabricator.wikimedia.org/T106386 (10Anomie) 05Open→03Stalled This can't be done until Flow's data isn't going to be deleted when the recompression is run. But now the tasks for moving Flow's data out of the MW External... [22:57:54] 10Operations, 10MediaWiki-Revision-backend: Compress data at external storage - https://phabricator.wikimedia.org/T106386 (10Anomie) [22:58:13] (03CR) 10SBassett: "> Meh, now I think maybe this should just be reverted." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477005 (owner: 10Brian Wolff) [23:14:37] (03CR) 10Brian Wolff: "yeah" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477005 (owner: 10Brian Wolff) [23:28:27] 10Operations, 10ops-eqiad, 10DC-Ops: mr1-eqiad.wikimedia.org - Duplicate IP on mgmt network - https://phabricator.wikimedia.org/T245320 (10wiki_willy)