[08:51:03] !log tools.zppixbot-test decom [08:51:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [08:51:08] Reception123: ^ [09:00:48] !log tools.zppixbot-test decom complete [09:00:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [09:07:33] 2/3 decom's done [09:22:47] !log tools.zppixbot moved meetbot stuff to new host [09:22:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [09:59:07] !log tools.wdmm deployed 1abeb708cf [09:59:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wdmm/SAL [10:02:17] !log tools.wdmm deployed e69222c7b6 (service.template) [10:02:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wdmm/SAL [10:02:50] !log tools.wdmm deployed e0b49bc2a8 (toolforge.org) [10:02:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wdmm/SAL [11:01:58] !log tools live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/608849 (T256737) [11:02:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:02:02] T256737: Toolforge email: proper prometheus integration - https://phabricator.wikimedia.org/T256737 [11:19:45] !log tools cleanup exim email queue (4 frozen messages) [11:19:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:40:32] Hi, my wikitech username is my LDAP username, right? [13:41:18] Tanvir: Not necessarily [13:42:00] Reedy, okay, is it possible to change my Wikitech username to my regular Wikimedia username? [13:42:51] I think so. We don't do many renames on wikitech, but htey do happen [13:43:20] Reedy, is it possible for you or anyone to do so per IRC request, or should I file something there? [13:45:45] I want to change my LDAP username also to Wikitanvir (avoiding mismatches). [13:52:30] I think in the case of ldap username (and hence, shell), you have to create a new account [13:55:33] Okay, then I'll wait for them. Nothing urgent. [13:56:02] Thanks for the information! [13:56:45] Reedy, on another topic, you notified me on my bnwp talk page about high login rate of one of my bots. I think, I took care of the issue. Could you please take a look once you have time. [13:56:56] Thanks in advance. :) [13:57:37] Ttanvir: You could follow https://www.mediawiki.org/wiki/How_to_report_a_bug and create a Phab task with the project tags #wikitech.wikimedia.org and #LDAP [13:58:18] andre__, for the renaming / creating new account, you mean? [13:58:27] yes [13:58:54] Thanks, will do so. [18:32:32] Are any WMF projects accessible over IPv6? If I ping wikipedia.org, I'll get an IPv6 response, but the address is for the load balancer so you can't actually browse Wikipedia using IPv6 https://[2620:0:861:ed1a::1]/ [18:36:55] freephile: this would get you gerrit: http://[2620:0:861:2:208:80:154:137] [18:37:01] if you just want something to test [18:37:13] note the [ ] around it or browsers won't get it [18:38:03] cool. [18:38:18] first website I've browsed using IPv6 [18:38:35] the issue with Wikipedia is just a certificate issue i think [18:39:42] mutante: is there anything I can read to learn more about the implementation? [18:40:35] there's not much to see [18:41:15] I've researched using Google of course for the generic, but usually there are details in Wikimedia ops or whatever [18:41:21] freephile: in the case of gerrit, it comes down to basically just 2 things. adding an AAAA record in DNS and adding a Listen line in Apache config [18:41:40] OK [18:41:43] you can clone the DNS repo if you want, it's public. the line looks like this: [18:41:50] gerrit1001 1H IN AAAA 2620:0:861:2:208:80:154:136 [18:42:01] then in /etc/apache2/ports.conf you need something like: [18:42:07] Listen [2620:0:861:2:208:80:154:137]:80 [18:42:10] and that is basically it [18:42:25] the only things we don't have on IPv6 yet are wmflabs.org and toolforge.org iirc [18:43:05] this was a simpler example because it is not behind caching /loadbalancers [18:43:22] it just has a public IP(s) [18:43:56] right. I'm trying to figure out whether I can implement a full native IPv6 deployment, or go dual-stack and translate at the load balancer [18:46:50] freephile: the actual application servers that have mediawiki on them and are behind load balancer are one of the things that do not have IPv6 records yet. [18:47:08] freephile: but.. i don't think there is a translation. the LVS uses host names like in https://config-master.wikimedia.org/pybal/eqiad/appservers-https [18:47:20] so it just gets the v4 IP so far [18:47:58] aha [18:49:01] https://wikitech.wikimedia.org/wiki/LVS [18:49:19] not sure if that helps.. but yea [18:51:54] freephile: btw curl -6 https://en.wikipedia.org/wiki/Main_Page works and -6 tells curl to only use v6 when resolving the name [18:52:16] the issue you saw would happen for both IPv4 and IPv6 when using an IP, because of the TLS cert [18:53:42] thank you very much for the link and info [18:53:52] yw [19:38:01] !log toolserver-legacy creating instance relic-buster to eventually replace the existing relic-stretch (deprecated image) [19:38:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolserver-legacy/SAL [20:57:31] ia-upload fixes are in :) [21:02:51] nice [21:11:54] Uh, PDF. Didn't see that coming. [21:11:57] well, to be reviewed [21:12:04] no? how come? [21:52:39] because of past licensing concerns? [22:00:31] !log tools.xslack Killed eggdrop processes running on dev.toolforge.org bastion [22:00:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.xslack/SAL [22:02:00] !log tools.xslack Killed snitchbot.py process running on dev.toolforge.org bastion [22:02:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.xslack/SAL [22:02:09] fwiw, i just looked at the old "relic" instance that is there to redirect old toolserver.org URLs.. and it's surprisingly busy in the apache logs. tons of hits for osm-labels and tiles.. something using OSM is still linking to it .. hrmm [22:02:32] mutante: old URLs never die :) [22:03:10] yea.. i mean i expected a few hits, but not that kind of scrolling when i do tail -f , heh [22:03:18] and yeah, there are many many OSM legacy users that "leach" off of the maps project's tile server [22:03:27] a.www.toolserver.org b.www.toolserver.org ... [22:04:06] we should really move those to be directly on the new maps ingress server we built [22:04:42] * bd808 wonders how difficult that would be in practice [22:04:49] theh real reason i went to it is because "how is it possible this still uses letsencrypt::cert::integrated when almost nothing else uses it and where it was used it broke due to upstream changes.. but here it still works and the cert isnt expired [22:05:29] i don't know the answer yet, it should not work but does, haha [22:06:27] I'm not sure everyone else agrees with me, but I think we should move the toolserver redirector into the tools project. One benefit of that is that tools now has an acme-chief server to hand out the LE certs [22:07:17] mutante: I thought that LE integration method was going to continue to work for another 6 months or so? Its using a deprecated, but not disabled protocol right? [22:07:19] almost everything switched to acme-chief. this is one of the few remaining [22:07:47] yeah, K.renair made a phab task about that manifest somewhere [22:08:08] bd808: ah.. there we have the answer. ""Account creation on ACMEv1 is disabled." [22:08:16] it gets away with it because it already has an account [22:08:25] T252199 [22:08:25] T252199: Stop using letsencrypt::cert::integrated - https://phabricator.wikimedia.org/T252199 [22:08:32] but when we applied a role using it to a new instance.. you cant use it anymore [22:08:40] fun times [22:08:43] which would be fixed by https://gerrit.wikimedia.org/r/c/operations/puppet/+/602722 [22:08:49] most likely [22:09:10] anyways, i just checked what is still using that at all and the list is short [22:09:21] profile::mail::smarthost [22:09:21] toolserver_legacy [22:09:21] tlsproxy::localssl [22:09:22] that's it [22:09:47] cool. you may have jsut nerdsniped me into poking at toolserver_legacy [22:10:05] that's what made me do it :) i checked openstack-browser for "legacy" [22:10:11] to see if it's still even there [22:10:19] there is another "legacy" role but that is not this one [22:11:04] another way is to just install certbot and setup a cron for autorenewal [22:11:30] yeah. certbot would be the way to go for a new LE setup [22:11:41] i did that to work around it for gerrit in cloud [22:11:45] the meet.wmcloud.org instance is using it [22:11:55] because it's a separate cloud VPS project [22:12:01] ah, ack [22:12:30] it works fine for instances with a floating IP [22:12:41] inside prod there is no need because of acme-chief [22:12:41] as long as the security rule is there [22:13:10] and we now have acme-chief in the vps project that does the HTTP ingress for most other projects [22:13:32] yea, but that is just for deployment-prep [22:13:37] before the code was like "if in prod acme_chief, if in cloud letsencrypt::cert::integrated" [22:13:51] because the latter is broken, i replaced it with certbot [22:15:13] ah so someone else does use puppet haha [22:15:18] rabbit hole.. what i really wanted to say was "either we can replace the last 3 remaining ones using letsencrypt::cert and then delete that entire module... or we can merge paladox' change to sync acme-tiny with upstream to v2" [22:15:21] it's not in the cool club anymore but I like it [22:17:03] ningu: if we didn't have ~10 years of momentum behind using it I would switch to ansible in seconds [22:17:29] Puppet is very difficult for most folks to learn to use well [22:17:40] fair enough. I haven't tried ansible more than very briefly [22:17:52] what I like about puppet is also what's most difficult though -- the dependency graph [22:17:54] and we have a not great system here where we also mostly refuse to use manifests that others wrote [22:18:16] so its all bespoke and a lot of extra work for the Cloud Services team [22:18:28] I still don't really understand how ansible handles things that have to happen in a certain order, and making sure they are idempotent [22:18:37] most {{cn}} OpenStack deploys use Ansible [22:18:50] I guess you can explicitly specify the order but that seems tricky too [22:19:38] trickier than Puppet? [22:19:54] maybe I am just used to it [22:19:55] resource graph ordering is the worst in Puppet [22:20:55] for a while in the MediaWiki-Vagrant project we actually used the Puppet graph random order option just to help track down the unspecified dependencies in the graph [22:21:00] haha [22:21:37] because we had a large number of bug reports that turned out to be caused by ordering changes from slightly different ruby versions [22:21:49] yeah that is definitely a problem, I agree [22:55:41] Is it possible someone can get rid of the 2FA on my wikitech account... I think I lost that code a while ago... I'm happy to make an edit on my enwp user page verifying the request if needed [22:59:03] RichSmith: we have a process! [22:59:20] bd808: Where! [22:59:39] oh now it needs to be a documented process? ;) [22:59:53] HAHA [23:00:00] * bd808 looks to see if we actually have end user docs [23:00:05] Oh, the best features are undoumented [23:00:11] undocumented* [23:00:43] https://wikitech.wikimedia.org/wiki/Password_reset#For_users [23:00:59] Reedy: Cheers! [23:01:42] thanks Reedy. not where I was looking at all [23:02:02] google is great at indexing our sites [23:04:01] I was expecting it somewhere near https://wikitech.wikimedia.org/wiki/Password_reset#Reset_two_factor_authentication [23:40:23] Reedy: I want to make a wm-bot alias for the 2fa reset instructions. What you would you remember as a label for that? `!2fa`? [23:40:54] probably works, yeah [23:41:20] !2fa is https://wikitech.wikimedia.org/wiki/Password_reset#For_users [23:41:21] Key was added [23:41:35] !2fa|Reedy [23:41:41] !2fa | Reedy [23:41:41] Reedy: https://wikitech.wikimedia.org/wiki/Password_reset#For_users [23:41:46] w00t [23:42:00] I made that section fancier too [23:44:02] fancy indeed [23:45:02] !unicorn [23:45:02] 🦄 [23:45:19] I forgot about all these commands I setup ~3 years ago [23:46:29] !goat [23:46:42] such fail :( [23:47:16] !goat is 🐐 [23:47:16] Key was added [23:47:24] !goat [23:47:24] 🐐 [23:51:21] !beta alias deployment-prep [23:51:21] Created new alias for this key [23:51:35] !highfive act highfives $infobot_nick! [23:51:35] Key was added [23:51:38] !highfive [23:51:39] * wm-bot highfives bd808! [23:57:54] bd808: Thanks for that :) [23:58:05] RichSmith: yw [23:58:21] * RichSmith re-adds to Authy... [23:59:11] "Twillio Authy" now. Gotta get the new brand in there :) [23:59:30] +1 for Authy making 2fa easier to deal with [23:59:48] {{cn}}