[08:43:20] !log tools running disk performance test on etcd-4 [08:43:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:47:33] !log admin upgrading cinder on codfw cloudcontrol2* nodes (T278845) [08:47:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:47:37] T278845: cinder: Upgrade to same in codfw than eqiad - https://phabricator.wikimedia.org/T278845 [09:05:13] !log tools running disk performance test on etcd-8 [09:05:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:44:52] !log tools running disk performance test on etcd-4 (round2) [09:44:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:39:05] !log tools relocate some of the hiera keys for email server from project-level to prefix [14:39:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:45:30] !log tools created VM `tools-mail-03` as Debian Buster (T278538) [14:45:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:45:35] T278538: Toolforge: migrate mail server to Debian Buster - https://phabricator.wikimedia.org/T278538 [14:54:59] !log tools point floating IP 185.15.56.63 to tools-mail-03 (T278538) [14:55:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:55:07] T278538: Toolforge: migrate mail server to Debian Buster - https://phabricator.wikimedia.org/T278538 [14:56:45] !log tools shutoff tools-mail-02 (T278538) [14:56:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:03:35] !log tools created DNS A record `mail.tools.wmcloud.org` pointing to 185.15.56.63 [15:03:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:04:02] !log tools created MX record for `tools.wmcloud.org` pointing to `mail.tools.wmcloud.org` [15:04:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:57:22] !log tools rebooting `tools-mail-03` after enabling NFS (T [15:57:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:57:35] !log tools rebooting `tools-mail-03` after enabling NFS (T267082, T278538) [15:57:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:57:40] T278538: Toolforge: migrate mail server to Debian Buster - https://phabricator.wikimedia.org/T278538 [15:57:40] T267082: Rebuild Toolforge servers that should not have NFS mounted (and with affinity) - https://phabricator.wikimedia.org/T267082 [16:45:11] joakino: I'm testing out the new replica servers, and I'm finding a behavior difference... the `change-tag` table isn't working the same way. [16:45:47] how is it different ragesoss ? [16:45:58] In particular, I have some tests that run against this code on wikiedudashboard.toolforge.org https://github.com/WikiEducationFoundation/WikiEduDashboardTools/blob/master/public_html/revisions.php#L23 [16:46:58] It idea is to verify that a certain revision that was made via OAuth registers correctly as having been made by Wiki Education Dashboard. [16:48:02] It does so correctly on eqiad.wmflabs but that revision doesn't show up as having the relevant change tag when run on wikimedia.cloud replica. [16:49:43] ragesoss: sorry i'm on a meeting for the next 45', I'll have a look after it [16:50:00] ragesoss: is there a particular wiki that is failing for, or is it all of them? [16:50:05] no problem, I'm not in a rush. [16:50:20] My test runs against enwiki bd808 [16:50:25] in theory the view layer is the same, but maybe in practice there is some drift [16:52:32] Here's a test case... [16:52:45] New database; https://wikiedudashboard.toolforge.org/v8/revisions.php?db=enwiki&usernames[]=Petra+Sen&oauth_tags[]=OAuth+CID%3A+252&oauth_tags[]=OAuth+CID%3A+212&oauth_tags[]=OAuth+CID%3A+542&start=20160920003430&end=20160922003430 [16:53:01] Old database; https://wikiedudashboard.toolforge.org/revisions.php?db=enwiki&usernames[]=Petra+Sen&oauth_tags[]=OAuth+CID%3A+252&oauth_tags[]=OAuth+CID%3A+212&oauth_tags[]=OAuth+CID%3A+542&start=20160920003430&end=20160922003430 [16:54:15] I have a copy of the query code with the changed server naming in `/v8` so everything is identical between those two queries except for the switch to new replicas. [16:55:28] The new replicas return 'system: false' for the first revision (ie, not an edit made automatically by the Dashboard) while the second correctly returns 'system; true' for that first revision. [16:58:49] ragesoss: would you mind starting a phab task to track this? I have a feeling that it will need some digging into replication to figure out why the two clusters are giving different responses [16:59:00] sure [16:59:48] If you can isolate a query that we can run against the prod servers to decide which of the two responses is canonically correct that would be helpful. [17:04:44] right now PAWS can connect to both old and new cluster, may be useful to compare and share. here is an example connecting to both https://public.paws.wmcloud.org/User:JHernandez_(WMF)/Accessing%20the%20new%20replicas,%20changes%20from%20the%20previous%20cluster.ipynb [17:12:01] I added the query here; https://phabricator.wikimedia.org/T278983 [17:14:57] thanks ragesoss [18:22:13] !log toolsbeta redeploy ingress-admission controller with `kubectl apply -k deploys/toolsbeta` from the repo T275478 [18:22:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:22:17] T275478: toolsbeta: ingress admission controller doesn't accept valid FQDN patterns - https://phabricator.wikimedia.org/T275478