[00:01:08] (03CR) 10Dzahn: [C: 032] "noop" [puppet] - 10https://gerrit.wikimedia.org/r/476996 (owner: 10Dzahn) [00:05:27] (03PS3) 10Cwhite: role, profile: install, run, and collect icinga exporter metrics [puppet] - 10https://gerrit.wikimedia.org/r/476431 (https://phabricator.wikimedia.org/T208066) [00:06:23] (03PS4) 10Cwhite: role, profile: install, run, and collect icinga exporter metrics [puppet] - 10https://gerrit.wikimedia.org/r/476431 (https://phabricator.wikimedia.org/T208066) [00:14:21] (03PS7) 10Paladox: phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 [00:14:27] (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [00:22:11] (03PS5) 10Faidon Liambotis: quotereviewer: move Declined Remote Consulting Service back to optional [software] - 10https://gerrit.wikimedia.org/r/476981 (owner: 10RobH) [00:22:26] (03CR) 10Faidon Liambotis: [C: 032] quotereviewer: move Declined Remote Consulting Service back to optional [software] - 10https://gerrit.wikimedia.org/r/476981 (owner: 10RobH) [00:23:39] (03Merged) 10jenkins-bot: quotereviewer: move Declined Remote Consulting Service back to optional [software] - 10https://gerrit.wikimedia.org/r/476981 (owner: 10RobH) [00:27:53] (03PS1) 10Bstorm: sonofgridengine: found where the unconfigurable pid file is [puppet] - 10https://gerrit.wikimedia.org/r/477002 (https://phabricator.wikimedia.org/T200557) [00:31:39] (03CR) 10Bstorm: [C: 032] sonofgridengine: found where the unconfigurable pid file is [puppet] - 10https://gerrit.wikimedia.org/r/477002 (https://phabricator.wikimedia.org/T200557) (owner: 10Bstorm) [01:07:53] (03PS8) 10Paladox: phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 [01:08:48] (03CR) 10jerkins-bot: [V: 04-1] phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [01:14:57] PROBLEM - ensure kvm processes are running on labvirt1014 is CRITICAL: PROCS CRITICAL: 0 processes with regex args /usr/bin/kvm [01:15:30] Oh man, there's no limit to how many times I can trigger that. Sorry all. [01:16:26] (03PS1) 10Brian Wolff: Also log authevents channel. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477005 [01:16:29] ACKNOWLEDGEMENT - ensure kvm processes are running on labvirt1014 is CRITICAL: PROCS CRITICAL: 0 processes with regex args /usr/bin/kvm andrew bogott This is on purpose -- sorry everyone! [01:18:46] (03PS9) 10Paladox: phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 [01:19:50] (03CR) 10jerkins-bot: [V: 04-1] phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [01:21:25] (03PS10) 10Paladox: phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 [01:22:48] (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [01:24:30] (03PS11) 10Paladox: phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 [01:24:36] (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [01:26:06] (03PS12) 10Paladox: phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 [01:26:15] (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [01:29:12] (03PS13) 10Paladox: phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 [01:29:19] (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [01:53:42] (03PS1) 10Andrew Bogott: Rename labvirt1014 to cloudvirt1014 [puppet] - 10https://gerrit.wikimedia.org/r/477008 (https://phabricator.wikimedia.org/T210904) [01:59:53] (03CR) 10Gergő Tisza: [C: 031] "Does what it says. I think it largely duplicates other logs and the data it has is more specialized and less useful, though - in the long " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477005 (owner: 10Brian Wolff) [03:11:07] 10Operations, 10Cloud-Services, 10cloud-services-team (Kanban): tools-k8s-master-01 has two floating IPs - https://phabricator.wikimedia.org/T164123 (10GTirloni) * All ~/.kube/config files use https://k8s-master.tools.wmflabs.org:6443 * k8s-master.tools.wmflabs.org resolves to 10.68.17.142 on all tools serve... [03:13:28] (03PS14) 10Paladox: phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 [03:14:35] (03CR) 10jerkins-bot: [V: 04-1] phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [03:17:45] (03PS1) 10GTirloni: openstack: Fix unsafe keystone token cleanup SQL query [puppet] - 10https://gerrit.wikimedia.org/r/477012 (https://phabricator.wikimedia.org/T210405) [03:20:05] (03CR) 10GTirloni: [C: 032] openstack: Fix unsafe keystone token cleanup SQL query [puppet] - 10https://gerrit.wikimedia.org/r/477012 (https://phabricator.wikimedia.org/T210405) (owner: 10GTirloni) [03:25:56] (03PS15) 10Paladox: phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 [03:26:02] (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [03:28:03] (03PS16) 10Paladox: phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 [03:28:09] (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [03:32:27] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 868.11 seconds [03:32:29] (03Abandoned) 10Paladox: phabricator: Replace mod_php with php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/407958 (https://phabricator.wikimedia.org/T182832) (owner: 10Paladox) [03:43:13] (03CR) 10Brian Wolff: "I was going back and forth between this patch & changing confirmedit to consistently use captcha as the lig channel" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477005 (owner: 10Brian Wolff) [04:10:39] PROBLEM - Wikitech-static main page has content on labweb1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:10:53] PROBLEM - Wikitech-static main page has content on labweb1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:11:45] RECOVERY - Wikitech-static main page has content on labweb1002 is OK: HTTP OK: HTTP/1.1 200 OK - 33882 bytes in 2.369 second response time [04:11:57] RECOVERY - Wikitech-static main page has content on labweb1001 is OK: HTTP OK: HTTP/1.1 200 OK - 33880 bytes in 0.251 second response time [04:12:20] andrewbogott: ^^ [04:26:23] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 172.65 seconds [05:43:54] 10Operations, 10Cloud-Services, 10cloud-services-team (Kanban): tools-k8s-master-01 has two floating IPs - https://phabricator.wikimedia.org/T164123 (10GTirloni) a:03GTirloni [06:07:08] (03PS1) 10GTirloni: toolforge: Increase shinken 'High iowait' time period to 30min [puppet] - 10https://gerrit.wikimedia.org/r/477013 (https://phabricator.wikimedia.org/T161898) [06:07:53] (03CR) 10GTirloni: [C: 032] toolforge: Increase shinken 'High iowait' time period to 30min [puppet] - 10https://gerrit.wikimedia.org/r/477013 (https://phabricator.wikimedia.org/T161898) (owner: 10GTirloni) [06:28:08] PROBLEM - Check systemd state on netmon2001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [06:28:20] PROBLEM - netbox HTTPS on netmon1002 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 547 bytes in 0.009 second response time [06:29:45] PROBLEM - Check systemd state on netmon1002 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [06:37:52] RECOVERY - netbox HTTPS on netmon1002 is OK: HTTP OK: HTTP/1.1 302 Found - 348 bytes in 0.550 second response time [06:38:06] RECOVERY - Check systemd state on netmon1002 is OK: OK - running: The system is fully operational [06:38:54] RECOVERY - Check systemd state on netmon2001 is OK: OK - running: The system is fully operational [06:42:28] PROBLEM - Check systemd state on netmon2001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [07:08:40] RECOVERY - Check systemd state on netmon2001 is OK: OK - running: The system is fully operational [09:00:08] RECOVERY - Logstash rate of ingestion percent change compared to yesterday on icinga1001 is OK: (C)130 ge (W)110 ge 50.08 https://grafana.wikimedia.org/dashboard/db/logstash?orgId=1panelId=2fullscreen [09:09:36] PROBLEM - pdfrender on scb1002 is CRITICAL: connect to address 10.64.16.21 and port 5252: Connection refused [10:27:49] (03CR) 10Legoktm: [C: 031] "+2" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/476280 (https://phabricator.wikimedia.org/T210528) (owner: 10Anomie) [11:41:04] PROBLEM - Logstash rate of ingestion percent change compared to yesterday on icinga1001 is CRITICAL: 131.4 ge 130 https://grafana.wikimedia.org/dashboard/db/logstash?orgId=1panelId=2fullscreen [12:33:26] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:52] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.002 second response time [13:26:27] (03PS1) 10Revi: Add SPF record for wikidata.org [dns] - 10https://gerrit.wikimedia.org/r/477034 (https://phabricator.wikimedia.org/T210134) [13:45:45] 10Operations, 10DNS, 10Mail, 10Traffic, and 3 others: wikidata.org lacks SPF record - https://phabricator.wikimedia.org/T210134 (10Krenair) [13:47:06] (03CR) 10Alex Monk: "It looks like wikipedia.org also has this, should it be applied to most wikimedia-controlled domains?" [dns] - 10https://gerrit.wikimedia.org/r/477034 (https://phabricator.wikimedia.org/T210134) (owner: 10Revi) [14:44:55] 10Operations, 10Gadgets, 10MediaWiki-Cache, 10MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), and 4 others: Mcrouter periodically reports soft TKOs for mc[1,2]035 leading to MW Memcached exceptions - https://phabricator.wikimedia.org/T203786 (10Nikerabbit) Is the value so big that every time it is SET it causes... [15:56:15] (03PS2) 10Fomafix: Avoid redirects from HTTPS to HTTP and back to HTTPS [puppet] - 10https://gerrit.wikimedia.org/r/469262 [16:07:38] (03PS4) 10Daimona Eaytoy: Update AbuseFilter config to keep the status quo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/475772 [16:07:48] (03PS4) 10Daimona Eaytoy: Clarify docs for AbuseFilter emergency threshold [mediawiki-config] - 10https://gerrit.wikimedia.org/r/475337 [16:09:12] (03CR) 10jerkins-bot: [V: 04-1] Update AbuseFilter config to keep the status quo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/475772 (owner: 10Daimona Eaytoy) [16:25:18] (03PS17) 10Paladox: phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 [16:25:31] (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [16:26:51] (03CR) 10jerkins-bot: [V: 04-1] phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [16:28:30] (03CR) 10jerkins-bot: [V: 04-1] phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [16:31:39] (03PS18) 10Paladox: phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 [16:32:50] (03CR) 10jerkins-bot: [V: 04-1] phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [16:39:46] (03PS19) 10Paladox: phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 [16:39:52] (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [16:48:26] !log rebuilding labvirt1014 as cloudvirt1014, T210904 [16:48:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:48:30] T210904: Rename labvirt1014 to cloudvirt1014, move to eqiad1 - https://phabricator.wikimedia.org/T210904 [16:49:32] (03PS2) 10Andrew Bogott: Rename labvirt1014 to cloudvirt1014 [puppet] - 10https://gerrit.wikimedia.org/r/477008 (https://phabricator.wikimedia.org/T210904) [16:51:24] (03PS3) 10Andrew Bogott: Rename labvirt1014 to cloudvirt1014 [puppet] - 10https://gerrit.wikimedia.org/r/477008 (https://phabricator.wikimedia.org/T210904) [16:53:11] (03CR) 10Andrew Bogott: [C: 032] Rename labvirt1014 to cloudvirt1014 [puppet] - 10https://gerrit.wikimedia.org/r/477008 (https://phabricator.wikimedia.org/T210904) (owner: 10Andrew Bogott) [16:54:02] (03PS1) 10Andrew Bogott: rename labvirt1014 to cloudvirt1014 [dns] - 10https://gerrit.wikimedia.org/r/477053 (https://phabricator.wikimedia.org/T210904) [16:54:18] (03PS2) 10Andrew Bogott: rename labvirt1014 to cloudvirt1014 [dns] - 10https://gerrit.wikimedia.org/r/477053 (https://phabricator.wikimedia.org/T210904) [16:54:49] (03CR) 10Andrew Bogott: [C: 032] rename labvirt1014 to cloudvirt1014 [dns] - 10https://gerrit.wikimedia.org/r/477053 (https://phabricator.wikimedia.org/T210904) (owner: 10Andrew Bogott) [16:59:17] (03PS1) 10Andrew Bogott: clean up remaining labvirt1014.mgmt entries [dns] - 10https://gerrit.wikimedia.org/r/477056 (https://phabricator.wikimedia.org/T210904) [17:00:49] (03PS1) 10Andrew Bogott: cloudvirt1014: turn on monitoring and add to the scheduler pool [puppet] - 10https://gerrit.wikimedia.org/r/477057 (https://phabricator.wikimedia.org/T210904) [17:02:47] (03PS20) 10Paladox: phabricator: Add support for php-fpm [puppet] - 10https://gerrit.wikimedia.org/r/476985 [17:02:54] (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/476985 (owner: 10Paladox) [17:28:25] (03CR) 10Andrew Bogott: [C: 032] clean up remaining labvirt1014.mgmt entries [dns] - 10https://gerrit.wikimedia.org/r/477056 (https://phabricator.wikimedia.org/T210904) (owner: 10Andrew Bogott) [17:50:37] (03PS1) 10Daimona Eaytoy: Move all AbuseFilter config to abusefilter.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477063 [17:51:02] (03Abandoned) 10Daimona Eaytoy: Clarify docs for AbuseFilter emergency threshold [mediawiki-config] - 10https://gerrit.wikimedia.org/r/475337 (owner: 10Daimona Eaytoy) [17:51:22] (03CR) 10jerkins-bot: [V: 04-1] Move all AbuseFilter config to abusefilter.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477063 (owner: 10Daimona Eaytoy) [17:53:36] (03PS2) 10Daimona Eaytoy: Move all AbuseFilter config to abusefilter.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477063 (https://phabricator.wikimedia.org/T145931) [18:42:49] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:13:47] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [19:41:39] RECOVERY - Logstash rate of ingestion percent change compared to yesterday on icinga1001 is OK: (C)130 ge (W)110 ge 106.5 https://grafana.wikimedia.org/dashboard/db/logstash?orgId=1panelId=2fullscreen [19:51:18] 10Operations, 10Traffic, 10Wikimedia-General-or-Unknown, 10media-storage: Loading of the full version of images from Commons stucks - https://phabricator.wikimedia.org/T210890 (10Aklapper) (Removing #Wikimedia-production-error tag as we don't have a stacktrace of a crash, but not sure about the correct tag... [19:51:59] 10Operations, 10Traffic, 10Wikimedia-General-or-Unknown, 10media-storage: Loading full versions of larger images from Commons stucks / repeatedly gets interrupted after a few MBs - https://phabricator.wikimedia.org/T210890 (10Aklapper) [21:09:52] (03Abandoned) 10Jon Harald Søby: Enable arbitrary Wikidata access for oldwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/454570 (https://phabricator.wikimedia.org/T202543) (owner: 10Jon Harald Søby) [21:49:03] Evening [21:49:18] Got a note about yet more failed login attempts [21:49:44] I appreciate that this has been going on for some time but it's not funny anymore [21:50:17] When are WMF secuirty going to start cutting loose any ISP that's acting as platform for this? [22:02:54] As in massive range blocks [22:11:39] ShakespeareFan00, you're proposing that WMF start blocking entire ISPs from connecting to wikipedia because some of them have users which are making dodgy login attempts? [22:13:14] Yes. [22:13:36] At least until those ISP do something about zombie proxies and users doing iffy things [22:14:15] I.e "Terminating" the ability of what seems to be a botnet to operate [22:15:15] if it's a botnet rather than the attackers directly sending traffic from their own machines you'll have even less luck with that approach [22:15:37] Krenair: Something has to be done [22:15:44] because it's beyond a joke now [22:16:00] so you don't care whether anything effective is done [22:16:05] just that someone is seen to be doing something [22:16:16] Krenair: I'd like an effective response as well [22:16:27] have you considered a career in politics? [22:16:33] The 'termination' of those responsible from being able to continue it [22:16:51] What would you consider to be an effective response? [22:17:10] Presumably ISP' abuse teams are already being contacted? [22:17:36] account security increases [22:18:05] Well, I already have a strong password [22:18:18] (I can't use 2FA as I don't have a suitable device) [22:18:39] then why do you care? [22:19:04] Because I don't like organised crackers attacking ... [22:19:17] They should be "terminated" from being able to continue [22:19:32] You can't stop them trying [22:19:42] if you started denying all login attempts from them, you'd still have to record an attempt was made [22:19:42] I can't indviaully [22:20:05] Yes and use that as evidence to pass the authorities to get them shut down [22:20:44] its not that easy though..... [22:21:03] Also blocking entire ISP's will lock other users out from accessing wikipedia [22:21:20] I think he's suggesting that be used as leverage paladox [22:22:28] I would oppose that. [22:24:41] Krenair: A sort of Wikipedia equivalent of the old school Usenet Death Penalty [22:24:52] (If you know what that was) [22:25:28] ? [22:26:26] UDP was where an entire news server was de-peered because it had failed to tackle issues like bulk unsolicited commerical postings for example [22:26:59] It was generally effective in getting news servers to implement proper acceptabel use policy [22:27:23] which worked until Google Groups... [22:28:24] Your asking wikimedia to block a entire ISP because what some users were miss using it? That would go against wikipedia's values of free information for everyone. [22:28:37] ShakespeareFan00: you can use 2FA with your desktop/laptop computer as the device [22:29:04] (i do it, works fine) [22:29:13] Okay noted... [22:30:14] paladox: It's not just misuse, it that's some users on that ISP's have either 'zombie' proxies or are actively trying to compromise accounts which is illegal [22:30:28] I am suspecting the former. [22:30:38] Yes, but in the process you will be eliminating good faith users. [22:31:05] ShakespeareFan00: if you're just bothered by the notification, i think you can turn them off in preferences. i'm unfamiliar with the situation and whether anything can or should be done to prevent the actual login attempts [22:31:20] if you're worried about your account, and can set up 2FA, then i recommend doing just that :) [22:31:34] MatmaRex: Can you set up a way to redirect Special:Login to a [22:32:03] "You can't login in right now because attempts were made to compromise accounts from this IP or device?" [22:32:10] type page [22:32:26] pretty sure there is already a rate limit on login attempts [22:32:46] which would be effectively the same thing, i guess? [22:32:52] Not necessarily [22:33:40] The page redirected to would also need to say something along the line that trying to compromise accounts is illegal and that WMF is logging all attempts [22:33:57] and will share those details with law enforcement. [22:34:09] https://meta.wikimedia.org/wiki/Help:Two-factor_authentication#Enabling_two-factor_authentication recommends some desktop 2FA applications (personally i use a Yubikey and their authenticator app, but you'd need to buy the dongle, so that might not an option) [22:34:34] ShakespeareFan00: surely whoever is trying doesn't care :D [22:35:04] (03CR) 10Andrew Bogott: [C: 032] cloudvirt1014: turn on monitoring and add to the scheduler pool [puppet] - 10https://gerrit.wikimedia.org/r/477057 (https://phabricator.wikimedia.org/T210904) (owner: 10Andrew Bogott) [22:35:37] MatmaRex: True, but by giving them the warning, it changes things legally I'm told [22:36:24] Sorry if I sound a bit angry [22:36:30] I'm fed up [22:37:06] Presumably WMF secuirty can't tlak very much about what they are doing due to the onging nature of any investigations [22:37:10] *security [22:38:54] ShakespeareFan00 fed up? Did you get the notifications? [22:40:09] At least a dozen the other day and more this evening [22:40:39] My views on what should happen to those trying to compromise accounts aren't printable [22:47:49] ShakespeareFan00, you think whoever is doing this is using Special:UserLogin? [22:51:20] Oh... [22:51:20] ShakespeareFan00 they could be using BT / any majour ISP to launch this attack. Though i highly dought they are using Special:UserLogin. [22:51:46] If its BT their abuse department is usally fairly robust [22:51:51] :) [22:52:04] Presumably someone is reporting all the IP's? [22:52:35] Krenair: The Special:userLogin redirect is so that users on that network/IP range get a message [22:52:50] I am well aware the brute forcers may be trying API methods... [22:53:33] I think you will want 2fa if your that concerned ShakespeareFan00 [22:53:48] (per MatmaRex) [22:54:23] I'm less concerned about my account, more about seeing those responsible get caught :) [22:54:50] Even if a few good faith users might have to be inconvenienced to do it ;) [23:06:09] I doin't think good faith users will agree with you. [23:06:11] oh he left [23:11:05] 10Operations, 10ops-eqiad, 10DC-Ops: Update label and switch to rename labvirt1014 to cloudvirt1014 - https://phabricator.wikimedia.org/T210927 (10Reedy) [23:44:24] (03PS2) 10Dduvall: deployment-prep: Try changing redis_lock entries to memc hosts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/475025 (https://phabricator.wikimedia.org/T210030) (owner: 10Alex Monk) [23:47:17] (03CR) 10Dduvall: [C: 032] deployment-prep: Try changing redis_lock entries to memc hosts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/475025 (https://phabricator.wikimedia.org/T210030) (owner: 10Alex Monk) [23:49:01] (03Merged) 10jenkins-bot: deployment-prep: Try changing redis_lock entries to memc hosts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/475025 (https://phabricator.wikimedia.org/T210030) (owner: 10Alex Monk) [23:58:06] (03CR) 10jenkins-bot: deployment-prep: Try changing redis_lock entries to memc hosts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/475025 (https://phabricator.wikimedia.org/T210030) (owner: 10Alex Monk)