[02:18:30] <zhuyifei1999_>	 !log toolsbeta manually created flannel etcd key T190893
[02:18:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[02:18:32] <stashbot>	 T190893: Setup the webservice-related instances in toolsbeta - https://phabricator.wikimedia.org/T190893
[02:29:58] <legoktm>	 @seen valhallasw
[02:29:58] <wm-bot>	 legoktm: Last time I saw valhallasw they were quitting the network with reason: Quit: leaving N/A at 4/11/2016 3:18:42 PM (756d11h11m15s ago)
[02:30:03] <legoktm>	 o.O
[02:48:54] <zhuyifei1999_>	 @seen valhallasw|cloud
[02:48:54] <wm-bot>	 zhuyifei1999_: I have never seen valhallasw|cloud
[02:49:52] <zhuyifei1999_>	 @seenrx valhallasw
[02:49:52] <wm-bot>	 zhuyifei1999_: Last time I saw valhallasw`cloud they were quitting the network with reason: Quit: Connection closed for inactivity at 3/11/2018 5:59:39 PM (57d8h50m12s ago) (multiple results were found: valhallasw, valhallasw`TLV, valhallasw`vecto, valhallasw`vect4, valhallasw[m])
[02:50:03] <zhuyifei1999_>	 legoktm: ^
[02:50:25] <legoktm>	 ah
[02:50:34] <legoktm>	 ty
[02:51:14] <zhuyifei1999_>	 np
[10:54:54] <Danny_B|backup>	 hi, any server admin around, please?
[13:30:02] <andrewbogott>	 Danny_B|backup: Sorry, a bunch of people are traveling right now.  I don't see why your VM stopped… it looks like something dramatic happened on that host last night but not dramatic enough to kill anything.
[13:32:54] <doctaxon>	 andrewbogott: Hi! Any logins of user TaxonBot (dewiki) return login {result Failed reason {You have made too many recent login attempts. Please wait 2 days before trying again.}}
[13:33:38] <doctaxon>	 Bot username TaxonBot@TaxonBot
[13:39:25] <andrewbogott>	 doctaxon: login to what?
[13:44:00] <doctaxon>	 andrewbogott fragt mich auf #wikimedia-cloud: "login to what?" wie meint er das wohl?
[13:44:53] <doctaxon>	 andrewbogott: login to dewiki
[13:45:02] <andrewbogott>	 ah, I see
[13:45:42] <andrewbogott>	 There was a bot attack a few days ago which attempted a million fraudulent logins into a million different accounts.  It's possible that that cause the bot to be blocked.
[13:46:04] <andrewbogott>	 But more likely is that the bot is making some login mistake (e.g. wrong password) and trying over and over resulting in a throttle.
[13:46:05] <doctaxon>	 can you unblock please
[13:46:15] <andrewbogott>	 I don't have any particular knowledge about dewiki though.
[13:46:57] <chicocvenancio>	 I don't think there is a per-user login throtle
[13:47:07] <doctaxon>	 andrewbogott: but it seems that there is nobody who can proceed
[13:47:12] <chicocvenancio>	 doctaxon: from where are those logins being tried?
[13:47:37] <andrewbogott>	 chicocvenancio: doesn't throttling happen automatically after x failed logins?
[13:47:46] <andrewbogott>	 I feel like that's happened to me when I forget my password :)
[13:48:04] <doctaxon>	 the password was and is okay
[13:48:09] <paladox>	 i think you may be hitting the mitigation happening in prod
[13:48:23] <doctaxon>	 chicocvenancio: from the instance dwl:taxonbot
[13:48:24] <paladox>	 to limit a hackers attempt at gaining access to users account.
[13:48:27] <chicocvenancio>	 andrewbogott: I'm pretty sure it is done by IP, not per-user
[13:48:31] <andrewbogott>	 ah, ok
[13:48:50] <paladox>	 https://phabricator.wikimedia.org/T193769
[13:48:55] <paladox>	 chicocvenancio andrewbogott  doctaxon ^^
[13:49:36] <chicocvenancio>	 I'm aware
[13:50:09] <paladox>	 https://grafana.wikimedia.org/dashboard/db/authentication-metrics?orgId=1&from=1525275869702&to=now&theme=dark&panelId=13&fullscreen&var-entrypoint=*
[13:50:23] <doctaxon>	 okay, but it is possible to set up the login retry time to zero
[13:51:05] <chicocvenancio>	 oooh, it was back yesterday
[13:51:07] <TheresNoTime>	 doctaxon: when did you last try logging in? I can see a failed at `2018-05-08T12:38:04` but that wasn't a throttle reason
[13:51:07] <doctaxon>	 but this has to do by an admin, I guess
[13:51:40] <andrewbogott>	 doctaxon: yeah, as far as I know that's an on-wiki change that would be made by a de admin
[13:52:28] <doctaxon>	 that isn't a throttle reason because it's two days blocked
[13:54:57] <doctaxon>	 can you assign me a tech admin who's on now?
[13:56:08] <jynus>	 doctaxon: to get formal help, you need to create a ticket on phabricator
[13:56:38] <jynus>	 for what I can see, the block is actually legitimate, although probably was a mistake
[13:57:36] <chicocvenancio>	 https://www.mediawiki.org/wiki/Manual:$wgPasswordAttemptThrottle indicates it is per ip and per user
[13:57:43] <doctaxon>	 yes, I DO understand, but you can drop down it to zero minutes, so you have done it some time ago too
[13:58:14] <chicocvenancio>	 I'm unsure if changing the botpassword account will be used as well
[14:00:11] <chicocvenancio>	 doctaxon: this channel is for cloud services things, this does not fall into that territory. And as already mentioned, without a Phab task it is unlikely you will find help
[14:03:49] <doctaxon>	 chicocvenancio: how should I tag the phab ticket?
[14:04:26] <jynus>	 #security #site-requests could work
[14:12:02] <doctaxon>	 okay: T194160
[14:12:03] <stashbot>	 T194160: Unlock the login of bot user TaxonBot@TaxonBot to dewiki - https://phabricator.wikimedia.org/T194160
[14:19:08] <chicocvenancio>	 doctaxon: did you manually add all those subscribers?
[14:19:19] <doctaxon>	 no
[14:19:32] <doctaxon>	 the subtask function has it done
[14:19:54] <chicocvenancio>	 ahh, makes sense
[14:20:05] <chicocvenancio>	 as I noted there, it is probably unrelated though
[14:20:27] <doctaxon>	 i have read
[14:21:37] <doctaxon>	 chicocvenancio: I cannot login with new bottpassword, too, the user login is blocked
[14:22:02] <doctaxon>	 changing botpassword does not help
[14:22:13] <doctaxon>	 so unbreak now  is correct
[14:22:27] <chicocvenancio>	 it is not, because you cannot work on that
[14:22:55] <Reedy>	 Unbreak now is the wikis is down. Not just 1 bot can't edit
[14:23:50] <chicocvenancio>	 ^that too, but my point is you should not set the priority on task you're creating (unless you will work on them yourself)
[14:23:50] <Reedy>	 doctaxon: Why is your bot logging in every minute?
[14:25:18] <doctaxon>	 Reedy: to many logins due to login to many tasks because login has been blocked
[14:26:21] <TheresNoTime>	 But it's logging in successfully  `authentication for 'TaxonBot' succeeded` on `test2.wikipedia.org`
[14:26:59] <Reedy>	 Seems to be logging in fine to many wikis
[14:27:00] <Reedy>	 2018-05-08 14:20:05 [WvGyFApAMDwAADo@T90AAABK] mw1225 dewikivoyage 1.32.0-wmf.2 authentication INFO: Login for TaxonBot succeeded {"user":"TaxonBot"} 
[14:27:00] <Reedy>	 2018-05-08 14:20:05 [WvGyFApAEDMAAA7g1EkAAAAN] mw1286 test2wiki 1.32.0-wmf.2 authentication INFO: Login for TaxonBot succeeded {"user":"TaxonBot"} 
[14:27:00] <Reedy>	 2018-05-08 14:20:06 [WvGyFApAEMYAAK@hfuUAAAAY] mw1317 dewiki 1.32.0-wmf.2 authentication INFO: Login for TaxonBot succeeded {"user":"TaxonBot"} 
[14:27:00] <Reedy>	 2018-05-08 14:21:04 [WvGyTwpAEMMAALnUXMUAAABB] mw1314 test2wiki 1.32.0-wmf.2 authentication INFO: Login for TaxonBot succeeded {"user":"TaxonBot"}
[14:27:20] <jynus>	 Reedy: that is not the issue
[14:27:40] <jynus>	 the bot did a burst of logins yesterday >50
[14:27:48] <jynus>	 all with bad password
[14:28:05] <Reedy>	 lol
[14:28:09] <jynus>	 the block is legitimate
[14:28:20] <jynus>	 although doesn't seem to me in bad faith
[14:28:30] <Reedy>	 Probably not, no
[14:29:05] <doctaxon>	 jynus, the password is correct, if the logins have been mine 
[14:29:10] <jynus>	 this is the summary https://logstash.wikimedia.org/goto/dd24dcbf5ce722b12b23276c91bed73e
[14:30:13] <doctaxon>	 Reedy: these logins are of user TaxonBot, not of bot user TaxonBot@TaxonBot
[14:31:20] <Reedy>	 But jynus is right doctaxon, there were failed logins yesterday
[14:31:47] <jynus>	 if he sais it is impossible he send a bad password
[14:31:57] <jynus>	 so the block actually protected his account
[14:32:08] <Reedy>	 jynus: look at the client ip on at least one of the requests
[14:32:09] <jynus>	 so not sure unblocking is wise
[14:32:14] <Reedy>	 150.23.68.10.in-addr.arpa	name = taxonbot.dwl.eqiad.wmflabs.
[14:32:23] <Reedy>	 It's from his own cloud vm
[14:32:30] <jynus>	 sure, his tools account may be compromised
[14:32:36] <jynus>	 may need more blocks!
[14:33:02] <Reedy>	 Just spot checked a handful, they're all from the same VM
[14:33:32] <jynus>	 doctaxon: are you sure there is not way you may have accidentaly try to log in with the wrong password?
[14:33:40] <doctaxon>	 how can the password get a bad one automatically
[14:34:08] <jynus>	 programming error, mistake, etc.?
[14:34:38] <doctaxon>	 I changed nothing on those programs 
[14:35:44] <doctaxon>	 jynus: can you unblock the bot account and I will try the login with mine bot password?
[14:37:37] <Reedy>	 Do you have your own logs of what the bot is doing?
[14:39:06] <doctaxon>	 Yes, I promise to do it carefully
[14:39:30] <Reedy>	 I mean trying to work out what went wrong yesterday?
[14:41:25] <doctaxon>	 jynus: how to get authorized for https://logstash.wikimedia.org/goto/dd24dcbf5ce722b12b23276c91bed73e
[14:41:51] <jynus>	 @doctaxon You can ignore the link, that is for those that can unblock you only.
[14:42:09] <doctaxon>	 who are them they can unblock me only
[14:42:38] <jynus>	 people with deployment or root privileges
[14:43:11] <doctaxon>	 jynus: chasemp ?
[14:43:37] <jynus>	 however, without a good reason to do so, it is complicated- justifying it was a mistake etc would help
[14:43:48] <jynus>	 otherwise, it could put at risk your account
[14:44:00] <doctaxon>	 I understand
[14:44:02] <annika>	 damn
[14:44:16] <jynus>	 e.g. if you have logs of why login failed
[14:44:52] <jynus>	 if you don't, maybe it wasn't you, so it is actually beneficial for you it is blocked
[14:45:49] <jynus>	 I have commented on the ticket with the link that will provide more information of what happened from the servers point of view
[14:46:15] <jynus>	 if you could provide more information from the client (where you execute teh bot) that would help
[14:47:51] <jynus>	 so, to be clear, this is not a punishment, the software detected a strange pattern and reacted to it- it cannot just be blindly unlocket, we need to understand what happend and correct if there is an ongoing issue
[15:48:53] <Neha16>	 zhuyifei1999_: Hi! I am getting familiar with toolsbeta. So I ssh into toolsbeta, migrate to /usr/lib/python2.7/dist-packages/toollabs and edit the code there using sudo. Shouldn't there be other folders of the repository such as scripts, debian etc?
[15:52:57] <zhuyifei1999_>	 Neha16: debian is for the packaging
[15:53:09] <zhuyifei1999_>	 it's related to how it is installed
[15:54:04] <zhuyifei1999_>	 for files inside scripts they are installed in one of the bin directories, most likely /usr/bin
[15:54:26] <zhuyifei1999_>	 just run `which` on it. eg `$ which webservice`
[15:56:57] <Neha16>	 zhuyifei1999_: Got it. Thank you so much
[15:58:31] <zhuyifei1999_>	 np
[15:58:57] <zhuyifei1999_>	 oh btw, as long as you are running the webservice under grid, it should be working
[15:59:25] <zhuyifei1999_>	 you can also test webservice shell for k8s, but the networking inside the k8s pod is broken
[15:59:33] <zhuyifei1999_>	 and I can't figure out why
[16:01:11] <zhuyifei1999_>	 but we'll get it working, hopefully soon
[16:01:18] <zhuyifei1999_>	 Neha16: ^
[16:03:03] <Neha16>	 zhuyifei1999_: That's ok. I still need to get familiar with a whole lot of things :)
[16:03:14] <zhuyifei1999_>	 k
[19:59:11] <legoktm>	 andrewbogott: should the openstack/horizon/wmf-sudo-dashboard repository notifications go to -cloud-feed or -operations? Right now it's ending up in -dev which seems wrong
[19:59:53] <andrewbogott>	 probably to -cloud-feed
[20:00:02] <andrewbogott>	 legoktm: what is it saying, btw?
[20:00:23] <legoktm>	 just the wikibugs_ gerrit notifications
[20:20:42] <ragesoss>	 woot. I moved the mysql data directory for outreachdashboard.wmflabs from /var/lib/mysql over to /srv and it worked!
[23:09:24] <hauskatze>	 Fatal error: Invalid serialization data for DateTime object in /mnt/nfs/labstore-secondary-tools-project/meta/git/wikimedia-contrib/tool-labs/backend/modules/Cacher.php on line 153
[23:14:53] <Niharika>	 bd808: Can you poke at https://tools.wmflabs.org/hatjitsu/? :(
[23:29:01] <bd808>	 Niharika: I don’t have a laptop around. I’m traveling. 
[23:29:19] <Niharika>	 bd808: No worries. We found an equivalent. :) 
[23:31:11] <bd808>	 bstorm_: can you check the hatjitsu tool? It’s a nodejs Kubernetes webservice
[23:31:49] <bd808>	 Niharika: remind me to add you as a maintainer. :)
[23:32:10] <Niharika>	 bd808: I shall do that!
[23:32:22] <bstorm_>	 I can look
[23:39:41] <bstorm_>	 It's definitely borked.  It's crashing.
[23:39:49] <legoktm>	 !log wikitextexp set $wgULSLanguageDetection = false; so ULS won't change wiki UI language based on Accept-Language headers
[23:39:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikitextexp/SAL
[23:46:28] <bstorm_>	 bd808: That's what's wrong with hatjitsu https://www.irccloud.com/pastebin/5BeqZSbF/
[23:46:41] <bstorm_>	 Not entirely sure what's up with that...
[23:47:20] <legoktm>	 looks related to zhuyifei1999_ 's changes to webservice?
[23:47:21] <bstorm_>	 No that's something else...that's what I got for logs lol
[23:47:26] <bstorm_>	 Yeah
[23:47:56] <bstorm_>	 I dunno
[23:48:52] <bstorm_>	 That may be the changes to webservice that I'm seeing...but I think that's preventing me from seeing what's wrong with hatjitsu :)  Since that's a node service, not python
[23:52:02] <zhuyifei1999_>	 bstorm_: just restart the the webservice
[23:52:08] <bstorm_>	 ok :)
[23:52:29] <zhuyifei1999_>	 the new script should mount that file
[23:53:09] <bstorm_>	 Ahhh, ok.  
[23:53:33] <bstorm_>	 Hasn't crashed yet...
[23:53:41] <bstorm_>	 So that's good
[23:53:45] <bstorm_>	 Watching it
[23:53:50] <zhuyifei1999_>	 actually, where is that logs from? (just to make sure it's from inside the container)
[23:54:14] <zhuyifei1999_>	 oh wait nevermind webservice-runner is the webservice bootstraper
[23:54:19] <bstorm_>	 :)
[23:54:20] <bstorm_>	 Yeah
[23:54:45] <bstorm_>	 That was the result of using `kubectl logs`
[23:54:53] <bstorm_>	 Looking good so far
[23:55:17] <zhuyifei1999_>	 k
[23:55:47] <bstorm_>	 404 is much better than 502 as far as http error codes go.  I have no idea how to use this tool, so I think that's an improvement?
[23:56:14] <zhuyifei1999_>	 Niharika: ^
[23:57:08] <Niharika>	 Oh well.
[23:57:20] <Niharika>	 It was working perfectly until last week.
[23:57:32] <bstorm_>	 Huh.
[23:57:39] <Niharika>	 I guess we'll have to wait for Brya.n to fix it.
[23:58:39] <bstorm_>	 All it's logs are from 2 years ago...and the access logs only show 404s
[23:58:40] <bstorm_>	 weird