[12:57:56] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: L10n messages missing: prefs-srf and srf-prefs-intro-text - https://phabricator.wikimedia.org/T130922#2150859 (10Dereckson)
[12:58:33] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: L10n messages missing: prefs-srf and srf-prefs-intro-text - https://phabricator.wikimedia.org/T130922#2150872 (10Dereckson)
[12:58:35] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: SRF preference messages broken - https://phabricator.wikimedia.org/T128027#2150874 (10Dereckson)
[13:11:57] <tom29739>	 How do I disable 2FA when I haven't got the phone app or anything set up? I accidentally pressed the enable button, and then pressed back in my browser, but it seems to be enabled anyway, and I haven't got a phone code to disable it.
[13:46:28] <Luke081515>	 tom29739: at wikitech or at phabricator?
[13:46:39] <Luke081515>	 At wikitech: Do you saved the emergency tokens?
[13:46:48] <Luke081515>	 *did you
[13:46:56] <tom29739>	 No.
[13:47:02] * tom29739 facepalms
[13:47:12] <tom29739>	 At wikitech.
[13:47:29] <Luke081515>	 Then you need to contact a member with database access there to disable it there
[13:47:35] <tom29739>	 Great.
[13:47:42] <Luke081515>	 I think the better solution is, if you create a phabricator task
[13:48:13] <Luke081515>	 reseting the setting at the database is the only way, IIRC, if you don't got the emergency tokens
[13:48:13] <tom29739>	 Because once I log out of wikitech, I can't access it without a code.
[13:49:01] <tom29739>	 the phab task, does it need to be a security one or anything?
[13:50:48] <Luke081515>	 tom29739: You're still logged in?
[13:50:54] <tom29739>	 Yeah.
[13:51:36] <Luke081515>	 Normally I guess the 2FA is only enabled if you entered a token? What does Special:Preferences say?
[13:51:44] <tom29739>	 If I get logged out because I close my browser, cookie runs out, etc. I can't log back in without a code.
[13:51:52] <tom29739>	 That's what I though.
[13:51:57] <tom29739>	 *thought.
[13:52:17] <Luke081515>	 Special:Preferences show "enable 2FA" or "reset 2fa / disable 2FA"?
[13:52:20] <Luke081515>	 *shows
[13:52:24] <tom29739>	 I seem to remember seeing a phab task about that. It's a bug.
[13:52:35] <Luke081515>	 yeah, I saw that too
[13:52:47] <Luke081515>	 no, your task don't need a security setting
[13:53:00] <Luke081515>	 just create one with:
[13:53:15] <Luke081515>	 "Reset 2FA for <user> at wikitech" with the wikitech project
[13:53:48] <tom29739>	 It says 'disable two-factor authentication'.
[13:54:13] <tom29739>	 When I click on it it says 'Enter a code from you mobile app to verify'.
[13:54:20] <tom29739>	 *your
[13:54:22] <Luke081515>	 try if that works without code too :D
[13:54:39] <Luke081515>	 so it's currently enabled, if that ^ don't works, you need to ask someone
[13:54:59] <tom29739>	 It doesn't
[13:55:22] <Luke081515>	 some bugs are only present at the wrong positions :-/
[13:55:22] <tom29739>	 It says 'Failed to validate two-factor credentials'
[13:55:27] <Luke081515>	 so you need to create a task
[13:57:32] <Luke081515>	 halfak: Do you got access to wikitech-DB?
[13:57:58] <halfak>	 o/ Luke081515 
[13:58:00] <halfak>	 As in the MediaWiki DB for the wikitech wiki?
[13:58:04] <halfak>	 Or labsDB?
[13:58:11] <Luke081515>	 MediaWiki DB
[13:58:18] <halfak>	 Hmm.. Let me find out. 
[13:58:25] <Luke081515>	 someone (not me) needs to reset 2FA
[13:59:15] <Luke081515>	 2FA was accidentally created, and no possibility to reset it (T130892)
[13:59:15] <stashbot>	 T130892: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892
[13:59:38] <halfak>	 Luke081515, oh!!  At best I could query it, but not modify it :/
[13:59:46] <Luke081515>	 :-/
[14:00:09] <Luke081515>	 tom29739: Try to keep logged in till the next admin arrives :P
[14:00:36] <halfak>	 FYI, looks like labswiki is not sync'd to the analytics database server. 
[14:00:51] <halfak>	 andrewbogott, ^ 
[14:00:58] <halfak>	 In case you're around already
[14:01:07] <tom29739>	 I've been logged in for the past 20 days or so, so I'll try to stay logged in.
[14:01:16] <halfak>	 ^ my plan too :) 
[14:01:17] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Reset 2FA for User:Tom29739 at wikitech - https://phabricator.wikimedia.org/T130926#2150962 (10tom29739)
[14:01:45] <halfak>	 As soon as a Labs root shows up, I'll be happy to run tests :) 
[14:01:50] <andrewbogott>	 halfak: I don't know what that means — why would it be synced?
[14:02:09] <halfak>	 andrewbogott, sorry meant to point higher.  Looks like there's a 2FA issue for logging into labs
[14:02:21] <Luke081515>	 andrewbogott: Do you got write access to mediawiki wikitech DB? Then please take a look at T130926
[14:02:22] <stashbot>	 T130926: Reset 2FA for User:Tom29739 at wikitech - https://phabricator.wikimedia.org/T130926
[14:02:55] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Reset 2FA for User:Tom29739 at wikitech - https://phabricator.wikimedia.org/T130926#2150977 (10Luke081515)
[14:03:54] <andrewbogott>	 tom29739: do you have ssh configured for tools and/or other labs projects?
[14:04:07] <tom29739>	 Yep, for tools.
[14:04:22] <tom29739>	 I'm logged in to tools-bastion-05 now.
[14:04:24] <Luke081515>	 !log rcm.cac starting the vm and running git-update
[14:04:27] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm.cac/SAL, Master
[14:04:49] <andrewbogott>	 tom29739: ok, can you please touch a file called '2fa_reset.txt' in your home directory?
[14:05:19] <tom29739>	 done.
[14:05:32] <andrewbogott>	 (Just an identity check — thanks)
[14:05:46] <andrewbogott>	 ok, confirmed, let me figure out how to do this and I'll reset things :)
[14:05:55] <tom29739>	 Thanks.
[14:07:16] <andrewbogott>	 tom29739: ok, done, how does it look now?
[14:07:49] <tom29739>	 It's disabled now, thanks!
[14:08:29] <andrewbogott>	 sure thing, you were a good test case for our new process :)
[14:08:31] <Luke081515>	 tom29739: So I guess it's time to close the phab task now? :D
[14:08:38] <andrewbogott>	 I can do it
[14:08:40] <tom29739>	 Doing
[14:08:40] <Luke081515>	 ok
[14:08:47] <Luke081515>	 a good win:win situation :D
[14:09:03] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Reset 2FA for User:Tom29739 at wikitech - https://phabricator.wikimedia.org/T130926#2151001 (10tom29739) 5Open>3Resolved a:3tom29739 @andrew reset it for me.
[14:09:17] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Reset 2FA for User:Tom29739 at wikitech - https://phabricator.wikimedia.org/T130926#2151005 (10Andrew) Tom verified his ID by touching a file on the tools bastion.  All set.
[14:09:42] <andrewbogott>	 relocating, back in a few
[14:10:23] <Luke081515>	 !log rcm.cac enabling role monobook
[14:10:26] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm.cac/SAL, Master
[14:11:29] <halfak>	 \o/
[14:42:38] <wikibugs>	 6Labs, 13Patch-For-Review, 5WMF-deploy-2016-03-29_(1.27.0-wmf.19): Switch to using Horizon/Designate for labs public dns - https://phabricator.wikimedia.org/T124184#2151103 (10Andrew)
[14:45:53] <wikibugs>	 6Labs, 10Tool-Labs, 6Community-Tech-Tool-Labs, 10MediaWiki-Vagrant, and 2 others: Tools, Labs, & MediaWiki-Vagrant help desk (#wikimedia-hackathon-2016) - https://phabricator.wikimedia.org/T130780#2151112 (10bd808)
[14:58:37] <wikibugs>	 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892#2149727 (10Luke081515) Another user was affected: T130926
[14:59:43] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Labs-Other-Projects: Cannot login into cac.rcm.eqiad.wmflabs - https://phabricator.wikimedia.org/T130471#2151142 (10Luke081515) 5Open>3Resolved Instance deleted and recreated
[15:02:31] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org: decom old wikitech-static machine - https://phabricator.wikimedia.org/T129391#2151149 (10Dzahn) a:3Dzahn
[15:04:35] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org: decom old wikitech-static machine - https://phabricator.wikimedia.org/T129391#2151154 (10Dzahn) @Krenair you think we can go ahead here?
[15:43:20] <shinken-wm>	 PROBLEM - Puppet run on tools-bastion-11 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [0.0]
[15:48:19] <shinken-wm>	 RECOVERY - Puppet run on tools-bastion-11 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:14:18] <Cyberpower678>	 kaldari, Niharika: so I'm going to need a password to access the API
[16:14:38] <Niharika>	 Cyberpower678: I'm sorry! I forgot about that. 
[16:14:44] <Niharika>	 Cyberpower678: Give me a minute. 
[16:15:46] <wikibugs>	 6Labs, 10Labs-Sprint-109: Remove reliance on ldap $::projectid from shinkengen - https://phabricator.wikimedia.org/T108625#2151317 (10Luke081515) Seems like there is no patch here?
[16:18:32] <Niharika>	 Cyberpower678: Done. 
[16:24:20] <Cyberpower678>	 Niharika, thanks
[16:27:32] <shinken-wm>	 RECOVERY - Puppet staleness on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [3600.0]
[16:59:48] <marxarelli>	 o/
[17:00:46] <marxarelli>	 we're seeing resolution failure for integration-slave-trusty-1002.eqiad.wmflabs from labs dns. any labs opsen around to take a look?
[17:01:41] <yuvipanda>	 marxarelli: can you see that host on wikitech?
[17:02:02] <marxarelli>	 yuvipanda: hmm. let me check ...
[17:02:32] <yuvipanda>	 in manage instances
[17:03:18] <marxarelli>	 yuvipanda: yep, it's there
[17:03:42] <marxarelli>	 looks like it's just ns2
[17:03:54] <marxarelli>	 (`dig +short @labs-ns2.wikimedia.org. integration-slave-trusty-1002.eqiad.wmflabs` gives nothing)
[17:04:13] <yuvipanda>	 andrewbogott: ^ designate failure?
[17:04:17] <marxarelli>	 but `dig +short @labs-ns1.wikimedia.org. integration-slave-trusty-1002.eqiad.wmflabs` gives me 10.68.17.12
[17:05:16] <yuvipanda>	 oooh
[17:05:17] <yuvipanda>	 fascinating
[17:05:19] <marxarelli>	 yuvipanda: andrewbogott has no available IRQs atm :)
[17:05:22] <yuvipanda>	 ah
[17:05:23] <andrewbogott>	 yuvipanda: probably but I don't have time to look now unless it's happening over and over
[17:05:24] <yuvipanda>	 ok
[17:05:33] <yuvipanda>	 and I'm not sure what to do...
[17:06:23] <marxarelli>	 yuvipanda: it's not critical. i've depooled the instance until y'all have time to investigate
[17:06:32] <marxarelli>	 ty!
[17:06:32] <yuvipanda>	 ok!
[17:06:34] <yuvipanda>	 marxarelli: can you file a ug?
[17:06:37] <yuvipanda>	 *bug
[17:06:39] <marxarelli>	 yep!
[17:07:29] <marxarelli>	 yuvipanda: blerg. i should have searched before bugging you https://phabricator.wikimedia.org/T129640 :)
[17:07:47] <marxarelli>	 i'll add a 'me too'
[17:07:56] <yuvipanda>	 :D ok
[17:10:38] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Cannot SSH to a few CI slaves - https://phabricator.wikimedia.org/T129640#2111479 (10dduvall) We're still seeing this. It looks like a DNS replication issue perhaps. Resolution from NS2 gives no answer while NS1 responds just fine.  ``` ~...
[17:11:15] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Cannot SSH to a few CI slaves due to DNS failure - https://phabricator.wikimedia.org/T129640#2151437 (10dduvall)
[17:11:55] <yuvipanda>	 marxarelli: given our current crunches, do you think you can just delete and recreate taht instance without too much trouble?
[17:13:43] <marxarelli>	 yuvipanda: i think we have plenty of trusty capacity atm, so it's possible that we can just delete the instances. i'll check in with hashar about it
[17:14:36] <yuvipanda>	 ok!
[17:34:38] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: SRF preference messages broken - https://phabricator.wikimedia.org/T128027#2151556 (10Krenair) does it need converting to the new i18n json format?
[17:38:44] <wikibugs>	 6Labs, 10Tool-Labs: Setup a proper deployment strategy for Kubernetes - https://phabricator.wikimedia.org/T129311#2151577 (10yuvipanda) Here's what needs to happen when we 'deploy a new version of kubernetes':  On a deployment/build host:  1. Push a tag to our kubernetes gerrit repo with the version we want to...
[17:38:51] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 6Operations: Update phantomjs to 2.1.1 on trusty - https://phabricator.wikimedia.org/T130940#2151595 (10Paladox)
[17:40:23] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org: decom old wikitech-static machine - https://phabricator.wikimedia.org/T129391#2151635 (10Krenair) Yep
[17:41:19] <wikibugs>	 6Labs, 10Tool-Labs: Setup a proper deployment strategy for Kubernetes - https://phabricator.wikimedia.org/T129311#2151640 (10yuvipanda) Things #scap3 needs to have to make this possible:  1. Ability to setup a deployment server without bringing in all of mediawiki, salt, trebuchet, trebuchet-trigger, redis, et...
[17:41:27] <wikibugs>	 6Labs, 10Tool-Labs, 3Scap3: Setup a proper deployment strategy for Kubernetes - https://phabricator.wikimedia.org/T129311#2151641 (10yuvipanda)
[18:00:16] <mutante>	 please make wikibugs talk in the other channels again
[18:00:26] <mutante>	 (happens after every restart afaict)
[18:03:16] <Krenair>	 what's wrong with it?
[18:03:28] <Krenair>	 it should join and talk in other channels as soon as there's something to talk about, shouldn't it?
[18:06:04] <mutante>	 i edit tickets and it says nothing about it
[18:06:12] <mutante>	 it's happened many times before
[18:06:18] <mutante>	 where it talks in -labs but not in -production
[18:25:17] <Luke081515>	 mutante: Which channel makes problems? in -releng and -operations the bot is active
[18:26:00] <Luke081515>	 normaly the bots joins only a channel after a restart, if there is something to speak about. (Labs is the only excemption)
[18:26:57] <mutante>	 Luke081515: -operations
[18:27:37] <mutante>	 it's in the channel but doesnt talk
[18:27:44] <mutante>	 it stopped a little while ago
[18:28:00] <mutante>	 .. eh.. and it started again
[18:28:13] <mutante>	 sorry, was just intermittent then
[18:28:18] <Luke081515>	 np
[18:28:44] * Luke081515 thinks wikibugs trys to annoy mutante :P
[18:29:11] <tom29739>	 ^ probably :D
[18:30:44] <Luke081515>	 like if (username == 'mutante') { return null; } :P
[18:32:49] <Luke081515>	 mutante: but seriously, this could have multiple reason, for example the phab conduit could be the reason too
[18:34:20] <Luke081515>	 seems like wikibugs works now clean again
[18:34:56] <mutante>	 Luke081515: is redis involved?
[18:35:04] <Luke081515>	 I think yeah
[18:35:17] <mutante>	 i think sometimes redis is missing a change
[18:35:21] <mutante>	 but then it continues normal
[18:35:37] <mutante>	 that's how it felt.. but it's ok now, yea
[18:36:07] <Luke081515>	 mutante: Yeah, redis is involed, see https://gerrit.wikimedia.org/r/#/c/263931/
[18:36:15] <Luke081515>	 *involved
[18:36:41] <mutante>	 ah, yea
[18:40:56] <shinken-wm>	 PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[18:41:19] <yuvipanda>	 ^ was me, fixed now
[18:50:06] <gifti>	 i cannot log in to wikitech anymore; it requires 2fa now, but i haven't activated it. yuvipanda? andrewbogott?
[18:52:02] <yuvipanda>	 did you activate it for horizon?
[18:52:45] <gifti>	 what is horizon? no
[18:55:29] <yuvipanda>	 ok, I know andrewbogott and csteipp have been playing with it, so I'm just going to have to wait for them (or Krenair) to chime in.
[18:55:52] <yuvipanda>	 I know that andrewbogot.t is busy with other things atm for the weekend switchover thouh
[18:56:02] <gifti>	 ah, ok, yuvipanda, shall i create a task?
[18:56:05] <Krenair>	 gifti, what makes you think that wikitech login requires 2fa?
[18:56:08] <yuvipanda>	 yes gifti
[18:56:27] <gifti>	 Krenair: "Login error  The two-factor authentication token provided was invalid."
[18:56:35] <Krenair>	 did you try to provide a token?
[18:56:41] <gifti>	 and also, 2fa is activated in my settings
[18:56:43] <gifti>	 no
[18:56:58] <gifti>	 (in the one session that i luckily still have)
[18:57:09] <Luke081515>	 seems like this is the next consequence of T130892?
[18:57:09] <stashbot>	 T130892: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892
[18:57:15] <gifti>	 (but i tried to add a key via my new phone)
[18:57:16] <Krenair>	 if 2fa is activated in your settings than you need to provide the token
[18:57:43] <gifti>	 i don't have activated it in the first place and i don't have a token
[18:57:58] <Krenair>	 <gifti> and also, 2fa is activated in my settings
[18:58:14] <gifti>	 thank you for your mathematician answer ;)
[18:58:17] <Luke081515>	 Krenair, this can happen, see T130892
[18:58:17] <stashbot>	 T130892: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892
[18:58:22] <Krenair>	 ugh
[18:58:30] <Luke081515>	 We already had this once today
[18:58:40] <Luke081515>	 Krenair: Do you got DB access to silber? (write)
[18:58:42] <Krenair>	 gifti, did you start trying to enable 2fa and then not validate it?
[18:58:43] <Krenair>	 Luke081515, yes
[18:58:47] <gifti>	 no
[18:58:47] <Krenair>	 I have deployment access remember?
[18:59:09] <gifti>	 i'm unnerved, i will now go out of this conversation, bye
[18:59:10] <Cyberpower678>	 yuvipanda, How can I see how much space a table uses up?
[18:59:12] <Luke081515>	 andrew made this procedure today: User should create a file at tools bastion home, and that's the confirmation for 2FA removal
[18:59:23] <yuvipanda>	 Cyberpower678: not sure. it's just a mysql server, google around to find out?
[18:59:40] <Luke081515>	 Krenair: Yeah, I remeber, but I'm not sure if this gives access to all servers etc
[18:59:44] <gifti>	 Luke081515: where is that documented?
[18:59:50] <gifti>	 or announced?
[18:59:52] <Cyberpower678>	 I did.  It lead me to an SQL query, that I got an access denied error
[18:59:55] <Krenair>	 Luke081515, it doesn't give access to all servers, but it does include silver
[19:00:02] <Luke081515>	 ok
[19:00:16] <Luke081515>	 the procedure of today is here: https://phabricator.wikimedia.org/T130926#2151005
[19:00:19] <yuvipanda>	 Cyberpower678: ah. In that case I'd suggest opening a specific task with the problem you have and what you are trying to do.
[19:00:28] <yuvipanda>	 hopefully one of our DBAs can advice.
[19:00:42] <Cyberpower678>	 yuvipanda, I just want to know how much space Cyberbot's DB takes up
[19:00:51] <gifti>	 ah, this …
[19:01:46] <yuvipanda>	 Cyberpower678: I'm just going to repeat what I said earlier :) please file a task.
[19:11:19] <wikibugs>	 6Labs: 2fa broke wikitech login - https://phabricator.wikimedia.org/T130953#2151905 (10Giftpflanze)
[19:12:03] <shinken-wm>	 PROBLEM - Host tools-docker-builder-01 is DOWN: CRITICAL - Host Unreachable (10.68.16.104)
[19:13:25] <wikibugs>	 6Labs: 2fa broke wikitech login - https://phabricator.wikimedia.org/T130953#2151917 (10Luke081515) Related: T130892
[19:29:43] <shinken-wm>	 RECOVERY - Host tools-docker-builder-01 is UP: PING OK - Packet loss = 0%, RTA = 0.99 ms
[19:35:57] <wikibugs>	 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892#2151965 (10Andrew) @csteipp I don't have backups of that table as far as I know.  This is a bug in the new OATH code, isn't it?  Can it be...
[19:36:29] <shinken-wm>	 PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[19:41:26] <wikibugs>	 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892#2151969 (10csteipp) @Andrew, we could disable it for everyone who has is_valid=0 in the DB. That will disable 2FA for any users who enabled...
[19:50:50] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-docker-builder-01 is OK: OK: All targets OK
[20:13:33] <wikibugs>	 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892#2152044 (10csteipp) More I think about it, probably just best to remove it for those accounts. I'm on vacation today, but if someone wants...
[20:23:20] <Krenair>	 !log deployment-prep fiddled around with puppet on deployment-cache-text04 earlier to fix certs etc.
[20:23:21] <stashbot>	 Please !log in #wikimedia-releng for beta cluster SAL
[20:23:24] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master
[20:23:44] <Krenair>	 !log deployment-prep repaired centralauth.spoofuser table on deployment-db1
[20:23:45] <stashbot>	 Please !log in #wikimedia-releng for beta cluster SAL
[20:23:47] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master
[20:23:53] <Krenair>	 !log deployment-prep started redis-server on deployment-redis01
[20:23:54] <stashbot>	 Please !log in #wikimedia-releng for beta cluster SAL
[20:23:56] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master
[20:26:26] <wikibugs>	 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892#2152064 (10csteipp) If we do that, someone should grab a copy of the table, and when jcrespo is back from Easter, and hopefully has a backu...
[20:29:11] <Krenair>	 shoo, stashbot 
[20:43:07] <wikibugs>	 6Labs, 13Patch-For-Review, 5WMF-deploy-2016-03-29_(1.27.0-wmf.19): Switch to using Horizon/Designate for labs public dns - https://phabricator.wikimedia.org/T124184#2152133 (10Andrew)
[20:44:30] <wikibugs>	 6Labs: Migrate DNS entries from ldap to designate - https://phabricator.wikimedia.org/T129272#2152148 (10Andrew) All done now except for:  probable domain conflict:  beta in wmflabs.org for 208.80.155.135 in wmflabsdotorg probable domain conflict:  tools in wmflabs.org for 208.80.155.131 in wmflabsdotorg  ...des...
[20:51:44] <wikibugs>	 6Labs, 10Phlogiston: phlogiston-2 hangs every week - https://phabricator.wikimedia.org/T129891#2152150 (10JAufrecht) Possibly related: was in a very weird state today, with load at 66 growing to 68.  PS -auxf led to a lot of these:  ``` root     12928  0.0  0.2 108296 35692 ?        D    15:52   0:00 /usr/bin/...
[20:54:57] <shinken-wm>	 RECOVERY - Puppet run on tools-docker-builder-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:00:06] <wikibugs>	 6Labs: Migrate DNS entries from ldap to designate - https://phabricator.wikimedia.org/T129272#2152162 (10Andrew) I hacked those entries into the designate record table, but they still don't resolve :(
[21:39:40] <shinken-wm>	 PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0]
[21:53:24] <wikibugs>	 6Labs, 15User-bd808: Migrate projects using ::role::deprecated::mediawiki::install to ::role::labs::mediawiki_vagrant - https://phabricator.wikimedia.org/T121476#2152263 (10bd808)
[21:56:53] <wikibugs>	 6Labs, 15User-bd808: Migrate projects using ::role::deprecated::mediawiki::install to ::role::labs::mediawiki_vagrant - https://phabricator.wikimedia.org/T121476#2152281 (10bd808)
[22:04:43] <wikibugs>	 6Labs, 15User-bd808: Migrate projects using ::role::deprecated::labsvagrant to ::role::labs::mediawiki_vagrant - https://phabricator.wikimedia.org/T121477#2152302 (10bd808)
[22:10:36] <Cyberpower678>	 kaldari, ping
[22:10:42] <Cyberpower678>	 Niharika, ping
[22:11:26] <kaldari>	 hello
[22:12:56] <Cyberpower678>	 kaldari, So what's the status of the checkIfDead class.  When I inadvertantly activated it, it immediately got stuck and stalled the bot.  It's going to need some timeout feature to prevent it from hanging.
[22:13:40] <kaldari>	 Cyberpower678: Any suggestion how long to set the timeout?
[22:13:51] <Cyberpower678>	 kaldari, I'm thinking 3 seconds max.
[22:14:10] <Cyberpower678>	 Text should transfer quick if the site was working.
[22:14:16] <kaldari>	 ok, I'll get that added
[22:15:14] <Cyberpower678>	 kaldari, Would you also object to me modifying the IA multiquery function for use on the checkIfDead class.  Calling all the sites to be checked at once would make things go faster.
[22:15:46] <kaldari>	 that's a good idea
[22:17:03] <Cyberpower678>	 Oh and, we need to be able to detect when the bot gets blacklisted from sites.  It may think the site is dead when in fact the bot simply got banished.
[22:18:09] <kaldari>	 Cyberpower678: If it fails the 3 second timeout, should it return it as dead? Seems a bit dangerous to assume that.
[22:18:19] <gifti>	 Cyberpower678: can you let me know, when you find a way for that?
[22:18:55] <Cyberpower678>	 If the bot gets banned isn't there usually an error returned in the header or the code.
[22:19:32] <kaldari>	 I imagine it would return an error code, but not sure which.
[22:19:51] <Cyberpower678>	 kaldari, the bot runs a link 3 times in seperate instances before flagging a site as dead.  The status is cached in the DB.
[22:19:53] <kaldari>	 Disallowed? Don't remember what the number for that is though
[22:20:54] <Cyberpower678>	 kaldari, so if a site fails a check 3 times, it's considered dead.
[22:21:05] <kaldari>	 OK, so the function in checkifdead should just return false if it fails the 3 second limit then, correct?
[22:21:13] <Cyberpower678>	 Yep
[22:21:17] <kaldari>	 got it
[22:21:26] <wikibugs>	 6Labs, 10Horizon, 13Patch-For-Review: Horizon dashboard for managing http proxies for labs instances - https://phabricator.wikimedia.org/T129245#2152331 (10Andrew) this is now merged and running on horizon.wikimedia.org and looks pretty good!
[22:21:39] <wikibugs>	 6Labs: Migrate DNS entries from ldap to designate - https://phabricator.wikimedia.org/T129272#2152332 (10Andrew) It turns out to be possible to create these, I just had to specify the fqdn:  openstack recordset create eae60a3b-a0df-47b2-9492-5fab480514fe tools.wmflabs.org. --type A --records "208.80.155.131"
[22:22:18] <Cyberpower678>	 And if we can figure out how to detect a bot block, the returned value for that should be null, as Cyberbot will no longer be able to accurately tell what the status is, and will set the status to 5.
[22:23:56] <Cyberpower678>	 Cybebot's Live scale is: 5->bot cannot tell, and will no longer query the site; 4->Default indicating status unknown; 3->Alive; 2->Failed one check; 1->Failed two checks; 0->Dead
[22:25:45] <wikibugs>	 6Labs: Migrate DNS entries from ldap to designate - https://phabricator.wikimedia.org/T129272#2152333 (10Andrew) @valhallasw is going to sort out the spf bits; everything else is migrated.
[22:26:16] <wikibugs>	 6Labs, 13Patch-For-Review, 5WMF-deploy-2016-03-29_(1.27.0-wmf.19): Switch to using Horizon/Designate for labs public dns - https://phabricator.wikimedia.org/T124184#2152334 (10Andrew)
[22:37:07] <wikibugs>	 6Labs, 10Tool-Labs, 13Patch-For-Review, 3Scap3: Setup a proper deployment strategy for Kubernetes - https://phabricator.wikimedia.org/T129311#2152357 (10yuvipanda) Intermediate alternative if we can't actually get scap to do this in the meantime:  1. The k8s build script extracts the tar into a well known...
[22:39:40] <Cyberpower678>	 kaldari, logging mechanism installed
[22:39:52] <kaldari>	 awesome
[22:43:17] <wikibugs>	 6Labs, 13Patch-For-Review, 5WMF-deploy-2016-03-29_(1.27.0-wmf.19): Switch to using Horizon/Designate for labs public dns - https://phabricator.wikimedia.org/T124184#2152370 (10Andrew)
[22:56:29] <Cyberpower678>	 kaldari, you have a bug in your logging API.
[22:57:13] <kaldari>	 oh?
[22:57:25] <Cyberpower678>	 Click the dropdown for the bots. :p
[22:58:47] <Cyberpower678>	 kaldari, ^ do you see it?
[22:59:43] <kaldari>	 hmm, I'll fix it
[23:06:37] <Cyberpower678>	 kaldari, btw, I turned the bot loose
[23:07:15] <Cyberpower678>	 It's no longer restricted to the pages with tags on them
[23:07:31] <Cyberpower678>	 Seeing as it was completing a run in just 9 hours.
[23:16:37] <Cyberpower678>	 kaldari, I'd say we are at a point where we could discuss making an interface for wikipedians to use.
[23:19:14] <Cyberpower678>	 kaldari, Since the code is split apart, the engine for actually analyzing the page is already there, so we are essentially making a GUI.
[23:19:30] <Cyberpower678>	 kaldari, And I feel like that's your area of expertise.
[23:20:59] <kaldari>	 sure, we could either build an interface on Tool Labs or a gadget to create some kind of interface for the article you are currently on, or both.
[23:21:37] <Cyberpower678>	 The interface would be under the cyberbot tool, so it can directly access everything it needs to.
[23:22:17] <Cyberpower678>	 I have so many ideas for the interface, that I think you will love, and cannot possibly be made into a gadget. :D
[23:24:00] <SPF|Cloud>	 guys, it seems that I've deleted my wikitech entry in the Google Authenticator app without disabling 2FA on Wikitech
[23:24:27] <Cyberpower678>	 SPF|Cloud, do you still have your secret one time keys?
[23:24:38] <SPF|Cloud>	 I don't think so..
[23:24:50] <Cyberpower678>	 Then you're screwed. :-(
[23:25:01] <SPF|Cloud>	 no, I'm not
[23:25:25] <Cyberpower678>	 Those secret keys are the only way to recover you account.
[23:25:29] <bd808>	 SPF|Cloud: we just came up with a procedure for this. Let me find the wiki page
[23:25:36] <SPF|Cloud>	 https://wikitech.wikimedia.org/wiki/Help:Horizon_FAQ#What_happens_if_I_lose_my_phone_and_my_backup_codes.3F
[23:25:45] <Cyberpower678>	 SPF|Cloud, oh cool.
[23:25:50] <SPF|Cloud>	 I still have SSH access
[23:26:10] <Cyberpower678>	 SPF|Cloud, then don't mind me.  I'm just talking out of my ass. :p
[23:26:19] <bd808>	 sweet. can you touch a file in your homedir named "reset_2fa.txt"
[23:27:23] <SPF|Cloud>	 southparkfan@bastion-01:~$ ls
[23:27:23] <SPF|Cloud>	 reset_2fa.txt
[23:27:46] <wikibugs>	 6Labs, 10Tool-Labs: Upgrade to Kubernetes 1.2 - https://phabricator.wikimedia.org/T130972#2152462 (10yuvipanda)
[23:27:48] <Cyberpower678>	 kaldari, a sortable table for the logging interface would be nice. :D
[23:28:46] <bd808>	 SPF|Cloud: hmm. I
[23:28:54] <bd808>	 I'm not seeing the file
[23:29:07] <SPF|Cloud>	 Are you logged in at bastion-01?
[23:29:14] <SPF|Cloud>	 I can ssh to tools-dev if you want
[23:29:22] <bd808>	 I'll cehck on bastion-01
[23:29:28] <wikibugs>	 6Labs, 10Tool-Labs: Upgrade to Kubernetes 1.2 - https://phabricator.wikimedia.org/T130972#2152462 (10yuvipanda) With https://gerrit.wikimedia.org/r/#/c/279648/ and the 'operations/software/kubernetes' repo, I've already rebased the patch and fixed the test failures!   We still need to change the format of the...
[23:30:37] <bd808>	 SPF|Cloud: bah I can't sudo on bastion-01 (needs real root). Can you do it on tools-dev so I can see it?
[23:31:30] <SPF|Cloud>	 southparkfan@tools-bastion-02:~$ ls -l | grep 2fa
[23:31:30] <SPF|Cloud>	 -rw-r--r-- 1 southparkfan wikidev          0 Mar 25 23:31 reset_2fa.txt
[23:32:02] <bd808>	 SPF|Cloud: got it. Now to try out the db change for the first time :)
[23:32:16] <SPF|Cloud>	 okay
[23:34:13] <wikibugs>	 6Labs, 10Tool-Labs: Upgrade to Kubernetes 1.2 - https://phabricator.wikimedia.org/T130972#2152512 (10yuvipanda)
[23:34:15] <wikibugs>	 6Labs, 10Tool-Labs, 13Patch-For-Review, 3Scap3: Setup a proper deployment strategy for Kubernetes - https://phabricator.wikimedia.org/T129311#2152511 (10yuvipanda)
[23:34:46] <bd808>	 SPF|Cloud: should be fixed.
[23:37:07] <SPF|Cloud>	 yep. thanks :D
[23:37:30] <bd808>	 yw! Now make sure to print out the recovery keys when you re-enable :)
[23:52:13] <Technical_13>	 Puppet is failing to run on the "xtools-lb" instance in the Wikimedia Labs
[23:52:14] <Technical_13>	 project "xtools".
[23:52:42] <bd808>	 Technical_13: need some help figuring out why?
[23:52:45] <mutante>	 Technical_13: what's the error?
[23:52:50] <Technical_13>	 I have a newborn and no time to fix this. Is there a labs admin that could look in to it for me?
[23:53:07] <bd808>	 I can take a look
[23:54:12] <Technical_13>	 Thank you. The error isn't specified in the email.
[23:54:24] <bd808>	 !log xtools Added self (BryanDavis) as admin to debug puppet failure
[23:54:29] <Technical_13>	 And I have no time to set up a client on my new phone.
[23:55:51] <kaldari>	 Cyberpower678: Created a pull request for the bot listing bug. Just have to wait for Niharika to merge.
[23:56:21] <bd808>	 !log xtools Deleted stale puppet lock file on xtools-lb
[23:56:23] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Xtools/SAL, Master
[23:57:11] <bd808>	 Technical_13: fixed. easy peasy
[23:57:45] <bd808>	 !log xtools Removed self (BryanDavis) from project
[23:57:46] <Technical_13>	 Great, thanks bd808 
[23:57:47] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Xtools/SAL, Master
[23:59:13] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Cannot SSH to a few CI slaves due to DNS failure - https://phabricator.wikimedia.org/T129640#2152572 (10dduvall) I've gone ahead and replaced the integration instances per @yuvipanda in IRC (new instances are `integration-slave-trusty-102...