[02:03:19] <Lofhi>	 Hello, does anyone know what's going on with PAWS? I have error 500: Internal Server Error. Redirect loop detected. 
[02:11:39] <Zppix>	 madhuvishy: bd808 paws is acting up ^
[02:18:15] <Lofhi>	 And "Spawner failed to start [status=1]" error :>
[02:22:16] <Lofhi>	 And now " Failed to start your server. Please contact admin." :>
[02:32:49] <madhuvishy>	 Lofhi: what's your paws username?
[02:33:16] <Lofhi>	 Lofhi
[02:33:19] <Lofhi>	 madhuvishy ^
[02:37:28] <madhuvishy>	 Lofhi: are you still getting redirect loop?
[02:38:09] <Lofhi>	 madhuvishy <
[02:38:11] <Lofhi>	 <3
[02:38:14] <Lofhi>	 It works now!
[02:38:18] <madhuvishy>	 cool
[02:38:23] <Lofhi>	 Thanks :)
[02:40:31] <madhuvishy>	 yw :)
[02:41:48] <Zppix>	 Thanks madhuvishy
[02:56:55] <Lofhi>	 madhuvishy, can I bother you a little more?
[02:57:08] <madhuvishy>	 Lofhi: sure, what's up 
[02:57:29] <Lofhi>	 When I go on https://wikimedia.org/api/rest_v1/metrics/pageviews/top/fr.wikipedia.org/all-access/2018/01/07 with Google Chrome, it shows me
[02:57:32] <Lofhi>	 {"type":"https://mediawiki.org/wiki/HyperSwitch/errors/not_found","title":"Not found.","method":"get","detail":"The date(s) you used are valid, but we either do not have data for those date(s), or the project you asked for is not loaded yet.  Please check https://wikimedia.org/api/rest_v1/?doc for more information.","uri":"/analytics.wikimedia.org/v1/pageviews/top/fr.wikipedia.org/all-access/2018/01/07"}
[02:57:42] <Lofhi>	 But when I go on the same URL with Firefox
[02:57:57] <Lofhi>	 File not found
[02:57:57] <Lofhi>	 Firefox cannot find the file at https://wikimedia.org/api/rest_v1/metrics/pageviews/top/en. wikipedia. org/all-access/2018/01/07.
[02:58:17] <Lofhi>	 You know why there's such a difference?
[02:58:31] <madhuvishy>	 right, It's possible that the http content headers look different
[02:59:25] <Lofhi>	 Is it possible to fix it? I thought my code was wrong by checking it myself via Firefox :[
[02:59:26] <madhuvishy>	 I don't know of the top of my head what it could be though, but looks like a difference in how the 2 browsers handle the content-type
[02:59:53] <madhuvishy>	 aah, our team doesn't really maintain the apis
[03:00:05] <Lofhi>	 Arf...
[03:00:08] <musikanimal>	 maybe it's a "feature" of Firefox to show a formatted 404 page when it isn't served as HTML
[03:00:25] <musikanimal>	 but you request as HTML
[03:00:36] <madhuvishy>	 wikimedia-analytics and wikimedia-services are better channels for this specific query. but I recommend using an extension like Postman to test queries to REST APIs
[03:01:09] <madhuvishy>	 https://www.getpostman.com/ 
[03:02:05] <madhuvishy>	 or even curl
[03:02:16] <Lofhi>	 All right, thank you for all this valuable information! :)
[03:03:34] <madhuvishy>	 no worries.
[03:03:42] * madhuvishy goes afk for dinner plans now
[03:06:28] <Lofhi>	 good appetite
[12:02:37] <Hauskatze>	 bd808: https://tools.wmflabs.org/meta/userpages/Evolution+and+evolvability displays fatal errors due to missing dbs we dropped
[12:03:01] <Hauskatze>	 is this something specific to Pathoschild's tool or it requires general wmcs involvement?
[12:17:21] <zhuyifei1999_>	 why would we drop a replica database?
[12:23:24] <Wiki13>	 well alswikibooks doesnt exist anymore for example. it got "dropped" from production, so it would make sense the replicas would delete it aswell.
[12:25:01] <Wiki13>	 I think his question was more meant as to why it errors on a nonexisting database
[12:28:43] <Hauskatze>	 https://github.com/Pathoschild/Wikimedia-contrib/issues/95
[12:29:58] <Wiki13>	 I have been looking at the database.php file and bit down it shows that two wikis are ignored
[12:31:37] <Wiki13>	 https://github.com/Pathoschild/Wikimedia-contrib/blob/f94b7aa1e9a3c7b3e5899db33d939098c7335490/tool-labs/backend/modules/Database.php#L485
[12:31:58] <Wiki13>	 maybe that could be used as a work around here
[12:33:40] <Hauskatze>	 or maybe he's using in that very tool the old dbs and not the svc.analytics... ones?
[12:33:47] <Hauskatze>	 therefore getting old data?
[12:33:51] <Hauskatze>	 not sure
[12:34:09] <Wiki13>	 also possible
[12:36:49] <jynus>	 Hauskatze I do not know much about that, but there is a meta_p database, that should be used to check/locate database existance
[12:37:53] <Hauskatze>	 jynus: ni idea, entre el inglés y mi poca programación estoy perdido
[12:40:39] <Wiki13>	 https://github.com/Pathoschild/Wikimedia-contrib/blob/f94b7aa1e9a3c7b3e5899db33d939098c7335490/tool-labs/backend/modules/Wikimedia.php#L43
[12:40:59] <Wiki13>	 It does use the new servers and it seems to use meta_p
[12:41:34] <Wiki13>	 metawiki_p, oops
[13:13:20] <jynus>	 metawiki_p <> meta_p
[13:13:31] <jynus>	 metawiki_p => meta.wikimedia.org
[13:13:48] <jynus>	 meta_p => lookup table of wikis available on wikireplicas
[13:23:51] <Wiki13>	 yea, I was confused for a sec there though I do know the difference :p
[13:36:06] <arturo>	 jynus: is there something I can help with?
[13:36:24] <jynus>	 not me
[13:37:49] <jynus>	 I think some tools has bad code
[13:48:13] <Wiki13>	 arturo: Someone came in here to ask why https://tools.wmflabs.org/meta/userpages/Evolution+and+evolvability didnt work
[13:48:39] <Wiki13>	 it errors on a database that does no longer exist (got deleted in production)
[13:49:50] <Wiki13>	 He opened https://github.com/Pathoschild/Wikimedia-contrib/issues/95 tho, but I dont know how active Patschild even is atm
[13:50:58] <arturo>	 ok Wiki13 I will talk the to team today. Does a phabricator ticket exists?
[13:51:10] <Wiki13>	 I don't know
[13:51:24] <Wiki13>	 do you mean for the databases?
[13:51:53] <Wiki13>	 let me search in the SAL log for production it should be in there
[13:54:00] <arturo>	 mmm I would need additional context. I don't know who Pathoschild is i.e. is that person supposed to maintain the code? Should we patch the code in toolsforge ourselves and then send a pull request to github? Better to have bd808 to look at it. I can create a phabricator task for us to don't forget about this
[13:54:00] <Wiki13>	 arturo: The ticket about deleting some wikis (including the one this tool errors on) got deleted per https://phabricator.wikimedia.org/T181923
[13:54:58] <Wiki13>	 in principle, they dont exist so the tool shouldn't error on it
[13:55:02] <jynus>	 s/deleted/resolved/
[13:55:22] <Wiki13>	 thanks for correcting me
[13:56:29] <arturo>	 will create another ticket for us WMCS with all this information
[13:56:51] <jynus>	 it seems meta_p tables have not been updated
[13:57:39] <jynus>	 there is T181925
[13:57:39] <stashbot>	 T181925: Remove als.wik(ibooks|iquote|tionary), mo.wik(ipedia|tionary) views from replicas - https://phabricator.wikimedia.org/T181925
[13:58:14] <jynus>	 I would reopen that one
[13:58:53] <arturo>	 jynus: is about the code or only about meta_p tables? if is about code, I would open a new one
[13:59:41] <jynus>	 I don't know about other's code
[13:59:50] <jynus>	 I know meta_p seems to be wrong
[14:00:07] <arturo>	 fair enough :-)
[14:00:45] <Wiki13>	 The DB connect code lives here: https://github.com/Pathoschild/Wikimedia-contrib/tree/f94b7aa1e9a3c7b3e5899db33d939098c7335490/tool-labs/backend/modules 
[14:00:58] <Wiki13>	 (Database.php and Wikimedia.php)
[14:03:44] <arturo>	 I created T184433
[14:03:44] <stashbot>	 T184433: tools: missing databases creates some issues - https://phabricator.wikimedia.org/T184433
[14:04:00] <arturo>	 jynus: Wiki13 feel free to add content/comments
[14:12:23] <Wiki13>	 arturo: Just checked other tools from that same project and they error too 
[14:12:31] <Wiki13>	 https://tools.wmflabs.org/meta/stalktoy/213.137.0.0%2F16
[14:13:04] <Wiki13>	 so maybe bad code aswell, but it might be aswell due to the outdated meta_p
[14:13:09] <arturo>	 Wiki13: could you please follow-up on the ticket? I will try to update meta_p
[14:13:17] <Wiki13>	 k
[14:13:23] <Wiki13>	 will update there
[14:18:23] <arturo>	 jynus: do you know if there is a SAL for labsdbXXX ?
[14:20:40] <arturo>	 I can't find one at https://wikitech.wikimedia.org/w/index.php?title=Category:SAL
[14:28:07] <jynus>	 arturo: for DBA operations, we log to the operation one- https://wikitech.wikimedia.org/wiki/SAL
[14:28:16] <jynus>	 for non-DBA operations, I don't know, we do not do those
[14:28:17] <Wiki13>	 Would https://tools.wmflabs.org/sal/tools <- (most generic one) the best for this?
[14:29:31] <jynus>	 labsdb host are not tools, they are production hosts
[14:30:34] <jynus>	 note I am talking about the host actions/dba things
[14:30:56] <jynus>	 I would agree that database content would be related more to tools
[14:33:03] <Wiki13>	 ah I see
[14:33:46] <Wiki13>	 anyway, gtg soon. will be back on later though
[14:36:46] * arturo food
[15:38:54] <ASammour>	 Hello. I have a problem connecting to database using java project.
[15:39:39] <jynus>	 please explain what you are trying to do and how, and what errors you get
[15:40:19] <ASammour>	 I'm trying to run a SQL query for example in specific wiki project.
[15:40:34] <ASammour>	 Like getting number of articles created by IPs
[15:41:09] <jynus>	 please share _exactly_ what you are typing or the code you run :-)
[15:41:12] <ASammour>	 I know that I can use quarry to write SQL, but I need to do it with my toolforge account
[15:41:18] <jynus>	 that is ok
[15:42:07] <ASammour>	 See this for example, https://en.wikipedia.org/wiki/Wikipedia:Database_reports
[15:42:18] <ASammour>	 I want to run it using java language
[15:42:26] <jynus>	 you can paste code here: https://phabricator.wikimedia.org/paste/edit/form/14/ to share it
[15:42:36] <jynus>	 so we can have a look
[15:45:32] <chasemp>	 ASammour: no one can help effectively without seeing the exact code you are running with an explanation of what you intend the code to do
[15:46:27] <ASammour>	 Ok, I'm looking for the code I used before, please wait.
[15:50:27] <ASammour>	 I've tried this code using netbeans, but it seems it want some dependencies https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Java
[15:51:51] <ASammour>	 Is there anu framework, or library to easily connect to database (Like class maybe)?
[15:53:39] <chasemp>	 this confuses me String url = "jdbc:mysql://tools.db.svc.eqiad.wmflabs:3306/enwiki_p";
[15:53:44] <chasemp>	 would that...work?
[15:54:05] <chasemp>	 it seems like it conflates toolsdb and wikireplicas to me
[15:55:13] <paladox>	 nope that would not work.
[16:01:14] <Ivy>	 Hi.
[16:01:38] <chasemp>	 ASammour: there is no single "the database." There are databases for users/Tools to use interactivey and those that are read-only that house replica data from production.  Depends on what you are doing.  Based on referencing quarry I suspect you want that string to be somethign ike "enwiki.web.db.svc.eqiad.wmflabs"
[16:01:43] <Ivy>	 ASammour: For database reports specifically, the (old) underlying scripts are here: https://en.wikipedia.org/wiki/Wikipedia:Database_reports/Indefinitely_blocked_IPs/Configuration
[16:02:03] <Ivy>	 You can go to "/Configuration" for any report to see a copy of its source code.
[16:02:16] <Ivy>	 You may find Python to be more enjoyable than Java, fwiw.
[16:19:34] <ASammour>	 Thats what come to me: " java.net.UnknownHostException MESSAGE: tools.db.svc.eqiad.wmflabs"
[16:20:02] <ASammour>	 I've take this host url "String url = "jdbc:mysql://tools.db.svc.eqiad.wmflabs:3306/enwiki_p";"
[16:20:08] <ASammour>	 Is it right?
[16:20:14] <chasemp>	 ASammour: that host exists but that database will not exist on that host 
[16:20:17] <chasemp>	 ASammour: no
[16:20:26] <chasemp>	 ASammour: please explain what you are trying to do
[16:20:59] <ASammour>	 I'm trying to connect to enwiki_p database in my java project, in netbeans IDE
[16:21:06] <chasemp>	 ok
[16:21:23] <chasemp>	 ASammour: change that host to enwiki.web.db.svc.eqiad.wmflabs
[16:21:27] <ASammour>	 I've written this code: "        Class.forName("com.mysql.jdbc.Driver").newInstance();         String url = "jdbc:mysql://tools.db.svc.eqiad.wmflabs:3306/enwiki_p";         Connection conn = DriverManager.getConnection(url, " user","password");"
[16:21:48] <ASammour>	 This should work with normal database
[16:21:59] <ASammour>	 But I think the database url is wrong
[16:24:16] <ASammour>	 You mean like this "String url = "jdbc:mysql://enwiki.web.db.svc.eqiad.wmflabs";"?
[16:25:38] <chasemp>	 ASammour: based on that example try: enwiki.web.db.svc.eqiad.wmflabs:3306/enwiki_p
[16:27:22] <chasemp>	 ASammour: note this is a read-only database
[16:27:43] <andrewbogott>	 paladox: do you know anything about/have an opinion about https://phabricator.wikimedia.org/T184450 ?
[16:28:01] <paladox>	 andrewbogott hi, im not in that project
[16:28:05] <paladox>	 but wmf got rid of gitblit
[16:28:17] <andrewbogott>	 yeah, I just saw your name come up a fair bit in discussion about deprecating the phab tag and related things
[16:28:29] <paladox>	 ah yeh, i guess it can be deleted
[16:28:36] <paladox>	 as it dosen't look like it gets any activity.
[16:29:17] * andrewbogott nods
[16:29:23] <andrewbogott>	 I'll send a notification of deletion email
[16:29:33] <paladox>	 ok
[16:29:36] <paladox>	 thanks
[16:34:24] <andrewbogott>	 abartov: I would like to delete the VM 'openocr.openocr.eqiad.wmflabs' (which you created).  Do you have any objections?
[16:37:54] <ASammour>	 chasemp: It result this "java.sql.SQLException: No suitable driver found for enwiki.web.db.svc.eqiad.wmflabs:3306/enwiki_p" even I've put the mysql-connector in the project.
[16:56:43] <ASammour>	 Any hint?
[16:57:28] <chasemp>	 ASammour: taht seems like a java specific error and I'm not sure atm
[16:58:38] <ASammour>	 Well it occurs when I don't add mysql-connector jar file in the project.
[16:59:14] <ASammour>	 I googled about the error, and they say it also occurs when url host is incorrect.
[18:03:08] * zhuyifei1999_ thinks ASammour "jdbc:mysql://" part
[18:03:15] <zhuyifei1999_>	 * missed
[18:04:08] <zhuyifei1999_>	 and the host resolving error is that they are doing it locally rather than on cloud, considering that netbeans is an IDE
[18:04:19] <zhuyifei1999_>	 (with a gui)
[18:05:45] <chasemp>	 zhuyifei1999_: heh, nice catch -- yeah of course :D
[18:06:32] <wm-bot>	 I will let you know when I see ASammour and I will deliver that message to them
[18:06:32] <zhuyifei1999_>	 @notify ASammour Please see the logs of #wikimedia-cloud regarding your JDBC error
[18:35:54] <abartov>	 andrewbogott: no, go ahead. It turns out nobody used it.
[18:36:11] <andrewbogott>	 abartov: great.  Is that whole project defunct?
[19:12:56] <abartov>	 andrewbogott: yes. It's just a Labs instance of the vanilla openocr open source code. Nothing valuable there.
[19:26:07] <chasemp>	 !log tools sudo service docker restart; sudo service flannel restart; sudo service kube-proxy restart on tools-proxy-02
[19:26:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:36:57] <andrewbogott>	 !log openocr deleting project and all VMs
[19:36:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openocr/SAL
[20:34:38] <madhuvishy>	 !log tools Restart kube services and uncordon tools-worker-1001
[20:34:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:37:35] <chasemp>	 madhuvishy: what was up with tools-worker-1001?
[20:38:14] <madhuvishy>	 chasemp: nothing, I had it cordoned as a canary when I restarted services every where else on friday
[20:38:18] <chasemp>	 kk
[21:53:26] <wikibugs>	 (03PS1) 10Ottomata: Mv varnishkafka profile certificate::ssl_key_password [labs/private] - 10https://gerrit.wikimedia.org/r/403061
[21:54:06] <wikibugs>	 (03CR) 10Ottomata: [V: 032 C: 032] Mv varnishkafka profile certificate::ssl_key_password [labs/private] - 10https://gerrit.wikimedia.org/r/403061 (owner: 10Ottomata)
[22:34:25] <Zppix>	 Is identd coming anytime in the future? I recall there was a discussion about it awhile back
[22:35:52] <harej>	 like, for IRC?
[22:37:47] <paladox>	 Zppix ^^
[22:39:00] <mutante>	 you would have to run identd where your IRC client runs
[22:39:12] <mutante>	 or bouncer
[22:39:33] <mutante>	 that would get rid of the dreaded ~ character in front of your nickname in /whois  ;)
[22:40:38] <Zppix>	 No i meant for toolforge/vps
[22:40:44] <bd808>	 there's a task, but nobody has worked on it
[22:41:26] <bd808>	 freenode has offered to raise the per-ip limits for us if we have solid identd support
[22:41:48] <bd808>	 us == cloud VPS instances and most strongly Toolforge
[22:41:53] <harej>	 Is there a clear benefit to it?
[22:42:10] <harej>	 (extremely product manager voice) who would be the customer for this service
[22:42:31] <bd808>	 right now we apply public ips to a lot of toolforge worker nodes so that freenode bots work as expected
[22:43:03] <bd808>	 so the customer is WMCS admins and conservation of our limited public IPv4 pool
[22:43:25] <bd808>	 but! IPv6 may be only 9-12 months out and make that all moot
[22:44:45] <harej>	 I seem to recall that IPv6 has existed for at least a decade ;)
[22:45:19] <bd808>	 harej: not if you are running nova-network as your OpenStack software defined networking layer ;)
[22:45:29] <harej>	 Well there you go
[22:45:53] <harej>	 So, what actually would this take? Just a cloud VPS server running some kind of server daemon and a public IP?
[22:46:52] <bd808>	 there's a special identd that can delegate to a backend server for ident verification. we'd need that setup on the Cloud VPS gateway and all the nodes that would be clients of it
[22:47:38] <bd808>	 if you don't have a static IP on your instance then all your traffic is SNAT translated to a single ip.
[22:48:16] <bd808>	 there are good reasons honestly that we would keep per-exec node public IPs for Toolforge even without Freenode though
[22:48:41] <chasemp>	 +1
[22:48:54] <bd808>	 like so that when website X complains that it was bing attacked we would have a more narrow list of tools to look at
[22:49:43] <bd808>	 T151704 is the identd task
[22:49:44] <stashbot>	 T151704: Freenode sometimes throttles bot connections from tools - https://phabricator.wikimedia.org/T151704
[22:50:06] <chasemp>	 if you gave me a big enough IP pool I would give every pod a unique ipv6 address
[22:50:14] <chasemp>	 :)
[22:50:27] <bd808>	 that should be easy to arrange, and yes correct
[22:51:55] <bd808>	 each should actually get a /64 or something really since that's the granularity that folks tend to use to throw ban hammers at IPv6
[22:52:54] <Krenair>	 <bd808> right now we apply public ips to a lot of toolforge worker nodes so that freenode bots work as expected
[22:53:09] <Krenair>	 As I recall, only some workers and not others, so it's random as to whether you end up on an instance with one or not?
[22:53:13] <Krenair>	 or was that resolved?
[22:53:29] <bd808>	 I think its mostly resolved these days
[22:53:45] <bd808>	 but I don't think we have an alert/report that checks that regularly
[22:56:41] <Krenair>	 "In theory, running either of these services on our outbound NAT host(s) and all of the grid engine exec nodes would allow an ident request to the NAT'ed ip to find its way to the appropriate grid node to determine the tool account that is actually opening the irc connection."
[22:56:44] <Krenair>	 very interesting
[22:57:18] <Krenair>	 what would happen when it is asked about a connection from something other than a tools grid engine exec node?
[22:57:29] <Krenair>	 just depends on whether that host runs an identd of its own?
[22:58:14] <bd808>	 I think so, yeah. The front identd would be setup to delegate. We'd have to test/read docs to see what happens when the delegation is not answered
[22:58:31] <Ivy>	 This is about running oidentd or something again?
[22:58:42] <bd808>	 Ivy: yup
[22:59:03] <bd808>	 Zppix poked about it in backscroll
[22:59:45] <Ivy>	 Ahh.
[22:59:52] <Krenair>	 so this would sit on the labnet host that handles everything into 208.80.155.128/25, and would intercept anything headed for the identd port?
[23:00:02] <bd808>	 correct
[23:00:05] <Ivy>	 Toolserver used oidentd with the "users can change their ident" option disabled.
[23:01:03] <Krenair>	 which right now appears to be labnet1001.eqiad.wmnet