[03:18:20] <Andy_>	 Hello everyone, I'm Andy.
[03:19:00] <Andy_>	 Will the project add user rights, is it looking for an administrator here?
[03:19:16] <Andy_>	 Add members and admin users to a project Project admins can use the following steps to add new members or grant administrative permissions to members.  Go to Special:NovaProject. Use Add member link to add new members. Use Add member link for the projectadmin role in the Role column. Log at #wikimedia-cloud connect that you added/granted admin to the member.
[03:20:19] <Andy_>	 I saw this message, so IRC came in and I hope you can specify it. Thank you.
[03:36:50] <wikibugs>	 (03PS2) 10Rosalieper: [WIP]Trying to use the async and await keywords for the download and twit [labs/tools/Commons-twitter-bot] (Promise-dev) - 10https://gerrit.wikimedia.org/r/441150
[07:22:32] <Nemo_bis>	 I suspect a job of mine has been stuck for 13 days now because other jobs on the same host take all the CPU
[07:26:03] <Nemo_bis>	 They all have the same nice level though
[07:31:29] <Nemo_bis>	 And ps says it's "Ssl" so it might be waiting for something... but what
[07:44:07] <Nemo_bis>	 Maybe for this
[07:44:07] <Nemo_bis>	 python2 4411 tools.oabot    3u  IPv4 49224430      0t0  TCP tools-exec-1401.tools.eqiad.wmflabs:39000->text-lb.eqiad.wikimedia.org:https (CLOSE_WAIT)
[07:57:21] <UnitedKingdumb>	 nigger
[07:57:32] <UnitedKingdumb>	 !ops
[07:57:32] <wm-bot>	 UnitedKingdumb: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[07:57:41] <UnitedKingdumb>	 !ops !ops
[07:57:41] <wm-bot>	 UnitedKingdumb: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[08:01:29] <UnitedKingdumb>	 nigger Matthew_ 
[08:01:36] <UnitedKingdumb>	 you forgot this chan
[08:01:39] <UnitedKingdumb>	 ha ha loser
[08:03:42] <UnitedKingdumb>	 !ops nigger cunt fuck faggot pussy
[08:03:42] <wm-bot>	 UnitedKingdumb: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[10:21:35] <Lucas_WMDE>	 !log wikidata-dev wikidata-lexeme resurrect lexeme demo system so SMalyshev can test search integration
[10:21:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL
[10:27:47] <Lucas_WMDE>	 !log wikidata-dev wikidata-lexeme `sudo do-release-upgrade` (we need a newer PHP version for recent MediaWiki versions)
[10:27:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL
[10:33:56] <Hauskatze>	 !help Is it possible to regenerate a replica.my.cnf file which was accidentally committed?
[10:33:57] <wm-bot>	 Hauskatze: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[10:51:18] <Lucas_WMDE>	 Hauskatze: from https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Regenerate_replica.my.cnf and https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore#maintain-dbusers, it looks to me like an admin would only have to delete your current account and then a new one would automatically be generated
[10:51:21] <Lucas_WMDE>	 but I’m not an admin
[10:52:07] <Hauskatze>	 Lucas_WMDE: I've added Bryan D. to the task
[10:54:02] <Lucas_WMDE>	 !log wikidata-dev wikidata-lexeme reboot after upgrade
[10:54:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL
[11:00:38] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: hieradata: add eqiad1 keystone admin_token [labs/private] - 10https://gerrit.wikimedia.org/r/442063 (https://phabricator.wikimedia.org/T196633)
[11:03:13] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [V: 032 C: 032] hieradata: add eqiad1 keystone admin_token [labs/private] - 10https://gerrit.wikimedia.org/r/442063 (https://phabricator.wikimedia.org/T196633) (owner: 10Arturo Borrero Gonzalez)
[11:19:22] <zhuyifei1999_>	 Nemo_bis: which job is stuck? I can look into it
[11:44:18] <Lucas_WMDE>	 !log wikidata-dev wikidata-lexeme converted ladsgroup’s cronjob to update MediaWiki into systemd units update-mediawiki.{service,timer}
[11:44:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL
[12:07:40] <Nemo_bis>	 zhuyifei1999_: oabot_prefill, process 4411 on tools-exec-1401, job ID 6533332
[12:07:46] <zhuyifei1999_>	 ok
[12:08:14] <Nemo_bis>	 In the worst case it's waiting for some response that got "lost" during the reboots, or something, and the connection needs to be closed in gdb AFAICT
[12:09:04] <zhuyifei1999_>	 strace says 'futex(0x2bac7f7c5a90, FUTEX_WAIT_PRIVATE, 0, NULL'. threads?
[12:09:25] <Nemo_bis>	 I could restart the job but it took 30798 min to reach this point
[12:09:45] <Nemo_bis>	 yes there are multiple threads
[12:11:17] <Nemo_bis>	 (very simplistic multi-threading)
[12:11:59] <zhuyifei1999_>	 thread 21779, 21717, and 7282 are reading on fd 229, 325, and 393 respectively
[12:12:51] <zhuyifei1999_>	 tools-exec-1401.tools.eqiad.wmflabs:45421->diego.dissem.in:https
[12:13:10] <zhuyifei1999_>	 all of them goes to that host
[12:13:16] <zhuyifei1999_>	 (diego.dissem.in:https_
[12:13:18] <zhuyifei1999_>	 *)
[12:13:35] <Nemo_bis>	 all of those can be closed/killed if useful
[12:13:54] <zhuyifei1999_>	 with gdb?
[12:15:04] <Nemo_bis>	 nevermind, I just killed the whole thing by mistake I guess
[12:15:11] <zhuyifei1999_>	 I can't see the process anymore
[12:15:14] <zhuyifei1999_>	 oh okay
[12:15:18] <Nemo_bis>	 I'll just restart and wait a few weeks :)
[12:15:37] <Nemo_bis>	 This host is too busy with mono bots anyway ;) 
[12:15:47] <Nemo_bis>	 Thanks for the help
[12:15:54] <zhuyifei1999_>	 np
[12:16:44] <zhuyifei1999_>	 too busy with mono bots... you remind me of something
[12:18:02] <zhuyifei1999_>	 https://phabricator.wikimedia.org/T195834
[12:33:30] <zhuyifei1999_>	 !log tools.dibot killed all jobs besides lighttpd T195834
[12:33:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dibot/SAL
[12:33:32] <stashbot>	 T195834: mono-based bot hangs after mono version upgrade - https://phabricator.wikimedia.org/T195834
[12:34:53] <zhuyifei1999_>	 Nemo_bis: should be much cleaner now :)
[12:42:12] <Nemo_bis>	 \o/
[12:45:27] <Nemo_bis>	 CPU usage just went down by more than two thirds :) https://tools.wmflabs.org/nagf/?project=tools
[12:47:23] <paladox>	 it has over 12tb of memory wow
[12:47:43] <paladox>	 uh i mean 1.2tb
[12:50:34] <paladox>	 hmm 
[12:50:35] <paladox>	 https://graphite-labs.wikimedia.org/render/?title=tools-worker-1017.tools.eqiad.wmflabs+Memory+last+day&width=800&height=250&from=-1day&hideLegend=false&uniqueLegend=true&target=alias%28color%28stacked%28diffSeries%28tools.tools-worker-1017.tools.eqiad.wmflabs.memory.MemTotal%2Ctools.tools-worker-1017.tools.eqiad.wmflabs.memory.%7BMemFree%2CBuffers%2CCached%7D%29%29%2C%22%235555cc%22%29%2C%22Used%22%29&target=alias%28color%28stacked%28tools.to
[12:50:35] <paladox>	 ols-worker-1017.tools.eqiad.wmflabs.memory.Cached%29%2C%22%2333cc33%22%29%2C%22Cached%22%29&target=alias%28color%28stacked%28tools.tools-worker-1017.tools.eqiad.wmflabs.memory.Buffers%29%2C%22%2399ff33%22%29%2C%22Buffers%22%29&target=alias%28alpha%28color%28stacked%28tools.tools-worker-1017.tools.eqiad.wmflabs.memory.MemFree%29%2C%22%23f0ffc0%22%29%2C0.4%29%2C%22Free%22%29&target=alias%28color%28stacked%28tools.tools-worker-1017.tools.eqiad.wm
[12:50:35] <paladox>	 flabs.memory.SwapCached%29%2C%22%239900CC%22%29%2C%22Swap%22%29&target=alias%28color%28tools.tools-worker-1017.tools.eqiad.wmflabs.memory.MemTotal%2C%22red%22%29%2C%22Total%22%29
[12:50:37] <paladox>	 uh
[12:50:42] <paladox>	 that is a large url
[12:51:06] <paladox>	 this https://phabricator.wikimedia.org/P7295 url is returning 500
[12:51:22] <paladox>	 "Graphite encountered an unexpected error while handling your request."
[12:51:29] <paladox>	 "Please contact your site administrator if the problem persists."
[12:51:38] <paladox>	 TypeError: reduce() of empty sequence with no initial value
[12:52:16] <paladox>	 !help ^^
[12:52:17] <wm-bot>	 paladox: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[12:53:40] <paladox>	 also in the topic it says "#wikimediacloudfeed", shoulden't that be #wikimedia-cloud-feed?
[12:56:55] <paladox>	 is graphite running on trusty?
[12:57:33] <paladox>	 I think it may be fixed in https://github.com/graphite-project/graphite-web/pull/646 https://github.com/graphite-project/graphite-web/issues/338
[13:07:48] <paladox>	 filed a task @ https://phabricator.wikimedia.org/T198209
[14:54:52] <wikibugs>	 (03CR) 10Andrew Bogott: "I'm not convinced about the elimination of hieradata/labs.yaml -- common.yaml applies to MANY more hosts and it seems useful to only set t" [labs/private] - 10https://gerrit.wikimedia.org/r/423189 (owner: 10EddieGP)
[16:05:47] <eddiegp>	 andrewbogott: re: https://gerrit.wikimedia.org/r/423189 - which are those "many more hosts"? I'm a bit confused by this. I looked at hierarchy definition in puppet (labs.hiera.yaml and production.hiera.yaml). According to those, both the labs.yaml and the common.yaml file are only loaded on cloud puppetmasters.
[16:06:13] <andrewbogott>	 eddiegp: ok, that confuses me, I'll have to look again
[16:06:30] <eddiegp>	 You did note this is the labs/private, NOT operations/puppet repo, right?
[16:07:34] <chasemp>	 partially agree and partially disagree eddiegp, andrewbogott I can try to comment there but it will be a day or two
[16:07:49] <chasemp>	 I think the analysis is right, but the outcomes I have in mind are different
[16:08:06] <andrewbogott>	 eddiegp: yes, but labs/private is meant to shadow the live private repo, so any change made there needs to be made in the other (invisible-to-you) repo.
[16:08:23] <chasemp>	 yes I agree that removing labs.yaml isn't in the cards atm
[16:08:23] <andrewbogott>	 But yeah, mostly I'll defer to Chase on this because he's thought about that hiera search path a lot more than I have.
[16:09:25] <eddiegp>	 So then we're keeping labs.yaml in the not-private private repo just for the sake of mirroring how the real private repo looks?
[16:10:45] <chasemp>	 outcome yes, reasoning no
[16:11:09] * chasemp afk
[16:18:48] <eddiegp>	 chasemp: I don't really understand what you were trying to say. Please comment on the patch when you find time. (Don't wanna argue about it, but am curious ;-) )
[16:41:36] <wikibugs>	 (03PS4) 10EddieGP: Remove labs/<projectname>/common.yaml hiera path [labs/private] - 10https://gerrit.wikimedia.org/r/423189
[16:45:12] <eddiegp>	 andrewbogott: ^ Updated, only changing the other thing then :)
[17:27:45] <Tobi_>	 Hello
[17:27:51] <Tobi_>	 Nice
[17:34:01] <zhuyifei1999_>	 paladox: try my kinda-old fork: https://tools.wmflabs.org/yifeibot/nagf/?project=tools :)
[17:34:17] <paladox>	 oh
[17:34:26] <paladox>	 that worked
[17:34:37] <paladox>	 i guess the latest version is broken
[17:34:40] <zhuyifei1999_>	 (because I like the color of ganglia :P)
[17:34:56] <paladox>	 heh
[17:34:58] <zhuyifei1999_>	 and added a few features before sumbitting a pull req upstream
[17:36:38] <paladox>	 thanks
[17:38:09] <zhuyifei1999_>	 :)
[17:38:48] <zhuyifei1999_>	 though, I do wish someone figures out what's wrong with upstream
[20:55:42] <wikibugs>	 (03PS3) 10Rosalieper: [WIP]Trying to use the async and await keywords for the download and twit [labs/tools/Commons-twitter-bot] (Promise-dev) - 10https://gerrit.wikimedia.org/r/441150
[22:43:48] <wikibugs>	 (03PS4) 10Rosalieper: [WIP]Trying to use the async and await keywords for the download and twit [labs/tools/Commons-twitter-bot] (Promise-dev) - 10https://gerrit.wikimedia.org/r/441150