[00:22:13] <wikibugs_>	 10Striker: Groups and tools only refreshed at login - https://phabricator.wikimedia.org/T144943#3557385 (10bd808) Places where Django Groups are used in #Striker:  `striker.goals.signals.handlers.check_goal_member()` * checks user.groups for settings.TOOLS_TOOL_LABS_GROUP_NAME  `striker.goals.signals.handlers.ch...
[00:34:54] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1431 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[00:35:10] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1404 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[01:09:53] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1431 is OK: OK: Less than 1.00% above the threshold [0.0]
[01:10:09] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0]
[02:23:45] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[03:03:45] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0]
[05:46:33] <wikibugs_>	 (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374116 (owner: 10L10n-bot)
[06:17:51] <wikibugs_>	 (03PS1) 10Legoktm: Re-enable all rules before running [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374126
[06:18:54] <wikibugs_>	 (03CR) 10Legoktm: [C: 032] Re-enable all rules before running [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374126 (owner: 10Legoktm)
[06:19:13] <wikibugs_>	 (03Merged) 10jenkins-bot: Re-enable all rules before running [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374126 (owner: 10Legoktm)
[06:26:23] <wikibugs_>	 (03PS1) 10Legoktm: Use Phabricator instead of Github [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374127
[06:27:22] <wikibugs_>	 (03PS1) 10Legoktm: Only write to the JSON file once [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374128
[06:27:39] <wikibugs_>	 (03CR) 10Legoktm: [C: 032] Use Phabricator instead of Github [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374127 (owner: 10Legoktm)
[06:27:58] <wikibugs_>	 (03Merged) 10jenkins-bot: Use Phabricator instead of Github [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374127 (owner: 10Legoktm)
[06:28:00] <wikibugs_>	 (03CR) 10Legoktm: [C: 032] Only write to the JSON file once [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374128 (owner: 10Legoktm)
[06:28:16] <wikibugs_>	 (03Merged) 10jenkins-bot: Only write to the JSON file once [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374128 (owner: 10Legoktm)
[06:48:03] <shinken-wm>	 PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[06:48:16] <wikibugs_>	 10cloud-services-team, 10Community-Tech, 10DBA, 10Security: create production ip_changes table for RangeContributions - https://phabricator.wikimedia.org/T173891#3557553 (10ArielGlenn) >>! In T173891#3546484, @Bawolff wrote: >>>! In T173891#3546429, @kaldari wrote: >>>The table should not be in dumps - it...
[07:28:04] <shinken-wm>	 RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[08:39:13] <wikibugs_>	 (03CR) 10Lokal Profil: "are there any changes compared to the cherry-picked branch?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/373878 (https://phabricator.wikimedia.org/T166528) (owner: 10Jean-Frédéric)
[08:50:24] <wikibugs_>	 10MediaWiki-extensions-OpenStackManager, 10Patch-For-Review: MW OpenStackManager: add support for ED25519 SSH keys - https://phabricator.wikimedia.org/T159070#3557708 (10Paladox) 05Open>03Resolved Closing as resolved as this should now work. Please re open if it does not.
[08:51:22] <wikibugs_>	 (03PS12) 10Paladox: Connect wikibugs to irc over ssl [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/328663 (https://phabricator.wikimedia.org/T141089)
[08:51:55] <wikibugs_>	 (03CR) 10Paladox: "> @Paladox got a wikibugs maintainer to deploy/restart/check on it?" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/328663 (https://phabricator.wikimedia.org/T141089) (owner: 10Paladox)
[10:11:23] <multichill>	 mark: What tool are you using again to document rackspace usage? Is it any good?
[11:16:17] <mark>	 multichill: check out netbox, we're just evaluating it
[11:32:13] <wikibugs_>	 10Data-Services, 10cloud-services-team (Kanban), 10Analytics, 10DBA, 10Research: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3558277 (10jcrespo) Duplicate of T156869? We should fix it for all users, not only datasets_p?
[12:30:11] <gehel>	 chasemp: Hello! For whenever you have some time, could you have a look at T169133 and let me know if you are missing anything to move forward?
[12:30:12] <stashbot>	 T169133: WDQS testing setup platform sizing - https://phabricator.wikimedia.org/T169133
[12:33:28] <wikibugs_>	 10Cloud-Services, 10DBA: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3558381 (10MarcoAurelio) Hmm. I though you had to check previously, I mean, before the DB is created if there's enough disk space, etc.? In any case, we'll notify you again after the DB is cre...
[12:34:45] <wikibugs_>	 10Cloud-Services, 10DBA: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3558388 (10Marostegui) >>! In T173027#3558381, @MarcoAurelio wrote: > Hmm. I though you had to check previously, I mean, before the DB is created if there's enough disk space, etc.? In any cas...
[12:36:02] <wikibugs_>	 10Cloud-Services, 10DBA: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3558396 (10MarcoAurelio) Okie dockie :)  Everything is set up already I think. We just need a sysadmin willing to do it and a deployment window.
[13:41:50] <Medlin>	 https://drive.google.com/file/d/1T_lEo8IVwwJxyGha4yjZ4SvJjdAdkg0IVtZWdKaTIIeE631nEFxmee0onkWSl4srDGPN0wXGPDNS94C2/view?usp=drivesdk
[14:00:01] <wikibugs_>	 10Cloud-VPS (Quota-requests), 10Discovery, 10Wikidata, 10Wikidata-Query-Service: WDQS testing setup platform sizing - https://phabricator.wikimedia.org/T169133#3558613 (10Andrew)
[14:19:48] <chasemp>	 gehel: it looks like enough for us to talk about resourcing in our meeting tomorrow.  heads up it wasn't on the radar for looking at until monday of last week as andrew added the quota-requests project we use.
[14:20:11] <chasemp>	 that's the beginning of the process though, not the end, etc :)
[14:21:32] <gehel>	 chasemp: Thanks! So for next time, I should tag such request with "Cloud-VPS (Quota-requests)"? (I do have another such request coming up one of theses days...)
[14:22:05] <chasemp>	 gehel: that is what puts it in the queue for review, fyi there is a specific format for the tasks too https://phabricator.wikimedia.org/project/view/2880/ 
[14:22:47] <chasemp>	 gehel: there is a new-projects tag and a quota-requests tag (new and existing projects)
[14:23:00] <gehel>	 chasemp: Ok, I'll try to keep that in mind for next time...
[14:24:13] <chasemp>	 gehel: there are two tasks pending somewhere or other, the WDQS one and a maps oen somewhere, the speculative ask for both could be as high as 4T iirc
[14:24:32] <chasemp>	 that's is a lot of usage compared to most projects, that's like beta and tools territory
[14:24:50] <chasemp>	 do you want to talk it over and whatever budget there was for doing that on bare metal sometime?
[14:24:55] <chasemp>	 we may need to wait or purchase
[14:25:18] <gehel>	 I know there was budget to refresh the maps hardware, not sure if it can be reused for cloud
[14:25:23] <gehel>	 (but it should...)
[14:25:31] <chasemp>	 let's find out and how much it was?
[14:25:45] <gehel>	 chasemp: I can ask Faidon...
[14:26:09] <gehel>	 what exactly is "beta" in this context?
[14:26:18] <chasemp>	 deployment-prep
[14:26:25] <chasemp>	 I just meant, thats notably big
[14:27:06] <gehel>	 moving the wdqs-beta WM to -prep does not actually change anything
[14:27:08] <chasemp>	 we had a 500GB instance ask awhile ago that had to wait for this round of new hw (4 or 5 months they waited?)
[14:27:27] <chasemp>	 that's not what I mean
[14:27:39] <gehel>	 for maps, I'm still looking at how to reduce the footprint (which is surprisingly much easier than for wdqs)
[14:27:50] <chasemp>	 what I mean is the asks are big enough that we need a lot of heads up and possibly $ to fulfill in the near future
[14:28:02] <gehel>	 yep, understood
[14:29:01] <gehel>	 You made me wonder what the scope of deployment-prep is. It looks like a staging environment for project that go to production, and if it is, the wdqs-beta wm might make sense to be there (not that it changes anything in term of overall capacity)
[14:29:52] <chasemp>	 ah, that is what is but with various levels of equity in scaled down parity to prod components
[14:30:05] <chasemp>	 but yes resources are resources etc no matter the project envelope
[14:31:25] * gehel is now wondering why the wdqs-beta VM isn't in deployment-prep
[14:31:35] * gehel guesses: "historical reasons"...
[14:31:53] <chasemp>	 gehel: greg-g would the person to ask if that's a sane place
[14:32:02] <chasemp>	 it comes with benefits like not recreating many wheels
[14:32:06] <chasemp>	 but probably some responsibilities too
[14:33:44] <gehel>	 chasemp: for wdqs, we are a bit stuck on our current VM in term of disk space. The RAM / CPU of the current VM is fine though. And the goal is to replace them, not to add any new one. 
[14:34:13] <gehel>	 so in the end, we are *only* looking for a few 100GB of disk
[14:34:31] <chasemp>	 storage is the most contentious resource in cloud
[14:34:34] <gehel>	 would that be easier to find in the short term?
[14:34:35] <chasemp>	 we are overcommitted at all times
[14:34:44] <gehel>	 Damn...
[14:35:15] <chasemp>	 we have one of the new labvirt's down due to hw failure too
[14:35:28] <gehel>	 Ok, so no short term solution... too bad...
[14:35:30] <chasemp>	 so it's a matter of reviewing pending asks and projected growth and all that
[14:35:41] <chasemp>	 we may be able to grant it 
[14:35:54] <gehel>	 Ok, thanks for all the info, it is a bit clearer to me now!
[14:37:19] <chasemp>	 we need as much lead time as possible, and the lack of requeset tag there means it was never seriously considered until this week as a true ask, it seemed like a working out the options task
[14:39:36] <gehel>	 my bad, will do better next time :)
[14:40:21] <chasemp>	 heh gehel no intention to chastise man, but if you can find out if there is $ in the banana stand we can at least move things forward
[14:40:46] <gehel>	 damn, the only thing anyone is interested in is always money... :)
[14:40:49] <chasemp>	 but that's weeks of lag time there if we did buy a dedicated virt or expand with this in mind
[14:41:23] <gehel>	 yep, understood, at least now I understand the constraints (and lead time) with which to work...
[14:43:45] <wikibugs_>	 10Cloud-VPS (Quota-requests), 10Discovery, 10Wikidata, 10Wikidata-Query-Service: WDQS testing setup platform sizing - https://phabricator.wikimedia.org/T169133#3558747 (10Gehel) To summarize an IRC discussion with @chasemp:  * The specs above (8 CPU, 16Gb RAM, 500Gb disk) are the same as the current wdq-de...
[15:21:03] <multichill>	 mark: That looks nice indeed. I wonder if it supports overlapping vlans and overlapping ip space (happens a lot in a service provider network ;-) )
[15:25:10] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1407 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[15:28:35] <wikibugs_>	 10Data-Services, 10cloud-services-team (Kanban), 10Analytics, 10DBA, 10Research: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3558849 (10bd808) >>! In T173511#3558277, @jcrespo wrote: > Duplicate of T156869? We should fix it for...
[15:30:56] <Urbanecm>	 Hello, what does 'Error: Command '['/data/project/commons-mass-description-test/www/python/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1' mean? I got this message after running Error: Command '['/data/project/commons-mass-description-test/www/python/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1
[15:31:08] <Urbanecm>	 Sorry python3 -m venv $HOME/www/python/venv
[15:31:34] <multichill>	 mark: http://netbox.readthedocs.io/en/stable/data-model/ipam/ <- it does, looks good :-D
[15:31:54] <mark>	 nice
[15:36:30] <multichill>	 I think I need to dig up my requirement list. All the difficult questions (multi-tenant, ldap for authentication and authorization, overlap) seem to be ok.
[15:36:45] <multichill>	 The data center team used to track the racks, but ehm, that's a different company now....
[15:38:10] <paravoid>	 it's pretty neat
[15:38:23] <paravoid>	 and actively being developed
[15:38:31] <paravoid>	 there's a branch where upstream is adding virtualization support too
[15:38:59] <paravoid>	 oh and there's already support for NAPALM to fetch Juniper's linecards etc. for inventory, although I haven't tested it
[15:40:17] <multichill>	 Yeah, the Napalm stuff looks cool
[15:40:46] <multichill>	 We have all of that, Cisco IOS, Cisco Nexus, Fortigate, Juniper
[15:41:23] <multichill>	 Is your testing instance somewhere on labs or just on your own laptop paravoid?
[15:41:48] <paravoid>	 there's one in labs, but it's behind a password
[15:42:35] <paravoid>	 I can give you an account, but I'm playing with the data import etc., so you can't mess around with what's in there (i.e. read only)
[15:42:54] <paravoid>	 that said, I set it up on my laptop too and it was pretty trivial, took maybe 10 minutes
[15:43:15] <multichill>	 That would be nice to have a look around
[15:43:16] <paravoid>	 so if you really want to try various features such as the IPAM, I recommend that instead
[15:43:55] <multichill>	 That's probably the next step. 
[15:44:07] <bd808>	 Urbanecm: are you trying to do that on an Ubuntu Trusty host (like a Toolforge bastion?)
[15:44:38] <Urbanecm>	 I!m doing it at tools-bastion-02
[15:45:08] <bd808>	 yeah. So Trusty shipped with a broken venv module for python3 :/
[15:45:29] <bd808>	 You can google and find some ways to try and work around it
[15:45:56] <bd808>	 Or you can try to use our kubernetes containers which are running Debian Jessie
[15:46:37] <bd808>	 Urbanecm: If you really need Trusty, try this workaround -- https://askubuntu.com/a/488530/46612
[15:47:03] <Urbanecm>	 When I'm doing it at the machine that appears after webservice --backend=kubernetes python shell, everything works correctly
[15:47:26] <bd808>	 *nod* that would be on Debian instead of Ubuntu
[15:47:50] <Urbanecm>	 Is the foundation going to fix it?
[15:48:17] <bd808>	 Are we going to fix Trusty's packages? Probably not
[15:48:32] <Urbanecm>	 No, to install system-wide workaround
[15:48:46] <bd808>	 we have :) Use Kubernetes
[15:49:15] <bd808>	 or stick with python2 on Trusty
[15:49:40] <Urbanecm>	 I just have to remember I must switch into the debian box... 
[15:52:36] <bd808>	 The super long term solution will be when we pick a PaaS to overlay on Kubernetes and people start using whatever container build process that PaaS provides.
[15:53:48] <bd808>	 That will not be happening any time soon unfortunately. We have more problems to fix than people to work on them and that fell down on the priority list
[15:55:15] <andrewbogott>	 this will also be addressed by supporting grid engine on Stretch, right?  Which we hope to do soon anyway.
[15:55:49] <bd808>	 yeah, if we can move all the Grid to Stretch that should also fix it
[15:56:55] <bd808>	 Or I geuss first step would be a Stretch bastion and N stretch grid nodes
[16:03:00] <wikibugs_>	 (03CR) 10BryanDavis: [C: 032] Guard striker.tools.utils.tools_admin for anons [labs/striker] - 10https://gerrit.wikimedia.org/r/374085 (owner: 10BryanDavis)
[16:04:11] <wikibugs_>	 (03Merged) 10jenkins-bot: Guard striker.tools.utils.tools_admin for anons [labs/striker] - 10https://gerrit.wikimedia.org/r/374085 (owner: 10BryanDavis)
[16:05:09] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:05:27] <wikibugs_>	 (03PS1) 10BryanDavis: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/374343
[16:07:47] <wikibugs_>	 (03CR) 10BryanDavis: [C: 032] Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/374343 (owner: 10BryanDavis)
[16:07:53] <wikibugs_>	 (03Merged) 10jenkins-bot: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/374343 (owner: 10BryanDavis)
[16:34:23] <shinken-wm>	 PROBLEM - Puppet errors on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[16:41:43] <wikibugs_>	 10Tools: Tool "wikidata-timeline" loads Google Analytics - https://phabricator.wikimedia.org/T172972#3559174 (10cdrini) Unfortunately I won't be able to address this until halfway through September; I will provide input then. Sorry for the delay!
[16:48:35] <wikibugs_>	 (03CR) 10Jean-Frédéric: "> are there any changes compared to the cherry-picked branch?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/373878 (https://phabricator.wikimedia.org/T166528) (owner: 10Jean-Frédéric)
[17:09:21] <shinken-wm>	 RECOVERY - Puppet errors on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:28:41] <hall1467>	 Does anyone know if there is a tool that can be used to query Wikipedia page view data back to around 2012?
[17:30:02] <hall1467>	 I'm aware of this tool: https://tools.wmflabs.org/pageviews/?project=en.wikipedia.org&platform=all-access&agent=user&range=latest-20&pages=
[17:30:46] <hall1467>	 However, the page view data does not go all the way back to 2012
[17:32:32] <harej>	 There are kind of two eras of page views: the official, Analytics-blessed dataset that starts in 2015, and the dark ages before then that didn't accurately capture mobile views and probably didn't do a good enough job filtering out robots.
[17:38:53] <hall1467>	 harej: I see, that's helpful. Is there a tool that queries the "dark age" page view data? Sounds like the data provided by such a tool wouldn't be as informative unfortunately due to unfiltered robots, etc...
[17:39:31] <harej>	 there was stats.grok.se but I am not sure if it loads anymore
[17:39:56] <harej>	 It'd be nice if Analytics backloaded that data into their current Restbase setup but I understand why they would hesitate to do so
[17:42:34] <wikibugs_>	 10cloud-services-team (FY2017-18), 10Wikimedia-Blog-Content: Publish a blog post covering the Cloud Services rebranding process - https://phabricator.wikimedia.org/T170288#3559475 (10bd808) @MelodyKramer Its probably time to start thinking about getting this blog post together so we can get it published in Q1....
[17:43:56] <hall1467>	 harej: Okay, doesn't seem to load for me. Oh well. Thanks for the information :)
[17:48:49] <quiddity>	 hall1467, see top item at https://tools.wmflabs.org/pageviews/faq/ :)
[17:49:03] <wikibugs_>	 10cloud-services-team (FY2017-18), 10Wikimedia-Blog-Content: Publish a blog post covering the Cloud Services rebranding process - https://phabricator.wikimedia.org/T170288#3559494 (10MelodyKramer) This is actually on my to-do list for this week and I have this YouTube video (https://www.youtube.com/watch?v=edK...
[17:49:49] <bd808>	 hall1467: there may be some nuggets of information in this recent analytics mailing list thread too -- https://lists.wikimedia.org/pipermail/analytics/2017-August/005978.html
[18:00:42] <hall1467>	 bd808 , quiddity:  Thanks! The links were helpful. Looks like there currently isn't a tool available with older than July 2015 per-article view data
[18:01:50] <bd808>	 I think that is correct with stats.grok.se having gone offline recently
[18:03:07] <wikibugs_>	 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3559555 (10madhuvishy) Update: We are still blocked on talking to HP Support about the disk shelves.
[18:03:42] <hall1467>	 Right, that's my understanding too
[18:18:03] <wikibugs_>	 (03PS1) 10Rush: mirror openstack refactor values up to changeset 11816fd1b2 [labs/private] - 10https://gerrit.wikimedia.org/r/374372
[18:20:39] <wikibugs_>	 (03CR) 10Rush: [V: 032 C: 032] mirror openstack refactor values up to changeset 11816fd1b2 [labs/private] - 10https://gerrit.wikimedia.org/r/374372 (owner: 10Rush)
[18:23:02] <wikibugs_>	 (03Draft1) 10MarcoAurelio: Remove LTA-Knowledgebase [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/374373
[18:23:16] <wikibugs_>	 (03PS2) 10MarcoAurelio: Remove LTA-Knowledgebase [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/374373
[18:25:26] <wikibugs_>	 (03CR) 10MarcoAurelio: "Evidence: https://phabricator.wikimedia.org/T157725#3076502" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/374373 (owner: 10MarcoAurelio)
[18:28:07] <Zppix>	 For the love of god im half tempted to find a way to prevent the underscore in wikibugs
[18:30:13] <wikibugs_>	 (03CR) 10Legoktm: [C: 032] Remove LTA-Knowledgebase [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/374373 (owner: 10MarcoAurelio)
[18:30:31] <wikibugs_>	 10Cloud-VPS (Quota-requests), 10Discovery, 10Wikidata, 10Wikidata-Query-Service: WDQS testing setup platform sizing - https://phabricator.wikimedia.org/T169133#3559594 (10Smalyshev) Yes, the main problem is the diskspace. Though the performance is kinda an issue too since we can't keep up with full update...
[18:30:52] <wikibugs_>	 (03Merged) 10jenkins-bot: Remove LTA-Knowledgebase [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/374373 (owner: 10MarcoAurelio)
[18:31:01] <wikibugs_>	 (03CR) 10jenkins-bot: Remove LTA-Knowledgebase [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/374373 (owner: 10MarcoAurelio)
[18:31:21] <TheresNoTime>	 g'bye project :(
[18:31:35] <Zppix>	 ??
[18:31:43] <legoktm>	 Zppix: the underscore really bothers you that much?
[18:31:50] <Zppix>	 legoktm: it drives me nuts
[18:31:55] <legoktm>	 why?
[18:32:14] <Zppix>	 legoktm: i just have a thing against bot nicks with underscores
[18:32:59] <Zppix>	 Hence is why i (iirc) wrote up a thing to stop jouncebot from doing that in -operations
[18:34:33] <wikibugs_>	 !log tools.wikibugs Updated channels.yaml to: 33cb046399985b6fdd2a0210b8975e651811df02 Remove LTA-Knowledgebase
[18:35:10] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[18:35:44] <Zppix>	 Now thats clever
[18:36:09] <Zppix>	 Umm i dont think it quite worked though
[19:05:07] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0]
[19:27:16] <wikibugs_>	 10Data-Services, 10DBA: Design a method for keeping user-created tables in sync across labsDBs - https://phabricator.wikimedia.org/T156869#2988296 (10bd808) I think we should stop allowing write access entirely on the wiki replica servers by Cloud Services users. I know this will cause some currently unspecifi...
[19:28:14] <tomthirteen>	 I was wondering if people had ideas on two issues
[19:28:40] <tomthirteen>	 first, I'm trying to do analysis on langlinks from wikipedia
[19:29:08] <wikibugs_>	 10Data-Services, 10DBA: Design a method for keeping user-created tables in sync across labsDBs - https://phabricator.wikimedia.org/T156869#3559949 (10Halfak) As the person who filed this task, I agree 100% with @bd808's assessment.  Shall we close this "declined"?
[19:29:15] <tomthirteen>	 however, i have noticed that if the page is being redirected from an original page and the original page had language links
[19:29:33] <tomthirteen>	 the second page won't have statitics on those language links
[19:29:43] <tomthirteen>	 am i understanding this correctly?
[19:32:15] <Reedy>	 halfak: You should probably file a task to disable writes to slaves first though ;)
[19:33:02] <wikibugs_>	 (03CR) 10Lokal Profil: [C: 032] Rename -lang argument -langcode [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/373878 (https://phabricator.wikimedia.org/T166528) (owner: 10Jean-Frédéric)
[19:33:15] <wikibugs_>	 10Data-Services, 10DBA: Design a method for keeping user-created tables in sync across labsDBs - https://phabricator.wikimedia.org/T156869#2988296 (10chasemp) @halfak +1.  It was great to get this all on-task though as we will no doubt reference it in the future.
[19:33:43] <halfak>	 Reedy, wat
[19:34:00] <wikibugs_>	 (03Merged) 10jenkins-bot: Rename -lang argument -langcode [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/373878 (https://phabricator.wikimedia.org/T166528) (owner: 10Jean-Frédéric)
[19:34:09] <Reedy>	 halfak: If you're gonna close T156869... A task needs filing to make the change bd808 suggested
[19:34:09] <stashbot>	 T156869: Design a method for keeping user-created tables in sync across labsDBs - https://phabricator.wikimedia.org/T156869
[19:34:17] <Reedy>	 So you can't write to arbitary slaves
[19:34:34] <halfak>	 This is about the new replicas.  You already can't write to those :) 
[19:34:39] <bd808>	 I think that is already the case on the new boxes
[19:34:49] <wikibugs_>	 (03CR) 10jenkins-bot: Rename -lang argument -langcode [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/373878 (https://phabricator.wikimedia.org/T166528) (owner: 10Jean-Frédéric)
[19:35:04] <wikibugs_>	 10Cloud-Services, 10DBA: Prepare and check storage layer for electcomwiki - https://phabricator.wikimedia.org/T174385#3559994 (10Framawiki)
[19:35:15] <wikibugs_>	 10Cloud-Services, 10DBA: Prepare and check storage layer for electcomwiki - https://phabricator.wikimedia.org/T174385#3559994 (10Framawiki)
[19:35:29] <bd808>	 they are using different authn/z than the current cluster
[19:36:31] <bd808>	 tomthirteen: I think you are asking about pageviews data. If so you may get better help in the #wikimedia-analytics channel or the analytics mailing list
[19:36:58] <wikibugs_>	 (03CR) 10Lokal Profil: "Would it be an idea to dump the report to a log?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/373880 (owner: 10Jean-Frédéric)
[19:37:24] <wikibugs_>	 10Cloud-Services, 10DBA: Prepare and check storage layer for electcomwiki - https://phabricator.wikimedia.org/T174385#3559994 (10Reedy) Not sure a private task was specifically needed...  https://gerrit.wikimedia.org/r/374384 adds it to private_wikis
[19:37:36] <tomthirteen>	 No, it's the language links
[19:38:19] <tomthirteen>	 I can't get 100% accurate number of language links for pages when I use the wikipedia slave database
[19:38:33] <tomthirteen>	 and it seem it's because of the redirecting of pages
[19:38:47] <tomthirteen>	 at least that's what is seems like
[19:39:26] <Reedy>	 Aren't language links in wikidata anyway?
[19:40:15] <tomthirteen>	 but in the langlinks.sql files it's not always complete
[19:40:31] <tomthirteen>	 they sometimes don't give the accurate list of links
[19:41:05] <Reedy>	 You mean the database dumps?
[19:42:39] <tomthirteen>	 yes
[19:43:40] <Reedy>	 Does the api resolve the redirected language links? I'm guessing not
[19:44:35] <tomthirteen>	 Am I understanding this correctly? Is there a work around if anybody knows?
[19:45:35] <Reedy>	 Usually, update the source links to point to the actual target
[19:45:55] <Reedy>	 Or, search for redirects to that page, and use that in your search for langlinks against a page id
[19:46:25] <tomthirteen>	 I'm only getting info for the present page. 
[19:46:41] <tomthirteen>	 I can seem to find the data for the redirect
[19:48:06] <wikibugs_>	 10Cloud-Services, 10Beta-Cluster-Infrastructure, 10Wikimedia-General-or-Unknown, 10Patch-For-Review: rename -labs.php to -beta.php - https://phabricator.wikimedia.org/T150268#3560052 (10Reedy) 05Open>03stalled
[20:07:09] <wikibugs_>	 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3560132 (10RobH) On labstore1006 when loading the HP raid utility via bios, it gives the following error: error: no such device: EMBEDDED250.  It doe...
[20:10:25] <wikibugs_>	 (03PS1) 10Lokal Profil: Allow skipping some configs during full harvest. [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/374403 (https://phabricator.wikimedia.org/T174334)
[20:11:24] <wikibugs_>	 (03CR) 10jerkins-bot: [V: 04-1] Allow skipping some configs during full harvest. [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/374403 (https://phabricator.wikimedia.org/T174334) (owner: 10Lokal Profil)
[20:19:00] <wikibugs_>	 (03PS2) 10Lokal Profil: Allow skipping some configs during full harvest. [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/374403 (https://phabricator.wikimedia.org/T174334)
[20:19:56] <wikibugs_>	 (03CR) 10Lokal Profil: "Do we document allowed monuments_config variables or command line arguments anywhere?" [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/374403 (https://phabricator.wikimedia.org/T174334) (owner: 10Lokal Profil)
[20:23:10] <wikibugs_>	 10Data-Services, 10Toolforge, 10Documentation: Document labsdb replication set up - https://phabricator.wikimedia.org/T85868#3560203 (10bd808)
[20:30:40] <wikibugs_>	 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3560242 (10Cmjohnson) I started a ticket with HP.  Your case was successfully submitted. Please note your Case ID: 5322481808 for future reference.
[20:31:37] <wikibugs_>	 10Striker, 10cloud-services-team (Kanban), 10Patch-For-Review, 10User-Zppix, 10User-bd808: Fatal error when SSH key data is invalid - https://phabricator.wikimedia.org/T174112#3560246 (10bd808) 05Open>03Resolved The UI in #striker will now mark keys that it can't read as "invalid" and allow the user...
[20:32:12] <wikibugs_>	 10Striker, 10cloud-services-team (Kanban), 10Patch-For-Review, 10User-Zppix, 10User-bd808: Fatal error when SSH key data is invalid - https://phabricator.wikimedia.org/T174112#3560248 (10Zppix) @bd808 sweet thanks for fixing this so quickly!!
[20:36:07] <wikibugs_>	 10Striker, 10cloud-services-team (Kanban), 10Patch-For-Review, 10User-bd808: Unable to add new LDAP user as maintainer of a tool - https://phabricator.wikimedia.org/T173963#3560255 (10bd808) 05Open>03Resolved
[20:36:21] <wikibugs_>	 10Cloud-Services, 10DBA: Prepare and check storage layer for electcomwiki - https://phabricator.wikimedia.org/T174385#3560260 (10Framawiki) >>! In T174385#3560023, @Reedy wrote: > Not sure a private task was specifically needed... >  > https://gerrit.wikimedia.org/r/374384 adds it to private_wikis  Task create...
[20:36:33] <wikibugs_>	 10Striker, 10cloud-services-team (Kanban), 10User-bd808: GID out of range warning while creating a new tool account - https://phabricator.wikimedia.org/T173958#3560261 (10bd808) 05Open>03Resolved
[20:37:48] <wikibugs_>	 10Striker, 10cloud-services-team (Kanban), 10Epic, 10Patch-For-Review, 10User-bd808: Manage shared tool accounts via Striker - https://phabricator.wikimedia.org/T149458#3560265 (10bd808) Code is live. Remaining work is to document the functionality and promote it on wikitech + tech blog/mailing list.
[20:39:38] <wikibugs_>	 10cloud-services-team (Kanban), 10DC-Ops, 10Operations, 10ops-eqiad: labvirt1015 crashes - https://phabricator.wikimedia.org/T171473#3560268 (10Cmjohnson) So far the error at least in the h/w log on the server has not returned.....keeping this open to monitor.
[20:44:05] <wikibugs_>	 10Cloud-Services, 10DBA: Prepare and check storage layer for electcomwiki - https://phabricator.wikimedia.org/T174385#3560299 (10Reedy) Bleugh. "private" should've been "subtask"  There's less to do for a private wiki, but just needs doing first
[20:48:54] <wikibugs_>	 10Tool-Zppixbot, 10User-Zppix: Fix adminlist module - https://phabricator.wikimedia.org/T174395#3560312 (10Zppix)
[20:49:13] <wikibugs_>	 10Tool-Zppixbot, 10User-Zppix: Fix adminlist module - https://phabricator.wikimedia.org/T174395#3560325 (10Zppix) p:05Triage>03High
[21:12:04] <wikibugs_>	 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3560381 (10madhuvishy) @Cmjohnson Thank you!
[21:15:05] <hashar>	 andrewbogott: in case you are around today. I found yet another intel paper about tunning openstack https://01.org/sites/default/files/performance_analysis_and_tuning_in_china_mobiles_openstack_production_cloud_2.pdf :)
[21:15:33] <hashar>	 found that paper while looking for the parameter  rpc_conn_pool_size
[21:16:52] <tabbycat>	 bd808:  User account Striker test 2017-08-28 <-- this was you, right?
[21:16:59] <bd808>	 tabbycat: yes
[21:17:09] <tabbycat>	 blocks for sockpuppetry :P
[21:17:15] <bd808>	 +1
[21:17:20] <tabbycat>	 :)
[21:17:35] <bd808>	 the account isn't attached on wikitech and I already 'lost' the password
[21:18:13] <Zppix>	 bd808: found it here you go xD
[21:18:23] <Zppix>	 It was in the trash :P
[21:18:24] <tabbycat>	 you can do a password reset or change its credentials manually via terbium :)
[21:18:38] <tabbycat>	 in case you really need to recover it
[21:18:59] <tabbycat>	 there's a script for that IIRC
[21:19:05] <bd808>	 yeah. its trivial to get back
[21:19:48] <bd808>	 I have email recovery easily too
[21:34:02] <wikibugs_>	 10Wikibugs, 10Africa-Wikimedia-Developers, 10MediaWiki-extensions-SendGrid, 10Patch-For-Review: Integrating SendGrid extension work board to wikibugs - https://phabricator.wikimedia.org/T174260#3560465 (10D3r1ck01) @Legoktm, this is waiting for you :).
[22:59:38] <wikibugs_>	 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3560733 (10RobH) I also checked other systems iwth HP raid controllers.  So these showing the raid1 as sdd is a problem, it should show as sda.  Othe...
[23:05:30] <tomthirteen>	 hi has anyone had experience loading langlinks.sql to heidisql?
[23:05:49] <tomthirteen>	 I'm trying to load the arabic sql and it gets through most of it 
[23:06:58] <tomthirteen>	 then I get "end of file block was cut within some multibyte characheter,
[23:07:28] <tomthirteen>	 at position x. increasing chunk size and retry
[23:07:39] <tomthirteen>	 but it just keeps retrying to no avail
[23:07:55] <tomthirteen>	 I don't know how to increase the chunk size
[23:54:39] <wikibugs_>	 10Cloud-Services, 10cloud-services-team (Kanban), 10Wikimedia-Mailing-lists: Rename labs-admin mailing list to cloud-admin - https://phabricator.wikimedia.org/T167155#3560843 (10bd808) @RobH, you certainly have more experience with this sort of thing than I do. If creating new lists and archiving the old lis...