[00:27:43] !log tools Updated misctools to 1.24 [00:27:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [00:42:52] 10Tools: SVGCheck tool unresponsive - https://phabricator.wikimedia.org/T177125#3663199 (10bd808) According to https://tools.wmflabs.org/admin/tool/svgcheck, @Jarry1250 is the maintainer of the svgcheck tool. [00:47:14] 10Tools: Make table at tool-extreg-wos sortable if possible - https://phabricator.wikimedia.org/T176873#3663202 (10bd808) According to https://tools.wmflabs.org/admin/tool/extreg-wos, @Legoktm and @Reedy are the maintainers of this tool. [00:50:42] 10Tools: Make table at tool-extreg-wos sortable if possible - https://phabricator.wikimedia.org/T176873#3663206 (10Reedy) OOI, why? [00:57:58] 10Tools, 10Wiki-Loves-Monuments: Redirect toolserver.org/~erfgoed/stream/ - https://phabricator.wikimedia.org/T175671#3599469 (10bd808) The redirects are setup in https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/toolserver_legacy/templates/www.toolserver.org.erb [03:17:42] 10Toolforge: Tool naming convention - https://phabricator.wikimedia.org/T177439#3663282 (10Sowjanyavemuri) Modified the tutorial to reflect the above changes. [03:22:54] 10cloud-services-team, 10DBA: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3663283 (10MusikAnimal) [03:23:02] 10cloud-services-team, 10DBA, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3663297 (10MusikAnimal) [03:33:19] 10Tools: Migrate to Tool Labs: https://toolserver.org/~vvv/adminstats.php - https://phabricator.wikimedia.org/T63030#677537 (10Liuxinyu970226) This is nowadays https://xtools.wmflabs.org/adminstats, close? [03:35:05] 10cloud-services-team, 10DBA, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3663307 (10MusikAnimal) Or perhaps that's just it... the connection limit was higher on `tools.labsdb` where the temporary tables w... [03:47:17] 10Toolforge, 10Documentation: Document Tool naming contraints - https://phabricator.wikimedia.org/T177439#3663331 (10bd808) [03:47:43] 10Toolforge, 10Documentation: Document Tool naming contraints - https://phabricator.wikimedia.org/T177439#3658679 (10bd808) 05Open>03Resolved Thanks @Sowjanyavemuri! [03:47:45] 10Toolforge, 10Outreachy (Round-15), 10Patch-For-Review: Outreachy - webservice microtask for Sowjanyavemuri - https://phabricator.wikimedia.org/T176624#3663335 (10bd808) [03:56:37] 10Data-Services, 10cloud-services-team, 10DBA, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3663345 (10bd808) [04:01:11] !log tools.wikibugs Updated channels.yaml to: b951ef6a2c714b1e9a316cbfe373d78e00d74c30 Add /.tox to .gitignore [04:02:44] bd808: Hm.. the screenshot on that Create new tool wikitech page shows underscores, which are invalid? [04:02:57] yeah :/ [04:03:10] k, just making sure I didn't miss understand. [04:03:14] we did not know that until later [04:03:40] kubernetes is really picky about namespace names and the tool name is used as that [04:08:18] !log tools.wikibugs Updated channels.yaml to: b951ef6a2c714b1e9a316cbfe373d78e00d74c30 Add /.tox to .gitignore [04:10:56] stashbot: you there? [04:10:56] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [04:11:24] hmmm... I wonder if I told stashbot to ignore wikibugs so that it didn't spam phab links... [04:12:33] yup... darn [04:13:12] legoktm: apparently I configured stashbot such that your cool announce deploys trick doesn't work [04:13:28] oh no :( [04:15:27] I can change the code so that it only ignores for doing the phab expansion stuff [04:15:46] that would be nice [04:15:47] that was really the point of the ignore list [04:17:36] * legoktm afk -> dinner [04:25:05] (03PS1) 10BryanDavis: Only ignore phab mentions [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/382642 [04:28:52] (03CR) 10BryanDavis: [C: 032] Only ignore phab mentions [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/382642 (owner: 10BryanDavis) [04:29:48] (03Merged) 10jenkins-bot: Only ignore phab mentions [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/382642 (owner: 10BryanDavis) [04:34:10] !log tools.stashbot Updated to 2cd2a3f (Only ignore phab mentions) [04:34:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [05:56:22] 10Cloud-Services, 10cloud-services-team, 10DBA: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3663430 (10Marostegui) Sanitarium host was sanitized and check_private_data reported no issues. So this is ready for #cloud-services-team to do their magic and resolve t... [06:00:17] 10Cloud-Services, 10Toolforge, 10MediaWiki-extensions-WikimediaMaintenance: Make maintance script for sending annual survey emails - https://phabricator.wikimedia.org/T148783#2732629 (10Tgr) Well, you'd have to opt out from all mass messages, which seems a bit overboard (although in general the ability to se... [06:22:42] 10Data-Services, 10cloud-services-team, 10DBA, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3663283 (10Marostegui) This user has no connection limits on the old servers (I guess for historic reasons?). Ri... [06:25:38] 10Cloud-Services, 10Toolforge, 10DBA: What would be the preferred way to run a series of read queries on DB replica, each on two *_p databases ? - https://phabricator.wikimedia.org/T124805#3663454 (10Marostegui) 05Open>03Resolved Going to mark this as resolved for now as it is an old one and as Jaime sai... [07:48:15] 10Toolforge, 10cloud-services-team (Kanban), 10User-bd808: tools-mail queue length alert from prometheus cron - https://phabricator.wikimedia.org/T177484#3663595 (10fgiunchedi) @chasemp I bet that's a side effect of {T166561}, is it persisting or has been transient during package upgrades? [08:10:38] Hi, On trying to launch/re-launch jobs they are failing with https://tools.wmflabs.org/paste/view/f91ff0cc could someone give it a poke. This happens with or without specifying the memory size. [08:29:26] 10Cloud-Services, 10Wikidata, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Sprint: Open view for term_full_entity_id in wb_terms table in labs - https://phabricator.wikimedia.org/T167114#3663723 (10Ladsgroup) 05stalled>03Open [13:18:37] 10Tools: Make table at tool-extreg-wos sortable if possible - https://phabricator.wikimedia.org/T176873#3664318 (10MarcoAurelio) >>! In T176873#3663206, @Reedy wrote: > OOI, why? I feel it makes easier to search for pending work if we could sort the table, instead to browsing up and down. If it is possible with... [13:26:11] 10Cloud-Services, 10Wikidata, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Sprint: Open view for term_full_entity_id in wb_terms table in labs - https://phabricator.wikimedia.org/T167114#3664340 (10Ladsgroup) It's blocking our work [13:56:00] 10Toolforge, 10Outreachy (Round-15), 10Patch-For-Review: Outreachy - webservice microtask for Sowjanyavemuri - https://phabricator.wikimedia.org/T176624#3664404 (10Sowjanyavemuri) Hi, @Andrew @bd808 @madhuvishy I have made few changes to the microtask patch(https://gerrit.wikimedia.org/r/#/c/380568/) adding... [14:10:25] 10Toolforge, 10cloud-services-team (Kanban), 10User-bd808: tools-mail queue length alert from prometheus cron - https://phabricator.wikimedia.org/T177484#3664428 (10chasemp) 05Open>03Resolved >>! In T177484#3663595, @fgiunchedi wrote: > @chasemp I bet that's a side effect of {T166561}, is it persisting o... [14:12:20] 10Data-Services, 10cloud-services-team, 10DBA, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3664436 (10chasemp) p:05Triage>03Normal We did profile things and put in per user limits on the new setup fo... [14:17:18] legoktm: https://gerrit.wikimedia.org/r/#/c/372213/ seems to have been caught by a new linting thing for module/profile/role idioms...it did -1 it. I was trying to push that out, sorry it sat for so long post fixes. [14:24:26] 10Data-Services, 10cloud-services-team, 10DBA, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3663283 (10jcrespo) Note *.analytics.db.svc.eqiad.wmflabs hosts are supposed to be for very long-running connect... [14:29:01] 10Cloud-Services, 10Community-Liaisons (Jul-Sep 2017), 10Design: Update Cloud Services logo+wordmark - https://phabricator.wikimedia.org/T174094#3664540 (10Qgil) [14:29:47] 10Tool-Attribution-Generator, 10Community-Liaisons, 10Structured-Data-Commons, 10TCB-Team, 10Wikidata: Attribution Generator and Structured Data on Commons discussion - https://phabricator.wikimedia.org/T173341#3664543 (10Qgil) [14:48:33] 10Cloud-VPS, 10Operations, 10Patch-For-Review: Ferm rules for labstore NFS hosts - https://phabricator.wikimedia.org/T165136#3664623 (10chasemp) p:05Triage>03Normal [15:11:34] Damianz: that looks like Python2 code being run with the Python3 interpreter [15:12:09] oh... crap. I thought I fixed that yesterday [15:16:56] 10Toolforge: jstart - TypeError: memoryview: str object does not have the buffer interface - https://phabricator.wikimedia.org/T177614#3664696 (10JJMC89) [15:20:59] (03PS1) 10BryanDavis: jstart: Convert stdin from string to bytes [labs/toollabs] - 10https://gerrit.wikimedia.org/r/382726 [15:21:41] (03PS2) 10BryanDavis: jstart: Convert stdin from string to bytes [labs/toollabs] - 10https://gerrit.wikimedia.org/r/382726 (https://phabricator.wikimedia.org/T177614) [15:22:40] (03CR) 10BryanDavis: [V: 031 C: 032] jstart: Convert stdin from string to bytes [labs/toollabs] - 10https://gerrit.wikimedia.org/r/382726 (https://phabricator.wikimedia.org/T177614) (owner: 10BryanDavis) [15:23:54] 10Toolforge, 10cloud-services-team (Kanban), 10Patch-For-Review, 10User-bd808: jstart - TypeError: memoryview: str object does not have the buffer interface - https://phabricator.wikimedia.org/T177614#3664713 (10bd808) p:05Triage>03Unbreak! a:03bd808 [15:24:07] (03Merged) 10jenkins-bot: jstart: Convert stdin from string to bytes [labs/toollabs] - 10https://gerrit.wikimedia.org/r/382726 (https://phabricator.wikimedia.org/T177614) (owner: 10BryanDavis) [15:25:57] (03PS1) 10BryanDavis: Debian version bump [labs/toollabs] - 10https://gerrit.wikimedia.org/r/382727 [15:26:17] I always forget that part :/ [15:26:28] (03CR) 10BryanDavis: [V: 031 C: 032] Debian version bump [labs/toollabs] - 10https://gerrit.wikimedia.org/r/382727 (owner: 10BryanDavis) [15:28:40] (03PS2) 10BryanDavis: Debian version bump [labs/toollabs] - 10https://gerrit.wikimedia.org/r/382727 [15:29:14] (03CR) 10BryanDavis: [V: 032 C: 032] Debian version bump [labs/toollabs] - 10https://gerrit.wikimedia.org/r/382727 (owner: 10BryanDavis) [15:30:33] (03Merged) 10jenkins-bot: Debian version bump [labs/toollabs] - 10https://gerrit.wikimedia.org/r/382727 (owner: 10BryanDavis) [15:33:39] !log tools Upgrade jobutils to 1.25 (T177614) [15:33:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:33:46] T177614: jstart - TypeError: memoryview: str object does not have the buffer interface - https://phabricator.wikimedia.org/T177614 [15:35:38] 10Toolforge, 10cloud-services-team (Kanban), 10Patch-For-Review, 10User-bd808: jstart - TypeError: memoryview: str object does not have the buffer interface - https://phabricator.wikimedia.org/T177614#3664781 (10bd808) This was caused by {rLTOL6a65130c50d49c8f4ef25f8154b745dcb6c09b7c}. I neglected to test... [15:36:05] 10Toolforge, 10cloud-services-team (Kanban), 10Patch-For-Review, 10User-bd808: jstart - TypeError: memoryview: str object does not have the buffer interface - https://phabricator.wikimedia.org/T177614#3664782 (10bd808) 05Open>03Resolved [15:40:57] Thanks for the quick response on T177614 bd808 [15:40:57] T177614: jstart - TypeError: memoryview: str object does not have the buffer interface - https://phabricator.wikimedia.org/T177614 [15:43:18] JJMC89|work: you filed it at the same time I was putting the patch into gerrit based on Damianz's report in backscroll here. Lucky timing. [15:43:37] I just sent an email to cloud@ about the problem. [15:52:20] 10Cloud-VPS (Project-requests): Request creation of reading-demos VPS project - https://phabricator.wikimedia.org/T177616#3664794 (10bmansurov) [15:59:08] bd808: fyi I saw this hit [15:59:09] RECOVERY - Start a job and verify on Trusty on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.401 second response time [15:59:20] so I wonder if jsub being broken didn't break that and we didn't notice [16:00:19] chasemp: :/ probably. I have to admit to not paying any attention to icinga messages [16:00:42] it's super super easy to miss in -ops [16:00:54] I thinking I"ll ping myself on checker.tools.wmflabs.org [16:01:07] but I rally really want to create a notification group for our stuff in icinga [16:01:22] that does a few things for those types of alerts besides irc only [16:01:23] and only -ops [16:01:28] that's such a busy channel etc [16:16:14] 10Cloud-VPS (Project-requests): Request creation of reading-demos VPS project - https://phabricator.wikimedia.org/T177616#3664917 (10Jdlrobson) [16:20:05] 10Cloud-Services, 10Toolforge, 10MediaWiki-extensions-WikimediaMaintenance: Make maintance script for sending annual survey emails - https://phabricator.wikimedia.org/T148783#3664943 (10bd808) >>! In T148783#3663433, @Tgr wrote: > Why not use the Newsletter extension for this? Seems like the exact use case i... [16:26:07] 10Data-Services, 10cloud-services-team, 10DBA, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3664959 (10MusikAnimal) >>! In T177570#3664528, @jcrespo wrote: > Note *.analytics.db.svc.eqiad.wmflabs hosts ar... [16:26:33] 10PAWS, 10Toolforge: PAWS Kubernetes cluster forgot how to forward packets between nodes - https://phabricator.wikimedia.org/T177363#3664978 (10bd808) [16:54:50] 10Cloud-VPS (Project-requests): Request creation of reading-demos VPS project - https://phabricator.wikimedia.org/T177616#3664794 (10bd808) The reading-web-staging project contains 8 virtual machines, so I'm a bit skeptical about the "our current (and only) instance" claim. Would this project remove the need for... [17:02:18] 10Cloud-VPS (Project-requests): Request creation of MWStake VPS project - https://phabricator.wikimedia.org/T177500#3660874 (10bd808) Is the intent of this project to host MWStake specific tools or are you aiming for something different in scope? The purpose statement sounds more oriented to either SemanticMedia... [17:05:22] 10Cloud-VPS (Quota-requests): Increase floating-ip quota for 'cvn' project (trusty to debian migration) - https://phabricator.wikimedia.org/T177299#3665161 (10bd808) p:05Triage>03Normal [17:05:41] 10Cloud-VPS (Quota-requests): Temporary increase of floating-ip quota for 'cvn' project (trusty to debian migration) - https://phabricator.wikimedia.org/T177299#3654178 (10bd808) [17:09:11] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Documentation: Document policy on Cloud VPS project scope - https://phabricator.wikimedia.org/T177632#3665179 (10bd808) [17:58:44] 10Toolforge, 10Outreachy (Round-15): Improvements for the Toolforge 'webservice' command - https://phabricator.wikimedia.org/T175768#3665399 (10bd808) [17:58:50] 10Toolforge, 10Outreachy (Round-15), 10Patch-For-Review: Outreachy - webservice microtask for Sowjanyavemuri - https://phabricator.wikimedia.org/T176624#3665395 (10bd808) 05Open>03Resolved @Sowjanyavemuri Your gerrit patch looks good. I think we can call this microtask done. Your next step is to write up... [18:21:14] 10Toolforge, 10cloud-services-team (Kanban), 10puppet-compiler: toolsbeta: set up puppet-compiler / temporary-apply - https://phabricator.wikimedia.org/T97081#1232397 (10Andrew) I'm trying to reproduce the tools puppet compiler described here. A few things have clearly changed since this was last built... T... [19:02:42] chasemp: ok...I have no idea how to fix those errors or what they mean :/ [19:03:53] my strategy for puppet is find something similar to what I need and copy/paste as necessary [19:04:49] legoktm: ok, I'll take a look at what it would mean to comply. [19:07:04] thanks [19:39:46] 10Toolforge, 10Outreachy (Round-15), 10Patch-For-Review: Outreachy - webservice microtask for Sowjanyavemuri - https://phabricator.wikimedia.org/T176624#3665839 (10Sowjanyavemuri) @bd808 Thanks for reviewing and verifying the patch:) Should I create a new task for the above steps describing the proposal on l... [19:49:01] 10Toolforge, 10Outreachy (Round-15), 10Patch-For-Review: Outreachy - webservice microtask for Sowjanyavemuri - https://phabricator.wikimedia.org/T176624#3665911 (10bd808) >>! In T176624#3665839, @Sowjanyavemuri wrote: > @bd808 Thanks for reviewing and verifying the patch:) > Should I create a new task for th... [19:52:40] 10Toolforge, 10Outreachy (Round-15), 10Patch-For-Review: Outreachy - webservice microtask for Sowjanyavemuri - https://phabricator.wikimedia.org/T176624#3665934 (10Sowjanyavemuri) @bd808 Sure will work on that. Thanks! [20:10:24] hi raynor. So you can login to bastion but not to a sprcific project instance? [20:10:28] *specific [20:10:50] yeah [20:11:06] do we need to setup permissions for specific instances? [20:11:10] let me get a shell on chromium-pdf.reading-web-staging.eqiad.wmflabs and check out the auth logs [20:11:12] yes, I can login into bastion [20:11:26] can you log into other reading-web-staging vms? [20:11:57] I tried chromium-pdf and reading-web-staging, both without success [20:13:35] raynor: what is your Wikitech user name? [20:13:55] Pmiazga [20:13:58] and shell name is pmiazga [20:14:16] I checked the ssh key in wikitech, it's correct [20:14:50] ok. I see you as a member of the reading-web-staging project [20:14:57] * bd808 greps auth log [20:15:53] I just tried many times with slight config changes and both keys (I have a prod ssh key, and dev ssh key) [20:16:01] hmmm.. I don't see auth attempts on chromium-pdf.reading-web-staging.eqiad.wmflabs with your username, so things may be breaking down before you get there [20:17:26] Ok. now I'm seeing "Failed publickey for pmiazga" in the logs [20:17:43] for the key "RSA 3c:b7:df:7d:eb:06:30:82:d3:88:df:55:6e:0b:01:4b" [20:18:30] ok, sec (now i tried pasting the full url instead of host name from the ssh config) [20:18:38] and it used the prod key, one sec [20:19:17] raynor: that used "RSA 3c:b7:df:7d:eb:06:30:82:d3:88:df:55:6e:0b:01:4b" again [20:19:33] "Accepted publickey for pmiazga" \o/ [20:19:35] ok, now it works, sorry to bother you [20:19:41] no worries [20:19:49] I had to mess up my ssh config ;/ [20:19:54] if I go with ssh pmiazga@chromium-pdf.reading-web-staging.eqiad.wmflabs it works [20:20:11] but by short `ssh chromium-pdf` it doesn't I'll fix it later. Thanks [20:21:17] I would only expect the latter to work if you have a "Host chromium-pdf" section in your ~/.ssh/config that gives the full hostname [20:22:31] You may want to read over https://wikitech.wikimedia.org/wiki/Managing_multiple_SSH_agents as well and split your prod and Cloud VPS keys into different agents [20:31:27] bd808: thanks for the link, that looks interesting. I'll read it later [20:40:55] 10Cloud-VPS, 10cloud-services-team (Kanban): Create banner image for Wikimedia Cloud VPS - https://phabricator.wikimedia.org/T177442#3666111 (10Quiddity) SVG and PNG: {F10039979} {F10039981} [21:30:16] 10Toolforge: Raise tool memory limit - similarity - https://phabricator.wikimedia.org/T176527#3666247 (10Surlycyborg) 05Open>03Invalid I'm most likely going to throw away this prototype, so there's no point pursuing this. Thank you all for your input! For the curious, https://lists.wikimedia.org/pipermail/c... [21:47:22] could you add me to the project "wikidata-dev" please? How should i request it? on wiki? [21:47:36] can i still add myself like back in wikitech before horizon? [21:54:00] scratch that, nothing needed. i got it :) [21:58:30] !log wikidata-dev added myself to project and projectadmins to work on T99531 [21:58:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [21:58:33] T99531: [Task] move wikiba.se webhosting to wikimedia misc-cluster - https://phabricator.wikimedia.org/T99531 [22:13:51] !log wikidata-dev creating new instance wikibase-stretch (doing stretch right away, existing one was jessie, but oldstable) [22:13:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [22:33:27] !log wikidata-dev added "Prefix Puppet" wikibase - added role::wikibase to instance wikibase-stretch (TT99531) [22:33:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [23:06:56] !log rcm Neon: running update [23:06:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL