[00:21:17] 10Cloud-Services, 10Cloud-VPS, 10DBA: Database upgrade MariaDB 10: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127#731912 (10kaldari) FYI, the lack of log indexes on some wikis is causing issues for XTools. See, for example, T171264#3468493. [00:22:13] 10Cloud-Services, 10Cloud-VPS, 10DBA: Database upgrade MariaDB 10: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127#3468673 (10kaldari) p:05Low>03Normal [00:40:08] 10Data-Services, 10cloud-services-team (Kanban), 10Operations, 10ops-eqiad, 10Patch-For-Review: Degraded RAID on labsdb1001 - https://phabricator.wikimedia.org/T171538#3468270 (10faidon) >>! In T171538#3468451, @chasemp wrote: > hmm > > ```# cat /proc/mdstat > Personalities : > unused devices: ```... [00:59:09] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [01:18:33] (03CR) 10Legoktm: [C: 032] De-reference symlinks in tarball [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/366414 (https://phabricator.wikimedia.org/T135194) (owner: 10Legoktm) [01:18:53] (03Merged) 10jenkins-bot: De-reference symlinks in tarball [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/366414 (https://phabricator.wikimedia.org/T135194) (owner: 10Legoktm) [01:33:36] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [01:35:39] 10Cloud-Services, 10Cloud-VPS, 10DBA: Database upgrade MariaDB 10: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127#3468760 (10MusikAnimal) Yeah, the `log_title_time` index seems especially helpful when doing things like looking up a block log. You could use `l... [02:18:13] PROBLEM - Puppet errors on tools-worker-1027 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:53:11] RECOVERY - Puppet errors on tools-worker-1027 is OK: OK: Less than 1.00% above the threshold [0.0] [03:51:56] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [04:06:55] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [04:51:26] 10Data-Services, 10cloud-services-team (Kanban), 10Operations, 10ops-eqiad, 10Patch-For-Review: Degraded RAID on labsdb1001 - https://phabricator.wikimedia.org/T171538#3468270 (10Marostegui) The disk failed is part of a HW RAID10: ``` # megacli -LDPDInfo -aAll Adapter #0 Number of Virtual Disks: 1 Virt... [05:13:29] 10Data-Services, 10DBA: Database upgrade MariaDB 10: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127#3468943 (10bd808) [07:08:56] (03CR) 10Lokal Profil: [C: 031] "Should we merge this into the wikidata branch? Is there anything missing in this patch (that cannot be handled in a later patch)?" [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/354961 (https://phabricator.wikimedia.org/T165988) (owner: 10Jean-Frédéric) [09:31:47] 10Tools, 10Commons: Zoomviewer is down - https://phabricator.wikimedia.org/T169864#3411054 (10zhuyifei1999) 05Open>03Resolved a:03dschwen [09:39:54] 10wikitech.wikimedia.org, 10DBA: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3469274 (10MarcoAurelio) p:05Triage>03Lowest [09:43:15] 10wikitech.wikimedia.org, 10DBA: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3469261 (10Marostegui) I guess you are aware that rename a database isn't something supported on MySQL/MariaDB at the moment. While you can rename a table, a database rename isn't supported yet. [09:44:18] (03CR) 10Jean-Frédéric: [C: 032] Proof of concept to harvest Wikidata into monuments database [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/354961 (https://phabricator.wikimedia.org/T165988) (owner: 10Jean-Frédéric) [09:45:24] (03Merged) 10jenkins-bot: Proof of concept to harvest Wikidata into monuments database [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/354961 (https://phabricator.wikimedia.org/T165988) (owner: 10Jean-Frédéric) [09:46:55] (03CR) 10jenkins-bot: Proof of concept to harvest Wikidata into monuments database [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/354961 (https://phabricator.wikimedia.org/T165988) (owner: 10Jean-Frédéric) [09:57:53] 10wikitech.wikimedia.org, 10DBA: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3469348 (10MarcoAurelio) 05Open>03stalled Forgot to set it to stalled. Yes, I was not given the exact reason, but I was told that it was not currently possible. Still this is a valid task that... [09:59:00] 10wikitech.wikimedia.org, 10DBA: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3469353 (10Marostegui) There are ways to rename a database, but not using a mysql native command, but exporting and importing into a new database with the correct name etc. It requires downtime, te... [10:03:05] 10cloud-services-team, 10wikitech.wikimedia.org: contentadmin has suddenly less permissions - https://phabricator.wikimedia.org/T171208#3469359 (10MarcoAurelio) According to [[ https://wikitech.wikimedia.org/wiki/Special:ComparePages?page1=User%3AMarcoAurelio%2FSandbox&rev1=1766077&page2=User%3AMarcoAurelio%2F... [10:06:42] (03CR) 10Jean-Frédéric: "Agree, good to go. thanks for pushing this through the finish line!" [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/354961 (https://phabricator.wikimedia.org/T165988) (owner: 10Jean-Frédéric) [10:08:51] 10wikitech.wikimedia.org, 10DBA: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3469378 (10MarcoAurelio) Doesn't look urgent enough to me to duplicate labswiki and paste it to a new wikitech DB, etc. However I leave that decision to the #cloud-services-team. It was also sugges... [10:09:49] 10Cloud-Services, 10wikitech.wikimedia.org, 10Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#3145305 (10MarcoAurelio) Would/Is T171570 a blocker for this one? [10:10:25] 10cloud-services-team, 10wikitech.wikimedia.org, 10DBA: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3469387 (10Marostegui) [11:43:09] rm: cannot remove ‘/data/project/maurelio/mwdev’: Is a directory [11:43:14] so... [11:44:11] rm -rf did it [12:32:55] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:35:30] 10cloud-services-team, 10DC-Ops, 10Operations, 10ops-eqiad: labvirt1015 crashes - https://phabricator.wikimedia.org/T171473#3469779 (10MoritzMuehlenhoff) a:03Cmjohnson [12:40:49] 10Cloud-VPS, 10Operations, 10Puppet: Move the main WMCS puppetmaster into the Labs realm - https://phabricator.wikimedia.org/T171188#3469789 (10MoritzMuehlenhoff) p:05Triage>03Normal [12:47:46] 10Cloud-VPS, 10Operations, 10monitoring, 10User-fgiunchedi: Diamond collectors collects NFS statistics on Cloud-VPS - https://phabricator.wikimedia.org/T171583#3469801 (10zhuyifei1999) [12:48:03] 10Cloud-VPS, 10Operations, 10monitoring, 10User-fgiunchedi: Diamond collectors collects NFS statistics on Cloud-VPS - https://phabricator.wikimedia.org/T171583#3469801 (10zhuyifei1999) [13:07:55] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [13:08:50] 10Cloud-VPS, 10Operations, 10monitoring, 10User-fgiunchedi: Diamond collectors collects NFS statistics on Cloud-VPS - https://phabricator.wikimedia.org/T171583#3469861 (10zhuyifei1999) Workaround (on render side): set a threshold of 1TB `aliasByNode(maximumBelow(maximumAbove(HOST.diskspace.*.byte_avail,0),... [13:20:23] 10Cloud-VPS, 10Operations, 10monitoring, 10User-fgiunchedi: Diamond collectors collects NFS statistics on Cloud-VPS - https://phabricator.wikimedia.org/T171583#3469876 (10zhuyifei1999) Created [[https://github.com/wikimedia/nagf/pull/16|PR to nagf]]. [14:04:08] 10Cloud-VPS, 10Operations, 10monitoring, 10User-fgiunchedi: Diamond collectors collects NFS statistics on Cloud-VPS - https://phabricator.wikimedia.org/T171583#3470079 (10zhuyifei1999) This is probably caused by [[https://github.com/python-diamond/Diamond/commit/1d85b93defecc338cbf0e9a7dee371204c8311dd|1d8... [14:14:36] 10Cloud-Services: Requesting access to RESOURCE for fajr18 - https://phabricator.wikimedia.org/T171591#3470166 (10Reedy) [14:18:31] 10Cloud-Services, 10Tracking, 10User-bd808: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#3470180 (10Andrew) [14:18:33] 10Cloud-VPS (Quota-requests), 10ORES, 10Scoring-platform-team, 10User-bd808: Request increase quota for ores-staging to 52GB RAM - https://phabricator.wikimedia.org/T169811#3470177 (10Andrew) 05Open>03Resolved a:03Andrew [14:24:36] 10Toolforge: Requesting access to RESOURCE for fajr18 - https://phabricator.wikimedia.org/T171591#3470213 (10chasemp) p:05Triage>03Normal Hi @Fajr18 I notice this account already exists in LDAP: > uid=17566(fajr18) gid=500(wikidev) groups=500(wikidev) Did you request access to the Tools project here https... [14:25:08] 10Toolforge: Requesting access to Toolforge for fajr18 - https://phabricator.wikimedia.org/T171591#3470218 (10chasemp) [14:29:11] bd808: any luck so far? :) [14:35:05] 10Toolforge: Requesting access to Toolforge for fajr18 - https://phabricator.wikimedia.org/T171591#3470096 (10MarcoAurelio) @chasemp I've found https://toolsadmin.wikimedia.org/tools/membership/status/51 and IIRC I saw this account registering yesterday on Wikitech. Maybe this is a duplicate of the toolsadmin re... [14:39:48] 10Toolforge: Requesting access to Toolforge for fajr18 - https://phabricator.wikimedia.org/T171591#3470263 (10Fajr18) Hi @Reedy seems i requested in wrong place : I need shell accsess. [14:41:41] 10Toolforge: Requesting access to Toolforge for fajr18 - https://phabricator.wikimedia.org/T171591#3470280 (10MarcoAurelio) a:05Fajr18>03None [14:49:16] PROBLEM - Puppet errors on tools-exec-1402 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:55:45] Cyberpower678: I made some progress but not as much as I had hoped. I should have more time to work on it after my morning meetings today. [14:55:50] 10Toolforge: Requesting access to Toolforge for fajr18 - https://phabricator.wikimedia.org/T171591#3470378 (10chasemp) 05Open>03Resolved a:03chasemp >>! In T171591#3470240, @MarcoAurelio wrote: > @chasemp I've found https://toolsadmin.wikimedia.org/tools/membership/status/51 and IIRC I saw this account reg... [14:56:00] bd808: :-) Thanks [15:10:53] 10cloud-services-team, 10wikitech.wikimedia.org, 10DBA: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3470481 (10bd808) I suppose it could happen as part of moving the database tables from hosting on silver.wikimedia.org to hosting in the s3 cluster ({T167973}), but honestl... [15:13:02] 10Cloud-Services, 10wikitech.wikimedia.org, 10Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#3470498 (10bd808) >>! In T161859#3469379, @MarcoAurelio wrote: > Would/Is T171570 a blocker for this one? {T171570} seems like a large potential for breaking things with a very small ove... [15:13:58] 10Cloud-Services, 10wikitech.wikimedia.org, 10Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#3470502 (10Paladox) Wouldn't this also break gerrit? [15:16:28] 10cloud-services-team, 10wikitech.wikimedia.org, 10DBA: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3470509 (10Marostegui) neither on DBAs :-) [15:16:43] 10Cloud-Services, 10wikitech.wikimedia.org, 10Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#3470510 (10zhuyifei1999) >>! In T161859#3470502, @Paladox wrote: > Wouldn't this also break gerrit? gerrit uses LDAP account, so as long as LDAP is sane gerrit is okay. [15:17:16] 10Cloud-Services, 10wikitech.wikimedia.org, 10Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#3470511 (10Paladox) Yep but would new accounts work? [15:29:17] RECOVERY - Puppet errors on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [15:51:20] 10cloud-services-team (Kanban): nova-compute process monitoring: check twice - https://phabricator.wikimedia.org/T171606#3470603 (10Andrew) [16:12:53] !log ores staging ores-wmflabs-deploy:68326ba [16:12:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [16:20:30] 10Cloud-Services, 10wikitech.wikimedia.org, 10Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#3470696 (10bd808) >>! In T161859#3470511, @Paladox wrote: > Yep but would new accounts work? See the outline at the top which clearly includes `Replace wikitech as source of LDAP accoun... [16:21:01] 10Cloud-Services, 10wikitech.wikimedia.org, 10Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#3470697 (10Paladox) Oh i see. Thanks. [16:48:09] !log ores deploying ores-wmflabs-deploy:68326ba [16:48:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [16:48:27] hello people [16:48:44] I am trying to spin up an instance in the analytics project and getting zk2-1 nslcd[487]: [b127f8] no available LDAP server found: Server is unavailable [16:48:56] (got from horizon) [16:49:41] elukey hi, do you have a custom puppet master? [16:49:49] nope [16:49:59] hmm, try restarting it :) [16:51:32] I can try but it seems a horrible error :D [16:53:08] elukey: that's on a brand new instance for its first boot? [16:54:16] yep, https://horizon.wikimedia.org/project/instances/b94faf19-3d4e-41a4-ad3a-6c6fa5abafb8/console is after reboot [16:55:03] This is going to be unsatisfying, but if the first puppet run fails the instance is functionally dead. [16:55:20] The standard fix is to delete and try again [16:55:41] (with a different name) [16:55:42] all right, will try to fix the puppet config then, I thought that it would have booted [16:55:45] thanks!!! [16:55:50] elukey: <3 [16:55:57] madhuvishy: o/ [16:59:49] 10Cloud-Services: Create a "state of the cloud" monthly report - https://phabricator.wikimedia.org/T171618#3470944 (10bd808) [17:02:53] 10Cloud-Services: Create a "state of the cloud" monthly report - https://phabricator.wikimedia.org/T171618#3470981 (10bd808) The trick to automating this is figuring out where we can gather the interesting metrics from and possibly where we could store them for month over month comparisons. Counting the number... [17:04:08] 10Cloud-Services: Create a "state of the cloud" monthly report - https://phabricator.wikimedia.org/T171618#3470985 (10bd808) The metric(s) that come out of {T167556} would probably be reasonable to include on this report. [17:13:46] 10Cloud-Services: Create a "state of the cloud" monthly report - https://phabricator.wikimedia.org/T171618#3471051 (10bd808) LDAP account creations since date: ``` $ ldapsearch -xLLL -P 3 -E pr=40000/noprompt -o ldif-wrap=no -b"ou=people,dc=wikimedia,dc=org" '(&(objectClass=posixaccount)(createTimestamp>=2017070... [17:16:57] hi [17:18:16] hello fajjj [17:21:41] (03PS1) 10Jcrespo: rddmarc: Add fake dbpassword for the database application [labs/private] - 10https://gerrit.wikimedia.org/r/367702 (https://phabricator.wikimedia.org/T170158) [17:21:49] hello [17:21:53] i am former user of toolserver [17:22:13] and wanna be a member of bot projects [17:22:32] (03CR) 10Jcrespo: [V: 032 C: 032] rddmarc: Add fake dbpassword for the database application [labs/private] - 10https://gerrit.wikimedia.org/r/367702 (https://phabricator.wikimedia.org/T170158) (owner: 10Jcrespo) [17:24:57] rrrrr_: the "bot" project is obsolete, what you may want is toolforge [17:25:35] https://wikitech.wikimedia.org/wiki/Portal:Toolforge [17:29:50] 10Data-Services, 10DC-Ops, 10Operations: Split up labstore external shelf storage available in codfw between labstore2001 and 2 - https://phabricator.wikimedia.org/T171623#3471092 (10madhuvishy) [17:32:34] bd808: (if you have time) - even old hosts like zk1-1.analytics.eqiad.wmflabs that are not accessible anymore for some reason follow the puppet suggestion that you gave me? [17:33:32] elukey: let me take a look. We had some ldap craziness last week that may still be causing you issues. [17:33:42] thanks a lot! [17:34:53] !log analytics Puppet broken on zk1-1.analytics.eqiad.wmflabs with "$clusters[$cluster_name] is :undef, not a hash or array at /etc/puppet/modules/profile/manifests/zookeeper/server.pp:22" [17:34:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Analytics/SAL [17:35:24] elukey: ^ puppet isn't working on that instance so it does not have the updated LDAP x509 cert that it needs [17:35:50] ah I just flipped the instance to the new profile (since it was still using a old role not existing anymore) [17:36:17] *nod* [17:36:57] weird, will keep checking.. but IIUC once puppet runs it will unblock right? [17:37:13] the quickest fix to get you back into the instance would be to remove the profile. Then I can force a puppet run and get your LDAP + ssh access working again [17:37:41] then you can turn the profile back on and start the whack-a-mole game of getting your project heira right [17:38:30] This is fall out from https://wikitech.wikimedia.org/wiki/Incident_documentation/20170719-ldap [17:38:32] 10Cloud-Services: Create a "state of the cloud" monthly report - https://phabricator.wikimedia.org/T171618#3470944 (10Jprorama) Have often found number of unique log ins to a systems per month to be a useful stat: (Drop the wc to see the distribution of logins per user.) $ last |cut -f1 -d" " | head -n -2 | sor... [17:38:34] ahhhh now I think I found the issue, can I try to update hiera and bother you one more time? [17:38:39] sure [17:41:32] bd808: done, it should work now [17:41:46] * bd808 forces puppet run [17:41:46] if not I'll buy you a beer and remove the puppet prefix from horizon :D [17:42:58] elukey: I think you will be able to ssh in now [17:43:16] elukey@zk1-1:~$ [17:43:21] * elukey dances [17:43:25] thankssssss!! [17:43:41] !log analytics Forced puppet run on zk1-1.analytics.eqiad.wmflabs after elukey fixed hiera settings [17:43:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Analytics/SAL [17:52:41] 10Cloud-Services: Create a "state of the cloud" monthly report - https://phabricator.wikimedia.org/T171618#3471372 (10chasemp) /me waves to @Jprorama [18:13:26] :thumbsup: [18:39:36] bd808: btw, I think toolsadmin needs an update? Tool Labs => Toolforge [18:40:34] Sagan: https://gerrit.wikimedia.org/r/#/c/364134/ [18:40:59] bd808: ah, ok [18:41:08] this is all going to take months :) [18:41:40] like we have still toolserver URLS I guess :D [18:47:12] !log wikidata-dev shutting down structured-commons instance [18:47:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [19:06:41] 10Data-Services, 10cloud-services-team (Kanban), 10Operations, 10ops-eqiad, 10Patch-For-Review: Degraded RAID on labsdb1001 - https://phabricator.wikimedia.org/T171538#3471836 (10Cmjohnson) Disk replaced and rebuilding Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Onlin... [19:12:25] 10Cloud-Services, 10Operations, 10ops-eqiad: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3471871 (10Cmjohnson) [19:13:47] 10Cloud-Services, 10Operations, 10ops-eqiad: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3352141 (10Cmjohnson) @chasemp Which vlan are these going in...I racked in row A and D....i see the instruction say it's public but I see a comment that it's labs-support.... [19:18:31] 10Cloud-Services, 10Operations, 10ops-eqiad: rack/setup/install labmon1002 - https://phabricator.wikimedia.org/T165784#3471907 (10Cmjohnson) [19:19:54] 10Data-Services, 10cloud-services-team (Kanban), 10Operations, 10ops-eqiad, 10Patch-For-Review: Degraded RAID on labsdb1001 - https://phabricator.wikimedia.org/T171538#3471914 (10chasemp) thanks you @Cmjohnson [19:24:00] 10cloud-services-team: Change nova fullstack test back to testing all hosts - https://phabricator.wikimedia.org/T171641#3471935 (10Andrew) [20:59:50] 10Cloud-Services, 10Operations, 10ops-eqiad: rack/setup/install labmon1002 - https://phabricator.wikimedia.org/T165784#3472470 (10RobH) [21:17:26] 10Cloud-Services, 10MediaWiki-extensions-OpenStackManager, 10Toolforge: The future of service groups and service users on Labs - https://phabricator.wikimedia.org/T162945#3180968 (10Krinkle) Please don't delete `cvn.cvnservice` from CVN project without prior coordination first. This user is in active use. O... [21:26:08] bd808: thanks for the review, I put it up for swat [21:26:11] :) [21:26:36] cool. thanks for caring about wikitech! [21:30:04] the janitor :) [21:33:02] not sure if I'll be around 1 AM my time for the swat deploy though [21:33:05] I'll try [21:36:04] TabbyCat: there is an EU timezone SWAT these days too I think [21:40:25] bd808: I know, I use that a lot. I added that to the next avalaible swat window which happens to be the evening SF SWAT [21:41:27] bd808 yep the eu swat is at 2pm i think (utc +1 time not sure what TabbyCat is possibly 3pm?) [21:41:44] 3 PM is correct [21:41:57] :) [21:42:02] 2 PM in winter, hope it comes soon [21:42:18] lol, that is 1pm here :) [22:40:57] 10cloud-services-team, 10wikitech.wikimedia.org: Remove 'importers' (note the ending 's') group from wikitech - https://phabricator.wikimedia.org/T171682#3472955 (10MarcoAurelio) [23:35:10] 10cloud-services-team (FY2017-18), 10Research, 10Goal: [FY17-18] Program 4: Technical community building - https://phabricator.wikimedia.org/T171120#3473179 (10leila) [23:42:49] 10cloud-services-team (FY2017-18), 10Research, 10Goal: [FY17-18] Program 4: Technical community building - https://phabricator.wikimedia.org/T171120#3473227 (10leila) @bd808 I'm removing Research from the list of Tags. We still have our subtasks listed as part of this program. (this makes it more readable on... [23:43:02] 10cloud-services-team (FY2017-18), 10Goal: [FY17-18] Program 4: Technical community building - https://phabricator.wikimedia.org/T171120#3473228 (10leila)