[00:01:53] bd808: huh, that's surprisig - i thought it had been completely offline actually, due to the file permissions/deletion issue.. [00:01:59] ..until i restored it this monday [00:02:13] the app.py file was missing [00:02:37] or could that have been it? (i.e. webservice not finding the app.py counting as a restart) [00:05:11] HaeB: ah. that could be it [00:14:49] HaeB: it looks like it has been running since 2017-07-24T02:55:56. So it was probably just in that horrible restart loop until you fixed it [00:15:13] ah good [00:16:11] BTW i read chasemp's notes about the issue (https://lists.wikimedia.org/pipermail/labs-announce/2017-July/000246.html ), and my files and directories actually didn't have o+w set when this happened - at least according to the backup [00:16:29] ... where the permissions reset on the backup? [00:16:35] HaeB: I changd it on restore as restoring files w/ bad perms seemed silly :) [00:16:48] i see [00:16:53] Yeah. teh bad perms were fixed after the restore [00:28:37] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [00:35:06] 10Cloud-VPS, 10Puppet: role::puppetmaster::standalone: Unable to locate package geoipupdate - https://phabricator.wikimedia.org/T171916#3480251 (10MaxSem) [00:35:34] 10Cloud-VPS, 10Puppet: role::puppetmaster::standalone: Unable to locate package geoipupdate - https://phabricator.wikimedia.org/T171916#3479983 (10MaxSem) Still fails, with even more errors (I tried on a fresh VM). [00:57:15] 10Cloud-VPS, 10Puppet: role::puppetmaster::standalone: Unable to locate package geoipupdate - https://phabricator.wikimedia.org/T171916#3480258 (10bd808) >>! In T171916#3480251, @MaxSem wrote: > Still fails, with even more errors (I tried on a fresh VM). Is this a jessie|stretch base image? For //"reasons"//... [00:58:47] 10Cloud-VPS, 10Puppet: role::puppetmaster::standalone: Unable to locate package geoipupdate - https://phabricator.wikimedia.org/T171916#3480261 (10MaxSem) Stretch. [01:06:40] MaxSem: I'd be close to betting that the WMF stretch repo doesn't have the geoipupdate package in it [01:07:39] that's a package we load locally and I would guess that it hasn't been built for stretch yet [01:08:22] * MaxSem misses Windows where the same .msi packages work across ten years of versions! [01:08:29] * paravoid would like to place that bet [01:08:37] well ok [01:08:37] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [01:08:38] to be fair [01:08:49] geoipupdate isn't part of stretch-wikimedia, that's true [01:09:02] but it doesn't need to be, I uploaded it to Debian and it's part of stretch proper now :) [01:09:08] https://packages.debian.org/stretch/geoipupdate [01:09:34] why is it needed for freaking puppetmaster anyway? :P [01:09:48] I guess contrib isn't enabled in labs? [01:09:58] probably not... [01:10:04] MaxSem: because in prod, we use it to distribute paid-for GeoIP databases [01:11:35] PROBLEM - Puppet errors on tools-exec-1404 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:13:19] 10Cloud-VPS, 10Puppet: role::puppetmaster::standalone on stretch: Unable to locate package geoipupdate - https://phabricator.wikimedia.org/T171916#3480267 (10bd808) [01:14:01] enabling contrib in labs should be fine by its ToS I think [01:14:21] but a better solution would probably be to untangle the geoip distribution stuff from the puppetmaster manifests [01:14:27] because you're right, it's not strictly needed [01:14:35] just an artifact of the prod setup [01:14:42] +1 for that [01:14:52] this isn't the first time its been tripped over [01:15:32] MaxSem: the "make your life easy" answer is to use jessie instead of stretch for now [01:15:51] or just add "contrib" to your sources.list, that should work [01:15:58] or a new sources.list.d [01:16:13] for this issue at least, not sure if there are other stretch/puppet issues not discovered yet :) [01:16:20] depends on how adventurous you want to be :) [01:22:17] PROBLEM - Puppet errors on tools-exec-1402 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:29:13] Hello [01:30:24] !help [01:30:24] Venkat: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [02:02:18] RECOVERY - Puppet errors on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [02:16:35] RECOVERY - Puppet errors on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [02:21:40] 10cloud-services-team, 10Operations: notebook100[12] - Invalid relationship: Apt::Pin[r-base] - https://phabricator.wikimedia.org/T171924#3480338 (10Dzahn) [02:23:20] PROBLEM - Puppet errors on tools-exec-1402 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [02:37:35] PROBLEM - Puppet errors on tools-exec-1404 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [03:28:18] RECOVERY - Puppet errors on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [03:48:17] PROBLEM - Puppet errors on tools-exec-1438 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [03:49:19] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [04:18:15] RECOVERY - Puppet errors on tools-exec-1438 is OK: OK: Less than 1.00% above the threshold [0.0] [04:24:21] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [04:42:36] RECOVERY - Puppet errors on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [04:49:10] PROBLEM - Puppet errors on tools-worker-1005 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:24:08] RECOVERY - Puppet errors on tools-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [06:31:29] 10Cloud-Services, 10Toolforge: http://tools.wmflabs.org/?status is no longer sortable - https://phabricator.wikimedia.org/T157648#3480485 (10zhuyifei1999) 05Open>03Resolved Looks sortable now. Perhaps fixed in T140254? [06:31:31] 10Cloud-Services, 10Toolforge: Sorting by CPU/VMEM columns doesn't sort by their value on http://tools.wmflabs.org/?status - https://phabricator.wikimedia.org/T69737#3480489 (10zhuyifei1999) [06:59:38] PROBLEM - Puppet errors on tools-exec-1406 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:33:36] PROBLEM - Puppet errors on tools-exec-1404 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:39:35] RECOVERY - Puppet errors on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [07:48:57] 10cloud-services-team, 10Operations: notebook100[12] - Invalid relationship: Apt::Pin[r-base] - https://phabricator.wikimedia.org/T171924#3480579 (10MoritzMuehlenhoff) p:05Triage>03High Seems like a side effect of 7dfe90c0d494999e2cfc05b12169401d40d54c99 ? [08:02:37] 10Tool-Zppixbot: add a feature to remind a certain user to do something - https://phabricator.wikimedia.org/T171931#3480605 (10Reception123) [08:13:37] RECOVERY - Puppet errors on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [09:04:35] PROBLEM - Puppet errors on tools-exec-1404 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:44:35] RECOVERY - Puppet errors on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [10:31:59] 10wikitech.wikimedia.org: Create a Gadget to easily add/remove/modify patches for SWAT at wikitech:Deployments - https://phabricator.wikimedia.org/T171940#3480850 (10MarcoAurelio) [10:52:27] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:15:09] PROBLEM - Puppet errors on tools-worker-1005 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:27:32] is tools-login working good for you? [11:29:58] TabbyCat: no [11:30:12] I had to login from dev.tools [11:30:13] someone took down the IO I think [11:34:37] http://tools.wmflabs.org/nagf/?project=tools#h_tools-bastion-03_load load is skyrocketing [11:50:21] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:53:11] 10Toolforge: Find out how to throttle per-process / per-user NFS IO on tools-login so they can't unintentionally eat up all the bandwidth - https://phabricator.wikimedia.org/T171944#3480963 (10zhuyifei1999) [11:55:10] RECOVERY - Puppet errors on tools-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [11:57:25] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [12:35:19] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [12:42:01] http://tools.wmflabs.org/?tool=tool-account did someone name their tool 'tool-account'?! lol [12:43:19] 10Cloud-Services, 10DBA: Prepare and check storage layer for hi.wikiversity - https://phabricator.wikimedia.org/T171829#3481097 (10Urbanecm) p:05Triage>03Low [13:26:10] zhuyifei1999_: seems to have recovered [13:26:14] if we can pin down what they were doing we can possibly compensate [13:58:56] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1407 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:03:01] I have no idea who was doing it [14:03:33] during the whole time I was unable to login, much less than getting a process list [14:20:44] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Operations: instance root passwords vs. multiple puppetmasters - https://phabricator.wikimedia.org/T171959#3481362 (10Andrew) [14:38:55] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [15:07:47] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Operations: Switch to new labs puppetmasters - https://phabricator.wikimedia.org/T171786#3481437 (10Andrew) [15:08:26] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Operations: Switch to new labs puppetmasters - https://phabricator.wikimedia.org/T171786#3476152 (10Andrew) [15:37:17] Hello, I'm Krishn. I'm interesting in contributing to wikimedia TechOps. My background is computer science and my skills include python,linux,jenkins,and knowledge on puppet. Can anybody help me in getting started. Thanks. [15:44:15] mkrish_: you are in the user channel for cloud services, general tech ops would be discussed in wikimedia-operations fyi. To learn about cloud services read https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction and https://wikitech.wikimedia.org/wiki/Help:Getting_Started and for general Operations read https://wikitech.wikimedia.org/wiki/Get_involved. There is some upfront setup to getting involved I [15:44:15] think but the best bet is to go through open tasks (lik ehttps://phabricator.wikimedia.org/maniphest/?project=PHID-PROJ-msyn2z45n7mw45bfuscb&statuses=open()&group=none&order=newest#R) and lurk in irc to see what get an idea of what is going on [15:46:09] Thanks for your help :) [16:04:02] isn't wikimedia cloud hiring an ops? [16:12:56] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:29:44] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:37:58] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [16:43:38] zhuyifei1999_: yes, we are indeed [16:44:06] * bd808 has reviewed >200 resumes for that in the last 4 months [16:47:44] PROBLEM - Puppet errors on tools-exec-1441 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:48:33] o.O [17:09:46] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:01] PROBLEM - Puppet errors on tools-exec-1418 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:33:54] PROBLEM - Puppet errors on tools-worker-1009 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:50:03] RECOVERY - Puppet errors on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [18:13:53] 10cloud-services-team (FY2017-18), 10Goal, 10Patch-For-Review: Refactor openstack Puppet to account for Neutron - https://phabricator.wikimedia.org/T171494#3482061 (10chasemp) Deployed https://gerrit.wikimedia.org/r/#/c/368321/ and rolled it back yesterday, nothing blew up but I figured out I had been testin... [18:13:56] RECOVERY - Puppet errors on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0] [18:49:56] Hello, is it possible to change location of jsub jobname.out and jobname.err files? I prefer creating a directory in the homedir and directing all those files to that directory. Or, is it possible to just suppress their creation (for example when I have 30 jobs producing no output I have 60 meaningless files I just delete)? [18:50:29] Urbanecm: yes you can use the "-o" and "-e" flags to change output and error output files, path is relative to job dir if I recall [18:50:39] ah, to log nothing I'm not sure [18:50:55] good question I'll see what qsub/jsub say [18:51:20] I think -o /dev/null will supress [18:51:25] likewise for -e [18:51:34] that should do it I bet yeah [18:52:11] I have had odd behavior with trying to get tricky w/ -o and -e tho as it literally tries to open a file iirc instead of doing redirection [18:52:18] but I think /dev/null is amendable to that [18:52:20] chasemp, jpr: Thank you both. Can I set a directory for them? E.g. directing all of them t ~/outputs? [18:52:35] Urbanecm: we have a "logs" directory in the home of every tool iirc [18:52:47] that would be ideal as it marks it as ephemeral data esp when we go to do cleanup [18:53:46] chasemp, but jsub adds output to the homedir [18:54:12] yes that's a thing we need to change honestly [18:54:20] It'll be great! [18:54:39] I think webservices use log dir by default already but jsub does not [18:55:06] jpr: jsub is our wrapper around qsub for (in theory) user friendliness [18:55:26] :) [18:55:41] chasemp, really? I have access.log/error.log for the old grid way :) [18:55:46] (directly in homedir) [18:56:07] Urbanecm: maybe it's only k8s webservices or I'm misremembering [18:56:09] both are possible [18:57:03] I once went on the war path for saner log handling and then got derailed, that was like a year ago :) [18:57:43] Thank you all! [19:04:09] access.log/error.log is with lighttpd iirc [19:06:11] Urbanecm: regarding ~/outputs. I think qsub -o and -e are just path names and if you specify a dir it will but the default output file names there. You may want to just use -o output/ since relative paths default to $HOME (unless the -cwd job flag is set). from the jsub code it looks like these pass through unchanged [19:06:38] s/but the /put the/ [19:07:33] the docs the -e flag for qsub has all the details of what can be achieved. http://gridscheduler.sourceforge.net/htmlman/htmlman1/qsub.html [19:52:48] 10wikitech.wikimedia.org, 10MediaWiki-extensions-Linter, 10Parsoid, 10Services (designing): On wikis without changeprop enabled, lint errors don't update after page edits - https://phabricator.wikimedia.org/T171788#3482369 (10GWicke) [20:31:18] 10Tool-fatameh: More descriptions for Fatameh - https://phabricator.wikimedia.org/T171995#3482470 (10XXN) [20:34:58] 10Tool-fatameh: More descriptions for Fatameh - https://phabricator.wikimedia.org/T171995#3482498 (10XXN) [20:37:22] tom29739 are you around? [20:38:10] 10Tool-fatameh: More descriptions for Fatameh - https://phabricator.wikimedia.org/T171995#3482529 (10XXN) [20:46:49] 10Cloud-Services, 10wikitech.wikimedia.org, 10BetaFeatures, 10Edit-Review-Improvements, and 3 others: ERI requesting opt-in on wikitech but not available - https://phabricator.wikimedia.org/T165822#3482630 (10jmatazzoni) 05Open>03Resolved [22:25:02] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:41:35] 10cloud-services-team (FY2017-18), 10Goal, 10Patch-For-Review: Refactor OpenStack Puppet to account for Neutron - https://phabricator.wikimedia.org/T171494#3483013 (10chasemp) [23:00:00] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [23:01:40] 10Toolforge, 10Tools-Kubernetes, 10Kubernetes: Update Toolforge k8s nodejs images to 6.11 - https://phabricator.wikimedia.org/T170716#3483049 (10bd808) [23:01:42] 10Cloud-Services, 10Tools-Kubernetes, 10Kubernetes: newer npm for nodejs Kubernetes instances - https://phabricator.wikimedia.org/T169451#3483050 (10bd808) [23:01:44] 10Toolforge, 10Tools-Kubernetes, 10Kubernetes: Consider moving Tool Forge flannel backend to host-gw - https://phabricator.wikimedia.org/T167084#3483053 (10bd808) [23:01:46] 10Toolforge, 10Tools-Kubernetes, 10Kubernetes: Periodic cleanup of Docker registry - https://phabricator.wikimedia.org/T169366#3483051 (10bd808) [23:01:48] 10Cloud-Services, 10PAWS, 10Tools-Kubernetes, 10Kubernetes, 10Patch-For-Review: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3483052 (10bd808) [23:01:50] 10Cloud-Services, 10Tools-Kubernetes, 10Kubernetes: Make tools-webservice use the official kubernetes python client rather than pykube - https://phabricator.wikimedia.org/T159892#3483057 (10bd808) [23:01:52] 10Cloud-VPS, 10Toolforge, 10Tools-Kubernetes, 10Kubernetes: Homedir/UID info breaks after a while in Tools Kubernetes (can't read replica.my.cnf) - https://phabricator.wikimedia.org/T166949#3483055 (10bd808) [23:01:52] phab spam incoming [23:01:54] 10Toolforge, 10Tools-Kubernetes, 10cloud-services-team (Kanban), 10Kubernetes: Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3483054 (10bd808) [23:01:56] 10Toolforge, 10Tools-Kubernetes, 10Documentation, 10Kubernetes: Document basics of using a custom Kubernetes "Deployment" to operate a tool - https://phabricator.wikimedia.org/T165403#3483056 (10bd808) [23:01:58] 10Toolforge, 10Tools-Kubernetes, 10Kubernetes: Make maintain-kubeusers run on first attempt - https://phabricator.wikimedia.org/T158453#3483058 (10bd808) [23:02:00] 10Toolforge, 10Tools-Kubernetes, 10Kubernetes: k8s webservice restart failure with `ValueError: get() more than one object; use filter` - https://phabricator.wikimedia.org/T156626#3483060 (10bd808) [23:02:02] 10Toolforge, 10Tools-Kubernetes, 10Kubernetes: Make webservice backend for lighttpd default to kubernetes - https://phabricator.wikimedia.org/T154506#3483061 (10bd808) [23:02:04] 10Toolforge, 10Tools-Kubernetes, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Unify k8s roles between prod and tools - https://phabricator.wikimedia.org/T158452#3483059 (10bd808) [23:02:07] 10Toolforge, 10Tools-Kubernetes, 10Kubernetes, 10Tracking: Make webservice backend default to kubernetes - https://phabricator.wikimedia.org/T154504#3483062 (10bd808) [23:02:09] 10Toolforge, 10Tools-Kubernetes, 10Community-Tech-Tool-Labs, 10Kubernetes: My first kubernetes + python3 + django app tutorial - https://phabricator.wikimedia.org/T149191#3483063 (10bd808) [23:02:12] 10Toolforge, 10Tools-Kubernetes, 10Kubernetes: Move kubernetes authentication to using X.509 client certs - https://phabricator.wikimedia.org/T144153#3483064 (10bd808) [23:02:14] 10Toolforge, 10Tools-Kubernetes, 10Kubernetes: Setup Kubernetes Masters in a HA setup - https://phabricator.wikimedia.org/T142862#3483065 (10bd808) [23:02:16] 10Cloud-Services, 10Tools-Kubernetes, 10Kubernetes: Provide a migration path for tools running tomcat - https://phabricator.wikimedia.org/T141396#3483068 (10bd808) [23:02:18] 10Toolforge, 10Tools-Kubernetes, 10Kubernetes: Build replacement for the webservice toolschecker test - https://phabricator.wikimedia.org/T142164#3483066 (10bd808) [23:02:20] 10Cloud-Services, 10Tools-Kubernetes, 10Kubernetes: Monitor kube2proxy failures - https://phabricator.wikimedia.org/T140988#3483072 (10bd808) [23:02:22] 10Cloud-Services, 10Tools-Kubernetes, 10Kubernetes: Create failover host for docker registry - https://phabricator.wikimedia.org/T141030#3483071 (10bd808) [23:02:24] 10Cloud-Services, 10Tools-Kubernetes, 10Kubernetes: Tools with names longer than 24 characters cannot start kubernetes webservices - https://phabricator.wikimedia.org/T141100#3483070 (10bd808) [23:02:29] 10Cloud-Services, 10Tools-Kubernetes, 10Kubernetes, 10Patch-For-Review, 10User-bd808: Add a easy way to run a ruby webservice on tools - https://phabricator.wikimedia.org/T141388#3483069 (10bd808) [23:02:31] 10Toolforge, 10Tools-Kubernetes, 10Kubernetes: Monitor that not too many replicasets have a big difference between desired and current+pending - https://phabricator.wikimedia.org/T140561#3483074 (10bd808) [23:02:42] 10Toolforge, 10Tools-Kubernetes, 10Kubernetes: Health check for k8s etcd - https://phabricator.wikimedia.org/T140247#3483078 (10bd808) [23:02:42] 10Cloud-Services, 10Tools-Kubernetes, 10Kubernetes: Move tools-db and tools-redis into DNS - https://phabricator.wikimedia.org/T139190#3483081 (10bd808) [23:02:44] 10Cloud-Services, 10Tools-Kubernetes, 10Kubernetes, 10Tracking: Issues with 'webservice' kubernetes backend (tracking) - https://phabricator.wikimedia.org/T139107#3483082 (10bd808) [23:02:46] 10Toolforge, 10Tools-Kubernetes, 10Community-Tech-Tool-Labs, 10Epic, 10Kubernetes: Evaluate Kubernetes based workflow replacement options for SGE - https://phabricator.wikimedia.org/T136264#3483084 (10bd808) [23:02:49] 10Toolforge, 10Tools-Kubernetes, 10Community-Tech-Tool-Labs, 10Kubernetes: Develop evaluation criteria for comparing Platform as a Service (PaaS) solutions - https://phabricator.wikimedia.org/T136265#3483083 (10bd808) [23:02:52] 10Cloud-Services, 10Tools-Kubernetes, 10Kubernetes: Decide on upgrade policy for Kubernetes - https://phabricator.wikimedia.org/T133598#3483085 (10bd808) [23:02:54] 10Toolforge, 10Tools-Kubernetes, 10Kubernetes: Set up (admin-only for now) kubernetes dashboard - https://phabricator.wikimedia.org/T133098#3483086 (10bd808) [23:02:56] 10Toolforge, 10Tools-Kubernetes, 10Kubernetes: Setup monitoring for kubernetes core components. - https://phabricator.wikimedia.org/T131929#3483087 (10bd808) [23:04:21] 10Data-Services, 10Toolforge: Find out how to throttle per-process / per-user NFS IO on tools-login so they can't unintentionally eat up all the bandwidth - https://phabricator.wikimedia.org/T171944#3483093 (10bd808) [23:10:48] 10Cloud-VPS, 10Puppet: role::puppetmaster::standalone on stretch: Unable to locate package geoipupdate - https://phabricator.wikimedia.org/T171916#3483101 (10bd808) Discussed a bit on irc with @faidon. The recommended short term fix is to use jessie instead of stretch. The next tier of fix is for us to fix op... [23:11:15] 10Cloud-VPS, 10Puppet: role::puppetmaster::standalone on stretch: Unable to locate package geoipupdate - https://phabricator.wikimedia.org/T171916#3483104 (10bd808) p:05Triage>03Normal [23:22:56] 10cloud-services-team (FY2017-18), 10Goal: Program 10 Outcome 2: Rebranding - https://phabricator.wikimedia.org/T166404#3483132 (10bd808) [23:22:58] 10cloud-services-team (FY2017-18), 10Goal, 10Patch-For-Review, 10User-bd808: Perform initial Cloud Services rebranding - https://phabricator.wikimedia.org/T168480#3483131 (10bd808) [23:23:00] 10cloud-services-team (Kanban), 10Project-Admins, 10User-bd808: Rename and update Cloud Services Phabricator projects - https://phabricator.wikimedia.org/T167244#3483129 (10bd808) 05Open>03Resolved Declaring victory here. If we find more phab things to fix up for the renaming effort we can create new tasks.