[00:01:25] 10Tool-Labs-tools-Pageviews, 07I18n: massviews-category-description lego for "category" - https://phabricator.wikimedia.org/T146973#2918407 (10MusikAnimal) 05declined>03Open OK thanks for the help, all! And I love the babbles, no need to be sorry :) [00:07:40] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:34:10] 06Labs, 10Tool-Labs, 06Developer-Relations: Provide an easy way for Tool Labs tools to expose their source code - https://phabricator.wikimedia.org/T102081#2918542 (10bd808) [02:07:12] 06Labs: Revert increased quota for services-test labs project - https://phabricator.wikimedia.org/T153711#2918568 (10Eevans) >>! In T153711#2917384, @Andrew wrote: > Done -- please notify us when you're done with the additional instance. Thanks; I'll let you know! [02:31:17] 06Labs, 10wikitech.wikimedia.org, 05WMF-deploy-2017-01-03_(1.29.0-wmf.7): Wikitech blank page and no logs with mediawiki 1.29.0-wmf.7 - https://phabricator.wikimedia.org/T154618#2918592 (10Reedy) This is basically T153257 again.. :( Easiest "fix" for now, is to manually checkout the 3.7 tag, and push that a... [02:33:34] PROBLEM - Puppet run on tools-exec-1414 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [02:34:38] PROBLEM - Puppet run on tools-exec-1416 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [02:35:52] 06Labs: Upgrade Labs to OpenStack Mitaka - https://phabricator.wikimedia.org/T145919#2918599 (10Andrew) [02:35:55] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: nova-api: fix (or at least understand) deprecated configs before Mitaka - https://phabricator.wikimedia.org/T150776#2918597 (10Andrew) 05Open>03Resolved a:03Andrew [03:13:32] RECOVERY - Puppet run on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [03:14:40] RECOVERY - Puppet run on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [04:15:21] is the job queue for wikitech completely messed up? [04:15:38] I'm not seeing category changes recorded [04:18:51] categoryMembershipChange: 1 queued; 5460 claimed (79 active, 5381 abandoned); 0 delayed [04:18:57] that doesn't look right [04:20:35] * bd808 looks at T153618 [04:20:49] stashbot: you there? [04:20:49] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [04:27:44] 06Labs, 10wikitech.wikimedia.org: Job queue has 119663 entries - https://phabricator.wikimedia.org/T153618#2918697 (10bd808) I forgot about this problem and then found it again today when trying to add some new category links on Wikitech. ``` $ mwscript showJobs.php --wiki=labswiki --group refreshLinksPrioriti... [06:40:38] PROBLEM - Puppet run on tools-exec-1416 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:15:36] Hello, I'm new and may be able to help with Puppet code. It wasn't clear whether Projects puppet or puppet-cleanup were still after contributors. What's the next step? [07:15:38] RECOVERY - Puppet run on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:32] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/330639 (owner: 10L10n-bot) [08:11:23] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 15User-bd808: Facilitate Volunteer NDA application process for potential Tool Labs standards committee appointees - https://phabricator.wikimedia.org/T154625#2918846 (10Ladsgroup) I already signed the volunteer NDA {T134651}. Is there anything else I need to... [09:25:08] 06Labs, 10Tool-Labs, 10DBA: Spatial database for tool-labs - https://phabricator.wikimedia.org/T154497#2918930 (10Tobias1984) @akosiaris That would be great if I could get an account and the information on how to access the database from wmflabs. [09:47:03] PROBLEM - Free space - all mounts on tools-worker-1003 is CRITICAL: CRITICAL: tools.tools-worker-1003.diskspace._var_lib_docker.byte_percentfree (No valid datapoints found) tools.tools-worker-1003.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-worker-1003.diskspace.root.byte_percentfree (<100.00%) [10:08:42] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:34:55] 10Tool-Labs-tools-Xtools, 06Community-Tech: Investigation: Plan for rewriting XTools - https://phabricator.wikimedia.org/T154551#2919064 (10zhuyifei1999) >>! In T154551#2917712, @Matthewrbowker wrote: > @bd808 I understand. My comment with divorcing the code from Tool Labs more had to do with fun like [[https... [11:06:00] 06Labs, 10Tool-Labs, 10DBA: Spatial database for tool-labs - https://phabricator.wikimedia.org/T154497#2919157 (10akosiaris) >>! In T154497#2918930, @Tobias1984 wrote: > @akosiaris That would be great if I could get an account and the information on how to access the database from wmflabs. Sure, what's your... [11:48:38] 06Labs, 10Tool-Labs, 10DBA: Spatial database for tool-labs - https://phabricator.wikimedia.org/T154497#2919252 (10Tobias1984) Sorry. I forgot that I registered like this: https://wikitech.wikimedia.org/wiki/User:Tobias47n9e [12:19:34] Change on 12www.mediawiki.org a page OAuth/es was modified, changed by Sabbut link https://www.mediawiki.org/w/index.php?diff=2339894 edit summary: Created page with "Usar OAuth en sitios Wikimedia" [12:28:52] Change on 12www.mediawiki.org a page OAuth/es was modified, changed by Sabbut link https://www.mediawiki.org/w/index.php?diff=2339896 edit summary: Created page with "extensión que hace uso de la extensión OAuth" [12:33:08] 06Labs: Request creation of wikidata-federation labs project - https://phabricator.wikimedia.org/T154659#2919282 (10WMDE-leszek) [13:01:24] [= [14:22:21] 06Labs: Request creation of wikidata-federation labs project - https://phabricator.wikimedia.org/T154659#2919282 (10chasemp) Can you be more specific about several instances and what size instances those are meant to be? Our default quota allowance may not be enough depending. [15:05:33] 06Labs: Revert increased quota for services-test labs project - https://phabricator.wikimedia.org/T153711#2919537 (10Eevans) >>! In T153711#2917324, @chasemp wrote: > Hey @Eevans thanks for being patient with the holidays we are just now back in an almost normal session. Seems cool, thanks for noting this is te... [15:09:51] 06Labs: Request creation of wikidata-federation labs project - https://phabricator.wikimedia.org/T154659#2919538 (10WMDE-leszek) So basic setup would be two instances (standard m1.medium size should be just fine). For more advanced testing we might want to add 2-3 more instances. But I believe the standard quota... [15:28:54] 06Labs: Request creation of wikidata-federation labs project - https://phabricator.wikimedia.org/T154659#2919556 (10WMDE-leszek) p:05Triage>03Normal [15:35:05] 06Labs, 10DBA: page_lang column of the page table is not replicated to Labs - https://phabricator.wikimedia.org/T154355#2919584 (10chasemp) If I understand correctly this is not entirely a view issue but once it's fixed and replicated we still need to run maintain-views to establish them for the rest of the un... [15:35:55] 06Labs, 10Tool-Labs, 10Prod-Kubernetes, 10Tools-Kubernetes, 07kubernetes: Kubernetes discussion at Dev Summit - https://phabricator.wikimedia.org/T153943#2919586 (10chasemp) p:05Triage>03Normal [15:36:54] 06Labs, 10DBA: page_lang column of the page table is not replicated to Labs - https://phabricator.wikimedia.org/T154355#2919601 (10jcrespo) The column is on all wikis, except on labsdb1001:enwiki, where it cannot be added, and will never be. [15:39:21] 06Labs, 10Labs-Infrastructure: Duplicate /proc/mounts entries on labstore1001 - https://phabricator.wikimedia.org/T153728#2919609 (10chasemp) [15:39:27] 06Labs, 10Labs-Infrastructure: Duplicate /proc/mounts entries on labstore1001 - https://phabricator.wikimedia.org/T153728#2888830 (10chasemp) p:05Triage>03Normal [15:40:32] 06Labs: Update keystone hooks and monkeypatches for Mitaka - https://phabricator.wikimedia.org/T153469#2919612 (10chasemp) p:05Triage>03Normal [15:56:21] 06Labs, 10Tool-Labs, 10DBA: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2804720 (10chasemp) @yuvipanda can you let @marostegui know if this all ready for https://phabricator.wikimedia.org/T151014#2893169? [15:56:57] bd808: Is there an actual rule that tools on Tool Labs shouldn't be editing wikis anonymously? [15:57:56] anomie: I think there is an actual IP block (at least on some wikis) that enforces it [15:58:43] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: maintain-kubeusers broken again - https://phabricator.wikimedia.org/T150725#2919635 (10chasemp) 05Open>03Resolved a:03chasemp [16:00:24] chasemp: Although an IP block catches ISPs that use the same private addresses. I learned that a few years ago when I tried to make such a block on enwiki. :/ I think AbuseFilter might be able to at least deny the kinds of contributions AbuseFilter covers. [16:03:03] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Set up backups of tools and misc data from labstore1004/5 in labstore2003/4 - https://phabricator.wikimedia.org/T149870#2919663 (10chasemp) @madhuvishy can you update status here? iiuc a flock like thing is in play but we are still keeping only 1 historical... [16:05:52] I haven't seen a codified rule for Tools that I recall which says anon editing is not allowed, but it would be certainly considered bad manners, It seems worth considering if there are issues with limiting it otherwise. [16:07:18] I thought there were rules about bots and anon editing generally tbh [16:07:36] but it's a large and varied discussion from wht I can tell that has a lot of per wiki/per-project idioms [16:11:37] I have a patch in the works to basically let us add non-XFF-following IP blocks in the server config. Someone came along complaining about the possibility. [16:14:27] what is the value in XFF-following IP blocks I wonder? it seems like it would always result in odd outcomes if it follows the XFF chain blocking all or something or maybe just the first entry. That's all gameable anyway tho. Unless it uses x-real-ip (or whatever that is called now) which is our curated version as I understand it [16:14:48] and shouldn't suffer from teh same problem as I understand it [16:15:29] anyways, good luck :) let us know if we can assist from this side [16:16:11] 06Labs, 15User-Nikerabbit: Revert in 01/2017: Request creation of wmwcourse labs project - https://phabricator.wikimedia.org/T144388#2919708 (10chasemp) >>! In T144388#2916973, @Nikerabbit wrote: > I started shutting down instances that are no longer in use. Please don't delete the ones still running just yet.... [16:16:48] It can catch some cases where someone tries to get around a block by using a web proxy. At least that's the justification in the description of T25343. [16:16:48] T25343: Allow blocking open proxies based on X-Forwarded-For header (XFF) - https://phabricator.wikimedia.org/T25343 [16:17:41] huh anomie, that makes some sense but it would seem that exluding all RFC 1918 addresses would be pragmattic [16:18:04] that of course would not allow blocking from the 10.x space for labs as well I imagine [16:18:06] so it's a pickle I suppose [16:18:45] but at that point any private addressing has to be under our control in some form or another so banning private addresses from anon editing would work out [16:18:55] that's 60s of thought of course [16:19:01] probably much harder in practice to work through this [16:20:09] 06Labs: Clean up data in /data/scratch/mwoffliner - https://phabricator.wikimedia.org/T144025#2919722 (10chasemp) @Kelson can we wipe this off of scratch now? [16:23:27] 06Labs, 10wikitech.wikimedia.org, 05WMF-deploy-2017-01-03_(1.29.0-wmf.7): Wikitech blank page and no logs with mediawiki 1.29.0-wmf.7 - https://phabricator.wikimedia.org/T154618#2919724 (10thcipriani) >>! In T154618#2918592, @Reedy wrote: > This is basically T153257 again.. :( Well, wish I would've seen tha... [16:43:47] 06Labs, 10Tool-Labs: About 71 users are missing replica.my.cnf - https://phabricator.wikimedia.org/T140592#2919782 (10chasemp) @yuvipanda can we close this? [16:45:37] anomie: As far as I know such rules would be up to the wikis themselves. Setting policy rules like that for Labs/Tools would at best be a recommended practice since we don't do any deep inspection on outbound traffic [16:47:08] it would be whackamole enforcement (not that it's not an ok thing to have) [16:52:49] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2919816 (10chasemp) [17:11:29] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2919907 (10chasemp) [17:31:00] 10Tool-Labs-tools-Xtools, 06Community-Tech: Investigation: Plan for rewriting XTools - https://phabricator.wikimedia.org/T154551#2919965 (10bd808) >>! In T154551#2919064, @zhuyifei1999 wrote: > @bd808 I wonder if toollabs-vagrant is worth considering? It'll make many tools that are hardwired to tool labs (dire... [17:36:53] 06Labs, 06Operations, 13Patch-For-Review: Kill the labtest $realm - https://phabricator.wikimedia.org/T148717#2919989 (10Andrew) modules/puppetmaster/files/labtest.hiera.yaml is still used, and appears to still be useful. [17:36:59] preguiça* [17:37:05] ops [17:45:07] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 15User-bd808: Facilitate Volunteer NDA application process for potential Tool Labs standards committee appointees - https://phabricator.wikimedia.org/T154625#2920021 (10bd808) >>! In T154625#2918846, @Ladsgroup wrote: > I already signed the volunteer NDA {T1... [18:00:27] PROBLEM - High iowait on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: tools.tools-webgrid-lighttpd-1404.cpu.total.iowait (>11.11%) [18:04:02] chasemp: hi [18:05:11] chasemp: I will clean piece by piece this directory [18:05:28] RECOVERY - High iowait on tools-webgrid-lighttpd-1404 is OK: OK: All targets OK [18:06:46] 06Labs: Clean up data in /data/scratch/mwoffliner - https://phabricator.wikimedia.org/T144025#2920185 (10Kelson) @chasemp I'll clean it piece by piece to see if it works fine without. Until now I have only verified other aspects (than storage) of the full dumping. [18:13:41] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:20:15] 06Labs, 10Tool-Labs, 10DBA: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2920233 (10jcrespo) Check also on the other labs servers, just in case (1004, 5 and 9-11). [18:24:48] 06Labs, 10Tool-Labs, 10DBA: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2920254 (10yuvipanda) We're ready for it now, as long as I'm around to verify that things are working just fine :) could be now, could be later! [18:29:07] Kelson: cool, thank you. No rush just checking on status there. [18:37:58] Danny_B: could you look at VMs in teh grid here https://phabricator.wikimedia.org/T143349 and see if we can delete those outright (or plan on moving away from them) [18:45:40] chasemp: I have an email thread going with danny about those, I owe him a response still [18:45:48] andrewbogott: ok thanks [19:10:07] 06Labs: Revert increased quota for services-test labs project - https://phabricator.wikimedia.org/T153711#2920452 (10Eevans) >>! In T153711#2917384, @Andrew wrote: > Done -- please notify us when you're done with the additional instance. I'm not seeing enough capacity to launch an m1.xlarge instance. The //ser... [19:49:27] 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad, 07Wikimedia-Incident: Replace fans (or paste) on labservices1001 - https://phabricator.wikimedia.org/T154391#2920724 (10Andrew) Note to self: Before outage, silence icinga for labservices1001 and checker.tools.wmflabs.org [20:01:36] 06Labs: Revert increased quota for services-test labs project - https://phabricator.wikimedia.org/T153711#2920775 (10Andrew) Sorry, due to a typo when I first did this, your quotas were still wrong. I just now tried again... should be better. Apologies for the confusion! [20:09:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:12:26] chasemp: trying to add (already allocated) disk to phlogiston-3, but on the wikitech control page I don't have the "configure" button that the instructions say I should have. [20:12:47] jaufrecht: what instructions are you reading? [20:12:53] https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space [20:13:14] I only have "delete", "reboot", "get console output" [20:13:30] jaufrecht: that may just be obsolete as puppet management to add the role::labs::lvm::srv has moved to horizon.wikimedia.org [20:13:34] let me see if we have new docs [20:13:45] andrewbogott: ^ is that correct? [20:15:36] trying to figure it out now; if you don't have docs I can try updating the existing. [20:17:23] jaufrecht: yeah if you can get into horizon it may be more self-explanatory w/ the puppet management panel adn being able to see the role in the list of avail [20:17:26] I'm not sure [20:18:13] updated in docs: https://wikitech.wikimedia.org/w/index.php?title=Help%3AAdding_Disk_Space&type=revision&diff=1271488&oldid=618061 [20:18:35] thanks jaufrecht kudos [20:21:31] 06Labs, 10wikitech.wikimedia.org, 05WMF-deploy-2017-01-03_(1.29.0-wmf.7): Wikitech blank page and no logs with mediawiki 1.29.0-wmf.7 - https://phabricator.wikimedia.org/T154618#2920835 (10thcipriani) 05Open>03Resolved a:03thcipriani group1 has 1.29.0-wmf.7 and wikitech yet lives. Closing. [20:25:17] chasemp, jaufrecht, correct, at the moment disk volumes are only done via puppet [20:55:53] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2565438 (10fgiunchedi) re: ` filippo-test-precise2` please nuke [21:14:40] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:30:59] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2921002 (10chasemp) [21:32:25] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2565438 (10chasemp) @Danny_B Can we remove these instances? > wildcat dannyb-large > wildcat dannyb-test > wildcat dannyb [21:38:03] chasemp: no, you can not. as andrewbogott pointed out earlier, we are working (actually, going to work to be more precise) on their migration [21:40:23] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2921017 (10chasemp) @akosiaris or @Jgreen do you guys know what the story is with the `otrs-test2` instance? seems like we can purge it but I wanted to check. [21:43:00] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2921022 (10chasemp) @Rillke from looking at the SAL it seems like the `mediahandler-tests-static ` instance is probably removable, can you confirm? [21:46:52] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2921039 (10chasemp) [21:47:54] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2565438 (10chasemp) @dschwen how do things look for removing? [21:54:30] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2921057 (10chasemp) @Mattflaschen-WMF (listed as contact here https://wikitech.wikimedia.org/wiki/Nova_Resource:Editor-engagement) do you know if there is a plan to migrate: editor-engagement docs... [21:56:02] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2921069 (10dschwen) Please do not remove the fastcci or maps-wma1 instances! They are being used. [21:56:28] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2921070 (10chasemp) @Slevinski, HI. Do you know if there is a plan to migrate the `signwriting-ase-wiki` instance off of precise? [21:56:37] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2921072 (10Jgreen) If my user is associated with the instance then it is ancient and purgeworthy. The last work I did on OTRS was for the previous upgrade, Alex did the recent upgrade. [21:59:01] Anyone know what the licence on labs_mediawiki_logo.png e.g. https://visualeditor-test.wmflabs.org/labs_mediawiki_logo.png is (I can't find it anywhere in a repo) [22:00:08] James_F: nope! [22:00:17] Ho-hum. [22:00:50] Maybe we should just say it's a Labs logo and so CC-BY-SA. [22:00:55] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2921093 (10chasemp) [22:01:14] I like it but have no idea where it came from [22:02:03] I mean, I'd assume it originated with River or with Ryan Lane given the unicorn/rainbow motif. ;-) [22:03:17] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, and 2 others: Developing community norms for vital bots and tools - https://phabricator.wikimedia.org/T149312#2748486 (10cscott) At the editing team off-site we also raised the idea of improving our frameworks so that gadgets/bots/exte... [22:04:21] we have all come to identify w/ the pointy horse James_F, but yes probably, although I thought https://wikitech.wikimedia.org/w/images/thumb/6/60/Wikimedia_labs_logo.svg/180px-Wikimedia_labs_logo.svg.png and https://upload.wikimedia.org/wikipedia/commons/thumb/a/a4/Tool_labs_logo.svg/180px-Tool_labs_logo.svg.png [22:04:24] were submitted by volunteers [22:04:27] I could be wrong [22:04:53] Yes, though they're "recent". [22:05:08] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2921104 (10Slevinski) @chasemp , We will not be migrating signwriting-ase-wiki. It is fine if it shuts down in March. The ASL Wikipedia is available on Incubator. [22:05:40] James_F: lost to the sands of time my friend [22:05:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [22:09:36] 06Labs, 06Operations, 10hardware-requests: Site: (1) hardware access request for labtest - https://phabricator.wikimedia.org/T154706#2921133 (10chasemp) [22:13:34] 06Labs, 06Operations, 10hardware-requests: Site: (1) hardware access request for labtest - https://phabricator.wikimedia.org/T154706#2921186 (10chasemp) [22:15:00] 06Labs, 06Operations, 10hardware-requests: Site: (1) hardware access request for labtest - https://phabricator.wikimedia.org/T154706#2921133 (10chasemp) 05Open>03stalled [22:21:59] 06Labs, 13Patch-For-Review: promethium.wikitextexp.eqiad.wmflabs (10.68.16.2, labs baremetal host) has strange DNS A record result, and missing PTR - https://phabricator.wikimedia.org/T139438#2921225 (10Krenair) a:05AlexMonk-WMF>03Krenair [22:22:21] 06Labs, 10Horizon, 13Patch-For-Review: Switch dynamicproxy to point back to IP rather than domain names - https://phabricator.wikimedia.org/T133554#2921231 (10Krenair) a:05AlexMonk-WMF>03Krenair [22:22:57] 06Labs, 10Labs-Sprint-109: Remove reliance on ldap $::projectid from shinkengen - https://phabricator.wikimedia.org/T108625#2921242 (10Krenair) a:05AlexMonk-WMF>03Krenair [23:01:01] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, and 2 others: Developing community norms for vital bots and tools - https://phabricator.wikimedia.org/T149312#2921640 (10bd808) >>! In T149312#2921097, @cscott wrote: > > Right now the entry point for many community developers is a cl... [23:15:42] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:16:51] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, and 2 others: Developing community norms for vital bots and tools - https://phabricator.wikimedia.org/T149312#2921658 (10Anomie) >>! In T149312#2921097, @cscott wrote: > Right now the entry point for many community developers is a clie... [23:49:59] 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad, 07Wikimedia-Incident: Replace fans (or paste) on labservices1001 - https://phabricator.wikimedia.org/T154391#2921778 (10Cmjohnson) I purchased some thermal paste today and the entire process should not take longer than 10mins.