[00:36:38] 10Striker, 10cloud-services-team (Kanban), 10User-bd808: Unable to add new LDAP user as maintainer of a tool - https://phabricator.wikimedia.org/T173963#3547536 (10bd808) {F9161389} {F9161391} [00:56:22] 10Cloud-Services, 10cloud-services-team (Kanban), 10Privacy, 10Security: "last" command on WMF Labs/Tools allows users to view IPs of other toolforge users - https://phabricator.wikimedia.org/T172650#3547545 (10bd808) [00:56:24] 10Cloud-VPS, 10cloud-services-team (Kanban), 10WMF-Legal, 10User-bd808: Add warning about ssh ip address visibility to Cloud VPS TOU - https://phabricator.wikimedia.org/T173846#3547542 (10bd808) 05Open>03Resolved a:03bd808 https://wikitech.wikimedia.org/w/index.php?title=MediaWiki:Signupstart&diff=17... [00:56:53] 10Data-Services, 10DC-Ops, 10Operations, 10ops-codfw: Split up labstore external shelf storage available in codfw between labstore2001 and 2 - https://phabricator.wikimedia.org/T171623#3547547 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['labstore2001.codfw.wmnet'] ``` Of which those **FAI... [00:57:37] 10Cloud-Services, 10cloud-services-team (Kanban), 10Privacy, 10Security, 10User-bd808: "last" command on WMF Labs/Tools allows users to view IPs of other toolforge users - https://phabricator.wikimedia.org/T172650#3547551 (10bd808) 05Open>03Resolved a:03bd808 Account creation in both #wikitech.wiki... [00:59:32] 10Striker, 10cloud-services-team (Kanban), 10User-bd808: Unable to add new LDAP user as maintainer of a tool - https://phabricator.wikimedia.org/T173963#3546209 (10bd808) p:05Triage>03High [01:00:21] 10Striker, 10cloud-services-team (Kanban), 10User-bd808: GID out of range warning while creating a new tool account - https://phabricator.wikimedia.org/T173958#3547567 (10bd808) p:05Triage>03Normal a:03bd808 [01:45:05] PROBLEM - Puppet errors on tools-exec-1421 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [02:20:05] RECOVERY - Puppet errors on tools-exec-1421 is OK: OK: Less than 1.00% above the threshold [0.0] [02:54:54] 10Horizon, 10cloud-services-team (Kanban): puppet configuration broken on Horizon - https://phabricator.wikimedia.org/T173982#3547649 (10Zoranzoki21) [02:55:13] 10Tools: Tool "mediawiki-feeds" loads bootstrap and jquery from bootstrapcdn and ajax.googleapis.com - https://phabricator.wikimedia.org/T172663#3547650 (10Samwilson) p:05Triage>03Normal [03:03:09] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3547656 (10Samwilson) [03:03:12] 10Tools: Tool "mediawiki-feeds" loads bootstrap and jquery from bootstrapcdn and ajax.googleapis.com - https://phabricator.wikimedia.org/T172663#3547654 (10Samwilson) 05Open>03Resolved Fixed in https://github.com/samwilson/mediawiki-feeds/commit/8702e6900f4015a91e9c59d81fa5e8bbccf52865 and deployed on toolfo... [04:20:36] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3547669 (10Mmarx) [04:20:38] 10Tools: Tool "sqid" loads assets from bootstrapcdn, fonts.googleapis.com, and code.jquery.com - https://phabricator.wikimedia.org/T172833#3547667 (10Mmarx) 05Open>03Resolved Merged and deployed. [05:27:54] 10Tools: Add code snippets with integrity hashes to CDNJS interface - https://phabricator.wikimedia.org/T173996#3547691 (10Samwilson) [05:31:40] 10cloud-services-team, 10Community-Tech, 10DBA, 10Security: create production ip_changes table for RangeContributions - https://phabricator.wikimedia.org/T173891#3547704 (10Marostegui) [05:37:38] 10cloud-services-team, 10Community-Tech, 10DBA, 10Security: create production ip_changes table for RangeContributions - https://phabricator.wikimedia.org/T173891#3547705 (10Marostegui) >>! In T173891#3546800, @kaldari wrote: > Ran `foreachwiki sql.php /srv/mediawiki/php/maintenance/archives/patch-ip_change... [06:01:24] 10cloud-services-team, 10Community-Tech, 10DBA, 10Security: create production ip_changes table for RangeContributions - https://phabricator.wikimedia.org/T173891#3547706 (10Marostegui) >>! In T173891#3545400, @kaldari wrote: >>if possible try to avoid assigning it directly to Jaime > I assigned it to Jaime... [06:52:04] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:30:52] PROBLEM - Puppet errors on tools-exec-1428 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [07:32:03] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [08:10:51] RECOVERY - Puppet errors on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [10:14:40] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1427 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:49:43] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1427 is OK: OK: Less than 1.00% above the threshold [0.0] [12:20:03] PROBLEM - Puppet errors on tools-exec-1433 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:45:01] 10Tool-fatameh: Author doubled in fatameh - https://phabricator.wikimedia.org/T174015#3548269 (10Fnielsen) [12:47:31] 10Tool-fatameh: Author doubled in fatameh - https://phabricator.wikimedia.org/T174015#3548300 (10Fnielsen) There are many of these problems for Karl Zilles, see: https://tools.wmflabs.org/sourcemd/new_resolve_authors.php?name=Karl+Zilles&doit=Look+for+author [13:00:01] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [14:35:15] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:08:12] PROBLEM - Puppet errors on tools-exec-1405 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:20:16] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:22:24] PROBLEM - Puppet errors on tools-proxy-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:29:59] 10Toolforge, 10Structured-Data-Commons, 10Wikidata: Provide backwards compatibility views for toolforge replica [MCR] - https://phabricator.wikimedia.org/T174047#3549198 (10daniel) [15:35:52] 10Tools: Add code snippets with integrity hashes to CDNJS interface - https://phabricator.wikimedia.org/T173996#3547691 (10bd808) >>! In J65#654, @bd808 wrote: > @Samwilson this sounds like something worth opening a task for. We don't run the same frontend or build script as the upstream, so what needs to be don... [15:47:32] 10Tools: Add code snippets with integrity hashes to CDNJS interface - https://phabricator.wikimedia.org/T173996#3549258 (10bd808) The current frontend (rTCJS) is pretty bare bones. It's a single static page that is generated from the combined json manifest that [[https://phabricator.wikimedia.org/source/operatio... [15:48:12] RECOVERY - Puppet errors on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [15:56:06] 10Horizon, 10cloud-services-team (Kanban): puppet configuration broken on Horizon - https://phabricator.wikimedia.org/T173982#3549313 (10Andrew) 05Open>03Resolved I believe this is now fixed. There were a few issues: - the new puppetmasters didn't allow horizon to access the puppet API to get a list of a... [16:00:16] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [16:02:26] RECOVERY - Puppet errors on tools-proxy-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:03:17] PROBLEM - Puppet errors on tools-redis-1001 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:27:02] 10Striker: Create new OAuth grant named "Toolforge admin console" - https://phabricator.wikimedia.org/T174052#3549429 (10bd808) [16:27:34] 10Striker: Create new OAuth grant named "Toolforge admin console" - https://phabricator.wikimedia.org/T174052#3549447 (10bd808) [16:27:36] 10cloud-services-team (FY2017-18), 10Goal, 10Patch-For-Review: Program 10 Outcome 2: Rebranding - https://phabricator.wikimedia.org/T166404#3549446 (10bd808) [16:31:54] 10cloud-services-team (Kanban): Time to renew *.wmflabs.org cert - https://phabricator.wikimedia.org/T174053#3549456 (10Andrew) [16:32:02] 10cloud-services-team (Kanban): Time to renew *.wmflabs.org cert - https://phabricator.wikimedia.org/T174053#3549468 (10Andrew) p:05Triage>03High [16:35:39] 10cloud-services-team, 10Operations: Time to renew *.wmflabs.org cert - https://phabricator.wikimedia.org/T174053#3549484 (10Andrew) [16:38:20] RECOVERY - Puppet errors on tools-redis-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [17:05:18] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [17:38:28] 10Data-Services, 10DC-Ops, 10Operations, 10ops-codfw: Split up labstore external shelf storage available in codfw between labstore2001 and 2 - https://phabricator.wikimedia.org/T171623#3549730 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by madhuvishy on neodymium.eqiad.wmnet for hosts: ```... [18:44:12] (03PS1) 10BryanDavis: Fix spelling of "Maintainers" [labs/striker] - 10https://gerrit.wikimedia.org/r/373642 [18:44:14] (03PS1) 10BryanDavis: Check tools.admin membership for admin rights [labs/striker] - 10https://gerrit.wikimedia.org/r/373643 [18:44:16] (03PS1) 10BryanDavis: striker.tools.models.Maintainer: fix __str__ [labs/striker] - 10https://gerrit.wikimedia.org/r/373644 (https://phabricator.wikimedia.org/T173963) [18:46:25] (03PS2) 10BryanDavis: striker.tools.models.Maintainer: fix __str__ [labs/striker] - 10https://gerrit.wikimedia.org/r/373644 (https://phabricator.wikimedia.org/T173963) [18:53:10] (03CR) 10BryanDavis: [C: 032] Fix spelling of "Maintainers" [labs/striker] - 10https://gerrit.wikimedia.org/r/373642 (owner: 10BryanDavis) [18:54:29] (03Merged) 10jenkins-bot: Fix spelling of "Maintainers" [labs/striker] - 10https://gerrit.wikimedia.org/r/373642 (owner: 10BryanDavis) [18:54:33] (03CR) 10BryanDavis: [C: 032] Check tools.admin membership for admin rights [labs/striker] - 10https://gerrit.wikimedia.org/r/373643 (owner: 10BryanDavis) [18:56:13] (03CR) 10BryanDavis: [C: 032] striker.tools.models.Maintainer: fix __str__ [labs/striker] - 10https://gerrit.wikimedia.org/r/373644 (https://phabricator.wikimedia.org/T173963) (owner: 10BryanDavis) [18:57:05] (03Merged) 10jenkins-bot: Check tools.admin membership for admin rights [labs/striker] - 10https://gerrit.wikimedia.org/r/373643 (owner: 10BryanDavis) [18:58:30] (03Merged) 10jenkins-bot: striker.tools.models.Maintainer: fix __str__ [labs/striker] - 10https://gerrit.wikimedia.org/r/373644 (https://phabricator.wikimedia.org/T173963) (owner: 10BryanDavis) [19:14:01] 10Striker: toolsadmin not recognising that user is assocaited with tool accounts - https://phabricator.wikimedia.org/T165009#3254084 (10bd808) I see @FriyMan listed as a maintainer on: * https://toolsadmin.wikimedia.org/tools/id/friybot * https://toolsadmin.wikimedia.org/tools/id/rfahelper There is no obvious r... [19:28:52] 10Striker: Striker does not recognize tools as maintainers - https://phabricator.wikimedia.org/T158478#3550381 (10bd808) Fixed as part of {T149458}. See https://toolsadmin.wikimedia.org/tools/id/merlbot2 as an example of the display. [19:30:36] bd808: I created a new tool via striker :D I'm wondering how to integrate the "or later" part of GPLs without adding a bunch more confusing options.. [19:30:44] 10Striker, 10User-bd808: Striker does not recognize tools as maintainers - https://phabricator.wikimedia.org/T158478#3550399 (10bd808) 05Open>03Resolved a:03bd808 [19:31:12] legoktm: a good question. I punted on it. :) [19:31:53] The best way is probably a proper LICENSE/COPYING fie in the tool itself [19:34:29] hmm [19:34:43] bd808: my new tool's shell is set to /bin/sh, not bash? [19:35:01] ugh. probably something I typoed... [19:35:28] * bd808 looks at source [19:37:13] apparently I didn't set it explicitly to anything at all, so that must be the LDAP posixUser default [19:37:15] I'll fix it [19:39:28] (03PS1) 10BryanDavis: Set tool account's shell to /bin/bash [labs/striker] - 10https://gerrit.wikimedia.org/r/373675 [19:40:07] 10Data-Services, 10DC-Ops, 10Operations, 10ops-codfw: Split up labstore external shelf storage available in codfw between labstore2001 and 2 - https://phabricator.wikimedia.org/T171623#3550443 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['labstore2002.codfw.wmnet'] ``` Of which those **FAI... [19:42:20] legoktm: thanks for catching that! the LDAP records have no shell at all so the local system is just filling something in [19:42:50] (03CR) 10Legoktm: [C: 031] Set tool account's shell to /bin/bash [labs/striker] - 10https://gerrit.wikimedia.org/r/373675 (owner: 10BryanDavis) [19:42:53] :) [19:43:24] next question... [19:43:27] http://tools.wmflabs.org/mwpackages/ says not serviced [19:43:35] tools.mwpackages@tools-bastion-03:~$ webservice status [19:43:36] Your webservice is running [19:43:43] this is a k8s/python3/flask tool [19:44:02] sometimes the tools proxy loses track of things starting... [19:44:16] I'd try `webservice restart` first [19:44:36] our ingress proxy needs to be replaced. [19:44:55] restarted, still not working :/ [19:44:55] invisible-unicorn was a cool hack be we should be able to do better today [19:45:46] hmmm... I see the pod running [19:46:21] Try webservice stop n start maybe restart didnt complete properly [19:47:12] legoktm: maybe we should fix the shell first just in case thats some related problem [19:47:52] tools.mwpackages@tools-bastion-03:~$ chsh -s /bin/bash [19:47:52] You may not change the shell for 'tools.mwpackages'. [19:48:18] yeah I think I have to fix it for you with LDAP superpowers [19:48:21] 10Toolforge, 10User-bd808: Update code and/or docs for "How can I detect if I'm running in Labs?" - https://phabricator.wikimedia.org/T174082#3550488 (10Quiddity) [19:49:14] is there maintenance going on for openstack? Nodepool seems unhappy: timeout trying to delete/build instances [19:49:47] thcipriani: no, but andrewbogott saw rabbitmq being angry earlier today [19:50:03] andrewbogott: nodepool is sucking for CI :/ [19:50:31] 10Toolforge, 10User-bd808: Update [[Help:Toolforge/Developing]] docs - https://phabricator.wikimedia.org/T174084#3550517 (10Quiddity) [19:50:32] thcipriani: when are you going to be using docker instead? ;) [19:50:33] I'm here, will keep looking... [19:50:35] I saw he restarted it, and initially nodepool seemed ok about it...took a few minutes and nodepool started showing timeouts [19:50:39] I suspect it's just overloaded... [19:50:44] Sometimes i wish nodepool was better..... [19:51:02] every day I wish we had not decided to use nodepool [19:51:13] bd808: Someday™ [19:51:24] I see builds piling up into ERRORs as well now [19:51:34] thcipriani: I've heard that estimate before ;) [19:51:45] thcipriani: can we untrademark that and trademark right-now [19:51:49] seems like it gets stuck at | OS-EXT-STS:task_state | spawning andrewbogott [19:51:54] not even getting scheduled? [19:52:49] I'm going to just stop nodepool for a minute and see if things calm down [19:53:24] andrewbogott: we should purge the intances in flight? [19:53:30] stuck in error etc? [19:53:34] yep [19:53:35] I can do that [19:53:39] ok [19:55:38] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Continuous-Integration-Infrastructure, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3550552 (10chasemp) We have been having rabbitmq and/or timeout issues with operations this... [19:56:34] chasemp: andrewbogott fyi T174083 [19:56:35] T174083: zuul seems stuck and has some jobs running for 20-40 minutes already - https://phabricator.wikimedia.org/T174083 [19:56:43] legoktm: ok, your tool's shell should be sane now [19:57:47] !lot tools restarted nslcd and nscd on tools-bastion-02 [19:58:17] legoktm: if you are logged into bastion-03 it may not notice yet... [19:58:55] bd808: !log [19:59:12] !log tools restarted nslcd and nscd on tools-bastion-02 [19:59:13] ffs [19:59:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:59:23] * legoktm logs out and in again [19:59:27] Its that kind of day i feel you [19:59:41] !log tools restarted nslcd and nscd on tools-bastion-03 [19:59:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:59:54] bd808: looks like that did it? http://tools.wmflabs.org/mwpackages/ :D [20:00:16] that's interestingly messed up [20:00:39] AGPL? [20:00:42] really? [20:00:59] yes...? most of my tools are AGPL [20:01:22] do you ever hot patch them to test a fix? [20:01:33] if so you are violating your own license. :) [20:02:33] legoktm: i suggest investing in atleast the apache license [20:02:53] Apache and AGPL are very different [20:03:03] Exactly [20:03:06] bd808: maybe my workflow sucks, but normally I just write code locally, git commit push, then pull on the server restart, and if it doesn't work, try all again [20:03:45] I just have a thing against the actual wording of AGPL. Its a nice idea but the "must publish *before* running in public" part is kind of problematic [20:03:50] (and that's unrelated to my license choice) [20:04:09] Yes it is [20:04:23] (Referring to bd808) [20:04:32] robla and I talked to Luis years ago about AGPL for MW and found that trap [20:04:48] technically AGPL would not allow us to run security patches [20:05:00] I had a conversation with slaporte about AGPL at Wikimania a few weeks ago :) [20:05:12] heh [20:05:16] did he have good things to say? [20:05:53] I'm pro copyleft these days though :) [20:06:00] was not always the case [20:06:35] his position was similar "good idea in spirit, potentially problematic in practice", but he said they have an internal checklist, and if they can ensure we'll meet the terms of the license, then its OK to use [20:08:23] if you ever want to find LDAP records that are missing an attribute (like loginShell) -- ldapsearch -xLLL -P 3 -E pr=40000/noprompt -o ldif-wrap=no -b 'ou=people,ou=servicegroups,dc=wikimedia,dc=org' '(&(objectClass=posixAccount)(!(loginShell=*))) [20:08:44] I think at least for labs where I have no need for security patches and want to ensure source code is available, AGPL works [20:09:13] yeah. I should not have acted like it was crazy. sorry for that [20:09:18] what about EUPL? [20:09:19] no worries :) [20:09:29] but it is a challenging license to comply wiht [20:09:42] plus, what kind of FSF/SFC card-carrying member would I be if I didn't use and encourage strong copyleft? ;) [20:10:15] * bd808 leaves his card in a drawer at home most days ;) [20:10:44] I'm ready for the post-copyright world to win [20:11:13] the whole universe is just a big tumblr of mashups :) [20:11:19] heh, class time now, ttyl. thanks for help :) [20:12:32] bd808: strange as it is, I'm all for proprietary licenses sometimes :P [20:13:01] my PhD is all rights reserved for example :P [20:13:08] TabbyCat: ewww i love foss licenses or atleast os licenses [20:13:56] my 'leftist' rant is that copyright only really protects the wealthy actors in the system. its a means to suppress competition that is expensive to actually enforce [20:14:50] there is no copyright public defender pool nor right to state provided representation if you can't afford to enforce the terms of your license [20:15:19] so Disney benefits from it but a random street artist does not [20:15:29] the Attorney General of my country does [20:15:37] I mean, on some cases [20:21:43] 10Cloud-Services, 10Design: Update Cloud Services logo+wordmark - https://phabricator.wikimedia.org/T174094#3550694 (10Quiddity) [20:22:01] 10Cloud-Services, 10Design: Update Cloud Services logo+wordmark - https://phabricator.wikimedia.org/T174094#3550730 (10Quiddity) [20:22:04] 10cloud-services-team (FY2017-18), 10Goal, 10Patch-For-Review, 10User-bd808: Perform initial Cloud Services rebranding - https://phabricator.wikimedia.org/T168480#3550731 (10Quiddity) [20:28:22] (03PS2) 10BryanDavis: Set tool account's shell to /bin/bash [labs/striker] - 10https://gerrit.wikimedia.org/r/373675 [20:29:13] 10cloud-services-team (FY2017-18), 10Goal, 10Patch-For-Review, 10User-bd808: Perform initial Cloud Services rebranding - https://phabricator.wikimedia.org/T168480#3550746 (10Quiddity) [20:43:25] (03CR) 10BryanDavis: "recheck" [labs/striker] - 10https://gerrit.wikimedia.org/r/373675 (owner: 10BryanDavis) [20:46:08] TabbyCat: you know that script of yours that fixes redirects? [20:46:27] meow? [20:46:39] Meooow? [20:47:19] harej: I operate a couple of scripts that fix redirects but they're part of the pywikibot suite [20:47:37] I lately only run redirect.py cronned [20:48:02] TabbyCat: is it based on inbound links or does it go to a given page and check for redirects? [20:48:40] harej: redirect.py has two modes afaik: fetching special:(double|broken)redirects and fixing what's in there [20:49:02] or the "live" option which do not wait and fix double/broken redirects as they appear [20:49:11] I think AvicBot@meta uses that [20:49:39] it only fixes page moves though, not links on them [20:52:38] harej: got any double or broken redirect fixing to do? [20:53:14] Not double or broken, no [20:53:55] More that ever since I moved all of those pages out of the grants namespace, a ton of links are redirects which is mildly annoying. But nothing actually broken. [20:58:37] (03CR) 10BryanDavis: [V: 032 C: 032] Set tool account's shell to /bin/bash [labs/striker] - 10https://gerrit.wikimedia.org/r/373675 (owner: 10BryanDavis) [20:59:43] hmm, maybe it's the fixing_redirects.py script what we need [21:00:19] but it's an annoying script to run as we have to go page by page; that might have changed in the meanwhile though. It's been some time I don't use it. [21:48:45] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:13:21] 10Striker, 10cloud-services-team (Kanban), 10Patch-For-Review, 10User-bd808: Unable to add new LDAP user as maintainer of a tool - https://phabricator.wikimedia.org/T173963#3551227 (10bd808) This really feels like an in-process cache bug the more I work on it. Each time the wsgi container in either of my t... [22:18:49] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [22:24:49] 10Tools: Add code snippets with integrity hashes to CDNJS interface - https://phabricator.wikimedia.org/T173996#3551255 (10Samwilson) Could we just scrape the upstream page and extract the hashes or whole snippets from there? Or is that juts a bit too hacky? :-) [22:44:04] 10Striker, 10User-Zppix: Striker: 500 error when going to settings -> ssh keys - https://phabricator.wikimedia.org/T174112#3551272 (10Zppix) [22:51:04] 10Data-Services, 10DC-Ops, 10Operations, 10ops-codfw: Split up labstore external shelf storage available in codfw between labstore2001 and 2 - https://phabricator.wikimedia.org/T171623#3551301 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by madhuvishy on neodymium.eqiad.wmnet for hosts: ```... [22:58:46] (03PS1) 10BryanDavis: Break in-process cache of model choice fields [labs/striker] - 10https://gerrit.wikimedia.org/r/373708 (https://phabricator.wikimedia.org/T173963) [22:58:48] (03PS1) 10BryanDavis: Guard against LDAP modify with no changes for maintainers [labs/striker] - 10https://gerrit.wikimedia.org/r/373709 [23:16:03] 10Striker, 10User-Zppix: Fatal error when SSH key data is invalid - https://phabricator.wikimedia.org/T174112#3551396 (10bd808) [23:18:53] 10Striker, 10User-Zppix: Fatal error when SSH key data is invalid - https://phabricator.wikimedia.org/T174112#3551403 (10Zppix) How are my ssh keys imvalid? I use them all the time. [23:21:49] 10Tools: Add code snippets with integrity hashes to CDNJS interface - https://phabricator.wikimedia.org/T173996#3551422 (10bd808) >>! In T173996#3551255, @Samwilson wrote: > Could we just scrape the upstream page and extract the hashes or whole snippets from there? Or is that juts a bit too hacky? :-) The hard... [23:55:18] 10Striker, 10User-Zppix: Fatal error when SSH key data is invalid - https://phabricator.wikimedia.org/T174112#3551272 (10bd808) There are 11 `sshPublicKey` attributes attached to the `uid=zppix1,ou=people,dc=wikimedia,dc=org` LDAP object. As shown in the stack traces above, 3 of these keys are encoded in such...