[00:02:21] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [00:35:03] PROBLEM - Puppet errors on tools-exec-1428 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:44:24] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:45:12] 10Toolforge, 10Collaboration-Team-Triage, 10Flow, 10labs-sprint-119: Make Flow database available / accessible on Labs/Tools - https://phabricator.wikimedia.org/T69397#3476856 (10Mattflaschen-WMF) [00:45:28] 10Toolforge, 10Collaboration-Team-Triage, 10Flow, 10labs-sprint-119: Make Flow database available / accessible on Labs/Tools - https://phabricator.wikimedia.org/T69397#3476859 (10Mattflaschen-WMF) [00:45:30] 10Cloud-Services, 10Toolforge: Set up replica from extension1 cluster - https://phabricator.wikimedia.org/T121742#3476858 (10Mattflaschen-WMF) [00:45:41] 10Toolforge, 10Collaboration-Team-Triage, 10Flow, 10labs-sprint-119: Make Flow database available / accessible on Labs/Tools - https://phabricator.wikimedia.org/T69397#713063 (10Mattflaschen-WMF) [00:46:22] 10Data-Services, 10Collaboration-Team-Triage, 10Flow, 10labs-sprint-119: Make Flow database available / accessible on Labs/Tools - https://phabricator.wikimedia.org/T69397#3476861 (10bd808) [00:48:31] 10Data-Services, 10Collaboration-Team-Triage, 10Flow, 10labs-sprint-119: Make Flow database available / accessible on Labs/Tools - https://phabricator.wikimedia.org/T69397#713063 (10Mattflaschen-WMF) [01:10:01] RECOVERY - Puppet errors on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [01:24:26] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [02:58:21] 10wikitech.wikimedia.org, 10MediaWiki-extensions-Linter, 10Parsoid, 10Services: On wikis without changeprop enabled, lint errors don't update after page edits - https://phabricator.wikimedia.org/T171788#3476964 (10bd808) [03:03:24] 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10MW-1.30-release-notes (WMF-deploy-2017-07-25_(1.30.0-wmf.11)): novaadmin removed from many keystone projects - https://phabricator.wikimedia.org/T171313#3476970 (10bd808) a:03Andrew [03:10:54] 10Tool-Pageviews: Totals for Edits and Editors are not updated when pages are removed from the analysis - https://phabricator.wikimedia.org/T169848#3476975 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal This has been fixed! Thank you for the report :) [03:13:10] If I authorized a new key via wiki tech, how long until that key is syndicated across the cloud VPSes? [03:13:47] I can SSH into Bastion but it complains when I try to then log in to wpx-data-01 [03:26:42] Never mind, figured it out [03:32:09] 10Tool-Pageviews: Add support for the uselang (or similiar) parameter - https://phabricator.wikimedia.org/T171015#3450823 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal Done! http://tools.wmflabs.org/pageviews/?project=de It looked like the i18n library I was using would automatically support qqx, but... [03:51:44] PROBLEM - Puppet errors on tools-exec-1440 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [04:11:44] RECOVERY - Puppet errors on tools-exec-1440 is OK: OK: Less than 1.00% above the threshold [0.0] [05:09:31] 10cloud-services-team (FY2017-18), 10Goal: Program 10 Outcome 3: Outreach - https://phabricator.wikimedia.org/T166406#3477026 (10bd808) [05:09:33] 10cloud-services-team (Kanban), 10User-bd808: Design and submit Wikimania poster art for "What is Cloud Services?" - https://phabricator.wikimedia.org/T169927#3477024 (10bd808) 05Open>03Resolved https://commons.wikimedia.org/wiki/File:What_is_Cloud_Services%3F_poster.pdf [05:10:01] 10cloud-services-team (FY2017-18), 10Goal: Program 10 Outcome 3: Outreach - https://phabricator.wikimedia.org/T166406#3295299 (10bd808) [05:10:03] 10cloud-services-team (Kanban), 10User-bd808: Design and submit Wikimania poster art for "FLOSS best practices for bots and tools" - https://phabricator.wikimedia.org/T169919#3477027 (10bd808) 05Open>03Resolved https://commons.wikimedia.org/wiki/File:FLOSS_Best_Practices_for_Bots_and_Tools_poster.pdf [05:17:17] !help I setup-ed the webservice for python, so how can I "use only browser to run my code" now...? [05:17:18] r96340: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [05:18:06] r96340: what's the name of your tool? it should be running at tools.wmflabs.org/ [05:22:17] So I just put a link to my python code on index page, it will run itself instead of download it? [05:24:29] r96340: I'm not sure what you mean, if you run a python webservice by doing webservice start on your tool account, you can access it from that link [05:27:21] I wanna build a browser-active pywikibot, not using any clinet, and someone tell me I need to setup a webservice for python. [05:36:32] browser-active pywikibot => PAWS [05:36:41] the service exists already [05:48:05] I will check it. thx [06:58:58] PROBLEM - Puppet errors on tools-bastion-05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:38:55] RECOVERY - Puppet errors on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:53:21] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:10:53] 10Cloud-VPS, 10Operations, 10monitoring, 10Patch-For-Review, 10User-fgiunchedi: Diamond collectors collects NFS statistics on Cloud-VPS - https://phabricator.wikimedia.org/T171583#3477178 (10fgiunchedi) @zhuyifei1999 looks like this is fixed, confirmed? [08:10:55] 10PAWS: PAWS doesn't display space character - https://phabricator.wikimedia.org/T171820#3477179 (10R96340) [08:13:22] 10PAWS: PAWS doesn't display space character - https://phabricator.wikimedia.org/T171820#3477198 (10R96340) [08:28:22] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [10:20:09] 10Cloud-Services, 10DBA: Prepare and check storage layer for hi.wikiversity - https://phabricator.wikimedia.org/T171829#3477438 (10MarcoAurelio) [10:21:04] 10Cloud-Services, 10DBA: Prepare and check storage layer for hi.wikiversity - https://phabricator.wikimedia.org/T171829#3477438 (10jcrespo) Thanks, please let us know when the database is actually created. [10:29:08] 10Cloud-Services, 10DBA: Prepare and check storage layer for hi.wikiversity - https://phabricator.wikimedia.org/T171829#3477438 (10MarcoAurelio) @jcrespo You're welcome. I think @Dereckson said he'll do that so I'll add him to this task so he can notify you. [10:31:44] 10VPS-project-Wikistats: Add hi.wikiversity to wikistats - https://phabricator.wikimedia.org/T171831#3477487 (10MarcoAurelio) [10:41:00] 10Cloud-VPS, 10Operations, 10monitoring, 10Patch-For-Review, 10User-fgiunchedi: Diamond collectors collects NFS statistics on Cloud-VPS - https://phabricator.wikimedia.org/T171583#3477525 (10zhuyifei1999) 05Open>03Resolved a:03chasemp Yep, thanks :) [11:39:54] ㅆ지 [11:40:24] Worng channel, please forget above msg [11:57:22] 10Cloud-Services, 10cloud-services-team (FY2017-18), 10Datasets-General-or-Unknown: Setup periodic rsync jobs from dataset1001/dumpsdata1001|2 to labstore1006|7 - https://phabricator.wikimedia.org/T171541#3477771 (10ArielGlenn) My thoughts on this task: - If you need the data to be available right away so... [12:00:38] 10cloud-services-team (FY2017-18), 10Datasets-General-or-Unknown, 10Goal: Begin migrating customer-facing Dumps endpoints to Cloud Services - https://phabricator.wikimedia.org/T168486#3477774 (10ArielGlenn) A small correction to the above: there are indeed two dump runs per month, but the first run, starting... [14:17:08] 10Cloud-VPS (Quota-requests), 10Recommendation-API, 10User-schana: Request custom instance for recommendation-api labs project - https://phabricator.wikimedia.org/T169766#3478186 (10schana) [14:23:01] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:56:58] 10Cloud-Services, 10Operations: wikitech api list=novainstances not returning list of instances - https://phabricator.wikimedia.org/T171280#3478310 (10Andrew) [14:57:02] 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10MW-1.30-release-notes (WMF-deploy-2017-07-25_(1.30.0-wmf.11)): novaadmin removed from many keystone projects - https://phabricator.wikimedia.org/T171313#3478308 (10Andrew) 05Open>03Resolved Attached patch should resolve the ultimate cause. [14:57:59] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [14:59:34] (03CR) 10BryanDavis: [C: 032] Update composer.json [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/367879 (owner: 10MarcoAurelio) [15:00:01] (03Merged) 10jenkins-bot: Update composer.json [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/367879 (owner: 10MarcoAurelio) [15:23:59] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:59:02] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [16:10:27] !log tools Disabled puppet on tools-static-11 to test https://gerrit.wikimedia.org/r/#/c/357878 [16:10:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:11:39] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Operations: Switch to new labs puppetmasters - https://phabricator.wikimedia.org/T171786#3478669 (10Andrew) [16:25:55] bd808: I'll merge that now for you [16:26:06] thanks chasemp [16:27:01] !log tools Enabled puppet on tools-static-11 [16:27:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:28:04] (jeopardy theme for jenkins...) [16:28:58] I think it's more the baby elephant waltz [16:29:43] https://www.youtube.com/watch?v=oYu7c4Vkmp0 [16:30:14] hm [16:30:16] I thought of [16:30:16] https://www.youtube.com/watch?v=PrQQZuLru9s [16:31:01] it uh forced me to rebase by the time jenkins caught up :) [16:32:05] (that inevitably leads to https://www.youtube.com/watch?v=5dhSdnDb3tk obv) [16:36:40] chasemp: heh. I can sing all the songs from that movie. I even have stuffed a Baloo and King Louie to act out the scenes. [16:38:04] :) [16:42:45] bd808 chasemp: looks live now, thanks [16:43:54] something in https://tools.wmflabs.org/fontcdn/ is still trying to hit fonts.googleapis [16:45:13] yeah, I'm looking into it [16:45:32] a lot of 404s as well [16:46:54] zhuyifei1999_: one of the 404's is "/fontcdn/l/font?kit=wwAFjRMmw2eVBgWuYkS8oaMc2z3SecJ4mIP4cJh0JHIlDVtwWiDWhiSptYdXoExr6TMu0t-E2O7euEkpxJohTO2w0J-avqcUChVUvGkjwhk&skey=a0a0114a1dcab3ac&v=v16" [16:47:04] we don't have a rule for /l/ [16:47:30] hmm [16:47:55] looking at the source page they are something like https://fonts.gstatic.com/l/font?kit=wwAFjRMmw2eVBgWuYkS8oaMc2z3SecJ4mIP4cJh0JHIlDVtwWiDWhiSptYdXoExr6TMu0t-E2O7euEkpxJohTO2w0J-avqcUChVUvGkjwhk&skey=a0a0114a1dcab3ac&v=v16 [16:48:20] weird [16:49:02] sounds like that needs another block like the "location ^~ /fontcdn/s/" one [16:49:21] !log git cherrypicking https://gerrit.wikimedia.org/r/#/c/368196/ to see if it has side affects [16:49:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [16:50:15] zhuyifei1999_: maybe we should change the rules so that /css gets the special rewrite and then the rest just passes to fonts.gstatic? [16:50:16] !log git restarting gerrit-test3 gerrit to pickup https://gerrit.wikimedia.org/r/#/c/368196/ [16:50:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [16:50:27] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:50:58] but what is /l/font? [16:51:06] * zhuyifei1999_ gogles [16:51:59] https://github.com/alexvcasillas/Google-Fonts-JSON-to-Google-Font-Format/blob/master/google-fonts.json SVGs? [16:52:27] its returning mime-type font/woff2 [16:54:44] it looks like it does several things.. not sure if it embeds in a way that needs rewriting... [16:55:05] hmm... https://tools-static.wmflabs.org/fontcdn/css?family=Roboto%7COpen+Sans%7CLato%7CSlabo+27px%7CRoboto+Condensed%7COswald%7CMontserrat%7CSource+Sans+Pro&text=abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 [16:55:11] they are referred to here [16:56:12] probably that's what l/font is for, generating font files dynamically based on eg. '&text=abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789' [16:56:46] ah that would make some sense. for limiting the embedded glyphs [16:57:30] look a this nice write up :) -- https://www.codeday.top/2017/07/02/29658.html [16:58:30] I've got some time, so let me see if I can make a new config and test it out [16:59:09] k thanks [17:00:16] I still find it odd how my browser still seems to be caching the font css. They don't seem to appear in the 'Network' tab but appear in 'Style editor' tab of my console. [17:00:24] (Firefox here) [17:25:28] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:37] zhuyifei1999_: I think I got it working at https://tools-static-beta.wmflabs.org/fontcdn/... -- /css, /s, and /l are all responding now [17:28:13] ok [17:28:40] I need to figure out how to force reload the font css in webfontloader [17:40:39] Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://tools-static.wmflabs.org/fontcdn/l/font?kit=XoJC5RLXTdH-FJ_w8JTaFaMc2z3SecJ4mIP4cJh0JHIlDVtwWiDWhiSptYdXoExr6TMu0t-E2O7euEkpxJohTO2w0J-avqcUChVUvGkjwhk&skey=653237e53512d0de&v=v13. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). [17:40:41] o.O [17:41:16] zhuyifei1999_: I just applied the /l/ change there. Like after your irc message... [17:42:01] hmm [17:42:22] but it is still 404'ing? .... [17:42:47] no, CORS seems blocking it [17:44:02] our response is a 404 which is probably the CORS issue [17:44:15] * bd808 checks that request with upstream [17:45:03] zhuyifei1999_: that's a 404 at fonts.gstatic too [17:46:29] uh, no period in the end. that's part of the console mesage in Firefox [17:46:51] I do see a "Access-Control-Allow-Origin: *, *" in the response headers though [17:48:13] https://stackoverflow.com/questions/42338672/cors-header-access-control-allow-origin-does-not-match-null-but-it-is-not [17:48:16] (03PS1) 10Andrew Bogott: add some stub certs to fix puppet compiler [labs/private] - 10https://gerrit.wikimedia.org/r/368222 [17:48:27] The 'Access-Control-Allow-Origin' header contains multiple values '*, *', but only one is allowed [17:48:30] huh? [17:48:54] ah... we are adding and there is one from upstream too [17:49:22] * zhuyifei1999_ tries chromium [17:49:42] https://phabricator.wikimedia.org/P5812 -- "Access-Control-Allow-Origin: *" is in there twice [17:50:32] (03CR) 10Andrew Bogott: [V: 032 C: 032] add some stub certs to fix puppet compiler [labs/private] - 10https://gerrit.wikimedia.org/r/368222 (owner: 10Andrew Bogott) [17:50:43] o.O [17:50:54] since we always add it I guess we should be stripping any Access-Control-Allow-Origin header from the upstream [17:53:49] actually we should just not add it [17:54:05] the upstream headers here will be correct for * origin [17:54:40] the upstream headers here will be correct for * origin <= I don't think I get this [17:55:08] gstatic will always add when appropriate [17:55:17] so we don't need to force it as well [17:55:51] so instead of adding our own header and hiding theirs we can just not add the header [17:59:08] yeah, I just checked the css and they are set as well [18:00:17] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Operations: Add AAAA records for labpuppetmaster1001 and 1002 - https://phabricator.wikimedia.org/T171880#3479078 (10Andrew) [18:13:11] zhuyifei1999_: double access-control headers should be fixed now. [18:13:28] k [18:14:00] I also found where I screwed up in the loading URIs [18:16:07] bd808: all is well now? [18:16:19] chasemp: I hope so :) [18:16:45] tools-static is fine. I hope we swatted the little bug [18:17:57] !log tools.stewardbots Merge "Update composer.json" [18:18:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [18:19:10] * zhuyifei1999_ is having trouble rebuilding the js. not a js native here :/ [18:19:45] zhuyifei1999_: it never helps that every js project uses a completely different tool chain [18:19:57] lol [18:22:33] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Operations, 10Patch-For-Review: Add AAAA records for labpuppetmaster1001 and 1002 - https://phabricator.wikimedia.org/T171880#3479152 (10Dzahn) a:05Andrew>03Dzahn [18:25:27] 10cloud-services-team (FY2017-18), 10Goal: Refactor openstack Puppet to account for Neutron - https://phabricator.wikimedia.org/T171494#3479169 (10chasemp) === The case of multiple deployments of interrelated roles within a site that need to share inter-profile values Prod puppet backends and tree: ``` -... [18:29:21] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Operations, 10Patch-For-Review: Add AAAA records for labpuppetmaster1001 and 1002 - https://phabricator.wikimedia.org/T171880#3479190 (10Dzahn) ``` @labpuppetmaster1001:~# host labpuppetmaster1001 labpuppetmaster1001.wikimedia.org has address 208.80.154.158 la... [18:30:54] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Operations, 10Patch-For-Review: Add AAAA records for labpuppetmaster1001 and 1002 - https://phabricator.wikimedia.org/T171880#3479200 (10Dzahn) ``` @labpuppetmaster1001:~# ping6 labpuppetmaster1001.wikimedia.org PING labpuppetmaster1001.wikimedia.org(labpuppet... [18:33:12] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Operations: Switch to new labs puppetmasters - https://phabricator.wikimedia.org/T171786#3479205 (10Dzahn) [18:33:15] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Operations, 10Patch-For-Review: Add AAAA records for labpuppetmaster1001 and 1002 - https://phabricator.wikimedia.org/T171880#3479204 (10Dzahn) 05Open>03Resolved [18:34:14] !log tools.fontcdn attempting to update to e900070 (weird git hash, I know) [18:34:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.fontcdn/SAL [18:48:57] !log tools.fontcdn Updating to 022bb0a [18:48:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.fontcdn/SAL [18:50:24] bd808: https://tools.wmflabs.org/fontcdn/ looks all from tools-static now. thanks again :) [18:58:00] zhuyifei1999_: nice! Now we need to figure out how to promote it and cdnjs [18:58:40] Would you be interested in writing a blog post that we put on the cloud services phame blog? [18:59:04] hmm [18:59:28] I don't think I'm good at writing blogs [18:59:48] $ git grep '[Ss]earch tool for' [18:59:48] README.md:FontCDN is a search tool for [Google Fonts](https://www.google.com/fonts). Check it out at [fontcdn.org](http://fontcdn.org). [18:59:48] index.html: FontCDN: A search tool for Google web fonts [18:59:48] index.html: [18:59:57] * zhuyifei1999_ needs to fix this ^ [19:01:40] zhuyifei1999_: maybe you could get quiddity to work on it with you. I bet if you helped him understand the "why" he'd be able to write it up. [19:02:00] * bd808 is trying not to lick another cookie ;) [19:02:59] quiddity: oh uh [19:03:17] ;) ikr! But yeah, I can probably help with that. [19:03:45] let me fix index.html and readme.md first [19:20:42] !log tools.fontcdn Updating to a6c5276 [19:20:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.fontcdn/SAL [19:23:38] quiddity: ok so basically some tools like to use fonts to decorate them [19:24:12] google has a large collection of open source and freely licensed fonts [19:25:24] it also has a nice CSS api for browser compatibility [19:26:02] so many people wants to use, well, google fonts [19:26:40] but here's the problem: when people download the fonts, they download from google servers [19:28:01] so google may be able to track our users with IPs, referrer headers, etc. [19:28:56] that isn't very privacy-policy-friendly. last time I checked I was unable to find a discussion on it [19:29:59] nod. [19:30:09] to workaround this, prior to T110027, tool owners have to fork the fonts and store them locally in the tool [19:30:09] T110027: Create a fonts CDN for use on Tool Labs - https://phabricator.wikimedia.org/T110027 [19:31:19] along with the css for browser compatibility. that is obviously less than ideal. (eg. what if you want to change a font?) [19:31:30] see also: T103934, T129936, T133919 [19:31:30] T133919: [EPIC] Protect end-user privacy by restricting non-consentual third-party browser interactions - https://phabricator.wikimedia.org/T133919 [19:31:30] T103934: Convert all Labs tools to use cdnjs for static libraries and fonts - https://phabricator.wikimedia.org/T103934 [19:31:30] T129936: Ensure that Terms of Use document restrictions on third-party web interactions - https://phabricator.wikimedia.org/T129936 [19:31:45] (eg. what if the google upstream updates the font) [19:32:52] another workaround is to remove webfonts from the tool, but that's not the point [19:33:45] so we first checked if the font files are available via bulk download anywhere, sort of like cdnjs [19:34:06] but nothing... [19:34:28] so we went ahead and created a reverse proxy [19:35:37] another font-searching interface is added to the tool to ease the finding of altered font css urls [19:36:55] now with the proxy whenever you fetch the fonts, you fetch from the proxy, which removes all the referrer, cookie, ip information and ask google for the actual css / font files [19:37:15] so google can't track our users :) [19:37:45] I think the story to tell is basically that we want to make it easy for our Toolforge developers to protect the privacy of their users, so we have /cdnjs/ and now /fontcdn/ as replacements for popular 3rd party CDNs. [19:38:01] both really need wikitech page(s) too [19:38:11] and cross linking from webservice docs [19:38:43] building the service is easy, getting it used is the hard part :) [19:38:51] btw, I find my title "Search and Proxy for Google web fonts" awkward, any suggestions? [19:38:55] bd808: yeah [19:41:06] the discussions are really early. the tool "static" was probably created before all the (related) tasks in your list [19:42:08] "static" itself was to have a central place to store all the cloned/forked js/css libraries like jquery and bootstrap [19:42:58] https://tools.wmflabs.org/static-browser/ 500 o.O [19:43:22] it says "FastCGI-stderr: TemplateNotFound: index.html" [19:44:03] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login, OAuth error: API error mwoauth-invalid-authorization - https://phabricator.wikimedia.org/T136114#3479496 (10MisterSynergy) @yuvipanda and others: I am experiencing this problem as well in PAWS (`WARNING: API error mwoauth-invalid-au... [19:44:18] https://github.com/tool-labs/static/commits/master?after=ff2064eed3c127570154a413a2e17596dbfc56ec+34 the repo started in oct 1 2013 [19:51:39] 10Toolforge, 10Patch-For-Review: Create a fonts CDN for use on Tool Labs - https://phabricator.wikimedia.org/T110027#3479502 (10zhuyifei1999) Mirroring the fontcdn web interface repo to R2118 [19:56:02] bd808: I thought we had a ToU change consultation from legal ages ago. I'm wondering, did anything change? [19:56:21] zhuyifei1999_: it's still stuck in review with WMF legal [19:56:37] * zhuyifei1999_ facepalm [19:56:39] I poked some folks about it a couple of weeks ago [20:05:36] PROBLEM - Puppet errors on tools-exec-1404 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:23:11] zhuyifei1999_, something like this, perhaps? https://etherpad.wikimedia.org/p/cloudfontcdn --- I've got to focus on something else for a few hours, but will be back. [20:24:35] * zhuyifei1999_ needs to sleep in a few mins [20:26:48] nod, no rush. :) I need to get lunch, too... [20:32:53] Cyberpower678: since I noticed your iabot interface use fonts.gstatic last time, you might want to try it :) uris for the fonts provided in https://tools.wmflabs.org/fontcdn/. oh and tell us in case you like your tool as an "example of the feature in action" [20:45:35] RECOVERY - Puppet errors on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [21:12:02] 10cloud-services-team (FY2017-18), 10Goal: Refactor openstack Puppet to account for Neutron - https://phabricator.wikimedia.org/T171494#3479777 (10chasemp) [21:16:11] !log tools Disabled puppet on tools-proxy-01 to test nginx proxy config changes [21:16:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:21:30] !lot tools.admin Updated tool-admin-web to 3830db8 [22:21:43] !log tools.admin Updated tool-admin-web to 3830db8 [22:21:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [22:22:20] !log tools.admin Attempting to switch webservice to tool-admin-web [22:22:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [22:27:07] !log tools.admin tool-admin-web is live. Switch back with ~/old.sh if things go sideways. (T140254) [22:27:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [22:27:09] T140254: Modernize the admin tool's codebase - https://phabricator.wikimedia.org/T140254 [22:29:57] 10Cloud-Services, 10Toolforge, 10cloud-services-team (Kanban), 10User-bd808: Modernize the admin tool's codebase - https://phabricator.wikimedia.org/T140254#3479923 (10bd808) 05stalled>03Open It's alive! I'm going to leave this open for a couple of days in case the prior broken behavior resurfaces. [22:38:14] 10Cloud-VPS, 10Puppet: role::puppet::self: Unable to locate package geoipupdate - https://phabricator.wikimedia.org/T171916#3479983 (10MaxSem) [22:38:59] 10Cloud-VPS, 10Puppet: role::puppet::self: Unable to locate package geoipupdate - https://phabricator.wikimedia.org/T171916#3479996 (10MaxSem) [22:54:41] 10cloud-services-team (FY2017-18), 10Goal, 10Patch-For-Review: Create a detailed migration plan for implementing Neutron as our OpenStack SDN layer - https://phabricator.wikimedia.org/T167559#3480046 (10chasemp) groupings at the moment ```labcontrol1001.wikimedia.org labcontrol1002.wikimedia.org labcontrol1... [22:56:43] bd808: is there any way to have puppet to re-load whatever it loader from Hiera: space on wikitecth? I've edited it but local puppet still seems to have the old data [22:57:21] changes should take effect on the next puppet run... [22:57:30] I don't think there is any caching there [22:57:50] bd808: hmm... then I must be missing something [22:57:52] did you force a puppet run after the edit? [22:58:08] yes... and I get this: Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://wdqs-puppetmaster.wikidata-query.eqiad.wmflabs/plugins: getaddrinfo: Name or service not known [22:58:26] this is an old puppetmaster setting, which I deleted... but somehow it still is active [22:58:55] hmmmm [22:59:20] should I manually edit puppet.conf? [22:59:26] so, are you trying to change a host to a new puppetmaster? [22:59:43] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Operations: Switch to new labs puppetmasters - https://phabricator.wikimedia.org/T171786#3480067 (10Andrew) [23:00:11] well, kinda... I created a new one but turns out I already had this name a while back, and it had custom puppetmaster setting in hiera [23:00:26] so I'm trying to remove that since I no longer use custom puppetmaster [23:00:50] bd808: ah, looks like manually editing puppet.conf worked [23:00:50] *nod* [23:01:09] hopefully nothing would explode later... [23:01:22] yeah. thats pretty much all you can do if the file is pointing to a dead puppetnmaster [23:01:33] yep that was the case [23:02:49] it will probably be ok. if you are back to pointing at the main Cloud VPS puppetmaster and never got a good run in-between the certs and everything should still be in-place on both sides [23:03:09] ok, cool, thanks [23:09:19] !lot tools Re-enabled puppet on tools-proxy-01 [23:17:22] !log tools.grid-jobs Re-enabled hourly cron after patching for T168653 with 3f1ea69 [23:17:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.grid-jobs/SAL [23:17:25] T168653: grid-jobs tool broken; loads forever with no actual response - https://phabricator.wikimedia.org/T168653 [23:18:42] 10Tools, 10cloud-services-team (Kanban), 10User-bd808: grid-jobs tool broken; loads forever with no actual response - https://phabricator.wikimedia.org/T168653#3480115 (10bd808) 05Open>03Resolved a:05bd808>03zhuyifei1999 I pushed the patch, but @zhuyifei1999 did all the hard work. :) [23:21:25] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:25:52] When will people learn not to run heavy things on tools-login? [23:27:24] !log tools Killed python procs owned by sdesabbata on tools-login that were stealing all cpu/io [23:27:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:29:24] bd808: any idea what they were running? [23:30:08] "python Code/get_geopages_users_wmflabtools.py" [23:30:42] someone else is running celery here too! [23:30:48] grumble [23:31:09] hm [23:38:12] celery is horrible, gets stuck between my teeth. [23:54:49] !lot tools.zppixbot Adjusted cron to only update once very 10 minutes instead of once per minute [23:55:05] nice [23:56:25] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [23:58:55] HaeB: I think your blogconverter tool is sick. webservice has started it 14,737 times in the last week.