[00:13:20] RECOVERY - Puppet errors on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [00:13:25] PROBLEM - Puppet errors on tools-exec-1401 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:20:28] 10Cloud-Services, 10Toolforge, 10cloud-services-team (Kanban): Build new tools puppetmaster - https://phabricator.wikimedia.org/T169350#3488189 (10bd808) A local machine setup for using clush can be found at https://github.com/bd808/wikimedia-cloud-vps-hostgroup-generator and P5831. This can temporarily brid... [00:48:22] RECOVERY - Puppet errors on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [01:03:32] PROBLEM - Puppet errors on tools-exec-1417 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:43:32] RECOVERY - Puppet errors on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [05:32:29] 10cloud-services-team, 10DBA: Compress InnoDB on db1102 - https://phabricator.wikimedia.org/T172169#3488463 (10Marostegui) [07:04:34] 10Cloud-Services, 10PAWS, 10Tools-Kubernetes, 10Kubernetes, 10Patch-For-Review: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3488524 (10yuvipanda) 05Open>03Resolved a:03yuvipanda Yup, new cluster! [07:05:07] 10cloud-services-team (FY2017-18), 10Goal: Program 10 Outcome 2: Rebranding - https://phabricator.wikimedia.org/T166404#3488530 (10yuvipanda) [07:05:11] 10Cloud-Services: Use wmcloud.org domain for PAWS - https://phabricator.wikimedia.org/T168686#3488528 (10yuvipanda) 05Open>03declined Nah, don't care for now. Can move whenever other stuff moves :) [07:50:23] PROBLEM - Puppet errors on tools-exec-1433 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:30:24] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [11:21:43] hmmmmm [11:51:19] !log phabricator cherry picking https://gerrit.wikimedia.org/r/#/c/369001/ to test for any errors. [11:51:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL [12:51:19] 10cloud-services-team, 10DBA: Compress InnoDB on db1102 - https://phabricator.wikimedia.org/T172169#3489024 (10Marostegui) [12:51:42] !log quarry Deployed ba54a61 on quarry-main-01 T164390 [12:51:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [12:51:45] T164390: Quarry should remember my login - https://phabricator.wikimedia.org/T164390 [12:57:06] 10Quarry, 10Patch-For-Review: Quarry should remember my login - https://phabricator.wikimedia.org/T164390#3489031 (10zhuyifei1999) 05Open>03Resolved a:03zhuyifei1999 It should now remember login for 31 days (default). If this is too long or too short, or if a "remember my login" checkbox should be added,... [13:59:18] hi yall! [13:59:26] "In a project you can create as many instances as you'd like up to its quota. A wikitech admin can increase a project's limits as needed." :) [14:01:08] ottomata: ? [14:01:14] https://phabricator.wikimedia.org/T172186 [14:01:44] ottomata: I'll put it on the list for today's meeting [14:01:54] thank you [14:01:57] 10Cloud-Services: Increase quota for analytics project in Cloud - https://phabricator.wikimedia.org/T172186#3489174 (10Ottomata) [14:33:13] 10Tool-stewardbots, 10Continuous-Integration-Config, 10Patch-For-Review, 10User-MarcoAurelio: Implement jenkins tests on labs/tools/stewardbots - https://phabricator.wikimedia.org/T128503#3489258 (10MarcoAurelio) 05Open>03Resolved a:03MarcoAurelio I think this is resolved for now. In case we need mor... [15:04:57] 10Cloud-VPS (Quota-requests): Increase quota for analytics project in Cloud - https://phabricator.wikimedia.org/T172186#3489462 (10bd808) [15:05:46] ottomata: did the page you found that description on not have a link to https://phabricator.wikimedia.org/project/view/2880/ ? [15:06:31] apparently, no. [15:06:34] * bd808 fixes [15:11:15] nope! :) [15:11:33] ottomata: it does now. wikis are cool [15:12:07] thank you bd808, your wiki skills are impressive [15:12:33] only over here in the weird wiki that everyone ignores ;) [15:15:25] 10Cloud-VPS (Quota-requests): Increase quota for analytics project in Cloud - https://phabricator.wikimedia.org/T172186#3489484 (10chasemp) p:05Triage>03Normal [15:16:34] 10Cloud-VPS (Quota-requests): Increase quota for analytics project in Cloud - https://phabricator.wikimedia.org/T172186#3489163 (10chasemp) I'm reading this as requesting total capacity increase on top of existing for 4 medium sized instances. [15:20:13] 10Cloud-VPS (Quota-requests): Temporary increased quota for analytics project in Cloud - https://phabricator.wikimedia.org/T172186#3489515 (10chasemp) [15:23:15] 10Cloud-VPS (Project-requests): Request creation of pluggableauth VPS project - https://phabricator.wikimedia.org/T172034#3489519 (10chasemp) p:05Triage>03Normal [15:23:19] 10Cloud-VPS (Project-requests): Request creation of pluggableauth VPS project - https://phabricator.wikimedia.org/T172034#3483597 (10chasemp) +1 [15:24:26] 10Cloud-VPS (Quota-requests): Temporary increased quota for analytics project in Cloud - https://phabricator.wikimedia.org/T172186#3489525 (10chasemp) +1 [15:45:14] 10Cloud-VPS (Quota-requests): Temporary increased quota for analytics project in Cloud - https://phabricator.wikimedia.org/T172186#3489568 (10Ottomata) Sure, i can probably work with that, thanks. Its mostly the # of instances we need increased, more so than the ram/cpus available. [17:03:36] madhuvishy: fixed :) [17:04:06] That makes me on-call for the wikimania week I guess [17:04:13] which should work out ok [17:17:39] heh sorry [17:44:04] 10Tool-Zppixbot: ZppixBot - Feature requests - https://phabricator.wikimedia.org/T172206#3489973 (10Reception123) [17:53:21] !log git upgrading gerrit on gerrit-new.wmflabs.org to 2.14.2-2449-gc8fa38496c (bazel build release) (master branch). Testing wikimedia branding for prod and the move change feature [17:53:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [18:13:46] 10PAWS, 10Toolforge, 10Kubernetes: Expand PAWS cluster to be about 10 nodes - https://phabricator.wikimedia.org/T172209#3490113 (10yuvipanda) [18:29:17] 10PAWS, 10Toolforge, 10Kubernetes: Expand PAWS cluster to be about 10 nodes - https://phabricator.wikimedia.org/T172209#3490199 (10yuvipanda) To be clear, I don't actually care too much about the size of Tools' k8s cluster :) I only want about 8-10 nodes for the PAWS cluster :) Am happy to let toolforge admi... [18:36:44] heya, i forget [18:36:50] what was the problem with new instances not being able to spawn? [18:36:54] ldap_start_tls_s() failed (uri=ldap://ldap-labs.eqiad.wikimedia.org:389): Connect error: (unknown error code) [18:36:56] ? [18:37:06] is it trusty? [18:37:06] i've made 2 new instances today, and can't get past that [18:37:08] no [18:37:08] stretch [18:37:39] try rebooting to see if that forces a puppet run. [18:38:00] yeah tried that, hang on, mabye i have prefix puppet stuff that is causing problems? [18:38:12] ah [18:38:41] try not adding any roles to any isntances when creating them, add the class when you can ssh in :) [18:39:31] ottomata: could be related to the ldap certificate issues from a week(?) ago? Is the puppetmaster you're using fully up to date? [18:40:13] yeah, i didn't realize the prefix stuff was applying to these names [18:40:14] i forgot [18:40:17] changing names and dtrying [18:40:20] valhallasw`cloud: its just regular labs puppetmaster [18:42:37] andrewbogott: seems like ottomata is having some trouble with making new instances [18:43:33] ottomata: I can look when I'm done with the thing I'm doing now [18:43:36] do prefeixes apply as wildcard? [18:43:37] like [18:43:38] or [18:43:53] just as created with the - prefix [18:43:57] i.e. [18:44:07] does kafka20-1 match a prefix puppet confi for kafka2 [18:44:07] ? [18:45:56] yes, that's what a prefix is [18:45:59] if I'm understanding the question [18:48:40] ok, i think this was my fault [18:48:44] sorry for the false alarm [18:48:48] the instance i just made seems towork [18:48:53] that didn't match this prefix [18:49:27] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:49:51] 10Tool-Zppixbot, 10Goal, 10User-Zppix: Complete config - https://phabricator.wikimedia.org/T148944#3490268 (10Zppix) 05Open>03Resolved a:03Zppix this should of been resolved ages ago [18:57:51] !log analytics Bumped instance quota to 24 instances (nova quota-update analytics --instances 24) [18:57:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Analytics/SAL [18:58:48] 10Cloud-VPS (Quota-requests): Temporary increased quota for analytics project in Cloud - https://phabricator.wikimedia.org/T172186#3490346 (10madhuvishy) 05Open>03Resolved a:03madhuvishy It seems like there's enough RAM/CPUs allocated to accommodate 4 more medium instances. I've bumped up the instance quot... [19:01:39] 10Cloud-VPS (Quota-requests): Temporary increased quota for analytics project in Cloud - https://phabricator.wikimedia.org/T172186#3490359 (10madhuvishy) 05Resolved>03Open [19:02:22] 10Cloud-VPS (Quota-requests): Temporary increased quota for analytics project in Cloud - https://phabricator.wikimedia.org/T172186#3489163 (10madhuvishy) @Ottomata We are marking this as temporary quota increase, do you have an idea when we can drop the quota back down? [19:03:49] thanks madhuvishy ! :) [19:03:58] oh, does it have to be temporary? [19:04:11] i need this to test migrations, but i'd like to not have to ask every time i need to do this [19:05:27] ottomata: I think that's what we understood from 'need this for practice' - can you comment that you'd like it to be permanent, I'll check with the team and update :) [19:06:09] ok [19:07:38] 10Cloud-VPS (Quota-requests): Temporary increased quota for analytics project in Cloud - https://phabricator.wikimedia.org/T172186#3490426 (10Ottomata) Analytics mostly uses labs so that we can test our clustered software somewhere other than production. We often have to do migrations between cluster hardware,... [19:09:16] 10Cloud-VPS (Quota-requests): Increased quota for analytics project in Cloud - https://phabricator.wikimedia.org/T172186#3490441 (10madhuvishy) [19:10:02] 10Cloud-VPS (Quota-requests): Increased quota for analytics project in Cloud - https://phabricator.wikimedia.org/T172186#3489163 (10madhuvishy) 05Open>03Resolved Thanks @Ottomata for clarifying! That seems fine. +1. [19:24:26] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:46] 10Tool-Zppixbot: ZppixBot - Feature requests - https://phabricator.wikimedia.org/T172206#3489973 (10Zppix) Interesting... I may look into doing some of them however the ping suggestion, was already implemented (i understand the suggestion correctly) see converse.py [19:25:57] 10Tool-Zppixbot, 10User-Zppix: ZppixBot - Feature requests - https://phabricator.wikimedia.org/T172206#3490566 (10Zppix) a:03Zppix [19:26:36] 10Tool-Zppixbot, 10User-Zppix: ZppixBot - Feature requests - https://phabricator.wikimedia.org/T172206#3490569 (10Reception123) [19:26:38] !log pluggableauth Created project with user Cicalese as projectadmin [19:26:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Pluggableauth/SAL [19:26:50] 10Tool-Zppixbot, 10Documentation, 10Tracking, 10User-Zppix: Write documentation - https://phabricator.wikimedia.org/T149500#3490570 (10Zppix) p:05Triage>03Low [19:27:20] 10Cloud-VPS (Project-requests): Request creation of pluggableauth VPS project - https://phabricator.wikimedia.org/T172034#3490574 (10madhuvishy) 05Open>03Resolved a:03madhuvishy This is done. https://wikitech.wikimedia.org/wiki/Nova_Resource:Pluggableauth [19:29:17] 10Cloud-VPS (Project-requests): Request creation of pluggableauth VPS project - https://phabricator.wikimedia.org/T172034#3490583 (10cicalese) Thank you! Cindy [19:43:08] bd808 hi, im wondering could you review https://gerrit.wikimedia.org/r/#/c/369461/ please? :) [19:45:24] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Continuous-Integration-Infrastructure, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3490615 (10hashar) @Andrew wrote: > This was at 6, which has historically been mostly safe,... [20:02:37] paladox: the "owner" mediawiki-vagrant's node stuff is the services team. [20:02:44] ok [20:03:07] 10Cloud-Services: Please install a recent version of node and npm by default on new labs instance - https://phabricator.wikimedia.org/T157368#3002816 (10hashar) For the CI slaves we rely on the npm 1.4.21 package (in Jessie) to install a more recent npm from npmjs.org. A summarized version would be: ``` lang=rub... [20:47:14] !log tools.grid-jobs Deployed df4df3f [20:47:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.grid-jobs/SAL [20:50:39] * zhuyifei1999_ can't find df4df3f anywhere [20:52:10] zhuyifei1999_: I'm force pushing patches because I'm a cheater [20:52:26] !log tools.grid-jobs Deployed d2b3ba4 [20:52:27] :( [20:52:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.grid-jobs/SAL [20:52:35] that one should exist [20:53:25] not in phab nor gerrit [20:53:47] I do see https://phabricator.wikimedia.org/R2080:5ee462b08dc471b047f477c9dbe3c2187b134ced though [20:54:03] it takes the phab ui a bit to catch up [20:54:15] there is another amended force push of that [20:54:17] 30 secs [20:54:20] if the repo is active [20:54:52] "Commit Still Parsing" -- https://phabricator.wikimedia.org/R2080:d2b3ba4 [20:55:20] that's slow [20:56:36] now [20:56:45] *now I see [20:56:57] Yep [20:57:05] it's slow to prevent overloading phabricator [20:57:13] as it continously checks gerrit's for updates [20:57:23] which can or might overload the mysql servers [20:57:29] or the phab server it's self [20:57:34] Total jobs that ran: 251650 wow [20:58:59] im pretty sure we have less tasks then that even open in last 2 years [21:01:34] * zhuyifei1999_ contribute < 2% [21:22:10] !log mobile deleted instance zimbuilder; let's not use a Trusty image when we set up its successor. [21:22:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile/SAL [21:23:00] !log mobile deleting mobile-puppetmaster and resurrecting as a Stretch instance [21:23:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile/SAL [21:25:46] (03Draft1) 10Paladox: gerrit: Copy gerrit sh keys into gerrit_root [labs/private] - 10https://gerrit.wikimedia.org/r/369551 [21:25:49] (03PS2) 10Paladox: gerrit: Copy gerrit sh keys into gerrit_root [labs/private] - 10https://gerrit.wikimedia.org/r/369551 [21:26:43] (03PS3) 10Paladox: gerrit: Copy gerrit ssh keys into gerrit_root [labs/private] - 10https://gerrit.wikimedia.org/r/369551 [21:26:57] (03CR) 10Chad: [V: 032 C: 032] gerrit: Copy gerrit ssh keys into gerrit_root [labs/private] - 10https://gerrit.wikimedia.org/r/369551 (owner: 10Paladox) [21:27:08] (03CR) 10Chad: [V: 032 C: 032] gerrit: Copy gerrit ssh keys into gerrit_root [labs/private] - 10https://gerrit.wikimedia.org/r/369551 (owner: 10Paladox) [21:27:30] (03PS1) 10Chad: nvm I'm dumb: Revert "gerrit: Copy gerrit ssh keys into gerrit_root" [labs/private] - 10https://gerrit.wikimedia.org/r/369552 [21:27:36] (03CR) 10Chad: [V: 032 C: 032] nvm I'm dumb: Revert "gerrit: Copy gerrit ssh keys into gerrit_root" [labs/private] - 10https://gerrit.wikimedia.org/r/369552 (owner: 10Chad) [21:34:08] !log mobile launched new zim-master stretch instance for zim creation prototyping (once more, with feeling) [21:34:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile/SAL [21:41:56] I wish I could afford computing systems with absurdly high levels of RAM :/ [21:43:39] harej my mac only has 3.99 gb left of 16gb [21:43:53] Unused ram is wasted ram [21:44:06] I have ram clearing app though [21:44:11] only gains me 3-4gb [21:44:27] But the point of RAM is to use it [21:47:35] But yes, for this project I'm working on, having, oh, around 100 GB of RAM would be nice :) [21:48:33] lol that's alot [21:48:40] phabricator has 60+ gb of ram [21:49:20] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:49:47] MediaWiki has over 3TB [21:50:34] lol [21:50:38] wow [21:51:44] 6.5TB in EQIAD alone [21:52:11] Over 8 in EQIAD [21:52:18] lol that is a wow from me [21:52:34] how much on avage is used? [21:54:41] halfak: np [21:54:59] :))))\ [21:55:20] zhuyifei1999_, did you have a hand in the new "Explain" button? [21:55:46] yeah I added it yesterday or the day before [21:56:04] do you like it? :P [21:56:16] https://phabricator.wikimedia.org/T146483 [21:56:25] Allow me to copy-paste from #wikimedia-research the other day. [21:56:27] [14:47:20] OMG OMG OMG [21:56:27] [14:47:26] You can "explain" inside of quarry now [21:56:27] [14:47:28] OMG [21:56:27] [14:47:29] OMG [21:56:27] [14:47:32] :D!!!! [21:56:28] [14:48:13] I'm filled with surprise and delight. [21:56:43] ;) [21:57:50] Reedy: sure, but I don't have a budget in the millions of dollars, unfortunately [21:58:10] Last I checked, 100GB of ram isn't $1M+ :P [21:58:29] It's still more than I could afford. Darn rent :/ [21:59:14] lol [22:00:39] * zhuyifei1999_ use up my ram all the time due to having around 600 tabs open in firefox [22:00:57] I once had a VM with 64 GB of RAM. It was niiiiice. It would've cost me a fortune had it had it more than like 10 hours. [22:07:01] halfak: though, re-making quarry in redash might be out of my abilities unless I really invest a lot of time on it [22:07:28] zhuyifei1999_, totally understood. That seems pretty long term. [22:07:31] or if yuvi make it possible, but at current status it's unlikely [22:07:44] I'm not even that excited for redash honestly. [22:07:57] I think quarry is nice as a tool that has a very limited scope. [22:10:02] What is the benefit for the customer of rewriting Quarry in Redash? [22:10:02] the theoretical benefit of redash is that other people use it and build on it [22:10:36] so instead of being a thing that only we work on it has a large community [22:10:52] well quarry is a bit stagnant and eventually cruft sets in to outweigh what love we can give it [22:11:01] and that community fixes bugs and adds features [22:11:09] timeline on that may be pretty generous for us but that's the mid-term fear I think [22:12:35] bd808: Hello :) [22:13:05] hey d3r1ck [22:24:23] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [22:27:45] 10Cloud-VPS: Please install a recent version of node and npm by default on new labs instance - https://phabricator.wikimedia.org/T157368#3491266 (10bd808) @Jdlrobson, so you ask is for nodejs and npm to be added to the base system build of all virtual machines in #Cloud-VPS? The only default packages we have rea... [22:32:49] Anyone here know how many files Commons had in January 2015? [22:39:30] harej: https://stats.wikimedia.org/wikispecial/EN/TablesWikipediaCOMMONS.htm 26 million or 28 million [22:39:39] thank you [22:39:48] np [22:42:13] bd808 i got a question about oauth got a min? [22:44:13] o/ d3r1ck [22:44:47] Zppix: what's the question? [22:45:51] bd808 so we're trying to make it so when wikilabels goes thru oauth when it goes to mediawiki.org for oauth login/token/whatever we want it so we can have it automatically determine the user's lang so we dont have to setup sending users to their home wiki [22:46:22] is this possible with oauth [22:47:01] I don't understand the question. Can you rephrase? [22:47:21] the "best" wiki to do the oauth handshake with is meta [22:47:34] the idea of a "home wiki" is difficult [22:48:11] there is a wiki that centralauth has marked for each person as their home wiki, but that is really just the wiki that they were first mead global from [22:48:59] zppix: I don't understand the question. Can you rephrase? the "best" wiki to do the oauth handshake with is meta [22:49:22] bd808 sorry i got disconnected, anyway for example go to labels.wmflabs.org and click on say enwiki it takes you to the oauth popup is there a way to make that popup (always when possible) show up in that users lang [22:49:22] zppix: there is a wiki that centralauth has marked for each person as their home wiki, but that is really just the wiki that they were first mad global from [22:49:33] zppix: automatically determine the user's lang <= iirc you can get the user's interface language [22:49:56] meta would be localized to their ui language I think [22:50:13] PROBLEM - Puppet errors on tools-exec-gift-trusty-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:50:22] thats what i thought we'll have to try that out [22:51:02] ty bd808 [22:51:28] mediawiki would too in theory, but people are much more likely to have setup their ui language on meta [22:51:47] ah I went cheaty https://github.com/toollabs/video2commons/blob/master/video2commons/frontend/app.py#L145 [22:51:52] using mediawiki as the main oauth target is pretty dated. [22:52:23] bd808 thats what were currently using but no luck [22:52:27] so i guess well try meta [22:52:39] zhuyifei1999_: if I understood zppix's question it was about the oauth dialog itself [22:52:57] that needs to be handled in the wiki (and should be already) [22:53:12] ah ok [22:53:15] bd808 thats is exactly what i mean [22:53:20] unless OAuth messages are always in the wiki's lang ... [22:53:28] which actually might be the case [22:53:42] messages and l10n are a pain that way [22:53:56] I've had a lot of "fun" dealing with that. [22:54:16] we have concepts of a "UI language" which is what the user pref and ?uselang= control [22:54:17] I have some pages on Meta that adjust depending on your interface language. It just assumes you're logged in and *shrug* if otherwise. [22:54:31] Because you have to be logged in to benefit from ULS. [22:54:39] and an concept of the "content language" which is global to the wiki [22:55:00] PROBLEM - Host tools-paws-worker-1004 is DOWN: CRITICAL - Host Unreachable (10.68.21.233) [22:55:02] on meta and mw.o the content language is english [22:55:06] i wish it was a standard to have browsers always send browser-lang [22:55:09] It'd be a lot better if logged out users could set their interface language (or if there was some kind of content negotiation) [22:55:12] but on way eswiki it is spanish [22:55:22] harej: check out commons :) [22:55:36] harej: well, until you realize that would melt our servers ;) [22:55:40] https://commons.wikimedia.org/wiki/MediaWiki:AnonymousI18N.js [22:55:52] bd808: commons already melted it :) [22:56:12] luckily nobody uses commons [22:56:16] * bd808 ducks [22:56:38] the Wikimedia Resource Center serves content based on UI language but in the grand scheme of things no one uses the Resource Center ;) [22:56:40] PROBLEM - Host tools-paws-worker-1002 is DOWN: CRITICAL - Host Unreachable (10.68.23.186) [22:56:41] bd808 you gotta love it when projects boycott what we dont add as a feature :P [22:56:59] zhuyifei1999_: that trick just adds ?uselang= magically right? [22:57:13] uh no [22:57:31] it has a few functions, but the most important are: [22:57:55] 1. keep uselang= in links [22:58:13] 2. suggest interface language based on browser language [22:58:22] https://phabricator.wikimedia.org/T58464 is the ticket btw [22:58:41] yeah. the uselang= rewrite client side is the main magic [22:59:06] that fragments the front-end varnish cache, but at least in a semi-reasonable way [22:59:47] if usage of that were wide spread on say enwiki it would cause big problems though [23:00:17] the only way that enwiki works at the anon request scale we get is that most things are served from cache [23:00:48] enwiki is for English speakers anyways [23:00:51] if all the requests hit the backend mediawiki servers and databases we would need 100x as many servers [23:01:20] zhuyifei1999_: thankfully the urgency of ULS-for-all is only for relatively small sites [23:01:24] 10PAWS: Install Python 2.7 in PAWS - https://phabricator.wikimedia.org/T167926#3491398 (10yuvipanda) 05Open>03declined Heya! Thanks for filing this! Unfortunately I don't think we want to encourage Python2.7 use long term on PAWS. And maintaining two images (for 3 and 2) is going to add a lot more added com... [23:02:11] 10PAWS: Pandas to_clipboard function does not work in remote environments like PAWS - https://phabricator.wikimedia.org/T155763#3491400 (10yuvipanda) 05Open>03declined Closing, since the modle of pasting won't really ever work with remote kernels like PAWS :) [23:04:56] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Continuous-Integration-Infrastructure, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3491412 (10chasemp) I did a small bit of poking today. It seems we are using swap on labco... [23:32:09] 10PAWS: PAWS with bot accounts - https://phabricator.wikimedia.org/T120558#3491526 (10yuvipanda) 05Open>03Resolved a:03yuvipanda Going to close this for now! [23:50:12] RECOVERY - Puppet errors on tools-exec-gift-trusty-01 is OK: OK: Less than 1.00% above the threshold [0.0]