[00:06:30] PROBLEM - Host tools-paws-worker-1004 is DOWN: CRITICAL - Host Unreachable (10.68.20.63) [00:11:41] !log tools.listeria Added -N ... labels to croned jobs [00:11:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.listeria/SAL [00:15:56] !lot tools.listeria 72 active jobs running. crontab makes me think this should be more like 2-3 running jobs [00:18:38] stashbot: ? [00:18:38] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [00:18:45] !log tools.listeria 72 active jobs running. crontab makes me think this should be more like 2-3 running jobs [00:18:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.listeria/SAL [00:19:56] !log tools.listeria Killed all running jobs [00:19:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.listeria/SAL [00:25:41] 10Cloud-Services, 10PAWS, 10Toolforge: Setup a devpi server to help speedup pip installs - https://phabricator.wikimedia.org/T132025#3491649 (10yuvipanda) 05Open>03declined Not worth it. [00:43:32] 10PAWS, 10Toolforge, 10Kubernetes: Expand PAWS cluster to be about 10 nodes - https://phabricator.wikimedia.org/T172209#3491747 (10yuvipanda) It's at 5 nodes now, and I've run out of quota in tools now. @bd808 @chasemp what do you think of me depooling & deleting 5 k8s tools nodes and creating 5 paws ones?... [00:45:37] 10PAWS, 10Toolforge, 10Kubernetes: Expand PAWS cluster to be about 10 nodes - https://phabricator.wikimedia.org/T172209#3491751 (10bd808) Lets just bump the quota. We can always dial it back later. [00:51:42] can someone help me to start my webservice? The server appears to be confused. [00:51:45] tools.magog@tools-bastion-03:~$ webservice --backend=kubernetes start [00:51:45] Looks like you already have another webservice running, with a gridengine backend [00:51:45] You should stop that webservice by issuing: [00:51:45] webservice --backend=gridengine stop [00:51:46] And then start it again with backend kubernetes by issuing: [00:51:48] webservice --backend=kubernetes start [00:51:50] tools.magog@tools-bastion-03:~$ webservice --backend=gridengine stop [00:51:52] Your webservice is not running [00:54:49] I meant to say "please" [00:56:07] 10PAWS, 10Toolforge, 10Kubernetes: Expand PAWS cluster to be about 10 nodes - https://phabricator.wikimedia.org/T172209#3491754 (10bd808) >>! In T172209#3491751, @bd808 wrote: > Lets just bump the quota. We can always dial it back later. Added 40G of ram quota: ``` $ nova quota-show --tenant tools +-------... [00:56:40] Magog_the_Ogre: heh. it can get confused sometimes [00:56:48] what;s the tool name? [00:57:01] tools.magog [00:57:01] bd808: look before the $ :P [00:57:10] thanks Reedy :) [00:57:26] Reedy: you expect me to be able to read? [00:57:40] bd808: sudo look before the $ :P [00:59:36] as a developer, I am sick and darned tired of people constantly asking me to read. [00:59:41] !log tools.magog `rm service.manifest service.log` to try and fix confused webservice command [00:59:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.magog/SAL [01:01:06] that worked. Thanks everyone! :) [01:01:57] Magog_the_Ogre: there was a running kubernetes pod but the webservice control file thought that you were running on grid engine. [01:02:02] made things crazy [01:02:38] I did `webservice --backend=kubernetes restart` after removing those state tracking files [01:09:31] PROBLEM - Puppet errors on tools-exec-1417 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:36:39] 10PAWS, 10Toolforge, 10Kubernetes: Expand PAWS cluster to be about 10 nodes - https://phabricator.wikimedia.org/T172209#3491787 (10yuvipanda) 05Open>03Resolved tvym, @bd808! Since labs in general hates it when I try to create new instances, it took me almost 20 tries with lots of instances failing, but... [01:43:31] 10Quarry: Gigantic query results cause a SIGKILL and the query status do not update - https://phabricator.wikimedia.org/T172086#3491791 (10zhuyifei1999) The current storing query status in the same process as the storing query results isn't going to work. SIGKILL cannot be caught, so only the celery master proce... [01:44:33] RECOVERY - Puppet errors on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [01:47:57] PROBLEM - Puppet errors on tools-paws-worker-1002 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:55:24] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:22:59] RECOVERY - Puppet errors on tools-paws-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [02:25:20] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [02:46:27] PROBLEM - Puppet errors on tools-paws-worker-1013 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [03:20:37] 10PAWS: Paws display "502 - Bad gateway error" for specific user - https://phabricator.wikimedia.org/T172080#3491934 (10yuvipanda) @Ebramino can you try https://paws.tools.wmflabs.org? It's going to replace paws.wmflabs.org shortly. Same everything, just a different underlying base. [03:26:28] RECOVERY - Puppet errors on tools-paws-worker-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [03:52:34] PROBLEM - Puppet errors on tools-exec-1404 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [04:12:35] RECOVERY - Puppet errors on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [05:46:46] 10PAWS: Paws display "502 - Bad gateway error" for specific user - https://phabricator.wikimedia.org/T172080#3492021 (10Ebraminio) 05Open>03Resolved a:03Ebraminio Great. It works there! And nice, an updated Python! Thank you very much! [06:53:39] 10Cloud-VPS, 10cloud-services-team (Kanban): New Trusty base images hang on boot - https://phabricator.wikimedia.org/T172064#3492048 (10Andrew) I am able to build booting images if I roll back to puppet patch ffdfa2821bca02a0ec013d1e618d4d9690f7ec7d So... despite trying very carefully to eliminate puppet chan... [07:45:26] 10PAWS: Paws display "502 - Bad gateway error" for specific user - https://phabricator.wikimedia.org/T172080#3492107 (10yuvipanda) @Ebraminio cool! I'm going to upgrade Python to 3.6 as well (it's currently 3.5 I think) :) npm will also be available by default. [08:47:11] PROBLEM - Puppet errors on tools-worker-1027 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:00:47] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login, OAuth error: API error mwoauth-invalid-authorization - https://phabricator.wikimedia.org/T136114#3492237 (10yuvipanda) @MisterSynergy can you (and others!) try out a new install at https://paws.tools.wmflabs.org? All your old files... [09:04:21] 10PAWS: R kernel not available for PAWS - https://phabricator.wikimedia.org/T164198#3225260 (10yuvipanda) Heya! https://paws.tools.wmflabs.org is the newer version of PAWS (URLs will switch shortly!). It does have an R kernel! Can you check it out and confirm? Thanks! [09:05:02] 10PAWS, 10MediaWiki-extensions-OAuth: Oauth for PAWS fails - presumably because of username change - https://phabricator.wikimedia.org/T161696#3139903 (10yuvipanda) can you try https://paws.tools.wmflabs.org? It's a new installation of PAWS (you get to keep all your old files!). I'll switch out the URL in a co... [09:06:40] 10PAWS: Paws display 504 - Bad gateway time-out - https://phabricator.wikimedia.org/T143493#3492276 (10yuvipanda) 05Open>03Resolved a:03yuvipanda [09:10:06] 10PAWS: Point paws.wmflabs.org to new PAWS setup - https://phabricator.wikimedia.org/T172257#3492294 (10yuvipanda) [09:27:13] RECOVERY - Puppet errors on tools-worker-1027 is OK: OK: Less than 1.00% above the threshold [0.0] [09:59:02] quiddity: https://phabricator.wikimedia.org/phame/post/view/65/toolforge_provides_proxied_mirrors_of_cdnjs_and_now_fontcdn_for_your_usage_and_user-privacy/ nice [10:09:47] 10Tool-Zppixbot, 10Documentation, 10User-Zppix: Write documentation - https://phabricator.wikimedia.org/T149500#3492391 (10Aklapper) not #tracking [10:17:22] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:29:34] 10Tool-fatameh: More descriptions for Fatameh - https://phabricator.wikimedia.org/T171995#3492444 (10Tarrow) Thanks, this should probably be included in the upstream package Wikidata Integrator. I'll pass this ticket on to them. I'll mark this as resolved when/if they patch your suggestion in and I include the... [10:46:22] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login, OAuth error: API error mwoauth-invalid-authorization - https://phabricator.wikimedia.org/T136114#3492483 (10Dvorapa) @yuvipanda It works as expected for me. [10:57:21] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [11:19:51] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login, OAuth error: API error mwoauth-invalid-authorization - https://phabricator.wikimedia.org/T136114#3492618 (10MisterSynergy) Works for me as well. [11:21:31] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login, OAuth error: API error mwoauth-invalid-authorization - https://phabricator.wikimedia.org/T136114#3492652 (10MarcoAurelio) Will test [11:23:32] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login, OAuth error: API error mwoauth-invalid-authorization - https://phabricator.wikimedia.org/T136114#3492670 (10MarcoAurelio) @yuvipanda ``` marcoaurelio@PAWS:~$ pwb.py login -family:wikibooks -lang:es Traceback (most recent call last)... [11:25:08] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login, OAuth error: API error mwoauth-invalid-authorization - https://phabricator.wikimedia.org/T136114#3492673 (10MarcoAurelio) ``` marcoaurelio@PAWS:~$ pwb.py redirect do -family:wikibooks -lang:es Traceback (most recent call last): Fi... [11:46:51] 10PAWS, 10MediaWiki-extensions-OAuth: Oauth for PAWS fails - presumably because of username change - https://phabricator.wikimedia.org/T161696#3492699 (10FaFlo) 05Open>03Resolved a:03FaFlo Seems to work - thanks :) [11:58:21] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [12:03:57] 10Tools: Tool "adas" loads fonts from fonts.googleapis.com - https://phabricator.wikimedia.org/T172271#3492737 (10zhuyifei1999) [12:11:36] 10Tools: Tool "arkivbot" loads assets from bootsrtrapcdn and code.jquery.com - https://phabricator.wikimedia.org/T172272#3492762 (10zhuyifei1999) [12:12:46] 10Tools: Tool "arkivbot" loads assets from bootsrtrapcdn and code.jquery.com - https://phabricator.wikimedia.org/T172272#3492778 (10zhuyifei1999) @Danmichaelo and Profoss are the maintainers of this tool. I'm unable to find the Phabricator username of the latter. [12:16:12] 10Tools: Tool "arowf" loads jquery and bootstrap from cloudflare - https://phabricator.wikimedia.org/T172273#3492781 (10zhuyifei1999) [12:17:17] gonna do 10 tasks a day [12:19:34] 10Tools: Tool "ash-django" loads bootstrap from bootstrapcdn - https://phabricator.wikimedia.org/T172274#3492797 (10zhuyifei1999) [12:21:17] bd808: "http://www.w3.org/Icons/valid-xhtml10" isn't allowed either right? [12:23:00] 10Tools: Tool "blankpages" loads jquery from ajax.googleapis.com - https://phabricator.wikimedia.org/T172275#3492815 (10zhuyifei1999) [12:23:21] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [12:25:44] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login, OAuth error: API error mwoauth-invalid-authorization - https://phabricator.wikimedia.org/T136114#3492831 (10Dvorapa) @MarcoAurelio It looks like T142269 [12:27:02] 10Tools, 10Wikisource: Tool "bub" loads assets from ajax.googleapis.com - https://phabricator.wikimedia.org/T172276#3492836 (10zhuyifei1999) [12:29:08] 10Tools: Tool "bytesadded" loads jquery ui from ajax.googleapis.com - https://phabricator.wikimedia.org/T172277#3492850 (10zhuyifei1999) [12:31:01] 10Tools: Tool "cgstat" loads jquery from ajax.googleapis.com - https://phabricator.wikimedia.org/T172278#3492865 (10zhuyifei1999) [12:32:15] 10Tools: Tool "cgstat" loads jquery from ajax.googleapis.com - https://phabricator.wikimedia.org/T172278#3492880 (10zhuyifei1999) Oops forgot to CC @Addshore @Jkroll @WMDE-Fisch [12:35:49] 10Tools: Tool "citationhunt" loads assets from google - https://phabricator.wikimedia.org/T172279#3492887 (10zhuyifei1999) [12:40:51] 10Tools: Tool "cite-o-meter" loads fork-me-on-github ribbon from github - https://phabricator.wikimedia.org/T172280#3492927 (10zhuyifei1999) [13:32:05] !log tools.heritage Upgdate deployed pywikibot to 3.0.20170403 (T112460) [13:32:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [13:32:08] T112460: Source links in the monuments database get too long and are truncated - https://phabricator.wikimedia.org/T112460 [13:32:50] (03PS6) 10Lokal Profil: Store plain permalink instead of urlencoded one [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309858 (https://phabricator.wikimedia.org/T112460) [13:34:01] 10Tools: Tool "cgstat" loads jquery from ajax.googleapis.com - https://phabricator.wikimedia.org/T172278#3493219 (10Addshore) https://github.com/wmde/cgstat/pull/1 [13:34:08] 10Tools, 10User-Addshore: Tool "cgstat" loads jquery from ajax.googleapis.com - https://phabricator.wikimedia.org/T172278#3493220 (10Addshore) [13:34:14] 10Tools, 10User-Addshore: Tool "cgstat" loads jquery from ajax.googleapis.com - https://phabricator.wikimedia.org/T172278#3492865 (10Addshore) a:03Addshore [13:34:48] 10Tools, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3493223 (10zhuyifei1999) [13:40:31] 10Tools, 10Patch-For-Review, 10User-Addshore: Tool "cgstat" loads jquery from ajax.googleapis.com - https://phabricator.wikimedia.org/T172278#3493264 (10Addshore) [13:42:51] 10Tools, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3493291 (10Addshore) [13:42:53] 10Tools, 10Patch-For-Review, 10User-Addshore: Tool "cgstat" loads jquery from ajax.googleapis.com - https://phabricator.wikimedia.org/T172278#3493289 (10Addshore) 05Open>03Resolved Merged & Deployed! [13:44:30] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login, OAuth error: API error mwoauth-invalid-authorization - https://phabricator.wikimedia.org/T136114#3493304 (10yuvipanda) @MarcoAurelio - is that showing up at paws.tools.wmflabs.org and not at paws.wmflabs.org? Or is it showing up at... [14:09:28] 10Tools, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3493424 (10Danmichaelo) [14:09:30] 10Tools: Tool "arkivbot" loads assets from bootsrtrapcdn and code.jquery.com - https://phabricator.wikimedia.org/T172272#3493421 (10Danmichaelo) 05Open>03Resolved p:05Triage>03Normal a:03Danmichaelo [14:09:44] 10Tools: Tool "arkivbot" loads assets from bootsrtrapcdn and code.jquery.com - https://phabricator.wikimedia.org/T172272#3492762 (10Danmichaelo) Thanks for providing the replacement links! [14:23:13] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3493485 (10Cmjohnson) @chasemp Do you know the raid cfg you want? The server has (12) 3.5 6Tb disks and (2) 2.5" disk, the disk shelf has (12) 3.5"... [14:23:40] 10Tools, 10Community-Tech-Tool-Labs, 10Epic: Convert all Labs tools to use cdnjs for static libraries and fonts - https://phabricator.wikimedia.org/T103934#3493487 (10zhuyifei1999) [14:23:43] 10Toolforge, 10Patch-For-Review: Create a fonts CDN for use on Tool Labs - https://phabricator.wikimedia.org/T110027#3493486 (10zhuyifei1999) 05Open>03Resolved [14:25:43] 10Data-Services, 10DBA: HELP! Database is getting Slow: A test which took less than 10 min, now it takes 3 hours. I cannot do my experiments. - https://phabricator.wikimedia.org/T105964#3493503 (10zhuyifei1999) [14:26:31] 10Data-Services: max_user_connections is (too) low - running lots of simple queries - https://phabricator.wikimedia.org/T155025#3493506 (10zhuyifei1999) [14:27:03] 10Data-Services, 10DBA: HELP! Database is getting Slow: A test which took less than 10 min, now it takes 3 hours. I cannot do my experiments. - https://phabricator.wikimedia.org/T105964#1455384 (10Marostegui) @marcmiquel is this still an issue? [14:34:46] !help [14:34:46] hexmode: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [14:36:53] hexmode: hey, what's up? [14:37:49] Sagan: just looking for the right place to ask questions, passed the info along [14:38:12] Sagan: OBE has my question [14:39:04] Is there any way to free up ~1GB on one of the WMF labs' fresh installs? Need just a little more space for a database dump [14:39:29] you mean a normal labs instance or one at toolforge? [14:40:00] on normal instances you can get some extra space: https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space [14:40:02] Pretty sure we have a normal instance [14:40:13] Thanks! [14:43:19] 10Data-Services, 10DBA: HELP! Database is getting Slow: A test which took less than 10 min, now it takes 3 hours. I cannot do my experiments. - https://phabricator.wikimedia.org/T105964#3493551 (10marcmiquel) Nope, closed. Thank you În Mie, 2 aug. 2017, 16:27 Marostegui, a... [14:45:18] 10VPS-Projects, 10Beta-Cluster-Infrastructure, 10Operations, 10Release-Engineering-Team (Kanban), and 2 others: a lot of beta cluster instances are not reachable over SSH - https://phabricator.wikimedia.org/T171174#3493557 (10fgiunchedi) [14:45:47] 10Data-Services, 10DBA: HELP! Database is getting Slow: A test which took less than 10 min, now it takes 3 hours. I cannot do my experiments. - https://phabricator.wikimedia.org/T105964#3493560 (10zhuyifei1999) 05Open>03Resolved [14:46:15] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3493561 (10chasemp) >>! In T167984#3493485, @Cmjohnson wrote: > @chasemp Do you know the raid cfg you want? The server has (12) 3.5 6Tb disks and (... [14:48:47] 10Cloud-VPS: Please install a recent version of node and npm by default on new labs instance - https://phabricator.wikimedia.org/T157368#3493570 (10Jdlrobson) My frustration is that every time I setup a new instance I have to uninstall the existing outdated Node and reinstall the new one. My wish is that when s... [14:55:07] 10PAWS: R kernel not available for PAWS - https://phabricator.wikimedia.org/T164198#3493589 (10Halfak) I do not see an R option in the "new" dropdown. [14:55:27] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Continuous-Integration-Infrastructure, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3493590 (10chasemp) ```/home/rush# sudo bash swap_stat.sh inet_gethost (10199) 68 kB inet_g... [14:56:04] 10PAWS: R kernel not available for PAWS - https://phabricator.wikimedia.org/T164198#3493592 (10Halfak) Also I should confirm that I see the same issue described in the task when trying to open an R notebook. [15:20:43] !log git disabling puppet on gerrit-mysql temp for icinga2 update (changed configuation needs change in the repo) [15:20:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [15:24:12] (03Draft1) 10Paladox: Update configuration to match latest icinga2 update [labs/icinga2] - 10https://gerrit.wikimedia.org/r/369681 [15:24:14] (03PS2) 10Paladox: Update configuration to match latest icinga2 update [labs/icinga2] - 10https://gerrit.wikimedia.org/r/369681 [15:24:17] (03CR) 10Paladox: [C: 04-2] "Do not merge yet" [labs/icinga2] - 10https://gerrit.wikimedia.org/r/369681 (owner: 10Paladox) [15:41:53] !log tools.mix-n-match Killed 30+ overlapping mix-n-match-microsync jobs that seemed stuck [15:41:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mix-n-match/SAL [15:46:24] bd808: "http://www.w3.org/Icons/valid-xhtml10" isn't allowed either right? [15:46:49] zhuyifei1999_: is it hosted on a WMF server? ;) [15:46:56] no [15:48:19] oh and anything I should add/change in subtasks of https://phabricator.wikimedia.org/T172065 ? (except the "from from" typo I noticed after I filed todays batch) [15:49:31] planning to file 10 tasks a day :) so prefer if things are good before I file the rest, instead of after [15:50:00] (03PS3) 10Paladox: Update configuration to match latest icinga2 update [labs/icinga2] - 10https://gerrit.wikimedia.org/r/369681 [15:59:32] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3493754 (10Cmjohnson) [16:03:38] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3493767 (10Cmjohnson) @robh @chasemp The servers are racked and all preliminary work done. I connected the disk shelf to the server but it's not bei... [16:13:30] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3493786 (10Cmjohnson) They are connected like this on the P441 port 1E is connected DP1 (I/O module A) of the array and port 2E goes to DP1 (I/O... [16:14:20] (03PS4) 10Paladox: Update configuration to match latest icinga2 update [labs/icinga2] - 10https://gerrit.wikimedia.org/r/369681 [16:16:39] 10Tools, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3484471 (10Bawolff) If we want to flat outban this sort of thing, we could use csp to do it. [16:24:30] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3493823 (10chasemp) ping @madhuvishy hopefully will have some time to read up on the manuals :) [16:27:53] !log tools.dawikitool Deleted a bunch of flyt_overs and slet_overs jobs that looked stuck (running for >24 hours) [16:27:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dawikitool/SAL [16:38:53] 10Tools, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3493886 (10zhuyifei1999) >>! In T172065#3493809, @Bawolff wrote: > If we want to flat outban this sort of thing, we could use csp to do it. {T130748}. The ToU currently is curre... [16:39:41] 10Cloud-Services: Add Content-Security-Policy header enforcing 3rd party web interaction restrictions to proxy responses - https://phabricator.wikimedia.org/T130748#2145156 (10zhuyifei1999) [16:39:43] 10Tools, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3493890 (10zhuyifei1999) [16:40:47] 10Tools, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3484471 (10zhuyifei1999) [16:40:49] 10Tools, 10Community-Tech-Tool-Labs, 10Epic: Convert all Labs tools to use cdnjs for static libraries and fonts - https://phabricator.wikimedia.org/T103934#3493894 (10zhuyifei1999) [16:40:51] 10Cloud-Services, 10Toolforge, 10Monumental, 10Privacy: Monumental imports css from fonts.googleapis.com - https://phabricator.wikimedia.org/T168786#3493892 (10zhuyifei1999) [16:41:03] 10Cloud-Services, 10Toolforge, 10Monumental, 10Privacy: Monumental imports css from fonts.googleapis.com - https://phabricator.wikimedia.org/T168786#3376728 (10zhuyifei1999) [16:41:05] 10Toolforge, 10Patch-For-Review: Create a fonts CDN for use on Tool Labs - https://phabricator.wikimedia.org/T110027#3493896 (10zhuyifei1999) [16:59:05] !log tools Force deleted 6 jobs suck in 'dr' state [16:59:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:02:17] !log tools.drtrigonbot Deleted a large number of mail.tools.drtrigonbot jobs stuck in 'qw' state for months [17:02:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.drtrigonbot/SAL [17:21:41] bd808: fyi, -once can fail once in a long time (like a few months). happened a few times to some of my very long-running jobs in crontab [17:21:57] no way to reproduce so didn't file a ticket [17:22:26] zhuyifei1999_: fail in which direction? Allowing >1 or false positive blocking? [17:22:39] allowing >1 [17:22:44] *nod* [17:23:00] I wouldn't notice false positive blocking because they are in crontabs [17:23:41] its a bit of a hack in jsub to implement -once at all. Its not a native thing in grid engine [17:23:54] jsub is a hack :/ [17:24:07] its a convenience wrapper :) [17:24:26] convenience leaky wrapper :P [17:24:33] all abstractions are leaky as the CS folks like to say [17:25:36] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:31:14] 10Toolforge, 10Toolforge-standards-committee: Rename the Tool Labs standards committee - https://phabricator.wikimedia.org/T170363#3428649 (10MacFan4000) I changed Tool Labs to Toolforge [17:33:48] (03PS5) 10Paladox: Update configuration to match latest icinga2 update [labs/icinga2] - 10https://gerrit.wikimedia.org/r/369681 [17:50:20] (03PS6) 10Paladox: Update configuration to match latest icinga2 update [labs/icinga2] - 10https://gerrit.wikimedia.org/r/369681 [18:00:38] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [18:03:02] 10Tools: Tool "betabot" loads a badge from www.w3.org - https://phabricator.wikimedia.org/T172309#3494296 (10zhuyifei1999) [18:04:50] (03PS7) 10Paladox: Update configuration to match latest icinga2 update [labs/icinga2] - 10https://gerrit.wikimedia.org/r/369681 [18:05:32] (03CR) 10Paladox: [V: 032 C: 032] "Figures crossed that i did everything correctly :)" [labs/icinga2] - 10https://gerrit.wikimedia.org/r/369681 (owner: 10Paladox) [18:07:33] !log git enable puppet again on gerrit-mysql - deploying https://gerrit.wikimedia.org/r/#/c/369681/ [18:07:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [18:13:14] (03Draft1) 10Paladox: Fix syntax error in commands.conf [labs/icinga2] - 10https://gerrit.wikimedia.org/r/369705 [18:13:17] (03PS2) 10Paladox: Fix syntax error in commands.conf [labs/icinga2] - 10https://gerrit.wikimedia.org/r/369705 [18:13:20] (03CR) 10Paladox: [V: 032 C: 032] Fix syntax error in commands.conf [labs/icinga2] - 10https://gerrit.wikimedia.org/r/369705 (owner: 10Paladox) [18:21:54] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Continuous-Integration-Infrastructure, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3494394 (10chasemp) A few thoughts on this phenom. I'm not sure if rabbit components swapp... [18:23:22] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Continuous-Integration-Infrastructure, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3494400 (10chasemp) Also, not a terrible idea as we start forcing rabbit back into physical... [18:32:10] PROBLEM - Puppet errors on tools-exec-1431 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:36:58] 10PAWS: Paws display "502 - Bad gateway error" for specific user - https://phabricator.wikimedia.org/T172080#3494464 (10Ebraminio) 3.6 will be just fantastic for my use! Great news! Please also consider putting a node.js jupyter kernel also, I tried one locally and didn't find it very ready to use on my local t... [18:54:45] 10PAWS: R kernel not available for PAWS - https://phabricator.wikimedia.org/T164198#3494563 (10yuvipanda) @halfak that's strange. I totally do see an 'R' in the new dropdown menu.... [18:56:40] 10Data-Services, 10cloud-services-team, 10DBA, 10Security-Team, and 5 others: Make wbqc_constraints table available on Quarry et al. - https://phabricator.wikimedia.org/T170927#3494567 (10Bawolff) I approve on behalf of #security-team [19:00:43] 10PAWS: R kernel not available for PAWS - https://phabricator.wikimedia.org/T164198#3494572 (10Halfak) I stopped my server and restarted it. Same deal! I also tried a hard refresh to make sure it wasn't my local cache. Anything else I could try? [19:07:09] RECOVERY - Puppet errors on tools-exec-1431 is OK: OK: Less than 1.00% above the threshold [0.0] [19:07:40] 10Tool-Zppixbot, 10User-Zppix: ZppixBot - Feature requests - https://phabricator.wikimedia.org/T172206#3494602 (10Reception123) [19:08:30] 10Wikibugs, 10Anti-Harassment (AHT Sprint 2): Update Wikibugs to Point Anti-Harassment to #wikimedia-anti-harassment-tools - https://phabricator.wikimedia.org/T167516#3494609 (10TBolliger) [19:31:50] 10Tool-Zppixbot, 10User-Zppix: ZppixBot - Feature requests - https://phabricator.wikimedia.org/T172206#3494672 (10Zppix) p:05Triage>03Normal [19:40:33] 10Tools, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3494735 (10Beta16) [19:40:35] 10Tools: Tool "betabot" loads a badge from www.w3.org - https://phabricator.wikimedia.org/T172309#3494732 (10Beta16) 05Open>03Resolved p:05Triage>03Low Fixed! Thanks for reporting. [19:41:46] 10Cloud-Services, 10Operations, 10ops-eqiad: rack/setup/install labmon1002 - https://phabricator.wikimedia.org/T165784#3276770 (10Cmjohnson) [20:19:22] halfak: zppix was asking about OAuth and ui language stuff yesterday. Do you know if there is a bug filed about the issues your users/testing have seen? [20:21:12] bd808 https://phabricator.wikimedia.org/T166472 [20:21:22] thanks Zppix [20:21:27] np [20:21:29] I think awight has been looking into this a bit. [20:21:39] I've not really considered it carefully. [20:21:40] we both have halfak [20:21:44] roger [20:22:25] ok, so there's not a specific report about l10n not being done to ui language? [20:22:44] no, but upon testing it doesnt work bd808 [20:22:45] that was the bit I was interested in sending people looking into if it was happening [20:23:20] i dont know what awight tried but i tried using uselang= in the url to see if the oauth dialog box would l18n [20:24:03] uselang may or may not work. changing your user pref would be more definitive [20:24:28] if you can repro, file a bug and let me know. I think I know who to poke about looking into it [20:25:18] bd808 ill try that or find someone that is multilingual help me out [20:25:24] ill keep you updated okay ? [20:25:59] Zppix: I bet you can tell if the messages are in spanish or german or not if you set that lang [20:26:35] bd808: User prefs won’t help if the person is logged-out. AFAIK there are two ways to localize the OAuth login flow: one is to send the user to their home wiki, or at least the home wiki of the labeling campaign they’re logging into. The second is to fix long-standing i18n stuff in MediaWiki login, and send them to meta.wmo [20:26:47] if the skin messages change but the dialog doesn't then you'd have a smoking gun [20:26:47] bd808 i realised that after i said that :P [20:27:08] if you are logged out oauth will redir to login [20:27:21] bd808 let me test it out rn [20:27:21] After looking at the UX for the meta login, I’m mostly convinced that we need to send people to their home wiki, which I believe works around the need to fix anything else about the OAuth flow. [20:27:52] login l10n is separate from oauth dialog l10n [20:27:54] awight see thats where i disagree but i think we need to take this to the task instead of flooding -cloud? [20:28:15] awight: except you will probably find that "home wiki" is a nebulous concept [20:28:39] the data marked in CA is pretty arbirary [20:28:40] it does l18n [20:28:45] bd808 disregard [20:28:51] I’m mentioning it here cos I don’t want it to sound like we suddenly need to do work on OAuth l10n [20:28:51] sorry :/ [20:28:52] awight, home wiki is kind of spooky. [20:29:04] halfak awight it does l18n properly i tested with changing my user pref to es [20:29:14] Nice. So this is a non-issue? [20:29:25] halfak: Understood—let’s send users to the wiki of the labeling campaign. Home wiki is broken. [20:29:40] halfak i still think meta is a better place to send them rather then mw.org [20:29:46] awight, but even doing that is kind of funny. [20:29:55] but if no-one agrees then im okay with not changing [20:30:00] We haven't had users complain. Why are we fixing a non-broken thing? [20:30:08] actually they have [20:30:13] T166472 [20:30:13] T166472: Wikilabels should authenticate on the right wiki - https://phabricator.wikimedia.org/T166472 [20:30:22] No strong pref for MW or Meta -- whatever the OAuth folk want to standardize around. [20:30:25] I think sending to mediawikiwiki or meta.wmo is bad, so this is an issue. [20:30:34] bd808 is there a standard? [20:30:35] halfak: Why not labeling campaign wiki, though? [20:31:26] awight, pain in the ass [20:31:31] eh [20:31:44] Sometimes someone will auth without trying to access a campaign [20:31:52] metawiki is the 'normal' OAuth wiki these days [20:32:02] mw.o was originally because that was the first place it was deployed [20:32:04] good enough for me then [20:32:17] halfak so no change? [20:32:32] metawiki! [20:32:36] ok [20:32:41] well i have the pr already setup halfak [20:32:44] it just needs a merge [20:33:44] Zppix, can you put that task in the "Review" column: https://phabricator.wikimedia.org/tag/scoring-platform-team/ [20:33:49] yes [20:33:56] Once I see it there, I'll review and merge [20:34:18] done [20:34:43] cool, if we’re going to metawiki, then the l10n questions are relevant… [20:35:25] thanks again bd808 [22:23:41] bd808: I get permission denied when I try to ssh to a newly created vm [22:23:54] and the logs show that it can't communicate with ldap [22:24:09] XioNoX: lame. that's never a good thing. [22:24:13] what's the instance? [22:24:50] bd808: lookingglass.traffic.eqiad.wmflabs [22:26:05] XioNoX: it's rejecting my root key too. we have this occasionally. if the first puppet run doesn't work correctly then the instance is basically dead. [22:26:14] the "fix" is to delete and try again [22:26:29] using a different hostname can avoid other problems too. [22:26:36] okay [22:27:01] Poor Yuvi had to try 17 times yesterday to get 10 good vms up :/ [22:28:14] good to hear :) [22:28:39] bd808: same issue with traffic-lg.traffic.eqiad.wmflabs, or should I wait longer? [22:29:13] oh yeah, the logs show that it's not done yet [22:29:21] Its ssh server doesn't seem to be up yet [22:29:57] I'm in now! [22:30:00] thanks [22:58:48] 10PAWS: Point paws.wmflabs.org to new PAWS setup - https://phabricator.wikimedia.org/T172257#3495443 (10yuvipanda) I'm trying to set up auto deploys to PAWS, so I'll wait till I'm done with that. [22:59:34] ooh, new paws? [23:10:10] PROBLEM - Puppet errors on tools-bastion-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:11:11] 10Data-Services, 10cloud-services-team (Kanban), 10DBA, 10Security-Team, and 5 others: Make wbqc_constraints table available on Quarry et al. - https://phabricator.wikimedia.org/T170927#3495469 (10bd808) a:03madhuvishy @madhuvishy the patch at https://gerrit.wikimedia.org/r/#/c/365969/3 is ready for your... [23:38:09] 10Cloud-VPS: Please install a recent version of node and npm by default on new labs instance - https://phabricator.wikimedia.org/T157368#3495570 (10bd808) I empathize with your frustration, but I'm not seeing a clearly actionable request yet. The note from @Jhernandez sounds like it applies to Trusty based VMs.... [23:44:11] tools-login.wmflabs.org aka tools-bastion-03 is being extremely slow at the moment. [23:44:55] Seems filesystem related at first guess. [23:46:08] Yes, +1 to anomie [23:48:20] 10VPS-project-Wikistats: wikistats: add wikimania wikis - https://phabricator.wikimedia.org/T172342#3495618 (10Reedy) [23:55:00] its almost always someone hogging NFS io [23:55:32] * bd808 tries to check it out [23:55:48] * anomie figures it's something like that [23:55:54] load avg 21.69...