[00:25:33] jdlrobson: aren't you a maintainer of 'styleguide', according to http://tools.wmflabs.org/?tool=styleguide ? [00:27:29] you're free to unsubscribe, but CSP may prevent that tool from working in the future [00:33:05] (uh, CSP may not prevent it, but the webproxy can) [00:42:32] PROBLEM - Puppet errors on tools-exec-1442 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:12:16] PROBLEM - Puppet errors on tools-exec-1424 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:17:34] RECOVERY - Puppet errors on tools-exec-1442 is OK: OK: Less than 1.00% above the threshold [0.0] [01:52:16] RECOVERY - Puppet errors on tools-exec-1424 is OK: OK: Less than 1.00% above the threshold [0.0] [02:05:40] (03PS1) 10Lokal Profil: Make api handle sparql harvested source fields [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370770 (https://phabricator.wikimedia.org/T172841) [02:06:18] (03PS2) 10Lokal Profil: Make scripts dealing with the sparql source field deal with sparql harvested data [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370481 (https://phabricator.wikimedia.org/T171300) [02:06:34] (03CR) 10jerkins-bot: [V: 04-1] Make api handle sparql harvested source fields [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370770 (https://phabricator.wikimedia.org/T172841) (owner: 10Lokal Profil) [02:11:43] (03PS2) 10Lokal Profil: Make api handle sparql harvested source fields [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370770 (https://phabricator.wikimedia.org/T172841) [02:12:35] (03CR) 10jerkins-bot: [V: 04-1] Make api handle sparql harvested source fields [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370770 (https://phabricator.wikimedia.org/T172841) (owner: 10Lokal Profil) [02:13:16] PROBLEM - Puppet errors on tools-exec-1424 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [02:25:11] (03PS3) 10Lokal Profil: Make api handle sparql harvested source fields [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370770 (https://phabricator.wikimedia.org/T172841) [02:53:16] RECOVERY - Puppet errors on tools-exec-1424 is OK: OK: Less than 1.00% above the threshold [0.0] [03:35:45] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [03:48:03] (03PS1) 10Lokal Profil: Harvest monument_article via sparql [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370775 (https://phabricator.wikimedia.org/T172842) [03:56:34] 10wikitech.wikimedia.org, 10Deployment-Systems, 10User-MarcoAurelio: Create a Gadget to easily add/remove/modify patches for SWAT at wikitech:Deployments - https://phabricator.wikimedia.org/T171940#3480850 (10Legoktm) If it were possible to use VE to edit the Deployments page I think all of the pain would in... [04:00:44] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [04:56:18] 10wikitech.wikimedia.org, 10Deployment-Systems, 10User-MarcoAurelio: Create a Gadget to easily add/remove/modify patches for SWAT at wikitech:Deployments - https://phabricator.wikimedia.org/T171940#3511162 (10mmodell) @legoktm: indeed, that would be a fairly good solution. The current deployment calendar inv... [05:10:28] PROBLEM - Puppet errors on tools-worker-1011 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [05:31:05] 10PAWS, 10Wikimania-Hackathon-2017: Validating Jupyter notebooks cited from Wikimedia sites - https://phabricator.wikimedia.org/T172848#3511198 (10Daniel_Mietchen) [05:33:46] 10PAWS, 10Wikimania-Hackathon-2017: Validating Jupyter notebooks cited from Wikimedia sites - https://phabricator.wikimedia.org/T172848#3511210 (10Daniel_Mietchen) [05:45:28] RECOVERY - Puppet errors on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [07:02:24] PROBLEM - Puppet errors on tools-worker-1017 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:37:24] RECOVERY - Puppet errors on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [07:39:57] 10Data-Services, 10DBA: LabsDB infrastructure pending work - https://phabricator.wikimedia.org/T153058#3511295 (10jcrespo) [09:50:10] 10Tools, 10WikimediaUI Style Guide: Tool "styleguide" redirects to GitHub without consent - https://phabricator.wikimedia.org/T172834#3511505 (10Prtksxna) [09:50:22] 10Tools, 10WikimediaUI Style Guide: Tool "styleguide" redirects to GitHub without consent - https://phabricator.wikimedia.org/T172834#3510793 (10Prtksxna) a:03Prtksxna This would be my fault from {T154703}, sorry! I have removed the redirect now and added a notice instead. ``` $ curl -I http://tools.wmflabs... [09:53:36] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3511519 (10zhuyifei1999) [09:53:38] 10Tools, 10WikimediaUI Style Guide: Tool "styleguide" redirects to GitHub without consent - https://phabricator.wikimedia.org/T172834#3511517 (10zhuyifei1999) 05Open>03Resolved >>! In T172834#3511505, @Prtksxna wrote: > @zhuyifei1999 Do we need anything else to resolve this? Nope. Thanks! [10:42:31] PROBLEM - Puppet errors on tools-bastion-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:44:50] 10Toolforge, 10Documentation, 10Kubernetes: Create visual diagram of documented components of Toolforge Kubernetes cluster - https://phabricator.wikimedia.org/T172855#3511407 (10Aklapper) [11:00:45] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:17:30] RECOVERY - Puppet errors on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:21:07] 10Tool-stewardbots, 10User-MarcoAurelio: New page design - https://phabricator.wikimedia.org/T172870#3511751 (10MarcoAurelio) [11:21:23] 10Tool-stewardbots, 10User-MarcoAurelio: README.md is outdated - https://phabricator.wikimedia.org/T172871#3511765 (10MarcoAurelio) [11:24:50] 10Tool-stewardbots, 10User-MarcoAurelio, 10WorkType-Maintenance: Delete old data and/or stop logging to stewardbots' SULWatcher SQL DB - https://phabricator.wikimedia.org/T151113#3511780 (10MarcoAurelio) I'm going to re-ping @jcrespo and @Marostegui because I'm unsure about how to do this properly and withou... [11:33:57] hey after sql_local; how I can connect to the db I want? like s1234_abc? [11:34:12] sql s1234_abc does not work [11:34:46] i'll try "use" [11:35:16] 10Tool-stewardbots, 10User-MarcoAurelio, 10WorkType-Maintenance: Delete old data and/or stop logging to stewardbots' SULWatcher SQL DB - https://phabricator.wikimedia.org/T151113#3511817 (10jcrespo) For what I see, this is a user database (meaning it is not a replicated mediawiki table, right?). If that is t... [11:35:28] 10Tools: Tool "sqid" loads assets from bootstrapcdn, fonts.googleapis.com, and code.jquery.com - https://phabricator.wikimedia.org/T172833#3511818 (10Mmarx) p:05Triage>03Normal a:03Mmarx Taking this. Tracking progress in [Github issue #123](https://github.com/Wikidata/SQID/issues/123). [11:35:53] TabbyCat: can you expand on what you want and what you get? [11:36:18] use is the right sql command to change current database [11:36:32] you can also disconnect and conenct to a different db [11:36:54] 10Tool-stewardbots, 10User-MarcoAurelio, 10WorkType-Maintenance: Delete old data and/or stop logging to stewardbots' SULWatcher SQL DB - https://phabricator.wikimedia.org/T151113#3511824 (10MarcoAurelio) Some context: ``` MariaDB [s51541_sulwatcher]> describe logging; +-------------+------------------+----... [11:38:16] jynus: so I've got circa 300k useless records in the logging table and I want to get rid of them but at the same time I don't want the bot to stop logging there [11:39:00] well, for that, the table must allow row-level locking [11:39:09] 10Tool-stewardbots, 10User-MarcoAurelio, 10WorkType-Maintenance: Delete old data and/or stop logging to stewardbots' SULWatcher SQL DB - https://phabricator.wikimedia.org/T151113#3511826 (10Marostegui) In addition to Jaime's comments, when issuing drops, it is generally a good idea to do them with "if exists... [11:39:23] that usually means InnoDB [11:39:48] DELETE * FROM table_name; ? [11:39:50] on MyISAM any other write would block the table [11:40:12] if you want to delete all, TRUNCATE will be faster [11:40:33] but that will delete all rows [11:40:45] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [11:41:20] make sure you do after truncate, ALTER TABLE table_name ENGINE=InnoDB; that will avoid locking issues in the future [11:45:33] hmm, seems too complicated [11:46:09] can't I simply issue delete * from logging where l_timestamp = '20170301000000'; ? [11:46:24] older than I mean [11:47:10] hey, you were the first person to ask for advice :-) [11:48:17] I am not telling you what to do, just answering your question in the best way possible ;-) [11:48:33] I know, I know [11:48:56] but I'm also concerned that we're wasting resources with those records and I don't know what to do :( [11:51:58] your table takes 60MB, honestly, there are worse things [11:52:19] we have 300GB tables that users refuse to trim [11:52:40] I have a 25 GB table that I'll probably drop in a week :) [11:52:40] jynus: the unindexed flagged_revs tables with millions of records perhaps ;) [12:05:56] jynus: will the "delete * from loggin keep intact the index, etc or will it start from 1 again? [12:08:48] he left [13:17:53] 10Data-Services, 10XTools: s51187 and p50380g50692 database users are generating excessive lag on replica service - https://phabricator.wikimedia.org/T172882#3512008 (10jcrespo) [13:20:06] 10Data-Services, 10XTools: s51187 and p50380g50692 database users are generating excessive lag on replica service - https://phabricator.wikimedia.org/T172882#3512023 (10jcrespo) @madhuvishy @bd808 can you help me reach the owner(s) of p50380g50692__DPL_p , I don't know which tool corresponds to that database/u... [13:22:12] 10Data-Services, 10XTools: s51187 and p50380g50692 database users are generating excessive lag on replica service - https://phabricator.wikimedia.org/T172882#3512026 (10jcrespo) [13:32:53] PROBLEM - Puppet errors on tools-worker-1019 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:06:31] 10Toolforge, 10Wikisource, 10Bengali-Sites: Update the "tesseract-ben" package on Toolforge for OCR on Bengali Wikisource - https://phabricator.wikimedia.org/T167566#3512216 (10Tpt) Should be done. I am not able to make sure it's actually live. [14:07:52] RECOVERY - Puppet errors on tools-worker-1019 is OK: OK: Less than 1.00% above the threshold [0.0] [14:13:24] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3512226 (10Tgr) [14:13:26] 10Tools: Tool "remarkup2wikitext" loads assets from code.jquery.com and Google Analytics - https://phabricator.wikimedia.org/T172826#3512224 (10Tgr) 05Open>03Resolved Uhh, it must have been mindless copypasting of some HTML boilerplate. It uses neither jQuery nor GA (and GA still used the sample siteid `ŲA-X... [14:19:10] !log tools.heritage Deploy latest from Git master: 25023b6, d556d52, 56cd469, e15709d, 576a6d4, 550fb2d, 57d4f07, d2980f5 [14:19:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [14:28:43] !log tools.heritage Deploy latest from Git master: 4bb0c12 (T165759, T165759), 35b20ec, c3be5fe, 196c165, 7b8dcb2, eac0756, e15a912 (T172094), 6c195db, 85b415c (T112460) [14:28:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [14:28:51] T165759: [Wikidata2MonumentsDB] Store harvested wikidata back into monuments_all - https://phabricator.wikimedia.org/T165759 [14:28:51] T112460: Source links in the monuments database get too long and are truncated - https://phabricator.wikimedia.org/T112460 [14:28:51] T172094: npm 3 requires peer dependencies to be explicitly defined in package.json - https://phabricator.wikimedia.org/T172094 [14:31:03] jynus: re T172882 it seems to be dplbot [14:31:06] T172882: s51187 and p50380g50692 database users are generating excessive lag on replica service - https://phabricator.wikimedia.org/T172882 [14:31:32] (i did a grep 50692 ../*/...DATA.olduser) [14:31:48] annika: thanks [14:33:45] 10Data-Services, 10XTools: s51187 and p50380g50692 database users are generating excessive lag on replica service - https://phabricator.wikimedia.org/T172882#3512356 (10jcrespo) [14:35:29] 10Data-Services, 10XTools: s51187 and p50380g50692 database users are generating excessive lag on replica service - https://phabricator.wikimedia.org/T172882#3512008 (10jcrespo) I am told the DPL database may be dplbot tool. [14:36:53] 10Data-Services, 10XTools: s51187 and p50380g50692 database users are generating excessive lag on replica service - https://phabricator.wikimedia.org/T172882#3512372 (10MusikAnimal) We are very close to killing the old edit counter which creates these temporary tables. In the meantime, please feel free to bloc... [14:41:52] 10Data-Services, 10XTools: s51187 and p50380g50692 database users are generating excessive lag on replica service - https://phabricator.wikimedia.org/T172882#3512385 (10jcrespo) Thanks, converting them to InnoDB will make them not lock the database- I will do that for Xtools. [15:15:15] 10Toolforge: Require a Phabricator account as a prerequisite to getting tool labs access - https://phabricator.wikimedia.org/T172899#3512572 (10Reedy) [15:20:06] 10Toolforge: Require a Phabricator account as a prerequisite to getting tool labs access - https://phabricator.wikimedia.org/T172899#3512572 (10Paladox) I think all new accounts with ldap was stopped a while ago. But old accounts can still use ldap. All new accounts are from mediawiki. [15:20:43] 10Toolforge: Get toolforge users who don't currently have phabricator accounts to get them - https://phabricator.wikimedia.org/T172901#3512609 (10Reedy) [15:23:45] 10Toolforge: Require a Phabricator account as a prerequisite to getting tool labs access - https://phabricator.wikimedia.org/T172899#3512630 (10chasemp) p:05Triage>03Normal [15:24:14] 10Toolforge: Require a Phabricator account as a prerequisite to getting tool labs access - https://phabricator.wikimedia.org/T172899#3512638 (10Reedy) {F8998238} Looks fine to me? [15:25:56] 10Toolforge: Require a Phabricator account as a prerequisite to getting tool labs access - https://phabricator.wikimedia.org/T172899#3512644 (10Paladox) @Reedy they need to update that. I believe someone around wmf has been meaning to update it but did not find the source code to do that. Does new accounts work... [15:34:50] 10Toolforge: Require a Phabricator account as a prerequisite to getting tool labs access - https://phabricator.wikimedia.org/T172899#3512572 (10bd808) I have never heard of any plan to disable LDAP account linking. There may be some sort of desire to make the Wikimedia account show up first so that people find t... [15:35:24] 10Toolforge: Require a Phabricator account as a prerequisite to getting Toolforge access - https://phabricator.wikimedia.org/T172899#3512718 (10bd808) [15:37:44] 10Striker: Require a Phabricator account as a prerequisite to getting Toolforge access - https://phabricator.wikimedia.org/T172899#3512733 (10bd808) It would be pretty easy to enforce the Phabricator account linking that is done at https://toolsadmin.wikimedia.org/profile/settings/accounts as a part of the Toolf... [16:05:45] Getting reports about replica.cnf missing for one of my tools - https://github.com/Krinkle/mw-tool-orphantalk/issues/7 [16:06:09] Possibly a recurring problem that we saw earlier this year where UIDs changed underneath a running kubernetes container? [16:08:18] I can access it fine from tools-login and from webservice shell [16:08:21] The replica.my.cnf file [16:08:25] bd808: ^ [16:08:38] I'll leave it for debugging it you like, but can also just issue a restart which I know will fix it [16:08:48] https://tools.wmflabs.org/orphantalk/ [16:35:53] 10Data-Services, 10XTools: s51187 and p50380g50692 database users are generating excessive lag on replica service - https://phabricator.wikimedia.org/T172882#3513085 (10russblau) Well, this came as quite a surprise, but I've gone ahead and converted all of dplbot's user databases to use InnoDB. (Hard to believ... [16:37:49] (03CR) 10Jean-Frédéric: [C: 032] Harvest monument_article via sparql [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370775 (https://phabricator.wikimedia.org/T172842) (owner: 10Lokal Profil) [16:47:55] (03CR) 10Jean-Frédéric: Make scripts dealing with the sparql source field deal with sparql harvested data (031 comment) [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370481 (https://phabricator.wikimedia.org/T171300) (owner: 10Lokal Profil) [16:50:37] (03CR) 10Jean-Frédéric: "Posting comments." (032 comments) [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370770 (https://phabricator.wikimedia.org/T172841) (owner: 10Lokal Profil) [16:59:01] 10Tools: Tool "masscamps" loads jquery and angularjs from ajax.googleapis.com - https://phabricator.wikimedia.org/T172660#3513168 (10yuvipanda) I don't think anyone is using this project, and I've no time, so am happy to just shut the tool :) [17:28:13] Krinkle: I restarted it since it had sat for a bit, I don't think it was going to get the debug attention it deserves this week [17:28:24] !log tools webservices restart tools.orphantalk [17:28:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:33:58] chasemp: Thanks :) [17:38:56] (03PS1) 10Krinkle: api: Fix 'replag.lagged' data to look at all instead of just the last shard [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/370856 [17:39:01] (03CR) 10Krinkle: [C: 032] api: Fix 'replag.lagged' data to look at all instead of just the last shard [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/370856 (owner: 10Krinkle) [17:39:34] (03Merged) 10jenkins-bot: api: Fix 'replag.lagged' data to look at all instead of just the last shard [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/370856 (owner: 10Krinkle) [17:58:48] (03PS1) 10Mobrovac: Add certificates for restbase101[678] [labs/private] - 10https://gerrit.wikimedia.org/r/370859 [18:02:47] (03CR) 10Filippo Giunchedi: [C: 031] Add certificates for restbase101[678] [labs/private] - 10https://gerrit.wikimedia.org/r/370859 (owner: 10Mobrovac) [18:03:48] (03CR) 10Filippo Giunchedi: [V: 032 C: 032] Add certificates for restbase101[678] [labs/private] - 10https://gerrit.wikimedia.org/r/370859 (owner: 10Mobrovac) [18:17:36] 10Cloud-Services, 10Toolforge: #tool-labs workboard is crazy - https://phabricator.wikimedia.org/T157698#3513505 (10bd808) This was at least partially addressed by {T167244} which created a hierarchy of projects starting from the #cloud-services root (which was a rename of the prior `#labs` project). The #clou... [18:19:54] 10Data-Services, 10Toolforge: Drop database s53003__xtools_prod - https://phabricator.wikimedia.org/T170645#3437979 (10bd808) @Samwilson have you tried running `DROP DATABASE s53003__xtools_prod` as the `s53003` user? [18:47:10] 10Tools: Cross-site scripting warning between tools and tools-static - https://phabricator.wikimedia.org/T172932#3513601 (10Slashme) [18:50:46] 10Toolforge, 10Tools: Cross-site scripting warning between tools and tools-static - https://phabricator.wikimedia.org/T172932#3513641 (10Slashme) [18:54:31] 10Toolforge, 10Tools: Cross-site scripting warning between tools and tools-static - https://phabricator.wikimedia.org/T172932#3513601 (10valhallasw) That's not an XSS warning but a performance warning :-) This should be solveable by loading jquery directly in a script tag, rather than building-then-document.wr... [19:08:33] PROBLEM - Puppet errors on tools-docker-registry-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:18:12] 10cloud-services-team (FY2017-18), 10Goal: Define a metric to track OpenStack system availability - https://phabricator.wikimedia.org/T167556#3513806 (10Andrew) Here are some user-facing things that I'd like to have metrics for: * Openstack APIs ** Keystone API availability ** Nova API availability ** Designa... [19:21:12] PROBLEM - Puppet errors on tools-k8s-master-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:22:21] PROBLEM - Puppet errors on tools-docker-registry-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:22:45] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:24:25] PROBLEM - Puppet errors on tools-worker-1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:26:12] stashbot seems to be having problems bd808 ^^ [19:32:30] (03PS1) 10Jean-Frédéric: Fix case of `monument_article` parameter in ro_ro configuration [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/370871 (https://phabricator.wikimedia.org/T172931) [19:35:35] 10Toolforge: Get toolforge users who don't currently have phabricator accounts to get them - https://phabricator.wikimedia.org/T172901#3512609 (10Luke081515) Is there a quick method to check which LDAP accounts are not connected to Phabricator? Probably these users may already have Phab Accounts, but only with M... [19:39:23] RECOVERY - Puppet errors on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [19:41:46] bd808: re stashbot, I tried to restart and got: [19:41:46] tools.stashbot@tools-bastion-03:~$ ./bin/stashbot.sh tail [19:41:47] Error from server: container stashbot.bot-4055467412-iuwt7 is not valid for pod stashbot.bot-4055467412-af5u6 [19:41:57] !log tools.heritage Queried job status using `qstat`, scheduled the update_monument for deletion using `qdel`, and rescheduling a new one. [19:42:11] JeanFred: stashbot isn't here right now :( [19:42:36] the cloud team is all doing a presentation right now :) I'll get bryan on it after [19:44:22] greg-g: :-( [19:44:36] I'm not but I don't know anything about stashbot. I can look in a few minutes [19:44:46] andrewbogott: ohai :) [19:44:54] andrewbogott: they'll be done soon (15 minutes) [19:44:59] Thanks for the heads up :) [19:45:06] I wouldn't worry unless you want to investigate [19:45:26] !log's can be backfilled [19:46:37] (03CR) 10Lokal Profil: [C: 032] Fix case of `monument_article` parameter in ro_ro configuration [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/370871 (https://phabricator.wikimedia.org/T172931) (owner: 10Jean-Frédéric) [19:46:50] I'm getting 502s on tool labs... [19:47:15] Matthew_: say more. what do you see where doing what? [19:47:48] http://tools.wmflabs.org/matthewrbowker/ Accessing that returns a 502 bad gateway error. [19:48:04] http://tools.wmflabs.org/articlerequest/ same [19:48:20] (03Merged) 10jenkins-bot: Fix case of `monument_article` parameter in ro_ro configuration [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/370871 (https://phabricator.wikimedia.org/T172931) (owner: 10Jean-Frédéric) [19:49:37] random clicks on https://tools.wmflabs.org/admin/tools don't give me 502s [19:50:13] though I did get one on https://tools.wmflabs.org/icommons/ [19:51:30] Huh... is it just tools I maintain then? Because I'm getting it on matthewrbowker, matthewrbowker-dev, articlerequest, articlerequest-dev, xtools-ec, and xtools-articleinfo just in a quick spot check for tools I can remember off the top of my head. [19:52:26] static too... [19:52:38] https://tools.wmflabs.org/static/res/bootstrap/3.1.1/css/bootstrap.min.css [19:52:57] I don’t seem to be able to log in to Striker − is that a known issue ? (I might just be doing something wrong, in which case I’ll try harder :) [19:53:13] there were some alerts for k8s components, so it could be that just k8s webtools are affected? [19:53:43] stashbot error also suggests something off with k8s [19:53:55] JeanFred: works for me [19:54:31] (03PS1) 10Lokal Profil: Catch yet another exception [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/370878 [19:54:36] andrewbogott: Thanks − will try harder then :) [19:55:00] http://tools.wmflabs.org/ - Also 502 [19:55:15] looks like stashbot fails on any msg [19:55:36] any from irc [19:55:46] something wrong with the way he handles the msgs? [19:55:49] cc andrewbogott bd808 [19:55:58] articlerequest just recovered... [19:56:13] Sagan: I see it, looking... [19:56:40] andrewbogott: I believe both the webservice and stashbot are indications of something off with k8s. [19:56:42] the problem seemed to have started around 8:20pm [19:56:56] kubectl logs static-2745485710-ynhcp doesn't seem to work for tools.static (which is the webservice pod) [19:57:36] (03CR) 10Jean-Frédéric: [C: 032] Catch yet another exception [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/370878 (owner: 10Lokal Profil) [19:58:08] stashbot tries to post to /irc-2017.08/irc (elasticsearch, I assume), gets a 401 in return and crashes [19:58:14] kubectl logs stashbot.bot-4055467412-af5u6 [19:58:30] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 327 bytes in 1.003 second response time [19:59:23] « string 'Magnus' not found » :D [19:59:28] (03Merged) 10jenkins-bot: Catch yet another exception [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/370878 (owner: 10Lokal Profil) [20:00:00] test [20:00:08] meh [20:00:11] andrewbogott: we just got done presenting, hopping on now about? [20:00:44] looks like stashbot did not registered my msg, but crashed since jouncebot posted at operations [20:02:24] (03CR) 10jenkins-bot: Fix case of `monument_article` parameter in ro_ro configuration [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/370871 (https://phabricator.wikimedia.org/T172931) (owner: 10Jean-Frédéric) [20:02:45] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [20:03:27] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 4851 bytes in 0.029 second response time [20:04:16] PROBLEM - Puppet errors on tools-bastion-05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:06:15] RECOVERY - Puppet errors on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:06:53] bd808: Does Striker support well diacritics in Login names ? :) [20:07:43] !log tools.stashbot restarted k8s container via /data/project/stashbot/bin/stashbot.sh restart, still broken [20:08:12] (03CR) 10jenkins-bot: Catch yet another exception [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/370878 (owner: 10Lokal Profil) [20:08:33] RECOVERY - Puppet errors on tools-docker-registry-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:08:34] valhallasw`cloud: seems there was some event that effected k8s things and maybe is not happening atm? [20:08:50] !log tools.stashbot stopped stashbot; cannot authenticate to elasticsearch and is flapping [20:11:56] valhallasw`cloud: ehm... [20:13:17] better now? [20:13:20] hm, no [20:16:20] icinga says all k8 workers are okay at operations [20:16:28] and it looks like stashbot is stable again [20:16:34] JeanFred: try logging? [20:20:14] JeanFred: in theory, yes. In practice... do you have a reproducible problem? [20:20:37] !log tools.stashbot seems to be running again [20:20:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [20:21:45] Sagan: hahaha, I had not actively realized that !log tools.stashbot stashbot is broken doesn't work if stashbot is down :D [20:22:57] Sagan: Just did − no luck [20:23:12] JeanFred: it should work now again, like it does above [20:23:15] valhallasw`cloud: :) [20:23:46] bd808: Trying to log in as `Jean-Frédéric` [20:24:38] JeanFred: let me see if I can spot anything in the server logs... [20:26:41] JeanFred: I'm not seeing any errors logged, so whatever is happening is at least not a crash problem. [20:27:05] Thanks for looking bd808 [20:27:07] that doesn't mean there's not a bug though ;) [20:27:18] I’ll try changing passwords just in case [20:27:20] RECOVERY - Puppet errors on tools-docker-registry-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:28:11] a cli search for 'cn=Jean-Frédéric' works so if there is a problem then its probably in python string marshaling [20:31:24] Changed password − can log in fine on Wikitech but not on Striker :-( [20:33:20] ok. that smells a lot like a striker bug then [20:33:25] Hm. django-auth-ldap works all on unicode strings, the underlying pyldap library assumes everything is utf-8. [20:36:01] python utf8 is unicode... right? [20:36:56] utf8 is an encoding which supports all characters in unicode [20:37:08] JeanFred: would you mind filing a bug? I can see if I can recreate/debug in my local dev instance [20:37:27] what I mean with 'assumes everything is utf-8' is that all .encode() and .decode()'s use utf-8 as encoding (i.e. it assumes byte strings are utf-8 encoded text) [20:37:57] https://github.com/pyldap/pyldap/tree/0871162d05be6c7ae09221451521aafb5c12307a#bytestext-management has a note on bytes vs text [20:39:20] RECOVERY - Puppet errors on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [20:39:27] 10Striker, 10User-bd808: Can’t log into Striker, potentially because of accents in logins - https://phabricator.wikimedia.org/T172949#3514139 (10JeanFred) [20:39:35] Here you go :) [20:39:40] thank you [20:40:40] 10Tools: Tool "copyvios" loads assets from code.jquery.com - https://phabricator.wikimedia.org/T172397#3514157 (10Earwig) 05Open>03Resolved I fixed it. Thanks. [20:40:41] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3514159 (10Earwig) [21:04:18] what's the current fix for a newly created deployment-prep instance with this error? [21:04:21] Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster02.deployment-prep.eqiad.wmflabs] [21:04:49] godog: that's using a project-local puppetmaster, right? [21:04:55] I seemed to remember rm -rf /var/lib/puppet/ssl/ but that clearly didn't do it [21:04:57] So probably you have to rm -rf /var/lib/puppet/ssl [21:05:00] andrewbogott: that's deployment-prep yeah [21:05:08] and then probably sign things on the puppetmaster [21:05:13] I can have a look if you want [21:05:39] andrewbogott: thanks! yeah that'd be helpful [21:06:09] I seemed to remember autosign was on but clearly not [21:06:12] godog: what instance? [21:06:30] andrewbogott: deployment-puppetmaster02.deployment-prep.eqiad.wmflabs [21:06:35] no... [21:06:43] deployment-imagescaler02 [21:07:33] godog: now it's just saying 'no certificate found and waitforcert is disabled' [21:07:38] is that different from what you saw? [21:07:51] That seems like just waiting for signing on the master [21:08:05] yeah that's what I'm seeing too now after nuking the ssl dir [21:08:23] OK, doing 'puppet cert sign deployment-imagescaler02.deployment-prep.eqiad.wmflabs' on puppetmaster02 [21:09:11] yay! works now [21:09:14] thanks andrewbogott ! [21:09:28] have to sign the salt cert too, wherever that is... [21:10:02] * andrewbogott constantly surprised and alarmed by vim + stretch + mouse [21:10:23] yeah, we should get mouse-=a in production puppet [21:13:01] well, I tried fixing the salt key by deleting and re-signing and it's still complaining [21:13:14] so I'm done caring unless someone else wants me to [22:30:04] 10Cloud-Services, 10Toolforge: #tool-labs workboard is crazy - https://phabricator.wikimedia.org/T157698#3514477 (10zhuyifei1999) It's much better now, with the separation of #data-services from #toolforge [23:16:32] 10Tools: Tool "suggestor" loads botstrap from bootstrapcdn - https://phabricator.wikimedia.org/T172962#3514572 (10zhuyifei1999) [23:22:57] 10Tools: Tool "text2hash" loads assets from bootstrapcdn, unpkg.com, and google - https://phabricator.wikimedia.org/T172963#3514588 (10zhuyifei1999) [23:23:37] 10Tools: Tool "text2hash" loads assets from bootstrapcdn, unpkg.com, and google - https://phabricator.wikimedia.org/T172963#3514604 (10zhuyifei1999) [23:26:03] 10Tools: Tool "traffic-grapher" loads jquery from code.jquery.com - https://phabricator.wikimedia.org/T172964#3514609 (10zhuyifei1999) [23:26:23] 10Tools: Tool "traffic-grapher" loads jquery from code.jquery.com - https://phabricator.wikimedia.org/T172964#3514623 (10zhuyifei1999) The tool maintainer is KevinPayravi. I cannot find their Phabricator username. [23:28:16] (03CR) 10Lokal Profil: Make api handle sparql harvested source fields (032 comments) [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370770 (https://phabricator.wikimedia.org/T172841) (owner: 10Lokal Profil) [23:30:48] 10Tools: Tool "translate" loads bootstrap from bootstrapcdn - https://phabricator.wikimedia.org/T172965#3514629 (10zhuyifei1999) [23:34:07] 10Tools: Tool "videotutorials" loads buttons from i.creativecommons.org, w3.org, and licensebuttons.net - https://phabricator.wikimedia.org/T172966#3514648 (10zhuyifei1999) [23:34:58] PROBLEM - Puppet errors on tools-worker-1010 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [23:36:27] 10Tools: Tool "wdq2sparql" loads fork-me-on-github ribbon from github - https://phabricator.wikimedia.org/T172967#3514665 (10zhuyifei1999) [23:38:19] 10Tools: Tool "wembedder" loads jquery from cloudflare - https://phabricator.wikimedia.org/T172968#3514680 (10zhuyifei1999) [23:41:46] 10Tools: Tool "wikidata-nolabels" loads jquery and bootstrap from code.jquery.com and bootstrapcdn - https://phabricator.wikimedia.org/T172969#3514695 (10zhuyifei1999) [23:42:51] (03CR) 10Lokal Profil: Make scripts dealing with the sparql source field deal with sparql harvested data (031 comment) [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370481 (https://phabricator.wikimedia.org/T171300) (owner: 10Lokal Profil) [23:46:25] (03CR) 10Jean-Frédéric: [C: 032] Make api handle sparql harvested source fields [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370770 (https://phabricator.wikimedia.org/T172841) (owner: 10Lokal Profil) [23:46:37] (03CR) 10Jean-Frédéric: [C: 032] Make scripts dealing with the sparql source field deal with sparql harvested data [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370481 (https://phabricator.wikimedia.org/T171300) (owner: 10Lokal Profil) [23:47:16] 10Tools: Tool "wikidata-primary-sources" loads assets from online.swagger.io - https://phabricator.wikimedia.org/T172970#3514710 (10zhuyifei1999) [23:47:20] (03Merged) 10jenkins-bot: Make api handle sparql harvested source fields [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370770 (https://phabricator.wikimedia.org/T172841) (owner: 10Lokal Profil) [23:47:35] (03Merged) 10jenkins-bot: Make scripts dealing with the sparql source field deal with sparql harvested data [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370481 (https://phabricator.wikimedia.org/T171300) (owner: 10Lokal Profil) [23:47:40] (03Merged) 10jenkins-bot: Harvest monument_article via sparql [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370775 (https://phabricator.wikimedia.org/T172842) (owner: 10Lokal Profil) [23:48:03] 10Tools: Tool "wdq2sparql" loads fork-me-on-github ribbon from github - https://phabricator.wikimedia.org/T172967#3514728 (10Smalyshev) 05Open>03Resolved Replaced the ribbon with tools-static one. [23:48:11] (03CR) 10jenkins-bot: Make api handle sparql harvested source fields [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370770 (https://phabricator.wikimedia.org/T172841) (owner: 10Lokal Profil) [23:48:55] (03CR) 10jenkins-bot: Make scripts dealing with the sparql source field deal with sparql harvested data [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370481 (https://phabricator.wikimedia.org/T171300) (owner: 10Lokal Profil) [23:49:40] (03CR) 10jenkins-bot: Harvest monument_article via sparql [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/370775 (https://phabricator.wikimedia.org/T172842) (owner: 10Lokal Profil) [23:50:39] 10Tools: Tool "wikidata-slicer" loads fonts from google - https://phabricator.wikimedia.org/T172971#3514733 (10zhuyifei1999) [23:53:17] 10Tools: Tool "wikidata-timeline" loads Google Analytics - https://phabricator.wikimedia.org/T172972#3514749 (10zhuyifei1999)