[01:03:05] <Sagan>	 !log rcm CAC: running vagrant git-update
[01:03:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL
[01:18:02] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1432 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[01:53:03] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0]
[02:14:05] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1432 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[02:27:56] <shinken-wm>	 PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[02:49:04] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0]
[02:57:55] <shinken-wm>	 RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0]
[03:33:00] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1433 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[03:41:27] <zhuyifei1999_>	 !log video upgrading youtube_dl on encoding0{1..3} & frontend, from version 2017.6.18
[03:41:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[03:42:59] <bd808>	 !log tools.stewardbots Test
[03:43:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[04:02:58] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0]
[04:37:04] <wikibugs>	 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3504120 (10Ahecht)
[04:37:06] <wikibugs>	 10Tools: Tool "ircredirect" redirects to kiwiirc.com without consent - https://phabricator.wikimedia.org/T172612#3504118 (10Ahecht) 05Open>03Resolved I had some time, so I updated the tool. It will now display a disclaimer unless consent is given by appending a variable to the URL.
[04:47:44] <shinken-wm>	 PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[05:17:45] <shinken-wm>	 RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0]
[05:23:06] <JJMC89|Away>	 Hi! Could a Toolforge admin reboot https://tools.wmflabs.org/templatecount/ and https://tools.wmflabs.org/templatetransclusioncheck/. They are giving '502 Bad Gateway'. The issue has been reported by multiple users at https://en.wikipedia.org/w/index.php?oldid=794151556#Wikimedia_Toolforge_Templatecount_tool_not_working .
[05:30:41] <bd808>	 !log tools.templatecount Restarted webservice as requested on irc
[05:30:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.templatecount/SAL
[05:31:48] <bd808>	 !log tools.templatetransclusioncheck Restarted webservice as requested on irc
[05:31:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.templatetransclusioncheck/SAL
[05:31:53] <bd808>	 JJMC89: done
[05:32:28] <JJMC89>	 Thanks bd808!
[05:43:46] <shinken-wm>	 PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[06:18:46] <shinken-wm>	 RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0]
[06:32:11] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[06:34:53] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1414 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[07:07:06] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0]
[07:14:57] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:15:17] <shinken-wm>	 PROBLEM - High iowait on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: tools.tools-webgrid-lighttpd-1416.cpu.total.iowait (>33.33%)
[10:30:17] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1416 is OK: OK: All targets OK
[10:44:47] <shinken-wm>	 PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[11:02:38] <valerio-bozzolan>	 Hello! Is known a certain NFS slowness sometimes in the Tool Labs bastion 147?
[11:03:51] <valerio-bozzolan>	 I mean: https://tools.wmflabs.org/portal-stats/ access time > 60 secs sometimes without reasons.
[11:24:47] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1416 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[11:39:17] <shinken-wm>	 PROBLEM - High iowait on tools-exec-1416 is CRITICAL: CRITICAL: tools.tools-exec-1416.cpu.total.iowait (>11.11%)
[11:59:20] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1416 is OK: OK: All targets OK
[12:29:46] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0]
[13:27:48] <wikibugs>	 (03CR) 10D3r1ck01: [C: 032] Add configs, docs and credit contributors (031 comment) [labs/tools/Wikimedia-Emoji-Bot] - 10https://gerrit.wikimedia.org/r/348010 (owner: 10D3r1ck01)
[13:28:04] <wikibugs>	 (03CR) 10D3r1ck01: [V: 032 C: 032] Add configs, docs and credit contributors [labs/tools/Wikimedia-Emoji-Bot] - 10https://gerrit.wikimedia.org/r/348010 (owner: 10D3r1ck01)
[13:28:13] <wikibugs>	 (03Merged) 10jenkins-bot: Add configs, docs and credit contributors [labs/tools/Wikimedia-Emoji-Bot] - 10https://gerrit.wikimedia.org/r/348010 (owner: 10D3r1ck01)
[13:34:47] <shinken-wm>	 RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:40:10] <wikibugs>	 10VPS-project-Wikipedia-Requests: Request creation of deep-learning-services VPS project - https://phabricator.wikimedia.org/T172421#3504477 (10Ebraminio)
[16:12:13] <wikibugs>	 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3504573 (10valhallasw)
[16:12:15] <wikibugs>	 10Tool-Gerrit-Patch-Uploader: Tool "gerrit-patch-uploader" loads fork-me-on-github ribbon from Amazon AWS - https://phabricator.wikimedia.org/T172535#3504571 (10valhallasw) 05Open>03Resolved Thank you for noticing this and for the suggestions on how to fix it! I could not get the CSS option to work, so I wen...
[16:17:23] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: ee0c29f92059dfdf4a6abd3efadd073b03b32fe0 Merge "Due to rebranding of cloud i must again change my projects in this config."
[16:17:24] <wikibugs>	 10Wikibugs, 10Anti-Harassment (AHT Sprint 2): Update Wikibugs to Point Anti-Harassment to #wikimedia-anti-harassment-tools - https://phabricator.wikimedia.org/T167516#3335495 (10valhallasw) I have force-restarted the phabricator connector of wikibugs -- this should force-reload all channel config. Please let m...
[16:18:56] <wikibugs>	 10Cloud-Services, 10Patch-For-Review: Drop ldapsupportlib.py - https://phabricator.wikimedia.org/T114063#3504584 (10valhallasw) I believe this was added at some point as a backup in case someone accidentally removes themselves from the service group. @coren may remember the background story :-)
[16:34:24] <wikibugs>	 10wikitech.wikimedia.org: Labtestwikitech is completely broken while logged in - https://phabricator.wikimedia.org/T172648#3504599 (10Krenair)
[17:00:37] <Krenair>	 bd808, andrewbogott, ^
[17:01:24] <bd808>	 "fun"
[17:09:33] <wikibugs>	 10wikitech.wikimedia.org: Labtestwikitech is completely broken while logged in - https://phabricator.wikimedia.org/T172648#3504599 (10bd808) The error seems to be `Table 'labtestwiki.babel' doesn't exist`. Comes from `MediaWiki\Babel\Database::getForUser()` which is firing after login.
[17:09:57] <bd808>	 Krenair: can we actually just run update.php on that host?
[17:10:10] <Krenair>	 not sure
[17:10:34] <Krenair>	 it's been many months since I last touched this stuff
[17:10:38] <Krenair>	 but
[17:11:22] <Krenair>	 I'd think usually the restriction on update.php is because it's a terrible idea to run database schema updates against large tables with replication etc. involved
[17:11:50] <Krenair>	 might be okay to do it on labtestwikitech
[17:12:07] <bd808>	 yeah, that was kind of my thinking too
[17:12:26] <Krenair>	 (to run database schema updates automatically like that*)
[17:12:36] <bd808>	 "someday" we will move these dbs into the normal cluster and they we be forgotten less often
[17:12:50] <Krenair>	 to be honest
[17:13:14] <Krenair>	 I'd've just figured out what got changed in prod and done the equivalent
[17:13:20] <Krenair>	 manually in mysql
[17:13:48] <Krenair>	 you might be able to get away with using update.php on lab(s|test)wiki
[17:15:06] <Krenair>	 hm
[17:15:30] <Krenair>	 since it's unlikely to have been run before, you won't have anything in the wiki's updatelog table
[17:15:58] <Krenair>	 that might make it attempt some unnecessary things, not sure whether it'd be harmful or not without spending more time on it
[17:16:12] <bd808>	 yeah, update is empty there
[17:16:17] <Krenair>	 updatelog?
[17:16:32] <bd808>	 correct
[17:16:49] <bd808>	 select * from updatelog; Empty set (0.00 sec)
[17:17:54] <bd808>	 I'll just make the babel table for now
[17:18:29] <Krenair>	 for reference, my own local dev wiki (not touched for probably the same amount of time), has 32 rows there
[17:19:45] <Krenair>	 https://phabricator.wikimedia.org/P5861
[17:21:17] <Krenair>	 btw did you find that error on labtestweb2001 or fluorine/logstash/etc.?
[17:22:20] <wikibugs>	 10wikitech.wikimedia.org: Labtestwikitech is completely broken while logged in - https://phabricator.wikimedia.org/T172648#3504645 (10bd808) Table created.  ``` $ cd /srv/mediawiki/php-1.30.0-wmf.12/extensions/Babel $ sql --write labtestwiki < babel.sql $ sql labtestwiki (wikiadmin@labtestweb2001) [labtestwiki]>...
[17:22:26] <bd808>	 it was in logstash
[17:24:20] <bd808>	 login works for me, but I didn't try before adding the table.
[17:25:11] <Krenair>	 oh you know what
[17:25:13] <Krenair>	 I just remembered
[17:25:22] <Krenair>	 WikimediaMaintenance's createExtensionTables.php
[17:25:50] <Krenair>	 but your way works too
[17:26:14] <Krenair>	 https://gerrit.wikimedia.org/r/#/c/309962/
[17:28:12] <Krenair>	 bd808, yes I'm successfully logged in now
[17:28:18] <bd808>	 neat. I looked at $wgDBTableOptions before I did the manual create there just to make sure there wasn't some weird stuff it would miss
[17:29:01] <wikibugs>	 10wikitech.wikimedia.org, 10User-bd808: Labtestwikitech is completely broken while logged in - https://phabricator.wikimedia.org/T172648#3504651 (10Krenair) 05Open>03Resolved a:03bd808 That did the trick, thanks @bd808
[17:49:44] <wikibugs>	 10Toolforge: Node.js on gridengine - https://phabricator.wikimedia.org/T166830#3504665 (10edsu) The memory leak is a result of using the old version of node with the Cheerio library. If node can be upgraded that would help me. v0.10 is end-of-life and is [no longer maintained](https://github.com/nodejs/LTS#readme).
[17:54:59] <Zppix>	 bd808 is it possible to grant limited access to a tool (ssh-wise)?
[17:56:05] <valhallasw`cloud>	 Zppix: limited access in what sense?
[17:57:16] * zhuyifei1999_ does an exploit and gains full access (just kidding)
[18:00:52] <valhallasw`cloud>	 Zppix: you can give limited write rights on files, but you can't give limited 'run a command' rights
[18:01:00] <Zppix>	 ok
[18:04:01] <wikibugs>	 10Toolforge: Node.js on gridengine - https://phabricator.wikimedia.org/T166830#3504667 (10zhuyifei1999) >>! In T166830#3504665, @edsu wrote: > If node can be upgraded that would help me. v0.10 is end-of-life and is [no longer maintained](https://github.com/nodejs/LTS#readme).  >>! In T166830#3418722, @zhuyifei19...
[18:07:02] <wikibugs>	 10Toolforge: Node.js on gridengine - https://phabricator.wikimedia.org/T166830#3504668 (10zhuyifei1999) Btw, regarding EOL: because this is installed from ubuntu trusty apt, ubuntu will backport any security updates until trusty's EOL.
[20:20:50] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1402 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[20:32:38] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1440 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[21:00:51] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:07:38] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1440 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:28:21] <wikibugs>	 10Tools: Tool "jury" loads assets from a few sites, mixed http/https - https://phabricator.wikimedia.org/T172654#3504770 (10zhuyifei1999)
[22:28:32] <wikibugs>	 10Tools: Tool "jury" loads assets from a few sites, mixed http/https - https://phabricator.wikimedia.org/T172654#3504784 (10zhuyifei1999) The tool maintainer is Jörg Wiegels. I cannot find their Phabricator username.
[22:35:09] <wikibugs>	 10Tools: Tool "lalm" loads assets from OSM tiles and toolserver(?) - https://phabricator.wikimedia.org/T172655#3504785 (10zhuyifei1999)
[22:35:37] <wikibugs>	 10Tools: Tool "lalm" loads assets from OSM tiles and toolserver(?) - https://phabricator.wikimedia.org/T172655#3504799 (10zhuyifei1999) The tool maintainer is Jongleur. I'm unable to find their Phabricator username.
[22:44:33] <wikibugs>	 10Tools: Tool "languageproofing-ui" loads assets from www.languagetool.org - https://phabricator.wikimedia.org/T172656#3504804 (10zhuyifei1999)
[22:48:43] <wikibugs>	 10Tools: Tool "limesmap" loads assets from openlayers.org and opencyclemap.org - https://phabricator.wikimedia.org/T172657#3504819 (10zhuyifei1999)
[22:54:26] <wikibugs>	 10Tools: Tool "list" redirects to GitHub without consent - https://phabricator.wikimedia.org/T172658#3504837 (10zhuyifei1999)
[22:58:16] <wikibugs>	 10Tools: Tool "locator" loads Google Maps - https://phabricator.wikimedia.org/T172659#3504854 (10zhuyifei1999)
[22:59:02] <wikibugs>	 10Tools: Tool "locator" loads Google Maps - https://phabricator.wikimedia.org/T172659#3504868 (10zhuyifei1999) The other maintainer is Silver_Spoon. I cannot find their Phabricator username.
[23:04:00] <wikibugs>	 10Tools: Tool "masscamps" loads jquery and angularjs from ajax.googleapis.com - https://phabricator.wikimedia.org/T172660#3504869 (10zhuyifei1999)
[23:04:58] <wikibugs>	 10Tools: Tool "masscamps" loads jquery and angularjs from ajax.googleapis.com - https://phabricator.wikimedia.org/T172660#3504883 (10zhuyifei1999) Yuvi is the only maintainer of this tool. Should I poke @bd808? (Just poked anyhow)
[23:08:09] <wikibugs>	 10Tools: Tool "matthewrbowker-dev" loads bootstrap from bootstrapcdn - https://phabricator.wikimedia.org/T172661#3504888 (10zhuyifei1999)
[23:09:44] <wikibugs>	 10Tools: Tool "masscamps" loads jquery and angularjs from ajax.googleapis.com - https://phabricator.wikimedia.org/T172660#3504907 (10bd808) https://github.com/yuvipanda/masscamps
[23:13:05] <zhuyifei1999_>	 bd808: that was a response to yuvi saying that for most queries that would be directed to him, redirect to you :P I'm not sure if he still has the willingness to fix the tool
[23:13:40] <bd808>	 then he should shut it down :) Its not a WMCS product
[23:13:43] <zhuyifei1999_>	 (anyways, trying to figure out why the next tool loads youtube)
[23:13:46] <zhuyifei1999_>	 k
[23:18:58] <wikibugs>	 10Tools: Tool "media-reports" loads YouTube in an iframe - https://phabricator.wikimedia.org/T172662#3504914 (10zhuyifei1999)
[23:22:07] <wikibugs>	 10Tools: Tool "mediawiki-feeds" loads bootstrap and jquery from bootstrapcdn and ajax.googleapis.com - https://phabricator.wikimedia.org/T172663#3504928 (10zhuyifei1999)
[23:26:58] <wikibugs>	 10Toolforge: Node.js on gridengine - https://phabricator.wikimedia.org/T166830#3308975 (10bd808) Upgrading nodejs on Trusty hosts has been discussed before, but I'm not easily finding the phab tickets where it happened. the TL;DR is that we deploy the nodejs packages found in either the upstream vendor apt repo...