[00:00:04] 6Labs, 10Tool-Labs: Backport newer tesseract-ocr-* packages from sid - https://phabricator.wikimedia.org/T118643#1806054 (10Billinghurst) @valhallasw I think that my inner stupid has a (significant) knowledge gap on doing this on toollabs, well at least some guidance/handholding. Are you saying that anyone wit... [00:23:25] 6Labs: bigbrother trying to run and restart non-existing task (wikisource-bot) - https://phabricator.wikimedia.org/T118387#1806067 (10Billinghurst) 5Open>3Resolved a:3Billinghurst no email messages. Success. Thanks. [01:50:47] !log rcm rebooted rcm-2 [01:50:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL, Master [02:23:04] PROBLEM - Puppet failure on tools-exec-1209 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [0.0] [02:48:01] 6Labs, 10wikitech.wikimedia.org, 7Wikimedia-log-errors: Hook SMWParseData::onTitleMoveComplete has invalid call signature; Parameter 3 to SMWParseData::onTitleMoveComplete() expected to be a reference, value given - https://phabricator.wikimedia.org/T118649#1806200 (10Krenair) https://gerrit.wikimedia.org/r/... [02:48:22] 6Labs, 10wikitech.wikimedia.org, 7Wikimedia-log-errors: Hook SMWParseData::onTitleMoveComplete has invalid call signature; Parameter 3 to SMWParseData::onTitleMoveComplete() expected to be a reference, value given - https://phabricator.wikimedia.org/T118649#1806202 (10Krenair) https://gerrit.wikimedia.org/r/... [03:03:10] RECOVERY - Puppet failure on tools-exec-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [03:15:48] 6Labs, 6operations: Investigate why nscd is used in labs - https://phabricator.wikimedia.org/T100564#1806213 (10yuvipanda) a:3yuvipanda [03:48:48] 10MediaWiki-extensions-OpenStackManager, 7I18n: GENDER support in openstackmanager-addedto, openstackmanager-failedtoadd - https://phabricator.wikimedia.org/T99063#1806223 (10Liuxinyu970226) [15:51:24] 6Labs, 10Labs-Infrastructure: labcontrol2001: What is it? What is it for? - https://phabricator.wikimedia.org/T118591#1806676 (10Andrew) I was in a meeting when I opened this so didn't have time to articulate further :) When we shut down pmtpa, labcontrol1001 was in danger of becoming the sole repository of... [21:32:43] legoktm: any idea what can cause [21:32:44] > mwapi.errors.APIError: rvmultpages: titles, pageids or a generator was used to supply multiple pages, but the limit, startid, endid, dirNewer, user, excludeuser, start and end parameters may only be used on a single page. -- None [21:32:56] I'm not actually using any of those params [21:32:59] I'm using rvlimit though [21:33:47] action='query', prop='revisions', titles=titles, [21:33:50] rvprop='ids', rvlimit='1', formatversion='2' [21:33:52] that's all I have [21:34:24] /w/api.php?action=query&prop=revisions&format=json&rvprop=ids&rvlimit=1&titles=Monkey%7CUSA fails [21:34:54] it works if I take out rvlimi [21:34:56] t [21:34:59] remove the limit [21:35:00] lol [21:35:22] hmm [21:35:32] I just assumed there's an actual limit separate from rvlimit... [21:35:37] oh I see [21:35:40] the error is coming from rv [21:35:44] and not from the API in general [21:35:46] nvm [22:13:37] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Gomi was created, changed by Gomi link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Gomi edit summary: Created page with "{{Tools Access Request |Justification=Create a tool for Czech Wikipedia to suggest possible internal links in the article. |Completed=false |User Name=Gomi }}" [22:42:25] I just had a bit of puppet weirdness [22:42:49] I hacked /var/lib and moved it to /srv/lib, leaving a symlink behind [22:43:02] because limn1.eqiad.wmflabs was running out of space [22:43:09] and everything seems ok except I'm getting this now: [22:43:12] Warning: Unable to fetch my node definition, but the agent run will continue: [22:43:13] Warning: Error 400 on SERVER: Permission denied - /var/lib/puppet/yaml/node [22:43:13] Info: Retrieving plugin [22:44:29] milimetric: yeah, I think moving *all* of /var/lib to a symlink might fuck a bunch of things up [22:44:39] not sure if you can recover from that easily.. [22:44:52] (a lot of security-sensitive things won't follow symlinks) [22:45:08] although in this case it might just be permission errors from the symlink having different permissions [22:59:58] right. not sure if I have any other choice, it had 0% remaining on /var. I'll have to figure out what the permission issue is [23:01:14] good luck :) [23:19:31] ok, fixed. if anyone has this problem and has to resort to my drastic methods, the puppet permission fix is - go to /var/lib/puppet on the bad instance and a separate good instance, and just chmod the bad one until they match [23:20:36] :)