[11:14:15] !log admin create by hand /var/cache/labsaliaser/labs-ip-aliases.json in cloudservices2002-dev (T218575) [11:14:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:14:19] T218575: Reallocate LDAP database from labtestservices2001 - https://phabricator.wikimedia.org/T218575 [13:13:23] o/ Puppet in one of my labs-instances is failing because it cannot update the system and fails to reach http://mirrors.wikimedia.org. Has anything changed recently? How do I fix the issue? Thanks. [13:21:23] Why is the s8 replag so high? [13:21:53] On web.db.svc [14:39:50] bmansurov: which VM instance? [14:40:02] Zppix: no idea, I suggest you open a phab task [14:40:02] arturo: rec-wiki [14:40:23] looks like jessie-backports repo has been removed [14:40:56] I couldn't find any documentation about how I can cleanly update the instance. [14:48:27] bmansurov: the jessie-backports repository is no longer available, Debian dropped it [14:49:21] bmansurov: if your instance requires packages from jessie-backports, my suggestion is to use stretch instead [14:49:38] arturo: it doesn't, but puppet is still failing [14:49:53] arturo: I'm creating a new streatch instance too. Thanks for the suggestion. [14:50:53] Zppix, It's only 5 seconds now, It might be a fluke, I've seen worse across the lot at times [14:50:59] bmansurov: I'm doing some checks anyway [14:51:07] ok, great! [14:51:30] bmansurov: puppet agent runs fine for me [14:51:31] https://www.irccloud.com/pastebin/PCYKLvgR/ [14:51:44] arturo: because I disabled jessie-backports [14:51:47] and some other failing repos [14:51:57] bmansurov: then you are all set :-) [14:52:24] arturo: I thought it was hacky and wanted to know if there was another way of fixing the issue. ;) [14:52:57] hopefully if you delete a file which is present in puppet, the agent should re-create it [14:53:34] ok [15:03:19] I've started getting Puppet failure emails for wikiedubackups.globaleducation.eqiad.wmflabs starting a few days ago. Not sure where to start looking for the source of the problem... is this because it's debian 8.1? [15:06:49] ragesoss: try doing sudo apt-get update, and if you see errors related to jessie-backports, then you you're having issues similar to me. See above for discussion. [15:08:41] ah, yeah, that looks like it's probably the issue. [15:09:05] I don't think there's anything from backports that I'm actually using. [15:14:43] I disabled the backports .list file, now there's just one failure with `sudo apt update`, "Failed to fetch http://mirrors.wikimedia.org/debian/dists/jessie-updates/InRelease Unable to find expected entry 'main/source/Sources' in Release file (Wrong sources.list entry or malformed file)" [15:17:23] I disabled the jessie-updates entry in sources.list as well, then I had to re-install cron, and now apt runs without error. Hopefully that will take care of it. [15:23:29] 👍 [19:47:33] is there a way to submit !log messages directly from a Toolforge shell? my current internet connection is good enough for mosh but not really for IRC [19:47:50] (case in point: I already had one reconnect while typing the above message, heh) [19:49:01] I think prod has ircecho/logmsgbot, but I don’t know if those are available on Toolforge [19:52:18] (there is a `log-command-invocation` in the $PATH which sounds very promising but turns out to be a no-op, left only to avoid breaking things that used an earlier script by the same name) [20:01:59] !log tools.quickcategories deployed 9c4941bc63 [20:02:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [20:16:28] !log tools.quickcategories enable background runs for everyone [20:16:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL