[00:00:01] <YuviPanda>	 ashwinpp: depending on what you need it for, ORES' enwp10 model might also be 'good enough'
[00:00:41] <ashwinpp>	 I'll start out from English
[00:02:09] <grrrit-wm>	 (03PS1) 10Jean-Frédéric: Map source 'comcat' to commonscat for (it,it) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235644 (https://phabricator.wikimedia.org/T111286) 
[00:02:23] <ashwinpp>	 So you are saying that for example categorylinks table contains good article category, would that be right?
[00:02:47] <ashwinpp>	 YuviPanda: Can you explain what exactly is the enwp10 model?
[00:02:55] <YuviPanda>	 ashwinpp: you should talk to halfak  :)
[00:03:06] <ashwinpp>	 okay
[00:03:06] <YuviPanda>	 I only have vague ideas of these, he knows them a lot better
[00:16:21] <grrrit-wm>	 (03PS1) 10Jean-Frédéric: Map source 'categoría-Commons' to commonscat for (es,es) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235648 (https://phabricator.wikimedia.org/T111288) 
[00:18:27] <grrrit-wm>	 (03CR) 10Jean-Frédéric: [C: 032 V: 032] Map source 'comcat' to commonscat for (it,it) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235644 (https://phabricator.wikimedia.org/T111286) (owner: 10Jean-Frédéric)
[00:18:37] <grrrit-wm>	 (03CR) 10Jean-Frédéric: [C: 032] Map source 'categoría-Commons' to commonscat for (es,es) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235648 (https://phabricator.wikimedia.org/T111288) (owner: 10Jean-Frédéric)
[00:18:46] <grrrit-wm>	 (03CR) 10Jean-Frédéric: [V: 031] Map source 'categoría-Commons' to commonscat for (es,es) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235648 (https://phabricator.wikimedia.org/T111288) (owner: 10Jean-Frédéric)
[00:18:54] <grrrit-wm>	 (03CR) 10Jean-Frédéric: [V: 032] Map source 'categoría-Commons' to commonscat for (es,es) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235648 (https://phabricator.wikimedia.org/T111288) (owner: 10Jean-Frédéric)
[00:32:17] <grrrit-wm>	 (03PS1) 10Jean-Frédéric: Fix flake8 issues [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235653 
[00:32:59] <grrrit-wm>	 (03PS1) 10Jean-Frédéric: Add tox.ini file to test code [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235654 
[00:45:00] <James_F>	 Any cloudadmin around willing to make me a member of ForrestBot? :-)
[00:45:56] <Krenair>	 Isn't that a tool?
[00:46:14] <Krenair>	 legoktm, valhallasw`cloud?
[00:47:35] <grrrit-wm>	 (03CR) 10Jean-Frédéric: [C: 032 V: 032] Fix flake8 issues [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235653 (owner: 10Jean-Frédéric)
[00:48:00] <James_F>	 Krenair: They're both busy. Well, valhallasw`cloud's probably asleep. :-)
[00:52:01] <grrrit-wm>	 (03CR) 10Krinkle: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235654 (owner: 10Jean-Frédéric)
[00:53:42] <grrrit-wm>	 (03CR) 10Krinkle: [V: 032] Add tox.ini file to test code [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235654 (owner: 10Jean-Frédéric)
[00:54:59] <grrrit-wm>	 (03CR) 10Krinkle: [C: 032] Add tox.ini file to test code [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235654 (owner: 10Jean-Frédéric)
[00:56:49] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add tox.ini file to test code [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235654 (owner: 10Jean-Frédéric)
[01:03:22] <grrrit-wm>	 (03CR) 10Jean-Frédéric: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235244 (https://phabricator.wikimedia.org/T110829) (owner: 10Jean-Frédéric)
[01:04:58] <grrrit-wm>	 (03Abandoned) 10Jean-Frédéric: Do not add interwiki's on unused images pages to unbreak bot [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235244 (https://phabricator.wikimedia.org/T110829) (owner: 10Jean-Frédéric)
[01:25:06] <wikibugs>	 6Labs, 6Analytics-Backlog, 10Wikimedia-Apache-configuration, 6operations, and 3 others: https://wikitech.wikimedia.org/beacon/statsv 404 Not Found - https://phabricator.wikimedia.org/T104359#1600598 (10Krenair) 5Open>3Resolved
[01:32:19] <halfak>	 ashwinpp, still around?
[01:39:16] <grrrit-wm>	 (03PS1) 10Jforrester: Format log messages to include a timestamp [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/235663 (https://phabricator.wikimedia.org/T103416) 
[01:42:45] <grrrit-wm>	 (03PS1) 10Jforrester: Support Task: too [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/235664 
[02:30:16] <bd808>	 !log tools.forrestbot Added Jforrester as project member
[02:30:22] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.forrestbot/SAL, Master
[03:25:13] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL 28.57% of data above the critical threshold [0.0]
[04:05:13] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK Less than 1.00% above the threshold [0.0]
[04:37:29] <wikibugs>	 6Labs, 10Tool-Labs, 6Commons, 10pywikibot-core, and 2 others: Pywikibot : Fix Commons scripts broken by toolserver.org to labs migration - https://phabricator.wikimedia.org/T78462#1600865 (10jayvdb)
[05:40:42] <legoktm>	 !log tools.forrestbot switched over config to use "ReleaseTaggerBot" account
[05:40:47] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.forrestbot/SAL, Master
[05:43:44] <grrrit-wm>	 (03PS1) 10Legoktm: Forrestbot → ReleaseTaggerBot [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/235674 (https://phabricator.wikimedia.org/T100811) 
[05:43:59] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Forrestbot → ReleaseTaggerBot [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/235674 (https://phabricator.wikimedia.org/T100811) (owner: 10Legoktm)
[05:44:11] <grrrit-wm>	 (03Merged) 10jenkins-bot: Forrestbot → ReleaseTaggerBot [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/235674 (https://phabricator.wikimedia.org/T100811) (owner: 10Legoktm)
[05:46:02] <ircnotifier>	 !log tools.wikibugs legoktm: Deployed 1564da8bd2a53f9899e93497f03ba13e4a6b734f Forrestbot → ReleaseTaggerBot wb2-irc
[05:46:07] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master
[05:51:32] <grrrit-wm>	 (03CR) 10Legoktm: [C: 04-1] ""Task:" is not a recognized header per https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines" [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/235664 (owner: 10Jforrester)
[05:52:54] <grrrit-wm>	 (03CR) 10Legoktm: "Can we also keep the info level and log group?" [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/235663 (https://phabricator.wikimedia.org/T103416) (owner: 10Jforrester)
[06:04:36] <valhallasw`cloud>	 legoktm: hah. Finally :D
[06:06:23] <ostriches>	 legoktm: Ideally we'd support (Bug|Task):
[06:07:39] <legoktm>	 ostriches: ideally we'd standardize on one thing and use it :P if we do want to start using Task, there are other things that need to be updated as well. And the last time I checked, Gerrit only supported "Bug" for searching :/
[06:08:33] <ostriches>	 legoktm: Ideally, I wouldn't be working at 11pm either but we can't all get what we want! ;-)
[06:58:17] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1215 is CRITICAL 66.67% of data above the critical threshold [0.0]
[07:07:35] <valhallasw`cloud>	 !log tools uppet failure on tools-exec-1215 is CRITICAL 66.67% of data above the critical threshold -- but /var/log/puppet.log doesn't exist?!
[07:07:41] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[07:07:50] <valhallasw`cloud>	 !log tools err, is empty.
[07:07:55] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[07:09:05] <valhallasw`cloud>	 !log tools last message in puppet.log.1.gz is Error: /Stage[main]/Toollabs::Exec_environ/Package[fonts-ipafont-gothic]/ensure: change from 00303-5 to latest failed: Could not get latest version: Execution of '/usr/bin/apt-cache policy fonts-ipafont-gothic' returned 100: fonts-ipafont-gothic: (...) E: Cache is out of sync, can't x-ref a package file
[07:09:10] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[07:09:46] <valhallasw`cloud>	 !log tools and just re-running puppet solves the issue. Sigh.
[07:09:51] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[07:23:14] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1215 is OK Less than 1.00% above the threshold [0.0]
[07:50:21] <grrrit-wm>	 (03CR) 10Hashar: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235630 (https://phabricator.wikimedia.org/T111263) (owner: 10Jean-Frédéric)
[08:22:47] <wikibugs>	 10Tool-Labs-tools-Other: Filter "lonely pages" by namespace - https://phabricator.wikimedia.org/T111316#1601242 (10Nemo_bis) 3NEW
[08:52:26] <wikibugs>	 10Wikibugs: Assigning a task to someone with a name beginning with a number causes IRC colour issues - https://phabricator.wikimedia.org/T111214#1601309 (10valhallasw) We should probably use \x0303 instead of \x033.
[08:55:02] <grrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 04-1] "That would also be good, yes. I think we should set it on the output rather than on the logger object, though (pythons logging infrastruct" (031 comment) [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/235663 (https://phabricator.wikimedia.org/T103416) (owner: 10Jforrester)
[10:30:05] <wikibugs>	 6Labs, 10Labs-Infrastructure, 5Continuous-Integration-Scaling: Labs project admin can not delete per project image on Horizon - https://phabricator.wikimedia.org/T110936#1601631 (10hashar)
[10:47:12] <wikibugs>	 6Labs, 10Datasets-Archiving, 10Datasets-General-or-Unknown, 10Labs-Infrastructure, 10Wikidata: [Bug] Wikidata JSON dumps gets deleted after every new Wikidata dump - https://phabricator.wikimedia.org/T107226#1601729 (10JanZerebecki) p:5Normal>3High
[11:13:28] <grrrit-wm>	 (03PS1) 10Jean-Frédéric: Update (it,it) config for 2015 data [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235703 (https://phabricator.wikimedia.org/T111334) 
[11:25:40] <grrrit-wm>	 (03CR) 10Jean-Frédéric: "Recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235703 (https://phabricator.wikimedia.org/T111334) (owner: 10Jean-Frédéric)
[11:27:56] <grrrit-wm>	 (03CR) 10Jean-Frédéric: [C: 032] Update (it,it) config for 2015 data [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235703 (https://phabricator.wikimedia.org/T111334) (owner: 10Jean-Frédéric)
[11:28:02] <grrrit-wm>	 (03Merged) 10jenkins-bot: Update (it,it) config for 2015 data [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235703 (https://phabricator.wikimedia.org/T111334) (owner: 10Jean-Frédéric)
[13:26:13] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL 44.44% of data above the critical threshold [0.0]
[14:01:14] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK Less than 1.00% above the threshold [0.0]
[14:20:35] <myrcx>	 I'm certain I read something related day before yesterday but my memory is awful, what's up with labs/xtools-ec and etc? 
[14:21:48] <YuviPanda>	 myrcx: labs in general is pretty ok atm, as for xtools please ask the xtools maintainers? 
[14:22:41] <myrcx>	 YuviPanda: ah good to hear, was there something the other day which affected xtools and a couple of other projects or am I imagining reading that? o.O
[14:23:01] <YuviPanda>	 myrcx: there was a few hours of outage on sunday
[14:23:14] <YuviPanda>	 xtools in general is flaky sometimes, even if labs is healthy
[14:23:41] <myrcx>	 thanks YuviPanda :)
[14:38:19] <wikibugs>	 6Labs, 10wikitech.wikimedia.org, 3Labs-sprint-112, 5Patch-For-Review, and 3 others: Can't list instances on Special:NovaInstance - https://phabricator.wikimedia.org/T110629#1602403 (10Andrew) ok... I don't know what's going on now, but ping be on irc sometime and I'll turn on some logfiles.
[14:39:15] <YuviPanda>	 andrewbogott: ok, so I have ^ now
[14:39:21] <YuviPanda>	 andrewbogott: logged out and back in, all empty still
[14:40:22] <andrewbogott>	 YuviPanda: it’s happening to you too?
[14:40:31] * andrewbogott tries it
[14:41:26] <andrewbogott>	 works for me, dammit
[14:41:48] <YuviPanda>	 andrewbogott: I'm going to try logging out and back in again
[14:46:01] <wikibugs>	 6Labs, 10wikitech.wikimedia.org, 3Labs-sprint-112, 5Patch-For-Review, and 3 others: Can't list instances on Special:NovaInstance - https://phabricator.wikimedia.org/T110629#1602440 (10Samtar) Tools and bastion empty for myself, logged out and back in
[14:47:51] <YuviPanda>	 andrewbogott: logged out and back in, no dice
[14:48:08] <andrewbogott>	 ok, hang on, lemme turn on some logging
[14:48:08] <YuviPanda>	 andrewbogott: should I try removing and re-adding myself/
[14:48:09] <YuviPanda>	 ?
[14:48:11] <YuviPanda>	 andrewbogott: ok
[14:48:14] * YuviPanda waits
[14:55:09] <andrewbogott>	 well, great, logging is broken now
[14:55:36] <YuviPanda>	 haha
[14:57:31] <wikibugs>	 6Labs, 10Tool-Labs, 7Database: Potential templatelinks data integrity issue on Tool Labs' enwiki_p - https://phabricator.wikimedia.org/T111371#1602483 (10MZMcBride) 3NEW
[15:02:39] <Niharika>	 Hi, is NFS still slow? I have a job which started 40 minutes ago according to qstat stats. Another is on for 33 minutes. 
[15:03:53] <wikibugs>	 6Labs, 10Tool-Labs: Set versions in man pages from debian/changelog - https://phabricator.wikimedia.org/T87344#1602521 (10scfc)
[15:04:54] <YuviPanda>	 andrewbogott: should I remove and re-add myself?
[15:05:00] <andrewbogott>	 please don't
[15:05:05] <andrewbogott>	 can you just hit ‘reload’?
[15:05:16] <YuviPanda>	 Niharika: hi! it's a bit slow, but jobs can run for 40mins or so. how long does it usually tkae?
[15:05:18] <YuviPanda>	 andrewbogott: done
[15:05:21] <YuviPanda>	 still blank
[15:05:30] <Niharika>	 YuviPanda: Under 20. 
[15:05:46] <YuviPanda>	 Niharika: is it doing a lot of file IO?
[15:06:05] <wikibugs>	 6Labs, 10Tool-Labs: Set versions in man pages from debian/changelog - https://phabricator.wikimedia.org/T87344#1602527 (10scfc) Several sources state that the date should reflect the last time the //man page// was updated, so inserting the date of the last `changelog` entry would give a false sense of currentn...
[15:06:22] <YuviPanda>	 Niharika: and yes, NFS is still a bit slow, but shouldn't be slow enough to cause 2x decreases unless you're doing a *lot* of IO
[15:06:45] <andrewbogott>	 YuviPanda: do you have, like, 14 projects selected in your project filter?
[15:06:52] <YuviPanda>	 andrewbogott: no, one
[15:06:54] <YuviPanda>	 k8s-eval
[15:07:17] <YuviPanda>	 andrewbogott: I have, in the not-so-distant-past (a few weeks ago?) had a lot of projects selected. Might have been 14
[15:07:59] <Niharika>	 YuviPanda: Nope, not a lot of IO. It edits just one page with 1000 rows of data.
[15:08:15] <andrewbogott>	 YuviPanda: ok, reload once more please?
[15:08:35] <YuviPanda>	 Niharika: yeah, shouldn't be NFS. so two thinsg: check logs and ssh to the instance it is running on (you can see that from qstat) and see if that process is actually running?
[15:08:43] <YuviPanda>	 andrewbogott: just did, no dice
[15:09:15] <Niharika>	 YuviPanda: I'm sshed in and qstat tells me it's running. Any other way to check if it is running or not? 
[15:10:34] <Niharika>	 Oh you mean ssh to the instance. Hang on.
[15:10:38] <YuviPanda>	 Niharika: yeah, ssh into the host itself (like, tools-exec-something) and see with ps auxf
[15:10:38] <YuviPanda>	 yeah
[15:13:04] <andrewbogott>	 YuviPanda: can you confirm that the project filter shows all the projects that you’d expect?
[15:13:20] <YuviPanda>	 andrewbogott: the dropdown you mean?
[15:13:24] <Niharika>	 ssh tools-exec-1213.eqiad.wmflabs -- ssh: Could not resolve hostname tools-exec-1213.eqiad.wmflabs: nodename nor servname provided, or not known
[15:13:32] <andrewbogott>	 YuviPanda: right
[15:13:37] <YuviPanda>	 Niharika: did you try that from inside tools-login?
[15:13:48] <Niharika>	 YuviPanda: Nope. /me tries again
[15:13:49] <YuviPanda>	 andrewbogott: yeah, seems to be.
[15:15:07] <andrewbogott>	 YuviPanda: what about other pages that show project-specific information?  e.g. proxies, sudo policies, service groups...
[15:15:10] <andrewbogott>	 they all empty too?
[15:15:23] <YuviPanda>	 andrewbogott: novaproject seems ok
[15:16:21] <andrewbogott>	 can you view project quotas?
[15:17:58] <YuviPanda>	 andrewbogott: checking
[15:18:07] <YuviPanda>	 andrewbogott: no, is empty
[15:18:22] <Niharika>	 ps auxf lists only one of my two jobs. 
[15:18:27] <andrewbogott>	 puppet classes?  proxies?  sudo policies?
[15:19:22] <YuviPanda>	 andrewbogott: no security groups, no proxies, yes puppet classes, yes sudo
[15:19:36] <andrewbogott>	 ko
[15:19:40] <andrewbogott>	 *ok
[15:20:14] <andrewbogott>	 YuviPanda: ok, I need to fix logging before I can figure out what’s happening.  And it’s going to be a while.
[15:20:26] <andrewbogott>	 Can you use horizon in the meantime and leave wikitech broken?
[15:21:04] <YuviPanda>	 andrewbogott: hmm, I need to apply puppet roles to instances...
[15:21:10] <YuviPanda>	 andrewbogott: I can probably manage for the day, however.
[15:21:17] <andrewbogott>	 thanks
[15:24:17] <andrewbogott>	 Krenair: any thoughts why $wgDebugLogGroups["ldap"] = "/tmp/ldap-s-1-debug.log” doesn’t work on wikitech anymore?
[15:24:33] <Krenair>	 I think we spoke to bd808 about this at one point?
[15:25:07] <Krenair>	 https://gerrit.wikimedia.org/r/#/c/221825/
[15:25:14] <andrewbogott>	 About logging in general, but this is a specific log to a specific file...
[15:25:28] <chasemp>	 seems pretty unusual
[15:25:57] <andrewbogott>	 um...
[15:26:00] <andrewbogott>	 ok, let me rephrase...
[15:26:04] <andrewbogott>	 ‘where are my logs?'
[15:26:21] <andrewbogott>	 bd808: same question :)
[15:26:30] <grrrit-wm>	 (03CR) 10Jforrester: "Neither is "Closes:" but we support that. Be generous in what you accept and strict in what you emit and all that…" [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/235664 (owner: 10Jforrester)
[15:26:46] <wikibugs>	 6Labs, 10Tool-Labs, 7Database: Potential templatelinks data integrity issue on Tool Labs' enwiki_p - https://phabricator.wikimedia.org/T111371#1602622 (10Legoktm) From the enwiki master: ``` mysql:wikiadmin@db1052 [enwiki]> select tl_namespace, tl_title from page join templatelinks on tl_from = page_id where...
[15:28:09] <andrewbogott>	 Krenair: I don’t have any context for that patch (which, in any case, isn’t merged.)  Does that imply that the logs are going to some central server now instead?
[15:28:41] <Krenair>	 andrewbogott, I think it inplies that wgDebugLogGroups is not working now
[15:31:59] <andrewbogott>	 hm, inserting the code from that patch doesn’t get me any further
[15:40:26] <Krenair>	 andrewbogott, yes well, we did already try the code and find it didn't work
[15:40:30] <Krenair>	 I'm not that surprised
[15:40:47] <andrewbogott>	 I figured — but hope springs eternal :)
[15:45:36] <grrrit-wm>	 (03PS2) 10Niedzielski: Copy alpha build from merge job instead of making [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/231697 (https://phabricator.wikimedia.org/T99115) 
[15:46:20] <grrrit-wm>	 (03CR) 10Niedzielski: "ready 4 review" [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/231697 (https://phabricator.wikimedia.org/T99115) (owner: 10Niedzielski)
[16:02:27] <wikibugs>	 6Labs, 10Tool-Labs, 7Database: Potential templatelinks data integrity issue on Tool Labs' enwiki_p - https://phabricator.wikimedia.org/T111371#1602719 (10Krenair) db1055 returns empty set, but the three labsdb hosts all give the result shown in the task description. I was wondering about db1069 (the host bet...
[16:17:18] <grrrit-wm>	 (03PS1) 10Greg Grossmeier: Deprecate gitblit, and migrate gerrit [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/235754 
[16:26:35] <grrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 032] Deprecate gitblit, and migrate gerrit [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/235754 (owner: 10Greg Grossmeier)
[16:26:48] <grrrit-wm>	 (03Merged) 10jenkins-bot: Deprecate gitblit, and migrate gerrit [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/235754 (owner: 10Greg Grossmeier)
[16:26:57] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: 6790b1bed753260168db20bb78dcfa5726be2aec Deprecate gitblit, and migrate gerrit
[16:27:00] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master
[16:37:30] <James_F>	 valhallasw`cloud: Sanity-check – https://phabricator.wikimedia.org/T101919 is effectively done now we just add tags and don't comment, right? The only issue is if you re-run after a new branch is cut and we label everything as wmf-X+1 rather than wmf-X, but that's a different issue about assuming things…
[16:38:06] <valhallasw`cloud>	 James_F: yes, and there's probably an edge case where the tag was removed after it was added
[16:38:24] <James_F>	 valhallasw`cloud: Hmm. Good enough to mark 'resolved', though?
[16:38:25] <YuviPanda>	 andrewbogott: any luck btw?
[16:38:36] <valhallasw`cloud>	 James_F: I think so. When would we run old emails through again?
[16:39:23] <James_F>	 valhallasw`cloud: If the bot dies and gets restarted.
[16:39:51] <valhallasw`cloud>	 James_F: oh, when there's an exception. Yeah, I see
[16:40:11] <valhallasw`cloud>	 I'd consider it solved as far as that is concerned
[16:40:22] <James_F>	 valhallasw`cloud: Also https://phabricator.wikimedia.org/T101920 :-)
[16:43:33] <greg-g>	 thanks for the merge valhallasw`cloud 
[16:47:37] <grrrit-wm>	 (03CR) 10Jforrester: Format log messages to include a timestamp (031 comment) [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/235663 (https://phabricator.wikimedia.org/T103416) (owner: 10Jforrester)
[16:47:44] <grrrit-wm>	 (03PS2) 10Jforrester: Format log messages to include a timestamp [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/235663 (https://phabricator.wikimedia.org/T103416) 
[16:51:08] <grrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 032] Format log messages to include a timestamp [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/235663 (https://phabricator.wikimedia.org/T103416) (owner: 10Jforrester)
[16:52:36] <James_F>	 valhallasw`cloud: Thanks. :-)
[16:52:42] <James_F>	 valhallasw`cloud: What about https://gerrit.wikimedia.org/r/#/c/235664/ ? ;-)
[16:54:24] <valhallasw`cloud>	 James_F: that probably needs some code in another file as well? this just makes Task: available in the dict
[16:54:53] <James_F>	 valhallasw`cloud: Oh, yeah.
[16:54:54] * James_F fixed.
[16:54:56] <James_F>	 Err.
[16:54:58] * James_F fixes.
[16:56:38] <grrrit-wm>	 (03PS2) 10Jforrester: Support Task: too [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/235664 
[16:56:43] <James_F>	 valhallasw`cloud: Good thing one of us is paying attention. ;-)
[16:58:01] <grrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 032] "when we get the message, we're already too late to complain about the wrong keyword, so I think it's better to try our best in being accep" [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/235664 (owner: 10Jforrester)
[16:59:08] <Cyberpower678>	 YuviPanda, why am seeing multiple instances of tasks running in my bot tools account
[16:59:18] <andrewbogott>	 YuviPanda: I need bd808 to show up and explain about how to make logs work
[16:59:20] <Cyberpower678>	 YuviPanda, did a restore go wrong?
[16:59:37] <YuviPanda>	 Cyberpower678: some are probably ghosts and not actually running
[16:59:47] <YuviPanda>	 Cyberpower678: ssh into the instances listed and check if they are actually running?
[16:59:50] <YuviPanda>	 Cyberpower678: is this wikihistory?
[17:00:00] <Cyberpower678>	 YuviPanda, Cyberbot
[17:00:10] <Cyberpower678>	 Everything is listed twice.
[17:00:45] <YuviPanda>	 Cyberpower678: so one of the after effects is 'ghost' jobs that don't exist in reality but just in qstat
[17:00:54] <YuviPanda>	 Cyberpower678: can you check the hosts to see if it actually is running?
[17:01:05] <YuviPanda>	 you can ssh to the hosts  printed in qstat from tools-login
[17:01:10] <YuviPanda>	 and use ps to check if they exist
[17:01:59] <Cyberpower678>	 The host is cut off.
[17:02:35] <Cyberpower678>	 YuviPanda, ^
[17:02:42] <YuviPanda>	 Cyberpower678: 'qstat -xml' or you can use 'qstat -j <job-id>'
[17:03:18] <Cyberpower678>	 access denied
[17:03:27] <Cyberpower678>	 unable to ssh into the node
[17:03:54] <Cyberpower678>	 YuviPanda, ^
[17:04:04] <YuviPanda>	 Cyberpower678: unfortunately I don't have time to debug this right now :( can you file a bug, or alternatively just delete the ones you don't want?
[17:04:21] <Cyberpower678>	 I don't which one is running or not.
[17:04:49] <gifti>	 i can ssh there …
[17:05:09] <YuviPanda>	 Cyberpower678: ok, then file a bug and I can look into it tomorrow?
[17:05:09] <Cyberpower678>	 YuviPanda, Can you delete all of the jobs except deadlinksbot?
[17:05:17] <YuviPanda>	 Cyberpower678: you can do that too
[17:05:26] * YuviPanda is going to go off soon, is 7pm here.
[17:05:34] <Cyberpower678>	 There has to be a fast way to do it.
[17:05:43] <Cyberpower678>	 There are over 30 items listed.
[17:05:51] <YuviPanda>	 well, awk and xargs?
[17:06:20] <Cyberpower678>	 ??
[17:06:31] <YuviPanda>	 those are linux commands that might be able to help
[17:06:34] <Cyberpower678>	 YuviPanda, must I remind you that I suck with linux
[17:06:38] <Cyberpower678>	 :\
[17:06:40] <YuviPanda>	 Cyberpower678: yes, and I suggest you fix that :)
[17:06:45] <YuviPanda>	 and am offering you pointers on what to learn
[17:07:17] <YuviPanda>	 Tool Labs *is* a linux system.
[17:07:31] <Cyberpower678>	 I know
[17:07:39] <Cyberpower678>	 I'm not that dumb. :p
[17:12:00] <Cyberpower678>	 There we go
[17:12:09] <Cyberpower678>	 YuviPanda, I feel smart now. :p
[17:19:53] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1408 is CRITICAL 60.00% of data above the critical threshold [0.0]
[17:34:42] <Krenair>	 andrewbogott, want to do https://gerrit.wikimedia.org/r/235768 now?
[17:34:50] <andrewbogott>	 Krenair: yes please, if you do not mind
[17:34:53] <andrewbogott>	 I was just about to ask :)
[17:35:15] <andrewbogott>	 YuviPanda: how is wikitech looking now?
[17:35:52] <Luke081515>	 Wikitech works for me
[17:36:08] <YuviPanda>	 andrewbogott: ok for me as well
[17:36:19] <andrewbogott>	 YuviPanda: can you see instances now?
[17:37:18] <Krenair>	 urrrggghhh
[17:37:47] <YuviPanda>	 andrewbogott: yeah, but i could since I removed myself and re-added
[17:38:02] <andrewbogott>	 YuviPanda: wait, you removed and re-added yourself to one project, and that fixed /all/ projects?
[17:38:35] <YuviPanda>	 andrewbogott: uh, I don't know if I had trouble with other projects - I had only one thing selected in my project filter at that time
[17:38:48] <andrewbogott>	 Krenair: urgh?
[17:39:13] <Krenair>	 andrewbogott, sorry, just the same mess I told you about yesterday
[17:39:16] <Krenair>	 figured it out and sync'd
[17:39:21] <andrewbogott>	 ah, right
[17:39:23] <andrewbogott>	 ok, thank you!
[17:39:43] <andrewbogott>	 Krenair: I’m sorry to let that patch die, but I really need to stop fixing wikitech so I can get back to killing wikitech :)
[17:39:49] <Krenair>	 heh
[17:40:48] <wikibugs>	 6Labs, 10wikitech.wikimedia.org, 3Labs-sprint-112, 5Patch-For-Review, and 3 others: Can't list instances on Special:NovaInstance - https://phabricator.wikimedia.org/T110629#1603382 (10Andrew) So... once again I believe this to be resolved (mostly because I reverted the patches that most likely caused it.)
[17:45:11] <Luke081515>	 Maybe this could interest you: Today, at the morning, I see this bug, but at, I guess 11:45 h UTC I could see my instances. But at the moment I can't see instances
[17:50:36] <Luke081515>	 andrewbogott: Was there a change on wikitech between 8 AM UTC and 11:45 AM UTC?
[17:51:12] <andrewbogott>	 Luke081515: we just now made some changes that made things better for most people.  Can you try logging out and back in?
[17:51:21] <andrewbogott>	 (I’m sure this will the the 100th time today :(  )
[17:51:21] <Luke081515>	 ok, wait a moment
[17:52:27] <Luke081515>	 andrewbogott: I can't see instances at the moment
[17:52:39] <andrewbogott>	 goddamn
[17:52:45] <andrewbogott>	 ok, hang on
[17:53:36] <Luke081515>	 andrewbogott: The button to create new instances is also not visible
[17:53:39] <Luke081515>	 Maybe this helps
[17:54:50] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1408 is OK Less than 1.00% above the threshold [0.0]
[18:29:33] <grrrit-wm>	 (03CR) 10Multichill: [C: 04-1] "Seems to be overlapping with the existing one." (032 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235630 (https://phabricator.wikimedia.org/T111263) (owner: 10Jean-Frédéric)
[18:47:32] <grrrit-wm>	 (03CR) 10Jean-Frédéric: "What do you mean by overlapping? These are different lists, with different monument type." (032 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/235630 (https://phabricator.wikimedia.org/T111263) (owner: 10Jean-Frédéric)
[18:52:17] <andrewbogott>	 Luke081515: any different now?
[19:19:23] <symac>	 Hi all, trying to use the ProxyCommand option to access my instance (as explained there: https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyCommand_ssh_option_.28recommended.29 ) I get the following error message: "channel 0: open failed: administratively prohibited: open failed" any idea of what I might be missing?
[19:21:00] <valhallasw`cloud>	 symac: incorrect hostname?
[19:26:37] <symac>	 valhallasw`cloud: I have copied the config to the end of my .ssh/config and am now trying to ssh symacbot.eqiad.wmflabs as explqined on the help page, where should a hostname be missing?
[19:27:17] <wikibugs>	 6Labs, 10wikitech.wikimedia.org, 3Labs-sprint-112, 5Patch-For-Review, and 3 others: Can't list instances on Special:NovaInstance - https://phabricator.wikimedia.org/T110629#1603733 (10scfc) Hurray! … in the sense that it works now for both tools and toolsbeta for "Tim Landscheidt" :-).  I'm hesitant to clo...
[19:27:51] <valhallasw`cloud>	 valhallasw@bastion-01:~$ ping symacbot.eqiad.wmflabs
[19:27:51] <valhallasw`cloud>	 ping: unknown host symacbot.eqiad.wmflabs
[19:29:16] <valhallasw`cloud>	 symac: oh! do you mean you want to login to the symacbot tool on tool labs?
[19:30:02] <valhallasw`cloud>	 an 'instance' in wikitech-speak is a server, so a tool is not an instance.
[19:30:19] <valhallasw`cloud>	 instead, login to login.tools.wmflabs.org, then `become symacbot`
[19:30:28] <symac>	 oh my bad :( 
[19:30:49] <symac>	 valhallasw`cloud: that is what I am doing at the moment, I wanted to bypass the become instruction, seems it is not possible
[19:33:02] <symac>	 valhallasw`cloud: thanks for your help
[19:33:10] <valhallasw`cloud>	 you're welcome!
[19:33:23] <valhallasw`cloud>	 unfortunately, I don't think there's an easy way to automatically become
[19:35:49] <Luke081515>	 andrewbogott: Sorry, I was away till now, but it works know, thanks
[21:07:02] <wikibugs>	 6Labs, 10Tool-Labs, 7Database: Potential templatelinks data integrity issue on Tool Labs' enwiki_p - https://phabricator.wikimedia.org/T111371#1604222 (10MZMcBride)
[21:07:04] <wikibugs>	 6Labs, 10Wikimedia-Labs-General, 6operations, 7Database, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1604221 (10MZMcBride)
[21:07:55] <wikibugs>	 6Labs, 10Tool-Labs, 7Database: Potential templatelinks data integrity issue on Tool Labs' enwiki_p - https://phabricator.wikimedia.org/T111371#1602483 (10MZMcBride) This task seems to be very similar to T106470.
[21:11:41] <grrrit-wm>	 (03PS1) 10Dzahn: CI: add fake SSH keys to unbreak compiler [labs/private] - 10https://gerrit.wikimedia.org/r/235871 
[21:13:39] <grrrit-wm>	 (03CR) 10John F. Lewis: [C: 031] CI: add fake SSH keys to unbreak compiler [labs/private] - 10https://gerrit.wikimedia.org/r/235871 (owner: 10Dzahn)
[21:14:09] <grrrit-wm>	 (03PS2) 10Dzahn: CI: add fake SSH keys to unbreak compiler [labs/private] - 10https://gerrit.wikimedia.org/r/235871 (https://phabricator.wikimedia.org/T111450) 
[21:14:42] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] CI: add fake SSH keys to unbreak compiler [labs/private] - 10https://gerrit.wikimedia.org/r/235871 (https://phabricator.wikimedia.org/T111450) (owner: 10Dzahn)
[21:14:55] <grrrit-wm>	 (03CR) 10Dzahn: [V: 032] CI: add fake SSH keys to unbreak compiler [labs/private] - 10https://gerrit.wikimedia.org/r/235871 (https://phabricator.wikimedia.org/T111450) (owner: 10Dzahn)
[22:09:28] <James_F>	 valhallasw`cloud: Did you mean for https://gerrit.wikimedia.org/r/#/c/235663/ and https://gerrit.wikimedia.org/r/#/c/235664/ to merge and be deployed? ;-)
[22:19:43] <valhallasw`cloud>	 James_F: heh, yes
[22:19:49] <James_F>	 valhallasw`cloud: :-)
[22:20:27] <valhallasw`cloud>	 (Asleep now)
[22:46:09] <wikibugs>	 6Labs, 7Tracking: Create project for Phlogiston - https://phabricator.wikimedia.org/T111461#1604703 (10JAufrecht) 3NEW a:3bd808
[22:53:54] <wikibugs>	 6Labs, 7Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#1604731 (10bd808)
[22:53:56] <wikibugs>	 6Labs, 7Tracking: Create project for Phlogiston - https://phabricator.wikimedia.org/T111461#1604729 (10bd808) 5Open>3Resolved https://wikitech.wikimedia.org/wiki/Nova_Resource:Phlogiston
[23:23:06] <wikibugs>	 6Labs: Create project for Phlogiston - https://phabricator.wikimedia.org/T111461#1604765 (10Legoktm)