[00:36:10] Hey. I have a labs instance - commtech-1 on commtech which hasn't come back up since the reboot yesterday. Its a mwvagrant service so I tried doing a `vagrant up` but I get the message - The provider 'lxc' could not be found, but was requested to [00:36:10] back the machine 'default'. Please use a provider that exists. [00:36:37] The instance was working fine since a long while and has some data/settings that I wouldn't want to lose. [00:36:42] IdeaS? [01:04:33] Niharika: that sounds like the shell magic to replace `vagrant` with `/usr/local/bin/mwagrant` isn't happening [01:04:42] *mwvagrant [01:04:57] without that wrapper script things go sideways [01:07:52] Niharika: hmmm... even calling the script directly I get that message. something is a bit messed up [01:09:11] Niharika: puppet is broken there due to missing packages. I'll see if I can fix it [01:18:38] !log commtech Manually addd missing trusty-backports repo to /etc/apt/sources.list on commtech-1 [01:18:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commtech/SAL [01:21:14] !log commtech Forced puppet run and ran /usr/local/bin/mwvagrant up on commtech-1 [01:21:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commtech/SAL [01:21:32] Niharika: ^ it should be working now. No idea how it got that messed up [03:30:38] bd808: Thank you!! I owe you a cupcake! [07:41:52] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/404921 (owner: 10L10n-bot) [11:36:43] !log codesearch legoktm@codesearch2:~$ sudo systemctl enable hound_proxy [11:36:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL [11:42:00] !log tools T178717 aborrero@tools-clushmaster-01:~$ clush -w @all 'sudo puppet agent --test' [11:42:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:42:06] T178717: Upgrade wmcs instances and masters to puppet 4.8 - https://phabricator.wikimedia.org/T178717 [11:46:47] (03PS1) 10Legoktm: Use `docker rm -f` to ensure previous image is always gone [labs/codesearch] - 10https://gerrit.wikimedia.org/r/404964 [11:46:49] (03PS1) 10Legoktm: Add enable.sh to easily run `systemd enable hound-[...]` [labs/codesearch] - 10https://gerrit.wikimedia.org/r/404965 [11:49:59] (03CR) 10Legoktm: [V: 032 C: 032] Use `docker rm -f` to ensure previous image is always gone [labs/codesearch] - 10https://gerrit.wikimedia.org/r/404964 (owner: 10Legoktm) [11:50:08] (03CR) 10Legoktm: [V: 032 C: 032] Add enable.sh to easily run `systemd enable hound-[...]` [labs/codesearch] - 10https://gerrit.wikimedia.org/r/404965 (owner: 10Legoktm) [12:11:06] !log tools T178717 aborrero@tools-webgrid-generic-1402:~$ sudo apt-upgrade trusty-wikimedia -v [12:11:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:11:11] T178717: Upgrade wmcs instances and masters to puppet 4.8 - https://phabricator.wikimedia.org/T178717 [12:24:17] !log tools T178717 aborrero@tools-exec-1401:~$ sudo apt-upgrade trusty-wikimedia -v [12:24:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:24:23] T178717: Upgrade wmcs instances and masters to puppet 4.8 - https://phabricator.wikimedia.org/T178717 [13:32:18] !log codesearch rebooting codesearch2 to see how well it recovers [13:32:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL [13:37:38] it recovered properly \o/ [13:39:14] :) [13:44:52] !log tools upgrade wikimedia packages on tools-bastion-05 [13:44:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:52:31] !log tools T178717 aborrero@tools-clushmaster-01:~$ clush -f 1 -w @all 'sudo facter | grep lsbdistcodename | grep trusty && sudo apt-upgrade trusty-wikimedia -v' [13:52:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:52:37] T178717: Upgrade wmcs instances and masters to puppet 4.8 - https://phabricator.wikimedia.org/T178717 [14:02:08] chasemp: Hey! I have a minor cleanup (https://gerrit.wikimedia.org/r/#/c/404980) which seems fine according to puppet compiler on production, but I'm wondering if there might be strange use cases on labs that I'm not aware of... [14:02:47] Basically, I'm just removing a default conf file that isn't used for the mediawiki class, as it is overriden in the corresponding profile. [14:02:58] gehel: I think it's ok, but can you ping andrew and bd808 on that task as they would be more savvy about it? [14:03:00] * chasemp nods [14:03:13] chasemp: sure, will add them as reviewers [14:03:14] yeah seems reasonable, if I were writing the sam epatch though I would bug bryan :) [14:03:34] ok, so let's bug the right people! [14:03:58] heh [14:05:10] Thanks! [14:14:46] (03PS1) 10Legoktm: Add OOUI [labs/codesearch] - 10https://gerrit.wikimedia.org/r/404989 (https://phabricator.wikimedia.org/T185157) [14:15:39] (03CR) 10Legoktm: [V: 032 C: 032] Add OOUI [labs/codesearch] - 10https://gerrit.wikimedia.org/r/404989 (https://phabricator.wikimedia.org/T185157) (owner: 10Legoktm) [15:14:37] (03PS1) 10Legoktm: Add basic health check end point [labs/codesearch] - 10https://gerrit.wikimedia.org/r/404996 [15:14:56] (03CR) 10Legoktm: [V: 032 C: 032] Add basic health check end point [labs/codesearch] - 10https://gerrit.wikimedia.org/r/404996 (owner: 10Legoktm) [15:18:25] (03PS1) 10Legoktm: Fix "starting up" health check [labs/codesearch] - 10https://gerrit.wikimedia.org/r/404999 [15:18:35] (03CR) 10Legoktm: [V: 032 C: 032] Fix "starting up" health check [labs/codesearch] - 10https://gerrit.wikimedia.org/r/404999 (owner: 10Legoktm) [15:24:00] (03PS1) 10Legoktm: Link to health check in exception message [labs/codesearch] - 10https://gerrit.wikimedia.org/r/405002 [15:24:28] (03CR) 10Legoktm: [V: 032 C: 032] Link to health check in exception message [labs/codesearch] - 10https://gerrit.wikimedia.org/r/405002 (owner: 10Legoktm) [15:42:32] !log tools T178717 aborrero@tools-clushmaster-01:~$ clush -w @all 'sudo puppet agent -t -v' [15:42:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:42:37] T178717: Upgrade wmcs instances and masters to puppet 4.8 - https://phabricator.wikimedia.org/T178717 [15:48:18] chasemp: I see lots of this [15:48:24] https://www.irccloud.com/pastebin/HUEe0SoN/ [15:48:51] arturo: I think that those messages were holding on for the Puppet 4 upgrade and now andrewbogott can do a thing to remove it [15:48:58] I hae a vague sense of the pending work there only tho [15:49:04] s/hae/have [15:49:16] if toolforge is upgraded to 4.x then yeah I can stop those [15:49:32] https://gerrit.wikimedia.org/r/#/c/404480/ [15:49:57] sweet [15:50:11] great [15:50:23] arturo: is it all upgraded? [15:50:52] andrewbogott: it should be. I can get a report on dpkg -s puppet across the fleet to make sure [15:51:02] sure [15:51:06] and I'll merge that patch [15:51:12] want to re-title it first [15:51:21] I see other weird things, like: `tools-clushmaster-01.tools.eqiad.wmflabs: Warning: Found multiple default providers for service: debian, runit; using debian` [15:51:52] arturo: ah, that came w/ Puppet 4 too, someone specific init or someone isntalled runit there by accident or dep I imagine [15:52:18] afaik runit should be nowhere but it creeped in as a dep of something in some places [15:54:45] arturo: do a sweep to see if runit is anywhere and why? [15:55:26] more weird things: [15:55:31] https://www.irccloud.com/pastebin/4HjetLdf/ [15:55:50] chasemp: sure, give me a minute [15:57:16] andrewbogott: all is puppet 4 [15:57:23] great! [15:57:54] more weird things: [15:57:58] https://www.irccloud.com/pastebin/Gf8sHKmU/ [15:58:11] should we open task for them? [15:59:25] more warnings, this one seems unrelated to the upgrade, but NFS? [15:59:29] https://www.irccloud.com/pastebin/QSTY3Zi1/ [16:00:18] arturo: that happens every once in awhile and we are not sure why [16:00:26] arturo: is that persistent if you run puppet again there? [16:00:43] chasemp: not tested, but is the first time I see it [16:00:55] arturo: there is at least one task with a rabbit hole of information [16:01:11] this one seems related to grid engine/NFS [16:01:14] basically peridically a check that tries to make sure the pupept run is happening while NFS is mounted doesn't return well [16:01:16] https://www.irccloud.com/pastebin/Xppr4FQs/ [16:01:23] yes [16:01:33] there is an ealry check for some grid work in puppet that makes sure NFS is mounted [16:01:42] when that fails it's a long list of dependency things that warn [16:01:46] okay [16:01:56] but the weird thing is it's not consistent and difficult to catch [16:02:18] I don't really know what's up, we thought for a long time it was because of load on the isntances / periods of high IO squeezing out Puppet [16:02:21] but I don't think that's true [16:02:27] it shows up in places where that wouldn't make sense [16:02:49] if you can debug that please do, I reached a place where I realized it wasn't a trivial issue and havent been able to come back [16:05:59] arturo: it's maybe a procedural issue with how NFS is checked, or it's a bug we keep tripping, or it really is a load issue we don't understand [16:06:02] ok [16:06:03] I have many questions [16:07:08] chasemp: the runit thing: is installed because a thing called 'vblade', no idea if that's something we use [16:07:14] https://www.irccloud.com/pastebin/jz7IdOvp/ [16:07:29] arturo: no, and it was other places too [16:07:35] hang on [16:07:52] arturo: https://phabricator.wikimedia.org/T182781 [16:07:53] similar [16:08:43] I see [16:08:55] so, shall I just purge it? [16:10:15] makes sense to me [16:11:45] !log tools aborrero@tools-clushmaster-01:~$ sudo aptitude purge vblade vblade-persist runit (for something similar to T182781) [16:11:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:11:53] T182781: Puppet broken on labstore1004 - https://phabricator.wikimedia.org/T182781 [16:11:53] arturo: someone at some point (tm) installed vblade tooling and I have no idea why [16:11:58] so it's all very reassuring [16:14:47] np [17:16:22] hello, is the "Rolling of infrastructure for kernel patching" still happening? Not sure if you forgot to update the channel topic (which everyone forgets about) [17:16:35] andrewbogott hi, im wondering could there be some kind of support for using a hiera code to allow us to add more repos that should be updated with the git sync script please? [17:17:25] I've created a customised icinga2 module that's not prod ready so i have to add it seperatly. But the puppetmaster is using labs and the instances are using the puppetmaster. [17:17:34] (I've pinged you as you built the script :)) [17:18:07] you're talking about the script that updates the puppet repo on local puppetmasters? Or something else? [17:18:25] Yep the script that updates the repo on the local puppetmaster [17:21:53] what's the name of the script? [17:22:21] I suspect that you'll be best off just copy/pasting and adding a second cron but I can look at having it take n repos rather than just one [17:22:59] andrewbogott git-sync-upstream [17:24:44] hm… right now that script has a dependency link built in (if it fails to update the main repo then it doesn't update the main repo). So it's not as simple as just injecting a list vs. a hard-coded repo name [17:26:12] paladox: go ahead and make a phab task for this — I don't know that I'll be able to work on it right away. It's not a terrible idea though [17:26:27] Ok thanks [17:39:57] andrewbogott i guess what it needs to do is iterate over for example hiera_example: - : and use that in repo_rebase [17:55:37] bd808: Thanks for the explanation on T115650. Appreciated! [17:55:38] T115650: Create an authoritative and well promoted catalog of Wikimedia tools - https://phabricator.wikimedia.org/T115650 [17:56:25] musikanimal: I think you're right that we are done now. I'll fix the topic [17:57:32] d3r1ck: you're welcome. It is a bit surprising when you first run into it. The Cloud projects are some of the only ones in a deep tree [17:58:26] Yeah! Actually I had faced this issue with a task I filed on phab [17:58:32] I added "Developer-Relations" [17:58:55] Then when I tried adding "Developer-Relations (Jan - March)", Developer-Relations was removed [17:59:12] Now, the explanation you gave cleared everything in my mind, thanks [17:59:24] It's really strange when this is encountered the first time. :) [18:40:41] Is it possible to adjust permissions/add users to user databases? [18:41:13] Would like to access a user database from another tool, not vital but preferable to the alternative. [18:41:36] (that database's creator is fine with it, he just isnt sure if he can change permissions for it) [18:43:33] Samwalton9: I can't remember anyone ever asking before, this would need a task before we do anything, but has the other user tried to allow this already? [18:45:36] He think he tried and wasn't allowed in the past. Basically the use case is I set up a tool for testing changes to another tool, and would like to access a database set up by the main tool. Maybe that wasn't the best way to go about it, and I can just set up a dummy/copy db for the testing tool if it's not possible. [18:47:21] Samwalton9: must it be restricted to these two people? Otherwise you could make it a public database by putting "_p" after the name [18:48:49] I guess not, I'll suggest that and if not just go with a copy/dummy db. [18:49:32] Samwalton9: it has been done in the past for sure. I don't think we have any standard policy for how to track this today, but we should probably come up with something. A phab task with the request/requirements would be a good place to start. [18:50:13] the old wiki replica servers had a lot of one-off hand built things that we are trying to automate instead for the new servers [18:51:29] modules/role/files/labs/db/maintain-dbusers.py has the grants hardcoded in it it seems so that's the place to start, and I agree with bd808 this would to be codified and repeatable to be sane [18:56:30] Created a task here, let me know if anything in the task description is unclear. Feel free to update the description with info from above: https://phabricator.wikimedia.org/T185242 [21:41:05] !help sorry technical difficults. What exactly is the different in the logging table between log action "modify" and "protect"? Thanks! [21:41:05] tomthirteen: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [21:45:28] hi tomthirteen. !help is usually more about something related to Cloud VPS or Toolforge infrastructure, but lets see if I can help you figure this out [21:46:26] !log phabricator deleting puppetdb-phabricator1 as it's unused (puppetdb is now hosted on puppet-phabricator) [21:46:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL [21:46:40] tomthirteen: have you tried looking in https://www.mediawiki.org/wiki/Manual:Database_layout for more info? [21:48:10] tomthirteen: I think if you drill into the logging table description there and then into the log_action section you'll eventually find https://www.mediawiki.org/wiki/Manual:Log_actions [21:48:22] o/ Can anyone help me with ssh'ing to a labs instance? I can ssh to instances that were created some time ago, but the ones that I created the past two days don't allow me to ssh. [21:48:53] bmansurov: what error are you getting, and how are you trying to connect? [21:49:23] chicocvenancio: I have set up a proxy in ~/.ssh/config and the error is bmansurov@wiki.recommendation-api.eqiad.wmflabs: Permission denied (publickey). [21:49:53] tomthirteen: it looks like "protect" is for the first protection action (from unprotected to protected) and modify is for changes to the protection (like changing the duration) [21:50:26] bd808 thank you. Yes, I understand that. My question has to do with IP edits. [21:50:51] It seems that there is a status like semi-protected where no IP address can edit the page. [21:51:27] almost any protection would disable ip edits I think. autoconfirmed certainly would [21:51:28] But there is a status for a page where an IP address can edit a page, but it needs to be approved first by an administrator [21:51:48] I cannot see in the logging page where this applies [21:52:37] hmm... approved by an admin sounds like https://www.mediawiki.org/wiki/Extension:Approved_Revs [21:52:40] (03PS1) 10Gerrit Patch Uploader: Remove uneeded hosts [labs/icinga2] - 10https://gerrit.wikimedia.org/r/405201 [21:52:43] (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [labs/icinga2] - 10https://gerrit.wikimedia.org/r/405201 (owner: 10Gerrit Patch Uploader) [21:52:59] Zppix you know you can use inline edits to create changes? [21:53:01] bmansurov: can you send your ssh config? [21:53:10] (03CR) 10Paladox: [C: 032] Remove uneeded hosts [labs/icinga2] - 10https://gerrit.wikimedia.org/r/405201 (owner: 10Gerrit Patch Uploader) [21:53:27] (03CR) 10Zppix: [V: 032] Remove uneeded hosts [labs/icinga2] - 10https://gerrit.wikimedia.org/r/405201 (owner: 10Gerrit Patch Uploader) [21:53:41] chicocvenancio: pm'ed [21:54:05] How can I tell in the logging table, or whichever table, where a page status allows IP address to edit, but with approval only [21:54:10] bmansurov: that host may not have come up correctly. I can't get in with my root key either [21:54:32] Sorry to be a pain, but I can figure this out. [21:54:33] bd808: I see, that's the 3rd or 4th instance that I recreated [21:54:53] Stretch is the problem I guess? [21:55:12] bmansurov: is there a project local puppet master? [21:55:17] tomthirteen: page_restrictions table? [21:55:39] bd808: not sure [21:55:45] How do I find out? [21:57:51] bmansurov: let me poke at the project in horizon and see if there is anything that makes sense for the failures [21:57:59] Reedy, thank you but how on this page do I figure it out https://www.mediawiki.org/wiki/Manual:Page_restrictions_table#pr_type [21:58:04] bd808: ok, thanks! [21:58:11] chicocvenancio: thanks for starting on that one! :) [21:58:16] tomthirteen: Query for it? [21:58:41] Get the page id, filter by pr_page [21:58:45] https://tools.wmflabs.org/openstack-browser/project/recommendation-api [21:58:55] bd808 bmansurov ^^ [21:59:06] dosen't look like a puppetmaster is in there [21:59:30] I see that you applied role::labs::mediawiki_vagrant. [21:59:45] Was that added straight away? [21:59:52] Ah [22:00:05] bd808 role::labs::mediawiki_vagrant isen't there a problem on stretch with that class? [22:00:20] ah! yes paladox. [22:01:01] yep. [22:01:12] bmansurov: try making your base VM jessie. I have an open bug to figure out what's wrong with role::labs::mediawiki_vagrant on strech hosts [22:01:26] also, "wiki" is a bad hostname for weird reasons [22:01:34] https://phabricator.wikimedia.org/T180377 [22:01:36] bd808: should I put a guard on that class so it errors out on !jessie? [22:01:53] bd808: OK, I'll try that. [22:01:57] andrewbogott: that might be a good idea actually. [22:02:08] bd808: wikipedia any better? [22:02:51] bmansurov: something like rec-wiki would be better. Out hostname space is actually flat. the project names are not taken into account [22:03:08] its a weird thing that we will fix someday™ [22:03:14] oh I see [22:03:28] it was actually surprised that you got wiki for a new host [22:03:42] Reedy, is there a manual of the commands so I'm not just hunting for answers? [22:03:53] I also tried wikipedia and that one worked too 😄 [22:04:10] bmansurov: I guess it's ok then :) [22:04:12] you are the winner [22:04:23] tomthirteen: For what? How to write SQL queries? [22:04:24] I wanna win :( [22:04:42] tomthirteen: you mean like https://www.w3schools.com/sql/ ? [22:05:04] bd808: on another note, is there a way to change the project name? I want to host not only recommendation related stuff in that project. [22:05:30] bd808: https://gerrit.wikimedia.org/r/#/c/405203/ [22:05:57] No, meaning I can figure out just how the status are applied. I can write the sql. If an administer puts a certain status on a page, what would this be in the restrictions page [22:06:32] -- The protection type (edit, move, etc) [22:06:32] pr_type varbinary(60) NOT NULL, [22:06:32] -- The protection level (Sysop, autoconfirmed, etc) [22:06:32] pr_level varbinary(60) NOT NULL, [22:08:07] Yes, I know those. But, as I have said, I think there is a status that also an IP address to edit a page, but it needs to be approved by an administrator. I can't seem to find this. [22:08:24] !log git Deployed https://gerrit.wikimedia.org/r/405201 to gerrit-mysql [22:08:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [22:10:04] !log git Deployed https://gerrit.wikimedia.org/r/404879 to gerrit-mysql for T185148 [22:10:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [22:10:07] T185148: Update docs, monitoring, etc. for new labs servers - https://phabricator.wikimedia.org/T185148 [22:10:32] Okay im done spamming yall :) [22:11:35] tomthirteen: Any sort of "approving" means Flagged revs [22:12:29] tomthirteen: Which wiki are you trying to get this information from? [22:12:48] quite a few [22:13:03] Well, give us an example [22:13:06] Not every wiki has flagged revs [22:13:14] So it's not the case that this is a problem on every wiki [22:14:55] bd808 andrewbogott i guess it is this error [22:14:56] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: No matching entry for selector parameter with value 'stretch' at /etc/puppet/modules/lxc/manifests/init.pp:23:18 on node phabricator-stretch.phabricator.eqiad.wmflabs [22:14:57] I'm looking at a couple of hundreds of pages. I think I have been doing it wrong. I'm trying to get the status for the pages in 2016 whether or not IP addresses could edit it or not. [22:15:43] reedy: so I'm trying to hunt for a case. But, what is the status in any table that would allow flagged reviews? [22:16:06] https://github.com/wikimedia/puppet/blob/production/modules/lxc/manifests/init.pp#L23 [22:17:48] paladox: ah! [22:18:00] :) [22:18:12] that would pretty obviously blow up on stretch [22:18:14] reedy: your help is much appreciated. [22:18:18] can the jessie ones be copied to stretch? [22:18:31] though maybe not [22:18:45] i wonder if a if statement would work so that we set the default to false. [22:18:53] paladox: well... probably it needs a guard that keeps stretch from pinning to backports [22:18:59] yep [22:19:06] sine presumably stretch won't need them [22:19:18] but I should do some testing [22:19:36] there could be other things that need tweaking for stretch [22:19:50] reedy or bd808 I am searching the spanish wikipedia for page 810149 title el clasico [22:19:59] Yep [22:20:03] nothings coming up but it is locked [22:21:07] https://github.com/wikimedia/mediawiki-extensions-FlaggedRevs/blob/master/backend/schema/mysql/FlaggedRevs.sql [22:23:51] reedy: this is for me i'm assuming. thank you. but is there nothing, no values, is any table that would say definitely whether a page is open to IP edits, IP edits with approval or locked to IP edits completely? [22:24:05] Probably not... [22:24:12] As what these config actually mean can depend on [22:24:23] the MW config for flagged revs [22:24:23] https://noc.wikimedia.org/conf/highlight.php?file=flaggedrevs.php [22:25:08] reedy: ugh i need to find an example of this... [22:25:27] I've been looking at a lot of pages, so I can't remember. [22:26:07] bd808 that works :). [22:26:50] reedy: restrictions only gives the status of the page presently, not its history, yes? [22:26:52] bd808 https://gerrit.wikimedia.org/r/#/c/405208/ [22:40:52] bd808 fails on start up with [22:40:54] /usr/lib/ruby/2.3.0/fileutils.rb:253:in `mkdir': Permission denied @ dir_s_mkdir - /srv/mediawiki-vagrant/.vagrant/machines (Errno::EACCES) [22:42:46] paladox: yeah. its going to need more tweaking :) [22:43:07] :) [22:43:17] I'll try to make time to poke at it in over the next week [22:44:01] ok thanks :)