[01:27:48] !log tools.csp-report Updated to 91c10d8 (Update for change in search results) and switch to elastic7 backend cluster [01:27:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.csp-report/SAL [01:35:14] !log tools.csp-report Updated to b58a6dc (Filter using keyword subfields) [01:35:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.csp-report/SAL [03:31:40] Could someone give me a hand with testing something, and delete this redirect from the move? https://wikitech.wikimedia.org/w/index.php?title=User:DSquirrelGM&redirect=no [03:39:54] (trying to check if it'll pull from meta) [16:17:53] Is there a wikitech, phab or toolforge admin here? [16:18:01] I need one of each [16:21:01] RhinosF1: I happen to be all 3. I'm in a meeting right now, but will have some time at the top of the hour [16:21:16] PM me if you would like [18:07:31] andrewbogott: bd808: I'm finding myself unable to access my server monitoring software on my VMs. The daemons are running, and the IP is still the same, but the my Mac/iPhone/iPad won't connect to them. Did anything change recently that I'm not aware of. [18:08:14] Cyberpower678: you are going to have to be much more specific. I have no idea what you are "aware of" or what you are reporting as broken [18:08:23] Cyberpower678: have you verified that you can access them locally? [18:08:37] (iow, from the host that they're running on — always the first thing to check) [18:08:56] I'm using iStat Server on all of my VMs which allow me to monitor resource usage very quickly on my devices wherever I am. [18:10:23] All three VMs won't connect now. [18:10:51] The daemon is running too. [18:11:15] And configuration looks correct, but I can't get it. [18:11:35] andrewbogott: I wouldn't know how to check locally. I'm not there. [18:11:52] do you know what port the service runs on? [18:12:03] 5109 [18:12:13] the first way to check is to connect to the host the service is running on and [18:12:23] $ telnet localhost [18:12:39] it will either connect and give you a cryptic prompt, or tell you there's nothing running there, etc. etc. [18:14:18] # telnet localhost 5109 [18:14:18] Trying ::1... [18:14:18] Trying 127.0.0.1... [18:14:20] Connected to localhost. [18:15:03] ok, the next thing is to check if you can connect like that between the different hosts (to make sure there aren't local firewall things happening) [18:15:50] and then do the same check from your local machine [18:16:02] (this is trying to figure out if/where the network interruption is) [18:16:29] you mean do telnet 185.15.56.22 5109 from the machine? [18:16:56] yeah [18:17:10] well, telnet 5109 [18:18:43] Yea, telnet's not getting anywhere with the public IP. Even on it's own VM [18:20:07] andrewbogott: ^ [18:20:18] 185.15.56.22 is the public ip? [18:20:22] Yes [18:20:42] # wget http://ipecho.net/plain -O - -q ; echo [18:20:42] 185.15.56.22 [18:20:46] huh, seems to work for me [18:20:50] https://www.irccloud.com/pastebin/bCWlF9oj/ [18:21:04] if wget is failing maybe the server itself is stuck [18:21:05] Not for me. [18:21:18] telnet works for me as well [18:21:47] that wget is just a call to a web service to find the public ip [18:21:52] # telnet 185.15.56.22 5109 [18:21:52] Trying 185.15.56.22... [18:21:52] telnet: Unable to connect to remote host: Connection timed out [18:22:06] And that's on the VM itself [18:22:24] on the vm itself it won't work [18:22:31] Oh? [18:22:35] the routing in Cloud VPS does not work like that [18:22:46] Ugh it's one of those. [18:22:47] yeah, the public IP is probably not active locally, you'd need to use the local IP there [18:22:55] (sorry, that's definitely not obvious) [18:23:10] public IPs are only on the Neutron gateway instances [18:23:31] it is all software defined networking magic to route that traffic back to a given Cloud VPS instance [18:23:39] and that routing only works from the outside in [18:24:06] But, both andrewbogott and I can connect from outside [18:24:21] so that seems like confirmation that the routing is in place [18:24:41] Okay Telnet connects. [18:24:46] yeah, so probably not a networking issue. I don't know where that leaves us outside of 'try restarting the server and looking at its logfiles' [18:25:05] From my PC, but none of my VMs actually connect on my monitoring apps. [18:25:43] They all went down at the same time. [18:25:50] Weird. [18:25:56] Cyberpower678: I don't think we can help you debug your 3rd party service integrations easily [18:26:08] I guess. [18:27:39] Okay, I restarted the daemons, and it came back. Weird that all three went down at the same time. [18:27:51] easy fix! [18:28:22] andrewbogott: but given how they coincidentally went down, one would hypothesize network issue first. :p [18:30:37] Cyberpower678: but one would also test that hypothesis [18:30:45] yep, although 'try restarting it' is always a good first step [18:31:04] andrewbogott: I would have it was just the one VM. :p [18:32:00] I mean how likely does remote monitoring software fail at the same time on 3 separate VMs when they share the same network? :p [18:37:32] DSquirrelGM: I see you asking for help, but I don't understand what help you are asking for. If you were trying to test if Global User Pages works on Wikitech, the answer is no. Wikitech has a separate user database from any of the main project wikis. This means that there is no way for GUP to know that your account on Wikitech is in any way related to an account on meta. [18:39:14] hmm ok then, don't worry about it - I'll move my user page back and then just have you delete the /test subpage [18:43:18] ok moved it back, ready to delete the subpage [18:44:27] DSquirrelGM: tag it with {{Delete}} and somebody will get to it [19:02:14] somewhat related - have you considered adding some links to some of the main maintenance categories to the recent changes page, similar to the mediawiki site? [19:02:47] I find features like that to be rather useful [19:04:41] DSquirrelGM: I think what you are really asking is if there is an active editor community on wikitech that is interested in UI customizations. I think the answer to that is mostly no. 90% or more of edits on wikitech are made by Foundation staff documenting their projects. It is much more like an intranet wiki than a collaborative content project today. [20:02:00] !log openstack add new ceph host aggregate T225320 [20:02:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [20:02:02] T225320: Ceph Proof of Concept Build and Testing - https://phabricator.wikimedia.org/T225320 [20:03:31] !log openstack add cloudvirt-wdqs100[123] to ceph host aggregate T221631 [20:03:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [20:03:33] T221631: Dedicated servers on WMCS to test WDQS scalability strategy - https://phabricator.wikimedia.org/T221631 [20:19:25] As one of the RC triagers, I'd also value at least one of those links! (I currently store the link to the deletion queue on my userpage, which isn't very efficient.) I've started a proposal at https://wikitech.wikimedia.org/wiki/MediaWiki_talk:Recentchangestext (ping@ DSquirrelGM ) [20:22:10] glad I'm not the only one that finds them useful [20:25:40] can't really think of anything else to add to what's there already [20:31:13] thinking maybe put a mockup on a 'talk:*/proposed' subpage to work on details [20:31:46] quiddity: JFDI. and if you need more rights to JFDI then poke me and I'll give them to you [20:34:10] DSquirrelGM, Just edit what's there if you think of anything, no need to complicate it further! bd808, Only full interface admins can edit MediaWiki namespace pages, but I'll ping you onwiki when it's ready to go. :) (for now I need lunch...) [20:37:16] hello! I'm guessing running the `maintain-meta_p` script is quick and perhaps low-risk? I was hoping we could do this soon for https://phabricator.wikimedia.org/T246056 [20:39:20] the discrepancy is causing some of my tools to break. I would just go off of the sitematrix API, but not all of those wikis are replicated. So I guess I need to cross-reference the sitematrix with meta_p.wiki in order to accurately know which dbs are present and queryable? or is there a better way? [20:56:31] if you happen to know what {{int:*}} to use for the label, go ahead and add/edit it, but figured out how to get the link itself to work [20:56:46] (quiddity) [21:03:21] musikanimal: I can give it a shot. The only thing that can make it difficult is row lock contention if I am remembering correctly. [21:05:33] !log admin Running `sudo maintain-meta_p --all-databases --purge` on labsdb1002 (T246056) [21:05:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:05:36] T246056: Remove any references to fixcopyrightwiki from the meta-index in Wikimedia Cloud Services - https://phabricator.wikimedia.org/T246056 [21:08:50] !log admin Running `sudo maintain-meta_p --all-databases --purge` on labsdb1011 (T246056) [21:08:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:08:59] doh. typo in that first one [21:09:09] labsdb1012, not 02 [21:09:27] * bd808 actually hates the meta_p db [21:10:22] I totally get why it is useful, but keeping it in sync seems to be tricky. Maybe we should just cron this to run every day? [21:10:56] !log admin Running `sudo maintain-meta_p --all-databases --purge` on labsdb1010 (T246056) [21:11:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:11:01] T246056: Remove any references to fixcopyrightwiki from the meta-index in Wikimedia Cloud Services - https://phabricator.wikimedia.org/T246056 [21:12:40] bd808, speaking as someone who struggled to get people to run it for ages I would love it to be run automatically every day [21:13:28] I'll start a task to give the DBAs a chance to tell me why its the worst idea ever :) [21:13:31] I've probably proposed that before and people were too scared [21:13:38] bd808: Thank you for running that, BTW. [21:14:38] !log admin Running `sudo maintain-meta_p --all-databases --purge` on labsdb1009 (T246056) [21:14:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:15:16] yes a cron for this would be fantastic [21:16:23] {{done}} with the updates [21:17:53] thanks! [21:18:01] though now I'm getting an error that ngwikimedia_p doesn't exist [21:19:04] Ha, that new one probably didn't get created yet. [21:19:25] musikanimal: because it that's true. T240772 jsut showed up on our workboard today [21:19:26] T240772: Prepare and check storage layer for ngwikimedia - https://phabricator.wikimedia.org/T240772 [21:19:41] this is the problem with meta_p honestly [21:19:47] yeah [21:20:03] it reflects the state of the slice data, not the state of the replicas [21:22:11] I can't manually fix the database either. :/ I only have sudo rights to run the maintenance script and not to directly interact with the mysql tables [21:22:31] in my application, I guess I have to do something like `USE whatever_p` and capture the "access denied" error to determine whether it not it can be queried [21:23:01] since `USE doesnotexist_p` gives `ERROR 1044: Access denied ...` [21:23:32] meta_p is a poor substitute for foreachwiki :) [21:24:27] eh, I need to run an actual query too! `ngwikimedia_p` does exist, but with no tables [21:24:35] yeah [21:24:42] no views [21:24:44] it's not exposed yet I presume [21:24:53] the tables are there too, but you never see them [21:25:03] maintain_views not yet run? [21:25:19] T240772 [21:25:20] T240772: Prepare and check storage layer for ngwikimedia - https://phabricator.wikimedia.org/T240772 [21:33:57] the user creation log was the only major one I thought to add so far - in case we get a repeat of a recent incident. [21:37:27] !log admin Running wmcs-wikireplica-dns to add service names for ngwikimedia.*.db.svc.eqiad.wmflabs (T240772) [21:37:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:37:30] T240772: Prepare and check storage layer for ngwikimedia - https://phabricator.wikimedia.org/T240772 [21:45:12] musikanimal: ngwikimedia_p exists now [21:46:46] yep, everything is in working order now :) Thank you! [21:54:28] !log tools.zppixbot - placing code for T246922 in sftp://login.tools.wmflabs.org/mnt/nfs/labstore-secondary-tools-project/zppixbot [21:54:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [21:58:35] looks good :) - https://wikitech.wikimedia.org/wiki/Special:RecentChanges [22:07:38] !log tools.zppixbot deploying hacked test in ##rhinosf1 [22:07:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [22:07:47] Zppix: ^ check code once I know it works [22:09:21] !log admin moving cloudvirt1006 into the maintenance aggregate for T246908 [22:09:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:09:23] T246908: drain cloudvirt1006 for battery replacement - https://phabricator.wikimedia.org/T246908 [22:22:38] !log admin upgrading designate on cloudservices1003/1004 to Queens [22:22:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:33:31] !log cloudinfra Shutoff cloudinfra-internal-puppetmaster01, replaced with -02 per T241719 [22:33:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [22:33:34] T241719: Migrate remaining self-hosted puppet masters to Puppet 5 / facter 3 - https://phabricator.wikimedia.org/T241719 [23:11:59] bd808: can you check why i keep getting permission denied trying to edit a file. [23:15:50] RhinosF1: I could, but maybe you could check yourself? `ls -lh` is a nice way to see permissions [23:16:10] hmm [23:16:39] ---------- 1 tools.zppixbot tools.zppixbot 1.0K Mar 4 23:09 abuse.py [23:16:51] bd808: what do I need for read/write [23:16:56] Zppix: you broke it [23:17:09] chmod 0660 abuse.py [23:17:24] that was what i was trying to do but i suck at chmod [23:17:31] or better: chmod g+rw,u+rw abuse.py [23:18:33] bd808: fyi take example doesnt do chmod 0660 like it should [23:18:58] it just does rw-r--r-- [23:19:18] Zppix: fyi, if that's a bug report I'm not phabricator [23:19:32] bd808: i dont know if its a bug [23:19:38] im not sure if its supposed too [23:20:09] Zppix: why did the databases just error [23:21:10] fixed