[07:31:51] https://tools.wmflabs.org does not resolve for me] [07:31:57] All other websites load [07:32:26] "Status: Known problems" is this the problem [07:34:14] !help ^ [07:34:14] Cyberpower678: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [07:41:11] Cyberpower678: Known problems was added ~15 hours ago, when the DDoS startet and Amsterdam was not reachable [07:41:39] It seems the DDoS is over now as I have no issues loading Wikipedia from Germany [07:42:25] Wurgl: ^ [07:42:36] When you look at: https://twitter.com/UKDrillas … it may come again [07:43:16] BTW: I could ssh on sgebastian just now [07:44:36] ssh failed. [07:44:43] * Cyberpower678 fires up the VPN to bypass. [07:58:41] Cyberpower678: nah. i dont think that DNS issue is part of that problem [08:00:18] mutante: Any ideas what it could be/ [08:01:22] Cyberpower678: not really, but i would expect it to be limited to one provider [08:01:35] it works for me just fine [08:02:07] let me know if it works after you start VPN [08:02:37] My VPN won't connect [08:02:41] :-( [08:02:51] I'm pegging it's a driver issue. [08:04:38] Cyberpower678: you can edit your hosts file manually [08:04:44] tools.wmflabs.org has address 185.15.56.5 [08:04:47] and add that ^ [08:04:54] :_) [08:05:45] Cyberpower678: what happens if you use "host" or "dig" command on command line? [08:07:40] $ host tools.wmflabs.org [08:07:40] Host tools.wmflabs.org not found: 2(SERVFAIL) [08:09:21] Which nameserver do you use? [08:10:27] I have 8.8.8.8 and 8.8.4.4 (both are google) as additional servers to the one of my provider [08:11:12] 1.1.1.1 is Cloudflare [08:11:27] I'm not sure. I'm on a mobile hotspot [08:14:18] well. SERVFAIL tells us the server isn't able to answer properly but the name does exist [08:14:24] as opposed to NXDOMAIN [08:15:18] mutante: adding the entry to hosts, makes it accessible. [08:16:08] Wurgl: alright, for some reason my GDNS went missing. :/ [08:16:12] check what nameserver is in /etc/resolv.conf .. that you got via DHCP [08:16:33] like is it the IP of your local router [08:17:18] $ ssh cyberbot-exec-iabot-01.eqiad.wmflabs [08:17:18] ssh: Could not resolve hostname primary.bastion.wmflabs.org: nodename nor servname provided, or not known [08:17:19] ssh_exchange_identification: Connection closed by remote host [08:17:25] Wurgl: ^ [08:17:53] It's the IP of my phone [08:17:57] mutante: ^ [08:18:07] Cyberpower678: restart the phone ? [08:18:10] I use that: tools-login.wmflabs.org [08:18:43] ssh: Could not resolve hostname cyberbot-exec-iabot-01.eqiad.wmflabs: Name or service not known [08:18:53] Does not work for me :-/ [08:19:21] well, if the bastion fails to resolve the internal instance name then that's a different issue i guess [08:19:35] but also cloud-vps != toollabs [08:20:06] Wurgl: That's normal for you as you do not have a forward entry in you ssh config. [08:20:15] tools-login for tools and primary.bastion for VPS [08:21:03] $ ssh login-stretch.tools.wmflabs.org [08:21:03] ssh: Could not resolve hostname login-stretch.tools.wmflabs.org: nodename nor servname provided, or not known [08:21:05] https://phabricator.wikimedia.org/T164420 [08:21:44] that old ticket could use an update [08:22:26] but that is 2 different things [08:25:39] Going AFK> [08:25:50] reboot the phone first [08:56:56] is there any way to find where my script hangs? [08:56:58] 8024697 process_templatedata persondata Continuous / Running 2019-09-04 11:28:02 1h37m37s 64.5M/585.9M (peak 85.2M) [08:57:30] It does not work since 18:07 UTC yesterday? [10:15:48] Wurgl: after completely rebooting Wifi, and resetting my DNS hosts (8.8.8.8, 8.8.4.4, and 1.1.1.1) I'm able to connect to labs again. [10:16:28] I don't trust DNS hosts other than Google normally. [10:16:34] fine [10:17:04] * Cyberpower678 needs to get the script executed before the next DDoS [12:36:17] Hello, can't check the channel logs so might already be answered: Is dns acting up for tools.wmflabs.org (*.wmflabs.org) a known issue? [12:36:57] there have been reports of that [12:38:35] multichill, FWIW though I can get good DNS results by querying against cloud-ns0.wikimedia.org and cloud-ns1.wikimedia.org right now [12:38:40] against 8.8.8.8 too [12:39:36] Krenair: Yeah, dig +trace works, but just lookup gives a SERVFAIL [12:39:56] So probably my ISP's recursor has a problem reaching it [12:40:29] possibly [12:43:13] Yup, if I do a DNS lookup for en.wikipedia.org using their dns servers, it works, for wmflabs.org it gives SERVFAIL [12:44:55] fun [12:53:09] updated https://phabricator.wikimedia.org/T232250 Krenair [12:53:54] TTL is a day so I guess it will start working again in a couple of hours when that expired [14:27:30] i think wikibugs needs restarting, it dosen't seem to be reporting phab tasks. [15:03:38] https://phabricator.wikimedia.org/T232247 was created for it [15:05:36] It worked in -operations a few hours ago [15:06:16] And it just reported something to #pywikibot [15:06:46] it's reporting gerrit patches AntiComposite [15:07:25] -interactive had some phab tasks half an hour ago [15:07:43] wait no that was days ago [17:16:15] Hi, can a tool admin fix ownership in /data/project/totoazero/pywikibot by chown files and dirs to tools.totoazero user? [17:18:43] zhuyifei1999_: ^ if you're available [17:21:51] The command "take" does not work for you [17:32:27] wow just discovered this command, thanks Wurgl ! [17:49:25] (sorry, late :P) [19:25:03] tools.totoazero@tools-sgebastion-08:~$ du -sh logs [19:25:04] 42G logs [19:25:06] yup [19:32:57] wow [20:27:06] Hi, a tool of mine failes to connect to the mysql-servers. Is this a consequence of the DDOS or any other known problems? Or am I dooing something wrong? [20:28:06] !help ^ [20:28:06] lucaswerkmeister: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [20:28:30] (I assume that command does more than just the wm-bot message…) [20:36:06] Logically it triggers all irc clients that are configured to notify when this keyboard appears, like mine. If it was added in the topic, wmcs team have probably set this up too :) [20:38:30] nudin[m]: what tool? [20:38:48] "machtsinn" [20:39:32] I just set it up a few days ago and it worked, today I wanted to continue and can't connect to the db [20:39:43] I can connect with your db creds to the replicas just fine [20:40:01] You are likely connecting to toolsdb... [20:40:33] I can connect there fine as well [20:41:34] `mysql tools.db.svc.eqiad.wmflabs -us54145 -p` failes: Access denied [20:42:51] It worked with the sql shortcut. Odd [20:43:11] Oh I see, that's a syntax error in your mysql command [20:43:29] `mysql -h tools.db.svc.eqiad.wmflabs -us54145 -p` [20:43:33] Try that way ^^] [20:44:07] So ruling out the DB itself...what's wrong with your connection from the app I wonder (because I see the logs) [20:44:07] ah, that works… [20:45:21] !log tools.totoazero Deployed til ea19353, file restructuring so that the scripts used are in a clean repo cloned from github [20:47:17] pm'd you quick about something there [20:49:25] stashbot down? it is not present in this chan nor -operations [20:50:15] Could have died in the DDoS and not come back up? [20:50:20] Framawiki: I can log it for you if you want [20:53:12] nudin[m]: So, I'm coming up blank on your tool so far, but it seems like something in the code since we can get to it. If you want to make a phabricator task to dig around in case it is not a code issue, and it is a worker node issue that might not be bad. I'd just take a quick look around the code first just in case. [20:54:20] bstorm_: want me to try with my creds? [20:54:36] Zppix: it works with the tools creds [20:54:40] oh [20:54:51] But it is having a db connection failure on the running tool [20:54:55] So it could be a few things [20:55:16] Yeah [20:55:34] hopefully its just some jumper cables are needed [22:28:05] !log tools.pbbot Install wiki-java fork @ d1def2e [22:30:02] stashbot is still down ig