[05:32:19] help. I'm getting, like, DDoS'd? [05:34:16] https://phabricator.wikimedia.org/P15679 - just thousands of these weird requests. I've set them to 403, but... I don't have much info to go off of w/o IPs [05:38:31] the log has user agent and referer at the end of each line. the weird thing is uwsgi is misparsing some of the http protocol lines (?) - stuff like (- http://hasty.ai HTTP/1.1 403) which should just be (HTTP/1.1 403). I have no clue what the raw traffic must look like to do that [07:57:51] you can try to filtering by referrer url [08:03:25] !log admin Safe rebooting 'cloudvirt1024.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus [08:03:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:20:33] !log admin Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus [08:20:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:34:31] !log admin Safe rebooting 'cloudvirt1025.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus [08:34:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:10:28] !log admin Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus [09:10:29] !log admin Safe rebooting 'cloudvirt1026.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus [09:10:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:10:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:38:33] !log citelearn delete 'affinity' server group, is preventing us from draining an hypervisor [09:38:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Citelearn/SAL [09:39:35] !log citelearn creating 'affinity' server groups is strongly discouraged (T276963) [09:39:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Citelearn/SAL [09:39:38] T276963: Horizon: add doc links and discouragement to the 'server groups' UIs - https://phabricator.wikimedia.org/T276963 [10:04:05] !log admin Safe rebooting 'cloudvirt1026.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus [10:04:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:47:01] !log tools rebase & resolve merge conflicts in labs/private.git [10:47:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:47:10] !log toolsbeta rebase & resolve merge conflicts in labs/private.git [10:47:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:09:02] hi, re: cloudmetrics1002, I see the ceph alerts as unknown in icinga since 12 hours, known? [13:08:40] godog: got more info? [13:08:51] (link to the alert or similar) [13:11:54] cloudmetrics1002 has some mystery hardware issue [13:12:04] I see, https://alerts.wikimedia.org/?q=instance%3D~alert right? alert1001 saying that timed out when trying to get cloudmetrics1002 for ceph osd down alert [13:12:06] T275605 [13:12:08] T275605: cloudmetrics1002: mysterious issue - https://phabricator.wikimedia.org/T275605 [13:12:47] probably it's dead again :S [13:15:37] seems deat to me v.v [13:19:46] !log admin rebooting cloudmetrics1002, got stuck again (T275605) [13:19:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:19:50] T275605: cloudmetrics1002: mysterious issue - https://phabricator.wikimedia.org/T275605 [13:22:56] godog: thanks for the ping [13:24:03] they should go away in a few min [13:25:14] the warn is real though, it's ok [13:27:26] interesting, cloudvirt1001 now paged [13:27:54] I was just looking at cloudmetrics1002.. [13:27:56] *cloudmetrics [13:29:57] hmmm. rsync for graphite dead because of the reboot... maybe that's what's causing the hardwar issue to trigger? (lots of io+cpu) [13:30:53] balloons: getting my headphones one sec [13:30:54] I'm going to open a ticket to troubleshoot/replace cloudmetrics1002 [13:31:23] I was doing it this morning when it flaked again.. good reminder :-) [14:11:28] arturo dcaro: ack! thanks for looking into it [14:11:40] and balloons too ! [15:15:35] !log admin Safe rebooting 'cloudvirt1026.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus [15:15:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:19:10] !log admin Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus [15:19:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:22:49] !log admin Safe rebooting 'cloudvirt1027.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus [15:22:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:23:26] !log tools upgrading exim4-daemon-heavy in tools-mail-03 [15:23:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:28:02] !log cloudinfra created anti-affinity server group 'mx' [15:28:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [15:31:03] !log cloudinfra bump instance quota from 12 to 14 [15:31:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [15:33:27] !log cloudinfra created VMs mx-out03/mx-out04 as debian buster [15:33:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [15:44:46] !log admin Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus [15:44:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:45:57] !log admin Safe rebooting 'cloudvirt1028.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus [15:45:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:56:30] !log cloudinfra relocate floating IPs 185.15.56.18 and .19 from mx-out01/mx-out02 to mx-out03/mx-out04 [15:56:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [15:58:01] !log cloudinfra shutoff mx-out01 and mx-out02 (migrated to mx-out03/mx-out04) [15:58:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [16:05:57] !log admin Safe reboot of 'cloudvirt1028.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus [16:06:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:13:43] !log tools.replag Updated to use new *.db.svc.wikimedia.cloud naming scheme [17:13:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.replag/SAL [18:10:15] Is there any way to connect to the console of a WMCS VM (particularly for rescue mode)? [18:12:24] dancy: I'm afraid there isn't without shell access to the hypervisors themselves [18:13:11] dancy: you saw the console thing in Horizon? [18:13:42] you can see the console output there, but afaik can't interact [18:13:59] yeah, I see the console. [18:14:05] waiting for me to type commands. :-) [18:18:02] are you trying to debug why you cant ssh to it? [18:18:33] some have root access to cloud VPS that doesnt rely the LDAP part [18:18:51] that would be the next thing to try ssh root@ [18:19:06] hmm. I didn't try root@... I'll try that next time. [18:27:50] dancy: it depends if your key is in the special authorized_keys file for that in the repo [18:30:39] !log tools.wikibugs restarting wb2-irc, bot timed out from irc [18:30:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [22:57:36] !log clouddb-services manually added cnames for toolsdb, osmdb and wikilabelsdb in db.svc.wikimedia.cloud zone T278252 [22:57:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [22:57:39] T278252: Make alias for tools.db.svc.wikimedia.cloud - https://phabricator.wikimedia.org/T278252