[02:07:30] !log tools reloaded ferm on tools-flannel-etcd-0[1-3] to get the k8s node moves to register [02:07:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [03:24:20] hi folks...any Wikimedia folks going to the Google Cloud Next thingy at Moscone in SF this week? [03:24:28] I'm actually not planning on going, but some of my colleagues at Internet Archive are, so I wanted to make sure y'all had a chance to say "hi" to each other. [03:48:03] Hello robla! I don't think any of the cloud people are in CA at the moment. I'd be interested if I were around. [03:48:19] !log tools moving tools-worker-1008 and tools-worker-1009 to eqiad1-r [03:48:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:02:05] andrewbogott: bummer there aren't WMF Cloud folks in town (and btw: long time, no see!). Like I said, I'm not planning to go, but this seemed like a good opportunity to make some virtual introductions [04:04:54] we're in the process of piloting a few k8s-managed services [07:35:08] * legoktm waves to robla [10:27:32] A couple of times in the past month I've received a MemoryError on a regular jsub task - is this an indication that the script I'm running is using too much memory, or is it dependent on what else is running at that time? [10:30:10] I would have just assumed it was an issue with my script, but I've had this two times with two entirely different processes now, both of which run daily with no problem [13:47:58] Samwalton9: just allocate more memory [13:49:36] grid is nuts https://phabricator.wikimedia.org/T219351#5060921 [14:07:35] !log tools moving tools-worker-1010, tools-worker-1011, tools-worker-1001 to eqiad1-r [14:07:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:56:56] !log tools cleared 4 queues on gridengine of E status (ldap again) [14:56:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:02:22] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @amir1 & @subbu - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:25:41] Why did eqiad1-r get created? [15:26:17] Zppix: ? [15:26:38] not sure I understand the question [15:26:39] arturo: why did we start using eqiad1-r what was wrong with the other network? [15:26:56] Zppix: the old deployment was using nova-network [15:27:11] Ah i forgot we moved from that [15:27:23] and eqiad1-r is using neutron as the software defined networking mechanism [15:27:49] nova-network is deprecated upstream, and openstack mitaka (the current version in use in cloudvps) is the last release supporting nova-network [15:28:00] so we were forced to migrate [15:28:25] Seems like being forced to migrate is becoming a recent trend :D [15:31:17] reactive vs proactive maintenance [15:32:08] :P [15:35:17] that is related to the amount of resources available I guess [15:40:43] arturo: more than likely [15:51:14] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @amir1 & @subbu - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:52:19] bd808, do you know why wm-bot is announcing the wrong date? This is tomorrow at 8 PT. [15:56:23] !log tools moving tools-worker-1012, tools-worker-1013 to eqiad1-r [15:56:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:46:51] Hello everyone. I got `error: cannot open .git/FETCH_HEAD: Permission denied` from `deployment-chromium01.deployment-prep.eqiad.wmflabs`. That means I am not included on the security groups for that instance? [16:47:53] Following this tutorial btw https://wikitech.wikimedia.org/wiki/Proton#Deploying_the_latest_version_of_Proton [17:05:32] !log tools migrating tools-k8s-etcd-01 to eqiad1-r [17:05:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:36:27] !log tools moving tools-worker-1014, tools-worker-1015 to eqiad1-r [17:36:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:00:11] !log tools migrating tools-flannel-etcd-01 to eqiad1-r [18:00:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:02:01] ~log tools migrating tools-k8s-etcd-02 to eqiad1-r [18:05:12] andrewbogott: I assume that ~log should be !log? [18:05:20] hm, yep [18:05:26] !log tools migrating tools-k8s-etcd-02 to eqiad1-r [18:05:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:36:49] !log tools moving tools-worker-1016, tools-worker-1017 to eqiad1-r [18:36:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:28:40] !log tools.montage-beta restarted webservice because it was hung in uninterruptible sleep and not working--causing very high load as well [19:28:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.montage-beta/SAL [19:36:58] !log tools.montage-beta T220543 stopped webservice because it runs up load on any node it runs on in its current state and needs fixing before it is run [19:37:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.montage-beta/SAL [19:37:01] T220543: montage-beta tool seems to be running up load and hanging in uninterruptible sleep - https://phabricator.wikimedia.org/T220543 [19:54:50] !log tools moving tools-flannel-etcd-02 to eqiad1-r [19:54:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:04:40] !log moving tools-k8s-etcd-03 to eqiad1-r [20:04:41] andrewbogott: Unknown project "moving" [20:04:54] !log tools moving tools-k8s-etcd-03 to eqiad1-r [20:04:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:05:03] at least no one thinks I'm a bot [20:43:19] !log tools moving tools-worker-1018, 1019, 1020, 1021 to eqiad1-r [20:43:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:44:43] !log tools moving tools-flannel-etcd-03 to eqiad1-r [20:46:21] i doin't think that !log went through andrewbogott [20:48:59] !log tools moving tools-flannel-etcd-03 to eqiad1-r [20:49:04] hm, I wonder if I killed the bot [20:49:09] seems like it should restart on its own [20:57:53] !log tools moving tools-flannel-etcd-03 to eqiad1-r [20:58:12] huh [20:58:21] stashbot, are you there? [21:12:36] stashbot, feeling better? [21:12:43] !log tools moving tools-flannel-etcd-03 to eqiad1-r [21:12:59] !log tools moving tools-flannel-etcd-03 to eqiad1-r [21:13:06] stashbot seems sullen [21:14:42] !log tools moving tools-flannel-etcd-03 to eqiad1-r [21:16:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:16:41] welp [21:39:24] beep [22:04:20] !log tools added the new region on port 80 to the elasticsearch security group for stashbot [22:04:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:04:26] :) [22:05:32] It's not working quite right yet with elasticsearch because of an exponential backoff, of course [23:26:20] !log tools.stashbot Test [23:26:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [23:32:18] !log tools.taim-bot Updated crontab to correct day of week (Wed==3) and hour (German summer time) [23:32:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.taim-bot/SAL