[00:14:27] bd808: Thanks a lot! T218072: Please install locale information on (kubernetes) webserver [00:14:28] T218072: Please install locale information on (kubernetes) webserver - https://phabricator.wikimedia.org/T218072 [00:14:36] It works [00:14:48] Wurgl: awesome! glad to help [12:28:22] hello world [13:31:47] !log openstack add a couple of puppet prefixes for common VMs to test puppet code [13:31:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [16:10:32] !log openstack created and deleted a bunch of testing VMs, like 'cloudvirt-stretch-test-X' and 'cloudnet-stretc-test-X' [16:10:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [16:42:52] !log admin published new debian-9.8 image (T218314) [16:42:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:42:55] T218314: Create Debian 9.8 image - https://phabricator.wikimedia.org/T218314 [17:05:29] !log admin-monitoring Updated fullstackd to use debian-9.8 [17:05:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin-monitoring/SAL [18:13:20] !log tools drained tools-worker-1028 for reboot (processes in D state) [18:13:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:04:13] !log admin bstorm started nfsd on labstore1006 (T218341) [19:04:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [19:04:16] T218341: labstore1006 - Various VMs are stuck - https://phabricator.wikimedia.org/T218341 [19:07:49] !log tools rebooted tools-sgewebgrid-lighttpd-0914 [19:07:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:08:30] !log tools rebooted tools-sgewebgrid-lighttpd-0914 (T218341) [19:08:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:08:37] !log tools rebooted tools-worker-1028 (T218341) [19:08:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:11:53] !log tools.bambots restarted webservice [19:11:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bambots/SAL [19:14:19] !log deployment-prep restart logstash on deployment-logstash2 to re-read and re-create apifeatureusage template [19:14:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [20:22:55] hey all.. I broke my vagrant again.. https://reading-web-staging.wmflabs.org/w/index.php any one able to help me avoid having to destroy it? [20:23:22] I'm seeing "No wiki found" [20:27:15] Now when I do vagrant up I get "NFS requires a host-only network to be created. Please add a host-only network to the machine (with either DHCP or a static IP) for NFS to work." [20:27:58] jdlrobson: the only "fix" I know for that bug in the LXC provider is to keep trying and/or reboot the host instance [20:28:47] "no wiki found" is usually -- https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS#Server_not_found [20:31:39] all I did was vagrant enable-role visualeditor [20:31:41] vagrant provision [20:31:53] it's terrifying that it can bring down a critical shared instance with just those 2 lines :( [20:32:10] i'm soft rebooting now [20:35:34] ok soft rebooting seemed to solve it [20:36:29] !log Tools depooled and rebooted tools-sgeexec-0908 [20:36:30] bd808: Unknown project "Tools" [20:36:36] !log tools depooled and rebooted tools-sgeexec-0908 [20:36:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:40:31] and broken again.. and soft rebooting again.. [20:44:52] !log tools Repooled tools-sgeexec-0908 after reboot [20:44:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:47:48] !log tools Depooled and rebooted tools-sgeexec-0917 [20:47:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:52:00] !log tools Hard reboot of tools-sgeexec-0917 via horizon [20:52:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:53:09] !log tools Repooled tools-sgeexec-0917 following reboot [20:53:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:54:42] !log tools Depooled and rebooted tools-sgeexec-0941.tools.eqiad.wmflabs [20:54:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:57:12] !log tools Hard reboot of tools-sgeexec-0941 via horizon [20:57:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:58:17] !log tools Repooled tools-sgeexec-0941 following reboot [20:58:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:19:56] !log tools Killed jobs still running on tools-exec-14{33,34,35,36,37,38,39,40,41,42}.tools.eqiad.wmflabs 2 weeks after being depooled (T217152) [21:19:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:20:00] T217152: Monitor and scale in the Trusty grid - https://phabricator.wikimedia.org/T217152 [21:23:00] !log tools rebooted tools-sgeexec-0919, tools-sgeexec-0934, tools-worker-1018 (T218341) [21:23:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:23:02] T218341: labstore1006 - Various VMs are stuck - https://phabricator.wikimedia.org/T218341 [21:32:41] !log tools rebooted tools-exec-1020 (T218341) [21:32:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:32:44] T218341: labstore1006 - Various VMs are stuck - https://phabricator.wikimedia.org/T218341 [21:38:51] !log tools rebooted tools-sgewebgrid-generic-0904 (T218341) [21:38:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:38:54] T218341: labstore1006 - Various VMs are stuck - https://phabricator.wikimedia.org/T218341 [21:48:27] !log tools Removed tools-exec-14{33,34,35,36,37,38,39,40,41,42} from the grid and shutdown instances via horizon (T217152) [21:48:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:48:30] T217152: Monitor and scale in the Trusty grid - https://phabricator.wikimedia.org/T217152 [21:55:08] !log tools Removed submit host flag from tools-bastion-05.tools.eqiad.wmflabs, removed floating ip, and shutdown instance via horizon (T217152) [21:55:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:55:18] T217152: Monitor and scale in the Trusty grid - https://phabricator.wikimedia.org/T217152 [22:10:52] !log tools Depooled tools-webgrid-lighttpd-142{0,1,2,5} (T217152) [22:10:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:10:56] T217152: Monitor and scale in the Trusty grid - https://phabricator.wikimedia.org/T217152 [22:20:27] !log tools Removed tools-webgrid-lighttpd-142{0,1,2,5} from the grid and shutdown instances via horizon (T217152) [22:20:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:20:40] T217152: Monitor and scale in the Trusty grid - https://phabricator.wikimedia.org/T217152 [22:30:06] !log tools Removed obsolete submit hosts from Trusty grid config [22:30:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:05:32] !log wildcat Deleted danny-b.wildcat.eqiad.wmflabs (T204703) [23:05:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wildcat/SAL [23:05:35] T204703: cloudvps: wildcat project trusty deprecation - https://phabricator.wikimedia.org/T204703 [23:06:38] !log telnet Deleted telnet2.telnet.eqiad.wmflabs (T204694) [23:06:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Telnet/SAL [23:06:40] T204694: cloudvps: telnet project trusty deprecation - https://phabricator.wikimedia.org/T204694 [23:09:09] !log queryrapi Deleted queryr-db-01 and queryr-api-01 (T204683) [23:09:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Queryrapi/SAL [23:09:11] T204683: cloudvps: queryrapi project trusty deprecation - https://phabricator.wikimedia.org/T204683 [23:11:33] !log osmit Deleted osm-serv and osmit-due (T204527) [23:11:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Osmit/SAL [23:11:36] T204527: cloudvps: osmit project trusty deprecation - https://phabricator.wikimedia.org/T204527 [23:13:53] !log math Deleted drmf, math-ru, mathoid2 (T204509) [23:13:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Math/SAL [23:13:55] T204509: cloudvps: math project trusty deprecation - https://phabricator.wikimedia.org/T204509 [23:15:32] !log getstarted Deleted vrt (T204508) [23:15:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Getstarted/SAL [23:15:34] T204508: cloudvps: getstarted project trusty deprecation - https://phabricator.wikimedia.org/T204508 [23:15:58] bd808: You are wonderful. Thank you for all your work. [23:16:15] :) thanks James_F [23:18:43] !log dumps Deleted dumps-stats and bugzilla (T204503) [23:18:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Dumps/SAL [23:18:46] T204503: cloudvps: dumps project trusty deprecation - https://phabricator.wikimedia.org/T204503 [23:20:47] !log design Deleted living-style-guide and lsg-01 (T204502) [23:20:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Design/SAL [23:20:49] T204502: cloudvps: design project trusty deprecation - https://phabricator.wikimedia.org/T204502 [23:28:40] !log tools Deleted tools-bastion-05 (T217152) [23:28:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:28:43] T217152: Monitor and scale in the Trusty grid - https://phabricator.wikimedia.org/T217152 [23:52:19] !log tools Disabled job queues and rescheduled continuous jobs away from tools-exec-14{21,22,23,24,25,26,27,28,29,30,31,32} (T217152) [23:52:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:52:23] T217152: Monitor and scale in the Trusty grid - https://phabricator.wikimedia.org/T217152