[00:36:24] Hello [02:01:35] anyone familiar with innodb error recovery? [02:02:12] the server aborts startup at the the transaction rollback phase [02:02:44] I don't care about rolling them back but the documentation does not mention any way of not doing that [02:03:08] the table is huge, dumping and reloading it would be problematic [02:13:55] alternatively, is it possible to increase the disk size of a VM (attach a new virtual disk)? [02:26:21] tgr, I don't think it's currently possible to attach extra storage to an instance [05:42:45] Is there a good channel to discuss bots and the Mediawiki API? [05:45:06] I tried #wikipedia but it was a bit dead/lacking [06:30:43] Maybe #wikimedia-tech [14:58:56] !log toolsbeta moving toolsbeta-grid-master and toolsbeta-puppetmaster-02 to labvirt1003 [14:58:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:00:44] !log paws deleting the long-shut-down paws-proxy-01 [15:00:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [15:01:16] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @Tonina_WMDE & @tgr - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:03:59] !log tools moving tools-exec-1413 and tools-exec-1442 [15:04:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:50:54] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @Tonina_WMDE & @tgr - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:53:27] !log tools moving tools-worker-1017, tools-worker-1027, tools-worker-1028 [15:53:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:56:23] !log tools moving tools-paws-worker-1003 [16:56:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:58:05] hello friends! when you have a moment, could a Toolforge admin please review T216268? I worry there may be more issues with my Ruby tools on Stretch, and I'd like ample time to make sure everything is working before the full switchover next month [16:58:06] T216268: Stretch grid problem: libssl.so.1.0.0 missing - https://phabricator.wikimedia.org/T216268 [17:22:56] musikanimal: I'll check it out. We have had a few other missing libs like that one [17:25:09] thanks! [18:13:25] !log tools.wikibugs Updated channels.yaml to: 43339cbca935d094fa81c5cb641805b00192e13f Remove Quarry from #wikimedia-research [18:13:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [18:54:46] hello [18:58:34] * bd808 is nervous that wm-bot may be becoming self-aware [19:02:08] I, for one, welcome our wm-bot overlord [19:17:08] !log moving tools-bastion-02 to labvirt1004 [19:17:09] andrewbogott: Unknown project "moving" [19:17:20] !log tools moving tools-bastion-02 to labvirt1004 [19:17:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:52:35] Is it possible to prevent puppet from modifying a specific file, so that tests can be performed without disabling it entirely? [20:15:18] !log quarry disabling puppet temporary on -web-01 to test csp conf T214637 [20:15:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [20:15:24] T214637: Setup CSP http header - https://phabricator.wikimedia.org/T214637 [20:21:18] framawiki: Not that I know of, no. [20:38:18] !log quarry re-activating puppet on -web-01, csp conf looks good T214637 [20:38:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [20:38:25] T214637: Setup CSP http header - https://phabricator.wikimedia.org/T214637 [20:59:12] !log quarry framawiki: Deployed 8f72587 on -web-01 T216581 [20:59:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [21:21:02] !log tools.meetbot Migrated to Stretch job grid [21:21:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.meetbot/SAL [21:34:48] !log tools moving the tools-static IP from tools-static-13 to tools-static-12 [21:34:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:57:07] !log tools moving tools-static-13 to a new virt host [21:57:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:58:22] !log admin silencing shinken and disabling puppet on shinken-02 for now [21:58:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:34:31] !log tools.totoazero disabled forum_nouveaux cronjob, broken, missing data in sql tables? [22:34:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.totoazero/SAL [23:06:20] valhallasw`cloud or legoktm hi, around? :) [23:17:29] !log tools begin build new tools-webservice package T178601 T193646 T215683 [23:17:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:17:35] T178601: Add charset=utf-8 by default to lighttpd - https://phabricator.wikimedia.org/T178601 [23:17:35] T193646: Can't access /mnt from kubernetes nodejs shell - https://phabricator.wikimedia.org/T193646 [23:17:36] T215683: Debian Stretch lighttpd does not allow overriding existing mimetype.assign values - https://phabricator.wikimedia.org/T215683 [23:17:50] legoktm valhallasw`cloud for this https://gerrit.wikimedia.org/r/#/c/labs/tools/wikibugs2/+/491868/ (please review :)) [23:19:21] !log tools this was built for stretch. hopefully it works for all distros [23:19:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:23:34] zhuyifei1999_: yeah, it will [23:24:00] :) [23:25:17] !log tools upgraded toollabs-webservice on tools-bastion-02 to 0.44 (newly-built version) [23:25:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:28:17] bd808: could you update https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Docker_Images with docker-builder-06 instead of 05? I don't have 2fa with me right now so can't login to wikitech [23:28:32] (/me don't have 2fa 90% of time) [23:29:22] heh. sure. Looks liek we name -05, -06, and -03 in that page in various places [23:30:52] !log tools begin rebuilding all docker images T178601 T193646 T215683 [23:30:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:30:57] T178601: Add charset=utf-8 by default to lighttpd - https://phabricator.wikimedia.org/T178601 [23:30:58] T193646: Can't access /mnt from kubernetes nodejs shell - https://phabricator.wikimedia.org/T193646 [23:30:58] T215683: Debian Stretch lighttpd does not allow overriding existing mimetype.assign values - https://phabricator.wikimedia.org/T215683 [23:37:39] https://www.irccloud.com/pastebin/rKmQNqtb/ [23:37:41] :( [23:38:35] zhuyifei1999_: hmmm... is that a base image that is Stretch instead of Jessie? [23:38:42] * zhuyifei1999_ looks [23:39:16] I think this is jessie [23:39:28] that's really weird... [23:41:14] looks like some of our Dockerfiles mkdir that path first and other do not [23:41:49] yeah [23:41:52] zhuyifei1999_: I'll make a patch [23:41:57] ok thanks [23:42:47] comparing the commands, I think this is the static-web container [23:43:23] static and php5.6 were the two that didn't mkdir before chmod [23:43:32] yeah [23:43:42] https://gerrit.wikimedia.org/r/#/c/operations/docker-images/toollabs-images/+/491873/ [23:45:03] ok, re-building [23:57:19] done. let's see if everything works [23:58:12] oh no [23:58:21] pykube.exceptions.HTTPError: /mnt/nfs/ is not in allowed host paths nor allowed host path prefixes [23:59:00] blerg. something we have to change on the Kuberntes config side too? [23:59:20] I remember doing this. let me check