[00:18:41] PROBLEM - Puppet run on tools-worker-1029 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [00:21:48] PROBLEM - Puppet run on tools-worker-1028 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [00:50:55] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:30:52] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [03:05:32] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 13Patch-For-Review: k8s webservice restart failure with `ValueError: get() more than one object; use filter` - https://phabricator.wikimedia.org/T156626#3073637 (10Legoktm) ``` tools.checker@tools-bastion-02:~/www/python/src$ webservice restart Restarting webservice... [04:12:14] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 13Patch-For-Review: k8s webservice restart failure with `ValueError: get() more than one object; use filter` - https://phabricator.wikimedia.org/T156626#3073644 (10zhuyifei1999) @Legoktm What's the output of `kubectl get pods`, `kubectl get services`, `kubectl get d... [04:13:13] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 13Patch-For-Review: k8s webservice restart failure with `ValueError: get() more than one object; use filter` - https://phabricator.wikimedia.org/T156626#3073645 (10Legoktm) Sorry, I needed to get the tool back up. I'll remember to check those for next time. [04:24:42] legoktm: I think I can reproduce it, hang on [04:24:49] :D [04:24:53] but I need to go to dinner :P [04:25:48] k [04:37:54] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 13Patch-For-Review: k8s webservice restart failure with `ValueError: get() more than one object; use filter` - https://phabricator.wikimedia.org/T156626#3073668 (10zhuyifei1999) Reproduced: {P5016} [04:43:37] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 13Patch-For-Review: k8s webservice restart failure with `ValueError: get() more than one object; use filter` - https://phabricator.wikimedia.org/T156626#3073670 (10zhuyifei1999) Actually, no. This traceback is the original traceback. [12:19:16] 06Labs, 10Labs-Infrastructure, 10DBA: Data integrity issue with enwiki_p user_groups on Wikimedia Tool Labs (missing rows) - https://phabricator.wikimedia.org/T159493#3073791 (10jcrespo) >>! In T159493#3071970, @MZMcBride wrote: > Related question: in scripts, I connect to `enwiki_p` for the database name an... [12:31:14] 06Labs, 10DBA: labsdb1004 MySQL crash - https://phabricator.wikimedia.org/T159572#3073825 (10jcrespo) @Marostegui That log by itself should not make the server crash. s51412__data having wrong permissions is something we did on purpose during the failover, so I would expect some of those happening in the past. [12:38:43] 06Labs, 10DBA: labsdb1004 MySQL crash - https://phabricator.wikimedia.org/T159572#3073826 (10jcrespo) I have restarted the slave ignoring a couple of extra heavy hitters, that should avoid crashes for now. I have also enabled GTID to mitigate problems on crash. [12:59:12] 06Labs, 10DBA: labsdb1004 MySQL crash - https://phabricator.wikimedia.org/T159572#3073827 (10jcrespo) BTW, labsdb1005 run out of /tmp space during the weekend- we need to move it to /srv, somewhere. Both cases are probably the same origin, too much load from 1 or several users. [14:51:53] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:53:18] 06Labs, 10DBA: labsdb1004 MySQL crash - https://phabricator.wikimedia.org/T159572#3073901 (10Marostegui) >>! In T159572#3073825, @jcrespo wrote: > @Marostegui That log by itself should not make the server crash. s51412__data having wrong permissions is something we did on purpose during the failover, so I woul... [15:31:52] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [16:05:31] Hi. One of the Admins in my team wiped and re-created an instance for the math project and now for some reason I can't run php. The instance is drmf-beta.eqiad.wmflabs does anyone have idea what might be going on? [16:06:00] hcohl@drmf-beta:~$ php -bash: php: command not found [16:07:49] it's probably not installed by default, did you try installing it? [16:08:04] how can I install it> [16:08:09] apt-get? [16:08:19] apt-get php? [16:08:37] sudo apt-get php? [16:09:20] We have an entire MediaWiki instance there. It's hard to imagine that php was not installed. Isn't it required to install MediaWiki? [16:09:36] http://drmf-beta.wmflabs.org/wiki/DRMF [16:11:16] are you sure drmf-beta.wmflabs.org points to drmf-beta.math.eqiad.wmflabs ? [16:14:33] http://drmf-beta.wmflabs.org/w/index.php?title=Main_Page&curid=1&diff=3&oldid=1 shows that this is actually a vagrant install [16:14:49] so you probably have a VM running inside drmf-beta that actually has HHVM/PHP. installed [16:17:56] you probably need to find the vagrant directory (maybe in /srv ?) and type 'vagrant ssh' [16:18:40] Howie [16:21:25] There is mediawiki-vagrant directory there. [16:22:23] Howie, well, try going there and running vagrant ssh? [16:23:03] ok that worked [16:23:40] and is php installed in there? [16:24:16] yes [16:25:32] But I want to run php in my home direcory, not in /srv/mediawiki-vagrant [16:27:15] okay, but vagrant ssh has presumably given you a shell prompt inside the VM? [16:28:52] yes [16:29:09] so you're not in /srv/mediawiki-vagrant, you're somewhere in the VM [16:29:31] yes i'm in /home/vagrant [16:30:10] test [16:30:17] The directory /home/hcohl does not exist [16:30:29] within ssh vagrant [16:30:39] yeah you're not going to have a user account within the vagrant VM [16:30:51] why? [16:31:07] I want to scp some xml dumps and import them. [16:32:09] google 'vagrant scp' [16:32:45] you could probably also use the default vagrant keypair I guess... [16:35:23] Ok. I think I'm out of my element now. I will wait on our Admin to explain how we should proceed. Thanks so much for helping me up to this point Krenair. [16:35:31] np [20:24:28] hi it seems that chrome and other browsers consider http connections as insecure... is there a way to set up https web proxies for labs instances? [20:24:58] yes [20:25:28] e.g. https://drmf.wmflabs.org/wiki/Main_Page [20:26:46] ah that's great, it just works out of the box [20:26:59] who ever did that, did an excellent job [20:28:00] the application below it has to not screw things up [20:28:41] e.g. drmf-beta is misconfigured to redirect some things from https to non-https [20:29:25] ^^was [20:32:59] yeah so the proxy just has a *.wmflabs.org cert [20:34:10] is that a bad thing? [20:35:18] no [20:35:32] that's just how it works [20:36:11] ok great thank you [23:46:25] anyone familiar with node.js? [23:47:06] is there a way to make an app think the user is visiting tools.wmflabs.org/ and not tools.wmflabs.org// ? [23:47:22] express does not seem to have any non-moronic way of handling base URLs