[17:14:28] Krinkle: are you onboard with switching XHGui today? [17:15:28] yep [17:16:21] Cool. We' [17:16:55] 're going to to need to manually copy over the settings to mwdebug*, right? Or am I missing a "these hosts only" option in scap sync-file? [17:18:41] dpifke: not sure I follow. [17:19:07] re: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/619886 [17:19:31] When staging a wmf config change, we pull it down to deploy1001 in -staging, and then `scap pull` to an mwdebug host, and then scap sync-file to all hosts once verified. [17:20:17] OK. I was thinking that we'd try to avoid touching the non-debug hosts. [17:20:54] So maybe skip the (global) sync-file at the end and just `scap pull` on the debug hosts? [17:20:56] right, we will test is without syncing everywhere, but the code should (and already is) present on all app servers. XWD code is inert on other hosts. [17:21:09] no, the prod cluster should not be out of sync with git [17:21:29] that would unknowingly deploy this some days in the future with an unsuspecting deployer doing it [17:21:35] Right, it's just the configuration change. And it's low risk. I'm maybe being over-sensitive to making this change right before a three day weekend. [17:21:40] as part of a larger sync [17:21:46] oh I see. nah that's fine. [17:21:53] we're not rolling out MW code [17:22:31] OK, sounds good. [17:22:42] https://wikitech.wikimedia.org/wiki/How_to_deploy_code for MediaWiki servers [17:22:52] it's quite the list. if you haven't before, I could do this one for now. [17:22:56] opening logstash meanwhile [17:23:50] Yup, just reviewed that. We walked through it once my ~first week, I think I'm comfortable with it. [17:24:13] OK. I dont' recall it for MW, I thought that was scap/webperf stuff but okay [17:26:22] I'm happy you're around just in case I screw it up. :) [17:31:39] OK, I think I have everything up. Watching logstash, will grab that one change into git repo in /srv/mediawiki-staging on deploy1001, `scap pull` on mwdebug1001, and then if all is good, `scap sync-file` on deploy1001. [17:34:27] merge it in Gerrit then pul down on deploy1001 [17:35:12] Got it. I don't have +2 rights in mediawiki-config, so will just need your +2 for that. [17:35:45] okay [17:37:56] I usually have a screen with two tabs, where one is `touch /var/lock/scap-global-lock` and the other is the git/scap stuff [17:37:59] (on deploy1001) [17:38:51] Oh, good idea. [18:59:00] Well I'm stumped. Profiles aren't making it into the new DB. Still are being written fine to MongoDB (as expected). Nothing in the logs. Manually connecting to the DB from mwdebug1002 with the credentials works, and can successfully insert rows which are visible on xhgui1001. Don't see any differences between prod & beta as to configuration, code, or DB schema. [19:43:25] dpifke: ack, will take a peek in a bit to see if anything stands out. [19:44:26] Thanks. I'll look again with fresh eyes after lunch, maybe I'm missing something obvious. [20:22:20] dpifke: I've added [PDO::ATTR_ERRMODE => PDO::ERRMODE_WARNING] to the new PDO call on mwdebug1002 (live hack) [20:22:25] https://logstash.wikimedia.org/app/kibana#/dashboard/mwdebug1002 [20:22:30] String data, right truncated: 1406 Data too long for column 'profile' at row 1 [20:22:40] Ah ha! [20:22:42] https://www.php.net/manual/en/pdo.error-handling.php [20:22:48] silient is the default, naturally. :P [20:23:06] except for "really important" issues like connection errors, which throw by default. [20:23:30] and for the "I got this issue and have php commit access" which warn by default. [20:24:04] I've scap-pulled my patch away , but will leave it further to you [20:24:45] I did not remember this, I thought warning was the default. Oh well. [20:24:50] Very much appreciated. Looking at what's different with the schema for that column, which I thought I already checked.