[09:45:25] 10serviceops, 10Operations, 10User-Joe: SRE FY2019 Q3 goal: Ramp-up serving traffic to PHP 7 - https://phabricator.wikimedia.org/T212828 (10Joe) [09:45:48] 10serviceops, 10Operations, 10User-Joe: SRE FY2019 Q3 goal: Ramp-up serving traffic to PHP 7 - https://phabricator.wikimedia.org/T212828 (10Joe) [10:22:34] 10serviceops, 10Operations, 10Performance-Team (Radar), 10User-Elukey, 10User-jijiki: Upgrade memcached for Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10elukey) [10:22:37] 10serviceops, 10Operations, 10Performance-Team (Radar), 10User-Elukey, 10User-jijiki: Test different growth factors for memcached (prep step for upgrade to newer versions) - https://phabricator.wikimedia.org/T217020 (10elukey) 05Open→03Resolved a:03elukey [11:27:00] jijiki: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/494200 Let's start sending traffic to citoid @ kubernetes. Wanna have a look ? [12:46:14] akosiaris: sure, sorry I was without internet for a while [12:48:51] jijiki: ok. So is what the patch does clear? [12:49:40] * akosiaris added bug: line as well, should be good to go [12:49:52] kubernetes1001.eqiad.wmnet: [citoid] n [12:49:59] this is for the migration part [12:50:01] yes ? [12:50:13] we will remove it in the next patch ? [12:50:27] ah indeed lemme post all 3 patches [12:50:35] that should make it a tad clearer [12:51:05] ha, found a bug as well [12:51:06] lol [12:51:42] you have 3 [12:51:42] oh no [12:51:43] ok [13:08:25] jijiki: https://gerrit.wikimedia.org/r/#/q/topic:T213194+(status:open+OR+status:merged) [13:08:44] I got a bit overzealous and went to started cleaning up as well [13:08:58] the last patch should probably wait 1 month or so before being merged [13:09:06] the other 3 should have been merged by the end of the week [13:10:14] but those 3 should help understand the flow of changes required to shift the traffic to be served from the kubernetes cluster and clean up after the migration [13:21:10] ok so 494215 will be merged last [13:22:02] and 494214 [13:22:52] when we are sure that citoid on k8s works as it should [13:31:47] and 494214 as well [13:31:52] and 494213 [13:32:14] 494214 and 494213 (the prereq) should probably happen end of this week (assuming no gotchas) [13:32:57] 494200 right about now and then we should use confctl to add the kubernetes hosts with a small weight as pooled (the default is unpooled for this cluster) [13:34:24] ok give me one sec [14:27:48] Hi, I'm trying to set up our termbox service to be run by service-runner. Currently I've tried using both the example config and altering the config.yaml used by mathoid. I get trace like that below. I guess because my config is invalid or broken. Any hints as to where I should look? [14:28:01] TypeError: Cannot set property 'worker_id' of undefined\n at forEach (..../home/tom/src/wikimedia/mediawiki/extensions/Wikibase/view/lib/wikibase-termbox/node_modules/service-runner/lib/worker.js:157:33 [14:28:50] config.yaml looks like this: https://www.irccloud.com/pastebin/lxZqHFsI/ [14:32:23] mobrovac: care to help ^ ? [14:36:05] tarrow: ['worker_id', 'logging', 'metrics', 'num_workers'].forEach((k) => { [14:37:02] yep, I guess because this.config.services wasn't correctly built from the config file or something? [14:37:18] hmmm, that worker_id is one thing that is not in the standard config [14:38:32] but you have passed num_workers: 0 so by default worker_id should be 0 [14:41:15] tarrow: I am guessing you used https://github.com/wikimedia/service-template-node, right? [14:41:16] inserting a console.log above line 157 suggests service.conf is undefined [14:42:49] akosiaris: actually no, I tried just using service-runner following it's readme because we already have quite some code [14:43:30] I tried to follow the "usage as a binary" step [14:45:05] hm, maybe the config is missing some required stanza failing to load (and unfortunately not reporting it) [14:45:16] like worker_heartbeat_timeout: [14:45:48] if this is it, it's definitely a bug and should be fixed [14:46:56] I did try to copy pretty much everything from mathoid (which seems to call service-runner in the binary way) but then ripped it out again to try and get a minimal file. I'll try keeping more of it in this time [15:32:47] 10serviceops, 10Operations, 10Wikidata, 10Wikidata-Termbox-Hike, and 4 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10akosiaris) Per some IRC discussions we had in #wikimedia-serviceops, the code should be updated to be service-runner compatible as this wil... [15:47:30] Hm.. Interestingly if I copy the mathoid config exactly I actually get a "Error while reading config file: YAMLException: duplicated mapping key at line 81, column 8:" I guess there is some merging or something going on [15:53:42] 10serviceops, 10Operations, 10Wikidata, 10Wikidata-Termbox-Hike, and 4 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10Tarrow) I am indeed already working on it. Just so you know the current state: we are already using blubber for the CI i.e. we have 'servi... [16:00:35] 10serviceops, 10Operations, 10Wikidata, 10Wikidata-Termbox-Hike, and 4 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10akosiaris) >>! In T212189#4998182, @Tarrow wrote: > I am indeed already working on it. > > Just so you know the current state: we are alre... [16:06:03] Is there someone I should poke my service-runner questions at or would it be best to raise a ticket? I suspect I'm just doing something wrong/stupid [16:14:37] tarrow: I 'd create a ticket to at least track it. It looks like it ain't something simple. Lemme know the task number and I 'll add the correct people [16:15:30] akosiaris: awesome. I'll craft it now [17:38:21] 10serviceops, 10Operations, 10Core Platform Team (PHP7 (TEC4)), 10Core Platform Team Kanban (Done with CPT), and 4 others: Set up a beta feature offering the use of PHP7 - https://phabricator.wikimedia.org/T213934 (10CCicalese_WMF) [19:35:23] 10serviceops, 10Scap, 10Release-Engineering-Team (Kanban): Scap: server_groups regression - https://phabricator.wikimedia.org/T217597 (10thcipriani) [21:37:02] 10serviceops, 10Scap, 10Patch-For-Review, 10Release-Engineering-Team (Kanban): Scap: server_groups regression - https://phabricator.wikimedia.org/T217597 (10Reedy) [21:37:16] 10serviceops, 10Scap, 10Patch-For-Review, 10Release-Engineering-Team (Kanban): Scap: server_groups regression - https://phabricator.wikimedia.org/T217597 (10Reedy) p:05Triage→03High [21:43:28] 10serviceops, 10Scap, 10Patch-For-Review, 10Release-Engineering-Team (Kanban): Scap: server_groups regression - https://phabricator.wikimedia.org/T217597 (10thcipriani) [21:58:05] 10serviceops, 10Citoid: JSTOR is blocking citoid IPs - https://phabricator.wikimedia.org/T216456 (10Mvolz) 05Open→03Resolved [21:58:51] 10serviceops, 10Citoid: JSTOR is blocking citoid IPs - https://phabricator.wikimedia.org/T216456 (10Mvolz) They've whitelisted our IPs on request, so this is now resolved. [23:20:36] 10serviceops, 10Scap, 10Release-Engineering-Team (Kanban): Deploy scap 3.9.2-1 - https://phabricator.wikimedia.org/T217611 (10thcipriani)