[06:55:06] 10serviceops, 10Operations, 10User-jijiki: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10jijiki) [07:41:27] 10serviceops, 10Operations, 10User-jijiki: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10aaron) >>! In T223310#5185185, @Addshore wrote: > For things added my wmf.3, it looks like there were indeed some chronology protector changes https://github.com/wik... [08:09:49] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, and 5 others: Introduce wikidata termbox SSR to kubernetes - https://phabricator.wikimedia.org/T220402 (10Pablo-WMDE) Hi @akosiaris, could you please provide a (curl) example of how the request is supposed to look like? Seems we w... [09:01:52] 10serviceops, 10Operations, 10User-jijiki: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10elukey) >>! In T223310#5186065, @aaron wrote: > > I would expect https://github.com/wikimedia/mediawiki/commit/1ef64b2f2683051066a33ba34c6d04723252c3c0 to have some... [09:24:59] 10serviceops, 10Operations, 10User-jijiki: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10elukey) I grabbed a pcap (~200MB) from 11:03:25 to 11:04:41 of traffic directed to the mc1033's redis port (only one direction to avoid having to deal with huge pcap... [09:40:43] 10serviceops, 10Operations, 10service-runner, 10Patch-For-Review, 10Services (later): Re-evaluate service-runner's (ab)use of statsd timing metric for nodejs GC stats - https://phabricator.wikimedia.org/T222795 (10Pchelolo) a:05holger.knust→03None [10:08:50] 10serviceops, 10CX-cxserver, 10Beta-Cluster-reproducible, 10Services (next): Cxserver container: Container does not send fatal errors to docker logs via stdout? - https://phabricator.wikimedia.org/T223395 (10akosiaris) streams: - host: deployment-logstash2.deployment-prep.eqiad.wmflabs port: 12201... [10:10:10] a little bit offtopic [10:10:17] how are we regarding capex? is it final? [10:10:34] because i was thinking about etcd, it gives me the chills thinking they are vms [10:10:57] and those vms are backed by a DRBD filesystem [10:12:33] https://github.com/helm/helm/releases/tag/v3.0.0-alpha.1 [10:12:36] \o/ [11:00:10] fsero: it's not final, but it's hard to impossible to add new items at this phase; that said, we always allocate a little bit of money for small/one-off allocations, so etcd could draw from that pool [11:08:15] great to know paravoid if we face problems then we should react, and probably recycle others [12:41:39] 10serviceops, 10CX-cxserver, 10Beta-Cluster-reproducible, 10Services (next): Cxserver container: Container does not send fatal errors to docker logs via stdout? - https://phabricator.wikimedia.org/T223395 (10Krenair) 05Open→03Invalid ... thanks :) [12:41:42] 10serviceops, 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, and 4 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) [12:45:03] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, and 5 others: Introduce wikidata termbox SSR to kubernetes - https://phabricator.wikimedia.org/T220402 (10mobrovac) @Pablo-WMDE all you have to do is have the request be of this form: ` URI: http://appservers.discovery.wmnet/{+you... [13:17:43] 10serviceops, 10CX-cxserver, 10Beta-Cluster-reproducible, 10Services (next): Cxserver container: Container does not send fatal errors to docker logs via stdout? - https://phabricator.wikimedia.org/T223395 (10mobrovac) Doing `s/module: src\/app.js/module: app.js/` should fix your issue. [13:47:50] 10serviceops, 10CX-cxserver, 10Beta-Cluster-reproducible, 10Services (next): Cxserver container: Container does not send fatal errors to docker logs via stdout? - https://phabricator.wikimedia.org/T223395 (10Krenair) I did `module: ../app.js` to solve that, was fine once I managed to get it writing error l... [14:33:48] <_joe_> is anyone in our team on top of the pdu swap? [14:37:05] per https://phabricator.wikimedia.org/T223126 doesnt seem to affect us directly right? [14:37:59] <_joe_> oh I was looking at the wrong rack it seems [14:38:26] im waiting for a review so i can read #operations [14:38:27] ahem ahem [14:38:33] <_joe_> ahahah [14:38:42] <_joe_> I did finish the first of your two reviews [14:38:49] <_joe_> the painful one [14:39:49] this does X person to aling with values question reminds me to https://en.wikipedia.org/wiki/Alignment_(Dungeons_%26_Dragons) [14:52:42] <_joe_> fsero: done :) [14:52:49] <_joe_> good job [14:52:52] thank you so mch :) [14:53:06] about the alerting.. email to ops@ is enough? [14:53:09] or something else? [14:53:29] i would prefer to downtime icinga but i guess is not feasible to do everywhere [14:58:57] https://external-preview.redd.it/OSRbAOGSOQwTbd1bSGdQQ2otfl14OF7mSrxcBFD4yEY.jpg?auto=webp&s=148f5fe355a4ca46fe34920f8db61db4c24f897b [14:59:01] oops sorry [14:59:07] https://external-preview.redd.it/OSRbAOGSOQwTbd1bSGdQQ2otfl14OF7mSrxcBFD4yEY.jpg [14:59:43] apergos: :D [15:00:30] in my time here I've managed to hit exactly the four corners of that chart :-D [15:26:25] 10serviceops, 10Operations, 10Core Platform Team Backlog (Watching / External), 10Patch-For-Review, and 2 others: Use PHP7 to run all async jobs - https://phabricator.wikimedia.org/T219148 (10jijiki) [15:28:46] <_joe_> I'll be a couple minutes late to our meeting [15:31:12] joning [15:31:27] mutante: [15:31:32] woops you're there [15:31:40] akosiaris: meet meet meet [15:37:01] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10mobrovac) >>! In T220401#5172116, @akosiaris wrote: > @Clarakosi @Eevans. I 've updated the chart to also conditional... [16:30:20] fsero: so seriously I don't look pixelated anymore? [16:30:24] yey [16:30:44] and when you open your mouth the sounds come after some ms, not s! [16:30:49] it's amazing [16:31:04] 150mbps can do that to you :p [16:31:26] lol [16:31:40] i liked the birds singing in the background during the meeting [16:32:00] hahah yeah it was a nice touch really [16:32:13] I was thinking the same thing [16:32:21] makes any message more positive, haha [16:32:40] mutante: I will go to my village in about 2 weeks, and birds chirp there as well [16:32:51] I will make sure I am outside during the meeting :p [16:33:01] and we will get to hear them in HD quality :)) [16:33:15] oh well, not that many mbps there [16:33:20] but it is 4G so [16:33:20] hah, ok [16:33:24] it should sufficient [16:35:08] :) one more thing i wanted to mention quickly: wikitech now has PHP 7.2 packages installed and also php-fpm is now running. I just did not change the Apache config yet. So it still talks to hhvm. The nice part is confirmed it's not a problem to have hhvm and php-fpm running at the same time and the actual switch is much smaller and easy to revert.. rather than doing one large change all at [16:35:14] once [16:35:49] and now getting some breakfast first.. then be back [16:43:45] <_joe_> fsero: I did +1 your cchange; do you want to merge it tomorrow morning and also do the cert refresh? [16:44:04] i want to address the volans nits [16:44:09] and merge the check if possible today [16:44:14] <_joe_> of course he had nits [16:44:39] * volans vanishes in a nit [16:44:42] cert refresh tomorrow or monday [16:44:47] at the very last [16:45:06] <_joe_> but they all make sense as usual [16:45:08] <_joe_> damn [16:45:24] i like his nits, they come also pretty quick [16:45:28] :P [16:48:02] mark: are you counting? :-P [16:50:07] <_joe_> just seven [16:50:15] <_joe_> that's under par [16:50:31] <_joe_> we should do volans CR golf [16:51:53] rotfl [16:59:52] how is that played? [16:59:55] I want in [17:00:33] <_joe_> I'm still defining the rules [17:00:39] <_joe_> but the gist is [17:01:21] <_joe_> you prepare a patch, a group of umpires (or an AI - BINGO! ) determines the expected number of nits for such a change [17:01:31] <_joe_> that's the par [17:01:47] <_joe_> if he adds less, you've made a birdie/eagle/whatever [17:01:58] <_joe_> if he adds more, then you're over par [17:02:13] you can nominate luca as par-generator until the AI is ready ;) [17:02:13] <_joe_> we can add scores during the year and make up a tournament [17:02:25] <_joe_> that seems right [17:02:31] lol [17:03:14] <_joe_> we can define the rules of the game better at the summit [17:03:39] * jijiki makes a notes [17:03:51] we may have access to actual clubs [17:04:02] as long as it does not end in a bloodbath [17:04:41] there is a golf course on the hotel premises in Dublin i heard [17:04:55] so that's the perfect location [18:18:37] 10serviceops, 10Mobile-Content-Service, 10Operations, 10Page Content Service, and 5 others: New Service Request: wikifeeds - https://phabricator.wikimedia.org/T223469 (10mobrovac)