[02:48:57] 10Cloud-VPS, 10cloud-services-team (Kanban): Create banner image for Wikimedia Cloud VPS - https://phabricator.wikimedia.org/T177442#3668142 (10MusikAnimal) >>! In T177442#3666111, @Quiddity wrote: > SVG and PNG: > > {F10039979} > > {F10039981} I like it! Is it possible to center the text, like it was done... [06:21:29] 10Data-Services, 10cloud-services-team, 10DBA, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3668279 (10Marostegui) I gave 30 to s51187 on .web. Let's see how that goes... [07:50:48] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3668422 (10jcrespo) Quarry switched to the new servers on Tue, Sep 26 (less than 2 weeks ago), so if that has been happening "in last months", it is not that. The only change I can see, which applies to both s... [07:58:18] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3668444 (10jcrespo) Actually, running your first query, takes: ``` 8 rows in set (7.85 sec) ``` on the new analytics server, while I it took >200 seconds on labsdb1001: ``` 8 rows in set (3 min 52.05 sec)... [08:20:02] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3668482 (10jcrespo) I had to kill the second query after 1000 seconds: ``` | Query | 983 | Queried about 131240000 rows | select distinct(con... [08:45:33] 10Cloud-VPS, 10Graphite: grafana-labs often fails to generate graphs with c.datapoints is undefined - https://phabricator.wikimedia.org/T177747#3668609 (10hashar) [08:52:25] 10Cloud-VPS, 10Continuous-Integration-Infrastructure (shipyard): integration-slave-docker-1001 deadlocked since Friday Oct 6th ~ 19:00 utc - https://phabricator.wikimedia.org/T177749#3668644 (10hashar) [09:17:31] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3668723 (10IKhitron) Hi, @jcrespo. All that you told me, I don't think you should it to me. I just use quarry, don't choose between analytics and something, I don't know what they are. You don't need to compar... [09:21:18] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3668749 (10jcrespo) @IKhitron It is ok, quarry maintainers will know what to do with the information I have written :-) [09:28:42] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3668761 (10IKhitron) Sure, @jcrespo, but you talked to me ("your query"), so I answered. [09:44:40] 10cloud-services-team, 10Operations, 10Puppet, 10User-Joe: Upgrade to puppet 4 (4.8 or newer) - https://phabricator.wikimedia.org/T177254#3668801 (10Joe) [09:45:28] 10Cloud-VPS, 10Continuous-Integration-Infrastructure (shipyard): integration-slave-docker-1001 deadlocked since Friday Oct 6th ~ 19:00 utc - https://phabricator.wikimedia.org/T177749#3668803 (10hashar) 05Open>03Resolved a:03hashar Hard rebooted, force ran puppet. The Jenkins channel to communicate with... [09:47:25] 10cloud-services-team, 10Operations, 10Puppet, 10User-Joe: Upgrade to puppet 4 (4.8 or newer) - https://phabricator.wikimedia.org/T177254#3668818 (10Joe) [10:07:10] 10cloud-services-team, 10Operations, 10Puppet, 10User-Joe: Upgrade to puppet 4 (4.8 or newer) - https://phabricator.wikimedia.org/T177254#3668864 (10Joe) [12:36:36] 10Cloud-VPS, 10Graphite: grafana-labs often fails to generate graphs with c.datapoints is undefined - https://phabricator.wikimedia.org/T177747#3669196 (10Gilles) It's Graphite intermittently serving junk to Grafana. It's a PNG image instead of JSON being sent back. I've seen that before, but I can't remember... [12:45:34] 10Cloud-VPS, 10Graphite: grafana-labs often fails to generate graphs with c.datapoints is undefined - https://phabricator.wikimedia.org/T177747#3669224 (10Gilles) Yes, we run into the same problem occasionally in production: https://phabricator.wikimedia.org/T153169#3042185, but it's really painful on Labs Gra... [13:39:07] (03PS1) 10Andrew Bogott: Move wmcs-related notices to #wikimedia-cloud-feed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/383126 (https://phabricator.wikimedia.org/T177427) [16:46:23] we are going to have some lag on labsdb-enwiki; we are working on it [17:26:50] (03CR) 10Quiddity: [C: 031] Move wmcs-related notices to #wikimedia-cloud-feed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/383126 (https://phabricator.wikimedia.org/T177427) (owner: 10Andrew Bogott) [17:33:03] 10Cloud-Services: Organize additional projects as #Cloud subprojects - https://phabricator.wikimedia.org/T177787#3670088 (10Quiddity) [17:40:54] 10Cloud-Services, 10cloud-services-team (Kanban), 10Patch-For-Review: Remove non-interactive bots from #wikimedia-cloud - https://phabricator.wikimedia.org/T177427#3670103 (10Quiddity) >>! In T177427#3661640, @chasemp wrote: > I don't mind the idea of `unbreak now` still showing up via wikibugs. [in the main... [17:57:18] 10Cloud-Services, 10cloud-services-team (Kanban), 10Patch-For-Review: Remove non-interactive bots from #wikimedia-cloud - https://phabricator.wikimedia.org/T177427#3670111 (10Andrew) >>! In T177427#3670103, @Quiddity wrote: >>>! In T177427#3661640, @chasemp wrote: >> I don't mind the idea of `unbreak now` st... [19:28:46] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3670403 (10zhuyifei1999) 05Open>03Resolved Dunno if anything needs to be done here. [19:30:55] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3670406 (10IKhitron) >>! In T160188#3670403, @zhuyifei1999 wrote: > Dunno if anything needs to be done here. Do you mean it will stay work quickly like this for good? [19:34:06] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3670411 (10zhuyifei1999) >>! In T160188#3670406, @IKhitron wrote: > Do you mean it will stay work quickly like this for good? Quarry will use the new faster analytic servers, yes. But how the query executes i... [19:35:39] 10Wikibugs: Allow configuring which priority of tasks are sent to a channel - https://phabricator.wikimedia.org/T177798#3670413 (10Quiddity) [19:35:43] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3670428 (10IKhitron) >>! In T160188#3670411, @zhuyifei1999 wrote: >>>! In T160188#3670406, @IKhitron wrote: >> Do you mean it will stay work quickly like this for good? > > Quarry will use the new faster anal... [19:37:01] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3670432 (10zhuyifei1999) Or make faster :) [19:37:23] 10Cloud-Services, 10cloud-services-team (Kanban), 10Patch-For-Review: Remove non-interactive bots from #wikimedia-cloud - https://phabricator.wikimedia.org/T177427#3658254 (10Quiddity) >>! In T177427#3670111, @Andrew wrote: > Definitely possible but will require some coding in the wikibugs tool. I'm not sur... [19:38:45] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3670437 (10IKhitron) >>! In T160188#3670432, @zhuyifei1999 wrote: > Or make faster :) Make faster it's great, but if it's possible that it will make slower, then this task isn't closed forever, I'm affraid. [20:02:26] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3670531 (10zhuyifei1999) >>! In T160188#3670437, @IKhitron wrote: > Make faster it's great, but if it's possible that it will make slower, then this task isn't closed forever, I'm affraid. In that case, if on... [20:50:05] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3670621 (10IKhitron) Not 5 times, at least 1800 times. I see, thankmyou for this explanation, and for your help, @zhuyifei1999. [21:15:55] 10Wikibugs: Allow configuring which priority of tasks are sent to a channel - https://phabricator.wikimedia.org/T177798#3670413 (10valhallasw) Rough changes needed: - pass priority into the channel lookup @ https://github.com/wikimedia/labs-tools-wikibugs2/blob/master/redis2irc.py#L97 - extend the channel look...