[00:41:02] <wikibugs>	 10Cloud-VPS, 10cloud-services-team (Kanban), 10Continuous-Integration-Infrastructure, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3433128 (10bd808) I ran `sudo rabbitmqctl purge_queue` for the ceilometer_notifications.* q...
[01:09:40] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1417 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[01:44:40] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0]
[03:44:42] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[04:38:46] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1427 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[04:39:40] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0]
[05:18:46] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1427 is OK: OK: Less than 1.00% above the threshold [0.0]
[05:29:43] <zhuyifei1999_>	 bd808: I'm wondering, is it somehow possible to make https://tools-static.wmflabs.org/cdnjs/ajax/libs/bootswatch/3.2.0/css/yeti/bootstrap.min.css not import from fonts.googleapis.com?
[05:31:02] <bd808>	 zhuyifei1999_: it would be ugly. we mirror the upstream repo with no changes. We would have to patch it and keep patching for each new version
[05:31:21] <zhuyifei1999_>	 I realized quarry's star icon doesn't work anymore and I suspect it's related to paths, so want to try moving all local css'es to use cdnjs
[05:31:28] <zhuyifei1999_>	 uh :(
[06:12:16] <legoktm>	 zhuyifei1999_: maybe we could have some lua rewrite it on the fly or something? it theoretically is just a find/replace right?
[06:30:50] <zhuyifei1999_>	 legoktm: could be possible via nginx I think
[06:31:54] <zhuyifei1999_>	 but we can't specify that in the html (i.e. can't say <link href='blah' replace='s/blah/blag/g'>
[06:33:00] <legoktm>	 yeah, I was thinking of doing it on the webserver side
[06:34:01] <zhuyifei1999_>	 well, cdnjs is kind of global and if we screw up :(
[06:34:50] <zhuyifei1999_>	 bd808: can some sort of rewrite be added to cdnjs? or is that too unmaintainable?
[06:35:18] <zhuyifei1999_>	 I mean on the nginx server
[06:37:38] <bd808>	 zhuyifei1999_: yeah. it could be done the same way we did it for /fontcdn/
[06:37:41] <bd808>	 in https://github.com/wikimedia/puppet/blob/production/modules/toollabs/templates/static-server.conf.erb
[06:38:44] <bd808>	 it will break content hashes though
[06:39:13] <bd808>	 so that's a problem
[06:40:02] <legoktm>	 oh right, ugh
[06:51:14] <zhuyifei1999_>	 most of them shouldn't import
[06:51:41] <zhuyifei1999_>	 those that needs a rewrite should be a minority
[06:54:29] <zhuyifei1999_>	 and the tools that loads cdnjs modules that external resources *should* already forked the module to not load external resources
[06:54:49] <zhuyifei1999_>	 actually this remines me of a ticket
[06:54:54] <zhuyifei1999_>	 *reminds
[06:55:12] <zhuyifei1999_>	 https://phabricator.wikimedia.org/T128841
[06:57:31] <bd808>	 zhuyifei1999_: 1,267,520 results -- https://github.com/search?utf8=%E2%9C%93&q=googleapis+extension%3Acss&type=Code
[06:57:57] <bd808>	 oh... I left the repo..
[06:57:57] <zhuyifei1999_>	 sigh
[06:58:03] <zhuyifei1999_>	 lol
[06:59:55] <bd808>	 apparently github won't tell me
[07:00:16] <bd808>	 it says there are 0 hits for googleapis in that repo which we know to be false
[07:01:11] <zhuyifei1999_>	 I'll clone the repo somewhere
[07:01:31] <bd808>	 I'm running a search on the clone on tools-static-11
[07:01:42] <bd808>	 its slow. it's a freaking huge repo
[07:03:11] <wikibugs>	 10Quarry: The star icon will not show/load in Quarry's star button - https://phabricator.wikimedia.org/T175466#3594195 (10zhuyifei1999)
[07:04:35] <zhuyifei1999_>	 how large is the repo? /me is thinking whether or not to use my laptop for this
[07:06:26] <bd808>	 like 100G
[07:06:36] <zhuyifei1999_>	 uh
[07:07:59] <bd808>	 I've got the search running in a screen. it will have an answer eventaully
[07:08:16] <bd808>	 git grep -l '@import' | grep -e '\.css$'|wc -l
[07:08:31] <zhuyifei1999_>	 ok thanks
[07:09:37] <zhuyifei1999_>	 https://phabricator.wikimedia.org/T128841#2391357 I dunno where I ran this from though
[07:13:20] <zhuyifei1999_>	 in that list only ng-wig seems to be importing something other than google fonts
[07:15:38] <bd808>	 3994 files matched `git grep -l '@import' | grep -e '\.css$'|wc -l`
[07:15:48] <zhuyifei1999_>	 o.O
[07:16:32] <bd808>	 there are 3147 libraries * N versions per lib :)
[07:16:52] <zhuyifei1999_>	 yeah sigh
[07:17:24] <bd808>	 scfc's comment at https://phabricator.wikimedia.org/T128841#2845667 is very relevant I think :)
[07:18:05] * bd808 goes to bed
[07:18:21] <zhuyifei1999_>	 well, it kind of sucks to have to fork the script and fork the dependencies as well
[07:18:34] <zhuyifei1999_>	 yeah I should do as well, 2am
[07:18:53] <bd808>	 o/
[07:19:34] <zhuyifei1999_>	 actually right I guess I'll change relative to absolute path for the fonts
[07:19:38] <zhuyifei1999_>	 gtg
[10:41:43] <sau226>	 !help unable to make user page on beta cluster. Get not logged in error message even though logged in (opened new tab).
[10:41:43] <goat-bot>	 sau226: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[10:41:48] <sau226>	 Yes sir
[10:44:30] <sau226>	 Cancelled as i fixed it
[10:44:42] <sau226>	 sorry
[12:48:48] <sau226>	 Phew
[13:39:34] <sau226>	 Is it possible for me to test the use of user right assignment in the test wiki on the beta cluster?
[13:45:58] <valhallasw`cloud>	 sau226: unless you are a sysop there, no. It's not a test wiki in that sense, it's a pre-production wiki to see if things don't fall over when the code is changed.
[13:46:13] <sau226>	 Thanks
[13:57:06] <wikibugs>	 10Cloud-Services, 10MediaWiki-Vagrant: Vagrant reload fails when reloading mediawiki-vagrant on cloud - https://phabricator.wikimedia.org/T175484#3594612 (10Paladox)
[16:10:33] <wikibugs>	 10Data-Services: SQL database ignores logs for hours - https://phabricator.wikimedia.org/T175487#3594755 (10Reedy)
[16:22:21] <zhuyifei1999_>	 !log quarry Deployed a6173a2 on quarry-main-01 T175466
[16:22:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[16:22:25] <stashbot>	 T175466: The star icon will not show/load in Quarry's star button - https://phabricator.wikimedia.org/T175466
[16:23:13] <wikibugs>	 10Quarry, 10Patch-For-Review: The star icon will not show/load in Quarry's star button - https://phabricator.wikimedia.org/T175466#3594762 (10zhuyifei1999) 05Open>03Resolved a:05zhuyifei1999>03Ricordisamoa Thanks!
[16:24:08] <Zppix>	 zhuyifei1999_:  my ssh magically fixed itself fyi
[16:24:24] <zhuyifei1999_>	 o.O
[16:24:36] <Zppix>	 the wonders of pcs
[16:51:04] <wikibugs>	 10Data-Services: SQL database ignores logs for hours - https://phabricator.wikimedia.org/T175487#3594707 (10bd808) https://tools.wmflabs.org/replag/ currently shows ~22 hours of replication lag for the s1, s2, and s4 wikis. This explains the query results.
[16:53:17] <wikibugs>	 10Data-Services: SQL database ignores logs for hours - https://phabricator.wikimedia.org/T175487#3594790 (10IKhitron) >>! In T175487#3594788, @bd808 wrote: > https://tools.wmflabs.org/replag/ currently shows ~22 hours of replication lag for the s1, s2, and s4 wikis. This explains the query results.  The problem...
[16:53:48] <wikibugs>	 10Data-Services: high replication lag on tool labs - https://phabricator.wikimedia.org/T175487#3594791 (10Bawolff)
[16:54:07] <wikibugs>	 10Data-Services, 10DBA: Significant replication lag for the s1, s2, and s4 wikis on labsdb100[13] - https://phabricator.wikimedia.org/T175487#3594792 (10bd808)
[16:57:56] <wikibugs>	 10Data-Services, 10DBA: Significant replication lag for the s1, s2, and s4 wikis on labsdb100[13] - https://phabricator.wikimedia.org/T175487#3594794 (10IKhitron) @bd808, pay attention: The hewiki numbers are 0:00:00, and it has the same problem: https://quarry.wmflabs.org/query/21487
[17:03:12] <wikibugs>	 10Data-Services, 10DBA: Significant replication lag for the s1, s2, and s4 wikis on labsdb100[13] - https://phabricator.wikimedia.org/T175487#3594820 (10bd808) Results from `wikireplica-web.eqiad.wmnet` match production, so this is just a replication issue with the labsdb100[13] cluster.  ``` $ mysql --default...
[17:04:41] <wikibugs>	 10Data-Services, 10DBA: Significant replication lag for the s1, s2, and s4 wikis on labsdb100[13] - https://phabricator.wikimedia.org/T175487#3594821 (10IKhitron) >>! In T175487#3594820, @bd808 wrote: > Results from `wikireplica-web.eqiad.wmnet` match production, so this is just a replication issue with the la...
[17:08:44] <wikibugs>	 10Data-Services, 10DBA: Significant replication lag for the s1, s2, and s4 wikis on labsdb100[13] - https://phabricator.wikimedia.org/T175487#3594835 (10bd808) @IKhitron please calm down. I am just adding more information to your bug report so that the #DBA team can figure out what is going on easier. My comme...
[17:11:56] <wikibugs>	 10Data-Services, 10DBA: Significant replication lag for the s1, s2, and s4 wikis on labsdb100[13] - https://phabricator.wikimedia.org/T175487#3594838 (10IKhitron) >>! In T175487#3594835, @bd808 wrote: > @IKhitron please calm down.  @bd808, you didn't understand me. I don't think anybody should hurry. All I sai...
[18:35:14] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1441 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[19:07:05] <bawolff>	 Is it just me, or do the wikireplicas seem slow today?
[19:07:48] <paladox>	 bawolff i think some filled a task for that, i think it's this one https://phabricator.wikimedia.org/T175487 ?
[19:08:30] <bawolff>	 Well probably related, albeit not exactly the same thing
[19:10:15] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1441 is OK: OK: Less than 1.00% above the threshold [0.0]
[19:26:16] <wikibugs>	 10wikitech.wikimedia.org, 10Deployment-Systems, 10Release-Engineering-Team (Backlog), 10User-MarcoAurelio: Create an easier way to  add/remove/modify patches for SWAT - https://phabricator.wikimedia.org/T171940#3594906 (10MarcoAurelio) Has any discussion happened about this?  My preference would be to use...
[19:33:37] <shinken-wm>	 PROBLEM - Puppet errors on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[19:42:16] <wikibugs>	 10wikitech.wikimedia.org, 10Deployment-Systems, 10Release-Engineering-Team (Backlog), 10User-MarcoAurelio: Create an easier way to  add/remove/modify patches for SWAT - https://phabricator.wikimedia.org/T171940#3480850 (10Zppix) why not use a system like google forms (but a open source/wikimedia maintained)
[20:04:52] <wikibugs>	 10wikitech.wikimedia.org, 10Deployment-Systems, 10Release-Engineering-Team (Backlog), 10User-MarcoAurelio: Create an easier way to add/remove/modify patches for SWAT - https://phabricator.wikimedia.org/T171940#3595002 (10Reedy)
[20:08:36] <shinken-wm>	 RECOVERY - Puppet errors on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0]
[20:35:06] <bawolff>	 ugh, why doesn't my query on tool labs want to use the rc_timestamp index...
[20:43:55] <valhallasw`cloud>	 bawolff: what's the query?
[20:44:20] <bawolff>	 select count(*) from recentchanges where   rc_source != 'wb' and rc_timestamp between  '20170903000000' and '20170910000000';
[20:44:26] <bawolff>	 I redid it using rc_id
[20:44:45] <bawolff>	 which seemed to be better, I guess because its doing primary key, so the actual rows will be in the right order
[20:45:14] <bawolff>	 compared to the other case, where mysql was probably thinking the extra layer of indirection was not worth it
[20:45:37] <bawolff>	 my new query is still running though
[20:45:53] <bawolff>	 There are an unbelievable number of wikidata edits in the commons RC table
[20:47:04] <valhallasw`cloud>	 yeah, if there's no index with rc_source, so I can imagine a plain table scan was considered to be more effective
[20:47:47] <valhallasw`cloud>	 it's only ~10M rows after all
[20:48:58] <bawolff>	 Well the query was running for like 15 minutes, I just gave up and posted my bug comment anyways
[20:49:23] * bawolff was looking for patterns in which wikis strike the watchlist bug https://phabricator.wikimedia.org/T171027#3595047
[20:49:47] <valhallasw`cloud>	 bawolff: https://quarry.wmflabs.org/query/21493 just returned for enwiki. 1732913 entries
[20:50:29] <bawolff>	 Specificly I was looking at commons, as I think its one of the wikis unduly affected
[20:50:47] <valhallasw`cloud>	 ah, right.
[20:53:07] <bawolff>	 Fun fact, english wikipedia has half the number of wikidata edits as romanian wikipedia
[20:53:42] <valhallasw`cloud>	 select count(*) from recentchanges where rc_source != 'wb' and rc_id > (select rc_id from recentchanges 
[20:53:42] <valhallasw`cloud>	 where rc_timestamp > '20170903000000' order by rc_timestamp asc limit 1) and rc_id < (select rc_id from recentchanges where rc_timestamp < '20170910000000' order by rc_timestamp desc limit 1)
[20:53:46] <valhallasw`cloud>	   seems OK for enwiki (few seconds). But I guess that's basically whta you tried for commons :-)
[20:58:52] <bawolff>	 yeah basically (Except i did the two queries in separate steps)
[21:17:40] <wikibugs>	 10wikitech.wikimedia.org, 10Deployment-Systems, 10Release-Engineering-Team (Backlog), 10User-MarcoAurelio: Create an easier way to add/remove/modify patches for SWAT - https://phabricator.wikimedia.org/T171940#3595113 (10mmodell) >>! In T171940#3594906, @MarcoAurelio wrote: > Has any discussion happened ab...
[21:19:33] <wikibugs>	 10wikitech.wikimedia.org, 10Deployment-Systems, 10Release-Engineering-Team (Backlog), 10User-MarcoAurelio: Create an easier way to add/remove/modify patches for SWAT - https://phabricator.wikimedia.org/T171940#3595129 (10Zppix) >>! In T171940#3595113, @mmodell wrote: > I looked into the PageForms extension...
[23:18:56] <Dragonfly6-7>	 is CropTool broken?