[00:00:02] 10Cloud-VPS, 10cloud-services-team (Kanban): New Trusty base images hang on boot - https://phabricator.wikimedia.org/T172064#3495669 (10Andrew) Bisect blames 2b18741526dff42582d84c25e0a8a7fddec080f0 which is very surprising! I need to test more but it seems to be right. [00:00:12] !log tools Restarted kube-proxy service on bastion-03 [00:00:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [00:04:21] 10PAWS: R kernel not available for PAWS - https://phabricator.wikimedia.org/T164198#3495675 (10yuvipanda) 05Open>03Resolved a:03yuvipanda @halfak should be all sorted out now! [00:05:18] 10Cloud-Services: Can't create or manage domains in the PAWS project - https://phabricator.wikimedia.org/T172344#3495678 (10yuvipanda) [00:13:49] 10Cloud-Services: Can't create or manage domains in the PAWS project - https://phabricator.wikimedia.org/T172344#3495701 (10yuvipanda) Here's what I want to do: 1. Move paws.wmflabs.org from a proxy to a DNS domain 2. Set up an A record for paws.wmflabs.org to point to a floating IP (this floating IP is in the... [00:13:59] 10Cloud-Services: Can't create or manage domains in the PAWS project - https://phabricator.wikimedia.org/T172344#3495702 (10yuvipanda) a:03Andrew [00:25:57] !help [00:25:57] annika: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [00:26:06] what happened to job 7913348? [00:26:22] annika: on what host? [00:26:27] i'm unable to find out because of my internet connection [00:26:59] do you need the host with the grid? [00:27:13] ah a grid job. looking.... [00:27:29] qstat is taking a long time to respond... [00:28:13] huh, this is doable with qstat? [00:29:03] it should be I think... [00:29:21] the job doesn't seem to be running according to http://tools.wmflabs.org/admin/oge/status [00:29:41] but qstat is hanging ... [00:30:11] it was aborted with "unable to run job" after running for a while [00:30:11] RECOVERY - Puppet errors on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:32:28] qstat is hanging? grid kaboom? [00:33:00] I think the NFS server may be melting :/ [00:33:14] madhuvishy: are you around? [00:33:55] oh no~ [00:35:51] bd808: nfs on tools-bastion-05 is fast [00:36:17] yeah qstat works [00:36:18] how are you on -05? [00:36:30] bastion-02 and 03 are pretty much unusable [00:36:34] Zppix: ssh [00:36:42] :P [00:36:55] zhuyifei1999_ special access to isnt it :P [00:37:08] bastion 02 is unusable as well? uh oh [00:37:26] Zppix: it's bastion, why special access? [00:37:28] let me try [00:37:41] some bastions are password and username'd [00:37:51] meh [00:38:08] either 05 isn't hooked up as a submission host or the grid is in real trouble [00:38:20] bd808 even connecting to tools-dev.wmflabs.org is slow [00:38:32] that's bastion-02 [00:38:41] ik [00:38:53] "bastion-02 and 03 are pretty much unusable" [00:38:56] I knew that [00:39:00] well, at least qstat works on 05 [00:39:15] oh i missread -02 [00:39:17] :P [00:39:37] what do you ssh to to getinto -05 zhuyifei1999_ [00:39:41] i got to do somethings :P [00:39:56] ssh tools-bastion-05.tools.eqiad.wmflabs [00:40:15] via the normal cloud vps jump servers [00:40:16] yep, the cool way [00:40:43] or some hidden dark magic that you can view on openstack-browser [00:40:49] zhuyifei1999_: you're right, I was jsut typing a nonsense qstat command :/ [00:40:52] bd808 sorry this is an obvous question did you try restarting them? :) [00:42:08] kicking everyone out of toolforge is a last ditch response [00:44:09] well, the good part is that NFS isn't breaking every single month like a few year ago :) [00:45:07] the team has put a lot of work into it [00:46:23] yeah this is rediculous. I'm going to have to reboot -03 [00:46:57] !lot tools tools-bastion-03 not usably responsive to interactive commands; will reboot [00:47:03] !log tools tools-bastion-03 not usably responsive to interactive commands; will reboot [00:47:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [00:48:59] PROBLEM - Puppet errors on tools-paws-worker-1002 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [00:50:25] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:50:31] https://phabricator.wikimedia.org/T171944 sigh [00:51:01] my guess is a big scp/sftp transfer [00:51:16] I'm on -02 and got a partial ps auxf [00:52:43] -02 seems to be a bit better after I rebooted -03 [00:54:16] Uhhh sorry bd808 at gym. Can hop on laptop if you need me [00:54:19] I find the most suspicious process is puppet actually [00:54:41] Puppet spikes the load a bit but should be momentary [00:54:48] puppet pops way up on top when it runs but that is io/wait [00:56:11] madhuvishy: you and your healthy life. :) [00:56:42] https://phabricator.wikimedia.org/P5845 [00:57:22] (nda-ing this paste, haven't checked if there are anything sensitive like passwords in the commands) [00:57:28] can't hop in atm but iotop -ao [00:57:31] is usually useful [00:58:39] heavy hitters are annika's attempts to run qacct actually [00:59:02] what is qacct? [00:59:03] interesting [00:59:21] that's the way you look for info on a finished job [00:59:25] https://linux.die.net/man/1/qacct reading [00:59:54] zhuyifei1999_: its the cli to the accounting file that we read for grid-jobs [00:59:57] PROBLEM - Puppet errors on tools-bastion-05 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [01:00:35] ah they can kill nfs [01:01:05] yeah, we need to get that file off of NFS and also rotate it more often... [01:02:08] annika: are you running that over and over? [01:02:26] it always fails for me [01:02:49] its killing things [01:02:54] What's the current status [01:02:56] please stop for a while [01:03:02] maybe i'm strangling ssh to death [01:03:05] ok [01:03:26] annika: its making the NFS server sad basically [01:03:37] and not stopping when you try to exit out I guess [01:03:42] madhuvishy: basically tools-bastion-03 is rebooted, -02 I got the ps auxf [01:03:47] bd808: may be let's rotate the accounting file? [01:03:53] madhuvishy: I think we're ok [01:03:57] Okay.. [01:04:10] we should rotate it though, yeah [01:04:16] i'm sorry, i wasn't aware of that beforehand [01:04:33] I kind of want to rotate it daily and keep 30 or so of them [01:04:43] annika: yeah, not mad at you :) [01:05:28] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [01:06:50] Thanks bd808! [01:08:16] annika: that wc -l you are running really eats up NFS io too [01:08:53] i know … [01:09:52] May be submit those as jobs instead of running on bastion, so it doesn't overload the bastions [01:09:53] you can run that on the grid and that makes it easier for people to work on the bastions [01:10:05] :) [01:11:27] bd808: sshd: bd808@pts/13,pts/0 <= Just wondering, why do you have two pts? [01:11:53] zhuyifei1999_: *shrug* [01:12:32] oh that may have been when I was active with 2 shells that are sharing a master conenction [01:12:50] https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing [01:13:54] things look tame right now [01:14:04] I'm going to go start cooking dinner :) [01:14:52] wow didn't know this [01:23:59] RECOVERY - Puppet errors on tools-paws-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [01:24:19] PROBLEM - Puppet errors on tools-worker-1021 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:24:55] 10Toolforge, 10Tools-Kubernetes, 10Kubernetes, 10Tracking: Make webservice backend default to kubernetes - https://phabricator.wikimedia.org/T154504#2913589 (10bd808) What if instead we require `--backend=...` and stop with an error if it is not provided? If that were combined with {T148872} it would be si... [01:26:07] PROBLEM - Puppet errors on tools-exec-1415 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:40:00] RECOVERY - Puppet errors on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [01:51:08] RECOVERY - Puppet errors on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [01:52:00] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [02:01:38] bd808 hi, i found more then one occurence of the word "Wikimedia Labs" in https://wikitech.wikimedia.org/wiki/Operations_requests :) [02:02:10] It's a wiki? [02:03:33] Reedy nope, i meant it needs replacing with the word Wikimedia Cloud :) [02:03:38] due to the rebrand :) [02:03:49] It's a wiki [02:03:52] Change it yourself [02:04:51] i am [02:05:32] https://wikitech.wikimedia.org/w/index.php?title=Operations_requests&type=revision&diff=1767127&oldid=819332 [02:12:03] ok, finally [02:12:07] we will speed things up now [02:12:13] i just couldnt get online [02:44:33] 10VPS-project-Wikistats: wikistats: add wikimania wikis - https://phabricator.wikimedia.org/T172342#3496012 (10Dzahn) a:03Dzahn [02:59:20] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [03:01:59] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [03:20:04] bd808: ping? [03:20:20] hey SMalyshev, what's up? [03:21:03] bd808: I'm trying to figure out why logstash hates https://gerrit.wikimedia.org/r/#/c/299825/7... is there any logging/debug output there except for the failure tag? [03:22:25] I think all you get on a grok failure is the tag... [03:22:47] maybe there is something in the logstash log on disk though [03:23:02] bd808: nothing in /var/log/logstash [03:23:49] ok, then another question - is it ok to edit configs in deployment-logstash2 a bit and try to figure out what's going on? [03:24:10] (which probably will require restarting logstash to pick new configs if I understand it correctly) [03:24:34] totally. you probably want to disable puppet with `sudo puppet --disable "debugging logstash"` before you do [03:24:44] oh, ok [03:24:54] or puppet will fight you :) [03:25:15] bd808: yep :) so for logstash I just can du sudo service logstash restart? [03:25:28] yeah, that shoudl do it [03:25:41] its a systemd unit [03:27:47] debugging grok parse failures is annoying [03:30:19] yeah I imagine since it's for stream processing not much diagnostics available [03:35:06] 10Cloud-VPS, 10cloud-services-team (Kanban): New Trusty base images hang on boot - https://phabricator.wikimedia.org/T172064#3496040 (10Andrew) - Old images use puppet 3.4.3 - New image suse puppet 3.8.5 In both puppet versions, if daemonize = true (or, presumably, 'no') and I run 'puppet agent' (with no args... [03:35:32] bd808: Hm.. It's nice that PHP 5.6 is supported (instead of PHP 5.5) via Kubernetes, and composer is now pre-insttalled on tools-login. But I can't run 'composer update' there because Composer rejects the PHP version from tools (which is 5.5.9) [03:35:43] And insie the kube shell, composer isn't installed. [03:35:59] Is there a login host with PHP 5.6, and/or could we install composer in the pod? [03:36:09] Krinkle: we should fix that! [03:36:50] 10Cloud-VPS, 10cloud-services-team (Kanban): New Trusty base images hang on boot - https://phabricator.wikimedia.org/T172064#3496041 (10Andrew) btw, it turns out the init script does not observe the START=no setting in /etc/default/puppet which I tried ages ago and which is why I decided that this wasn't a pup... [03:37:08] I'm surprised I haven't run into it earlier. I've been doing 'git pull && composer update' for most of my tools for a while now [03:37:21] but now it fails for https://tools.wmflabs.org/blankpages/ because I added a requirement php >= 5.6 [03:37:26] yeah [03:37:30] I'll manually install it in the pod for now [03:37:48] Im trying to find the tracking bug for adding packages to the k8s containers... [03:38:56] 10Toolforge, 10Kubernetes, 10Tracking: Packages to be installed in Tool Labs Kubernetes Images (Tracking) - https://phabricator.wikimedia.org/T140110#3496045 (10bd808) [03:40:44] 10Toolforge, 10Kubernetes: Add composer to PHP base images - https://phabricator.wikimedia.org/T172358#3496046 (10bd808) [03:41:57] 10Tools: Tool "blankpages" loads jquery from ajax.googleapis.com - https://phabricator.wikimedia.org/T172275#3496062 (10Krinkle) 05Open>03Resolved p:05Triage>03Normal This was fixed a while ago (~ 2015) in the base class for my tools, but BlankPages was still using an old toolserver-compat version of my... [03:41:58] 10Tools, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3496065 (10Krinkle) [03:53:03] bd808: thx [03:53:28] the way we install that is ... kind of gross [03:54:00] PROBLEM - Puppet errors on tools-exec-1423 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [03:54:07] its a clone of a git repo of composer installed by composer -- https://phabricator.wikimedia.org/diffusion/CICP/repository/master/ [03:57:28] bd808: Wait, that one is used by tools-login? [03:57:38] yup [03:57:46] Eww [03:58:08] and puppet updates the clone on each run [03:58:28] there seems to be a package in stretch [04:04:36] 10Toolforge, 10Kubernetes: Add composer to PHP base images - https://phabricator.wikimedia.org/T172358#3496089 (10bd808) The deployment on the Tooforge bastions is from [[https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/toollabs/manifests/composer.pp|`::toollabs::composer`]]... [04:08:18] bd808: looks like something wrong with either quotes or HTTPDATE - all the rest seems to parse fine... [04:10:00] you could probably skip parsing the date and just keep the timestamp that the syslog message will already cary [04:11:18] bd808: weird thing is the date pattern looks fine... I'll dig a bit more if I don't figure it out I'll do that [04:19:04] RECOVERY - Puppet errors on tools-exec-1423 is OK: OK: Less than 1.00% above the threshold [0.0] [04:20:11] bd808: I am thinking - maybe it is because we're writing into @timestamp? some typing is needed or something? [04:20:28] because @timestamp seems to have type, but HTTPDATE just matches string... [04:20:33] that could be something, yeah [04:20:55] maybe just change that var name and see what happens? [04:21:49] yeah doing so [04:22:22] the @timestamp is magic in logstash's data [04:23:00] yup looks like with different field name it works [04:23:12] \o/ [04:23:25] live hacking has its benefits :) [04:23:40] e.g. https://logstash-beta.wmflabs.org/app/kibana#/doc/logstash-*/logstash-2017.08.03/wdqs?id=AV2mVJ_khvX_CkfKbvQO&_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-15m,mode:quick,to:now)) [04:25:47] you might want to call "date" something like "http_date" or "str_date" [04:26:16] elasticsearch has this annoying thing where all fields with the same name have to be typed the same [04:26:33] it use to split on doc type but they changed that [04:26:38] https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html [04:26:44] looks like this is what we want [04:27:40] or maybe not if we don't want to change @timestamp... but looks like this is the way to change it [04:30:11] yeah, a date { .. target => "@timestamp" } might work, but its probably not needed [04:30:42] the timestamp from the event and from syslog should be really close to each other [04:31:10] here's a recipe for doing it -- https://stackoverflow.com/a/25156839/8171 [04:32:56] yeah thinking about it I'm not convinced we really need it [04:35:13] 10Toolforge, 10Kubernetes, 10Tracking: Packages to be installed in Toolforge Kubernetes Images (Tracking) - https://phabricator.wikimedia.org/T140110#3496123 (10bd808) [04:38:32] 10Toolforge, 10cloud-services-team (Kanban), 10Kubernetes, 10Patch-For-Review, 10User-bd808: Add composer to PHP base images - https://phabricator.wikimedia.org/T172358#3496124 (10bd808) a:03bd808 [04:41:51] bd808: ok, updated the patch to working config, please take a look and see if I didn't break anything :) [04:42:34] re-enable puppet and update the cherry-pick [04:42:39] that will test it for you :) [04:43:08] bd808: how you update the cherry-pick? [04:43:58] lets see if there are instructions. if there aren't I'll write some :) [04:44:27] yeah I'm not sure I even did non-master puppet config... [04:44:44] https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated#Updating.2Fremoving_a_cherry-pick [04:44:48] bd808: and btw thanks for helping with it :) [04:45:07] yw :) [04:50:59] interestingly enough, updating confs does not seem to make puppet restart logstash [04:51:16] either logstash auto-picks the confs, or puppet is lazy :) [04:51:57] ok, puppet run complete, parsing still works [04:51:59] \o/ [05:13:40] 10Data-Services, 10cloud-services-team (Kanban), 10DBA, 10Security-Team, and 5 others: Make wbqc_constraints table available on Quarry et al. - https://phabricator.wikimedia.org/T170927#3496145 (10Marostegui) labsdb1011 will have puppet disable till Monday most likely due to maintenance, but if you let me... [05:14:05] bd808: RE: composer k8s patch, depending on how usage is intended from your perspective, might want to pin with a git-checkout to a specific version. I know for apt-get this is not conventional, but for other run commands, deterministic might be useful, so that you can do a rebuild after changing a different command without also updating everything else. [05:14:53] Anyway, if images are kept in a stable/versioned way and rebuild is only done to intentinally update stuff than another implicit update wouldn' tbe too bad as you could then address that and keep iterating before promoting anything. [05:15:12] my only fear there is that if I pin it we will never remeber to update the Dockerfile [05:15:39] 10PAWS: HTTP 404 error (or sometimes 502 error) when trying to access the Control Panel of PAWS - https://phabricator.wikimedia.org/T140525#3496147 (10R96340) I tried to restart the notebooks, terminals, server, all computers logged-in my account, re-log-in my account in all of my computers ,and it still happene... [05:25:19] PROBLEM - Puppet errors on tools-worker-1021 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:43:33] 10cloud-services-team, 10DBA: Compress InnoDB on db1102 - https://phabricator.wikimedia.org/T172169#3496169 (10Marostegui) [06:26:58] 10Tool-Erwin's-tools: 502 Bad Gateway - https://phabricator.wikimedia.org/T172360#3496182 (10Supernino) [06:30:21] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [06:41:04] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1427 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:42:21] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:50:00] PROBLEM - Puppet errors on tools-exec-1430 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:03:14] 10PAWS: HTTP 404 error (or sometimes 502 error) when trying to access the Control Panel of PAWS - https://phabricator.wikimedia.org/T140525#3496200 (10Strainu) Re-adding @yuvipanda since I believe he was accidentally removed. [07:17:22] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [07:21:07] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1427 is OK: OK: Less than 1.00% above the threshold [0.0] [07:29:59] RECOVERY - Puppet errors on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0] [09:02:25] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Patch-For-Review: New Trusty base images hang on boot - https://phabricator.wikimedia.org/T172064#3484445 (10hashar) `random: nonblocking pool is initialized` I got that one on all instances I looked at and that is the last message in dmesg after boot. That is... [09:27:26] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:02:25] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [10:09:32] 10Cloud-VPS: Please install a recent version of node and npm by default on new labs instance - https://phabricator.wikimedia.org/T157368#3496521 (10Jhernandez) Thanks for digging in @bd808! It seems pretty clear for now: * For easy set up choose jessie as the base image when creating an instance * `apt-get i... [10:18:16] 10Tools, 10Privacy: Tool "cite-o-meter" loads fork-me-on-github ribbon from github - https://phabricator.wikimedia.org/T172280#3496555 (10Aklapper) [10:26:18] 10Cloud-VPS: Please install a recent version of node and npm by default on new labs instance - https://phabricator.wikimedia.org/T157368#3496564 (10hashar) Ah I missed Jon request about `Labs has an old version of node installed (0.10)`. So yeah definitely switch to Jessie and you will have Node 6 maintained by... [10:27:38] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:46:23] morning [10:46:34] Quarry appears to be adrift again [10:46:38] https://quarry.wmflabs.org/query/18908 [10:46:58] Is including rows that should not be present, having checked the relevant file on English Wikipedia [10:47:17] I can't see an obvious logic error so am I puzzled [10:57:09] Anyone? [11:07:35] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [11:07:56] ShakespeareFan00: I looked at the current revision of that query, you might have hit https://phabricator.wikimedia.org/T170464 so the query runs forever [11:08:16] I'm not sure what you mean by "Is including rows that should not be present, having checked the relevant file on English Wikipedia" [11:08:44] Did you read the query? [11:08:45] since the query did not run successfully [11:08:56] 18908? yes [11:09:15] it supposed to find very large images and exclude those already tagged [11:09:21] Which is the NOT IN part of the clause [11:09:28] However it's not doing that [11:09:43] It's still returning rows that ARE in the relevant category [11:09:53] 18908 has no results [11:10:00] It should have? [11:10:03] Hmm [11:10:06] https://quarry.wmflabs.org/query/18908 [11:10:20] It says it's still running [11:10:20] I just explained the reason [11:10:25] Right [11:10:32] So a valid query doesn't run [11:10:33] :( [11:10:35] (sigh) [11:10:54] selecting c1.*, c2.* at the same time just doesn't work, same solumn names [11:11:04] https://phabricator.wikimedia.org/T170464 [11:11:28] So how do you ACTUALLY do the intended query? [11:11:49] Namely find stuff in one category that aren't in another [11:11:52] ? [11:12:01] hmm [11:12:12] Categorylinks seems to be the correct table to use [11:12:19] But if you can't alias the same table... [11:12:33] Quarry is broken [11:12:52] because it can't do something most SQL based systems should [11:12:58] INNER JOIN categorylinks c2 ON c2.cl_from = p.page_id AND c1.cl_from = c2.cl_from AND c2.cl_to NOT IN ('Wikipedia_non-free_file_size_reduction_requests') [11:13:03] ^ bad sql [11:13:17] Whats wrong with it? [11:13:20] it means all images have a category that is not 'Wikipedia_non-free_file_size_reduction_requests' [11:13:38] AND c1.cl_from = c2.cl_from is unnecessary as well [11:13:45] Okay [11:13:55] How do you do the intended subset then? [11:14:10] Because first what wanted is a set of all Non-free media [11:14:20] And then the ones that haven't been checked for size [11:14:33] And then ONLY the ones that haven't already been tagged [11:14:35] and if you get rid if i.*, c1.*, c2.*, the query should run and result results, although wrong results [11:14:37] a sec [11:16:02] try left join with NULL match [11:16:10] or NOT IN with a subquery [11:17:13] How about instead I just delete the query, and we start again? [11:17:19] uh sure [11:17:44] Line by line.. [11:18:02] We are using en-wiki : "USE enwiki_p;" [11:18:21] We want image names - "SELECT DISTINCT i.img_name FROM image i" [11:18:24] Okay so far? [11:18:39] DISTINCT should be unnecessary [11:19:02] unless query results says otherwise [11:19:15] "WHERE i.img_width * i.img_height > 1e+6LIMIT 50" will give all the really big images [11:19:32] "LIMIT 50" so we don't overload the server [11:19:48] yeah sure [11:19:54] Now the next step is to reduce the intial query based on categorisation [11:20:06] But we can't use categorylinks twice due to bug [11:20:42] you can [11:20:50] just don't return the results [11:20:51] "INNER JOIN page p ON p.page_title = i.img_name AND p.page_namespace = 6" gives us a name [11:21:05] i.e. don't add to the select statement [11:21:17] "INNER JOIN categorylinks c1 ON c1.cl_from = p.page_id AND c1.cl_to = 'All_non-free_media' " is all non-free media [11:21:28] The next line is where the problems start [11:21:39] "INNER JOIN categorylinks c2 ON c2.cl_from = p.page_id AND c2.cl_to NOT IN ('Wikipedia_non-free_file_size_reduction_requests') " [11:21:48] Because this should be a subset... [11:22:02] And at the moment it's not quite doing that [11:22:05] that's probably the only problem afaict [11:22:20] I'd write it as LEFT JOIN categorylinks c2 ON c2.cl_from = p.page_id AND c2.cl_to == 'Wikipedia_non-free_file_size_reduction_requests' [11:22:33] add a where condition c2.cl_to is NULL [11:24:12] Typo somewhere :( [11:24:21] ? [11:24:41] My error == vs = [11:24:55] Works [11:24:57] :) [11:25:19] yeah, 50 at a time is very unnessary afaict [11:25:21] but still gives results that it shouldn''y [11:25:45] which? [11:25:50] zhuyifei1999_: When i ran it first time round it came back with about 5000 rows which is hard on a browser annd the server [11:26:18] https://en.wikipedia.org/wiki/File:AllChinaGamesFlag.svg is tagged for non-free reduce and thus shouldn't be in the results [11:26:21] ah, some people try to SELECT * FROM page [11:26:30] looking [11:26:35] So there's a logical error somewhere [11:26:55] I don't see the category [11:27:08] you expect Wikipedia_non-free_file_size_reduction_requests [11:27:21] Yeah [11:27:22] actual: Wikipedia non-free file size reduction requests for manual processingVector images of trademarksNon-free logos [11:27:30] where is Wikipedia_non-free_file_size_reduction_requests [11:27:43] == is an exact match [11:27:50] Non free reduce should categorise [11:27:56] nope [11:28:01] I'm not seeing an obvious match either [11:28:25] if you want Wikipedia_non-free_file_size_reduction_requests use can use LIKE [11:28:51] or if you want subcategories support you can use an IN clause with subqueries [11:29:12] I think I will need someone else to write this :( [11:29:19] I feel stupid right now [11:29:39] A ike clause didn't change anything [11:29:41] https://quarry.wmflabs.org/query/20466 <= see eg. [11:29:58] LIKE "Wikipedia_non-free_file_size_reduction_requests%" deh [11:30:01] *duh [11:30:15] % means match any length [11:30:47] zhuyifei1999_: Please assume I am a robot... [11:30:49] ;) [11:30:52] 20466 is about IN / NOT IN with subcategories [11:30:59] * zhuyifei1999_ facepalm [11:31:16] And I can't at the moment I am not fully understanding the query you linked :( [11:31:23] And yes faceplam is appropriate [11:31:26] you must be a very intelligent robot with a very advanced AI :P [11:32:01] ignore the fire few lines, start from SELECT ipb_address, ipb_reason, ipb_by_text [11:32:06] *first [11:32:16] Well not very advanced... I still can't write SQL queries [11:32:17] XD [11:32:20] Anyway [11:32:48] Seems to be working using % [11:32:54] And I've learnt something [11:33:33] ;) [11:34:04] Namely that I am ill-educated about SQL :( [11:34:38] google and stackoverflow are your friend :) [11:34:49] if you have issues you can ask here as well [11:35:45] And thanks for your patience [11:36:07] btw, if you have any idea about a fix for the bug on select c1.*, c2.*, feel free to leave a comment on the ticket or submit a patch [11:36:42] basically query results are stored in tables, and it cannot handle duplicate column names [11:36:47] np [11:36:51] Can MysqL? [11:37:00] nope [11:37:04] Ah... [11:37:13] Is there a way to alias them? [11:37:38] well, it's posslibs, but we need backworks-compatibility [11:37:45] Okay [11:37:48] above my ability [11:37:50] ;) [11:38:04] ie. I'm not gonna alter table on 10k sqllite databases [11:40:15] Quite [11:40:16] :) [11:47:27] PROBLEM - Puppet errors on tools-paws-worker-1013 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:22:29] RECOVERY - Puppet errors on tools-paws-worker-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [12:49:08] 10Cloud-Services, 10Huggle: huggle.huggle.eqiad.wmflabs does not have puppet installed - https://phabricator.wikimedia.org/T166588#3496851 (10Petrb) hmm, that I probably did, I have very bad experience with puppet, it happened to me couple of times that puppet completely ruined the instance and made it defunct... [13:01:42] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Continuous-Integration-Infrastructure, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3496881 (10hashar) Eventually I was looking at the rabbitMQ reported statistics on https://... [13:15:50] 10Tool-fatameh: Dot in title - https://phabricator.wikimedia.org/T172383#3496916 (10Fnielsen) [13:16:42] 10Tool-fatameh: Dot in title - https://phabricator.wikimedia.org/T172383#3496929 (10Fnielsen) See also https://www.wikidata.org/w/index.php?title=User_talk:GZWDer&oldid=531264407#Dot_after_title [13:33:33] PROBLEM - Puppet errors on tools-checker-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:36:05] PROBLEM - Puppet errors on tools-checker-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:55:03] 10Tool-fatameh: Fatameh adds a short author name that is already there. - https://phabricator.wikimedia.org/T172385#3497021 (10Fnielsen) [14:10:15] 10Tools: Tool "codeqc" loads assets from google and cloudflare - https://phabricator.wikimedia.org/T172389#3497104 (10zhuyifei1999) [14:11:52] o/ i'm working on a m1.xlarge instance (zim-proto.mobile.eqiad.wmflabs) and i think it should have 160gb storage, but in fact it looks like i only have 18gb available. am i missing something? [14:12:02] https://www.irccloud.com/pastebin/ZtYoK1K8/ [14:13:15] er, 19gb available and 18 (but 100%) used? anyway, i'm out and it's a surprise. [14:14:14] 10Tools: Tool "comidentgen" loads bootstrap and jquery from ajax.googleapis.com and bootstrapcdn - https://phabricator.wikimedia.org/T172391#3497146 (10zhuyifei1999) [14:23:33] mdholloway hi, you need to add a puppet class which adds the extra storage [14:23:35] role::labs::lvm::srv [14:23:40] mdholloway: cloud VPS only allocates 20 GB by default. You can have the remaining 140GB allocated in the /srv [14:23:57] paladox: harej: i see. thanks! [14:24:01] By enabling the puppet role in Horizon, yeah [14:24:06] got it [14:24:08] your welcome :) [14:24:21] 10Tools: Tool "commons-coverage" loads map tiles from OSM - https://phabricator.wikimedia.org/T172395#3497223 (10zhuyifei1999) [14:24:48]