[02:12:11] Are normal corporations not allowed to define their entity on wikidata? [02:15:22] I mean I used entity links from the US state that incorporated us and still my pages were deleted as advertising [02:15:54] Think I'll write a review of the Gestapo of Wikipedians [09:41:48] PROBLEM - check if wikidata.org dispatch lag is higher than 2 minutes on wikidata is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1438 bytes in 0.122 second response time [12:14:22] mh i think a bot is causing the lag [12:14:29] https://www.wikidata.org/wiki/User:BotNinja [12:19:29] * aude waves [12:20:13] agree about the bot [12:22:02] !admin [12:22:10] jzerebecki: hi [12:23:03] JohnFLewis: any better idea than temporarily blocking the bot until lag is down again? [12:23:27] no better ideas, there is a lag, looking [12:25:32] jzerebecki: I've blocked the bot for 24 hours stating dispatch stats [12:25:44] thx [12:28:55] jzerebecki: / aude: do we have an API for that? I heard of it being discussed but never saw it added or so [12:29:15] no db lag, only dispatch. api, jobrunner server load is low. [12:30:27] JohnFLewis: to query the dispatch lag? yes there is an api. it is what we also use to make icinga complain when it gets too high. [12:30:47] I'll find it from the icinga check then quickly [12:32:59] got it [12:34:21] https://www.wikidata.org/w/api.php?action=query&meta=siteinfo&format=json&siprop=statistics [13:00:29] JohnFLewis: jzerebecki: thank you! is the lag actually going down now? [13:01:10] Lydia_WMDE: just looked, gone (stalest) from 3 hours 10 to 3 hours 34 :/ [13:01:26] seems at least all other bots are not editing because the lag is too high. that's good [13:01:30] JohnFLewis: mpfh [13:01:49] jzerebecki: do we know if the dispatching is happening at all? [13:05:35] Lydia_WMDE: dispatcher works fine [13:05:41] ok [13:06:00] i think sometimes dispatch stats appears to go up, but is really catching up [13:06:14] i would look at average [13:06:21] *nod* [13:06:26] ok then let's wait it out i guess [13:06:36] let's see [13:07:16] * aude can check in an hour or so (after breakfast) [13:08:08] :) [13:08:10] thanks [13:09:09] average also seems to be increasing but I'll leave it to you lot and I'll just hover around if people need me for stuff :) [13:18:44] Lydia_WMDE: autoupdating should work now, I had problems with my crontab config [13:18:55] benestar: thanks! [13:18:55] and also the bot should to it again [13:20:48] benestar: there is still a dupe in there at least that was fixed [13:21:03] or did the page not update yet? [13:21:20] * hoo waves [13:21:50] Lydia_WMDE: Did anyone look at the dispatch lag, yet? [13:21:58] Lydia_WMDE: when was it fixed? [13:22:01] hoo: yeah jzerebecki and aude [13:22:07] So, it's fixed? [13:22:18] benestar: a while ago - it's author [13:22:29] hoo: supposedly yes [13:22:46] I only see "author abbreviation" on the page [13:23:07] Oo [13:23:16] it was there when i just checked [13:23:20] let me check again [13:23:41] benestar: still there - Autor - de [13:24:47] oh, I searched for author [13:25:37] Lydia_WMDE: well, it's still in wdqs :S http://tinyurl.com/ndxuut4 [13:25:48] hmmmmm [13:26:29] the time goes up because time progresses but the number of pending goes down, that mean it will catch up [13:26:54] SMalyshev: how's the current lag of the query service? [13:28:39] Lydia_WMDE: either the lag is >2 days or the update script missed those edits [13:28:42] jzerebecki: Ok, started another dispatcher now, in hte hope that it will pick that up [13:29:03] benestar: meh [13:29:23] hoo: do you know what's the current status of the query service (lag, update script etc.) [13:29:38] I can check... the one in production or the one on labs? [13:36:58] hoo: is one dispatcher using up a cpu? [13:37:08] Probably mine [13:37:13] running with a very large batch size [13:37:32] terbium is under heavy load due to us [13:37:34] but that's normal [13:38:03] no i'm only asking to know if the normal dispatcher is cpu bound [13:38:58] I think it spends a bit of its time waiting for the DB, but usually it is [13:40:44] https://ganglia.wikimedia.org/latest/?r=week&cs=&ce=&m=cpu_report&c=Miscellaneous+eqiad&h=terbium.eqiad.wmnet&tab=m&vn=&hide-hf=false&mc=2&z=small&metric_group=ALLGROUPS [13:42:33] is that our dispatcher that made the jobqueue grow?: https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Miscellaneous%20eqiad&h=terbium.eqiad.wmnet&r=week&z=small&jr=&js=&st=1436621739&v=3439901&m=Global%20JobQueue%20length&z=largehttps://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Miscellaneous%20eqiad&h=terbium.eqiad.wmnet&r=week&z=small&jr=&js=&st=1436621739&v=3439901&m=Global%20JobQueue%20length&z=large [13:43:38] jzerebecki: Probably [13:43:54] Also arbitrary access needs an awful lot of jobs [13:44:03] for invalidating caches and stuff [13:44:04] jzerebecki: hoo: the number of pending changes still seems to be going up [13:44:20] * Lydia_WMDE has to leave now [13:44:27] I started an own dispatcher [13:44:29] hoo: so we have two bottlenecks creating jobs through dispatcher and jobs not actually getting run as fast as we have cpu [13:44:32] and it's doing crazy stuff [13:45:17] seems like most of these are restbase jobs [13:45:45] Lydia_WMDE: i read it as going down [13:47:26] oh ok you are right it is going up now [13:49:39] it seems the change rate is going up, this bot started: https://www.wikidata.org/wiki/Special:Contributions/InwBot [13:50:09] the bot is not the issue [13:50:33] our change dispatching is awful [13:51:18] yes, but that it is awful means we can not cope with a higher change rate, even though we have the resources [13:56:57] mh... maybe the problem is partly the damn low hit ratio for some wikis [13:57:34] makes the dispatcher run forever and some other dispatcher will invalidate its lock, assuming it no longer runs (because it takes more than 60s to find a couple of changes that actually need to be dispatched) [13:58:19] jzerebecki: My dispatcher just dispatched 2 changes to tnwiki out of several hundred thousands [13:58:50] ow :( [13:59:40] That's only a problem if there are to many hcanges for them to dispatch within 60s [14:07:27] 1 change to iswikiquote out of 804,000+ [14:07:32] * 840,000 [14:20:46] hoo: so we add a limit to the amount of changes filtered in one go? [14:21:06] I think we should [14:21:14] That could be 10 or 100 times the batch size [14:32:16] dispatch stats appear to be improving [14:34:03] * aude off until maybe later [14:42:55] hoo: i filed that as https://phabricator.wikimedia.org/T105592 but as it is progressing we'll not attempt that now, right? [14:44:14] Not needed now [14:44:21] thanks :) [17:38:22] Dispatch lag is down! :) [17:55:18] RECOVERY - check if wikidata.org dispatch lag is higher than 2 minutes on wikidata is OK: HTTP OK: HTTP/1.1 200 OK - 1410 bytes in 0.452 second response time