[12:19:24] PROBLEM - High lag on wdqs1003 is CRITICAL: 3609 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:07:39] PROBLEM - High lag on wdqs1007 is CRITICAL: 3626 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:10:29] PROBLEM - High lag on wdqs1007 is CRITICAL: 3603 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:20:27] PROBLEM - High lag on wdqs1008 is CRITICAL: 3606 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:20:49] mmm [15:20:54] that looks bad [15:26:09] not **too** bad, big flow of edits on wikidata, wdqs can't keep up with the updates [15:26:19] PROBLEM - High lag on wdqs2006 is CRITICAL: 3602 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:26:44] if anyone knows of a bot that is generating a lot of edits at the moment, now would be a good time to raise a hand! [15:26:52] ACKNOWLEDGEMENT - High lag on wdqs1008 is CRITICAL: 3731 ge 3600 Mathew.onipe This is a known problem occuring a - The acknowledgement expires at: 2019-06-03 23:23:11. https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:26:52] people could just respect the API etiquette [15:26:52] ACKNOWLEDGEMENT - High lag on wdqs2006 is CRITICAL: 3602 ge 3600 Mathew.onipe This is a known problem occuring a - The acknowledgement expires at: 2019-06-03 23:23:11. https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:27:01] gehel: http://wikidata.wikiscan.org/?menu=live&filter=all&sort=weight&date=6&list=users [15:27:42] sjoerddebruin: oh nice! I did not know about those metrics! [15:28:01] it's a very awesome site, but it is also having some troubles in the last few days it seems [15:29:44] also, all servers are lagging which is very weird: https://phabricator.wikimedia.org/T224829 [15:29:58] normally only three would leg [15:30:13] not all that weird [15:30:25] when there is a high edit rate, updates will start to lag [15:30:42] ACKNOWLEDGEMENT - High lag on wdqs1003 is CRITICAL: 7063 ge 3600 Mathew.onipe known issue but this time a bit different - The acknowledgement expires at: 2019-06-03 23:29:51. https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:30:42] ACKNOWLEDGEMENT - High lag on wdqs1007 is CRITICAL: 3952 ge 3600 Mathew.onipe known issue but this time a bit different - The acknowledgement expires at: 2019-06-03 23:29:51. https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:30:51] they usually start to lag on the public cluster, which has a higher load related to user queries, but at some point, other servers will also lag [20:04:53] RECOVERY - High lag on wdqs1007 is OK: (C)3600 ge (W)1200 ge 1137 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [20:06:45] RECOVERY - High lag on wdqs2006 is OK: (C)3600 ge (W)1200 ge 1163 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [20:19:11] RECOVERY - High lag on wdqs1008 is OK: (C)3600 ge (W)1200 ge 1069 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen