[01:56:24] ACKNOWLEDGEMENT - MD RAID on wdqs2007 is CRITICAL: CRITICAL: State: degraded, Active: 7, Working: 7, Failed: 0, Spare: 0 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T281956 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Hardware_Raid_Information_Gathering [03:05:07] PROBLEM - WDQS high update lag on wdqs1011 is CRITICAL: 4428 ge 3600 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [03:05:19] RECOVERY - WDQS SPARQL on wdqs1011 is OK: HTTP OK: HTTP/1.1 200 OK - 689 bytes in 1.068 second response time https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [03:08:09] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs2001 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [03:08:11] PROBLEM - WDQS high update lag on wdqs2001 is CRITICAL: 1.08e+05 ge 4.32e+04 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [03:09:11] PROBLEM - Blazegraph process -wdqs-categories- on wdqs2007 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 499 (blazegraph), regex args ^java .* --port 9990 .* blazegraph-service-.*war https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [03:09:21] PROBLEM - Blazegraph Port for wdqs-categories on wdqs2007 is CRITICAL: connect to address 127.0.0.1 and port 9990: Connection refused https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [03:09:23] RECOVERY - WDQS SPARQL on wdqs2001 is OK: HTTP OK: HTTP/1.1 200 OK - 689 bytes in 1.200 second response time https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [03:09:27] RECOVERY - Check systemd state on wdqs2001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [03:09:33] PROBLEM - puppet last run on wdqs2007 is CRITICAL: CRITICAL: Puppet last ran 1 day ago https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [03:09:47] PROBLEM - Check systemd state on wdqs2007 is CRITICAL: CRITICAL - degraded: The following units failed: prometheus-blazegraph-exporter-wdqs-blazegraph.service,prometheus-blazegraph-exporter-wdqs-categories.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [03:09:47] RECOVERY - WDQS SPARQL on wdqs2007 is OK: HTTP OK: HTTP/1.1 200 OK - 689 bytes in 1.200 second response time https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [03:09:55] RECOVERY - Blazegraph process -wdqs-blazegraph- on wdqs2001 is OK: PROCS OK: 1 process with UID = 499 (blazegraph), regex args ^java .* --port 9999 .* blazegraph-service-.*war https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [03:10:03] RECOVERY - Query Service HTTP Port on wdqs2001 is OK: HTTP OK: HTTP/1.1 200 OK - 448 bytes in 0.143 second response time https://wikitech.wikimedia.org/wiki/Wikidata_query_service [03:11:31] PROBLEM - puppet last run on wdqs2001 is CRITICAL: CRITICAL: Puppet last ran 1 day ago https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [03:17:07] PROBLEM - WDQS SPARQL on wdqs1013 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [03:18:05] RECOVERY - puppet last run on wdqs2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [03:21:57] RECOVERY - Blazegraph process -wdqs-categories- on wdqs2007 is OK: PROCS OK: 1 process with UID = 499 (blazegraph), regex args ^java .* --port 9990 .* blazegraph-service-.*war https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [03:22:01] PROBLEM - WDQS high update lag on wdqs2007 is CRITICAL: 1.077e+05 ge 4.32e+04 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [03:22:05] RECOVERY - Blazegraph Port for wdqs-categories on wdqs2007 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9990 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [03:22:37] RECOVERY - puppet last run on wdqs2007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [03:38:05] RECOVERY - WDQS high update lag on wdqs1011 is OK: (C)3600 ge (W)1200 ge 1002 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [03:50:31] RECOVERY - Check systemd state on wdqs2007 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [10:37:41] RECOVERY - WDQS SPARQL on wdqs1013 is OK: HTTP OK: HTTP/1.1 200 OK - 689 bytes in 1.072 second response time https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [16:30:43] When I go to a wikidata page, there's a section call "Identifiers", this contains links to various websites. When I query the API I get the id part of the websites, but doesn't look like I get the full URLs like exists on the wikidata profile pages. How can I get the Identifiers for an entity including full links from the API? [17:48:32] RECOVERY - WDQS high update lag on wdqs2007 is OK: (C)4.32e+04 ge (W)2.16e+04 ge 2.148e+04 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [18:45:56] Hello! I'm having issues with HTTP ERROR 429: Too many Requests. It says to ask for more permissions, but I dont know where to ask for them. I've been googling and it says to just make the script wait until it's available again, but the "Retry-After" header isn't present. The header response is as follows: https://pastebin.com/F9UKfSDn [18:46:24] Btw, I'm using Python if it helps, with the requests module [18:46:59] Anyone has dealt with it before or has any clue about dealing with it? thank you in advance [19:13:01] Destokado: is every request blocked [19:13:30] No, the first ones perform good, its just after a while [19:14:01] Probably after the 60s limit https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual#Query_limits [19:16:52] RhinosF1: The 5 first ones went good, but after that just 429 error, then after some time a single one goes trhough and then error 429 again [19:22:18] Destokado: If it's a hard limit then time.sleep is by best recommendation. Not sure why retry after ain't there so maybe try a few seconds and exponentially back off each time you hit it [20:20:54] RhinosF1: It's strange, because it stop working properly after 1.7 seconds (8 requests this time), but the error code is the same and the headers are the same as i posted in the pastebin. Do you have any idea? May I be capped or something? [20:21:36] That's strange [20:22:18] I'd say given the missing header a phab task wouldn't be a bad idea [20:23:27] Destokado: it requires good UA [20:23:34] do you have a good UA set? [20:23:59] No, i'm just using the default (Python= [20:24:04] https://meta.wikimedia.org/wiki/User-Agent_policy [20:24:12] I thought UA but you said it doesn't fail first time [20:24:16] Which is weird [20:24:43] can't say it definitely the reason but it's better to have [20:25:25] True true [20:26:38] Hmmm i'm gonna try, but some weeks ago it worked way better (less errors). And if i execute it locally (it now runs on a server) it doesn't happen [21:06:06] Thank you so much guys, It was the User-Agent, now I've set it properly and it works just fine. Much WikiLove for you Rhinos [21:06:14] RhinosF1, Amir1 [21:06:48] I just know which people give good advice [21:07:05] He bribes me [21:15:01] I must give good bribes [21:16:06] In real life I'm normally the one with the bag of stuff. Less of a clue on wiki so I just know people. [21:16:44] By now I've just started in real life saying it's in my bag, just get it [21:20:49] Hahaha that's a very useful resource! I too know a lot of people, unfortunately (or not) they are not techies hehe [21:24:13] Knowing people is very good [21:24:22] Being the one to be known can be a curse and blessing [21:25:51] My bag has as much stuff in for other people then it does me [22:46:15] RhinosF1: we should compare bags the next time we are in the same place IRL :) For a hackathon especially I usually have a bunch of cables and plug converters that I don't need but guess that someone will stuck in with the stickers and power banks and N electronic devices that I always have. [23:27:01] PROBLEM - Host wdqs2007 is DOWN: PING CRITICAL - Packet loss = 100% [23:33:25] RECOVERY - Host wdqs2007 is UP: PING OK - Packet loss = 0%, RTA = 31.73 ms [23:39:37] PROBLEM - Check systemd state on wdqs2007 is CRITICAL: CRITICAL - degraded: The following units failed: prometheus-blazegraph-exporter-wdqs-categories.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [23:41:42] ACKNOWLEDGEMENT - MD RAID on wdqs2007 is CRITICAL: CRITICAL: State: degraded, Active: 7, Working: 7, Failed: 0, Spare: 0 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T282068 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Hardware_Raid_Information_Gathering [23:50:05] RECOVERY - Check systemd state on wdqs2007 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state