[00:38:26] a new user (User:CarexOntology) has created nearly a hundred mostly-empty items like "left pupil" and "bottom corner". how best to deal with this? [01:10:38] PROBLEM - SSH on wdqs1010 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:08] PROBLEM - SSH on wdqs1008 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:19] PROBLEM - SSH on wdqs2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:19] PROBLEM - SSH on wdqs2006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:09] PROBLEM - SSH on wdqs1009 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:28] PROBLEM - SSH on wdqs1007 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:39] PROBLEM - SSH on wdqs2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:59] PROBLEM - SSH on wdqs1005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:14:19] PROBLEM - SSH on wdqs2004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:14:48] PROBLEM - SSH on wdqs2005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:28] PROBLEM - SSH on wdqs1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:49] PROBLEM - SSH on wdqs1006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:24:09] PROBLEM - Host wdqs2005 is DOWN: PING CRITICAL - Packet loss = 100% [01:24:28] RECOVERY - Host wdqs2005 is UP: PING OK - Packet loss = 0%, RTA = 36.15 ms [01:24:58] RECOVERY - SSH on wdqs2005 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u3 (protocol 2.0) [01:28:38] PROBLEM - High lag on wdqs1005 is CRITICAL: 3608 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [01:28:39] PROBLEM - High lag on wdqs1003 is CRITICAL: 3606 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [01:28:49] PROBLEM - High lag on wdqs2004 is CRITICAL: 3601 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [01:29:09] PROBLEM - High lag on wdqs1008 is CRITICAL: 3640 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [01:29:09] PROBLEM - High lag on wdqs2001 is CRITICAL: 3623 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [01:29:18] PROBLEM - High lag on wdqs1006 is CRITICAL: 3647 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [01:29:19] PROBLEM - High lag on wdqs2002 is CRITICAL: 3632 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [01:29:28] PROBLEM - High lag on wdqs1009 is CRITICAL: 3656 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [01:29:28] PROBLEM - High lag on wdqs2005 is CRITICAL: 3637 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [01:29:29] PROBLEM - High lag on wdqs1004 is CRITICAL: 3658 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [01:29:30] PROBLEM - High lag on wdqs1010 is CRITICAL: 3663 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [01:29:38] PROBLEM - High lag on wdqs1007 is CRITICAL: 3670 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [01:29:48] PROBLEM - High lag on wdqs2006 is CRITICAL: 3658 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [01:53:39] PROBLEM - Check systemd state on wdqs1007 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [01:53:59] PROBLEM - Check systemd state on wdqs1009 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [01:58:18] PROBLEM - SSH on wdqs2005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:19] PROBLEM - Check systemd state on wdqs1007 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [02:17:39] PROBLEM - Check systemd state on wdqs1009 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [02:25:19] PROBLEM - Check systemd state on wdqs1007 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [02:25:39] PROBLEM - Check systemd state on wdqs1009 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [02:32:09] PROBLEM - Check systemd state on wdqs1007 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [02:35:58] PROBLEM - Host wdqs1009 is DOWN: PING CRITICAL - Packet loss = 100% [02:36:39] RECOVERY - Host wdqs1009 is UP: PING OK - Packet loss = 0%, RTA = 0.21 ms [02:36:59] RECOVERY - SSH on wdqs1009 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0) [02:36:59] RECOVERY - Check systemd state on wdqs1009 is OK: OK - running: The system is fully operational [02:42:18] PROBLEM - High lag on wdqs1009 is CRITICAL: 8026 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [02:53:48] PROBLEM - Check systemd state on wdqs1007 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [02:58:09] PROBLEM - Check systemd state on wdqs2006 is CRITICAL: CRITICAL - Failed to connect to bus: Resource temporarily unavailable: unexpected [03:03:49] PROBLEM - Check systemd state on wdqs1005 is CRITICAL: CRITICAL - Failed to connect to bus: Resource temporarily unavailable: unexpected [03:04:59] PROBLEM - Check systemd state on wdqs1007 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [03:08:39] PROBLEM - SSH on wdqs1009 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:46:48] RECOVERY - Check systemd state on wdqs1007 is OK: OK - running: The system is fully operational [03:55:29] RECOVERY - Check systemd state on wdqs1005 is OK: OK - running: The system is fully operational [04:00:09] PROBLEM - Check systemd state on wdqs2004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [04:10:09] RECOVERY - Check systemd state on wdqs2006 is OK: OK - running: The system is fully operational [04:14:58] PROBLEM - Check systemd state on wdqs2004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [04:26:19] PROBLEM - Check systemd state on wdqs2004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [04:44:19] PROBLEM - Check systemd state on wdqs2004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [04:56:39] PROBLEM - Check systemd state on wdqs2004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [05:14:38] PROBLEM - Check systemd state on wdqs2004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [05:26:59] PROBLEM - Check systemd state on wdqs2004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [05:44:48] PROBLEM - Check systemd state on wdqs2004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [05:50:09] PROBLEM - Check systemd state on wdqs1003 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [05:55:58] PROBLEM - Check systemd state on wdqs2004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [05:58:08] RECOVERY - Check systemd state on wdqs1003 is OK: OK - running: The system is fully operational [06:14:49] PROBLEM - Check systemd state on wdqs2004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [06:27:09] PROBLEM - Check systemd state on wdqs2004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [06:33:18] PROBLEM - Check systemd state on wdqs2003 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [06:33:29] PROBLEM - Check systemd state on wdqs1003 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [06:36:18] PROBLEM - Host wdqs1004 is DOWN: PING CRITICAL - Packet loss = 100% [06:37:08] RECOVERY - Host wdqs1004 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [06:37:09] RECOVERY - SSH on wdqs1004 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0) [06:40:18] RECOVERY - SSH on wdqs2001 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0) [06:45:09] PROBLEM - Check systemd state on wdqs2004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [06:47:08] PROBLEM - Host wdqs2002 is DOWN: PING CRITICAL - Packet loss = 100% [06:48:08] RECOVERY - Host wdqs2002 is UP: PING OK - Packet loss = 0%, RTA = 36.16 ms [06:48:18] RECOVERY - SSH on wdqs2002 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0) [06:49:38] PROBLEM - Host wdqs1005 is DOWN: PING CRITICAL - Packet loss = 100% [06:49:49] RECOVERY - SSH on wdqs1005 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0) [06:49:58] RECOVERY - Host wdqs1005 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [06:53:48] RECOVERY - Check systemd state on wdqs2003 is OK: OK - running: The system is fully operational [06:54:59] RECOVERY - Check systemd state on wdqs1003 is OK: OK - running: The system is fully operational [06:55:29] RECOVERY - Check systemd state on wdqs2004 is OK: OK - running: The system is fully operational [06:55:39] RECOVERY - SSH on wdqs2004 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u3 (protocol 2.0) [06:59:08] PROBLEM - Host wdqs1006 is DOWN: PING CRITICAL - Packet loss = 100% [06:59:28] RECOVERY - Host wdqs1006 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [06:59:29] RECOVERY - SSH on wdqs1006 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u3 (protocol 2.0) [07:00:09] RECOVERY - SSH on wdqs2005 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u3 (protocol 2.0) [07:02:58] PROBLEM - Host wdqs1007 is DOWN: PING CRITICAL - Packet loss = 100% [07:03:09] RECOVERY - SSH on wdqs1007 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u3 (protocol 2.0) [07:03:19] RECOVERY - Host wdqs1007 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [07:06:39] RECOVERY - SSH on wdqs2006 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u3 (protocol 2.0) [07:07:29] RECOVERY - SSH on wdqs1008 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u3 (protocol 2.0) [07:20:58] PROBLEM - Host wdqs1009 is DOWN: PING CRITICAL - Packet loss = 100% [07:21:19] RECOVERY - Host wdqs1009 is UP: PING OK - Packet loss = 0%, RTA = 1.02 ms [07:21:48] RECOVERY - SSH on wdqs1009 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0) [07:34:39] PROBLEM - Host wdqs1010 is DOWN: PING CRITICAL - Packet loss = 100% [07:35:18] RECOVERY - SSH on wdqs1010 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0) [07:35:19] RECOVERY - Host wdqs1010 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [11:32:45] What's up with the query service? :| https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen&from=now-24h&to=now [11:33:19] addshore: https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1 is set on 10 till 12 october for some reason [11:43:24] sjoerddebruin: 1. T207817, I think [11:43:25] T207817: WDQS Updater ran into issue and stopped working - https://phabricator.wikimedia.org/T207817 [11:43:35] 2. do you remember what it was set to before? [11:43:53] I would expect last 24 hours but I'm not sure [11:44:47] okay, I’ve set it to that now [11:45:00] better than the old default at least [11:59:06] thanks and thanks [14:01:17] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @Thiemo_WMDE & @Tim_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:50:58] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @Thiemo_WMDE & @Tim_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:45:47] RECOVERY - High lag on wdqs1006 is OK: (C)3600 ge (W)1200 ge 1151 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [15:53:46] RECOVERY - High lag on wdqs1007 is OK: (C)3600 ge (W)1200 ge 1119 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [16:05:18] RECOVERY - High lag on wdqs1008 is OK: (C)3600 ge (W)1200 ge 1128 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [16:23:27] RECOVERY - High lag on wdqs1009 is OK: (C)3600 ge (W)1200 ge 1172 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [16:31:58] RECOVERY - High lag on wdqs1010 is OK: (C)3600 ge (W)1200 ge 1143 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [17:11:40] duesen_: remember the other day I was asking about networking problems? do you have any idea what it could be if traceroute works, but ssh still says no route to host and browsers say the address is unreachable? [17:15:21] been trying to contact the isp but they're very elusive >_< [19:40:45] RECOVERY - High lag on wdqs2004 is OK: (C)3600 ge (W)1200 ge 1182 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [19:41:53] RECOVERY - High lag on wdqs2005 is OK: (C)3600 ge (W)1200 ge 1189 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [20:29:15] RECOVERY - High lag on wdqs2006 is OK: (C)3600 ge (W)1200 ge 1190 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [22:15:54] RECOVERY - High lag on wdqs2001 is OK: (C)3600 ge (W)1200 ge 1197 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [22:23:13] RECOVERY - High lag on wdqs2002 is OK: (C)3600 ge (W)1200 ge 1179 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen