[00:17:19] 10Operations, 10ops-eqiad, 10Data-Services, 10decommission: Decommission labsdb1006.eqiad.wmnet and labsdb1007.eqiad.wmnet - https://phabricator.wikimedia.org/T220144 (10wiki_willy) a:03Jclark-ctr [00:37:02] (03PS1) 10BryanDavis: toolforge: Update TOU link in exim warnings [puppet] - 10https://gerrit.wikimedia.org/r/595246 [00:45:15] RECOVERY - Check systemd state on an-launcher1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:50:49] PROBLEM - Check systemd state on an-launcher1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [02:29:38] (03PS3) 10Andrew Bogott: OpenStack: move all openstack API support to cloudcontrol1005 [puppet] - 10https://gerrit.wikimedia.org/r/595227 (https://phabricator.wikimedia.org/T252121) [02:29:40] (03PS1) 10Andrew Bogott: Define keystone_api_fqdn for codfw1dev instances [puppet] - 10https://gerrit.wikimedia.org/r/595249 [02:30:34] (03CR) 10Andrew Bogott: [C: 03+2] Define keystone_api_fqdn for codfw1dev instances [puppet] - 10https://gerrit.wikimedia.org/r/595249 (owner: 10Andrew Bogott) [02:55:55] PROBLEM - Varnish traffic drop between 30min ago and now at eqiad on icinga1001 is CRITICAL: 52.53 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [03:01:29] RECOVERY - Varnish traffic drop between 30min ago and now at eqiad on icinga1001 is OK: (C)60 le (W)70 le 93.89 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [05:00:19] RECOVERY - Check systemd state on labstore1007 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [05:01:03] RECOVERY - Check systemd state on labstore1006 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [07:00:04] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200509T0700) [09:50:31] (03CR) 10Daniel Kinzler: "> This will drop the following columns from the revision view:" [puppet] - 10https://gerrit.wikimedia.org/r/595201 (https://phabricator.wikimedia.org/T252219) (owner: 10Bstorm) [11:08:42] 10Operations, 10SRE-Access-Requests: Access to analytics-privatedata-users for Research intern - https://phabricator.wikimedia.org/T252129 (10Daniram3) [1] My username is Daniram3, and my shell account name is daniram. [2 - 4] Done. [5] I am not able to access Office Wiki. [6] To do. [11:47:02] 10Operations, 10SRE-Access-Requests: Access to analytics-privatedata-users for Research intern - https://phabricator.wikimedia.org/T252129 (10Krenair) I don't think `bastiononly` has existed for years. [11:53:33] 10Operations, 10SRE-Access-Requests: Access to analytics-privatedata-users for Research intern - https://phabricator.wikimedia.org/T252129 (10Miriam) >>! In T252129#6121161, @Krenair wrote: > I don't think `bastiononly` has existed for years. Sorry my bad, I copied from an old task! [11:53:43] 10Operations, 10SRE-Access-Requests: Access to analytics-privatedata-users for Research intern - https://phabricator.wikimedia.org/T252129 (10Miriam) [16:44:57] (03PS1) 10Andrew Bogott: Change cloudcontrol2001 and 2003 to buster [puppet] - 10https://gerrit.wikimedia.org/r/595281 (https://phabricator.wikimedia.org/T252121) [16:46:50] (03PS1) 10Andrew Bogott: Move cloudweb api service to cloudcontrol2004 [puppet] - 10https://gerrit.wikimedia.org/r/595282 (https://phabricator.wikimedia.org/T252121) [16:47:44] (03CR) 10Andrew Bogott: [C: 03+2] Change cloudcontrol2001 and 2003 to buster [puppet] - 10https://gerrit.wikimedia.org/r/595281 (https://phabricator.wikimedia.org/T252121) (owner: 10Andrew Bogott) [16:47:56] (03CR) 10Andrew Bogott: [C: 03+2] Move cloudweb api service to cloudcontrol2004 [puppet] - 10https://gerrit.wikimedia.org/r/595282 (https://phabricator.wikimedia.org/T252121) (owner: 10Andrew Bogott) [18:00:49] PROBLEM - HTTPS-dbtree on dbmonitor1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dbtree.wikimedia.org [18:04:27] RECOVERY - HTTPS-dbtree on dbmonitor1001 is OK: HTTP OK: HTTP/1.1 200 OK - 85088 bytes in 9.106 second response time https://wikitech.wikimedia.org/wiki/Dbtree.wikimedia.org [18:24:49] PROBLEM - HTTPS-dbtree on dbmonitor1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dbtree.wikimedia.org [18:30:19] RECOVERY - HTTPS-dbtree on dbmonitor1001 is OK: HTTP OK: HTTP/1.1 200 OK - 85467 bytes in 8.924 second response time https://wikitech.wikimedia.org/wiki/Dbtree.wikimedia.org [18:35:53] PROBLEM - HTTPS-dbtree on dbmonitor1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dbtree.wikimedia.org [18:43:23] RECOVERY - HTTPS-dbtree on dbmonitor1001 is OK: HTTP OK: HTTP/1.1 200 OK - 85560 bytes in 9.687 second response time https://wikitech.wikimedia.org/wiki/Dbtree.wikimedia.org [18:54:37] PROBLEM - HTTPS-dbtree on dbmonitor1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dbtree.wikimedia.org [19:03:53] RECOVERY - HTTPS-dbtree on dbmonitor1001 is OK: HTTP OK: HTTP/1.1 200 OK - 85565 bytes in 9.664 second response time https://wikitech.wikimedia.org/wiki/Dbtree.wikimedia.org [19:13:11] PROBLEM - HTTPS-dbtree on dbmonitor1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dbtree.wikimedia.org