[06:05:54] I am going to switch x3 master [08:24:28] I see 2 warning from db2239 on i/o [08:34:33] federico3: Same thing as yesterday, please review this: [08:35:09] 08:51:06 those are replication sources running their backups [08:35:09] 08:51:46 you can see that by checking their port, it is different from 3306 and they also have notifications disabled too [08:35:09] 08:51:52 So that's a backup source [08:37:22] I'm aware of the port but I'm not sure about the timing i.e. should we always ignore any io warning from those hosts on port != 3306 ? [08:38:54] federico3: For db* hosts running on different than 3306 yes, we can ignore as long as replication isn't broken: eg DUPLICATE ENTRY KEY blablabla [15:55:57] JennH: Do you want me to prepare pc2016 now and you can do it during your day? [15:59:30] marostegui: that works for me. [16:01:47] JennH: I will do it now and post on the task once it is ready, thanks you [16:04:59] np! [16:06:34] JennH: Posted on the task, pc2016 is now off and ready for you. I ran homer too [16:06:49] awesome ty! [22:56:05] FIRING: MysqlReplicationThreadCountTooLow: MySQL instance db1229:9104 has replication issues. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1229&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationThreadCountTooLow [22:56:48] FIRING: MysqlReplicationLagPtHeartbeat: MySQL instance db1229:9104 has too large replication lag (6h 1m 10s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1229&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [23:03:25] this is https://phabricator.wikimedia.org/T399249 [23:03:34] expired downtime [23:04:33] ^ extended the downtime by another 12h since this paged (https://sal.toolforge.org/log/jy-gGpgB8tZ8Ohr0zxPk)