[00:02:10] 10Puppet, 10Cloud-VPS, 10serviceops, 10Patch-For-Review: upgrade simplelamp class (apache -> httpd and mysql -> mariadb) or deprecate it - https://phabricator.wikimedia.org/T215662 (10Dzahn) @Slevinski One way we could do this is to create a new instance where all software versions are newer and we can con... [00:04:27] Reedy: Thrown together https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/489337 but untested and I wonder what other surprise changes will happen if MW users run that after a decade of changes… [00:04:53] We should probably be running it with OS upgrades too tbh [00:05:02] Oh, good point. [00:05:07] Improvements in exif libraries etc result in changes that might not get surfaced [00:05:11] At WMF Towers, not just for third parties. [00:05:14] * James_F nods. [00:06:12] I wonder if in MW releases we put it into every LTS or something [00:06:27] Or just every MW release? [00:06:47] So for getUpdateKey I'd return __CLASS__.$wgVersion. [00:06:49] We'd have to make sure we didn't end up with it doing every point release (unless we knew we fixed a bug in a point release) [00:07:00] Cause that would get annoying, and is probably pointless [00:07:00] Hmm, yeah. :-( [00:07:12] also, $wgVersion is going away ;P [00:07:15] And it's going to be very slow for people with lots of files. [00:07:26] Yeah yeah, WG_VERSION or whatever we call it. [00:07:29] We all know the media part of mediawiki is a lie [00:07:37] But how many people actually do store a lot of files? [00:07:38] And the 'wiki' part, in some ways. [00:07:51] I guess we could ask CindyCicaleseWMF who might know. :-) [00:08:34] https://wikistats.wmflabs.org/ has files [00:09:52] Something to think about, we don't need to fix it *now* obvs [00:09:55] OK, so ignoring WMF (no update.php), Wikia (no… update of MW), it gets pretty small (fewer than 100k files) pretty much immediately. [00:09:57] Yeah. [00:24:58] (03PS1) 10Nuria: Add ability for superset to connect to new staging DB [puppet] - 10https://gerrit.wikimedia.org/r/489338 (https://phabricator.wikimedia.org/T215680) [00:25:31] (03CR) 10jerkins-bot: [V: 04-1] Add ability for superset to connect to new staging DB [puppet] - 10https://gerrit.wikimedia.org/r/489338 (https://phabricator.wikimedia.org/T215680) (owner: 10Nuria) [00:25:49] (03PS1) 10Dzahn: create a role to replace role(simplelamp) [puppet] - 10https://gerrit.wikimedia.org/r/489339 (https://phabricator.wikimedia.org/T2156662) [00:26:28] (03CR) 10jerkins-bot: [V: 04-1] create a role to replace role(simplelamp) [puppet] - 10https://gerrit.wikimedia.org/r/489339 (https://phabricator.wikimedia.org/T2156662) (owner: 10Dzahn) [00:27:08] James_F: Reedy: not sure of the full context of the question, but few third parties that I have encountered store "a lot" of files. They may upload large files, but not many files. I can't speak for the entire community, but I have encountered people who considered using MediaWiki for a large scale file repository but decided against it for a few reasons: no hierarchical storage, increasing the likelihood of filename [00:27:08] clashes, no per file access control, etc.. As we all know, it wasn't really designed with that in mind. For those purposes, most link to files in other repos such as SharePoint. Most third party wikis are (much) smaller than Wikimedia wikis, so I would expect them to have proportionally fewer files. This does not go for all, however. Does that help? [00:30:16] (03Abandoned) 10Nuria: Adding mailto to camus job [puppet] - 10https://gerrit.wikimedia.org/r/362237 (https://phabricator.wikimedia.org/T169248) (owner: 10Nuria) [00:34:13] (03PS2) 10Dzahn: create a role to replace role(simplelamp), son_of_simplelamp or HMMP [puppet] - 10https://gerrit.wikimedia.org/r/489339 (https://phabricator.wikimedia.org/T2156662) [00:34:45] PROBLEM - MariaDB Slave Lag: s4 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 611.53 seconds [00:34:46] (03CR) 10jerkins-bot: [V: 04-1] create a role to replace role(simplelamp), son_of_simplelamp or HMMP [puppet] - 10https://gerrit.wikimedia.org/r/489339 (https://phabricator.wikimedia.org/T2156662) (owner: 10Dzahn) [00:40:05] (03PS3) 10Dzahn: create a role to replace role(simplelamp), son_of_simplelamp or HMMP [puppet] - 10https://gerrit.wikimedia.org/r/489339 (https://phabricator.wikimedia.org/T2156662) [00:40:43] (03CR) 10Bstorm: wiki replicas: Remove reference to old comment fields (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/489242 (https://phabricator.wikimedia.org/T212972) (owner: 10Anomie) [00:44:18] CindyCicaleseWMF: Sounds reasonable. Context is adding a moderately expensive file metadata refresh step to update.php, because MW (and underlying tools e.g. OS stuff) change and improve MIME detection, EXIF extraction, etc. over time. [00:44:24] (03CR) 10Dzahn: "@paladox if you wanna try this via your local puppetmaster let me know if it works on a fresh instance?:)" [puppet] - 10https://gerrit.wikimedia.org/r/489339 (https://phabricator.wikimedia.org/T2156662) (owner: 10Dzahn) [00:46:43] (03PS4) 10Dzahn: create a role to replace role(simplelamp), son_of_simplelamp or HMMP [puppet] - 10https://gerrit.wikimedia.org/r/489339 (https://phabricator.wikimedia.org/T2156662) [00:47:33] (03PS5) 10Dzahn: create a role to replace role(simplelamp), son_of_simplelamp or HMMP [puppet] - 10https://gerrit.wikimedia.org/r/489339 (https://phabricator.wikimedia.org/T2156662) [00:52:10] Reedy: Did the operation complete? [00:52:38] Nope, still got a looong way to go [00:53:27] (03PS6) 10Dzahn: create a role to replace role(simplelamp), son_of_simplelamp or HMMP [puppet] - 10https://gerrit.wikimedia.org/r/489339 (https://phabricator.wikimedia.org/T215662) [00:54:33] (03PS17) 10Jbond: Improve CI checks to ensure a basic catalogue compiles on all supported OS's [puppet] - 10https://gerrit.wikimedia.org/r/487882 (https://phabricator.wikimedia.org/T215275) [00:55:17] 10Puppet, 10Cloud-VPS, 10serviceops, 10Patch-For-Review: upgrade simplelamp class (apache -> httpd and mysql -> mariadb) or deprecate it - https://phabricator.wikimedia.org/T215662 (10Dzahn) I started making a new class to replace the current one without having to change it directly. This is to allow proj... [00:55:51] (03PS18) 10Jbond: Improve CI checks to ensure a basic catalogue compiles on all supported OS's [puppet] - 10https://gerrit.wikimedia.org/r/487882 (https://phabricator.wikimedia.org/T215275) [00:56:43] RECOVERY - MariaDB Slave Lag: s4 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 267.10 seconds [01:02:50] 10Puppet, 10Cloud-VPS, 10serviceops, 10Patch-For-Review: upgrade simplelamp class (apache -> httpd and mysql -> mariadb) or deprecate it - https://phabricator.wikimedia.org/T215662 (10Dzahn) [01:03:22] 10Puppet, 10Cloud-VPS, 10serviceops, 10Patch-For-Review, 10Technical-Debt: upgrade simplelamp class (apache -> httpd and mysql -> mariadb) or deprecate it - https://phabricator.wikimedia.org/T215662 (10Dzahn) [01:05:28] ACKNOWLEDGEMENT - Check systemd state on cloudcontrol1004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. GTirloni https://phabricator.wikimedia.org/T215544 [01:09:12] (03CR) 10Dzahn: "to be fair we are not 100% sure if we are actually already using the new config just that both are in the config and it works. not against" [puppet] - 10https://gerrit.wikimedia.org/r/489121 (https://phabricator.wikimedia.org/T212989) (owner: 10Paladox) [01:09:58] 10Operations, 10Traffic, 10Wikimedia-Incident: Puppet doesn't restart ferm on failure - https://phabricator.wikimedia.org/T206951 (10GTirloni) I don't know if this is related but today I noticed that, if iptables rules are cleared (`iptables -F`), subsequent puppet runs will not re-apply them. I also had to... [01:20:38] (03PS1) 10Dzahn: tor: add data types to parameters [puppet] - 10https://gerrit.wikimedia.org/r/489347 [01:21:12] (03CR) 10jerkins-bot: [V: 04-1] tor: add data types to parameters [puppet] - 10https://gerrit.wikimedia.org/r/489347 (owner: 10Dzahn) [01:22:17] (03PS2) 10Dzahn: tor: add data types to parameters [puppet] - 10https://gerrit.wikimedia.org/r/489347 [02:08:11] (03PS1) 10Paladox: gerrit: Remove css for iron-icons [puppet] - 10https://gerrit.wikimedia.org/r/489363 [02:08:53] (03PS2) 10Paladox: gerrit: Remove css for iron-icons [puppet] - 10https://gerrit.wikimedia.org/r/489363 [03:13:52] PROBLEM - Host ganeti2004.mgmt is DOWN: PING CRITICAL - Packet loss = 100% [03:16:42] RECOVERY - Host ganeti2004.mgmt is UP: PING OK - Packet loss = 0%, RTA = 36.97 ms [05:51:48] PROBLEM - Maps tiles generation on icinga1001 is CRITICAL: CRITICAL: 100.00% of data under the critical threshold [5.0] https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=8&fullscreen&orgId=1 [06:33:26] PROBLEM - puppet last run on analytics1048 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/R/biocLite.R] [06:34:32] PROBLEM - puppet last run on labtestvirt2001 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[tzdata],Package[ntfs-3g] [06:59:50] RECOVERY - puppet last run on analytics1048 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [07:00:54] RECOVERY - puppet last run on labtestvirt2001 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [07:05:57] (03CR) 10Dzahn: "> Patch Set 2:" [puppet] - 10https://gerrit.wikimedia.org/r/487917 (https://phabricator.wikimedia.org/T201491) (owner: 10Dzahn) [11:03:41] 10Operations, 10Advanced Mobile Contributions, 10Traffic, 10User-Joe: AMC – Opt-in for logged out users - https://phabricator.wikimedia.org/T215624 (10Joe) [11:55:29] (03PS2) 10Urbanecm: New throttle rule + removal of expired rules [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489207 (https://phabricator.wikimedia.org/T215446) [11:56:30] (03CR) 10jerkins-bot: [V: 04-1] New throttle rule + removal of expired rules [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489207 (https://phabricator.wikimedia.org/T215446) (owner: 10Urbanecm) [11:57:10] (03PS3) 10Urbanecm: New throttle rule + removal of expired rules [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489207 (https://phabricator.wikimedia.org/T215446) [11:57:31] a kind deployer available to deploy a throttle rule (https://gerrit.wikimedia.org/r/489207 / T215446)? [11:57:32] T215446: Request for temporary lift of account creation cap for Wikipedia edit-a-thon event (Feb 9) - https://phabricator.wikimedia.org/T215446 [12:04:41] PROBLEM - MariaDB Slave Lag: s4 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 616.51 seconds [12:10:30] dereckson, ? [12:24:15] PROBLEM - MariaDB Slave Lag: s4 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 601.31 seconds [12:37:09] PROBLEM - MariaDB Slave Lag: s4 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 651.27 seconds [13:07:43] PROBLEM - MariaDB Slave Lag: s4 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 607.06 seconds [13:15:27] PROBLEM - MariaDB Slave Lag: s4 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 619.33 seconds [13:43:06] (03CR) 10Elukey: Add ability for superset to connect to new staging DB (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/489338 (https://phabricator.wikimedia.org/T215680) (owner: 10Nuria) [13:57:13] RECOVERY - MariaDB Slave Lag: s4 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 295.45 seconds [14:40:03] 10Operations, 10serviceops, 10Performance-Team (Radar), 10User-Elukey, 10User-jijiki: Upgrade memcached for Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10elukey) Found https://github.com/memcached/memcached/issues/446 today, in which upstream warns about stability issues with 1.5.6... [14:41:34] 10Operations, 10MediaWiki-Cache, 10serviceops, 10Patch-For-Review, and 3 others: Apply -R 200 to all the memcached mw object cache instances running in eqiad/codfw - https://phabricator.wikimedia.org/T208844 (10elukey) a:05elukey→03jijiki [15:26:13] (03PS8) 10Zoranzoki21: Set wgRestrictionLevels for all Serbian projects to all groups (autopatrol, patroller, rollbacker, bot, sysop, bureaucrat) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/485903 (https://phabricator.wikimedia.org/T215653) [15:26:22] (03PS9) 10Zoranzoki21: Set wgRestrictionLevels for all Serbian projects to all groups (autopatrol, patroller, rollbacker, bot, sysop, bureaucrat) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/485903 (https://phabricator.wikimedia.org/T215653) [15:44:25] PROBLEM - MariaDB Slave Lag: s4 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 617.38 seconds [16:51:36] !log restarted icinga process on icinga1001 because of passive check alert-storm [16:51:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:02:05] 10Operations, 10Icinga, 10monitoring: Icinga passive checks go awal and downtime stops working - https://phabricator.wikimedia.org/T196336 (10Jgreen) Passive checks went awol again today (2019-02-09) starting around 16:25 UTC. I watched it for a few minutes and observed, on icinga1001 - loadavg overall on t... [18:29:36] PROBLEM - Check systemd state on ms-be2015 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [18:34:48] RECOVERY - Check systemd state on ms-be2015 is OK: OK - running: The system is fully operational [18:39:52] PROBLEM - MariaDB Slave Lag: s7 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 627.60 seconds [19:08:31] (03PS1) 10GTirloni: toolforge::clush::master - Convert cronjob to systemd timer [puppet] - 10https://gerrit.wikimedia.org/r/489393 (https://phabricator.wikimedia.org/T210818) [19:09:40] (03CR) 10jerkins-bot: [V: 04-1] toolforge::clush::master - Convert cronjob to systemd timer [puppet] - 10https://gerrit.wikimedia.org/r/489393 (https://phabricator.wikimedia.org/T210818) (owner: 10GTirloni) [19:19:12] (03PS1) 10GTirloni: wmcs::monitoring - Convert cronjob to systemd timer [puppet] - 10https://gerrit.wikimedia.org/r/489394 (https://phabricator.wikimedia.org/T210818) [19:20:08] (03PS2) 10GTirloni: toolforge::clush::master - Convert cronjob to systemd timer [puppet] - 10https://gerrit.wikimedia.org/r/489393 (https://phabricator.wikimedia.org/T210818) [19:20:10] (03CR) 10jerkins-bot: [V: 04-1] wmcs::monitoring - Convert cronjob to systemd timer [puppet] - 10https://gerrit.wikimedia.org/r/489394 (https://phabricator.wikimedia.org/T210818) (owner: 10GTirloni) [19:21:28] (03PS2) 10GTirloni: wmcs::monitoring - Convert cronjob to systemd timer [puppet] - 10https://gerrit.wikimedia.org/r/489394 (https://phabricator.wikimedia.org/T210818) [19:22:26] (03CR) 10jerkins-bot: [V: 04-1] wmcs::monitoring - Convert cronjob to systemd timer [puppet] - 10https://gerrit.wikimedia.org/r/489394 (https://phabricator.wikimedia.org/T210818) (owner: 10GTirloni) [19:27:14] (03PS3) 10GTirloni: wmcs::monitoring - Convert cronjob to systemd timer [puppet] - 10https://gerrit.wikimedia.org/r/489394 (https://phabricator.wikimedia.org/T210818) [19:35:52] (03PS1) 10BryanDavis: toolforge: Install Tesseract from stretch-backports [puppet] - 10https://gerrit.wikimedia.org/r/489396 (https://phabricator.wikimedia.org/T215693) [19:38:30] (03CR) 10BryanDavis: "Pretty certainly needs testing via cherry-pick in toolsbeta before merging." [puppet] - 10https://gerrit.wikimedia.org/r/489396 (https://phabricator.wikimedia.org/T215693) (owner: 10BryanDavis) [19:39:47] (03PS1) 10Paladox: gerrit: Make ci comments pretty under PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/489397 [19:47:45] (03PS2) 10Paladox: gerrit: Make ci comments pretty under PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/489397 [19:52:50] (03Abandoned) 10Urbanecm: New throttle rule + removal of expired rules [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489207 (https://phabricator.wikimedia.org/T215446) (owner: 10Urbanecm) [19:54:01] (03PS1) 10Urbanecm: Clean expired throttle rules [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489398 [19:57:12] (03PS1) 10Urbanecm: New throttle rule for Senior Citizens Write Wikipedia course [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489399 (https://phabricator.wikimedia.org/T215618) [19:58:18] (03CR) 10jerkins-bot: [V: 04-1] New throttle rule for Senior Citizens Write Wikipedia course [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489399 (https://phabricator.wikimedia.org/T215618) (owner: 10Urbanecm) [20:00:55] (03PS2) 10Urbanecm: New throttle rule for Senior Citizens Write Wikipedia course [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489399 (https://phabricator.wikimedia.org/T215618) [21:26:59] (03PS1) 10Jforrester: Stop setting wgSessionsInObjectCache, it's being removed from MW [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489408 [21:27:51] (03CR) 10Krinkle: [C: 03+1] Stop setting wgSessionsInObjectCache, it's being removed from MW [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489408 (owner: 10Jforrester) [21:32:27] (03PS1) 10BryanDavis: Set custom mime-types [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/489409 (https://phabricator.wikimedia.org/T178601) [21:33:03] (03CR) 10jerkins-bot: [V: 04-1] Set custom mime-types [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/489409 (https://phabricator.wikimedia.org/T178601) (owner: 10BryanDavis) [21:41:15] !log refreshImageMetadata.php for commonswiki done T215635 [21:41:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:41:18] T215635: Run refreshImageMetadata.php for new media type of DjVu files - https://phabricator.wikimedia.org/T215635 [21:42:00] !log running `foreachwiki refreshImageMetadata.php --mediatype BITMAP --mime image/vnd.djvu --force` on mwmaint1002 T215635 [21:42:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:47:52] (03PS2) 10BryanDavis: Set custom mime-types [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/489409 (https://phabricator.wikimedia.org/T178601) [22:03:29] PROBLEM - MariaDB Slave Lag: s7 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 633.44 seconds [22:21:37] RECOVERY - MariaDB Slave Lag: s7 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 254.43 seconds [22:30:33] Reedy: You're wonderful, thank you. [22:30:44] heh [22:30:53] * Reedy asks for a raise [22:31:19] lol [22:32:22] Reedy: You're doubly-wonderful, happy? ;-) [22:38:35] (03PS1) 10Zoranzoki21: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/operations/mediawiki-config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489419 [22:38:37] (03PS1) 10Zoranzoki21: Remove main page special casing from lawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489420 (https://phabricator.wikimedia.org/T215709) [22:38:55] (03Abandoned) 10Zoranzoki21: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/operations/mediawiki-config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489419 (owner: 10Zoranzoki21) [22:39:10] (03PS2) 10Zoranzoki21: Remove main page special casing from lawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489420 (https://phabricator.wikimedia.org/T215709) [22:49:49] (03CR) 10Reedy: Stop setting wgSessionsInObjectCache, it's being removed from MW (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489408 (owner: 10Jforrester) [22:50:40] 10Puppet, 10Cloud-VPS, 10serviceops, 10Patch-For-Review, 10Technical-Debt: upgrade simplelamp class (apache -> httpd and mysql -> mariadb) or deprecate it - https://phabricator.wikimedia.org/T215662 (10Slevinski) I just rebuilt these instances last month! Sigh... No worries, just let me know what new p... [22:55:41] (03PS3) 10Paladox: gerrit: Make ci comments pretty under PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/489397 (https://phabricator.wikimedia.org/T215658) [23:02:08] (03PS4) 10Paladox: gerrit: Make ci comments pretty under PolyGerrit [puppet] - 10https://gerrit.wikimedia.org/r/489397 (https://phabricator.wikimedia.org/T215658) [23:03:55] (03PS1) 10Paladox: Update bazlets to latest 2.15 commit [software/gerrit/plugins/wikimedia] - 10https://gerrit.wikimedia.org/r/489422 [23:04:27] (03PS2) 10Paladox: Update bazlets to latest 2.15 commit [software/gerrit/plugins/wikimedia] - 10https://gerrit.wikimedia.org/r/489422 [23:07:00] PROBLEM - MariaDB Slave Lag: s7 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 739.36 seconds [23:09:08] PROBLEM - Check systemd state on ms-be2014 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [23:36:14] 10Operations, 10MediaWiki-Revision-backend: Compress data at external storage - https://phabricator.wikimedia.org/T106386 (10Krinkle) [23:39:19] RECOVERY - Check systemd state on ms-be2014 is OK: OK - running: The system is fully operational [23:55:39] RECOVERY - MariaDB Slave Lag: s7 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 269.24 seconds