[03:11:01] 2014/03/04 03:10 OK ortelius / DISK OK - free space: / 6302 MB (21% inode=91%): [03:44:03] 2014/03/04 03:37 WARN hawthorn NTP NTP WARNING: Server has the LI_ALARM bit set, Offset 0.131085 secs [03:55:04] 2014/03/04 03:54 CRIT hawthorn NTP NTP CRITICAL: Server not synchronized, Offset unknown [03:56:04] 2014/03/04 03:55 WARN hawthorn NTP NTP WARNING: Server has the LI_ALARM bit set, Offset 0.000676 secs [04:02:04] 2014/03/04 03:55 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1971.000000 [04:03:04] 2014/03/04 03:56 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2019 [04:13:05] 2014/03/04 04:12 OK hawthorn NTP NTP OK: Offset 0.093334 secs [04:51:07] 2014/03/04 04:50 OK z-dat-s5-b MySQL slave Uptime: 3351114 Threads: 6 Questions: 2694396362 Slow queries: 13665 Opens: 913407 Flush tables: 2 Open tables: 255 Queries per second avg: 804.30 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1677 [04:51:07] 2014/03/04 04:50 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1741.000000 [05:09:08] 2014/03/04 05:02 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1855.000000 [05:10:08] 2014/03/04 05:09 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1525.000000 [09:26:19] 2014/03/04 09:19 WARN rosemary s1 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2143.000000 [09:27:19] 2014/03/04 09:20 WARN rosemary MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2188 [09:52:19] 2014/03/04 09:51 OK rosemary s1 replag QUERY OK: 'SELECT ts_rc_age()' returned 1758.000000 [09:53:19] 2014/03/04 09:52 OK rosemary MySQL slave Uptime: 1634373 Threads: 45 Questions: 781360853 Slow queries: 510124 Opens: 28146 Flush tables: 1 Open tables: 880 Queries per second avg: 478.79 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1715 [10:55:22] 2014/03/04 10:54 OK z-dat-s5-b /sql DISK OK - free space: /sql 138670 MB (12% inode=99%): [13:46:31] 2014/03/04 13:40 WARN z-dat-s5-b /sql DISK WARNING - free space: /sql 121390 MB (10% inode=99%): [14:07:33] 2014/03/04 14:07 OK z-dat-s5-b /sql DISK OK - free space: /sql 139226 MB (12% inode=99%): [15:15:40] 2014/03/04 15:09 WARN z-dat-s5-b /sql DISK WARNING - free space: /sql 120422 MB (10% inode=99%): [15:50:42] 2014/03/04 15:43 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [15:50:42] 2014/03/04 15:44 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (111) [15:50:42] 2014/03/04 15:44 CRIT z-dat-s7-a MySQL Can't connect to MySQL server on 'z-dat-s7-a' (111) [15:50:42] 2014/03/04 15:43 CRIT z-dat-s7-a MySQL slave Can't connect to MySQL server on 'z-dat-s7-a' (111) [16:02:43] 2014/03/04 16:02 OK z-dat-s7-a MySQL Uptime: 1131 Threads: 2 Questions: 45 Slow queries: 0 Opens: 25 Flush tables: 1 Open tables: 17 Queries per second avg: 0.39 [16:03:43] 2014/03/04 16:02 OK z-dat-s7-a MySQL slave Uptime: 1161 Threads: 1 Questions: 838 Slow queries: 0 Opens: 131 Flush tables: 1 Open tables: 122 Queries per second avg: 0.721 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1157 [16:10:45] 2014/03/04 16:10 OK z-dat-s3-a MySQL slave Uptime: 1611 Threads: 1 Questions: 53 Slow queries: 0 Opens: 34 Flush tables: 1 Open tables: 26 Queries per second avg: 0.32 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1618 [16:11:45] 2014/03/04 16:10 OK z-dat-s3-a MySQL Uptime: 1631 Threads: 3 Questions: 662 Slow queries: 1 Opens: 130 Flush tables: 1 Open tables: 120 Queries per second avg: 0.405 [16:21:46] 2014/03/04 16:15 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1940 [16:25:46] 2014/03/04 16:25 OK z-dat-s3-a MySQL slave Uptime: 2511 Threads: 4 Questions: 142233 Slow queries: 36 Opens: 5895 Flush tables: 1 Open tables: 5882 Queries per second avg: 56.643 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1748 [17:16:48] 2014/03/04 17:16 OK z-dat-s5-b /sql DISK OK - free space: /sql 138801 MB (12% inode=99%): [18:39:53] 2014/03/04 18:33 WARN z-dat-s5-b /sql DISK WARNING - free space: /sql 119263 MB (10% inode=99%): [20:00:56] 2014/03/04 20:00 OK z-dat-s5-b /sql DISK OK - free space: /sql 138710 MB (12% inode=99%): [20:24:58] 2014/03/04 20:18 CRIT cassia MySQL Can't connect to MySQL server on 'cassia' (111) [20:24:58] 2014/03/04 20:18 CRIT cassia MySQL slave Can't connect to MySQL server on 'cassia' (111) [20:24:58] 2014/03/04 20:18 CRIT cassia s4 replag QUERY CRITICAL: Can't connect to MySQL server on 'cassia' (111) [20:25:58] 2014/03/04 20:18 CRIT cassia s5 replag QUERY CRITICAL: Can't connect to MySQL server on 'cassia' (111) [20:26:58] 2014/03/04 20:26 OK cassia MySQL Uptime: 51 Threads: 1 Questions: 2713 Slow queries: 1 Opens: 85 Flush tables: 1 Open tables: 78 Queries per second avg: 53.196 [20:26:58] 2014/03/04 20:26 OK cassia MySQL slave Uptime: 54 Threads: 1 Questions: 2735 Slow queries: 1 Opens: 85 Flush tables: 1 Open tables: 78 Queries per second avg: 50.648 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 482 [20:26:58] 2014/03/04 20:25 OK cassia s5 replag QUERY OK: 'SELECT ts_rc_age()' returned 468.000000 [20:26:58] 2014/03/04 20:25 OK cassia wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 470.000000 [20:27:58] 2014/03/04 20:26 OK cassia s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 0.000000 [20:57:00] 2014/03/04 20:49 WARN cassia wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2206.000000 [20:57:00] 2014/03/04 20:50 WARN ortelius / DISK WARNING - free space: / 6232 MB (20% inode=91%): [21:21:02] 2014/03/04 21:19 CRIT cassia wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3646.000000