[03:04:20] HTTPS is intermittently broken? [03:48:12] 2013/12/02 03:41 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2047 [03:48:12] 2013/12/02 03:42 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2053.000000 [03:54:13] 2013/12/02 03:53 OK z-dat-s5-b MySQL slave Uptime: 431742 Threads: 5 Questions: 510985757 Slow queries: 4856 Opens: 29133 Flush tables: 1 Open tables: 255 Queries per second avg: 1183.544 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1606 [03:54:13] 2013/12/02 03:54 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1497.000000 [04:55:51] [[Special:Log/newusers]] create 10 * AdityaSharmaMalladi * (New user account) [07:32:30] amette_: ping [10:13:34] 2013/12/02 10:09 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110) [10:13:34] 2013/12/02 10:09 CRIT z-dat-s7-a MySQL slave Can't connect to MySQL server on 'z-dat-s7-a' (110) [10:14:34] 2013/12/02 10:07 CRIT z-dat-s3-a /tmp Timeout while attempting connection [10:14:34] 2013/12/02 10:13 OK z-dat-s7-a MySQL slave Uptime: 1028207 Threads: 14 Questions: 194731308 Slow queries: 68696 Opens: 1045080 Flush tables: 1 Open tables: 2455 Queries per second avg: 189.389 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 308 [10:15:34] 2013/12/02 10:14 OK z-dat-s3-a /tmp DISK OK - free space: /tmp 1079 MB (99% inode=99%): [12:50:39] 2013/12/02 12:44 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1964 [13:43:43] 2013/12/02 13:43 OK z-dat-s2-b MySQL slave Uptime: 6750865 Threads: 12 Questions: 2482226860 Slow queries: 61092 Opens: 23783058 Flush tables: 1 Open tables: 257 Queries per second avg: 367.690 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1599 [13:44:43] 2013/12/02 13:37 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2109 [13:44:43] 2013/12/02 13:38 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2114.000000 [15:09:49] 2013/12/02 15:09 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1788.000000 [15:10:49] 2013/12/02 15:09 OK z-dat-s5-b MySQL slave Uptime: 472302 Threads: 9 Questions: 602694663 Slow queries: 5432 Opens: 33699 Flush tables: 1 Open tables: 256 Queries per second avg: 1276.78 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1713 [16:10:54] 2013/12/02 16:04 CRIT nightshade aliasd Connection refused [16:36:28] [[Special:Log/newusers]] create 10 * Usha156 * (New user account) [16:45:25] [[Special:Log/newusers]] create 10 * Sherley977 * (New user account) [16:57:36] [[Special:Log/newusers]] create 10 * Valery842 * (New user account) [17:59:02] 2013/12/02 17:50 CRIT daphne s4 replag (Service Check Timed Out) [18:02:02] 2013/12/02 18:00 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 470.000000 [18:57:06] 2013/12/02 18:49 CRIT daphne s4 replag (Service Check Timed Out) [19:14:07] 2013/12/02 19:07 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2120 [19:14:07] 2013/12/02 19:07 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2079.000000 [19:16:07] 2013/12/02 19:14 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 852.000000 [19:43:10] 2013/12/02 19:43 CRIT z-dat-s5-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3602.000000 [19:44:10] 2013/12/02 19:43 CRIT z-dat-s5-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3649 [20:08:57] [[Special:Log/newusers]] create 10 * Jane321 * (New user account) [21:00:12] 2013/12/02 20:54 WARN daphne MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2019 [21:01:12] 2013/12/02 20:54 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2043.000000 [21:58:13] 2013/12/02 21:56 CRIT daphne s4 replag (Service Check Timed Out) [21:59:13] 2013/12/02 21:57 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3237.000000 [22:01:13] 2013/12/02 21:59 CRIT daphne s4 replag (Service Check Timed Out) [22:03:15] 2013/12/02 22:02 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3435.000000 [22:06:15] 2013/12/02 22:06 CRIT daphne MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3629 [22:08:15] 2013/12/02 22:06 CRIT daphne s4 replag (Service Check Timed Out) [22:39:17] 2013/12/02 22:32 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2098 [23:13:20] 2013/12/02 23:13 CRIT z-dat-s2-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3609