[04:36:47] @replag [06:15:06] [[Special:Log/newusers]] create 10 * Antonette32 * (New user account) [07:13:52] 2013/12/16 07:07 WARN rosemary MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2044 [07:13:52] 2013/12/16 07:06 WARN rosemary s1 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2030.000000 [08:06:55] 2013/12/16 08:06 CRIT rosemary MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3632 [08:06:55] 2013/12/16 08:05 CRIT rosemary s1 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3617.000000 [08:32:01] [[Special:Log/newusers]] create 10 * Louanne37 * (New user account) [09:16:58] 2013/12/16 09:10 WARN hawthorn NTP NTP WARNING: Server has the LI_ALARM bit set, Offset -0.131196 secs [09:45:00] 2013/12/16 09:44 OK hawthorn NTP NTP OK: Offset -0.067174 secs [10:15:02] 2013/12/16 10:11 CRIT z-dat-s7-a MySQL slave Can't connect to MySQL server on 'z-dat-s7-a' (110) [10:16:02] 2013/12/16 10:11 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110) [10:16:02] 2013/12/16 10:15 OK z-dat-s7-a MySQL slave Uptime: 2237907 Threads: 16 Questions: 441250227 Slow queries: 145183 Opens: 2314162 Flush tables: 1 Open tables: 2800 Queries per second avg: 197.170 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 255 [10:17:02] 2013/12/16 10:10 CRIT hyacinth RAID CHECK_NRPE: Socket timeout after 30 seconds. [10:18:02] 2013/12/16 10:11 CRIT hyacinth NTP CRITICAL - Socket timeout after 10 seconds [10:18:02] 2013/12/16 10:10 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (110) [10:18:02] 2013/12/16 10:10 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (110) [10:19:02] 2013/12/16 10:18 OK hyacinth NTP NTP OK: Offset -0.000381 secs [10:19:02] 2013/12/16 10:18 OK hyacinth RAID OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:19:02] 2013/12/16 10:17 OK z-dat-s3-a MySQL Uptime: 2238059 Threads: 26 Questions: 1237152565 Slow queries: 98644 Opens: 11579188 Flush tables: 1 Open tables: 16384 Queries per second avg: 552.779 [10:19:02] 2013/12/16 10:17 OK z-dat-s3-a MySQL slave Uptime: 2238059 Threads: 26 Questions: 1237152563 Slow queries: 98644 Opens: 11579188 Flush tables: 1 Open tables: 16384 Queries per second avg: 552.779 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 427 [10:19:02] 2013/12/16 10:17 OK z-dat-s6-a MySQL Uptime: 2672731 Threads: 5 Questions: 622435872 Slow queries: 211229 Opens: 1210299 Flush tables: 1 Open tables: 1276 Queries per second avg: 232.883 [10:45:04] 2013/12/16 10:41 CRIT z-dat-s7-a MySQL slave Can't connect to MySQL server on 'z-dat-s7-a' (110) [10:46:04] 2013/12/16 10:41 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110) [10:46:04] 2013/12/16 10:41 CRIT z-dat-s7-a MySQL Can't connect to MySQL server on 'z-dat-s7-a' (110) [10:47:04] 2013/12/16 10:46 OK z-dat-s6-a MySQL Uptime: 2674432 Threads: 4 Questions: 622663922 Slow queries: 211325 Opens: 1211130 Flush tables: 1 Open tables: 1278 Queries per second avg: 232.820 [10:47:04] 2013/12/16 10:46 OK z-dat-s7-a MySQL Uptime: 2239766 Threads: 5 Questions: 441460400 Slow queries: 145289 Opens: 2314164 Flush tables: 1 Open tables: 2802 Queries per second avg: 197.101 [10:47:04] 2013/12/16 10:45 OK z-dat-s7-a MySQL slave Uptime: 2239737 Threads: 7 Questions: 441458420 Slow queries: 145285 Opens: 2314164 Flush tables: 1 Open tables: 2802 Queries per second avg: 197.102 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 246 [11:59:08] 2013/12/16 11:58 OK z-dat-s3-a SMTP SMTP OK - 0.415 sec. response time [11:59:08] 2013/12/16 11:58 OK z-dat-s3-a SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:59:08] 2013/12/16 11:58 OK z-dat-s4-a SMTP SMTP OK - 0.003 sec. response time [11:59:08] 2013/12/16 11:58 OK z-dat-s4-a SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:59:08] 2013/12/16 11:58 OK z-dat-s6-a SMTP SMTP OK - 0.193 sec. response time [11:59:08] 2013/12/16 11:58 OK z-dat-s6-a SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:59:08] 2013/12/16 11:58 OK z-dat-s7-a SMTP SMTP OK - 0.412 sec. response time [11:59:08] 2013/12/16 11:57 OK z-dat-s7-a SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:07:08] 2013/12/16 12:06 WARN rosemary MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3578 [12:07:08] 2013/12/16 12:05 WARN rosemary s1 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3568.000000 [12:37:09] 2013/12/16 12:35 OK rosemary s1 replag QUERY OK: 'SELECT ts_rc_age()' returned 1797.000000 [12:38:09] 2013/12/16 12:37 OK rosemary MySQL slave Uptime: 3983644 Threads: 10 Questions: 1354945263 Slow queries: 1066802 Opens: 31868 Flush tables: 1 Open tables: 1457 Queries per second avg: 340.127 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1536 [12:38:09] 2013/12/16 12:35 CRIT z-dat-s7-a MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [12:42:09] 2013/12/16 12:36 CRIT daphne MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [12:42:09] 2013/12/16 12:35 CRIT z-dat-s6-a MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [13:08:12] 2013/12/16 13:01 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2176.000000 [13:29:13] 2013/12/16 13:28 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3400 [13:32:13] 2013/12/16 13:31 CRIT daphne s4 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3616.000000 [13:33:13] 2013/12/16 13:32 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3417 [13:38:15] 2013/12/16 13:37 OK z-dat-s6-a MySQL slave Uptime: 2684680 Threads: 8 Questions: 624550142 Slow queries: 211990 Opens: 1214832 Flush tables: 1 Open tables: 1249 Queries per second avg: 232.634 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1521 [13:43:15] 2013/12/16 13:36 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2165 [13:44:15] 2013/12/16 13:43 OK z-dat-s7-a MySQL slave Uptime: 2250396 Threads: 23 Questions: 442711490 Slow queries: 145997 Opens: 2314237 Flush tables: 1 Open tables: 2812 Queries per second avg: 196.726 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1648 [13:51:15] 2013/12/16 13:50 OK z-dat-s2-b MySQL slave Uptime: 157266 Threads: 25 Questions: 75648710 Slow queries: 2214 Opens: 1027339 Flush tables: 1 Open tables: 256 Queries per second avg: 481.23 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1744 [14:10:17] 2013/12/16 14:10 WARN daphne MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3513 [14:11:17] 2013/12/16 14:10 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3369.000000 [14:15:16] 2013/12/16 14:15 OK daphne MySQL slave Uptime: 20888913 Threads: 11 Questions: 2934811031 Slow queries: 2150088 Opens: 516610 Flush tables: 1 Open tables: 2418 Queries per second avg: 140.496 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1796 [14:16:16] 2013/12/16 14:15 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1557.000000 [14:53:19] 2013/12/16 14:50 CRIT z-dat-s7-a SSH CRITICAL - Socket timeout after 10 seconds [14:54:19] 2013/12/16 14:50 CRIT z-dat-s4-a SMTP CRITICAL - Socket timeout after 10 seconds [14:57:19] 2013/12/16 14:50 CRIT z-dat-s3-a SMTP CRITICAL - Socket timeout after 10 seconds [14:57:19] 2013/12/16 14:50 CRIT z-dat-s3-a SSH CRITICAL - Socket timeout after 10 seconds [14:57:19] 2013/12/16 14:50 CRIT z-dat-s4-a SSH CRITICAL - Socket timeout after 10 seconds [14:57:19] 2013/12/16 14:50 CRIT z-dat-s6-a SMTP CRITICAL - Socket timeout after 10 seconds [14:57:20] 2013/12/16 14:50 CRIT z-dat-s6-a SSH CRITICAL - Socket timeout after 10 seconds [14:57:20] 2013/12/16 14:50 CRIT z-dat-s7-a SMTP CRITICAL - Socket timeout after 10 seconds [23:03:55] @replag [23:07:40] Dispenser: s1-rr-a: 6w 1d 20h 51m 53s [+1.00 s/s]; s1-rr-a-wd: error; s1-user-c: 5w 3d 13h 47m 38s [+1.00 s/s]; s1-user-wd: 10w 5d 10h 27m 47s [+1.00 s/s]; s2-rr: 16s [-0.00 s/s]; s2-user: 17s [-0.00 s/s]; s2-user-c: error; s2-user-wd: 9w 15h 19m 1s [+1.00 s/s] [23:07:41] Dispenser: s3-user: 37s [-0.00 s/s]; s3-user-wd: 22w 3d 4h 45m 47s [+1.00 s/s]; s4-user-wd: 18w 2d 10h 2m 30s [+1.00 s/s]; s5-rr-a-wd: 10w 1d 5h 47m 10s [+1.00 s/s]; s5-user-c: 7w 20h 27m 53s [+1.00 s/s]; s6-user-wd: 18w 2d 10h 3m 10s [+1.00 s/s]; s7-user-wd: 18w 2d 10h 3m 17s [+1.00 s/s]