[06:35:28] 2013/07/30 06:31 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2001 [06:40:29] 2013/07/30 06:33 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2198 [07:02:30] 2013/07/30 07:01 CRIT z-dat-s3-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3607 [07:05:31] 2013/07/30 07:04 CRIT z-dat-s7-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3653 [07:10:31] 2013/07/30 07:06 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1950 [08:18:36] 2013/07/30 08:17 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [08:19:36] 2013/07/30 08:18 CRIT z-dat-s4-a MySQL slave (Service Check Timed Out) [08:21:36] 2013/07/30 08:16 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (110) [08:22:36] 2013/07/30 08:21 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 64648 MB (10% inode=99%): [08:22:37] 2013/07/30 08:18 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110) [08:23:37] 2013/07/30 08:17 CRIT hyacinth SSH CRITICAL - Socket timeout after 10 seconds [08:23:37] 2013/07/30 08:18 CRIT z-dat-s4-a MySQL (Service Check Timed Out) [08:23:37] 2013/07/30 08:17 CRIT z-dat-s6-a /sql Timeout while attempting connection [08:24:37] 2013/07/30 08:17 CRIT hyacinth Environment IPMI CHECK_NRPE: Socket timeout after 30 seconds. [08:24:37] 2013/07/30 08:17 CRIT hyacinth RAID CHECK_NRPE: Socket timeout after 30 seconds. [08:24:37] 2013/07/30 08:17 CRIT hyacinth SMTP CRITICAL - Socket timeout after 10 seconds [08:24:37] 2013/07/30 08:17 CRIT z-dat-s3-a Load avg. CHECK_NRPE: Socket timeout after 30 seconds. [08:24:37] 2013/07/30 08:17 CRIT z-dat-s6-a / Timeout while attempting connection [08:24:37] 2013/07/30 08:17 CRIT z-dat-s6-a /tmp Timeout while attempting connection [08:24:37] 2013/07/30 08:17 CRIT z-dat-s6-a Load avg. Timeout while attempting connection [08:24:38] 2013/07/30 08:16 CRIT z-dat-s6-a MySQL slave Can't connect to MySQL server on 'z-dat-s6-a' (110) [08:25:37] 2013/07/30 08:18 CRIT z-dat-s7-a MySQL Can't connect to MySQL server on 'z-dat-s7-a' (110) [08:26:36] 2013/07/30 08:25 OK hyacinth Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [08:26:37] 2013/07/30 08:26 OK hyacinth RAID OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:26:37] 2013/07/30 08:26 OK hyacinth SMTP SMTP OK - 0.003 sec. response time [08:26:37] 2013/07/30 08:26 OK hyacinth SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:26:37] 2013/07/30 08:25 OK z-dat-s3-a Load avg. OK - load average: 0.02, 0.17, 0.42 [08:26:37] 2013/07/30 08:25 OK z-dat-s3-a MySQL Uptime: 80323 Threads: 135 Questions: 117600804 Slow queries: 17001 Opens: 392057 Flush tables: 1 Open tables: 16384 Queries per second avg: 1464.98 [08:26:37] 2013/07/30 08:25 OK z-dat-s4-a MySQL Uptime: 338493 Threads: 1 Questions: 10489188 Slow queries: 1 Opens: 82 Flush tables: 1 Open tables: 74 Queries per second avg: 30.987 [08:26:38] 2013/07/30 08:25 OK z-dat-s6-a / DISK OK - free space: / 9345 MB (31% inode=86%): [08:26:38] 2013/07/30 08:26 OK z-dat-s6-a /sql DISK OK - free space: /sql 293550 MB (36% inode=99%): [08:26:39] 2013/07/30 08:26 OK z-dat-s6-a /tmp DISK OK - free space: /tmp 385 MB (99% inode=99%): [08:26:39] 2013/07/30 08:25 OK z-dat-s6-a MySQL Uptime: 514783 Threads: 59 Questions: 297347091 Slow queries: 34694 Opens: 45344 Flush tables: 1 Open tables: 4099 Queries per second avg: 577.616 [08:26:40] 2013/07/30 08:25 OK z-dat-s6-a MySQL slave Uptime: 514783 Threads: 60 Questions: 297347089 Slow queries: 34694 Opens: 45344 Flush tables: 1 Open tables: 4099 Queries per second avg: 577.616 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 538 [08:26:40] 2013/07/30 08:25 OK z-dat-s7-a MySQL Uptime: 334754 Threads: 118 Questions: 162023113 Slow queries: 15291 Opens: 499053 Flush tables: 1 Open tables: 3780 Queries per second avg: 484.6 [08:27:36] 2013/07/30 08:26 OK z-dat-s6-a Load avg. OK - load average: 1.17, 0.45, 0.50 [08:28:37] 2013/07/30 08:28 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3537 [08:34:36] 2013/07/30 08:33 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [08:35:37] 2013/07/30 08:34 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 64642 MB (10% inode=99%): [08:45:37] 2013/07/30 08:44 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [08:47:37] 2013/07/30 08:46 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 64636 MB (10% inode=99%): [08:53:37] 2013/07/30 08:52 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [08:54:37] 2013/07/30 08:53 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 64632 MB (10% inode=99%): [09:15:37] 2013/07/30 09:15 OK z-dat-s4-a MySQL slave Uptime: 341433 Threads: 1 Questions: 10631410 Slow queries: 1 Opens: 82 Flush tables: 1 Open tables: 74 Queries per second avg: 31.137 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1706 [09:31:38] 2013/07/30 09:30 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [09:33:38] 2013/07/30 09:32 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 64607 MB (10% inode=99%): [09:34:38] 2013/07/30 09:30 CRIT z-dat-s4-a MySQL slave (Service Check Timed Out) [09:35:39] 2013/07/30 09:34 OK z-dat-s4-a MySQL slave Uptime: 342603 Threads: 2 Questions: 10712129 Slow queries: 6 Opens: 82 Flush tables: 1 Open tables: 74 Queries per second avg: 31.266 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 351 [10:05:41] 2013/07/30 10:04 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3406 [10:09:41] 2013/07/30 10:08 OK z-dat-s7-a MySQL slave Uptime: 340951 Threads: 20 Questions: 163643163 Slow queries: 17602 Opens: 515053 Flush tables: 1 Open tables: 3808 Queries per second avg: 479.960 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1735 [10:15:42] 2013/07/30 10:12 CRIT z-dat-s5-b MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [10:18:42] 2013/07/30 10:12 CRIT cassia MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [10:23:43] 2013/07/30 10:22 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [10:24:43] 2013/07/30 10:23 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 64572 MB (10% inode=99%): [10:33:44] 2013/07/30 10:30 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [10:45:44] 2013/07/30 10:45 OK z-dat-s3-a MySQL Uptime: 900 Threads: 23 Questions: 914 Slow queries: 3 Opens: 327 Flush tables: 1 Open tables: 317 Queries per second avg: 1.15 [10:48:44] 2013/07/30 10:41 CRIT z-dat-s6-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 55478 [11:00:02] @replag [11:00:02] DaBPunkt: s1-rr-a-wd: 15h 19m 33s [+0.12 s/s]; s1-user-wd: 15h 19m 33s [+0.12 s/s]; s2-rr: 7m 57s [-0.00 s/s]; s2-user: 7m 57s [-0.00 s/s]; s2-user-c: error; s2-user-wd: 15h 19m 33s [+0.02 s/s]; s3-user: 4h 31m 38s [+0.04 s/s]; s3-user-wd: 15h 19m 33s [+0.12 s/s] [11:00:03] DaBPunkt: s4-user-wd: 3d 14h 1m 42s [-0.55 s/s]; s5-rr-a: 53m 46s [+0.01 s/s]; s5-rr-a-wd: 15h 19m 33s [+0.12 s/s]; s5-user: 13h 5m 57s [+0.11 s/s]; s6-user: 15h 21m 53s [+0.13 s/s]; s6-user-wd: 15h 19m 33s [+0.12 s/s]; s7-user-wd: 15h 19m 33s [+0.12 s/s] [11:00:18] @replag [11:00:19] DaBPunkt: s1-rr-a-wd: 15h 19m 49s [+0.98 s/s]; s1-user-wd: 15h 19m 49s [+0.98 s/s]; s2-rr: 8m 13s [+0.98 s/s]; s2-user: 8m 13s [+0.98 s/s]; s2-user-c: error; s2-user-wd: 15h 19m 49s [+0.98 s/s]; s3-user: 4h 31m 54s [+0.98 s/s]; s3-user-wd: 15h 19m 49s [+0.98 s/s] [11:00:20] DaBPunkt: s4-user-wd: 3d 14h 1m 58s [+0.99 s/s]; s5-rr-a: 47m 7s [-24.59 s/s]; s5-rr-a-wd: 15h 19m 49s [+0.99 s/s]; s5-user: 13h 3m 33s [-8.88 s/s]; s5-user-c: 10s [+0.00 s/s]; s6-user: 15h 21m 51s [-0.12 s/s]; s6-user-wd: 15h 19m 49s [+0.99 s/s]; s7-user-wd: 15h 19m 49s [+0.99 s/s] [11:00:44] 2013/07/30 11:00 WARN cassia MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2354 [11:00:44] 2013/07/30 10:59 WARN cassia s5 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3589.000000 [11:01:45] 2013/07/30 11:01 OK cassia MySQL slave Uptime: 3898540 Threads: 10 Questions: 3062147866 Slow queries: 56470 Opens: 15402 Flush tables: 1 Open tables: 14118 Queries per second avg: 785.460 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 536 [11:02:45] 2013/07/30 11:01 OK cassia s5 replag QUERY OK: 'SELECT ts_rc_age()' returned 309.000000 [12:24:51] 2013/07/30 12:24 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3589 [12:33:52] 2013/07/30 12:33 OK z-dat-s3-a MySQL slave Uptime: 7381 Threads: 23 Questions: 7392723 Slow queries: 591 Opens: 34819 Flush tables: 1 Open tables: 16384 Queries per second avg: 1001.588 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1676 [12:46:52] 2013/07/30 12:40 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (111) [12:47:52] 2013/07/30 12:40 CRIT z-dat-s2-b MySQL slave Can't connect to MySQL server on 'z-dat-s2-b' (111) [12:54:39] nightshade> ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0 [13:18:52] 2013/07/30 13:18 OK z-dat-s2-b MySQL Uptime: 2282 Threads: 5 Questions: 3622 Slow queries: 0 Opens: 294 Flush tables: 1 Open tables: 256 Queries per second avg: 1.587 [13:18:52] 2013/07/30 13:17 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2278 [13:22:52] 2013/07/30 13:22 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3345 [13:22:52] 2013/07/30 13:22 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3240.000000 [13:26:53] 2013/07/30 13:26 OK z-dat-s5-b MySQL slave Uptime: 1558900 Threads: 7 Questions: 2085514444 Slow queries: 14464 Opens: 1435381 Flush tables: 1 Open tables: 255 Queries per second avg: 1337.811 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1520 [13:26:53] 2013/07/30 13:26 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1360.000000 [13:32:54] 2013/07/30 13:31 OK z-dat-s2-b MySQL slave Uptime: 3070 Threads: 5 Questions: 229619 Slow queries: 14 Opens: 5133 Flush tables: 1 Open tables: 256 Queries per second avg: 74.794 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1757 [13:41:54] 2013/07/30 13:35 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (111) [13:42:54] 2013/07/30 13:35 CRIT z-dat-s2-b MySQL slave Can't connect to MySQL server on 'z-dat-s2-b' (111) [13:58:38] [[Special:Log/newusers]] create 10 * PortalandPortal2Rocks * (New user account) [14:16:55] 2013/07/30 14:16 OK z-dat-s2-b MySQL Uptime: 2465 Threads: 35 Questions: 437 Slow queries: 0 Opens: 153 Flush tables: 1 Open tables: 144 Queries per second avg: 0.177 [14:31:04] @replag [14:31:04] Merlissimo: s1-rr-a-wd: 18h 50m 35s [+1.00 s/s]; s1-user-wd: 18h 50m 35s [+1.00 s/s]; s2-rr: 1h 19m 15s [+0.34 s/s]; s2-user: 1h 19m 15s [+0.34 s/s]; s2-user-c: error; s2-user-wd: 18h 50m 35s [+1.00 s/s]; s3-user: 39s [-1.29 s/s]; s3-user-wd: 18h 50m 35s [+1.00 s/s] [14:31:05] Merlissimo: s4-user-wd: 3d 17h 32m 44s [+1.00 s/s]; s5-rr-a-wd: 18h 50m 35s [+1.00 s/s]; s5-user: 1m 45s [-3.71 s/s]; s6-user: 17h 16m 19s [+0.54 s/s]; s6-user-wd: 18h 50m 35s [+1.00 s/s]; s7-user-wd: 18h 50m 35s [+1.00 s/s] [15:12:56] 2013/07/30 15:12 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3353 [15:17:57] 2013/07/30 15:09 CRIT z-dat-s5-b s4 replag (Service Check Timed Out) [15:18:57] 2013/07/30 15:10 CRIT z-dat-s5-b wikidata replag (Service Check Timed Out) [15:25:58] 2013/07/30 15:25 OK z-dat-s5-b s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1629.000000 [15:25:58] 2013/07/30 15:25 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1819.000000 [15:28:58] 2013/07/30 15:25 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2003 [15:35:58] 2013/07/30 15:29 WARN z-dat-s5-b s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1878.000000 [15:37:59] 2013/07/30 15:37 OK z-dat-s5-b s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1740.000000 [15:41:59] 2013/07/30 15:41 OK z-dat-s2-b MySQL slave Uptime: 7573 Threads: 8 Questions: 1140697 Slow queries: 122 Opens: 23229 Flush tables: 1 Open tables: 256 Queries per second avg: 150.626 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1783 [16:09:02] 2013/07/30 16:08 OK z-dat-s5-b MySQL slave Uptime: 1568620 Threads: 7 Questions: 2088277507 Slow queries: 14804 Opens: 1435934 Flush tables: 1 Open tables: 256 Queries per second avg: 1331.283 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1720 [16:09:02] 2013/07/30 16:08 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1756.000000 [18:23:55] @replag [18:23:55] DaBPunkt: s1-rr-a-wd: 22h 43m 26s [+1.00 s/s]; s1-user-wd: 22h 43m 26s [+1.00 s/s]; s2-rr: 16s [-0.34 s/s]; s2-user: 16s [-0.34 s/s]; s2-user-c: error; s2-user-wd: 22h 43m 26s [+1.00 s/s]; s3-user: 1m 43s [+0.02 s/s]; s3-user-wd: 22h 43m 26s [+1.00 s/s] [18:23:56] DaBPunkt: s4-user-wd: 3d 21h 25m 35s [+1.00 s/s]; s5-rr-a-wd: 22h 43m 26s [+1.00 s/s]; s6-user: 2h 22m 10s [-8.35 s/s]; s6-user-wd: 22h 43m 26s [+1.00 s/s]; s7-user-wd: 22h 43m 26s [+1.00 s/s] [18:32:06] 2013/07/30 18:30 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3278 [18:35:06] 2013/07/30 18:33 OK z-dat-s6-a MySQL slave Uptime: 551290 Threads: 9 Questions: 304465345 Slow queries: 36134 Opens: 45665 Flush tables: 1 Open tables: 4142 Queries per second avg: 552.278 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1061 [21:42:18] @replag [21:42:18] DaBPunkt: s1-rr-a-wd: 1d 2h 1m 49s [+1.00 s/s]; s1-user-wd: 1d 2h 1m 49s [+1.00 s/s]; s2-user-c: error; s2-user-wd: 1d 2h 1m 49s [+1.00 s/s]; s3-user-wd: 1d 2h 1m 49s [+1.00 s/s]; s4-user-wd: 4d 43m 58s [+1.00 s/s]; s5-rr-a-wd: 1d 2h 1m 49s [+1.00 s/s]; s6-user-wd: 1d 2h 1m 49s [+1.00 s/s] [21:42:19] DaBPunkt: s7-user-wd: 1d 2h 1m 49s [+1.00 s/s] [22:19:34] nacht ts