[00:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:12:03] SSH on turnera is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [00:16:43] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 97200 MB (10% inode=99%): [00:22:03] SSH on damiana is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [00:24:15] @replag [00:24:15] Brooke: s1-user: 2w 1d 21h 38m 38s [-0.43 s/s]; s2-user: 25s [+0.00 s/s]; s3-rr-a: 3m 32s [+0.02 s/s]; s3-user: 3m 32s [+0.02 s/s] [00:34:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1373614 [00:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:41:03] SSH on damiana is CRITICAL: Server answer: [00:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:47:02] SSH on turnera is CRITICAL: Server answer: [00:53:03] SSH on damiana is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [00:54:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1373106.000000 [00:56:02] SSH on turnera is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [01:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:24:17] @replag [01:24:19] russblau: s1-user: 2w 1d 20h 59m 33s [-0.65 s/s]; s2-user: 10s [-0.00 s/s]; s3-rr-a: 28s [-0.05 s/s]; s3-user: 28s [-0.05 s/s] [01:34:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1371197 [01:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:42:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37631 MB (9% inode=99%): [01:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:47:12] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:47:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 88644 MB (9% inode=98%): [01:52:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:53:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:53:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:53:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 88626 MB (9% inode=98%): [01:53:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37756 MB (9% inode=99%): [01:54:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1370124.000000 [02:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:12:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:12:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:12:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 88644 MB (9% inode=98%): [02:12:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38732 MB (9% inode=99%): [02:31:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 88518 MB (9% inode=98%): [02:34:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1368433 [02:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:54:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1367672.000000 [03:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1366595 [03:35:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:54:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:54:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:54:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:54:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1366294.000000 [03:54:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 86782 MB (8% inode=98%): [03:54:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 86782 MB (8% inode=98%): [03:54:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38683 MB (9% inode=99%): [04:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:28:12] @replag [04:28:12] jeremyb: s1-user: 2w 1d 19h 20m 59s [-0.54 s/s]; s3-rr-a: 1m 52s [+0.01 s/s]; s3-user: 1m 52s [+0.01 s/s]; s6-rr-a: 23s [-0.01 s/s]; s6-user: 23s [-0.01 s/s] [04:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1365555 [04:35:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:54:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:54:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:54:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:54:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1365115.000000 [04:55:02] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38655 MB (9% inode=99%): [04:55:03] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 89117 MB (9% inode=98%): [04:55:03] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 89117 MB (9% inode=98%): [05:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1364468 [05:35:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:45:53] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 21459 MB (1% inode=97%): [05:54:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1364011.000000 [06:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1363541 [06:35:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:54:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1362894.000000 [07:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1361910 [07:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:36:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:55:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1361405.000000 [07:59:43] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23872 MB (5% inode=99%): [08:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:02:31] Is the toolserver down? I can't login on nightshade nor on willow [08:32:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42151 MB (10% inode=99%): [08:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1360342 [08:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:36:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:55:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1359343.000000 [09:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:27:14] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:27:14] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:27:14] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:27:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 88848 MB (9% inode=98%): [09:27:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 88848 MB (9% inode=98%): [09:27:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38471 MB (9% inode=99%): [09:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1358673 [09:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:36:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:37:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:55:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1358076.000000 [10:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:03:03] is globalblocking database available somewhere on toolserver? [10:12:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:12:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:12:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:12:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 88787 MB (9% inode=98%): [10:12:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38449 MB (9% inode=99%): [10:12:53] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 88787 MB (9% inode=98%): [10:35:24] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1357414 [10:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:36:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:46:02] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 69 MB (12% inode=98%): [10:47:03] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:52:43] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:52:43] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:52:53] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:53:02] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:53:12] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:53:22] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:53:33] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:54:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:54:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:54:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:55:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1357041.000000 [10:55:53] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:55:53] /home on hemlock is OK: DISK OK - free space: /home 19890 MB (39% inode=87%): [10:56:03] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 89419 MB (9% inode=98%): [10:56:03] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38430 MB (9% inode=99%): [10:56:03] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 89419 MB (9% inode=98%): [10:56:03] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 66 MB (12% inode=98%): [10:56:13] CAM on hemlock is OK: OK - cam detected no new errors [10:56:22] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [10:56:32] Load avg. on hemlock is OK: OK - load average: 0.20, 0.25, 0.82 [10:56:42] / on hemlock is OK: DISK OK - free space: / 7520 MB (37% inode=89%): [10:56:42] /aux0 on hemlock is OK: DISK OK - free space: /aux0 882340 MB (16% inode=58%): [11:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:03:13] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:03:23] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:20:23] @replag [11:20:23] Dispenser: s1-user: 2w 1d 16h 46m 57s [-0.37 s/s]; s2-user: 1h 20m 43s [+0.14 s/s] [11:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1356134 [11:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:36:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:54:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:54:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:54:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:55:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1355457.000000 [11:55:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 89257 MB (9% inode=98%): [11:55:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38400 MB (9% inode=99%): [11:55:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 89257 MB (9% inode=98%): [12:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:16:43] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 96923 MB (10% inode=99%): [12:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1354904 [12:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:36:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:48:01] hello all [12:48:04] DaB. * [Toolserver-announce] sql-s1 will be read-only on Wednesday [12:55:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1354420.000000 [13:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:23:38] hi [13:23:40] @replag [13:23:40] jeremyb: s1-user: 2w 1d 16h 3m 54s [-0.35 s/s]; s2-user: 2h 33m 15s [+0.59 s/s]; s3-rr-a: 14s [-0.00 s/s]; s3-user: 14s [-0.00 s/s] [13:23:47] so slow... [13:23:50] * jeremyb runs away [13:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1353550 [13:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:36:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:47:42] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23958 MB (5% inode=99%): [13:52:03] Load avg. on adenia is WARNING: WARNING - load average: 15.77, 14.52, 9.20 [13:55:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1353461.000000 [13:55:03] Load avg. on adenia is OK: OK - load average: 13.77, 14.55, 10.19 [14:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:15:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42573 MB (10% inode=99%): [14:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1353470 [14:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:36:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:55:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1352897.000000 [15:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1351961 [15:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:36:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:55:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1351441.000000 [16:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:01:58] @replag [16:01:59] Gciampaglia: s1-user: 2w 1d 15h 20m 28s [-0.27 s/s]; s2-user: 3h 38m 18s [+0.41 s/s]; s2-user-c: 20s [+0.00 s/s]; s5-rr-a: 28s [+0.00 s/s]; s5-user: 28s [+0.00 s/s]; s5-user-c: 20s [+0.00 s/s] [16:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1350608 [16:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:37:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:46:59] @replag [16:46:59] DaBPunkt: s1-user: 2w 1d 15h 4m 15s [-0.36 s/s]; s2-user: 3h 50m 7s [+0.26 s/s] [16:55:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1349971.000000 [17:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1349720 [17:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:37:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:37:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:45:54] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 23034 MB (1% inode=97%): [17:55:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1349594.000000 [18:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:33:05] Samuel Klein * Re: [Toolserver-l] s1 replag update, suggestion, and question [18:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1348890 [18:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:37:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:37:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:55:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1348560.000000 [19:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1348152 [19:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:37:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:55:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1347451.000000 [20:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1346574 [20:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:37:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:53:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:55:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1346092.000000 [21:00:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:48:17] Load avg. on ortelius is WARNING: WARNING - load average: 13.22, 15.12, 12.42 [21:49:17] Load avg. on ortelius is OK: OK - load average: 11.63, 14.41, 12.34 [21:52:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:53:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1344117 [21:53:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:55:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1344070.000000 [22:00:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:50:04] MZMcBride * Re: [Toolserver-l] sql-s1 will be read-only on Wednesday [22:52:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:53:16] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:53:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1342566 [22:53:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:55:16] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1342501.000000 [23:00:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:36:36] nacht ts [23:45:17] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 63.64, 53.46, 29.30 [23:52:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:53:16] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:53:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1340737 [23:53:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:55:16] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1340687.000000 [23:55:57] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 88412 MB (9% inode=98%): [23:55:57] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 88412 MB (9% inode=98%):