[00:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:16:43] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 94649 MB (9% inode=99%): [00:18:22] Load avg. on willow is WARNING: WARNING - load average: 14.50, 15.21, 12.83 [00:19:23] Load avg. on willow is OK: OK - load average: 12.46, 14.54, 12.74 [00:21:46] hello, humans! [00:22:32] can someone help me with a query I have running? [00:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [00:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [00:32:02] Load avg. on adenia is WARNING: WARNING - load average: 16.07, 15.67, 15.56 [00:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1465791 [00:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [00:40:32] hello hello [00:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:44:03] Load avg. on adenia is OK: OK - load average: 11.49, 13.71, 14.87 [00:53:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1464869.000000 [01:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:27:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [01:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [01:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1461801 [01:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [01:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:48:52] @replag [01:48:52] jeremyb: s1-user: 2w 2d 21h 45m 49s [-1.01 s/s]; s2-user: 10s [-0.00 s/s]; s3-rr-a: 2m 20s [+0.00 s/s]; s3-user: 2m 20s [+0.00 s/s]; s6-rr-a: 10s [-0.00 s/s]; s6-user: 11s [-0.00 s/s] [01:50:20] do multi maintainer projects have "user databases"? [01:50:30] if so, what is the naming convention? [01:52:05] dschwen: i assume p_ [01:52:31] ok [01:53:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1460389.000000 [01:53:22] hm [01:53:32] I might have to request the DB to be set up [01:55:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 91657 MB (9% inode=98%): [01:55:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 91657 MB (9% inode=98%): [01:55:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38769 MB (9% inode=99%): [02:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:12:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:12:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:12:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:16:22] Load avg. on willow is WARNING: WARNING - load average: 20.42, 17.93, 15.02 [02:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [02:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [02:31:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 91061 MB (9% inode=98%): [02:31:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 91061 MB (9% inode=98%): [02:31:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38760 MB (9% inode=99%): [02:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1456987 [02:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:36:22] Load avg. on willow is OK: OK - load average: 10.98, 13.91, 14.88 [02:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [02:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:53:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1454800.000000 [03:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:06:23] Load avg. on willow is WARNING: WARNING - load average: 15.43, 15.39, 14.38 [03:07:22] Load avg. on willow is OK: OK - load average: 13.81, 14.90, 14.27 [03:12:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:12:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:12:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:13:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 91502 MB (9% inode=98%): [03:13:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 91502 MB (9% inode=98%): [03:13:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39557 MB (9% inode=99%): [03:16:22] Load avg. on willow is WARNING: WARNING - load average: 17.08, 17.48, 15.95 [03:27:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [03:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [03:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1452319 [03:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [03:39:23] Load avg. on willow is OK: OK - load average: 10.48, 13.16, 14.76 [03:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:53:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1451185.000000 [04:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:11:23] Load avg. on willow is WARNING: WARNING - load average: 19.98, 17.36, 14.70 [04:19:22] Load avg. on willow is OK: OK - load average: 12.32, 14.84, 14.76 [04:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [04:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [04:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1448237 [04:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:36:22] Load avg. on willow is WARNING: WARNING - load average: 13.78, 15.05, 15.05 [04:37:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [04:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:53:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1447110.000000 [05:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:16:22] Load avg. on willow is WARNING: WARNING - load average: 16.44, 17.05, 15.53 [05:20:23] Load avg. on willow is OK: OK - load average: 14.56, 14.93, 14.95 [05:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [05:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [05:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1444872 [05:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [05:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:45:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 26319 MB (2% inode=97%): [05:51:19] Does anyone know what host and dbname the wikipedia1.0 data lives in? [05:52:32] i.e. the article assessments [05:53:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1444256.000000 [06:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:02:58] kaldari: I believe it is u_cbm_p on thyme? [06:03:15] thanks! [06:03:24] I'll check there [06:27:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1443269 [06:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:36:52] I thought it was running under its own multi-maintainer tool now. [06:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:53:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1441958.000000 [07:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1439409 [07:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:53:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1438306.000000 [08:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1435991 [08:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:53:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1434680.000000 [09:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:25:52] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 16.230 second response time [09:27:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1431830 [09:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:42:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [09:42:52] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 17.950 second response time [09:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:53:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1430708.000000 [09:55:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [10:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:28:12] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:28:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:28:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:30:02] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39475 MB (9% inode=99%): [10:30:03] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 86927 MB (8% inode=98%): [10:30:03] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 86927 MB (8% inode=98%): [10:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1429646 [10:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:53:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1428571.000000 [10:55:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [10:55:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:55:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:55:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:57:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 88994 MB (9% inode=98%): [10:57:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 88994 MB (9% inode=98%): [10:57:44] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39211 MB (9% inode=99%): [11:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:01:04] Russell Blau * Re: [Toolserver-l] s1 replag update, suggestion, and question [11:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1427163 [11:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:37:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:52:08] @replag [11:52:09] russblau: s1-user: 2w 2d 12h 2m 21s [-1.02 s/s]; s7-rr-a: 17s [-0.00 s/s]; s7-user: 17s [-0.00 s/s] [11:53:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1425620.000000 [11:55:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [12:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:11:57] @replag [12:12:12] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:12:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:12:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:13:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 90058 MB (9% inode=98%): [12:13:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 90058 MB (9% inode=98%): [12:13:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39148 MB (9% inode=99%): [12:14:47] russblau: s1-user: 2w 2d 11h 54m 34s [-0.39 s/s]; s3-rr-a: 3m 14s [+0.04 s/s]; s3-user: error; s4-user: 4m 52s [+0.00 s/s]; s6-rr-a: 5m 38s [+0.02 s/s]; s6-user: error; s7-rr-a: 7m 8s [+0.30 s/s]; s7-user: error [12:16:43] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 95060 MB (9% inode=99%): [12:17:50] :/ http://status.toolserver.org/ is down [12:19:04] John * [Toolserver-l] http://status.toolserver.org/ Down [12:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1424326 [12:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:53:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1423304.000000 [12:55:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [12:55:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:56:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 90067 MB (9% inode=98%): [12:56:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 90065 MB (9% inode=98%): [12:56:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39122 MB (9% inode=99%): [13:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:22:47] @replag [13:22:47] Dispenser: s1-user: 2w 2d 10h 51m 24s [-0.89 s/s]; s3-rr-a: 1m 41s [-0.02 s/s]; s3-user: 1m 41s [+0.01 s/s] [13:27:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1420953 [13:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:53:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1419985.000000 [13:54:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:54:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:54:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:55:33] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - 170 bytes in 0.534 second response time [13:55:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 89802 MB (9% inode=98%): [13:55:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 89802 MB (9% inode=98%): [13:55:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39061 MB (9% inode=99%): [13:56:43] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 274 bytes in 10.368 second response time [14:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:07:27] @replag [14:07:27] russblau: s1-user: 2w 2d 10h 18m 40s [-0.73 s/s]; s3-rr-a: 2m 30s [+0.02 s/s]; s3-user: 2m 30s [+0.02 s/s] [14:27:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:29:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:29:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37875 MB (9% inode=99%): [14:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1418925 [14:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:37:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:53:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1418152.000000 [15:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:22:22] @replag [15:22:22] Brooke: s1-user: 2w 2d 9h 43m 44s [-0.47 s/s]; s3-rr-a: 55s [-0.02 s/s]; s3-user: 55s [-0.02 s/s] [15:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1417028 [15:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:53:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1416360.000000 [16:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:20:05] Krinkle * Re: [Toolserver-l] http://status.toolserver.org/ Down [16:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [16:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [16:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1415012 [16:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [16:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:53:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1414073.000000 [17:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:29:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1412963 [17:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:45:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 28837 MB (2% inode=97%): [17:50:53] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 273 bytes in 20.076 second response time [17:51:52] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 18.550 second response time [17:53:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1412460.000000 [17:56:03] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 274 bytes in 20.760 second response time [18:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:17:53] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 17.200 second response time [18:29:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:29:57] @replag [18:29:58] se4598: s1-user: 2w 2d 8h 6m 53s [-0.40 s/s]; s2-user-c: 2m 46s [+0.00 s/s]; s3-rr-a: 58s [+0.00 s/s]; s3-user: 58s [+0.00 s/s]; s5-user-c: 2m 47s [+0.00 s/s] [18:31:02] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 273 bytes in 20.931 second response time [18:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:31:53] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 18.840 second response time [18:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1411451 [18:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:46:23] Load avg. on willow is WARNING: WARNING - load average: 19.82, 17.26, 13.45 [18:48:23] Load avg. on willow is OK: OK - load average: 11.13, 14.72, 12.95 [18:49:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:52:53] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 19.812 second response time [18:53:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1410615.000000 [18:56:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:17:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:29:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1409158 [19:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:53:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1408488.000000 [20:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:17:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [20:29:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1407447 [20:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:53:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1407167.000000 [21:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:14:43] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24101 MB (5% inode=99%): [21:17:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [21:29:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1406211 [21:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:40:10] okaay, thanks [21:40:18] @replag [21:40:20] se4598: s1-user: 2w 2d 6h 35m 4s [-0.44 s/s]; s2-user: 14s [+0.00 s/s]; s3-rr-a: 15s [-0.00 s/s]; s3-user: 15s [-0.00 s/s]; s5-user: 10s [-] [21:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:46:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42235 MB (10% inode=99%): [21:53:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1405824.000000 [22:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:17:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [22:23:08] @replag [22:23:08] Brooke: s1-user: 2w 2d 6h 25m 46s [-0.16 s/s]; s3-rr-a: 30s [-0.00 s/s]; s3-user: 30s [-0.00 s/s] [22:23:55] degraded [22:24:07] vs. last time above and yesterday [22:29:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [22:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [22:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1405403 [22:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [22:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:53:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1404947.000000 [23:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:17:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [23:29:04] DeltaQuad * Re: [Toolserver-l] http://status.toolserver.org/ Down [23:29:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [23:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [23:31:48] @replag [23:31:49] russblau: s1-user: 2w 2d 6h 2m 53s [-0.33 s/s]; s2-user: 18s [+0.00 s/s]; s3-rr-a: 24s [-0.00 s/s]; s3-user: 24s [-0.00 s/s] [23:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1404097 [23:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:37:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [23:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:53:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1403755.000000