[00:05:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:32:42] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [00:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 680200 [00:40:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30234 [00:41:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:43:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 477827 MB (9% inode=43%): [00:43:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:48:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:53:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30011.000000 [00:58:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 681104.000000 [01:06:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:06:42] Load avg. on willow is WARNING: WARNING - load average: 16.45, 16.30, 15.31 [01:23:43] Load avg. on willow is OK: OK - load average: 13.85, 14.61, 14.96 [01:32:42] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [01:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 682536 [01:40:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26761 [01:41:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:43:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 477766 MB (9% inode=43%): [01:43:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:49:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:53:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25989.000000 [01:58:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 683403.000000 [02:06:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:06:43] Load avg. on willow is WARNING: WARNING - load average: 16.29, 16.12, 15.22 [02:14:09] hey all [02:14:15] any maps people here? [02:14:48] I'm having a slow query on the osm_mapnik database [02:15:18] it should work in principle, as the tiles do get rendered [02:15:35] just when I'm trying to fetch the data myself it craps out [02:20:42] /sql on thyme is WARNING: DISK WARNING - free space: /sql 119036 MB (12% inode=99%): [02:32:42] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [02:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 684767 [02:41:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:41:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21393 [02:43:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 476032 MB (8% inode=43%): [02:43:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:49:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:53:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20920.000000 [02:56:41] Load avg. on willow is OK: OK - load average: 12.45, 13.30, 14.94 [02:58:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 685653.000000 [03:06:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:06:42] Load avg. on willow is WARNING: WARNING - load average: 18.43, 16.97, 16.09 [03:14:02] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42428 MB (10% inode=99%): [03:15:01] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 50558 MB (12% inode=99%): [03:32:42] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [03:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 687177 [03:41:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15370 [03:42:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:43:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 474886 MB (8% inode=43%): [03:43:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:49:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:53:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14377.000000 [03:58:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 688253.000000 [04:06:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:06:42] Load avg. on willow is WARNING: WARNING - load average: 18.52, 18.84, 17.57 [04:30:20] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=7464&oldid=7451&rcid=10164 * 91.198.174.202 * (+1) (updated page) [04:32:42] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [04:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 689841 [04:41:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10128 [04:42:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:43:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 475319 MB (8% inode=43%): [04:43:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:49:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:53:12] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9976.000000 [04:58:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 690641.000000 [05:00:42] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.03, 19.21, 17.47 [05:01:43] Load avg. on willow is WARNING: WARNING - load average: 22.17, 18.96, 17.50 [05:06:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:31:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 43741 MB (3% inode=98%): [05:32:41] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [05:32:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.06, 22.35, 18.46 [05:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 691883 [05:36:43] Load avg. on willow is WARNING: WARNING - load average: 16.38, 22.11, 19.53 [05:41:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6208 [05:43:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6279 [05:43:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:43:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:43:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 475192 MB (8% inode=43%): [05:49:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:52:33] Load avg. on willow is OK: OK - load average: 8.58, 11.80, 14.79 [05:53:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5427.000000 [05:59:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 692761.000000 [06:12:04] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3577 [06:12:33] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3528.000000 [06:26:33] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1754.000000 [06:27:03] MySQL slave on rosemary is OK: Uptime: 8953777 Threads: 14 Questions: 3603281897 Slow queries: 656156 Opens: 15447 Flush tables: 6 Open tables: 713 Queries per second avg: 402.431 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1656 [06:42:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 694487 [06:42:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:43:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:43:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:44:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 475076 MB (8% inode=43%): [06:49:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:59:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 695401.000000 [07:42:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 696949 [07:42:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:43:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:43:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:44:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 475031 MB (8% inode=43%): [07:49:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:59:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 697528.000000 [08:42:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 699007 [08:43:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:43:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:43:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:44:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 474939 MB (8% inode=43%): [08:49:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:59:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 699602.000000 [09:03:36] [[Special:Log/newusers]] create 10 * Brynspar * (New user account) [09:42:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 701103 [09:43:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:44:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:44:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:45:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 474855 MB (8% inode=43%): [09:49:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:59:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 701691.000000 [10:29:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 99142 MB (10% inode=98%): [10:29:54] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 99140 MB (10% inode=98%): [10:42:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 703055 [10:43:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:44:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:44:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:45:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 474745 MB (8% inode=43%): [10:49:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:59:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 703630.000000 [11:42:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 705323 [11:43:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:45:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:45:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:45:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 474120 MB (8% inode=43%): [11:49:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:59:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 705901.000000 [12:13:54] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40919 MB (10% inode=99%): [12:39:42] @replag [12:39:42] Dispenser: s1-user: 1w 1d 4h 28m 31s [+0.58 s/s]; s3-rr-a: 51s [+0.00 s/s]; s3-user: 51s [+0.00 s/s] [12:42:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 707409 [12:43:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:45:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:45:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 474268 MB (8% inode=43%): [12:46:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:49:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:56:11] @replag [12:56:11] DarkoNeko: s1-user: 1w 1d 4h 37m 37s [+0.55 s/s]; s3-rr-a: 19s [-0.03 s/s]; s3-user: 19s [-0.03 s/s]; s6-rr-a: 39s [+0.00 s/s]; s6-user: 39s [+0.00 s/s] [12:59:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 707962.000000 [13:00:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23782 MB (5% inode=99%): [13:42:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 709532 [13:43:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:45:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:45:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 474156 MB (8% inode=43%): [13:45:35] [[Special:Log/newusers]] create 10 * Davyk * (New user account) [13:46:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:49:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:50:19] [[Database access]] !M 10https://wiki.toolserver.org/w/index.php?diff=7465&oldid=7237&rcid=10167 * Davyk * (+26) (/* PHP */ ) [13:57:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:57:23] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:58:24] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 99849 MB (10% inode=98%): [13:58:24] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 99847 MB (10% inode=98%): [13:59:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 710085.000000 [14:19:54] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 49919 MB (12% inode=99%): [14:21:13] /sql on thyme is WARNING: DISK WARNING - free space: /sql 110223 MB (11% inode=99%): [14:42:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 711463 [14:43:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:45:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:46:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:46:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 474055 MB (8% inode=43%): [14:49:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:59:34] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 711905.000000 [15:13:14] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 106152 MB (10% inode=99%): [15:42:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 713154 [15:43:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:45:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:46:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:47:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 474453 MB (8% inode=43%): [15:49:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:51:55] @replag [15:51:55] Dispenser: s1-user: 1w 1d 6h 10m 35s [+0.53 s/s]; s2-user: 11s [-0.00 s/s]; s3-rr-a: 24s [+0.00 s/s]; s3-user: 24s [+0.00 s/s] [15:59:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 713628.000000 [16:42:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 714826 [16:44:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:45:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:46:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:47:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 474312 MB (8% inode=43%): [16:49:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:59:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 715208.000000 [17:42:23] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 44465 MB (3% inode=98%): [17:42:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 716573 [17:44:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:45:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:46:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:48:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 474217 MB (8% inode=43%): [17:49:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:59:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 717179.000000 [18:09:54] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40087 MB (9% inode=99%): [18:35:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23778 MB (5% inode=99%): [18:42:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 719509 [18:44:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:44:54] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 53677 MB (13% inode=99%): [18:45:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:46:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:48:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 474145 MB (8% inode=43%): [18:49:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:59:44] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 720371.000000 [19:35:30] [[Special:Log/newusers]] create 10 * TanishaC * (New user account) [19:42:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 721712 [19:44:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:45:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:46:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:48:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 473973 MB (8% inode=43%): [19:50:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:00:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 722210.000000 [20:43:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 723682 [20:44:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:46:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:46:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:48:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 473876 MB (8% inode=43%): [20:49:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [20:50:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:00:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 724193.000000 [21:19:04] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2197.000000 [21:43:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3637.000000 [21:43:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 725639 [21:44:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:46:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:46:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:48:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 473788 MB (8% inode=43%): [21:49:04] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [21:50:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:59:34] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [21:59:34] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [22:00:33] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [22:00:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 726114.000000 [22:03:43] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [22:29:03] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2188.000000 [22:29:33] s4 replag on z-dat-s4-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2212.000000 [22:43:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7238.000000 [22:44:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 727606 [22:44:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:47:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:47:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:48:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 473477 MB (8% inode=43%): [22:49:04] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [22:50:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:53:03] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3629.000000 [22:53:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3652.000000 [22:59:33] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [23:00:33] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [23:00:33] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [23:00:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 728106.000000 [23:03:43] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [23:17:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39615 MB (9% inode=99%): [23:39:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24388 MB (5% inode=99%): [23:43:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10838.000000 [23:44:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 729685 [23:44:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:48:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:48:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:48:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 472045 MB (8% inode=43%): [23:49:04] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [23:49:53] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 49535 MB (12% inode=99%): [23:50:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:53:03] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7228.000000 [23:54:33] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [23:59:33] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds)