[00:16:57] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 98523 MB (10% inode=99%): [00:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:30:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:43:37] Load avg. on willow is WARNING: WARNING - load average: 15.93, 16.13, 15.12 [00:48:36] Load avg. on willow is OK: OK - load average: 11.27, 14.34, 14.74 [00:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:54:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1271761 [00:57:46] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1271711.000000 [01:29:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:30:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:54:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1270239 [01:57:46] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1269924.000000 [02:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:30:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:34:37] Load avg. on willow is WARNING: WARNING - load average: 14.09, 17.47, 15.91 [02:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:53:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:54:37] Load avg. on willow is OK: OK - load average: 13.16, 14.02, 14.99 [02:54:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1265333 [02:57:46] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1265053.000000 [03:06:36] Load avg. on willow is WARNING: WARNING - load average: 17.03, 16.93, 16.34 [03:10:36] Load avg. on willow is CRITICAL: CRITICAL - load average: 37.30, 21.65, 18.09 [03:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:30:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:30:36] Load avg. on willow is WARNING: WARNING - load average: 22.37, 17.34, 17.02 [03:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:49:17] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:51:36] Load avg. on willow is OK: OK - load average: 13.17, 13.93, 14.95 [03:53:16] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:53:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:54:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1262566 [03:57:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1262492.000000 [04:13:57] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:30:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:30:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:36:36] Load avg. on willow is WARNING: WARNING - load average: 14.98, 15.41, 15.00 [04:37:37] Load avg. on willow is OK: OK - load average: 13.20, 14.84, 14.82 [04:40:26] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:40:26] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:40:27] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:40:57] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 88148 MB (9% inode=98%): [04:40:57] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37783 MB (9% inode=99%): [04:40:57] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 88168 MB (9% inode=98%): [04:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:54:36] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1260363 [04:56:36] Load avg. on willow is WARNING: WARNING - load average: 14.59, 15.62, 15.29 [04:57:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1260143.000000 [05:20:37] Load avg. on willow is WARNING: WARNING - load average: 16.91, 16.81, 17.15 [05:22:27] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:22:27] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:22:27] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:23:06] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 88196 MB (9% inode=98%): [05:23:07] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37765 MB (9% inode=99%): [05:23:07] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 88196 MB (9% inode=98%): [05:29:37] Load avg. on willow is OK: OK - load average: 10.86, 13.02, 14.96 [05:29:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:30:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:46:07] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 13617 MB (1% inode=95%): [05:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:54:26] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:54:26] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:54:26] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:54:36] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1258079 [05:55:57] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 88146 MB (9% inode=98%): [05:55:57] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37757 MB (9% inode=99%): [05:55:57] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 88146 MB (9% inode=98%): [05:57:56] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1257861.000000 [06:25:17] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 141911 MB (14% inode=99%): [06:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:30:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:32:37] Load avg. on willow is WARNING: WARNING - load average: 16.26, 18.27, 16.32 [06:53:16] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:54:36] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1255789 [06:57:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1255531.000000 [06:58:36] Load avg. on willow is OK: OK - load average: 11.09, 13.27, 14.81 [07:11:37] Load avg. on willow is WARNING: WARNING - load average: 18.90, 17.08, 15.75 [07:17:37] Load avg. on willow is OK: OK - load average: 9.83, 13.48, 14.70 [07:29:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:30:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:54:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1251645 [07:57:56] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1251300.000000 [08:08:56] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23918 MB (5% inode=99%): [08:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:30:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:30:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:36:56] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39549 MB (9% inode=99%): [08:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:53:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:54:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1248847 [08:55:26] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:55:27] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:55:27] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:55:57] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 87284 MB (8% inode=98%): [08:55:57] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38570 MB (9% inode=99%): [08:55:57] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 87284 MB (8% inode=98%): [08:58:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1248442.000000 [09:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:30:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:31:27] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:31:27] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:31:27] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:31:57] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 86655 MB (8% inode=98%): [09:31:57] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38550 MB (9% inode=99%): [09:31:57] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 86655 MB (8% inode=98%): [09:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:54:36] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1243951 [09:58:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1243573.000000 [10:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:30:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:53:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:54:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1240520 [10:58:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1240135.000000 [11:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:30:06] DaB. * Re: [Toolserver-announce] [Toolserver-l] sql-s1 will be read-only on Wednesday [11:30:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:34:06] DaB. * Re: [Toolserver-l] sql-s1 will be read-only on Wednesday [11:44:04] Mohamed ElGedawy * [Toolserver-l] Please approve [11:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:53:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:54:27] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:54:27] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:54:27] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:54:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1237656 [11:54:57] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 87309 MB (8% inode=98%): [11:54:57] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38440 MB (9% inode=99%): [11:54:57] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 87308 MB (8% inode=98%): [11:58:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1237881.000000 [12:04:05] Platonides * Re: [Toolserver-l] sql-s1 will be read-only on Wednesday [12:16:56] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 98053 MB (10% inode=99%): [12:23:57] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2038 [12:24:26] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2055.000000 [12:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:30:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:42:21] @replag [12:42:21] Dispenser: s1-rr-a: 46m 1s [+0.01 s/s]; s1-user: 46m 1s [-25.04 s/s]; s2-user: 14s [-0.18 s/s] [12:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:54:36] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1236422 [12:58:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1236045.000000 [13:06:27] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3610.000000 [13:06:56] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3625 [13:13:43] @replag [13:13:43] closedmouth: s1-rr-a: 1h 5m 6s [+0.61 s/s]; s1-user: 1h 5m 6s [+0.61 s/s]; s2-user: 10m 20s [+0.32 s/s]; s2-user-c: 31s [+0.00 s/s]; s5-user-c: 31s [+0.00 s/s] [13:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:30:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:30:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:54:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1228901 [13:58:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1228322.000000 [14:06:27] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4585.000000 [14:06:57] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4597 [14:29:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:30:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:53:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:54:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1221440 [14:58:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1221006.000000 [15:06:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5335.000000 [15:06:57] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5350 [15:26:27] @replag [15:26:27] jeremyb: s1-rr-a: 1h 36m 49s [+0.24 s/s]; s1-user: 1h 36m 49s [+0.24 s/s]; s3-rr-a: 1m 22s [-0.00 s/s]; s3-user: 1m 22s [-0.00 s/s] [15:26:36] + ???? [15:26:43] oh, that's only 1 hr though [15:26:50] no more 2 weeks! [15:27:41] oh, i guess it's still 2 weeks it's just not monitored by tsbot any more [15:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:30:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:30:54] http://munin.toolserver.org/Database/thyme/mysql_replication.html [15:30:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:32:57] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24358 MB (5% inode=99%): [15:53:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:54:36] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1213780 [15:55:57] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41645 MB (10% inode=99%): [15:58:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1213340.000000 [15:59:20] @replag [15:59:20] Jarry1250: s1-rr-a: 1h 51m 32s [+0.45 s/s]; s1-user: 1h 51m 32s [+0.45 s/s]; s2-user-c: 1m 42s [+0.01 s/s]; s3-rr-a: 1m 58s [+0.02 s/s]; s3-user: 1m 58s [+0.02 s/s]; s5-user-c: 1m 42s [+0.01 s/s] [16:06:27] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6855.000000 [16:06:57] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6871 [16:09:16] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:30:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:30:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:48:47] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:53:16] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:54:36] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1205865 [16:58:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1205386.000000 [17:06:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8259.000000 [17:06:57] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8266 [17:11:34] why is rosemary getting lagged now? [17:11:36] @replag [17:11:36] jeremyb: s1-rr-a: 2h 19m 46s [+0.39 s/s]; s1-user: 2h 19m 46s [+0.39 s/s]; s2-user: 43s [-0.04 s/s]; s3-rr-a: 15s [-0.02 s/s]; s3-user: 15s [-0.02 s/s] [17:12:00] and no DaB [17:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:30:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:30:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:46:07] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 16264 MB (1% inode=96%): [17:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:54:36] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1201029 [17:58:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1200491.000000 [18:06:57] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10098 [18:07:16] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10111.000000 [18:25:17] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 139461 MB (14% inode=99%): [18:25:57] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24018 MB (5% inode=99%): [18:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:30:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:53:06] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2162.000000 [18:53:16] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:54:36] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1192744 [18:55:57] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 25376 MB (6% inode=99%): [18:58:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1192277.000000 [18:59:56] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24385 MB (5% inode=99%): [19:06:57] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13518 [19:07:16] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13536.000000 [19:09:34] @replag [19:09:35] Earwig: s1-rr-a: 3h 48m 3s [+0.75 s/s]; s1-user: 3h 48m 4s [+0.75 s/s]; s3-rr-a: 60s [+0.01 s/s]; s3-user: 60s [+0.01 s/s] [19:09:53] err. [19:17:47] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3615.000000 [19:25:57] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 26111 MB (6% inode=99%): [19:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:30:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:30:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:32:57] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24258 MB (5% inode=99%): [19:39:37] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3467.000000 [19:45:37] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1481.000000 [19:46:57] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58407 MB (5% inode=97%): [19:46:57] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58407 MB (5% inode=97%): [19:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:54:36] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1184413 [19:58:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1183765.000000 [20:01:57] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 62276 MB (6% inode=97%): [20:01:57] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 62276 MB (6% inode=97%): [20:02:57] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36006 MB (8% inode=99%): [20:06:57] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16027 [20:07:57] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16067.000000 [20:16:57] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24351 MB (5% inode=99%): [20:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:30:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:30:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:45:56] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38690 MB (9% inode=99%): [20:53:16] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:54:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1177600 [20:58:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1177237.000000 [21:06:57] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17792 [21:07:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17828.000000 [21:29:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:30:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:54:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1167658 [21:57:48] @replag [21:57:49] russblau: s1-rr-a: 5h 17m 39s [+0.53 s/s]; s1-user: 5h 17m 39s [+0.53 s/s]; s6-rr-a: 13s [-0.00 s/s]; s6-user: 13s [-0.00 s/s] [21:58:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1167077.000000 [22:06:56] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19216 [22:07:57] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19241.000000 [22:20:43] greetings [22:21:01] can someone tell me how may i track the webserver log for something I have at the toolserver? [22:21:11] or the php log, to be sure... [22:21:57] Free Memory on damiana is WARNING: WARNING - 5.3% (443788 kB) free! [22:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:30:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:31:57] Free Memory on damiana is OK: OK - 7.3% (610176 kB) free. [22:37:34] web logs are in /var/log/http/ on ortelius and wolfsbane servers [22:46:57] Free Memory on damiana is WARNING: WARNING - 6.8% (572916 kB) free! [22:47:57] Free Memory on damiana is OK: OK - 7.1% (591888 kB) free. [22:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:54:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1158589 [22:55:01] Dispenser, thanks.... but I still can't see the errors in there.. [22:55:08] maybe the PHP log is sent somewhere else? [22:55:34] care to have a look? http://toolserver.org/~ntavares/patrimonio/api/api.php?action=statistics&stcountry=ad&format=html&limit=0 [22:56:00] this stopped working .. [22:56:46] same here: http://toolserver.org/~erfgoed/api/api.php?action=statistics&stcountry=at&format=html&limit=0 [22:57:08] i already put error_reporting(E_ALL); display_errors=on [22:57:11] to no avail.. [22:58:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1158126.000000 [22:59:28] Dispenser, before wasting your time, let me check another thing.. [23:04:33] Panora200, the php error server is indeed in /var/log [23:05:19] see /var/log/userlog [23:06:30] there doesn't seem to be errors by your tools [23:06:57] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19728 [23:07:48] Platonides, yup, weird... but I concluded that as well [23:08:09] i think their just empty record sets.. [23:08:27] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19771.000000 [23:08:30] what is giving you an error? [23:11:57] nothing so far.... the statistics module is returning empty results [23:12:41] which is not ok... [23:15:52] ermmm i think i found the problem lol [23:17:06] someone cleared the statistics table.. :/ [23:20:28] hmmm Platonides the monuments_all got 800k records... [23:21:03] that's nothing... and it's taking a lot to group by [23:21:14] maybe we should keep it warmed up eheh [23:29:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:30:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:30:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:53:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:53:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:54:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1154548 [23:58:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1154083.000000