[00:03:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31532 [00:15:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:15:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 428919 [00:21:38] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 429121.000000 [00:21:48] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 508538 MB (9% inode=45%): [00:23:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31771.000000 [00:28:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:43:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:55:28] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40016 MB (9% inode=99%): [01:03:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32645 [01:15:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:15:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 431214 [01:16:29] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23801 MB (5% inode=99%): [01:21:38] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 431402.000000 [01:21:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 510930 MB (9% inode=45%): [01:23:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32857.000000 [01:23:28] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 51621 MB (12% inode=99%): [01:29:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:04:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33260 [02:15:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:15:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 433298 [02:17:49] /sql on thyme is WARNING: DISK WARNING - free space: /sql 148975 MB (15% inode=99%): [02:21:38] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 433491.000000 [02:21:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 509912 MB (9% inode=45%): [02:23:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33331.000000 [02:28:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:43:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:05:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38559 MB (9% inode=99%): [03:05:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33961 [03:15:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:15:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 435275 [03:16:29] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23373 MB (5% inode=99%): [03:21:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 55908 MB (13% inode=99%): [03:21:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 435438.000000 [03:21:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 509333 MB (9% inode=45%): [03:23:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34109.000000 [03:28:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:43:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:05:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34621 [04:15:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:15:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 437478 [04:21:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 508908 MB (9% inode=45%): [04:22:38] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 437695.000000 [04:23:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34786.000000 [04:28:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:30:13] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=7451&oldid=7435&rcid=10124 * 91.198.174.202 * (+17) (updated page) [04:43:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:06:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35706 [05:15:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:15:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 439367 [05:21:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 508814 MB (9% inode=45%): [05:22:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 439536.000000 [05:23:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35982.000000 [05:26:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40627 MB (9% inode=99%): [05:27:48] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 50965 MB (4% inode=98%): [05:28:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:43:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:52:28] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23618 MB (5% inode=99%): [05:59:28] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 51481 MB (12% inode=99%): [06:06:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36516 [06:15:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:15:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 441155 [06:21:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 508739 MB (9% inode=45%): [06:22:38] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 441355.000000 [06:23:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36677.000000 [06:28:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:31:28] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:31:29] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:43:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:06:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37078 [07:15:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:15:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 442997 [07:21:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 508657 MB (9% inode=45%): [07:22:37] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 443182.000000 [07:24:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37361.000000 [07:28:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:31:28] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:31:28] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:43:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:06:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38003 [08:15:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:15:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:16:17] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 444722 [08:21:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 508584 MB (9% inode=45%): [08:22:38] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 444898.000000 [08:24:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38226.000000 [08:28:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:29:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 127216 MB (20% inode=99%): [08:31:28] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:31:29] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:43:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:48:31] hi [09:06:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38711 [09:11:47] [[Special:Log/newusers]] create 10 * NoelQkz * (New user account) [09:15:18] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2179.000000 [09:15:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:15:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:16:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 446336 [09:21:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 508439 MB (9% inode=45%): [09:22:38] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 446516.000000 [09:24:48] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39075.000000 [09:28:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:30:29] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 102793 MB (10% inode=98%): [09:30:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42931 MB (10% inode=99%): [09:30:38] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 102731 MB (10% inode=98%): [09:30:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126959 MB (20% inode=99%): [09:31:29] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:31:29] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:33:18] Load avg. on adenia is WARNING: WARNING - load average: 21.00, 17.02, 10.59 [09:44:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3607.000000 [09:44:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:47:18] Load avg. on adenia is OK: OK - load average: 7.93, 13.58, 13.10 [09:59:18] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3588.000000 [10:06:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39614 [10:12:29] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24273 MB (5% inode=99%): [10:14:19] Load avg. on adenia is WARNING: WARNING - load average: 18.01, 17.21, 15.59 [10:15:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:15:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 448270 [10:21:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 508329 MB (9% inode=45%): [10:22:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 448496.000000 [10:24:48] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39745.000000 [10:28:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:30:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126751 MB (20% inode=99%): [10:31:28] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:31:28] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:40:18] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3606.000000 [10:44:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:54:18] Load avg. on adenia is OK: OK - load average: 8.40, 12.07, 14.84 [10:57:18] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3581.000000 [11:04:18] Load avg. on adenia is WARNING: WARNING - load average: 18.45, 16.63, 15.67 [11:06:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40117 [11:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 450030 [11:16:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:16:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:21:18] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1781.000000 [11:21:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 508254 MB (9% inode=45%): [11:22:18] Load avg. on adenia is OK: OK - load average: 10.58, 13.68, 14.96 [11:22:58] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 450326.000000 [11:24:48] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40301.000000 [11:26:59] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:27:08] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:28:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:30:38] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 102038 MB (10% inode=98%): [11:30:39] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 102038 MB (10% inode=98%): [11:30:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126526 MB (20% inode=99%): [11:31:28] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:31:28] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:36:07] [[Special:Log/newusers]] create 10 * SusannaCO * (New user account) [11:45:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:53:58] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:54:08] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:56:28] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 103318 MB (10% inode=98%): [11:56:38] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 103318 MB (10% inode=98%): [12:06:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40932 [12:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 452229 [12:17:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:17:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:21:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 506282 MB (9% inode=44%): [12:22:58] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 452428.000000 [12:24:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41067.000000 [12:28:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:30:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126167 MB (20% inode=99%): [12:32:29] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:32:29] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:45:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:46:05] I need to fix this check… [12:59:49] [[Special:Log/newusers]] create 10 * GeraldoD * (New user account) [13:02:52] hello [13:02:58] i have to reboot a switch [13:03:08] whole toolserver might not be there for 5 min [13:03:11] at least [13:03:16] cross fingers.,.. [13:08:31] ok everything back up after a short while it seems [13:10:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:11:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:11:29] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 103105 MB (10% inode=98%): [13:11:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41368 [13:11:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126047 MB (20% inode=99%): [13:11:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 506580 MB (9% inode=44%): [13:11:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 453855 [13:17:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:22:58] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 453989.000000 [13:24:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41393.000000 [13:32:28] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:32:28] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:34:29] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:35:29] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:44:28] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [14:10:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:11:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41819 [14:11:48] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125803 MB (20% inode=99%): [14:11:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 506340 MB (9% inode=44%): [14:11:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:12:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:16:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 455327 [14:17:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:17:49] /sql on thyme is WARNING: DISK WARNING - free space: /sql 147023 MB (15% inode=99%): [14:23:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 455467.000000 [14:25:48] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41887.000000 [14:31:38] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 16.161 second response time [14:42:30] [[Special:Log/newusers]] create 10 * OKaren * (New user account) [14:45:29] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [15:10:38] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 274 bytes in 20.154 second response time [15:10:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:11:38] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 18.503 second response time [15:11:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42521 [15:11:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125519 MB (20% inode=99%): [15:11:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 506207 MB (9% inode=44%): [15:11:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:12:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:14:38] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 272 bytes in 20.836 second response time [15:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 456730 [15:17:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:23:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 456858.000000 [15:25:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42713.000000 [15:40:33] @replag [15:40:34] liangent: s1-rr-a: 11h 56m 15s [+0.17 s/s]; s1-user: 5d 6h 59m 45s [+0.39 s/s]; s2-user: 7m 17s [-0.29 s/s]; s3-rr-a: 26s [+0.00 s/s]; s3-user: 26s [+0.00 s/s] [15:45:29] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [16:07:45] hmm, I try to svn commit, but it complains no editor is set. However I did set the enviroment variables [16:10:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:11:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43457 [16:11:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125170 MB (20% inode=99%): [16:11:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 506271 MB (9% inode=44%): [16:11:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:12:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 458069 [16:17:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:23:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 458234.000000 [16:25:48] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43454.000000 [16:29:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 67449 MB (16% inode=99%): [16:37:38] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [16:45:28] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [17:10:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:11:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43699 [17:11:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124948 MB (20% inode=99%): [17:11:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505658 MB (9% inode=44%): [17:11:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:12:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 459443 [17:17:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:23:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 459620.000000 [17:25:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43729.000000 [17:27:49] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 50539 MB (4% inode=98%): [17:37:38] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [17:46:29] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [17:46:50] is it me, or does user table on metawiki_p really miss a huge chunk of data? [18:10:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:11:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44188 [18:11:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124736 MB (20% inode=99%): [18:11:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505558 MB (9% inode=44%): [18:11:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:12:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 460940 [18:17:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:23:28] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 461119.000000 [18:25:58] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44314.000000 [18:26:58] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:27:08] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:27:29] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 102612 MB (10% inode=98%): [18:27:38] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 102612 MB (10% inode=98%): [18:29:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42621 MB (10% inode=99%): [18:35:58] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:37:38] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:40:58] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:40:58] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:41:08] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:41:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37226 MB (9% inode=99%): [18:45:38] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [18:46:28] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [18:54:58] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:10:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:11:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44555 [19:11:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124481 MB (20% inode=99%): [19:11:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505410 MB (9% inode=44%): [19:11:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:12:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:13:38] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 102540 MB (10% inode=98%): [19:14:29] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 102539 MB (10% inode=98%): [19:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 462479 [19:17:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:18:38] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 19.934 second response time [19:19:38] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:23:28] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 462677.000000 [19:25:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44773.000000 [19:46:29] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [20:10:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:11:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 45307 [20:11:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124168 MB (20% inode=99%): [20:11:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505339 MB (9% inode=44%): [20:11:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:12:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 464130 [20:17:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:19:38] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [20:23:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 464295.000000 [20:26:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45409.000000 [20:44:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41428 MB (10% inode=99%): [20:46:28] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [21:10:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:11:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 45974 [21:11:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123839 MB (20% inode=99%): [21:11:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505194 MB (9% inode=44%): [21:11:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:13:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:16:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 465712 [21:17:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:19:38] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [21:24:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 465905.000000 [21:26:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46026.000000 [21:40:28] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24383 MB (5% inode=99%): [21:46:29] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [22:10:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:11:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46476 [22:11:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123306 MB (20% inode=99%): [22:11:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 502801 MB (9% inode=44%): [22:11:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:13:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:16:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 467123 [22:18:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:19:27] dungodung: What data is missing? [22:19:38] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [22:19:43] Akoopal: Did you figure out? If not, what's the exact error message? [22:19:49] figure it * [22:24:11] Brooke: yep [22:24:28] works now [22:24:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 467291.000000 [22:24:48] Cool. :-) [22:26:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46655.000000 [22:26:45] hmm, looked a bit better now, wtf is /opt/local/bin/setenv :-) [22:26:55] (yes, I am a tcsh user) [22:30:34] Brooke: metawiki_p tables at sql-s3 are missing some rows, but everything's alright at sql-s7 [22:34:29] dungodung: Hmm, you should file a JIRA ticket. [22:34:33] If there isn't one already. [22:34:38] I know enwiki_p on thyme is corrupt. [22:34:41] I can't be bothered [22:34:44] I haven't heard about metawiki_p. [22:34:53] at least not today [22:34:58] for I shall now sleep :P [22:34:58] :-) [22:35:14] Akoopal, I think setenv is used to set environment variables [22:35:16] metawiki moved from s3 to s7 long time ago – but the wmf-devs are unwilling to drop it on s3 [22:35:35] Platonides: in tcsh indeed [22:35:44] so I am used to that [22:36:04] normally I am triggered that I am in bash by an error message [22:36:23] but the script I mentioned causes the command to be silent, but ignored [22:36:36] which is very wrong imho [22:36:43] yes [22:37:30] seems a shell script to export the variables configured in ~/.environment [22:37:51] or rather, to print how to export them [22:38:32] wondering about the need [22:38:36] me too [22:38:57] DaBPunkt: can you quickly look who has a .enviroment in his homedir? [22:38:59] looks like something that would have to be called by another script to be useful [22:39:45] that script was probably river work [22:39:46] alebot/.environment alexz/.environment aude/.environment darkdadaah/.environment dcoetzee/.environment jcreus/.environment kate/.environment magnus/.environment master/.environment mzmcbride/.environment philip/.environment river/.environment rriver/.environment russell/.environment stanlekub/.environment sumurai8/.environment vishwin60/.environment whym/.environment [22:39:54] Hi. [22:40:16] Platonides: likely [22:40:41] specially since he seems to only have test lines on his .environment [22:41:21] is it perhaps called in the dotfiles? [22:41:29] startup files [22:42:44] look at the bottom of /etc/profile [22:42:50] # Set environment from user's $HOME/.environment [22:42:51] if [ -f "/opt/local/bin/setenv" ]; then [22:42:51] eval `/opt/local/bin/setenv` [22:42:51] fi [22:43:52] ahhhh [22:44:46] ok, rename the script and fix /etc/profile would do the trick then [22:46:28] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 61448 MB (15% inode=99%): [22:47:29] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [22:49:34] but bedtime for now, night [22:51:36] night, Akoopal [22:51:55] btw, you may want to set a different PS1 for bash vs tch [23:10:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:11:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46868 [23:11:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123090 MB (20% inode=99%): [23:11:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 502590 MB (9% inode=44%): [23:11:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:13:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:13:34] nacht ts [23:16:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 468369 [23:18:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:19:38] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [23:24:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 468575.000000 [23:25:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40576 MB (9% inode=99%): [23:26:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47007.000000 [23:30:32] Betacommand so I am trying to manage double redirects for all wikis (more or less) [23:30:43] often there are problems that cannot be fixed by bots [23:30:57] I want to have some "control" over this [23:31:18] for instance a tool that displays double redirects not yet fixed or modification of redirect.py would do the trick [23:31:56] ToAruShiroiNeko: We have told you before just parse the logs [23:40:44] I am unable to do this [23:41:16] it wouldnt be much of a problem if pywikipedia provided structured logs [23:41:43] *redirect.py that is [23:43:59] ToAruShiroiNeko: pywiki does [23:44:42] -log:_redirect.log [23:47:29] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL