[00:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:19:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:19:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:20:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:20:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:22:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:22:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 536346 MB (10% inode=45%): [00:22:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 161841 [00:24:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:24:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:35:07] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 177463 [00:37:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:38:25] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [00:43:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 199600 [00:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [00:47:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [00:50:46] Free Memory on turnera is CRITICAL: CRITICAL - 2.0% (167960 kB) free! [00:57:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72331 MB (11% inode=99%): [01:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:19:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:19:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:20:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:20:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:22:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:22:57] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 538378 MB (10% inode=45%): [01:22:57] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 164903 [01:24:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:25:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:32:37] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1899.000000 [01:32:47] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1904 [01:35:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 180318 [01:37:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [01:43:07] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 203151 [01:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [01:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [01:51:47] Free Memory on turnera is CRITICAL: CRITICAL - 2.1% (175764 kB) free! [01:57:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72229 MB (11% inode=99%): [02:11:37] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1794.000000 [02:11:47] MySQL slave on rosemary is OK: Uptime: 14468059 Threads: 16 Questions: 6344220079 Slow queries: 1935498 Opens: 252058 Flush tables: 6 Open tables: 3849 Queries per second avg: 438.498 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1785 [02:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:19:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:19:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:20:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:21:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:23:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:23:57] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 537133 MB (10% inode=45%): [02:23:57] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 168275 [02:24:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:26:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:35:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 183525 [02:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [02:38:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:42:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66690 MB (10% inode=99%): [02:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [02:44:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 206743 [02:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [02:51:47] Free Memory on turnera is CRITICAL: CRITICAL - 1.9% (161232 kB) free! [02:51:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69548 MB (11% inode=99%): [03:02:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 67028 MB (10% inode=99%): [03:07:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67687 MB (11% inode=99%): [03:11:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 67059 MB (10% inode=99%): [03:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:19:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:19:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:20:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:21:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:23:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:24:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:24:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 535643 MB (10% inode=44%): [03:24:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 171431 [03:26:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:35:07] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 186568 [03:38:27] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [03:38:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [03:44:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 210249 [03:47:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [03:51:47] Free Memory on turnera is CRITICAL: CRITICAL - 2.1% (175008 kB) free! [03:54:07] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108252 MB (11% inode=99%): [04:11:07] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106849 MB (10% inode=99%): [04:11:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58156 MB (9% inode=99%): [04:13:06] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107258 MB (11% inode=99%): [04:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:19:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:20:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:20:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:21:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:24:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:24:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:24:57] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 536239 MB (10% inode=45%): [04:24:57] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 174759 [04:26:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:35:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 189817 [04:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [04:38:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [04:44:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 213763 [04:46:06] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106873 MB (10% inode=99%): [04:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [04:51:47] Free Memory on turnera is CRITICAL: CRITICAL - 2.2% (181128 kB) free! [04:56:36] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 37948 MB (9% inode=99%): [04:57:46] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 79529 MB (8% inode=98%): [05:10:06] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:14:18] /aux0 on hemlock is CRITICAL: (Service Check Timed Out) [05:24:36] Free Memory on turnera is OK: OK - 34.7% (2905304 kB) free. [05:31:16] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45532 MB (7% inode=99%): [05:31:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:31:16] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 177858 [05:31:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:31:17] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 536048 MB (10% inode=45%): [05:31:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:31:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:31:27] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 37873 MB (9% inode=99%): [05:31:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:33:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:35:27] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 191968 [05:38:27] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [05:38:54] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:43:27] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [05:44:26] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 217148 [05:47:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:49:27] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [06:03:16] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 12946 MB (1% inode=95%): [06:21:17] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73903 MB (12% inode=99%): [06:21:27] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:22:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:31:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:32:16] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 180943 [06:32:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:32:17] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 535976 MB (10% inode=45%): [06:32:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:32:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:32:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:34:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:35:27] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 194434 [06:38:27] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [06:38:54] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:43:27] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [06:45:27] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 220542 [06:47:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:50:27] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [06:57:16] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 79353 MB (8% inode=98%): [07:12:35] SSH on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:12:44] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [07:22:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72940 MB (11% inode=99%): [07:22:27] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:23:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:31:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:32:15] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 183770 [07:32:15] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 535588 MB (10% inode=44%): [07:32:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:32:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:32:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:32:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:34:26] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:36:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 196810 [07:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [07:38:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:39:27] SSH on nightshade is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze2 (protocol 2.0) [07:39:27] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [07:40:25] Ariel T. Glenn * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [07:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [07:45:26] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 223958 [07:48:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:50:27] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [08:22:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:22:33] @replag [08:22:35] Merlissimo: s1-rr-a: 1m 22s [-0.03 s/s]; s1-user: 1m 23s [-0.03 s/s]; s2-user: 21s [+0.00 s/s]; s3-rr-a: 2d 14h 50m 55s [+0.97 s/s]; s3-user: 2d 14h 50m 55s [+0.97 s/s]; s6-rr-a: 2d 7h 21m 50s [+0.79 s/s]; s6-user: 2d 7h 21m 51s [+0.79 s/s]; s7-rr-a: 2d 3h 53m 9s [+0.87 s/s] [08:22:37] Merlissimo: s7-user: 2d 3h 53m 9s [+0.87 s/s] [08:23:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72610 MB (11% inode=99%): [08:23:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:31:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:32:16] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 187361 [08:32:16] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 535293 MB (10% inode=44%): [08:32:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:32:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:32:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:33:26] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:34:26] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:36:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 200103 [08:38:25] Federico Leva (Nemo) * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [08:39:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:39:26] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [08:42:35] @replag [08:42:53] Merlissimo: s1-rr-a: 17m 12s [+0.79 s/s]; s1-user: 17m 12s [+0.79 s/s]; s2-user: 20s [-0.00 s/s]; s3-rr-a: 2d 15h 8m 46s [+0.89 s/s]; s3-user: 2d 15h 8m 46s [+0.89 s/s]; s6-rr-a: 2d 7h 41m 0s [+0.96 s/s]; s6-user: 2d 7h 41m 15s [+0.96 s/s]; s7-rr-a: 2d 4h 13m 26s [+1.00 s/s] [08:42:54] Merlissimo: s7-user: 2d 4h 13m 27s [+1.00 s/s] [08:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [08:45:26] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 227463 [08:48:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:50:35] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [09:01:25] Federico Leva (Nemo) * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [09:03:05] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1931 [09:03:25] Peter Wendorff * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [09:03:26] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1922.000000 [09:05:05] MySQL slave on rosemary is OK: Uptime: 14492860 Threads: 15 Questions: 6355191614 Slow queries: 1940671 Opens: 253482 Flush tables: 6 Open tables: 3862 Queries per second avg: 438.505 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1734 [09:05:26] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1713.000000 [09:22:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:23:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72318 MB (11% inode=99%): [09:23:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:31:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:32:15] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 190705 [09:32:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 535102 MB (10% inode=44%): [09:32:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:32:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:32:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:33:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:35:26] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:37:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 203689 [09:39:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:39:25] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [09:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [09:45:27] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 230972 [09:48:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:50:35] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [10:20:40] does anybody know, how long are backups of /home kept? [10:23:16] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72002 MB (11% inode=99%): [10:23:25] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:24:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:31:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:33:16] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 194031 [10:33:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:33:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 534898 MB (10% inode=44%): [10:33:25] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:33:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:33:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:35:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:37:34] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 207118 [10:39:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:39:46] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [10:43:46] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [10:45:35] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 234488 [10:46:00] hi. Does anybody know if Kai already switched the OSM databases to the ODBL import? [10:48:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:50:45] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [11:24:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71683 MB (11% inode=99%): [11:24:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:24:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:31:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:33:15] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 197183 [11:33:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:33:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 534743 MB (10% inode=44%): [11:33:25] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:33:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:34:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:36:01] @replag [11:36:35] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:36:54] tsbot: ?? [11:37:15] jem-: s3-rr-a: 2d 17h 58m 40s [+0.98 s/s]; s3-user: 2d 17h 58m 41s [+0.98 s/s]; s6-rr-a: 2d 10h 26m 50s [+0.96 s/s]; s6-user: 2d 10h 26m 50s [+0.96 s/s]; s7-rr-a: 2d 6h 49m 4s [+0.90 s/s]; s7-user: 2d 6h 50m 17s [+0.90 s/s] [11:37:35] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 210481 [11:39:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:39:46] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [11:43:45] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [11:45:35] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 238074 [11:47:13] @replag all [11:47:28] jem-: s1-rr-a: 0s [-0.09 s/s]; s1-user: 0s [-0.09 s/s]; s2-user: 9s [-0.00 s/s]; s2-user-c: 0s [-0.00 s/s]; s3-rr-a: 2d 18h 9m 52s [+1.00 s/s]; s3-user: 2d 18h 9m 52s [+1.00 s/s]; s4-rr-a: 0s [-0.02 s/s]; s4-user: 0s [-0.02 s/s] [11:47:29] jem-: s5-rr-a: 1s [-0.02 s/s]; s5-user: 1s [-0.02 s/s]; s5-user-c: 0s [-0.00 s/s]; s6-rr-a: 2d 10h 37m 4s [+0.91 s/s]; s6-user: 2d 10h 37m 19s [+0.92 s/s]; s7-rr-a: 2d 6h 59m 28s [+0.91 s/s]; s7-user: 2d 6h 59m 28s [+0.90 s/s] [11:48:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:50:46] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:25:16] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71369 MB (11% inode=99%): [12:25:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:25:25] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:31:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:33:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:33:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 534355 MB (10% inode=44%): [12:33:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:33:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:34:15] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 200594 [12:34:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:36:36] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:38:36] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 213669 [12:39:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:39:45] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [12:43:46] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [12:45:36] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 241611 [12:48:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:50:45] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [13:25:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:25:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:26:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71067 MB (11% inode=99%): [13:27:44] hello all [13:30:34] hare krishna, allah akbar, shalom, amen! DaBPunkt is finally here, YAY! [13:30:59] Danny_B|backup: *g* Could nosy help you with restoring your file? [13:31:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:31:30] DaBPunkt: idk, she did not respond [13:31:51] Danny_B|backup: ok, then I will look [13:31:55] many thanks [13:32:29] and i realized that the protection is quite simple, just unset w for user ;-) [13:32:48] gonna do it on all sensitive files soon [13:33:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:33:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 534185 MB (10% inode=44%): [13:33:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:33:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:34:16] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 203949 [13:34:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:35:13] Danny_B|backup: when did the overwrite happen? [13:35:46] before or after 3:00 UTC? [13:36:35] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:39:35] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 216641 [13:40:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:40:46] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [13:43:46] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [13:45:34] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 245142 [13:46:32] DaBPunkt: Sep 26 00:34 utc [13:47:45] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:47:45] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:47:45] APT on mayapple is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:48:23] Danny_B|backup: ok, then I need the backup of the 25. [13:48:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:49:47] ill have a few minutes [13:49:50] +w [14:18:00] Danny_B|backup: sorry, I have to start a-new. My connection to hemlock was disconnected [14:18:21] nvm, take your time [14:18:56] actually, speaking about hemlock - it has many issues in last two days [14:25:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:25:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:26:16] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70773 MB (11% inode=99%): [14:31:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:33:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:33:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 533957 MB (10% inode=44%): [14:33:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:33:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:34:15] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 207351 [14:34:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:37:35] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:39:35] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 219635 [14:40:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:45:35] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 248645 [14:48:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:17:25] Ryan Lane * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [15:22:25] Andrei Cipu * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [15:25:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:25:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:27:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70452 MB (11% inode=99%): [15:31:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:34:16] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 210849 [15:34:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:34:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 533585 MB (10% inode=44%): [15:34:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:34:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:34:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:37:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:39:35] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 222572 [15:41:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:45:35] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 252140 [15:48:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:14:46] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:26:25] Jeremy Baron * [Toolserver-l] Fwd: [Wikitech-l] Toolserver is not opening for GU:WP [16:26:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:26:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:27:16] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70148 MB (11% inode=99%): [16:31:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:34:16] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 214236 [16:34:37] I will restart the mysql-process of z-dat-3-a. AFAIS hyacinth is swapping [16:34:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:35:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:35:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 533382 MB (10% inode=44%): [16:35:25] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:35:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:37:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:39:34] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 225736 [16:40:18] Danny_B|backup: your file should be in /home/danny_b/restored_from_backup [16:41:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:43:35] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [16:44:25] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:45:35] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [16:46:25] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 105040 MB (10% inode=99%): [16:48:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:58:05] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 78976 MB (8% inode=98%): [17:05:11] DaBPunkt: manz thanks [17:06:47] np [17:11:10] so what is the status of databases now? [17:14:05] Danny_B|backup: z-dat-s3-a is shutting down [17:14:15] should be back soon [17:14:42] awesome [17:15:19] and considering the latest list thread, should i assume we won't move to apache soon? [17:20:23] Danny_B|backup: no. It will take a little bit longer because the discussion takes time, but it would be unfair to stop now [17:25:26] Kolossos * Re: [Toolserver-l] Reasons for not migrating to Tool Lab-OSM [17:26:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:26:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:27:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69937 MB (11% inode=99%): [17:31:46] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 37236 MB (9% inode=99%): [17:32:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:34:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:35:15] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 217591 [17:35:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:35:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 531859 MB (10% inode=44%): [17:35:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:35:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:38:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:39:35] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 228595 [17:41:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:43:35] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [17:45:35] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [17:46:26] Carl (CBM) * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [17:48:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:51:12] s3 failed to restart or has a painfull shutdown ? [17:57:26] Platonides * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [18:01:25] Platonides * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [18:02:58] phe: 19:14:05 < DaBPunkt> Danny_B|backup: z-dat-s3-a is shutting down [18:03:12] (utc+2) [18:04:16] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 13846 MB (1% inode=95%): [18:21:15] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:21:35] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:21:46] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:23:05] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 37172 MB (9% inode=99%): [18:23:05] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 78944 MB (8% inode=98%): [18:23:15] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 78941 MB (8% inode=98%): [18:26:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:26:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:28:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69685 MB (11% inode=99%): [18:32:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:34:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:35:25] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 221016 [18:35:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:35:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 531229 MB (10% inode=44%): [18:35:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:35:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:38:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:39:35] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:39:45] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:39:55] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 231308 [18:41:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:43:35] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [18:45:34] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [18:46:15] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:48:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:49:47] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 37101 MB (9% inode=99%): [18:52:35] MySQL on z-dat-s3-a is OK: Uptime: 33 Threads: 1 Questions: 38 Slow queries: 0 Opens: 35 Flush tables: 1 Open tables: 28 Queries per second avg: 1.151 [18:59:26] DaB. * [Toolserver-announce] z-dat-s3-a (sql-s3) was restarted [19:00:28] \o/ [19:01:09] DaBPunkt: can you also reimport some db on s3 as they are corrupted? [19:01:31] I still wait for the s1-dump [19:02:26] John * Re: [Toolserver-l] [Wikitech-l] Toolserver is not opening for GU:WP [19:06:05] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 78923 MB (8% inode=98%): [19:06:15] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 78923 MB (8% inode=98%): [19:26:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:26:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:28:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69554 MB (11% inode=99%): [19:32:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:34:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:35:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:35:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 531087 MB (10% inode=44%): [19:35:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:35:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:35:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 223974 [19:38:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 230670 [19:42:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:45:35] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 261954 [19:48:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:23:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66972 MB (10% inode=99%): [20:26:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:26:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:33:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:35:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:35:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 530786 MB (10% inode=44%): [20:35:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:35:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:35:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:36:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 226589 [20:38:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 230484 [20:42:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:45:35] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 264936 [20:49:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:01:25] Federico Leva (Nemo) * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [21:12:49] sulutil doesn't reply http://toolserver.org/~quentinv57/sulinfo [21:13:04] should be: sulinfo doesn't respond [21:21:50] LA2: tell qientinv57 [21:23:16] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57088 MB (9% inode=99%): [21:26:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:26:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:28:37] See... there was an e-mail discussion that said MMP are alive and kicking on the toolserver, but when I need a tool, it has a single maintainer and all I can do is to contact that person. [21:29:34] Is it likely that this maintainer changed something, that caused the tool to stop working? It seems much more likely that some database got stuck in deadlock, or that replication halted. [21:30:31] LA2: for first: that is not a MMP. For second: Even IF it would be a MP you had to contact the developers of it too. [21:33:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:35:12] apmon, who grants access to ptolemy? [21:35:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:35:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 530655 MB (10% inode=44%): [21:35:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:35:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:35:32] do you have mapnik 2.1.0 installed there? [21:35:41] dschwen: No [21:35:50] It is still 0.7.1 [21:36:00] kolossos mentioned the render stack does not compile on solaris [21:36:05] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1959 [21:36:10] Never managed to get all of the dependencies compiled on Solaris [21:36:20] I upgraded the labs version of my renderer t 2.1 [21:36:25] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1968.000000 [21:36:29] there are quite few [21:36:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:36:46] I'd love to give it a try, if that is wanted [21:36:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 229758 [21:36:57] Yes, I think it is [21:37:11] but I still don;t have access to ptolemy [21:37:21] I'd want to update the stack to Mapnik 2.1 for a long time [21:37:25] do I need to bug annosy or DaB? [21:37:44] Are you member of the OSM MMP? [21:38:01] probably not [21:38:09] it is not in my groups list [21:38:25] It was still River who managed to compile mapnik 0.7.1 the last time [21:38:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:38:47] I must say I'm glad to have ubuntu on labs [21:38:47] Using the TSSpecs build system if I am not mistaken. But I know little about that [21:39:04] add a PPA and done [21:39:25] And you even have a working malloc on Ubuntu... ;-) [21:39:25] Free Memory on turnera is CRITICAL: CRITICAL - 3.4% (284408 kB) free! [21:39:46] the luxury! [21:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 233199 [21:41:44] dschwen: Currently river,mazder,cmarqu,kolossos,kayd,apmon are members of OSM [21:41:53] So yes, you don't look like you are in the group [21:42:35] and bug [21:43:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:43:46] I think you need to file a JIRA ticket to become member and that will hopefully also give you access to ptolemy [21:43:59] but it probably is worth mentioning that you need that in the Jira ticket [21:44:02] already done 1 minute ago [21:45:35] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 268458 [21:49:02] dschwen: I have now switched over the default OSM database to the ODBL import. [21:49:36] ok, I'll change my attribution [21:49:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:49:47] But I forgot to create the tags index, so that is still generating at the moment [22:02:25] Free Memory on turnera is WARNING: WARNING - 5.1% (424088 kB) free! [22:03:26] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (422656 kB) free! [22:04:26] Free Memory on turnera is WARNING: WARNING - 5.2% (435612 kB) free! [22:15:40] dschwen: Do you have experience "porting" things over to Solaris? [22:16:04] * DaBPunkt have – its horrible ;) [22:16:09] +' [22:16:31] DaBPunkt: I agree :-) [22:16:31] well, I had to wrestle with linux->solaris issues before, for my compiled codes and its dependencies [22:17:10] hello DaBPunkt ! [22:17:39] hello dschwen – it's not nice to lure away my users ;) [22:17:41] Things that caught me out were that some things are compiled with sun compiler and some with gcc. And that the default compile is 32bit and so some libraries are 32bit and some 64bit [22:18:01] hey , what do you mean, I'm giving you some room to breathe! [22:18:18] *g* it was a joke :) [22:18:22] I know [22:18:59] apmon: if we even get a new server for osm I will install debian on it [22:20:29] Although I like to complain about Solaris, I have mostly learnt to live with it on ptolemy and work around the issues. [22:20:51] But yes, I would prefer Debian [22:22:40] there are nice things in solaris (the stateful starting-system is useful sometimes), but sometimes I have the feeling it's a unix from bevor my time… [22:24:16] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54056 MB (8% inode=99%): [22:26:15] pstack is also nice to do some quick profiling (including I/O wait times) [22:26:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:26:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:28:25] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (378324 kB) free! [22:29:14] The main think I still haven't found a solution for is that unix domain sockets (in perl) don't seem to work (as expected) [22:29:23] a not so nice thing about solaris isthe memory-eating bug in the nfs-server… [22:30:36] Is that the same / similar issue as with malloc? [22:31:05] apmon: I have no idea [22:32:40] The standard malloc implementation of solaris never returns any memory to the OS. So if you malloc e.g. 10Gb and then immediately free it again, the 10Gb will never be returned until the process dies / ends. [22:32:51] Which is particularly problematic for long running server processes [22:33:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3607 [22:33:05] but 6.2gb kernel-memory-using can't be normal.. [22:33:09] these sound all like minor inconveniences compared to the fact that it is an absolute PITA to get up to date software installed [22:33:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:33:26] Free Memory on turnera is WARNING: WARNING - 5.2% (435016 kB) free! [22:33:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3614.000000 [22:33:58] latest boost we have is 1.42 :-( [22:34:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (374532 kB) free! [22:34:26] on debian I wouldn't even care about dependencies ike this [22:34:54] Well, if it is a consolidation, even on Ubuntu 12.04 boost is not new enough to compile mapnik 2.1 [22:35:05] but there are PPAs [22:35:09] dschwen: ee, debian has also 1.42 [22:35:14] and you DO NOT NEED TO COMPILE IT ;-) [22:35:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:35:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 530556 MB (10% inode=44%): [22:35:31] that's true :) [22:35:53] @replag [22:36:06] jem-: s1-rr-a: 1h 1m 12s [+0.09 s/s]; s1-user: 1h 1m 12s [+0.09 s/s]; s3-rr-a: 3d 3h 24m 3s [+0.85 s/s]; s3-user: 3d 3h 24m 15s [+0.85 s/s]; s6-rr-a: 2d 17h 38m 22s [+0.65 s/s]; s6-user: 2d 17h 38m 22s [+0.65 s/s]; s7-rr-a: 2d 16h 43m 47s [+0.90 s/s]; s7-user: 2d 16h 43m 47s [+0.90 s/s] [22:36:23] let's see... libboost1.50-all-dev [22:36:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:36:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:36:43] then again I'm crazy enough to have quantal in my VM at work [22:36:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:36:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 233050 [22:37:01] root@yarrow:~# dpkg -l|grep boost [22:37:02] ii libboost-all-dev 1.42.0.1 Boost C++ Libraries development files (ALL, default version) [22:37:13] dschwen: I have quantal on my main (work) laptop... ;-) [22:37:20] I'm not crazzy enough :) [22:37:44] DaBPunkt: Is there some hint about when the current issue with s3-7-etc. will be fixed? [22:38:25] jem-: I restarted s3 today. It will take a few hours to see if that helped. If not I will restart s6 next [22:38:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:38:51] Ok, thanks :) [22:39:00] s6 became also short of disc-space. nosy planed to fix this; maybe we will combine that [22:39:47] And btw, you have my support in all the current you-know-what :) [22:40:09] thanks :) [22:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 236490 [22:40:55] Maybe an IRC meeting of users would be a good idea if things get worse [22:41:20] ... But let's hope they don't get worse [22:43:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:45:35] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 272013 [22:49:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:51:25] DaB. * [Toolserver-announce] Maintenance-window for webservers Monday 20:00-22:00 UTC [22:55:52] nacht ts [23:02:26] Free Memory on turnera is CRITICAL: CRITICAL - 2.7% (229080 kB) free! [23:04:26] Free Memory on turnera is OK: OK - 13.7% (1149700 kB) free. [23:14:05] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3566 [23:14:25] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3560.000000 [23:24:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57469 MB (9% inode=99%): [23:26:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:26:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:33:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:35:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:35:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 528875 MB (9% inode=44%): [23:36:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:36:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:36:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 235982 [23:37:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:38:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 239012 [23:43:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:35] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 275522 [23:49:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default