[00:02:29] Load avg. on nightshade is WARNING: WARNING - load average: 17.15, 12.90, 10.16 [00:03:29] Load avg. on nightshade is OK: OK - load average: 14.80, 13.04, 10.38 [00:04:29] Free Memory on turnera.esi is WARNING: WARNING - 8.8% (370396 kB) free! [00:05:20] @replag [00:05:21] DaBPunkt: s1-sec: 3m 42s [+0.01 s/s]; s2-pri: 27s [-0.10 s/s]; s3-rr: 1d 44m 36s [+1.00 s/s]; s3-user: 1d 44m 36s [+1.00 s/s] [00:05:27] @replag [00:05:28] DaBPunkt: s1-sec: 3m 46s [+0.60 s/s]; s2-pri: 20s [-1.04 s/s]; s3-rr: 1d 44m 42s [+0.90 s/s]; s3-user: 1d 44m 42s [+0.90 s/s] [00:05:51] @replag [00:05:52] DaBPunkt: s1-sec: 3m 43s [-0.12 s/s]; s2-pri: 24s [+0.16 s/s]; s3-rr: 1d 45m 7s [+1.03 s/s]; s3-user: 1d 45m 7s [+1.03 s/s] [00:06:05] stupid thing, I see that it is decreasing [00:17:59] nacht ts [00:22:29] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [00:23:08] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:23:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:29:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:29:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:49:28] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [00:54:49] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [01:04:29] Free Memory on turnera.esi is WARNING: WARNING - 9.7% (406836 kB) free! [01:05:39] Free Memory on turnera.esi is OK: OK - 10.9% (458280 kB) free. [01:22:38] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [01:23:09] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:23:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:29:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:30:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:49:29] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [01:52:33] @replag [01:52:33] Betacommand: s2-pri: 36s [+0.00 s/s]; s3-rr: 1d 1h 50m 23s [+0.61 s/s]; s3-user: 1d 1h 50m 23s [+0.61 s/s]; s5-rr: 15s [+0.00 s/s]; s5-user: 15s [-0.28 s/s]; s7-rr: 1h 5m 24s [+0.08 s/s]; s7-user: 1h 5m 24s [+0.08 s/s] [01:54:59] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [02:02:34] Load avg. on nightshade is WARNING: WARNING - load average: 15.06, 14.94, 12.98 [02:03:29] Load avg. on nightshade is OK: OK - load average: 13.86, 14.59, 12.98 [02:04:49] Free Memory on turnera.esi is WARNING: WARNING - 9.6% (403532 kB) free! [02:05:49] Free Memory on turnera.esi is OK: OK - 11.0% (459104 kB) free. [02:22:38] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [02:23:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:23:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:29:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:30:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:32:38] Load avg. on nightshade is WARNING: WARNING - load average: 16.89, 14.63, 13.39 [02:33:39] Load avg. on nightshade is OK: OK - load average: 14.50, 14.36, 13.38 [02:49:39] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [02:54:59] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [03:21:29] /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 12072 MB (1% inode=99%): [03:22:49] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [03:23:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:23:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:29:19] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:30:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:49:39] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [03:54:58] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [04:04:50] Free Memory on turnera.esi is WARNING: WARNING - 8.9% (370944 kB) free! [04:05:50] Free Memory on turnera.esi is OK: OK - 10.1% (424304 kB) free. [04:22:49] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [04:23:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:23:50] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:29:19] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:30:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:49:49] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [04:54:59] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [04:59:49] Load avg. on nightshade is WARNING: WARNING - load average: 16.39, 13.74, 12.55 [05:00:49] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 31.38, 18.34, 14.25 [05:01:49] Load avg. on nightshade is WARNING: WARNING - load average: 25.35, 18.91, 14.71 [05:14:50] Load avg. on nightshade is OK: OK - load average: 13.78, 15.00, 14.80 [05:22:49] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [05:23:39] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:23:50] Load avg. on nightshade is WARNING: WARNING - load average: 16.21, 15.08, 14.77 [05:23:59] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:28:50] Load avg. on nightshade is OK: OK - load average: 14.14, 14.91, 14.81 [05:29:19] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:30:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:49:50] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [05:54:59] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [06:04:49] Free Memory on turnera.esi is WARNING: WARNING - 9.9% (416120 kB) free! [06:05:49] Free Memory on turnera.esi is OK: OK - 11.3% (471372 kB) free. [06:22:49] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [06:23:39] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:23:58] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:29:29] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:30:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:50:49] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [06:54:59] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [07:02:49] Load avg. on nightshade is WARNING: WARNING - load average: 15.01, 15.08, 14.06 [07:03:50] Load avg. on nightshade is OK: OK - load average: 14.28, 14.92, 14.07 [07:22:59] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [07:23:49] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:24:08] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:29:28] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:30:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:32:49] Load avg. on nightshade is WARNING: WARNING - load average: 16.52, 15.75, 14.58 [07:35:49] Load avg. on nightshade is OK: OK - load average: 14.03, 15.00, 14.50 [07:39:58] Load avg. on nightshade is WARNING: WARNING - load average: 17.62, 16.03, 15.00 [07:50:59] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [07:55:09] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [08:02:59] Load avg. on nightshade is WARNING: WARNING - load average: 15.75, 15.61, 14.89 [08:03:59] Free Memory on turnera.esi is WARNING: WARNING - 9.2% (383032 kB) free! [08:06:59] Load avg. on nightshade is OK: OK - load average: 13.80, 14.84, 14.74 [08:08:59] Free Memory on turnera.esi is OK: OK - 10.4% (436736 kB) free. [08:23:00] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [08:24:08] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:24:49] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:25:18] @replag all [08:25:19] nosy: s1-pri: 17s [+0.15 s/s]; s1-sec: 9s [+0.07 s/s]; s1-sec-c: 0s [-0.04 s/s]; s2-pri: 1s [-0.01 s/s]; s2/s5-pri-c: 21m 12s [+0.73 s/s]; s3-rr: 1d 8h 13m 27s [-2.13 s/s]; s3-user: 1d 8h 13m 27s [-2.13 s/s]; s4-rr: 21m 12s [+11.58 s/s] [08:25:20] nosy: s4-user: 1s [-0.03 s/s]; s5-rr: 1s [-]; s5-user: 10m 49s [+0.78 s/s]; s6-rr: 3s [-]; s6-user: 3s [-]; s7-rr: 4h 28m 34s [-30.18 s/s]; s7-user: 4h 28m 34s [-30.18 s/s] [08:29:39] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:30:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:43:59] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1849.000000 [08:46:45] @replag all [08:46:46] nosy: s1-pri: 5m 0s [+0.22 s/s]; s1-sec: 1s [-0.01 s/s]; s1-sec-c: 1s [+0.00 s/s]; s2-pri: 1s [-]; s2/s5-pri-c: 32m 6s [+0.51 s/s]; s3-rr: 1d 4h 34m 51s [-10.19 s/s]; s3-user: 1d 4h 34m 51s [-10.19 s/s]; s4-rr: 32m 6s [+0.51 s/s] [08:46:47] nosy: s4-user: 1s [-]; s5-rr: 3s [+0.00 s/s]; s5-user: 16m 57s [+0.29 s/s]; s6-rr: 4s [+0.00 s/s]; s6-user: 4s [+0.00 s/s]; s7-rr: 4h 31m 20s [+0.13 s/s]; s7-user: 4h 31m 20s [+0.13 s/s] [08:49:26] 3(created) [MNT-1192] s3 and s7 slaves were stopped; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1192> (Marlen Caemmerer) [08:50:59] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [08:51:25] 3(resolved) [MNT-1192] s3 and s7 slaves were stopped <10https://jira.toolserver.org/browse/MNT-1192> (Marlen Caemmerer) [08:53:59] Free Memory on turnera.esi is WARNING: WARNING - 9.4% (394876 kB) free! [08:55:09] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 95181 [08:56:59] Free Memory on turnera.esi is OK: OK - 10.5% (437452 kB) free. [08:58:01] @replag all [08:58:01] nosy: s1-pri: 1s [-0.44 s/s]; s1-sec: 1s [-]; s1-sec-c: 2s [+0.00 s/s]; s2-pri: 2s [+0.00 s/s]; s2/s5-pri-c: 36m 44s [+0.41 s/s]; s3-rr: 1d 1h 46m 32s [-14.95 s/s]; s3-user: 1d 1h 46m 32s [-14.95 s/s]; s4-rr: 2s [-2.85 s/s] [08:58:02] nosy: s4-user: 2s [+0.00 s/s]; s5-rr: 5s [+0.00 s/s]; s5-user: 21m 35s [+0.41 s/s]; s6-rr: 4s [-]; s6-user: 4s [-]; s7-rr: 3h 1m 13s [-8.01 s/s]; s7-user: 3h 1m 13s [-8.01 s/s] [09:03:00] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2720 [09:04:01] MySQL slave on z-dat-s7-a is OK: Uptime: 599482 Threads: 11 Questions: 70125135 Slow queries: 21889 Opens: 524066 Flush tables: 1 Open tables: 3659 Queries per second avg: 116.976 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1448 [09:04:01] Free Memory on turnera.esi is WARNING: WARNING - 8.5% (354464 kB) free! [09:06:24] 3(commented) [TS-1285] Retrieve data from expired user account <10https://jira.toolserver.org/browse/TS-1285> (Marlen Caemmerer) [09:09:23] 3(created) [MNT-1193] Restart MySQL s2/s5 on daphne; Maintenance; Emergency work <10https://jira.toolserver.org/browse/MNT-1193> (Marlen Caemmerer) [09:09:59] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [09:11:27] 3(commented) [TS-1287] Database S5 extremely slow <10https://jira.toolserver.org/browse/TS-1287> (Marlen Caemmerer) [09:11:29] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [09:11:59] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [09:11:59] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [09:11:59] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [09:16:29] s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 436.000000 [09:16:59] MySQL slave on daphne is OK: Uptime: 54 Threads: 13 Questions: 1863 Slow queries: 6 Opens: 208 Flush tables: 1 Open tables: 201 Queries per second avg: 34.500 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 474 [09:16:59] MySQL on daphne is OK: Uptime: 54 Threads: 13 Questions: 1865 Slow queries: 6 Opens: 208 Flush tables: 1 Open tables: 201 Queries per second avg: 34.537 [09:16:59] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2104.000000 [09:16:59] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3104.000000 [09:21:00] s5 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1679.000000 [09:23:00] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [09:24:29] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:24:49] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:26:59] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1635.000000 [09:29:23] 3(updated) [MNT-1193] Restart MySQL s2/s5 on daphne <10https://jira.toolserver.org/browse/MNT-1193> (Marlen Caemmerer) [09:30:39] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:30:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:52:59] Free Memory on turnera.esi is WARNING: WARNING - 8.6% (360212 kB) free! [09:55:19] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 69107 [09:55:59] Free Memory on turnera.esi is OK: OK - 10.3% (429160 kB) free. [10:04:00] Free Memory on turnera.esi is WARNING: WARNING - 9.1% (379392 kB) free! [10:23:00] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [10:24:29] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:25:00] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:30:38] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:30:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:55:29] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 52259 [11:03:59] Free Memory on turnera.esi is WARNING: WARNING - 8.7% (364748 kB) free! [11:05:59] Free Memory on turnera.esi is OK: OK - 10.1% (423372 kB) free. [11:22:59] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [11:24:29] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:24:59] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:30:48] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:30:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:43:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 40551 MB (4% inode=99%): [11:45:59] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/16:DOWN: 1 int NOK : CRITICAL [11:45:59] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/8:DOWN: 1 int NOK : CRITICAL [11:49:00] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/8:UP:1 UP: OK [11:49:00] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/16:UP:1 UP: OK [11:55:29] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26403 [12:03:09] Load avg. on nightshade is WARNING: WARNING - load average: 15.10, 13.48, 11.61 [12:04:09] Load avg. on nightshade is OK: OK - load average: 13.94, 13.49, 11.73 [12:04:10] Free Memory on turnera.esi is WARNING: WARNING - 9.0% (377456 kB) free! [12:06:08] Free Memory on turnera.esi is OK: OK - 10.3% (433080 kB) free. [12:12:26] 3(commented) [TS-1287] Database S5 extremely slow <10https://jira.toolserver.org/browse/TS-1287> [12:17:12] Free Memory on turnera.esi is WARNING: WARNING - 9.9% (412380 kB) free! [12:23:09] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [12:24:39] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:25:09] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:30:49] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:31:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:55:29] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5970 [13:04:19] Free Memory on turnera.esi is WARNING: WARNING - 9.2% (384548 kB) free! [13:05:19] Free Memory on turnera.esi is OK: OK - 10.3% (433160 kB) free. [13:05:38] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3442 [13:12:38] MySQL slave on z-dat-s3-a is OK: Uptime: 489322 Threads: 17 Questions: 449245946 Slow queries: 22658 Opens: 2460919 Flush tables: 1 Open tables: 16384 Queries per second avg: 918.98 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1782 [13:13:49] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:49] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:39] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.021 second response time [13:15:39] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.010 second response time [13:23:20] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [13:24:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:25:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:25:27] will take some database servers temporarily offline now to put some controllers in [13:30:49] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:31:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:32:43] now i do rosemary and adenia [13:32:49] ethernet 0/1/23 [rosemary.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/23:DOWN: 1 int NOK : CRITICAL [13:32:59] ethernet 0/1/3 [rosemary] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/3:DOWN: 1 int NOK : CRITICAL [13:36:49] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:DOWN: 1 int NOK : CRITICAL [13:37:08] ethernet 0/1/3 [cassini] on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/3:DOWN: 1 int NOK : CRITICAL [13:44:49] ethernet 0/1/23 [rosemary.mgmt] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/23:UP:1 UP: OK [13:45:09] ethernet 0/1/3 [rosemary] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/3:UP:1 UP: OK [13:49:49] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/14:UP:1 UP: OK [13:50:09] ethernet 0/1/3 [cassini] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/3:UP:1 UP: OK [13:52:19] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [13:52:19] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [13:52:39] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [13:52:39] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [13:53:49] s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1926.000000 [13:56:18] [[Admin:Datacenter-Tasks]] 10https://wiki.toolserver.org/w/index.php?diff=6669&oldid=6668&rcid=8802 * Nosy * (+32) () [13:56:29] MySQL on adenia is CRITICAL: Cant connect to MySQL server on adenia (146) [13:57:27] 3(created) [TS-1292] Disk Controller Error message on rosemary; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1292> (Marlen Caemmerer) [13:57:29] MySQL on adenia is OK: Uptime: 185 Threads: 24 Questions: 765 Slow queries: 3 Opens: 51 Flush tables: 1 Open tables: 36 Queries per second avg: 4.135 [13:58:09] ok...adenia and rosemary are back with san controllers inside [13:58:36] daphne (s2/s5) is next [14:01:03] stop the maintenance [14:01:08] i hate it [14:01:21] Stop trolling. [14:01:23] We hate you. [14:01:36] @replag [14:02:02] TBloemink i will knock up your daughter [14:02:08] Yes [14:02:09] !ops [14:02:24] narc [14:02:43] [[Admin:Service processors]] 10https://wiki.toolserver.org/w/index.php?diff=6670&oldid=6337&rcid=8803 * Nosy * (-25) (/* Systems on the SCS */ ) [14:02:58] wikirc_ go to hell [14:03:37] [[Admin:Service processors]] 10https://wiki.toolserver.org/w/index.php?diff=6671&oldid=6670&rcid=8804 * Nosy * (+25) (/* Sun X4100/X4150/X4250 (ILOM) */ ) [14:03:55] VERSION [14:04:19] Free Memory on turnera.esi is WARNING: WARNING - 9.1% (379440 kB) free! [14:05:22] Free Memory on turnera.esi is OK: OK - 10.3% (432728 kB) free. [14:07:09] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [14:07:39] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/2:DOWN: 1 int NOK : CRITICAL [14:08:19] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2494.000000 [14:08:19] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2493.000000 [14:08:38] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2486 [14:08:39] MySQL on rosemary is OK: Uptime: 1118 Threads: 17 Questions: 10630 Slow queries: 5 Opens: 112 Flush tables: 1 Open tables: 105 Queries per second avg: 9.508 [14:11:19] Free Memory on damiana.esi is WARNING: WARNING - 10.0% (418304 kB) free! [14:12:08] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/24:UP:1 UP: OK [14:12:28] thyme is next [14:12:36] last one to boot [14:15:18] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1745.000000 [14:16:49] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [14:22:00] Daniel_WMDE: who is op on this channel? [14:23:39] MySQL slave on rosemary is OK: Uptime: 2018 Threads: 15 Questions: 636538 Slow queries: 184 Opens: 294 Flush tables: 1 Open tables: 283 Queries per second avg: 315.430 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1780 [14:24:18] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1705.000000 [14:24:18] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [14:24:18] Free Memory on turnera.esi is WARNING: WARNING - 9.3% (389488 kB) free! [14:24:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:24:50] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/11:UP:1 UP: OK [14:25:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:27:19] Free Memory on turnera.esi is OK: OK - 10.2% (427460 kB) free. [14:28:19] Free Memory on damiana.esi is CRITICAL: CRITICAL - 6.0% (251528 kB) free! [14:29:19] Free Memory on damiana.esi is WARNING: WARNING - 8.7% (366172 kB) free! [14:30:49] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:31:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:31:50] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [14:31:59] MySQL slave on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [14:32:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:35:49] MySQL on thyme is OK: Uptime: 361 Threads: 17 Questions: 159 Slow queries: 2 Opens: 153 Flush tables: 1 Open tables: 143 Queries per second avg: 0.440 [14:35:59] MySQL slave on thyme is OK: Uptime: 372 Threads: 11 Questions: 1723 Slow queries: 8 Opens: 177 Flush tables: 1 Open tables: 166 Queries per second avg: 4.631 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1390 [14:36:18] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1394.000000 [14:52:10] [[Special:Log/newusers]] create 10 * Samford1993 * (New user account) [15:07:49] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/2:DOWN: 1 int NOK : CRITICAL [15:19:52] Alchimista: the only op currently online is Duesentrieb, but with a different account [15:20:29] Merlissimo: it's not needed anymore. One user was trolling on irc, but has gone away [15:21:39] yes just read it. you can ask chanserv for an access list: /msg chanserv access #wikimedia-toolserver list [15:22:51] nosy should get op status, i think [15:23:28] perhaps Daniel_WMDE can add her [15:24:15] thanks [15:24:28] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [15:24:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:28:20] Free Memory on damiana.esi is CRITICAL: CRITICAL - 5.7% (239768 kB) free! [15:29:20] Hello. If someone sees nosy, she should call me [15:29:20] Free Memory on damiana.esi is WARNING: WARNING - 8.1% (338616 kB) free! [15:30:50] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:32:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:32:50] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/3:DOWN: 1 int NOK : CRITICAL [15:33:25] DaBPunkt: we just talked about nosy because she isn't op in this channel ;-) [15:34:10] MySQL on cassia is CRITICAL: Cant connect to MySQL server on cassia (146) [15:34:10] MySQL slave on cassia is CRITICAL: Cant connect to MySQL server on cassia (146) [15:34:30] /sql on cassia is CRITICAL: DISK CRITICAL - /sql is not accessible: I/O error [15:34:30] SMF on cassia is CRITICAL: ERROR - maintenance: svc:/application/ts/mysql-51:default [15:34:50] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on cassia (146) [15:34:57] Merlissimo: That's because I haven't enough rights myself to give her and I didn't find time to let the access-list reset yet [15:36:00] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/3:UP:1 UP: OK [15:37:20] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [15:37:43] Trains of the netherlands are not so nice like the german ICEs – but they have wlan (more or less) ;-) [15:39:50] DaBPunkt: i'll notice apper because of the access list if i see him online [15:40:07] ok. I will try werdna [15:50:20] Free Memory on turnera.esi is WARNING: WARNING - 9.9% (414028 kB) free! [15:52:19] Free Memory on turnera.esi is OK: OK - 10.1% (424464 kB) free. [15:57:39] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.153320/1.75, alarm hl:np_load_avg=0.180664/2.00, alarm hl:mem_free=286.000000M/300M, alarm hl:available=1/0 [16:03:40] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [16:08:09] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/2:DOWN: 1 int NOK : CRITICAL [16:25:30] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:25:39] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [16:25:40] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:28:09] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL [16:28:20] Free Memory on damiana.esi is CRITICAL: CRITICAL - 5.5% (231804 kB) free! [16:28:40] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [16:29:20] Free Memory on damiana.esi is OK: OK - 8.0% (334532 kB) free. [16:31:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:32:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:34:11] MySQL on cassia is CRITICAL: Cant connect to MySQL server on cassia (146) [16:34:20] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [16:34:20] MySQL slave on cassia is CRITICAL: Cant connect to MySQL server on cassia (146) [16:34:30] SMF on cassia is CRITICAL: ERROR - maintenance: svc:/application/ts/mysql-51:default [16:35:10] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on cassia (146) [16:37:19] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [16:45:25] 3(assigned) [TS-1292] Disk Controller Error message on rosemary <10https://jira.toolserver.org/browse/TS-1292> (Marlen Caemmerer) [16:47:23] 3(created) [TS-1293] Add more fibre cables to the SAN installation for redundancy; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1293> (Marlen Caemmerer) [16:52:25] 3(commented) [TS-1293] Add more fibre cables to the SAN installation for redundancy <10https://jira.toolserver.org/browse/TS-1293> (Marlen Caemmerer) [16:54:24] 3(commented) [TS-1293] Add more fibre cables to the SAN installation for redundancy <10https://jira.toolserver.org/browse/TS-1293> (Marlen Caemmerer) [17:03:24] 3(created) [TS-1294] Configure second SAN switch; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1294> (Marlen Caemmerer) [17:03:26] 3(commented) [TS-1293] Add more fibre cables to the SAN installation for redundancy <10https://jira.toolserver.org/browse/TS-1293> (Marlen Caemmerer) [17:04:06] apmon: do you have some time? [17:04:17] yes [17:05:08] By the looks of things, postgresql still seems to be crashing... :-( [17:08:20] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/2:DOWN: 1 int NOK : CRITICAL [17:11:20] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/2:UP:1 UP: OK [17:13:20] FC 0/7 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/7:DOWN:1 UP: OK [17:13:40] /sql on cassia is OK: DISK OK - free space: / 12164 MB (60% inode=92%): [17:26:51] [[Admin:Datacenter-Tasks]] ! 10https://wiki.toolserver.org/w/index.php?diff=6672&oldid=6669&rcid=8806 * 85.17.198.101 * (+40) () [17:28:20] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL [17:28:40] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [17:31:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:31:32] nosy: I wonder if it might be something like the proj issue [17:32:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:32:30] MySQL on cassia is OK: Uptime: 611 Threads: 3 Questions: 16364 Slow queries: 0 Opens: 72 Flush tables: 1 Open tables: 63 Queries per second avg: 26.782 [17:32:45] some helper program in postgresql is crashing, which is triggered by specific queries that previously weren't used [17:34:21] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [17:34:40] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4592 [17:35:29] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3651.000000 [17:35:40] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3439 [17:36:30] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2050.000000 [17:37:29] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 518.000000 [17:37:30] apmon: proj issue? sorry i dont remember [17:37:40] MySQL slave on cassia is OK: Uptime: 921 Threads: 2 Questions: 194718 Slow queries: 6 Opens: 84 Flush tables: 1 Open tables: 75 Queries per second avg: 211.420 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 251 [17:40:31] nosy: perhaps similar to https://jira.toolserver.org/browse/OSM-2. I.e something like an incompatible library that only some querries use [17:41:31] and potentially those queries are now used more often, causing the more frequent crashes recently. [17:41:40] But that is pure speculation at the moment [17:42:49] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3223.000000 [17:45:50] s5 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1130.000000 [17:49:25] 3(commented) [OSM-3] Ptolemy Postgres crashed twice <10https://jira.toolserver.org/browse/OSM-3> (Marlen Caemmerer) [18:02:10] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 32.34, 22.18, 14.86 [18:03:10] Load avg. on nightshade is WARNING: WARNING - load average: 27.84, 22.73, 15.53 [18:06:10] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 28.12, 25.38, 17.88 [18:18:06] @replag [18:18:07] robla: s1-sec-c: 12m 17s [+0.02 s/s]; s2-pri: 1h 44m 20s [+0.19 s/s]; s2/s5-pri-c: 1h 9m 29s [+0.06 s/s]; s3-rr: 1m 7s [+0.00 s/s]; s3-user: 1m 7s [+0.00 s/s]; s4-rr: 12m 17s [-0.08 s/s] [18:18:20] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [18:18:29] hexmode: ^ looks like s3 is all caught up \o/ [18:18:30] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [18:18:50] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3822.000000 [18:18:50] s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6301.000000 [18:19:50] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3340.000000 [18:25:40] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [18:25:40] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:25:50] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:28:20] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL [18:28:40] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [18:31:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:32:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:34:30] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [18:40:26] 3(closed) [DRTRIGON-111] Commit clean_user_sandbox upstream (make it independent as well) <10https://jira.toolserver.org/browse/DRTRIGON-111> (drtrigon) [18:53:50] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3620.000000 [19:18:20] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [19:18:40] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [19:18:59] s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9904.000000 [19:23:00] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3497.000000 [19:25:50] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [19:25:50] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:26:00] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:28:30] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL [19:28:50] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [19:29:04] Russell Blau * Re: [Toolserver-l] Data of s1 (enwiki) probably corrupted [19:31:00] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1574.000000 [19:31:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:32:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:34:51] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [20:18:30] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [20:18:50] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [20:18:59] s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13512.000000 [20:25:50] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [20:25:50] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:25:59] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:28:30] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL [20:28:50] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [20:31:20] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:33:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:34:56] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [20:51:25] 3(commented) [DRTRIGON-112] subster_irc bot forgets wiki login when accessing other mediawiki project <10https://jira.toolserver.org/browse/DRTRIGON-112> (drtrigon) [21:04:55] Free Memory on damiana.esi is WARNING: WARNING - 6.2% (259396 kB) free! [21:07:54] Free Memory on damiana.esi is OK: OK - 7.4% (308688 kB) free. [21:18:54] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [21:19:03] s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17115.000000 [21:19:24] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [21:25:54] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [21:25:55] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:26:14] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:28:55] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL [21:28:55] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [21:31:23] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:33:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:35:04] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [22:19:03] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [22:19:24] s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20731.000000 [22:19:35] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [22:25:54] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [22:25:55] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:26:23] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:28:55] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL [22:28:55] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [22:31:34] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:33:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:35:04] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [23:00:25] 3(closed) [DRTRIGON-112] subster_irc bot forgets wiki login when accessing other mediawiki project <10https://jira.toolserver.org/browse/DRTRIGON-112> (drtrigon) [23:19:14] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [23:19:24] s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24335.000000 [23:19:44] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [23:26:14] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [23:26:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:26:24] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:29:04] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL [23:29:04] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [23:31:44] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:33:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:35:14] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [23:44:03] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 40438 MB (4% inode=99%):