[00:02:33] Load avg. on willow is WARNING: WARNING - load average: 19.34, 17.02, 13.39 [00:13:43] Load avg. on willow is OK: OK - load average: 12.17, 14.74, 14.19 [00:18:55] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [00:18:55] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [00:20:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:20:42] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:20:52] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [00:21:53] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [00:21:53] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [00:23:14] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [00:23:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:23:43] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [00:28:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:34:43] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [00:42:34] nacht ts [00:52:44] Load avg. on willow is WARNING: WARNING - load average: 16.80, 15.27, 14.12 [00:59:44] Load avg. on willow is OK: OK - load average: 14.17, 14.99, 14.48 [01:02:43] Load avg. on willow is WARNING: WARNING - load average: 19.42, 18.54, 16.11 [01:04:33] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1904.000000 [01:19:52] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [01:19:54] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [01:20:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:20:43] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:20:54] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [01:21:53] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [01:21:53] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [01:24:13] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [01:24:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:24:43] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [01:28:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:29:43] Load avg. on willow is OK: OK - load average: 12.97, 13.88, 14.98 [01:32:42] Load avg. on willow is WARNING: WARNING - load average: 15.61, 15.77, 15.63 [01:35:43] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [01:39:42] Load avg. on willow is OK: OK - load average: 13.98, 14.31, 14.95 [01:46:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3643.000000 [01:52:43] Load avg. on willow is WARNING: WARNING - load average: 14.85, 15.25, 15.02 [02:16:44] Load avg. on willow is WARNING: WARNING - load average: 17.58, 17.34, 16.78 [02:20:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:20:43] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:20:52] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [02:20:52] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [02:21:04] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [02:22:03] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [02:22:03] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [02:24:12] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [02:24:53] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [02:24:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:29:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:35:43] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [02:42:33] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [02:46:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6122.000000 [03:17:42] Load avg. on willow is WARNING: WARNING - load average: 13.94, 16.09, 16.59 [03:20:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:20:53] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [03:21:04] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [03:21:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:21:04] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [03:22:03] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [03:22:03] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [03:24:23] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [03:25:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:25:03] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [03:29:43] Load avg. on willow is OK: OK - load average: 11.77, 13.32, 14.92 [03:29:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:32:43] Load avg. on willow is WARNING: WARNING - load average: 13.43, 14.87, 15.38 [03:34:43] Load avg. on willow is OK: OK - load average: 10.79, 13.36, 14.75 [03:35:53] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [03:42:32] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [03:46:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6765.000000 [04:02:33] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2984.000000 [04:02:42] Load avg. on willow is WARNING: WARNING - load average: 17.57, 17.23, 14.81 [04:06:45] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1500.000000 [04:20:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:21:52] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [04:21:52] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:22:03] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [04:22:03] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [04:23:04] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [04:23:04] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [04:25:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:25:04] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [04:25:23] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [04:30:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:35:52] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [04:39:43] Load avg. on willow is OK: OK - load average: 13.58, 14.33, 14.91 [04:42:42] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [04:52:42] Load avg. on willow is WARNING: WARNING - load average: 14.32, 15.35, 15.04 [04:53:43] Load avg. on willow is OK: OK - load average: 13.64, 14.91, 14.90 [05:16:42] Load avg. on willow is WARNING: WARNING - load average: 15.25, 15.86, 15.77 [05:20:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:21:52] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [05:22:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:22:03] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [05:22:03] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [05:23:05] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [05:23:05] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [05:25:05] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [05:25:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:25:23] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [05:26:33] /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 7766 MB (0% inode=99%): [05:30:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:30:42] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.39, 19.89, 17.01 [05:31:43] Load avg. on willow is WARNING: WARNING - load average: 21.59, 19.18, 16.95 [05:36:52] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [05:42:43] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [06:00:42] Load avg. on willow is CRITICAL: CRITICAL - load average: 35.51, 21.67, 18.86 [06:01:42] Load avg. on willow is WARNING: WARNING - load average: 28.55, 22.21, 19.23 [06:21:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:22:04] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [06:22:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:22:04] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [06:22:04] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [06:24:04] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [06:24:04] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [06:25:04] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [06:25:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:25:23] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [06:30:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:37:04] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [06:43:43] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [07:00:43] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.66, 22.52, 19.36 [07:01:42] Load avg. on willow is WARNING: WARNING - load average: 25.28, 22.12, 19.42 [07:05:42] Load avg. on willow is CRITICAL: CRITICAL - load average: 37.11, 25.30, 21.04 [07:21:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:22:03] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [07:22:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:22:03] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [07:22:03] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [07:24:04] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [07:24:05] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [07:25:04] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [07:25:23] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [07:25:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:30:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:37:05] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [07:44:43] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [08:05:43] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.04, 19.70, 21.21 [08:09:42] Load avg. on willow is WARNING: WARNING - load average: 14.62, 17.20, 19.82 [08:10:42] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.76, 18.31, 20.03 [08:11:42] Load avg. on willow is WARNING: WARNING - load average: 19.65, 18.38, 19.94 [08:21:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:22:03] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [08:23:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:23:03] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [08:23:03] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [08:25:03] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [08:25:03] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [08:25:12] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [08:25:22] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [08:26:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:30:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:37:03] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [08:38:42] Load avg. on willow is WARNING: WARNING - load average: 15.23, 15.95, 17.07 [08:45:42] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [09:22:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:23:02] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [09:23:43] Load avg. on willow is OK: OK - load average: 10.38, 12.89, 14.79 [09:24:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:24:03] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [09:24:03] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [09:25:02] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [09:25:03] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [09:25:11] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [09:26:22] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [09:27:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:31:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:38:03] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [09:45:36] WHEN IS NIGHTSHADE GOING TO BE BACK... :( [09:46:42] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [09:50:16] never [10:02:43] Load avg. on willow is WARNING: WARNING - load average: 16.79, 16.40, 14.66 [10:08:42] Load avg. on willow is OK: OK - load average: 13.28, 14.91, 14.61 [10:12:42] Load avg. on willow is WARNING: WARNING - load average: 22.20, 17.86, 15.80 [10:22:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:23:02] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [10:25:03] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [10:25:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:25:03] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [10:25:03] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [10:25:03] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [10:25:13] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [10:26:22] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [10:27:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:29:42] Load avg. on willow is OK: OK - load average: 12.75, 14.09, 14.89 [10:32:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:32:43] Load avg. on willow is WARNING: WARNING - load average: 13.66, 14.72, 15.07 [10:34:43] Load avg. on willow is OK: OK - load average: 13.80, 14.45, 14.92 [10:38:03] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [10:46:43] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19778 [11:02:43] Load avg. on willow is WARNING: WARNING - load average: 19.35, 18.43, 15.96 [11:22:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:23:02] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [11:25:13] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [11:26:02] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [11:26:02] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [11:26:02] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [11:26:02] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [11:26:02] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:26:22] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [11:27:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:32:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:33:45] Hello all [11:38:02] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [11:46:43] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12771 [11:52:02] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42061 MB (4% inode=99%): [11:57:43] Load avg. on willow is OK: OK - load average: 12.97, 13.89, 14.91 [12:00:43] Load avg. on willow is WARNING: WARNING - load average: 29.67, 18.84, 16.54 [12:03:59] @replag [12:03:59] DaBPunkt: s1-sec-c: 1m 9s [-0.13 s/s]; s2-pri: error; s2/s5-pri-c: error; s3-rr: 1m 13s [-0.02 s/s]; s3-user: 1m 13s [-0.02 s/s]; s4-rr: 1m 9s [-0.13 s/s]; s5-user: error; s7-rr: 2h 40m 48s [-2.35 s/s] [12:04:00] DaBPunkt: s7-user: 2h 40m 48s [-2.35 s/s] [12:09:07] @replag [12:09:07] DaBPunkt: s2-pri: error; s2/s5-pri-c: error; s3-rr: 24s [-]; s3-user: 25s [-]; s5-rr: error; s5-user: error; s5-user-c: error; s7-rr: 2h 32m 10s [-] [12:09:08] DaBPunkt: s7-user: 2h 32m 10s [-] [12:09:26] 3(commented) [TS-1309] daphne s2/s5 corrupted and offline <10https://jira.toolserver.org/browse/TS-1309> (DaB.) [12:10:44] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.50, 20.40, 17.93 [12:12:11] @replag [12:12:11] DaBPunkt: s2-pri: error; s2/s5-pri-c: error; s5-user-c: error; s7-rr: 2h 24m 34s [-]; s7-user: 2h 24m 34s [-] [12:13:21] 3(created) [ACCAPP-458] Run bots in ptwiki; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-458> (Francisco Carvalho Venancio) [12:14:19] @replag [12:14:19] DaBPunkt: s2-pri: error; s2/s5-pri-c: error; s5-user-c: 1d 1h 47m 43s [-]; s7-rr: 2h 18m 21s [-2.92 s/s]; s7-user: 2h 18m 21s [-2.92 s/s] [12:20:05] @replag [12:20:05] DaBPunkt: s2-user: error; s2-user-c: error; s5-user-c: 1d 58m 58s [-]; s7-rr-a: error; s7-user: 1h 56m 6s [-] [12:22:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:23:03] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [12:24:20] 3(created) [MNT-1204] Made the tsbot-replag-display consistent; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1204> (DaB.) [12:24:26] 3(resolved) [MNT-1204] Made the tsbot-replag-display consistent <10https://jira.toolserver.org/browse/MNT-1204> (DaB.) [12:25:23] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [12:26:02] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [12:26:03] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [12:26:03] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [12:26:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:26:24] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [12:26:24] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [12:27:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:28:44] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3416 [12:30:44] Load avg. on willow is WARNING: WARNING - load average: 17.43, 14.68, 15.54 [12:31:44] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [12:32:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:33:44] Load avg. on willow is OK: OK - load average: 11.77, 13.36, 14.86 [12:37:45] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2485 [12:38:03] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [13:02:43] Load avg. on willow is WARNING: WARNING - load average: 17.48, 16.86, 15.03 [13:04:52] DaBPunkt, hello :) [13:08:54] Load avg. on willow is OK: OK - load average: 13.30, 14.85, 14.76 [13:12:53] Load avg. on willow is WARNING: WARNING - load average: 18.25, 17.26, 15.82 [13:22:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:23:03] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [13:25:24] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [13:26:04] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [13:26:04] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [13:26:05] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [13:26:13] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:26:24] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [13:26:24] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [13:27:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:32:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:38:04] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [13:40:21] 3(created) [GEOHACK-17] Wrong UK grid coordinates being generated (wrong Earth model?); GeoHack; Bug <10https://jira.toolserver.org/browse/GEOHACK-17> (TarquinWJ) [13:58:48] @replag [13:58:48] DaBPunkt: s2-user: error; s2-user-c: error; s3-rr-a: 57s [-]; s3-user: 57s [-]; s5-user-c: 3h 11m 4s [-13.25 s/s]; s7-rr-a: error [14:11:02] [[Admin:Puppet]] 10https://wiki.toolserver.org/w/index.php?diff=6695&oldid=6049&rcid=8842 * Dab * (+709) (+Adding a server to puppet ) [14:22:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:23:04] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [14:25:24] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [14:26:04] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [14:26:04] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [14:26:13] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:26:24] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [14:26:24] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [14:27:04] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [14:28:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:32:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:38:04] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [14:56:24] 3(created) [TS-1311] Install Perl modules LWP::Protocol::http; Toolserver; Minor Task <10https://jira.toolserver.org/browse/TS-1311> (Schutz ) [15:06:20] 3(updated) [TS-1311] Install Perl module LWP::Protocol::https <10https://jira.toolserver.org/browse/TS-1311> (Schutz ) [15:23:05] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [15:23:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:25:34] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [15:26:04] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [15:26:04] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [15:26:12] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:26:34] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [15:26:34] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [15:27:04] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [15:29:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:31:53] Load avg. on willow is WARNING: WARNING - load average: 23.97, 18.53, 14.22 [15:32:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:36:20] 3(commented) [ACCAPP-447] To run a Bot for scheduled work and routine event driven tasks on Marathi Wiipedia (mr.wiki). <10https://jira.toolserver.org/browse/ACCAPP-447> (Rahuldeshmukh101) [15:37:53] Load avg. on willow is OK: OK - load average: 10.75, 14.81, 14.09 [15:38:04] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [16:23:05] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [16:23:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:25:34] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [16:26:04] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [16:26:05] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [16:26:14] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:27:04] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [16:27:34] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [16:27:34] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [16:29:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:33:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:38:04] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [17:12:50] who is user dm? [17:15:09] ah, found it [17:23:10] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [17:24:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:25:37] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [17:26:09] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [17:26:09] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [17:26:16] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:27:08] /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 7776 MB (0% inode=99%): [17:27:09] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [17:27:37] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [17:27:46] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [17:30:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:31:12] [[Main Page]] ! 10https://wiki.toolserver.org/w/index.php?diff=6696&oldid=6664&rcid=8843 * Nemobis * (+1) (period) [17:33:38] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:38:07] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [17:38:43] DaBPunkt: hey :) are you around ? [17:42:57] or is there anybody else around who can tell me where we are in the building of the MMP interwiki-bot ? [17:44:51] Quentinv57: hello :) tu es là ? [17:45:22] oui salut Toto_Azero :) [17:45:43] Quentinv57: tu saurais pas où on en est avec le MMP toi ? [17:46:00] je pense qu'il doit �tre occup�, j'ai essay� de le joindre ce matin pour lui dire que pas mal de serveurs de la base de donn�es ne r�pondent plus [17:46:08] MMP = ? [17:46:20] multi-maintainer project [17:47:04] Toto_Azero, I don't really know about this... you should ask somebody else, Platonides seems to be aware ;) [17:47:18] I assume it's at /home/project/i/n/t/interwikibot [17:49:14] Platonides: thx :) do you know if there is any guy to look after this ? I mean, DaBPunkt recommended 5 persons if I remember well… [17:52:26] no ? [17:54:07] I don't see anyone belonging to that project yet [17:54:55] ok thx [18:17:26] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38099 MB (9% inode=99%): [18:23:09] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [18:25:08] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:25:46] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [18:26:16] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:27:09] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [18:27:09] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [18:27:09] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [18:27:27] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 117284 MB (29% inode=99%): [18:27:37] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [18:27:46] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [18:31:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:34:38] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:38:10] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [19:24:08] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [19:25:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:25:49] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [19:26:17] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:27:09] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [19:27:09] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [19:27:09] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [19:27:48] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [19:27:48] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [19:31:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:34:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:39:09] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [19:53:17] s4 replag on yarrow is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on yarrow (146) [19:53:49] MySQL slave on yarrow is CRITICAL: Cant connect to MySQL server on yarrow (146) [19:53:49] MySQL on yarrow is CRITICAL: Cant connect to MySQL server on yarrow (146) [19:53:49] s2 replag on yarrow is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on yarrow (146) [19:53:49] s5 replag on yarrow is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on yarrow (146) [19:54:09] / on yarrow is CRITICAL: Connection refused by host [19:54:09] Environment on yarrow is CRITICAL: Connection refused by host [19:54:09] Load avg. on yarrow is CRITICAL: Connection refused by host [19:54:09] /aux0 on yarrow is CRITICAL: Connection refused by host [19:54:09] Sun Grid Engine execd on yarrow is CRITICAL: Connection refused by host [19:54:10] SMF on yarrow is CRITICAL: Connection refused by host [19:54:10] Cluster on yarrow is CRITICAL: Connection refused by host [19:54:11] /tmp on yarrow is CRITICAL: Connection refused by host [19:54:11] SMTP on yarrow is CRITICAL: Connection refused [19:54:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:55:22] 3(created) [MNT-1205] Yarrow's DRAC needs a reset; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1205> (DaB.) [20:00:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:00:50] Load avg. on yarrow is CRITICAL: Connection refused by host [20:00:50] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:02:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:02:10] Environment on yarrow is CRITICAL: Connection refused by host [20:02:10] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [20:02:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:02:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:02:17] Load avg. on yarrow is CRITICAL: Connection refused by host [20:03:38] SMTP on yarrow is CRITICAL: Connection refused [20:03:47] / on yarrow is CRITICAL: Connection refused by host [20:22:08] Environment on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:22:18] Load avg. on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:22:27] SMF on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:22:47] / on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:22:48] /tmp on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:22:48] Sun Grid Engine execd on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:22:57] Cluster on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:27:27] SMF on yarrow is CRITICAL: Connection refused by host [20:27:48] / on yarrow is CRITICAL: Connection refused by host [20:27:48] /tmp on yarrow is CRITICAL: Connection refused by host [20:27:48] Sun Grid Engine execd on yarrow is CRITICAL: Connection refused by host [20:27:58] Cluster on yarrow is CRITICAL: Connection refused by host [20:28:08] Environment on yarrow is CRITICAL: Connection refused by host [20:28:17] Load avg. on yarrow is CRITICAL: Connection refused by host [20:29:28] SMF on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:29:48] / on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:29:48] /tmp on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:29:48] Sun Grid Engine execd on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:29:58] Cluster on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:30:07] Environment on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:30:17] Load avg. on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:44:48] / on yarrow is CRITICAL: Connection refused by host [20:44:48] /tmp on yarrow is CRITICAL: Connection refused by host [20:44:48] Sun Grid Engine execd on yarrow is CRITICAL: Connection refused by host [20:44:57] Cluster on yarrow is CRITICAL: Connection refused by host [20:45:08] Environment on yarrow is CRITICAL: Connection refused by host [20:45:18] Load avg. on yarrow is CRITICAL: Connection refused by host [20:45:27] SMF on yarrow is CRITICAL: Connection refused by host [20:53:27] SMF on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:53:48] / on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:53:48] /tmp on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:53:48] Sun Grid Engine execd on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:53:57] Cluster on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:54:08] Environment on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:54:17] Load avg. on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:01:27] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [21:01:37] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [21:01:48] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [21:01:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:01:57] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [21:01:57] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [21:02:07] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:02:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:02:07] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [21:02:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:02:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:02:17] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [21:02:57] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [21:03:49] SMTP on yarrow is CRITICAL: Connection refused [21:41:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:41:34] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [21:41:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:41:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:41:42] Load avg. on yarrow is OK: OK - load average: 0.00, 0.04, 0.01 [21:47:02] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - 170 bytes in 0.472 second response time [21:48:12] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 10.354 second response time [21:53:12] / on yarrow is OK: DISK OK - free space: / 3959 MB (88% inode=94%): [21:53:12] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [21:53:22] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [21:53:52] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [21:54:21] 3(commented) [ACCAPP-451] Urgent Account Creation <10https://jira.toolserver.org/browse/ACCAPP-451> (Cyberpower678) [21:54:31] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [21:59:11] [[User talk:Dab]] ! 10https://wiki.toolserver.org/w/index.php?diff=6697&oldid=6626&rcid=8844 * Josh * (+218) (/* Re-enable account "josh" */ new section) [22:08:13] @replag [22:08:13] Alchimista: s1-rr-a-c: 49s [-]; s2-user: error; s2-user-c: error; s3-rr-a: 57s [-]; s3-user: 57s [-]; s7-rr-a: error [22:35:21] 3(commented) [ACCAPP-458] Run bots in ptwiki <10https://jira.toolserver.org/browse/ACCAPP-458> (Francisco Carvalho Venancio) [22:37:35] having trouble wit a script for one tparis, anyone know of him or his script? [22:37:49] Its a replacement for soxreds script [22:38:46] seems like it works for en.wp but not for no.wp and the page itself says it should work for no.wp among other [22:39:47] jeblad: nowiki is on cluster 2 [22:39:58] so? [22:40:02] * jeblad dumb [22:40:06] we have no working copy of cluster 2 at the moment [22:40:17] aha [22:40:51] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [22:40:59] Then I tell the folks on Signpost that someone is working on it and they must sit down calmly and wait [22:41:02] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [22:41:02] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [22:41:12] SMTP on yarrow is CRITICAL: Connection refused [22:41:13] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [22:41:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:41:22] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [22:41:22] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [22:41:22] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [22:41:22] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:41:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:41:32] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [22:41:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:41:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:42:02] jeblad: we wait for a dump by the wmf-folks at the moment [22:42:45] okey, I'll throw the wild bunch at the signpost a bone ;) [22:53:12] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [22:53:22] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [22:54:01] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [22:54:31] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [23:10:11] toolserver: 03Ben Kurtovic 07master * r82bee17 10/ static/js/potd.js : Better handling of images less wide than our screen. - http://git.io/Ms6vRw [23:10:11] toolserver: 03Ben Kurtovic 07master * red086ef 10/ pages/earwigbot.mako : Grammar - http://git.io/O48hTw [23:10:12] toolserver: 03Ben Kurtovic 07master * rc482d9f 10/ (4 files in 3 dirs): [23:10:12] toolserver: Work on Copyvio Detector and integration with EarwigBot [23:10:12] toolserver: copyvios.mako can now import the earwigbot package and load pages [23:10:12] toolserver: gitignored config.ts-earwigbot.json - http://git.io/ce8jYA [23:12:26] toolserver: 03Ben Kurtovic 07master * r910a24a 10/ (3 files in 3 dirs): Minor updates. - http://git.io/YZgvvg [23:12:28] toolserver: 03Ben Kurtovic 07master * r7d92f7a 10/ (3 files in 3 dirs): Copyvio detection interface is (mostly) done. - http://git.io/k1BRaQ [23:17:38] what's this [23:18:23] oh, someone named his project 'toolserver' [23:23:13] ok, contacted Ben, it has been fixed [23:40:52] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [23:41:02] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [23:41:02] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [23:41:12] SMTP on yarrow is CRITICAL: Connection refused [23:41:12] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [23:41:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:41:22] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [23:41:22] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [23:41:22] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [23:41:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:41:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:41:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:41:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:42:31] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [23:49:54] @replag [23:49:55] DaBPunkt: s2-user: error; s2-user-c: error; s3-rr-a: 36s [-0.35 s/s]; s3-user: 36s [-0.35 s/s]; s7-rr-a: error [23:51:50] @replag [23:51:50] DaBPunkt: s1-rr-a-c: 31s [-]; s2-user: error; s2-user-c: error; s3-rr-a: 11s [-]; s3-user: 11s [-] [23:51:56] stupid typo… [23:52:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 41418 MB (4% inode=99%): [23:53:12] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [23:53:22] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [23:54:02] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [23:54:24] @replag [23:54:24] DaBPunkt: s1-rr-a-c: 1m 8s [-]; s2-user: error; s2-user-c: error [23:54:30] @replag all [23:54:31] DaBPunkt: s1-rr-a: 2s [-]; s1-rr-a-c: 1m 15s [+1.05 s/s]; s1-user: 1s [-]; s2-user: error; s2-user-c: error; s3-rr-a: 7s [-]; s3-user: 7s [-]; s4-rr-a: 1s [-] [23:54:32] DaBPunkt: s4-user: 1s [-]; s5-rr-a: 3s [-]; s5-user: 3s [-]; s5-user-c: 1s [-]; s6-rr-a: 3s [-]; s6-user: 3s [-]; s7-rr-a: 1s [-]; s7-user: 1s [-] [23:55:27] nacht ts [23:55:31] Environment on yarrow is UNKNOWN: NRPE: Unable to read output