[00:02:07] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [00:08:50] @replag [00:08:50] DaBPunkt: s2-user: 4d 9h 46m 55s [-1.79 s/s]; s3-rr-a: 11s [-0.00 s/s]; s3-user: 11s [-0.00 s/s]; s5-rr-a: 1h 51m 21s [+1.00 s/s]; s5-user: 1h 51m 21s [+1.00 s/s] [00:10:07] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [00:14:56] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [00:16:56] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [00:18:07] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7233.000000 [00:18:56] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [00:45:11] @replag [00:45:11] DaBPunkt: s1-rr-a-c: 15m 20s [+0.15 s/s]; s2-user: 4d 9h 1m 16s [-1.26 s/s]; s3-rr-a: 4m 11s [+0.11 s/s]; s3-user: 4m 11s [+0.11 s/s]; s5-rr-a: 2h 27m 42s [+1.00 s/s]; s5-user: 2h 27m 42s [+1.00 s/s]; s6-rr-a: 18m 23s [+0.03 s/s]; s6-user: 18m 23s [+0.03 s/s] [00:47:38] dewiki adds rev_sha1 at the moment, that will take some time [00:51:18] [[Dbbot-wm]] N 10https://wiki.toolserver.org/w/index.php?oldid=6708&rcid=8862 * Krinkle * (+431) (Created page with "{{DISPLAYTITLE:{{lcfirst:{{FULLPAGENAME}}}}}} '''dbbot-wm''' is a bot run by [[mw:User:Krinkle|krinkle]]. It runs [[Kribo]] with the [[wmfDbBot]] plugin. == Config == === Chan...") [00:52:06] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:52:16] [[WmfDbBot]] 10https://wiki.toolserver.org/w/index.php?diff=6709&oldid=6707&rcid=8863 * Krinkle * (+9) () [00:52:49] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [00:54:07] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [00:54:07] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [00:54:18] dewiki adds rev_sha1 at the moment, that will take some time [00:54:21] sorry [00:54:23] @replag [00:54:23] DaBPunkt: s1-rr-a-c: 21m 24s [+0.66 s/s]; s2-user: 4d 8h 50m 2s [-1.22 s/s]; s3-rr-a: 1m 33s [-0.29 s/s]; s3-user: 1m 33s [-0.29 s/s]; s5-rr-a: 2h 36m 54s [+1.00 s/s]; s5-user: 2h 36m 54s [+1.00 s/s]; s6-rr-a: 27m 35s [+1.00 s/s]; s6-user: 27m 35s [+1.00 s/s] [00:54:47] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1954 [00:55:47] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [00:55:49] nacht ts [00:57:48] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [00:57:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:58:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:58:47] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:58:47] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [00:59:49] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [00:59:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:02:07] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [01:09:16] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1886.000000 [01:11:06] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [01:14:55] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [01:16:56] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [01:18:16] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10842.000000 [01:22:02] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [01:22:48] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3634 [01:48:18] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3625.000000 [01:52:07] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:53:49] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [01:54:17] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [01:54:17] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [01:55:48] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [01:57:48] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [01:57:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:58:17] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.36, 31.50, 20.46 [01:58:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:59:47] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:59:47] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [02:00:48] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [02:00:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:02:08] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [02:02:17] Load avg. on willow is WARNING: WARNING - load average: 15.57, 23.24, 19.51 [02:12:06] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [02:14:56] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [02:16:56] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [02:17:16] Load avg. on willow is OK: OK - load average: 10.91, 12.93, 14.81 [02:18:18] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14443.000000 [02:18:56] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [02:22:17] Load avg. on willow is WARNING: WARNING - load average: 15.11, 14.50, 14.92 [02:22:48] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7234 [02:44:57] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1930 [02:49:16] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6137.000000 [02:52:07] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:53:58] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [02:54:17] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [02:54:17] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [02:55:58] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [02:57:58] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [02:57:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:58:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:59:58] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:59:58] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [03:00:58] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [03:00:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:03:08] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [03:12:08] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [03:12:57] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3612 [03:14:58] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [03:16:58] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [03:18:57] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [03:19:16] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18104.000000 [03:22:57] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10843 [03:49:17] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8548.000000 [03:52:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:54:08] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [03:54:17] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [03:54:17] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [03:56:17] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [03:58:07] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:58:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:58:57] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [04:00:06] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:00:06] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [04:01:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:01:57] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [04:03:17] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [04:12:58] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7213 [04:13:07] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [04:15:08] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [04:16:25] so…. more databases are down? [04:17:07] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [04:19:07] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [04:19:17] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21704.000000 [04:20:08] mys_721tx: They all seem to be up. [04:20:37] Pathoschild: that's great. Global contribution query works :-) [04:20:48] :) [04:23:06] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14448 [04:41:16] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1933.000000 [04:43:57] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1928 [04:45:29] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1962.000000 [04:46:57] MySQL slave on rosemary is OK: Uptime: 1004221 Threads: 19 Questions: 262665411 Slow queries: 97936 Opens: 1466 Flush tables: 1 Open tables: 646 Queries per second avg: 261.561 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 729 [04:49:28] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10959.000000 [04:52:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:54:16] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [04:54:28] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [04:54:29] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [04:56:17] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [04:58:07] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:58:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:58:57] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [05:01:06] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:01:06] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [05:01:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:02:56] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [05:03:17] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [05:06:58] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1929 [05:07:07] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1937 [05:13:20] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [05:13:20] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3640.000000 [05:13:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3645.000000 [05:13:57] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10873 [05:16:07] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [05:18:07] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [05:19:28] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25310.000000 [05:20:06] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [05:23:07] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18050 [05:30:57] /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 8353 MB (0% inode=99%): [05:33:07] MySQL slave on thyme is OK: Uptime: 1004599 Threads: 13 Questions: 424010004 Slow queries: 140918 Opens: 43447 Flush tables: 1 Open tables: 2606 Queries per second avg: 422.68 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1217 [05:34:57] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3611 [05:36:57] MySQL slave on rosemary is OK: Uptime: 1007222 Threads: 30 Questions: 262794856 Slow queries: 98290 Opens: 1503 Flush tables: 1 Open tables: 669 Queries per second avg: 260.910 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1451 [05:44:57] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1931 [05:45:07] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1937 [05:47:27] Load avg. on willow is WARNING: WARNING - load average: 15.52, 14.86, 13.89 [05:49:28] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8992.000000 [05:49:28] Load avg. on willow is OK: OK - load average: 12.96, 14.26, 13.79 [05:52:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:54:18] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [05:55:27] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [05:55:27] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [05:56:17] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [05:58:17] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:58:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:58:57] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [06:01:06] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:02:07] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [06:02:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:02:28] Load avg. on willow is WARNING: WARNING - load average: 22.06, 17.58, 15.32 [06:02:56] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [06:03:17] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [06:12:57] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3611 [06:13:07] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3617 [06:13:57] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14476 [06:14:17] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [06:14:17] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7298.000000 [06:14:28] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7304.000000 [06:16:06] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [06:18:06] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [06:20:26] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28972.000000 [06:21:07] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [06:21:27] Load avg. on willow is OK: OK - load average: 10.77, 13.29, 14.62 [06:24:06] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21710 [06:24:27] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3284.000000 [06:27:27] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1756.000000 [06:52:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:55:16] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [06:55:26] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [06:55:26] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [06:57:16] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [06:57:27] Load avg. on willow is WARNING: WARNING - load average: 20.35, 16.23, 14.52 [06:58:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:58:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:58:57] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [07:00:27] Load avg. on willow is CRITICAL: CRITICAL - load average: 36.78, 23.09, 17.43 [07:01:07] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:02:07] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [07:02:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:02:27] Load avg. on willow is WARNING: WARNING - load average: 25.57, 24.39, 18.69 [07:03:56] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [07:04:16] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [07:13:57] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7271 [07:13:57] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18076 [07:14:06] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7639 [07:14:16] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10898.000000 [07:15:15] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [07:15:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7439.000000 [07:16:06] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [07:19:07] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [07:21:26] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32631.000000 [07:22:06] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [07:24:07] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25310 [07:25:26] Load avg. on willow is WARNING: WARNING - load average: 22.54, 17.12, 17.14 [07:30:26] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.85, 20.16, 18.16 [07:32:26] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3557.000000 [07:32:26] Load avg. on willow is WARNING: WARNING - load average: 17.86, 20.26, 18.57 [07:33:06] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3342 [07:38:29] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1729.000000 [07:39:06] MySQL slave on thyme is OK: Uptime: 1012158 Threads: 8 Questions: 428121589 Slow queries: 141446 Opens: 43625 Flush tables: 1 Open tables: 2593 Queries per second avg: 422.979 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1461 [07:48:02] @replag all [07:48:03] nosy: s1-rr-a: 3h 35m 28s [+0.05 s/s]; s1-rr-a-c: 1m 23s [-0.05 s/s]; s1-user: 1s [-0.00 s/s]; s2-user: 4d 1h 17m 35s [-1.09 s/s]; s2-user-c: 6s [-0.08 s/s]; s3-rr-a: 54s [-0.00 s/s]; s3-user: 54s [-0.00 s/s]; s4-rr-a: 6s [+0.00 s/s] [07:48:04] nosy: s4-user: 6s [+0.00 s/s]; s5-rr-a: 9h 30m 34s [+1.00 s/s]; s5-user: 9h 30m 34s [+1.00 s/s]; s5-user-c: 6s [-0.08 s/s]; s6-rr-a: 7h 21m 15s [+1.00 s/s]; s6-user: 7h 21m 15s [+1.00 s/s]; s7-rr-a: 1s [-0.00 s/s]; s7-user: 1s [-0.00 s/s] [07:52:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:55:17] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [07:55:26] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [07:55:27] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [07:57:17] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [07:57:57] /sql on daphne is OK: DISK OK - free space: /aux0 143119 MB (14% inode=99%): [07:58:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:58:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:59:56] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [08:01:06] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:02:06] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [08:03:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:03:58] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [08:05:17] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [08:13:56] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11923 [08:13:56] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21676 [08:14:17] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11850.000000 [08:15:17] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [08:16:06] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [08:19:06] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [08:21:26] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36232.000000 [08:23:06] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [08:25:06] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28970 [08:27:26] Load avg. on willow is OK: OK - load average: 12.05, 13.84, 14.82 [08:31:21] 3(commented) [MNT-1192] s3 and s7 slaves were stopped <10https://jira.toolserver.org/browse/MNT-1192> (Marlen Caemmerer) [08:45:56] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3522 [08:46:26] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3344.000000 [08:50:27] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1797.000000 [08:50:56] MySQL slave on rosemary is OK: Uptime: 1018864 Threads: 19 Questions: 265621992 Slow queries: 99845 Opens: 1526 Flush tables: 1 Open tables: 685 Queries per second avg: 260.704 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1525 [08:52:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:55:17] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [08:55:27] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [08:55:27] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [08:57:17] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [08:58:17] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:58:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:59:56] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [09:01:06] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:02:08] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default svc:/application/ts/mysql-51:default [09:04:07] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [09:04:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:06:17] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [09:14:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25279 [09:14:17] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39680 MB (9% inode=99%): [09:15:17] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [09:16:06] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [09:19:16] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [09:20:17] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 98740 MB (24% inode=99%): [09:21:26] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39834.000000 [09:23:07] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [09:25:06] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32573 [09:52:24] 3(resolved) [GEOHACK-18] HTML that can't even stand a different font size <10https://jira.toolserver.org/browse/GEOHACK-18> (Dispenser) [09:52:56] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:55:17] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [09:56:25] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [09:56:26] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [09:57:17] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [09:58:17] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:58:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:00:56] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [10:01:06] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:02:07] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default svc:/application/ts/mysql-51:default [10:02:26] Load avg. on willow is WARNING: WARNING - load average: 16.76, 14.75, 13.16 [10:04:08] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [10:04:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:05:26] Load avg. on willow is OK: OK - load average: 14.00, 14.51, 13.36 [10:07:16] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [10:14:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28879 [10:15:17] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [10:16:07] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [10:19:17] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [10:21:26] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43434.000000 [10:23:06] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [10:25:07] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36173 [10:25:17] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37203 MB (9% inode=99%): [10:28:16] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24418 MB (5% inode=99%): [10:29:21] 3(created) [ACCAPP-463] Request for extension user:Ab2llah; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-463> (Abdollah Abdollahi) [10:35:27] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 26328 MB (6% inode=99%): [10:37:51] @replag al [10:37:52] nosy: s2-user: 4d 1h 11m 54s [-0.03 s/s]; s3-rr-a: 25s [-0.00 s/s]; s3-user: 25s [-0.00 s/s]; s5-rr-a: 12h 20m 22s [+1.00 s/s]; s5-user: 12h 20m 22s [+1.00 s/s]; s6-rr-a: 10h 11m 3s [+1.00 s/s]; s6-user: 10h 11m 3s [+1.00 s/s] [10:37:57] @¶eplag all [10:38:01] @replag all [10:38:02] nosy: s1-rr-a: 0s [-1.27 s/s]; s1-rr-a-c: 0s [-0.01 s/s]; s1-user: 0s [-0.00 s/s]; s2-user: 4d 1h 12m 4s [+1.00 s/s]; s2-user-c: 0s [-0.00 s/s]; s3-rr-a: 35s [+1.00 s/s]; s3-user: 36s [+1.08 s/s]; s4-rr-a: 1s [-0.00 s/s] [10:38:03] nosy: s4-user: 1s [-0.00 s/s]; s5-rr-a: 12h 20m 33s [+1.08 s/s]; s5-user: 12h 20m 33s [+1.08 s/s]; s5-user-c: 1s [-0.00 s/s]; s6-rr-a: 10h 11m 14s [+1.08 s/s]; s6-user: 10h 11m 14s [+1.08 s/s]; s7-rr-a: 3s [+0.00 s/s]; s7-user: 3s [+0.00 s/s] [10:46:20] 3(commented) [MNT-1200] Clean-up nagios-config <10https://jira.toolserver.org/browse/MNT-1200> (Marlen Caemmerer) [10:54:20] 3(commented) [ACCAPP-463] Request for extension user:Ab2llah <10https://jira.toolserver.org/browse/ACCAPP-463> (Abdollah Abdollahi) [11:28:40] Hello all [12:34:16] [[Main Page]] !M 10https://wiki.toolserver.org/w/index.php?diff=6710&oldid=6696&rcid=8864 * Sebastian Sooth (WMDE) * (-7) (url fixed) [12:43:38] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1.000000 [12:43:38] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:43:46] MySQL slave on cassia is OK: Uptime: 1020086 Threads: 29 Questions: 391132231 Slow queries: 58420 Opens: 1183994 Flush tables: 2 Open tables: 11370 Queries per second avg: 383.430 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [12:43:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:43:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:47:21] 3(commented) [GEOHACK-18] HTML that can't even stand a different font size <10https://jira.toolserver.org/browse/GEOHACK-18> (Jidanni) [13:18:46] [[Talk:Main Page]] ! 10https://wiki.toolserver.org/w/index.php?diff=6711&oldid=6623&rcid=8865 * 69.178.193.214 * (-1) () [13:42:55] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [13:43:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:43:05] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [13:43:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:43:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41425 [13:43:15] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [13:43:35] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:44:15] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [13:44:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:46:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:10:20] 3(updated) [MNT-52] Extending of user-account of purodha and updating ssh-key <10https://jira.toolserver.org/browse/MNT-52> (Marlen Caemmerer) [14:18:57] 3(closed) [MNT-1020] Add svn to MMT mwbot with fishEye-view <10https://jira.toolserver.org/browse/MNT-1020> (Marlen Caemmerer) [14:19:01] 3(closed) [MNT-775] Create MMT "newpagesdelayed" <10https://jira.toolserver.org/browse/MNT-775> (Marlen Caemmerer) [14:20:52] 3(updated) [MNT-1063] damiana is away <10https://jira.toolserver.org/browse/MNT-1063> (Marlen Caemmerer) [14:20:54] 3(updated) [MNT-900] Update debian-package for libgdiplus <10https://jira.toolserver.org/browse/MNT-900> (Marlen Caemmerer) [14:20:59] 3(closed) [MNT-950] Cretae view for ruewiki <10https://jira.toolserver.org/browse/MNT-950> (Marlen Caemmerer) [14:22:56] 3(commented) [OSM-7] Add extra indexes to make tile rendering faster <10https://jira.toolserver.org/browse/OSM-7> (Marlen Caemmerer) [14:23:00] 3(moved) [OSM-7] Add extra indexes to make tile rendering faster <10https://jira.toolserver.org/browse/OSM-7> (Marlen Caemmerer) [14:23:04] 3(commented) [MNT-715] Move the lib- and the webapp-directory of tomcat to /shared <10https://jira.toolserver.org/browse/MNT-715> (Marlen Caemmerer) [14:23:07] 3(assigned) [MNT-715] Move the lib- and the webapp-directory of tomcat to /shared <10https://jira.toolserver.org/browse/MNT-715> (Marlen Caemmerer) [14:26:54] 3(closed) [MNT-832] Update Munin config for OSM <10https://jira.toolserver.org/browse/MNT-832> (Marlen Caemmerer) [14:28:59] 3(updated) [MNT-709] Test FreeBSD performance on hyacinth <10https://jira.toolserver.org/browse/MNT-709> (Marlen Caemmerer) [14:29:00] 3(assigned) [MNT-668] Add missing quotas <10https://jira.toolserver.org/browse/MNT-668> (Marlen Caemmerer) [14:29:02] 3(closed) [MNT-761] Extend account of jylee <10https://jira.toolserver.org/browse/MNT-761> (Marlen Caemmerer) [14:29:05] 3(assigned) [MNT-400] Upgrade trainwreck <10https://jira.toolserver.org/browse/MNT-400> (Marlen Caemmerer) [14:35:53] 3(updated) [MNT-985] Install TStcl85-zlib <10https://jira.toolserver.org/browse/MNT-985> (Marlen Caemmerer) [14:36:00] 3(closed) [MNT-657] Change innodb_lock_wait_timeout to 3600 <10https://jira.toolserver.org/browse/MNT-657> (Marlen Caemmerer) [14:36:01] 3(updated) [MNT-453] Reinstall clematis <10https://jira.toolserver.org/browse/MNT-453> (Marlen Caemmerer) [14:42:53] 3(created) [MAGNUS-300] Replacing {{NowCommons}} by {{NowCommons (MtC drive)}}; Magnus' tools: CommonsHelper; Critical Task <10https://jira.toolserver.org/browse/MAGNUS-300> (Ebe123(b)) [14:42:55] 3(resolved) [TS-1212] Adjust logadm on wolfsbane <10https://jira.toolserver.org/browse/TS-1212> (Marlen Caemmerer) [14:43:06] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [14:43:06] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [14:43:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:43:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 45025 [14:43:06] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:43:16] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [14:43:35] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:44:15] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [14:45:06] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:45:06] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:46:52] 3(created) [CHTWO-43] Replacement of {{NowCommons}} by {{NowCommons (MtC drive)}} in March; Commonshelper2; Critical Task <10https://jira.toolserver.org/browse/CHTWO-43> (Ebe123(b)) [15:43:04] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [15:43:15] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48627 [15:43:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:43:15] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [15:43:35] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:44:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:44:05] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [15:44:15] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [15:45:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:46:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:47:55] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1962.000000 [16:05:54] 3(reopened) [DBQ-144] Query to create translated interlanguage links <10https://jira.toolserver.org/browse/DBQ-144> (Stavros Giannou) [16:12:55] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1604.000000 [16:42:56] 3(commented) [OSM-7] Add extra indexes to make tile rendering faster <10https://jira.toolserver.org/browse/OSM-7> (Kai Krueger) [16:43:06] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [16:43:15] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 52228 [16:43:16] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:43:16] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [16:43:35] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:44:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:44:15] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [16:44:25] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [16:45:06] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:45:52] 3(commented) [OSM-6] mapnik tirex-backend occasionally failing <10https://jira.toolserver.org/browse/OSM-6> (Kai Krueger) [16:46:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:52:15] [[User:Faris knight]] !M 10https://wiki.toolserver.org/w/index.php?diff=6712&oldid=6556&rcid=8866 * Faris knight * (-66) () [17:43:06] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [17:43:15] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 55828 [17:43:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:43:35] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:44:14] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [17:44:25] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [17:45:06] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:45:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:45:15] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [17:46:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:46:36] Load avg. on willow is WARNING: WARNING - load average: 24.14, 19.47, 15.98 [18:02:28] [[මුල් පිටුව]] ! 10https://wiki.toolserver.org/w/index.php?diff=6713&oldid=1472&rcid=8867 * 203.189.190.84 * (+0) () [18:04:27] [[Special:Log/newusers]] create 10 * Singhalawap * (New user account) [18:11:25] [[මුල් පිටුව]] ! 10https://wiki.toolserver.org/w/index.php?diff=6714&oldid=6713&rcid=8869 * Singhalawap * (+288) () [18:12:54] 3(resolved) [DBQ-144] Query to create translated interlanguage links <10https://jira.toolserver.org/browse/DBQ-144> (Hoo man) [18:13:11] isn't globalimagelinks available somewhere? [18:15:12] I see a globaldatabaselinks in s4, but it doesn't seem to have a globaldatabaselinks_p view [18:21:53] 3(created) [ACCAPP-464] I want this account to run my bot on wikipedia. it will do pywiki codes; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-464> (Abbas Salamat) [18:26:35] Load avg. on willow is OK: OK - load average: 14.03, 14.69, 15.00 [18:43:06] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [18:43:15] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 59428 [18:43:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:43:36] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:44:15] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [18:45:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:45:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:45:25] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [18:45:25] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [18:46:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:16:19] anyone? DaBPunkt, nosy? [19:16:50] Platonides: yes hello, i am around [19:17:04] you miss a view? [19:17:09] yep [19:17:25] do you know the hostname of the s4 you are using? [19:17:53] 3(assigned) [OSM-5] hstore operator ? no longer uses indexes <10https://jira.toolserver.org/browse/OSM-5> (Marlen Caemmerer) [19:18:19] I wanted to join it with a s4-user table [19:18:36] not sure if that's what you were asking [19:19:21] is globaldatabaselinks a table in commons? [19:19:49] it's another database [19:19:53] i only have globalimagelinks [19:19:53] but is related to commons [19:19:56] ah i see [19:20:46] wow i cannot find a database with this name [19:21:44] hmm... maybe it's a table inside commonswiki db? [19:21:51] it's not replicated in such case [19:21:53] 3(commented) [DBQ-144] Query to create translated interlanguage links <10https://jira.toolserver.org/browse/DBQ-144> (Stavros Giannou) [19:22:06] oh, it is [19:22:28] if i do show databases i cannot see it? [19:22:28] so I had it there the whole time! :/ [19:22:52] well ok, problem solved, but why cant i see it? [19:23:03] it was not on its own database [19:24:39] I apologise for my confusion, nosy [19:24:54] no problem if i have nothing to do fine :) [19:35:52] 3(commented) [DBQ-144] Query to create translated interlanguage links <10https://jira.toolserver.org/browse/DBQ-144> (Hoo man) [19:43:15] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [19:43:15] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 63028 [19:43:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:43:36] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:44:16] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [19:45:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:45:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:45:24] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [19:45:25] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [19:46:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:01:54] 3(commented) [DBQ-144] Query to create translated interlanguage links <10https://jira.toolserver.org/browse/DBQ-144> (Stavros Giannou) [20:04:09] re [20:04:12] @replag [20:04:13] DaBPunkt: s1-rr-a-c: 1m 17s [+0.00 s/s]; s2-user: 4d 6h 56m 57s [+0.61 s/s]; s3-rr-a: 20s [-0.00 s/s]; s3-user: 20s [-0.00 s/s]; s6-rr-a: 19h 37m 24s [+1.00 s/s]; s6-user: 19h 37m 24s [+1.00 s/s] [20:05:30] frwiki gets rev_sha1 – that cause the replag [20:14:35] Load avg. on willow is WARNING: WARNING - load average: 16.45, 14.56, 13.43 [20:16:35] Load avg. on willow is OK: OK - load average: 14.51, 14.71, 13.65 [20:32:37] Load avg. on willow is WARNING: WARNING - load average: 15.33, 14.98, 14.16 [20:43:15] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 66629 [20:43:15] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [20:43:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:43:46] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:45:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:45:16] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [20:45:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:45:25] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [20:45:25] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [20:46:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:54:16] if you have a MediaWiki Subversion account, you probably heard that we are switching to Git and Gerrit [20:54:21] 5 minutes until "Git, Gerrit, and You", my last G+ workshop on Git and Gerrit workflow [20:54:41] if you want to join, now is a good time to come into #mediawiki and get set up .... now is a good time to make sure you can log in to gerrit.wikimedia.org and to G+ [21:14:54] 3(commented) [DBQ-144] Query to create translated interlanguage links <10https://jira.toolserver.org/browse/DBQ-144> (Hoo man) [21:32:46] Load avg. on willow is WARNING: WARNING - load average: 15.33, 16.09, 15.40 [21:38:47] Load avg. on willow is OK: OK - load average: 13.71, 14.64, 14.96 [21:43:15] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 70229 [21:43:16] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [21:43:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:43:46] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:45:16] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [21:45:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:45:25] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [21:45:25] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [21:46:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:46:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:47:05] frwiki gets rev_sha1 – that cause the replag [21:47:06] @replag [21:47:07] DaBPunkt: s1-rr-a-c: 13m 29s [+0.12 s/s]; s2-user: 4d 7h 21m 21s [+0.24 s/s]; s3-rr-a: 58s [+0.01 s/s]; s3-user: 58s [+0.01 s/s]; s6-rr-a: 21h 20m 18s [+1.00 s/s]; s6-user: 21h 20m 18s [+1.00 s/s] [21:47:46] Load avg. on willow is WARNING: WARNING - load average: 16.17, 15.32, 15.10 [21:56:46] Load avg. on willow is OK: OK - load average: 12.15, 14.27, 14.84 [22:43:17] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 73829 [22:43:17] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [22:43:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:43:46] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:45:16] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [22:45:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:45:25] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [22:45:25] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [22:46:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:46:17] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:51:53] 3(work started) [UTRS-56] Duplicate action allowed, should disable 2nd action instead <10https://jira.toolserver.org/browse/UTRS-56> (Andrew Pearson) [22:51:55] 3(assigned) [UTRS-56] Duplicate action allowed, should disable 2nd action instead <10https://jira.toolserver.org/browse/UTRS-56> (Andrew Pearson) [22:55:54] 3(resolved) [UTRS-56] Duplicate action allowed, should disable 2nd action instead <10https://jira.toolserver.org/browse/UTRS-56> (Andrew Pearson) [22:58:52] 3(commented) [UTRS-54] Replace expired IP data with one-way hash <10https://jira.toolserver.org/browse/UTRS-54> (Andrew Pearson) [22:59:19] @replag [22:59:20] DaBPunkt: s1-user: 6m 51s [+0.01 s/s]; s2-user: 4d 7h 41m 12s [+0.27 s/s]; s6-rr-a: 22h 32m 31s [+1.00 s/s]; s6-user: 22h 32m 31s [+1.00 s/s] [23:14:46] Load avg. on willow is WARNING: WARNING - load average: 16.71, 19.39, 15.21 [23:18:45] Load avg. on willow is OK: OK - load average: 10.35, 14.58, 14.14 [23:31:27] @replag [23:31:27] DaBPunkt: s1-user: 20m 10s [+0.41 s/s]; s2-user: 4d 7h 45m 4s [+0.12 s/s]; s3-rr-a: 27s [-0.00 s/s]; s3-user: 27s [-0.00 s/s]; s6-rr-a: 23h 4m 39s [+1.00 s/s]; s6-user: 23h 4m 39s [+1.00 s/s] [23:34:50] Status: broken, lime my heart [23:34:53] *like [23:35:08] Typos ruin everything. [23:43:26] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [23:43:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:43:55] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:43:56] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1899.000000 [23:44:06] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1912 [23:44:15] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 77495 [23:45:25] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [23:45:25] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [23:45:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:45:25] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [23:46:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:46:26] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:55:45] @replag [23:55:45] DaBPunkt: s1-user: 43m 36s [+0.96 s/s]; s2-user: 4d 7h 59m 25s [+0.59 s/s]; s3-rr-a: 1m 14s [+0.03 s/s]; s3-user: 1m 14s [+0.03 s/s]; s5-rr-a: 13s [-0.93 s/s]; s5-user: 13s [-0.93 s/s]; s6-rr-a: 23h 28m 57s [+1.00 s/s]; s6-user: 23h 28m 57s [+1.00 s/s]