[00:21:33] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 22.22, 24.50, 26.60 [00:22:31] Free Memory on damiana is WARNING: WARNING - 6.6% (554980 kB) free! [00:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7165.000000 [00:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7157 [00:28:32] Free Memory on damiana is OK: OK - 7.2% (600584 kB) free. [00:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:43:31] Load avg. on ortelius is WARNING: WARNING - load average: 17.92, 17.78, 19.99 [00:44:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 28.64, 20.32, 20.72 [00:47:01] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2059.000000 [00:49:22] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 21690 MB (3% inode=99%): [00:54:31] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:01:22] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24228 MB (5% inode=99%): [01:07:32] Load avg. on ortelius is WARNING: WARNING - load average: 19.23, 19.35, 19.96 [01:19:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3636.000000 [01:20:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 26.48, 21.64, 20.10 [01:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7526.000000 [01:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7548 [01:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:36:32] Load avg. on ortelius is WARNING: WARNING - load average: 16.44, 17.97, 19.93 [01:46:21] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 34787 MB (8% inode=99%): [01:48:32] Load avg. on ortelius is OK: OK - load average: 7.45, 9.89, 14.50 [01:49:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 21362 MB (3% inode=99%): [01:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:19:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6592.000000 [02:20:22] Load avg. on willow is CRITICAL: CRITICAL - load average: 105.71, 62.39, 33.45 [02:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:26:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8189.000000 [02:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8187 [02:31:32] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 93062 MB (9% inode=98%): [02:31:32] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 93062 MB (9% inode=98%): [02:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:50:22] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 21290 MB (3% inode=99%): [02:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:19:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9081.000000 [03:20:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 72.15, 65.43, 65.14 [03:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7813.000000 [03:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7801 [03:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:50:22] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 21199 MB (3% inode=99%): [03:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:19:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10394.000000 [04:19:17] Betacommand (or anyone interested): http://toolserver.org/~alexz/coord/ [04:21:21] Load avg. on willow is CRITICAL: CRITICAL - load average: 69.81, 67.57, 65.88 [04:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:26:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8881.000000 [04:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8863 [04:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:50:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 21092 MB (3% inode=99%): [04:53:32] Load avg. on ortelius is WARNING: WARNING - load average: 20.08, 17.72, 15.14 [04:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:59:21] Load avg. on willow is WARNING: WARNING - load average: 10.35, 9.09, 19.48 [05:16:21] Load avg. on willow is OK: OK - load average: 11.52, 12.08, 14.93 [05:19:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10948.000000 [05:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9811.000000 [05:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9821 [05:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:47:33] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 16945 MB (1% inode=96%): [05:50:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 21010 MB (3% inode=99%): [05:53:32] Load avg. on ortelius is WARNING: WARNING - load average: 13.43, 14.82, 15.66 [05:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:01:32] Load avg. on ortelius is OK: OK - load average: 13.06, 14.03, 14.95 [06:16:21] Load avg. on willow is WARNING: WARNING - load average: 17.26, 17.73, 14.15 [06:19:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12439.000000 [06:19:22] Load avg. on willow is OK: OK - load average: 9.70, 14.30, 13.43 [06:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:25:32] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 124226 MB (12% inode=99%): [06:26:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10074.000000 [06:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10062 [06:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:41:32] Load avg. on ortelius is WARNING: WARNING - load average: 14.10, 15.07, 14.07 [06:51:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 20831 MB (3% inode=99%): [06:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:19:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13719.000000 [07:20:32] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (365284 kB) free! [07:21:32] Free Memory on damiana is OK: OK - 10.9% (916792 kB) free. [07:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:26:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9256.000000 [07:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9230 [07:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:34:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:41:32] Load avg. on ortelius is WARNING: WARNING - load average: 17.60, 17.13, 18.40 [07:51:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 20661 MB (3% inode=99%): [07:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:19:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14304.000000 [08:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11035.000000 [08:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11056 [08:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:41:32] Load avg. on ortelius is WARNING: WARNING - load average: 20.32, 18.55, 17.04 [08:51:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 20449 MB (3% inode=99%): [08:54:31] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:19:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15180.000000 [09:22:32] Load avg. on ortelius is OK: OK - load average: 13.79, 14.11, 14.99 [09:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11072.000000 [09:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11084 [09:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:51:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 20237 MB (3% inode=99%): [09:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:09:07] Prolineserver * [Toolserver-l] Table 'commonswiki_p.globalimagelinks' doesn't exist. [10:19:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16950.000000 [10:22:32] Load avg. on ortelius is WARNING: WARNING - load average: 17.21, 16.52, 14.34 [10:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11398.000000 [10:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11400 [10:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:42:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 25.83, 23.63, 20.21 [10:51:31] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 20006 MB (3% inode=99%): [10:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:11:42] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1912.000000 [11:19:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19304.000000 [11:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11877.000000 [11:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11866 [11:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:40:31] Load avg. on daphne is WARNING: WARNING - load average: 16.92, 15.66, 13.78 [11:40:42] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1762.000000 [11:42:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 24.79, 25.82, 26.73 [11:51:31] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 19780 MB (3% inode=99%): [11:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:56:21] Load avg. on willow is CRITICAL: CRITICAL - load average: 69.79, 52.60, 28.05 [11:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:59:32] Free Memory on damiana is WARNING: WARNING - 5.7% (478800 kB) free! [12:05:32] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (401872 kB) free! [12:08:32] Free Memory on damiana is OK: OK - 12.8% (1072300 kB) free. [12:19:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21700.000000 [12:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13144.000000 [12:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13169 [12:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:34:41] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2104.000000 [12:40:31] Load avg. on daphne is WARNING: WARNING - load average: 16.53, 16.42, 16.48 [12:42:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 21.64, 22.84, 24.21 [12:51:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 19533 MB (3% inode=99%): [12:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:56:21] Load avg. on willow is CRITICAL: CRITICAL - load average: 56.79, 55.86, 57.34 [12:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:07:31] Load avg. on daphne is OK: OK - load average: 13.31, 13.84, 15.00 [13:13:32] Load avg. on ortelius is WARNING: WARNING - load average: 12.86, 17.65, 19.90 [13:19:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24620.000000 [13:22:32] Free Memory on damiana is WARNING: WARNING - 5.3% (446680 kB) free! [13:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:26:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14177.000000 [13:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14188 [13:27:32] Load avg. on ortelius is OK: OK - load average: 6.71, 10.73, 14.47 [13:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:34:42] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2078.000000 [13:37:32] Free Memory on damiana is OK: OK - 7.5% (629096 kB) free. [13:39:32] Load avg. on daphne is WARNING: WARNING - load average: 15.21, 14.94, 14.39 [13:46:22] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 31525 MB (7% inode=99%): [13:49:32] Load avg. on ortelius is WARNING: WARNING - load average: 16.76, 16.44, 15.23 [13:51:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 19215 MB (3% inode=99%): [13:54:31] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:56:21] Load avg. on willow is CRITICAL: CRITICAL - load average: 4.17, 5.85, 22.26 [13:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:59:21] Load avg. on willow is WARNING: WARNING - load average: 4.70, 5.33, 19.11 [14:05:21] Load avg. on willow is OK: OK - load average: 5.17, 5.88, 14.94 [14:19:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27421.000000 [14:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:26:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14705.000000 [14:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14728 [14:26:32] Load avg. on ortelius is OK: OK - load average: 12.02, 13.47, 14.94 [14:31:32] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 93067 MB (9% inode=98%): [14:31:32] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 93067 MB (9% inode=98%): [14:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:34:41] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3027.000000 [14:39:32] Load avg. on daphne is WARNING: WARNING - load average: 16.58, 16.73, 16.63 [14:42:32] Load avg. on ortelius is WARNING: WARNING - load average: 19.28, 16.20, 15.34 [14:49:09] @replag [14:49:10] liangent: s1-rr-a: 4h 18m 52s [+0.16 s/s]; s1-user: 4h 18m 52s [+0.16 s/s]; s2-user: 4h 42m 57s [+0.20 s/s]; s2-user-c: 56m 19s [+0.03 s/s]; s3-rr-a: 1m 9s [+0.00 s/s]; s3-user: 1m 9s [+0.00 s/s]; s5-user-c: 56m 19s [+0.03 s/s] [14:51:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 18912 MB (3% inode=99%): [14:54:31] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:57:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 25.87, 22.66, 20.09 [14:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:00:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3602.000000 [15:19:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30740.000000 [15:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:26:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15283.000000 [15:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15286 [15:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:39:32] Load avg. on daphne is WARNING: WARNING - load average: 16.91, 17.13, 17.25 [15:46:21] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24456 MB (5% inode=99%): [15:51:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 18634 MB (3% inode=99%): [15:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:57:31] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 23.75, 28.84, 30.79 [15:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:00:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5068.000000 [16:19:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34039.000000 [16:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16617.000000 [16:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16609 [16:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:37:32] Free Memory on damiana is WARNING: WARNING - 6.9% (575816 kB) free! [16:38:32] Free Memory on damiana is OK: OK - 8.0% (669148 kB) free. [16:39:32] Load avg. on daphne is WARNING: WARNING - load average: 16.78, 16.93, 16.95 [16:48:22] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 28415 MB (6% inode=99%): [16:51:31] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 18376 MB (2% inode=99%): [16:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:57:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 35.29, 35.67, 34.21 [16:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:00:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6383.000000 [17:19:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36439.000000 [17:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16483.000000 [17:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16477 [17:30:22] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24291 MB (5% inode=99%): [17:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:39:32] Load avg. on daphne is WARNING: WARNING - load average: 16.76, 16.89, 16.97 [17:48:32] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 16331 MB (1% inode=96%): [17:51:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 18106 MB (2% inode=99%): [17:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:57:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 41.87, 41.12, 40.65 [17:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:00:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9853.000000 [18:19:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38975.000000 [18:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:25:32] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 118674 MB (12% inode=99%): [18:25:32] Free Memory on damiana is WARNING: WARNING - 6.9% (579336 kB) free! [18:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16489.000000 [18:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16511 [18:29:32] Free Memory on damiana is OK: OK - 7.1% (591596 kB) free. [18:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:36:32] Free Memory on damiana is WARNING: WARNING - 6.2% (515440 kB) free! [18:39:32] Load avg. on daphne is WARNING: WARNING - load average: 16.82, 16.98, 17.01 [18:51:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 17822 MB (2% inode=99%): [18:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:57:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 39.79, 34.45, 34.79 [18:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:00:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11713.000000 [19:08:15] hello [19:08:24] hi [19:08:43] hello Dispenser [19:08:51] hi! [19:09:59] nosy: long time no see! are you really in tahiti? [19:10:11] @replag [19:10:16] jeremyb: always ;) [19:10:16] jeremyb: s1-rr-a: 4h 59m 59s [+0.16 s/s]; s1-user: 4h 59m 58s [+0.16 s/s]; s2-user: 7h 3m 49s [+0.54 s/s]; s2-user-c: 3h 12m 6s [+0.52 s/s]; s5-user-c: 3h 12m 7s [+0.52 s/s] [19:10:27] nosy: {{fact}} [19:10:47] @replag all [19:10:51] nosy: s1-rr-a: 4h 59m 49s [-0.28 s/s]; s1-user: 4h 59m 50s [-0.22 s/s]; s2-user: 7h 4m 5s [+0.44 s/s]; s2-user-c: 3h 12m 9s [+0.08 s/s]; s3-rr-a: 35s [-0.00 s/s]; s3-user: 35s [-0.00 s/s]; s4-rr-a: 0s [-0.00 s/s]; s4-user: 3s [-0.00 s/s] [19:10:52] nosy: s5-rr-a: 6s [-0.00 s/s]; s5-user: 6s [-0.00 s/s]; s5-user-c: 3h 12m 10s [+0.09 s/s]; s6-rr-a: 1s [-0.00 s/s]; s6-user: 1s [-0.00 s/s]; s7-rr-a: 0s [-0.00 s/s]; s7-user: 0s [-0.00 s/s] [19:19:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41589.000000 [19:21:47] apmon: online? [19:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:24:27] Dispenser: what is backlinkscount.py? [19:25:06] wonder why a french user is hitting it so hard without leaving any contact details? [19:25:41] Do [19:25:53] Dispenser: no - wonder why ortelius has so much load [19:26:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18382.000000 [19:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18407 [19:28:02] how are servers named? [19:28:11] I think he's building a backlinks count table for the millions of pages on frwiki. I killed it once requiring him to add a user agent and even tried sending an email to his box. Somehow he think PHP_agent is good user agent [19:28:58] ortelius & wolfsbane - webserver; willow & nightshade log in servers (all poisonous plants) [19:29:38] err, ortelius was actually an OSM name [19:31:48] nosy, should I block him, rewrite backlinkscount.py as an fcgi script, or something else? [19:32:32] Load avg. on daphne is OK: OK - load average: 1.67, 9.77, 14.13 [19:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:33:43] Dispenser: if you think it is reasonable to block him and it is easy for you just do it - i dont understand if it is reasonable what he is doing but why does he need so many threads? [19:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:35:07] DaB. * Re: [Toolserver-l] Table 'commonswiki_p.globalimagelinks' doesn't exist. [19:35:49] I wish I knew too. If it's what I suspect, it would be easier for us to just run the SQL for 8 hours and give him the results. [19:36:28] sad he doesnt answer on your question [19:37:07] Dispenser: you could give him a 403 that says who to contact... [19:37:35] Dispenser: ortelius seems still to be fine probably it can run until tomorrow or something [19:39:05] If he doesn't finish up by Monday, I'll block him [19:50:23] apmon: ping [19:50:48] Dispenser: ok lets do that as long as ortelius is still working [19:51:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45528 MB (7% inode=99%): [19:54:31] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:56:06] @replag [19:56:07] jeremyb: s1-rr-a: 5h 33m 9s [+0.88 s/s]; s1-user: 5h 33m 9s [+0.88 s/s]; s2-user: 7h 33m 49s [+0.72 s/s]; s2-user-c: 3h 25m 20s [+0.34 s/s]; s3-rr-a: 1m 39s [+0.05 s/s]; s3-user: 1m 39s [+0.05 s/s]; s5-user-c: 3h 25m 20s [+0.34 s/s] [19:57:29] nosy: I am around now [19:57:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 27.59, 29.36, 32.88 [19:57:37] hello apmon [19:57:47] Dispenser: thanks for the memo [19:58:20] do you have any idea if syncing kolossos db needs to be done again if i did it an hour ago? [19:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:59:48] nosy: I think upates only run once a day, so it should be fine [19:59:54] although I don't know for sure. [20:00:18] * apmon needs to switch computers. Will be back in a short moment [20:00:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12343.000000 [20:03:59] wb apmon [20:04:16] imported all dbs afais and changed the config [20:04:43] Nice. Have you restarted the db already? [20:04:45] so postgres restart with the new data could be soon [20:04:48] nope [20:04:55] Otherwise I am currently stoping the import process [20:05:31] unfortunately it is just a little behind (resulting in it processing bigger chunks), but it should complete in a couple of minute [20:11:22] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 35196 MB (8% inode=99%): [20:13:41] nosy: The diff-import is done now [20:14:08] So apart from dschwen's processes, there is nothing accessing the db anymore I think [20:14:49] apmon: do you think we should wait for dschwens process to finish? [20:15:20] Those are as far as I know live process from WMA. So they won't finish [20:15:45] but given that they continued even while the db was constantly crashing, they seem to connect just fine again [20:16:02] apmon: ok then i will restart the db [20:16:22] thanks [20:19:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44264.000000 [20:21:02] apmon: its running now [20:21:31] cheers thanks. I'll check the things I can then now and post to the others to make sure no issues are ocuring for them [20:21:55] I'll also announce to maps-l that things seem to be working and uptodate again now (once I have verified that) [20:22:24] apmon: :) [20:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:22:42] i will keep the data for some days - probably one or two weeks? [20:23:37] Sounds good, thanks [20:25:04] looking at the activity of the db, the processes accessing u_kolossos and dschwen's have successfully reconnected to the db after restart [20:26:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20122.000000 [20:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20115 [20:28:24] @replag [20:28:26] liangent: s1-rr-a: 5h 34m 49s [+0.05 s/s]; s1-user: 5h 34m 47s [+0.05 s/s]; s2-user: 7h 45m 29s [+0.36 s/s]; s2-user-c: 3h 3m 24s [-0.68 s/s]; s3-rr-a: 14s [-0.04 s/s]; s3-user: 14s [-0.04 s/s]; s5-user-c: 3h 3m 24s [-0.68 s/s]; s7-rr-a: 27s [+0.01 s/s] [20:28:27] liangent: s7-user: 28s [+0.01 s/s] [20:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:41:30] @replag [20:41:31] liangent: s1-rr-a: 5h 35m 25s [+0.05 s/s]; s1-user: 5h 35m 25s [+0.05 s/s]; s2-user: 7h 47m 8s [+0.13 s/s]; s2-user-c: 2h 47m 34s [-1.21 s/s]; s3-rr-a: 1m 5s [+0.06 s/s]; s3-user: 1m 5s [+0.06 s/s]; s5-user-c: 2h 47m 33s [-1.21 s/s] [20:51:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53127 MB (8% inode=99%): [20:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:57:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 47.68, 42.07, 37.68 [20:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:00:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10140.000000 [21:19:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46990.000000 [21:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21727.000000 [21:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21746 [21:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:51:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53009 MB (8% inode=99%): [21:54:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:57:31] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 37.48, 36.44, 35.46 [21:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:00:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9850.000000 [22:03:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106891 MB (10% inode=99%): [22:19:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49508.000000 [22:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21988.000000 [22:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22012 [22:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:51:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52818 MB (8% inode=99%): [22:55:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:57:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 43.58, 43.18, 43.20 [22:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:00:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5259.000000 [23:13:41] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3495.000000 [23:19:41] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51802.000000 [23:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:24:41] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1735.000000 [23:26:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24649.000000 [23:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24650 [23:26:32] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 112457 MB (11% inode=99%): [23:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:34:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:51:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52570 MB (8% inode=99%): [23:55:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:57:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 48.06, 47.21, 46.57 [23:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default