[00:06:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104661 MB (17% inode=99%): [00:11:21] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3649.000000 [00:17:22] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [00:30:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:33:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:34:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:53:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:56:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:02:02] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24430 MB (5% inode=99%): [01:06:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104659 MB (17% inode=99%): [01:11:22] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7249.000000 [01:17:22] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [01:30:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:33:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:34:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:39:02] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39782 MB (9% inode=99%): [01:53:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:06:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104657 MB (17% inode=99%): [02:11:22] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10848.000000 [02:17:21] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [02:30:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:33:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:34:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:53:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:06:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104651 MB (17% inode=99%): [03:11:22] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14449.000000 [03:17:22] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [03:30:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:32:02] /sql on thyme is WARNING: DISK WARNING - free space: /sql 120674 MB (12% inode=99%): [03:33:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:34:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:53:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:06:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104649 MB (17% inode=99%): [04:11:22] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18049.000000 [04:17:22] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [04:30:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:33:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:34:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:39:21] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:44:02] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:53:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:06:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104646 MB (17% inode=99%): [05:11:22] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21648.000000 [05:17:21] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [05:30:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:33:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:34:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:47:12] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 30211 MB (2% inode=97%): [05:48:31] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:51:02] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36284 MB (8% inode=99%): [05:53:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:06:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104644 MB (17% inode=99%): [06:11:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25257.000000 [06:17:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [06:25:22] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 131834 MB (13% inode=99%): [06:30:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:33:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:34:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:53:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:06:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104638 MB (17% inode=99%): [07:11:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28858.000000 [07:17:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [07:30:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:33:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:34:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:53:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:06:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104628 MB (17% inode=99%): [08:09:59] @replag [08:09:59] liangent: s2-user: 7h 46m 12s [+0.28 s/s]; s2-user-c: 6m 25s [+0.00 s/s]; s3-rr-a: 19s [-0.11 s/s]; s3-user: 19s [-0.11 s/s]; s4-user: 8h 59m 35s [+0.07 s/s]; s5-user-c: 6m 25s [+0.00 s/s]; s7-rr-a: 10s [-0.00 s/s]; s7-user: 10s [-0.00 s/s] [08:11:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32457.000000 [08:17:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [08:30:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:33:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:34:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:53:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:06:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104621 MB (17% inode=99%): [09:11:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36057.000000 [09:17:31] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [09:24:02] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 98836 MB (10% inode=98%): [09:24:02] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 98836 MB (10% inode=98%): [09:30:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:33:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:34:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:53:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:06:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104613 MB (17% inode=99%): [10:11:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39658.000000 [10:17:31] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [10:28:37] @replag [10:28:37] liangent: s2-user: 10h 4m 50s [+1.00 s/s]; s3-rr-a: 3m 9s [+0.02 s/s]; s3-user: 3m 9s [+0.02 s/s]; s4-user: 11h 18m 13s [+1.00 s/s] [10:30:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:32:02] @replag [10:32:03] liangent: s2-user: 10h 8m 15s [+1.00 s/s]; s3-rr-a: 1m 11s [-0.57 s/s]; s3-user: 1m 11s [-0.57 s/s]; s4-user: 11h 21m 38s [+1.00 s/s] [10:33:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:34:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:53:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:53:32] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:55:22] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 98461 MB (10% inode=98%): [10:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:06:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104606 MB (17% inode=99%): [11:11:42] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43267.000000 [11:17:41] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [11:30:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:33:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:34:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:53:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:06:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104601 MB (17% inode=99%): [12:11:42] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46867.000000 [12:17:42] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [12:25:16] @replag [12:25:17] jeremyb: s1-rr-a: 1m 12s [-0.02 s/s]; s1-user: 1m 12s [-0.02 s/s]; s2-user: 12h 1m 29s [+1.00 s/s]; s3-rr-a: 8m 17s [+0.06 s/s]; s3-user: 8m 17s [+0.06 s/s]; s4-user: 13h 14m 52s [+1.00 s/s]; s6-rr-a: 14s [-0.00 s/s]; s6-user: 14s [-0.00 s/s] [12:25:18] jeremyb: s7-rr-a: 5m 18s [+0.02 s/s]; s7-user: 5m 18s [+0.02 s/s] [12:25:21] huh [12:25:25] 13h [12:30:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:31:02] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24365 MB (5% inode=99%): [12:33:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:34:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:53:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:53:32] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:53:42] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:54:02] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 100013 MB (10% inode=98%): [12:54:12] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 100014 MB (10% inode=98%): [12:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:06:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104589 MB (17% inode=99%): [13:07:02] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36917 MB (9% inode=99%): [13:11:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50468.000000 [13:17:42] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [13:27:42] Hi all [13:28:15] there is any toolserver's Administrators here? :-( [13:30:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:33:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:34:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:48:09] hello all [13:53:23] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:53:31] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:54:01] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36605 MB (8% inode=99%): [13:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:06:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104577 MB (17% inode=99%): [14:08:01] @replag [14:08:02] liangent: s1-rr-a: 1m 43s [+0.01 s/s]; s1-user: 1m 43s [+0.01 s/s]; s2-user: 13h 44m 14s [+1.00 s/s]; s3-rr-a: 23s [-0.08 s/s]; s3-user: 23s [-0.08 s/s]; s4-user: 14h 57m 37s [+1.00 s/s] [14:11:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54067.000000 [14:17:42] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [14:30:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:33:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:34:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:53:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:56:50] DaBPunkt: hallo DaB, hast Du schon das PostGIS upgrade ticket gesehen? Ist das viel Aufwand? Was meinst Du, ist es machbar? [14:57:15] dschwen: kann ich noch nicht abschätzen. Ich ahbe noch nie postGis geupdatet [15:04:26] DaBPunkt: replag is gone on thyme, could we make it an rr server now [15:06:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104570 MB (17% inode=99%): [15:11:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57667.000000 [15:17:41] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [15:17:55] switched sql-s1-rr to thyme as suggested [15:23:03] @replag [15:23:03] liangent: s1-rr-a: 5m 38s [+0.05 s/s]; s1-user: 5m 38s [+0.05 s/s]; s2-user: 14h 59m 16s [+1.00 s/s]; s4-user: 16h 12m 39s [+1.00 s/s] [15:23:22] DaBPunkt: s2 stopped replication [15:23:26] and s4 [15:27:05] liangent: I will look [15:30:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:30:57] @replag [15:30:58] DaBPunkt: s2-user: 15h 7m 11s [+1.00 s/s]; s3-rr-a: 43s [+0.00 s/s]; s3-user: 43s [+0.00 s/s]; s4-user: 16h 17m 59s [+0.67 s/s] [15:31:10] DaBPunkt: this happened several days ago and it restored automatically one or two days later [15:32:01] /sql on thyme is WARNING: DISK WARNING - free space: /sql 146680 MB (15% inode=99%): [15:33:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:34:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:36:53] Im getting "DBD::mysql::st execute failed: Lost connection to MySQL server during query" in one of my scripts [15:36:57] why? [15:37:41] simple DELETE query with SLOW_OK [15:38:32] on enwiki-p.userdb.toolserver.org [15:38:45] DaBPunkt: any idea? [15:42:36] hai [15:46:04] @replag [15:46:05] liangent: s2-user: 15h 22m 17s [+1.00 s/s]; s3-rr-a: 15s [-0.03 s/s]; s3-user: 15s [-0.03 s/s]; s4-user: 15h 7m 55s [-4.64 s/s] [15:46:35] liangent: I fiexed the porblem, the replag will increase [15:47:47] dschwen: the killer-log shows nothing AFAIS [15:47:54] odd [15:47:58] where is the script? [15:48:09] DaBPunkt: at a decreasing rate towards 0 then negative? [15:48:16] ~dschwen/wma/maintenance/build*.pl [15:48:56] launched like this: [15:48:58] qsub -l h_rt=6:30:00 -l virtual_free=300M -l arch=\* -l sql-s1-user=1 $HOME/wma/maintenance/update_lang.sh en [15:49:05] update_lang.sh is a wrapper [15:53:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:55:55] dschwen: can't see a problem. Did that happen more than 1 time? [15:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:59:46] yes [15:59:57] let me try again, I'll force another server [16:01:09] running on yarrow now [16:01:24] it'll take a few minutes to get to the part that fails [16:06:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104562 MB (17% inode=99%): [16:11:29] @replag [16:11:30] liangent: s1-rr-a: 3m 36s [-0.04 s/s]; s1-user: 3m 36s [-0.04 s/s]; s2-user: 15h 47m 42s [+1.00 s/s]; s3-rr-a: 23s [+0.01 s/s]; s3-user: 23s [+0.01 s/s]; s4-user: 9h 30m 13s [-13.28 s/s] [16:11:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34209.000000 [16:17:42] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30805 [16:29:12] DaBPunkt do you have a few moments? [16:30:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:33:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:34:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:53:23] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:56:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:03:59] DaBPunkt: the script ran, produced no error in the qsub logfiles, but also wrote no data to the db [17:04:07] it looks like it was interrupted [17:04:18] do you see anything in the logs? [17:06:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104555 MB (17% inode=99%): [17:06:42] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3475 [17:06:42] s4 replag on z-dat-s4-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3474.000000 [17:09:42] MySQL slave on z-dat-s4-a is OK: Uptime: 7758889 Threads: 7 Questions: 503002545 Slow queries: 97515 Opens: 179688 Flush tables: 1 Open tables: 1138 Queries per second avg: 64.829 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1722 [17:09:42] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 1722.000000 [17:11:01] /sql on thyme is OK: DISK OK - free space: /sql 218986 MB (22% inode=99%):