[00:00:21] 3(created) [MAGNUS-289] Possible script injection attack; Magnus' tools: CommonsHelper; Critical Bug <10https://jira.toolserver.org/browse/MAGNUS-289> (Magog the Ogre) [00:00:49] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:02:21] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1879.000000 [00:05:22] 3(commented) [MAGNUS-289] Possible script injection attack <10https://jira.toolserver.org/browse/MAGNUS-289> (Magog the Ogre) [00:11:21] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1800.000000 [00:18:08] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1939 [00:25:48] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1847 [00:39:18] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6541 [00:55:08] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [00:55:20] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [01:00:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 51636 MB (5% inode=99%): [01:00:38] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:00:38] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:01:50] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:18:08] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3445 [01:24:49] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1943.000000 [01:24:50] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1943.000000 [01:25:50] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3609 [01:39:30] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9601 [01:47:08] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3620 [01:52:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3628.000000 [01:52:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3630.000000 [01:55:08] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [01:56:19] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [02:00:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 51630 MB (5% inode=99%): [02:00:48] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:00:48] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:01:59] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:26:46] @replag [02:26:47] Joan: s1-pri: 1h 34m 25s [+0.02 s/s]; s1-sec: 1h 34m 26s [+0.02 s/s]; s2-pri: 11s [-0.00 s/s]; s3-rr: 3h 19m 12s [+0.04 s/s]; s3-user: 3h 19m 12s [+0.04 s/s]; s5-rr: 14s [-0.00 s/s]; s5-user: 14s [-0.00 s/s]; s6-rr: 1h 21m 19s [+0.01 s/s] [02:26:48] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6026 [02:26:49] Joan: s6-user: 1h 21m 19s [+0.01 s/s]; s7-rr: 1h 40m 33s [+0.01 s/s]; s7-user: 1h 40m 33s [+0.01 s/s] [02:40:30] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12551 [02:57:18] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [03:00:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 51632 MB (5% inode=99%): [03:00:48] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:01:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:02:08] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:26:50] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8146 [03:41:28] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13205 [03:44:17] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3459 [03:50:18] MySQL slave on z-dat-s6-a is OK: Uptime: 5115961 Threads: 10 Questions: 989312770 Slow queries: 344104 Opens: 10008699 Flush tables: 2 Open tables: 2860 Queries per second avg: 193.377 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1336 [03:53:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10890.000000 [03:53:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10893.000000 [03:55:09] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [03:57:18] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [04:00:39] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 51628 MB (5% inode=99%): [04:00:49] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:01:50] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:02:09] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:04:49] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3540 [04:07:49] MySQL slave on z-dat-s7-a is OK: Uptime: 2622535 Threads: 3 Questions: 630558595 Slow queries: 108659 Opens: 7568243 Flush tables: 1 Open tables: 3729 Queries per second avg: 240.438 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1040 [04:28:28] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2757 [04:30:14] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=6570&oldid=6551&rcid=8644 * 91.198.174.202 * (+2) (updated page) [04:30:28] MySQL slave on z-dat-s3-a is OK: Uptime: 3851460 Threads: 16 Questions: 4239082639 Slow queries: 337352 Opens: 53555356 Flush tables: 2 Open tables: 16384 Queries per second avg: 1100.643 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1505 [04:53:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14493.000000 [04:54:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14493.000000 [04:54:48] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.253418/1.75, alarm hl:np_load_avg=0.307129/2.00, alarm hl:mem_free=282.000000M/300M [04:55:48] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [04:56:08] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [04:57:18] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [05:00:45] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 51620 MB (5% inode=99%): [05:01:49] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:01:59] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:02:08] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:04:50] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.300781/1.75, alarm hl:np_load_avg=0.369629/2.00, alarm hl:mem_free=260.000000M/300M [05:36:21] 3(created) [MAGNUS-290] TreeViews page views problem; Magnus' tools; Bug <10https://jira.toolserver.org/browse/MAGNUS-290> (Lebron James) [05:53:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18092.000000 [05:53:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18094.000000 [05:55:08] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:19] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:49] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:56:08] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:56:08] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [05:58:18] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [06:01:39] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 51613 MB (5% inode=99%): [06:01:58] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:01:59] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:03:08] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:12:49] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.742676/1.75, alarm hl:np_load_avg=0.665039/2.00, alarm hl:mem_free=272.000000M/300M [06:13:48] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [07:05:14] [[Special:Log/newusers]] create 10 * Tinymatt16 * (New user account) [07:07:53] [[User talk:Tinymatt16]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6571&rcid=8646 * Tinymatt16 * (+352) (Created page with "With the [http://www.closedrhinoplasty.com closed rhinoplasty] technique, the entire procedure is performed from the inside of the nose without severing the columella, or delicat...") [07:09:21] [[User talk:Tinymatt16]] ! 10https://wiki.toolserver.org/w/index.php?diff=6572&oldid=6571&rcid=8647 * Tinymatt16 * (+2298) () [07:09:35] ... [07:09:55] [[Special:Log/move]] move 10 * Tinymatt16 * (moved [[02User talk:Tinymatt1610]] to [[Tips On Choosing A Great Closed Rhinoplasty Surgeon in LA]]) [07:33:10] @replag all [07:33:11] nosy: s1-pri: 6h 40m 50s [+1.00 s/s]; s1-sec: 6h 40m 50s [+1.00 s/s]; s1-sec-c: 1s [-0.00 s/s]; s2-pri: 28s [+0.00 s/s]; s2/s5-pri-c: 1s [-0.03 s/s]; s3-rr: 14s [-0.65 s/s]; s3-user: 14s [-0.65 s/s]; s4-rr: 1s [-0.00 s/s] [07:33:12] nosy: s4-user: 1s [-0.00 s/s]; s5-rr: 2s [-0.00 s/s]; s5-user: 2s [-0.00 s/s]; s6-rr: 3s [-0.27 s/s]; s6-user: 3s [-0.27 s/s]; s7-rr: 8s [-0.33 s/s]; s7-user: 8s [-0.33 s/s] [07:49:27] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [07:50:28] MySQL slave on z-dat-s7-a is OK: Uptime: 2635888 Threads: 4 Questions: 633873080 Slow queries: 109128 Opens: 7628506 Flush tables: 1 Open tables: 3924 Queries per second avg: 240.478 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [07:54:01] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25292.000000 [07:54:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25293.000000 [07:57:09] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [07:59:18] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [08:02:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 51609 MB (5% inode=99%): [08:03:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:03:00] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:03:08] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:10:18] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 35202 MB (3% inode=99%): [08:55:00] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28954.000000 [08:55:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28955.000000 [08:57:09] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [08:59:18] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [09:02:48] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 51605 MB (5% inode=99%): [09:03:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:03:01] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:03:10] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:55:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32558.000000 [09:55:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32557.000000 [09:57:19] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [09:59:19] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [10:02:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 51602 MB (5% inode=99%): [10:03:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:03:59] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:03:59] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:29:38] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.467773/1.00, alarm hl:np_load_long=0.724610/1.50, alarm hl:mem_free=24536.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.467773/1.10, alarm hl:np_load_long=0.724610/1.75, alarm hl:mem_free=24536.000000M/300M [10:32:38] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [10:46:22] 3(created) [TS-1270] Changed master database host for s1; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1270> (Marlen Caemmerer) [10:46:23] 3(resolved) [TS-1270] Changed master database host for s1 <10https://jira.toolserver.org/browse/TS-1270> (Marlen Caemmerer) [10:56:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36217.000000 [10:56:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36217.000000 [10:58:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 536543 [10:59:28] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 536205 [11:03:48] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44993 MB (4% inode=99%): [11:03:59] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:03:59] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:04:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:09:35] [[Special:Log/newusers]] create 10 * Cherishsantosh * (New user account) [11:30:28] [[Special:Log/newusers]] create 10 * Patrica * (New user account) [11:31:25] [[User talk:Patrica]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6575&rcid=8651 * Patrica * (+392) (Created page with "Americas Top Surgeons has been developed with one primary object in mind – to have quality Plastic Surgeons information available to the American public. Every year, demand for...") [11:36:14] [[User talk:Patrica]] ! 10https://wiki.toolserver.org/w/index.php?diff=6576&oldid=6575&rcid=8652 * Patrica * (+2514) () [11:36:15] [[Special:Log/delete]] delete 10 * Nosy * (deleted "[[02User talk:Patrica10]]": Vandalism: content was: "Americas Top Surgeons has been developed with one primary object in mind – to have quality Plastic Surgeons information available to the American public. Every ..." (and the only contributor was "[[Special:Contributions/Patrica|Patrica]]")) [11:36:20] [[Special:Log/newusers]] create 10 * Greenadamss * (New user account) [11:37:29] [[Special:Log/block]] block 10 * Nosy * (blocked [[02User:Patrica10]] with an expiry time of 3 days (account creation disabled): Spamming links to external sites) [11:44:38] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.446289/1.00, alarm hl:np_load_long=0.958008/1.50, alarm hl:mem_free=24108.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.446289/1.10, alarm hl:np_load_long=0.958008/1.75, alarm hl:mem_free=24108.000000M/300M [11:46:38] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [11:56:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39817.000000 [11:56:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39817.000000 [11:58:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 519450 [11:59:28] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 518005 [12:03:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 34417 MB (3% inode=99%): [12:03:59] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:03:59] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:05:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:12:59] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.385742/1.75, alarm hl:np_load_avg=0.363281/2.00, alarm hl:mem_free=248.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.385742/1.50, alarm hl:np_load_long=0.358399/1.75, alarm hl:mem_free=248.000000M/250M [12:14:00] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [12:52:13] [[Special:Log/newusers]] create 10 * General * (New user account) [12:56:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43417.000000 [12:56:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43417.000000 [12:58:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 498849 [12:59:29] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 501148 [13:03:48] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 34306 MB (3% inode=99%): [13:03:59] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:03:59] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:05:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:56:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47018.000000 [13:56:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47017.000000 [13:58:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 483659 [13:59:06] @replag [13:59:07] DaBPunkt: s1-pri: 13h 6m 45s [+1.00 s/s]; s1-sec: 13h 6m 45s [+1.00 s/s]; s2-pri: 6m 28s [+0.02 s/s]; s2/s5-pri-c: 2m 35s [+0.01 s/s]; s3-rr: 17s [+0.00 s/s]; s3-user: 17s [+0.00 s/s]; s7-rr: 10s [+0.00 s/s]; s7-user: 10s [+0.00 s/s] [14:00:29] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 488722 [14:03:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 35113 MB (3% inode=99%): [14:03:59] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:03:59] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:05:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:44:03] DaB. * [Toolserver-announce] Data of s1 (enwiki) probably corrupted [14:56:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50617.000000 [14:56:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50617.000000 [14:58:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 470554 [15:00:28] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 475308 [15:03:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 35905 MB (3% inode=99%): [15:04:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:04:09] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:05:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:10:06] DaBPunkt, why would the data be corrupt? [15:12:09] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.386719/1.75, alarm hl:np_load_avg=0.340332/2.00, alarm hl:mem_free=260.000000M/300M [15:12:23] Platonides: UPDATE table SET foo=foo+1 [15:13:10] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [15:13:15] ? [15:14:49] what did you do? start change replication to be from the wrong master (ie. a slave)? [15:18:03] Platonides: the wmf-guys had put the old master back (as a slave) too fast, before we made the switch. So we replicated from the old master since there. I planed to fix this today, but last night d32 was re-setuped, so a switch is not longer possible (my mistake) [15:18:17] since there → since then [15:18:53] but wouldn't the data continue from the same position? [15:19:16] if the position of the binlogs continued, it shouldn't matter if you were replicating from master or a slave [15:21:57] Platonides: in theorie: yes. But this is not longer possible, because of a mistake we re-started replication from the old-master-position [15:23:41] so you replicated a few statements twice? that would lead to problems, yes :( [15:57:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54277.000000 [15:57:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54278.000000 [15:58:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 451220 [16:00:02] DaB. * Re: [Toolserver-l] Extraordinary maintenance this week [16:00:28] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 454158 [16:04:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:04:11] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:04:48] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 36700 MB (3% inode=99%): [16:06:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:57:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57879.000000 [16:57:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57878.000000 [16:58:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 437485 [17:00:28] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 439750 [17:04:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:04:11] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:04:48] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 36525 MB (3% inode=99%): [17:06:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:12:38] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.635742/1.00, alarm hl:np_load_long=0.801758/1.50, alarm hl:mem_free=23876.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.635742/1.10, alarm hl:np_load_long=0.801758/1.75, alarm hl:mem_free=23876.000000M/300M [17:14:38] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [17:36:58] do the osm map guys have a separate channel? [17:49:47] any OSM guys in here? [17:49:58] avar [17:57:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61482.000000 [17:57:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61481.000000 [17:58:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 424634 [18:00:28] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 427288 [18:04:08] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:04:18] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:04:48] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 37294 MB (3% inode=99%): [18:06:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:07:00] !replag [18:07:17] @replag [18:07:18] BarkingFish: s1-pri: 17h 14m 56s [+1.00 s/s]; s1-sec: 17h 14m 56s [+1.00 s/s]; s3-rr: 34s [+0.00 s/s]; s3-user: 34s [+0.00 s/s] [18:07:24] holy cow [18:07:29] I thought that was a mistake :P [18:27:29] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:18] SMTP on hyacinth is OK: SMTP OK - 0.028 sec. response time [18:53:03] Russell Blau * Re: [Toolserver-l] Data of s1 (enwiki) probably corrupted [18:55:39] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:55:49] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:49] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:49] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:08] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [18:56:38] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [18:56:39] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [18:56:39] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [18:57:00] @replag [18:57:01] Jeff_G: s1-pri: 18h 4m 39s [+1.00 s/s]; s1-sec: 18h 4m 39s [+1.00 s/s]; s2-pri: 15s [-0.02 s/s]; s3-rr: 1m 24s [+0.02 s/s]; s3-user: 1m 24s [+0.02 s/s]; s6-rr: 50s [+0.00 s/s]; s6-user: 50s [+0.00 s/s]; s7-rr: 1m 1s [+0.00 s/s] [18:57:02] Jeff_G: s7-user: 1m 1s [+0.00 s/s] [18:57:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65087.000000 [18:57:18] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65087.000000 [18:58:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 413234 [19:00:30] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 413933 [19:04:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:04:29] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:04:48] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 37119 MB (3% inode=99%): [19:06:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:13:38] PING on amaranth is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [19:13:38] PING on web.amaranth is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [19:14:29] PING on amaranth is OK: PING OK - Packet loss = 0%, RTA = 110.00 ms [19:14:29] PING on web.amaranth is OK: PING OK - Packet loss = 0%, RTA = 110.00 ms [19:57:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68689.000000 [19:57:17] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68688.000000 [19:58:29] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 395949 [19:58:49] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 16.138 second response time [19:59:49] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 12.864 second response time [20:00:30] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 395894 [20:04:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:04:29] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:04:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 37364 MB (3% inode=99%): [20:06:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:10:39] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 32654 MB (3% inode=99%): [20:43:20] 3(resolved) [DBQ-174] Emijrp/List of Wikipedians by number of edits <10https://jira.toolserver.org/browse/DBQ-174> (madman) [20:43:22] 3(assigned) [DBQ-174] Emijrp/List of Wikipedians by number of edits <10https://jira.toolserver.org/browse/DBQ-174> (madman) [20:57:18] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72289.000000 [20:57:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72290.000000 [20:58:29] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 371751 [21:01:30] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 369001 [21:04:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:04:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 38507 MB (3% inode=99%): [21:05:29] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:07:29] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.319336/1.75, alarm hl:np_load_avg=0.310059/2.00, alarm hl:mem_free=294.000000M/300M [21:07:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:10:29] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [21:32:20] 3(created) [DBQ-175] checking cellistbot's it.wiki edits; Database Queries; Task <10https://jira.toolserver.org/browse/DBQ-175> (Nickanc) [21:40:39] SMTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:41:28] SMTP on amaranth is OK: SMTP OK - 0.455 sec. response time [21:44:39] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 513 bytes in 5.104 second response time [21:45:28] jira.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 513 bytes in 0.514 second response time [21:57:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75888.000000 [21:57:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75890.000000 [21:58:29] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 358777 [21:59:21] 3(commented) [DBQ-175] checking cellistbot's it.wiki edits <10https://jira.toolserver.org/browse/DBQ-175> (Hoo man) [22:01:29] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 355077 [22:04:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:05:48] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 38345 MB (3% inode=99%): [22:06:28] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:07:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:15:32] @replag [22:15:32] BarkingFish: s1-pri: 21h 23m 10s [+1.00 s/s]; s1-sec: 21h 23m 10s [+1.00 s/s]; s2-pri: 21s [+0.00 s/s]; s3-rr: 1m 32s [+0.00 s/s]; s3-user: 1m 32s [+0.00 s/s] [22:15:50] Anyone have any idea what the heck is going on with the replag and why it's so high? :) [22:25:28] BarkingFish: maybe the mailinglist? [22:25:54] No idea [22:27:25] I only spotted it again through the acc work, one of our tools keeps screaming "Replag very high - changes newer than 21h may not be shown!" etc... [22:27:40] I'm becoming immune to the messages, DaBPunkt :P [22:28:03] I guess I should go to bed, before I fall asleep here. [22:29:33] gn8 BarkingFish [22:29:55] n8 [22:57:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79489.000000 [22:57:18] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79489.000000 [22:58:29] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [23:02:29] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.388184/1.75, alarm hl:np_load_avg=0.337891/2.00, alarm hl:mem_free=295.000000M/300M [23:02:29] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [23:03:28] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [23:04:19] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:05:48] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 38277 MB (3% inode=99%): [23:06:28] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:07:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:57:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83089.000000 [23:57:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83089.000000 [23:59:12] nacht ts [23:59:29] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds)