[00:00:54] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11734 [00:02:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:06:54] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1933 [00:08:04] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.241211/1.75, alarm hl:np_load_avg=0.349609/2.00, alarm hl:mem_free=232.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.241211/1.50, alarm hl:np_load_long=0.349121/1.75, alarm hl:mem_free=232.000000M/250M [00:11:54] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13632 [00:12:54] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45377 MB (4% inode=99%): [00:14:04] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:16:05] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:16:05] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:42:54] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3633 [00:56:21] 3(updated) [UTRS-49] Reservation is until assignee navigates away from ticket, not for a specified time <10https://jira.toolserver.org/browse/UTRS-49> (Hersfold) [00:56:23] 3(commented) [UTRS-49] Reservation is until assignee navigates away from ticket, not for a specified time <10https://jira.toolserver.org/browse/UTRS-49> (Hersfold) [01:01:01] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14399 [01:02:25] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:02:26] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=0.906250/1.75, alarm hl:np_load_avg=0.809082/2.00, alarm hl:mem_free=192.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=0.906250/1.50, alarm hl:np_load_long=0.698242/1.75, alarm hl:mem_free=192.000000M/250M [01:04:35] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [01:12:22] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17191 [01:13:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45231 MB (4% inode=99%): [01:13:34] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.488770/1.75, alarm hl:np_load_avg=0.458496/2.00, alarm hl:mem_free=291.000000M/300M [01:16:35] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:16:35] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:16:35] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:40:41] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:41:33] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.007 second response time [01:43:22] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4637 [01:48:24] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [01:51:22] MySQL on z-dat-s7-a is OK: Uptime: 335 Threads: 2 Questions: 1034 Slow queries: 0 Opens: 197 Flush tables: 1 Open tables: 189 Queries per second avg: 3.86 [01:53:21] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3340 [01:55:22] MySQL slave on z-dat-s7-a is OK: Uptime: 575 Threads: 9 Questions: 92610 Slow queries: 14 Opens: 443 Flush tables: 1 Open tables: 427 Queries per second avg: 161.60 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1698 [02:01:21] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14508 [02:02:43] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:12:41] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.586426/1.75, alarm hl:np_load_avg=0.542969/2.00, alarm hl:mem_free=267.000000M/300M [02:13:21] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20614 [02:13:41] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [02:14:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45114 MB (4% inode=99%): [02:16:34] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:16:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:21:32] toolserver.org HTTP on wolfsbane is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 3.402 second response time [02:22:32] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.011 second response time [02:30:32] toolserver.org HTTP on wolfsbane is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.136 second response time [02:52:23] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1937 [02:57:23] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [02:59:21] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [03:00:24] MySQL on z-dat-s7-a is OK: Uptime: 210 Threads: 2 Questions: 1117 Slow queries: 0 Opens: 215 Flush tables: 1 Open tables: 206 Queries per second avg: 5.319 [03:00:24] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2312 [03:01:21] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17217 [03:03:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:13:21] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23696 [03:15:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45026 MB (4% inode=99%): [03:16:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:17:31] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:42:42] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.465820/1.75, alarm hl:np_load_avg=0.479492/2.00, alarm hl:mem_free=245.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.465820/1.50, alarm hl:np_load_long=0.477539/1.75, alarm hl:mem_free=245.000000M/250M [03:47:41] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [03:48:21] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2976 [03:50:21] MySQL slave on z-dat-s6-a is OK: Uptime: 25232 Threads: 10 Questions: 1425900 Slow queries: 3349 Opens: 39923 Flush tables: 1 Open tables: 521 Queries per second avg: 56.511 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1642 [03:56:42] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.595703/1.75, alarm hl:np_load_avg=0.538086/2.00, alarm hl:mem_free=246.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.595703/1.50, alarm hl:np_load_long=0.515137/1.75, alarm hl:mem_free=246.000000M/250M [04:01:41] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:02:32] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.011 second response time [04:03:51] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:13:21] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12908 [04:15:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44905 MB (4% inode=99%): [04:17:40] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:41] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:18:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:27:22] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3324 [04:29:21] MySQL slave on z-dat-s3-a is OK: Uptime: 27572 Threads: 20 Questions: 11625034 Slow queries: 9847 Opens: 82976 Flush tables: 1 Open tables: 16384 Queries per second avg: 421.624 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1539 [04:32:06] [[Talk:Main Page]] ! 10https://wiki.toolserver.org/w/index.php?diff=6622&oldid=6550&rcid=8727 * 101.160.176.5 * (+3) () [04:33:14] [[Talk:Main Page]] 10https://wiki.toolserver.org/w/index.php?diff=6623&oldid=6622&rcid=8728 * MZMcBride * (-243) (rv) [04:33:46] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:58.167.12.3910]] with an expiry time of infinite (account creation disabled): inappropriate behavior) [04:33:56] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:101.160.176.510]] with an expiry time of infinite (account creation disabled): inappropriate behavior) [04:40:33] toolserver.org HTTP on wolfsbane is CRITICAL: Connection refused [05:03:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:10:38] zzz =_= [05:11:42] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=0.865723/1.75, alarm hl:np_load_avg=0.836914/2.00, alarm hl:mem_free=253.000000M/300M [05:12:42] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [05:15:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44792 MB (4% inode=99%): [05:15:42] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.487305/1.75, alarm hl:np_load_avg=0.829590/2.00, alarm hl:mem_free=149.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.487305/1.50, alarm hl:np_load_long=0.608887/1.75, alarm hl:mem_free=149.000000M/250M [05:17:43] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [05:17:51] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:18:41] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:20:52] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.640625/1.75, alarm hl:np_load_avg=0.625000/2.00, alarm hl:mem_free=209.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.640625/1.50, alarm hl:np_load_long=0.573242/1.75, alarm hl:mem_free=209.000000M/250M [05:40:52] toolserver.org HTTP on wolfsbane is CRITICAL: Connection refused [05:43:26] [[Special:Log/newusers]] create 10 * Sellcars * (New user account) [05:44:09] Sellcars [05:45:32] sell cars . {watches contribs} [05:46:24] [[Are the Destroyed junk car LA easy to sale?]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6624&rcid=8732 * Sellcars * (+5588) (Created page with "How to sell your '''[http://www.sell-junkcars.com/cash-for-los-angeles-junk-cars/ junk cars LA]''' for some cash online on the internet that has been destroyed or turned into a p...") [05:47:47] Yep, called it! Sysops want to whack? [05:49:35] XD [05:51:21] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1928 [06:04:03] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:07:50] [[Are the Destroyed junk car LA easy to sale?]] ! 10https://wiki.toolserver.org/w/index.php?diff=6625&oldid=6624&rcid=8733 * Matthewrbowker * (+18) (Request deletion) [06:08:55] If someone wants to whack that user, it would be good :P [06:15:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44514 MB (4% inode=99%): [06:15:33] matthewrbowker for all we know thats a music group :p [06:15:59] or a new anime! :O [06:16:21] ToAruShiroiNeko: Of course, and the links to advertising sites make the background symphonies ;) [06:16:50] In all seriousness, it's advertising. I've read the article, and it has a lot of ELs to car selling sites. Not Toolserver Wiki material by any means. [06:18:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:18:53] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:22:23] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:Sellcars10]] with an expiry time of infinite (account creation disabled): inappropriate behavior) [06:22:35] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02Are the Destroyed junk car LA easy to sale?10]]": inappropriate content) [06:23:30] :) [06:23:52] matthewrbowker: Toolserver users are typically admins on the wiki. [06:24:09] Joan: Really? [06:24:13] Yes. [06:24:39] Not me, my groups are Autoconfirmed users, Users, confluence-users, jira-users [06:25:47] [[User talk:Dab]] 10https://wiki.toolserver.org/w/index.php?diff=6626&oldid=6530&rcid=8736 * MZMcBride * (+194) (/* Matthewrbowker */ new section) [06:25:52] Let's see if that does it. [06:26:16] Joan: Thanks :) [06:29:38] matthewrbowker: Your tool has an XSS vulnerability, it looks ilke... [06:29:41] like [06:29:44] http://toolserver.org/~matthewrbowker/serverinfo/answer.php?lang=en&family=wikibookss%3Ca%20href=%22http://google.com%22%3Ehello%3C/a%3E [06:30:21] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3655 [06:31:32] Joan: Thanks for the catch, I didn't see that. Whoops. [06:31:39] I'll look into fixing it right now. [06:40:51] toolserver.org HTTP on wolfsbane is CRITICAL: Connection refused [07:01:20] 3(created) [MATTHEWRBOWKER-6] Fix code vulnerabilities; Matthewrbowker's Tools: Article Information, Dabase Information, Project Information; Blocker Bug <10https://jira.toolserver.org/browse/MATTHEWRBOWKER-6> (Matthew Bowker) [07:04:44] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:09:25] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.368164/1.75, alarm hl:np_load_avg=0.423828/2.00, alarm hl:mem_free=279.000000M/300M [07:10:25] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [08:01:31] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.034 second response time [08:01:31] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45402 MB (4% inode=99%): [08:01:39] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:01:39] s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2161.000000 [08:03:00] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1558.000000 [08:03:09] MySQL slave on z-dat-s7-a is OK: Uptime: 18376 Threads: 12 Questions: 842282 Slow queries: 1295 Opens: 3521 Flush tables: 1 Open tables: 2434 Queries per second avg: 45.835 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1682 [08:03:49] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1842.000000 [08:03:50] s5 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1685.000000 [08:03:58] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2661 [08:03:58] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2139.000000 [08:03:58] MySQL slave on thyme is OK: Uptime: 2604816 Threads: 8 Questions: 901498569 Slow queries: 452809 Opens: 132089 Flush tables: 1 Open tables: 2821 Queries per second avg: 346.89 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1570 [08:04:19] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1911 [08:04:40] s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1757.000000 [08:04:50] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1587.000000 [08:04:51] MySQL slave on daphne is OK: Uptime: 6080841 Threads: 37 Questions: 5103424550 Slow queries: 760687 Opens: 33042257 Flush tables: 3 Open tables: 15973 Queries per second avg: 839.262 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1749 [08:05:00] Cluster on turnera.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [08:05:41] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:05:50] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:06:20] Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [08:06:59] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1635.000000 [08:07:09] MySQL slave on rosemary is OK: Uptime: 7653683 Threads: 53 Questions: 2307510798 Slow queries: 965179 Opens: 11519 Flush tables: 1 Open tables: 914 Queries per second avg: 301.490 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1618 [08:07:30] toolserver.org HTTP on wolfsbane is CRITICAL: Connection refused [08:17:00] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, [08:17:10] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [08:19:09] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 21039 MB (2% inode=99%): [08:22:19] MySQL slave on z-dat-s6-a is OK: Uptime: 41551 Threads: 23 Questions: 2202105 Slow queries: 5614 Opens: 47762 Flush tables: 1 Open tables: 1468 Queries per second avg: 52.997 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1674 [08:26:00] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3650 [08:41:39] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [08:41:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:42:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:42:09] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [08:42:20] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [08:42:29] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [08:42:30] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [08:43:19] MySQL on z-dat-s7-a is OK: Uptime: 231 Threads: 2 Questions: 1060 Slow queries: 0 Opens: 207 Flush tables: 1 Open tables: 198 Queries per second avg: 4.588 [08:43:29] MySQL slave on z-dat-s7-a is OK: Uptime: 241 Threads: 1 Questions: 2023 Slow queries: 1 Opens: 260 Flush tables: 1 Open tables: 251 Queries per second avg: 8.394 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 448 [08:44:30] MySQL on z-dat-s6-a is OK: Uptime: 298 Threads: 2 Questions: 120 Slow queries: 0 Opens: 22 Flush tables: 1 Open tables: 15 Queries per second avg: 0.402 [08:44:40] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1982 [08:46:40] MySQL slave on z-dat-s6-a is OK: Uptime: 429 Threads: 4 Questions: 26512 Slow queries: 5 Opens: 159 Flush tables: 1 Open tables: 150 Queries per second avg: 61.799 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1732 [08:50:09] MySQL on z-dat-s3-a is OK: Uptime: 641 Threads: 9 Questions: 2767 Slow queries: 2 Opens: 291 Flush tables: 1 Open tables: 267 Queries per second avg: 4.316 [08:59:58] [[Toolserver Intuition]] ! 10https://wiki.toolserver.org/w/index.php?diff=6627&oldid=6186&rcid=8737 * 203.189.188.105 * (-16937) (Blanked the page) [09:01:30] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45320 MB (4% inode=99%): [09:01:40] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:05:40] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:05:50] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:07:30] toolserver.org HTTP on wolfsbane is CRITICAL: Connection refused [09:09:19] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3467 [09:14:20] MySQL slave on z-dat-s3-a is OK: Uptime: 2092 Threads: 17 Questions: 974343 Slow queries: 87 Opens: 13581 Flush tables: 1 Open tables: 10417 Queries per second avg: 465.747 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1699 [09:19:00] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.575684/1.75, alarm hl:np_load_avg=0.532715/2.00, alarm hl:mem_free=202.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.575684/1.50, alarm hl:np_load_long=0.509766/1.75, alarm hl:mem_free=202.000000M/250M [09:35:51] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [10:01:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45186 MB (4% inode=99%): [10:01:39] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:05:20] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1949 [10:05:40] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:05:50] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:07:30] toolserver.org HTTP on wolfsbane is CRITICAL: Connection refused [10:10:30] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1932 [10:17:00] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.273438/1.75, alarm hl:np_load_avg=0.304199/2.00, alarm hl:mem_free=286.000000M/300M [10:17:00] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:17:51] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [10:20:40] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1910 [10:26:51] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:33:19] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3621 [10:41:02] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.421875/1.75, alarm hl:np_load_avg=0.308105/2.00, alarm hl:mem_free=273.000000M/300M [10:44:02] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [10:45:30] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3637 [10:51:29] huh, mysql is ssllooww [11:01:40] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:02:30] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45043 MB (4% inode=99%): [11:02:39] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3616 [11:05:41] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:06:01] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:08:28] toolserver.org HTTP on wolfsbane is CRITICAL: Connection refused [11:13:01] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.399902/1.75, alarm hl:np_load_avg=0.364258/2.00, alarm hl:mem_free=286.000000M/300M [11:14:01] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [11:24:42] HTTP proxy on ha-proxy.esi is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:25:02] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.138672/1.00, alarm hl:np_load_long=0.586914/1.50, alarm hl:mem_free=20683.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.138672/1.10, alarm hl:np_load_long=0.586914/1.75, alarm hl:mem_free=20683.000000M/300M [11:26:29] HTTP proxy on ha-proxy.esi is OK: HTTP OK: HTTP/1.0 302 Moved Temporarily - 759 bytes in 0.038 second response time [11:33:19] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7169 [11:34:01] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [11:46:29] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6023 [12:01:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:02:41] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44902 MB (4% inode=99%): [12:02:41] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5056 [12:03:00] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.322754/1.75, alarm hl:np_load_avg=0.371094/2.00, alarm hl:mem_free=299.000000M/300M [12:05:01] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [12:05:41] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:07:01] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:08:30] toolserver.org HTTP on wolfsbane is CRITICAL: Connection refused [12:13:01] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.340332/1.75, alarm hl:np_load_avg=0.323242/2.00, alarm hl:mem_free=257.000000M/300M [12:33:21] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10650 [12:33:40] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3533 [12:42:33] [[Special:Log/newusers]] create 10 * Kevin09 * (New user account) [12:46:29] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7101 [12:48:41] MySQL slave on z-dat-s6-a is OK: Uptime: 14956 Threads: 14 Questions: 922103 Slow queries: 2768 Opens: 502 Flush tables: 1 Open tables: 468 Queries per second avg: 61.654 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1782 [13:12:47] nosy: dns lookup isn't working anymore: $host -a de.wikipedia.org -> ;; connection timed out; no servers could be reached [13:13:18] oh dear....everthing is crashing today [13:13:22] must be monday [13:14:50] it depends where, nosy... ;-) [13:14:52] thats only since some minutes [13:15:45] Danny_B|backup: there should be only very few countries not having monday now [13:19:13] nosy and all: whoami is returning "cannot find name for user ID ...", don't know if related [13:19:46] contributors.php does not work right now [13:20:41] Hmmmm, and my IRC bot is not responding [13:21:53] qstat isn't working also [13:21:56] And I cannot login. [13:22:01] It's asking for password. [13:23:03] Oh, and tsnag and tsbot had a timeout 20 min ago [13:23:20] that because also local dns isn't working to, so qstat cannot find tunera [13:25:47] <|Danny_B|> huh - getpwuid() can't identify your account! [13:25:53] <|Danny_B|> what is that? [13:26:12] everytiem I come online, something crashes… [13:26:25] wait a moment, I will look for it [13:27:20] ldap has failed [13:27:36] seems turnera on heavy load? dabpunkt? [13:27:41] can you log in? [13:28:06] trying at the moment [13:28:21] do you have the ip? [13:28:43] I am now login to hemlock [13:29:03] <|Danny_B|> hmm, logging requires pwd and does not take the key from pageant [13:29:04] nosy: for turnera? 91.198.174.197 [13:29:11] on willow no filesystem (even local) is mounted [13:29:12] ok [13:29:22] gmm+ [13:29:35] ok bad things happening right now I see [13:29:39] that clears my doubts [13:30:26] nosy: I will try the service-console of turera [13:30:33] nightshade is refusing all my login attemps, I get a "key refused" message [13:30:43] DaBPunkt: good luck [13:30:52] i do not have the ip for this [13:31:03] drini: seems problems with auth and stuff [13:31:44] I bet more on the DNS-resolver [13:32:58] DaBPunkt: I have the feeling turnera is doing all this and should be under high load since i wait ages for the ssh prompt too [13:33:15] and nfs had a short hang [13:35:07] DaBPunkt: [13:35:11] problem: turnera's manegment-console is logout and needs a user/passwort-combination I don't know [13:35:13] switched ldap to damiana [13:35:39] great [13:35:40] :) [13:35:47] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:36:34] Ok, things seem to be back to normal :) [13:36:57] Load avg. on hyacinth is CRITICAL: Connection refused by host [13:36:57] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1939 [13:36:57] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1941.000000 [13:36:58] NTP on hyacinth is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset 0.007178 secs [13:36:59] DaBPunkt: are you booting turnera? [13:37:07] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1925.000000 [13:37:17] RAID on hyacinth is CRITICAL: Connection refused by host [13:37:17] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1880 [13:37:17] Cluster on damiana.esi is CRITICAL: turnera FAILED, damiana:nge1-turnera:nge1 faulted, damiana:nge0-turnera:nge0 faulted, vote turnera Offline, check nfs-hasp, nfs-home Online, [13:37:24] nosy: no, I still have no access [13:37:27] SMTP on hyacinth is CRITICAL: Connection refused [13:37:27] SSH on hyacinth is CRITICAL: Connection refused [13:37:36] its back again [13:37:36] /tmp on hyacinth is CRITICAL: Connection refused by host [13:37:46] /v/sql on hyacinth is CRITICAL: Connection refused by host [13:37:46] s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1905.000000 [13:37:48] seems it booted itself [13:37:56] Environment on hyacinth is CRITICAL: Connection refused by host [13:37:56] s5 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1713.000000 [13:38:06] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1747.000000 [13:38:13] am i here [13:38:16] MySQL slave on rosemary is OK: Uptime: 7673552 Threads: 8 Questions: 2311573402 Slow queries: 967034 Opens: 11534 Flush tables: 1 Open tables: 919 Queries per second avg: 301.239 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1725 [13:38:16] Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [13:38:36] NTP on turnera.esi is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.00657 secs [13:38:38] wb Danny_B|backup [13:38:47] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:39:46] s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1686.000000 [13:39:56] MySQL slave on daphne is OK: Uptime: 6100948 Threads: 24 Questions: 5117218210 Slow queries: 762898 Opens: 33144188 Flush tables: 3 Open tables: 16013 Queries per second avg: 838.757 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1660 [13:40:06] Cluster on turnera.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [13:41:26] turnera is acccessable by ssh again. I look in the syslog now [13:46:02] DaBPunkt: nge0 is flapping [13:46:11] yes, again [13:46:37] nosy: we should try to insert a direct cable between the 2 HA-hosts if possible [13:46:50] should be [13:47:42] DaBPunkt: good idea. unfortunatelly munin did not log anything special except higher disk usage [13:48:20] around 1:38 damiana's nginx became unstable and the service moved to turnera AFAIS [13:49:17] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [13:49:50] DaBPunkt: [13:49:51] rnosy@hemlock:~$ ssh hyacinth [13:49:51] ssh: connect to host hyacinth port 22: Connection refused [13:50:37] nosy: https://jira.toolserver.org/browse/MNT-1183 happend already yesterday [13:50:57] NTP on hyacinth is OK: NTP OK: Offset 0.003216 secs [13:50:58] DaBPunkt: i know [13:51:40] nosy: hyacinth is accessable by console [13:51:54] so maybe it just waits for ldap a little bit longer [13:53:36] NTP on turnera.esi is OK: NTP OK: Offset -0.002013 secs [13:55:31] DaBPunkt: are you using the console of hyacinth? [13:55:41] you can have it [13:55:47] ok thx [14:01:11] arround 7:18 damiana noticed network-problems. At 8:01 it came back from a reboot (I see no shutdown-messages), 8:00 turnera noticed that damiana was away and began to move a servies over [14:02:57] NTP on hyacinth is CRITICAL: NTP CRITICAL: No response from NTP server [14:03:57] NTP on hyacinth is OK: NTP OK: Offset -0.056946 secs [14:05:09] at 8:02 turnera's nge0 became flipping, damiana was still rebooting at the time [14:05:57] 08:03:58 both noticed each other again [14:10:52] at 08:04:09 turnera's nginx-check noticed that the network-interface is down and shutdown the ngixn-service [14:11:23] [[User talk:Kevin09]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=6628&rcid=8739 * Kevin09 * (+2925) (Created page with "At Always Fresh Rug cleaning we always strive to supply you the very best and safest carpet cleaning within Alexandria providers. We feel that a [http://sydneyscarpetcleaning.com...") [14:11:42] [[Special:Log/delete]] delete 10 * Dab * (deleted "[[02User talk:Kevin0910]]": SPAM) [14:17:15] nosy: both nodes showed a high memory-usage before the problems happen [14:31:34] DaBPunkt: hyacinth is in real trouble [14:31:35] root@hyacinth:/usr# /lib/svc/method/fs-root [14:31:35] ERROR: failed to mount /usr (see 'svcs -x' for details) [14:31:55] strange thing is: there is no /usr entry in vfstab [14:32:54] may I take a look? [14:34:47] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:35:36] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44611 MB (4% inode=99%): [14:35:47] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:36:56] Load avg. on hyacinth is CRITICAL: Connection refused by host [14:37:16] RAID on hyacinth is CRITICAL: Connection refused by host [14:37:26] SMTP on hyacinth is CRITICAL: Connection refused [14:37:27] SSH on hyacinth is CRITICAL: Connection refused [14:37:37] /tmp on hyacinth is CRITICAL: Connection refused by host [14:37:57] Environment on hyacinth is CRITICAL: Connection refused by host [14:38:46] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:40:12] nosy: I would say we reboot it and look what happen [14:40:28] no, wait please Dabpunkt [14:40:32] ok [14:41:05] ok, lets boot [14:41:20] I hate solaris… [14:42:45] DaBPunkt: http://tech.manunderstress.com/?p=90 [14:42:56] i eventually removed the last blank line [14:43:07] and i put it back in [14:43:41] if this really was the problem i'd want linux [14:47:01] does ping, no ssh-login, no serial console until now [14:48:27] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [14:48:27] SMTP on hyacinth is OK: SMTP OK - 0.129 sec. response time [14:48:27] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [14:48:37] /tmp on hyacinth is OK: DISK OK - free space: /tmp 43855 MB (100% inode=99%): [14:48:41] nosy: ok, online again [14:48:47] /v/sql on hyacinth is OK: DISK OK - free space: /v/sql 203840 MB (21% inode=99%): [14:48:56] Load avg. on hyacinth is OK: OK - load average: 0.91, 0.61, 0.31 [14:49:04] (that's the reason I put "#EOF" at the end of most config-files here ;)) [14:49:06] Environment on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [14:49:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:50:04] DaBPunkt: now the next problem [14:50:18] i got the volume back but i could not get the filesystem back [14:50:22] the are no problems, only solutions! ;) [14:50:30] i dont yet know what happened [14:50:31] for z-dat-s4-a? [14:50:34] yes [14:50:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:50:37] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6493 [14:50:37] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [14:50:37] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6199 [14:51:11] nosy: is the filesytem empty or destroyed? [14:51:16] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [14:51:28] the whole volume was destroyed [14:51:44] Then I guess there is no hope to restore it [14:51:58] i could put it back but when i configure like i did before i cant get no fs [14:52:47] and i read several things about vx raid setup backup with specific tools [14:52:51] mal auf deutsch: Das volumen WAR kaputt, das hast du repariert, aber es ist kein filesystem mehr drauf. Richtig verstanden? [14:52:57] we should think about it [14:52:59] ja genau [14:53:06] also mit dd sieht man noch was [14:53:13] aber das fsck kriegt es nicht hin [14:53:33] was sagt das fsck? [14:55:37] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3205 [14:55:39] dass es da kein valides fs oder irgendwelche superblöcke finden kann [14:55:50] du kannst ja mal gucken ob du optionen findest [14:56:44] plan: Du bereitest das Filesystem wieder her (formatieren, einbinden, etc) und am Mitwwoch während des Maintaince-Window schalten wir rosemary r/o und ziehen einen dump von commons. Den spielen wir dann auf z-dat-s4-a ein [14:56:48] ich habe gemacht: fsck -o full /dev/vx/rdsk/dat-s4-a/sql [14:57:09] ja das müssen wir wohl so machen [14:57:18] eigentlich wollte ich die wartung schon verschieben [14:57:37] MySQL slave on z-dat-s7-a is OK: Uptime: 558 Threads: 2 Questions: 264164 Slow queries: 15 Opens: 548 Flush tables: 1 Open tables: 531 Queries per second avg: 473.412 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1504 [14:57:46] weil ich hier so viel zeug um die ohren habe, dass ich es nicht geschafft habe, etwas sinnvolles vorzubereiten [14:58:37] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3461 [14:58:37] da wären immernoch: apache, php, mono, update von sun studio [14:58:55] also mir wäre sehr lieb, wenn ich nochmal ne wartung nächste woche machen könnte [14:59:01] nosy: ok, Donnerstag, Freitag oder erst die Woche drauf? [14:59:09] wenigstens die sun studio-patches wäre dann cool [14:59:21] naja, die commons-db sollten wir schon neu machen, oder? [14:59:53] klar, aber das kann ich selber machen, da muss ich dich nicht stören :) [15:00:56] ok...und ich mach dann nächste woche nochmal einige sachen [15:00:57] also planen wir einfach einen zusätzlichen Termin 1 Woche später ein? [15:01:05] das fänd ich gut [15:01:21] ok [15:02:02] s3 is slowly recovering [15:02:08] should be back up soon [15:02:13] nosy: I halted the tone for s4 again, no need to let it consume memory [15:02:20] tone → zone [15:02:29] true [15:02:37] MySQL slave on z-dat-s6-a is OK: Uptime: 857 Threads: 4 Questions: 239825 Slow queries: 25 Opens: 242 Flush tables: 1 Open tables: 212 Queries per second avg: 279.842 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1571 [15:03:16] MySQL on z-dat-s3-a is OK: Uptime: 898 Threads: 26 Questions: 13470 Slow queries: 1 Opens: 468 Flush tables: 1 Open tables: 355 Queries per second avg: 15.0 [15:05:10] [[Admin:Next maintenance]] 10https://wiki.toolserver.org/w/index.php?diff=6629&oldid=6537&rcid=8741 * Dab * (+71) (+structure) [15:09:23] 3(created) [TS-1277] Re-Setup z-dat-s4-a; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1277> (DaB.) [15:10:27] [[Admin:Next maintenance]] 10https://wiki.toolserver.org/w/index.php?diff=6630&oldid=6629&rcid=8742 * Dab * (+41) (/* This maintenance */ ) [15:11:25] 3(assigned) [TS-1277] Re-Setup z-dat-s4-a <10https://jira.toolserver.org/browse/TS-1277> (DaB.) [15:15:24] 3(assigned) [TS-1274] Update rsvg to 2.35.1 <10https://jira.toolserver.org/browse/TS-1274> (DaB.) [15:15:43] [[Admin:Next maintenance]] 10https://wiki.toolserver.org/w/index.php?diff=6631&oldid=6630&rcid=8743 * Dab * (+91) (/* This maintenance */ ) [15:19:15] dabpunkt what about MMP interwiki-bot? [15:19:38] pardon? [15:21:43] on 1 february you wrote a mail: [Toolserver-l] Adding of a new RULE for langlink-bots [15:22:04] saying "it will not longer be allowed to run most langlink-bots (aka interwiki-bots)" [15:22:32] and that the bot will be run by a MMP to be created [15:22:52] yes, and? [15:23:44] i wanted to know whether the mmp had been created [15:24:28] no. Until now no-one got active on that [15:26:31] i am concerned with it.wiki, and i know how difficult is keeping a big wiki up to date with langlinks, so i am interested in this thing [15:29:26] nickanc: Would you be interrested in organizing the creation of the MMP? Your main topic would be to find the 5 users who builds the MMP. [15:32:24] dabpunkt: i am not sure to be able to deal with every technical issue, anyway, i may try to find also users able to, ok? [15:34:57] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:35:31] nickanc: I don't care how the 5 users are choosen at the end – the ts-community should organize this. If you like to do this organizing, you are very welcome. The technical creation of the MMP at the end is my job, but that is the least problem [15:35:47] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45286 MB (4% inode=99%): [15:35:47] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:35:57] ok! [15:38:57] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:50:47] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7100 [15:58:47] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3338 [16:01:46] MySQL slave on z-dat-s3-a is OK: Uptime: 4411 Threads: 11 Questions: 3365808 Slow queries: 167 Opens: 23122 Flush tables: 1 Open tables: 16066 Queries per second avg: 763.48 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1584 [16:07:08] [[OpenStreetMap]] ! 10https://wiki.toolserver.org/w/index.php?diff=6632&oldid=5962&rcid=8744 * 84.50.62.182 * (+113) (/* projects */ ) [16:15:03] DaB. * Re: [Toolserver-l] dumps directory [16:18:22] Daniel_WMDE: Hm.. Just checking out CommonSense for something, and getting errors from curl in the output [16:18:26] https://toolserver.org/~daniel/WikiSense/CommonSense.php?u=en&i=Image%3AMonachus+tropicalis.gif&kw=Caribbean+monk+seal+Monachus+tropicalis&p=_20&go-clean=Find+Categories&cl=&w=en&v=0 [16:18:31] - curl returned an error: Couldn't resolve host 'rr.pmtpa.wikimedia.org' [16:18:32] about 20 times [16:18:44] seems like a weird host to be requested for resolving [16:18:52] why would commonsense connect to that [16:18:54] [[OpenStreetMap]] ! 10https://wiki.toolserver.org/w/index.php?diff=6633&oldid=6632&rcid=8745 * 84.50.62.182 * (+137) (/* projects */ ) [16:20:26] commonswiki-p.rrdb.toolserver.org ? [16:24:03] DaB. * [Toolserver-announce] Short outtake arround noon and s4-problems [16:34:57] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:35:47] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45151 MB (4% inode=99%): [16:35:47] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:38:56] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:34:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:35:46] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44956 MB (4% inode=99%): [17:35:57] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:36:20] 3(resolved) [UTRS-19] Add list of inactive users for account disabling (suspension after x days) <10https://jira.toolserver.org/browse/UTRS-19> (Thehelpfulone) [17:38:57] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:02:17] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.373047/1.75, alarm hl:np_load_avg=0.385742/2.00, alarm hl:mem_free=230.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.373047/1.50, alarm hl:np_load_long=0.356445/1.75, alarm hl:mem_free=230.000000M/250M [18:04:17] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [18:07:17] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.292969/1.75, alarm hl:np_load_avg=0.324219/2.00, alarm hl:mem_free=199.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.292969/1.50, alarm hl:np_load_long=0.335449/1.75, alarm hl:mem_free=199.000000M/250M [18:28:17] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.315430/1.00, alarm hl:np_load_long=0.788086/1.50, alarm hl:mem_free=21848.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.315430/1.10, alarm hl:np_load_long=0.788086/1.75, alarm hl:mem_free=21848.000000M/300M [18:29:16] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [18:34:56] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:35:47] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44791 MB (4% inode=99%): [18:35:57] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:38:57] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:12:26] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.407226/1.75, alarm hl:np_load_avg=0.359375/2.00, alarm hl:mem_free=240.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.407226/1.50, alarm hl:np_load_long=0.348633/1.75, alarm hl:mem_free=240.000000M/250M [19:13:27] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [19:25:02] [[User:Dab]] ! 10https://wiki.toolserver.org/w/index.php?diff=6634&oldid=2504&rcid=8746 * 87.96.84.8 * (-867) (Replaced content with "Siema eniu Macie bana tu robaczek") [19:26:08] Whats up position You have a worm banned here [19:26:11] interesting. [19:26:31] [[User:Dab]] M 10https://wiki.toolserver.org/w/index.php?diff=6635&oldid=6634&rcid=8747 * Valhallasw * (+867) (Reverted edits by [[Special:Contributions/87.96.84.8|87.96.84.8]] ([[User talk:87.96.84.8|talk]]) to last revision by [[User:Simetrical|Simetrical]]) [19:27:01] [[Special:Log/block]] block 10 * Valhallasw * (blocked [[02User:87.96.84.810]] with an expiry time of infinite (anonymous users only, account creation disabled): Intimidating behaviour/harassment) [19:27:46] [[Toolserver Intuition]] M 10https://wiki.toolserver.org/w/index.php?diff=6636&oldid=6627&rcid=8749 * Valhallasw * (+16937) (Reverted edits by [[Special:Contributions/203.189.188.105|203.189.188.105]] ([[User talk:203.189.188.105|talk]]) to last revision by [[User:94.15.30.112|94.15.30.112]]) [19:27:59] [[Special:Log/block]] block 10 * Valhallasw * (blocked [[02User:203.189.188.10510]] with an expiry time of infinite (anonymous users only, account creation disabled): Removing content from pages) [19:35:07] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:35:47] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44627 MB (4% inode=99%): [19:35:58] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:39:06] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:39:24] 3(commented) [ACCAPP-435] Running the bot for cleanup tasks like spell-check, transliterate, etc. <10https://jira.toolserver.org/browse/ACCAPP-435> [20:12:20] 3(created) [ACCAPP-448] Involvement in development of tools which are used in the russian wikipedia; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-448> (Dmitry Logvinenko) [20:19:26] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 20687 MB (2% inode=99%): [20:35:17] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:35:46] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44496 MB (4% inode=99%): [20:36:06] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:39:16] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:39:49] [[Datacenter-Tasks]] N 10https://wiki.toolserver.org/w/index.php?oldid=6637&rcid=8751 * Dab * (+34) (Created page with "*Reset the LOM-password of tunera.") [20:42:37] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.379395/1.75, alarm hl:np_load_avg=0.417480/2.00, alarm hl:mem_free=263.000000M/300M [20:43:37] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [20:45:06] [[Datacenter-Tasks]] 10https://wiki.toolserver.org/w/index.php?diff=6638&oldid=6637&rcid=8752 * Dab * (+349) (More stuff) [20:45:27] [[Special:Log/move]] move 10 * Dab * (moved [[02Datacenter-Tasks10]] to [[Admin:Datacenter-Tasks]]: It's a root-thing) [20:47:37] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.321289/1.75, alarm hl:np_load_avg=0.380371/2.00, alarm hl:mem_free=270.000000M/300M [21:23:04] zzzzz [21:35:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:35:56] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45255 MB (4% inode=99%): [21:36:17] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:39:17] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:13:47] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.470215/1.75, alarm hl:np_load_avg=0.470215/2.00, alarm hl:mem_free=289.000000M/300M [22:15:47] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [22:35:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:35:56] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45099 MB (4% inode=99%): [22:36:17] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:39:16] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:32:47] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.457520/1.75, alarm hl:np_load_avg=0.426270/2.00, alarm hl:mem_free=283.000000M/300M [23:35:26] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:35:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44935 MB (4% inode=99%): [23:36:27] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:38:47] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [23:39:26] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:55:06] nacht ts