[00:03:02] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.917969/2.00, alarm hl:mem_free=217.000000M/250M [00:03:52] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [00:05:03] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 19.547 second response time [00:05:04] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:07:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4577 [00:08:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4575.000000 [00:13:03] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 272 bytes in 20.232 second response time [00:13:04] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.924805/2.00, alarm hl:mem_free=154.000000M/250M [00:23:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [00:30:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 206522 MB (3% inode=26%): [00:32:33] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:32:33] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:33:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66094 MB (6% inode=99%): [00:34:03] /sql on adenia is OK: DISK OK - free space: /sql 119893 MB (18% inode=99%): [00:38:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [00:39:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:54:03] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3172 [00:57:06] nacht ts [01:07:12] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.171875/1.00, alarm hl:mem_free=15035.000000M/100M [01:08:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6614 [01:09:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6618.000000 [01:13:11] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [01:23:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:26:12] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.015625/1.00, alarm hl:mem_free=15127.000000M/100M [01:26:52] good night =_= [01:30:12] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 207503 MB (3% inode=26%): [01:32:43] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:32:43] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:34:02] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66916 MB (6% inode=99%): [01:37:12] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.711426/2.00, alarm hl:mem_free=239.000000M/250M [01:37:13] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [01:38:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [01:39:13] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:41:13] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:47:12] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.695312/2.00, alarm hl:mem_free=200.000000M/250M [01:49:24] 3(created) [ACCAPP-431] CLONE - Assist in Unblock request tool; Account Approval; Trivial New Account <10https://jira.toolserver.org/browse/ACCAPP-431> (Andrew Pearson) [01:51:20] 3(commented) [ACCAPP-431] CLONE - Assist in Unblock request tool <10https://jira.toolserver.org/browse/ACCAPP-431> (Andrew Pearson) [01:54:12] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3206 [02:00:24] 3(commented) [ACCAPP-431] CLONE - Assist in Unblock request tool <10https://jira.toolserver.org/browse/ACCAPP-431> (Brett Reynolds) [02:08:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6986 [02:09:12] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7032.000000 [02:11:12] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3631 [02:23:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [02:30:13] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 207167 MB (3% inode=26%): [02:33:41] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:33:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:34:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66802 MB (6% inode=99%): [02:38:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [02:39:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:46:13] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.048828/1.00, alarm hl:mem_free=15332.000000M/100M [02:49:12] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [03:08:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6701 [03:09:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6544.000000 [03:11:12] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5595 [03:23:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [03:31:13] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 233004 MB (4% inode=29%): [03:33:42] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:33:43] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:35:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66667 MB (6% inode=99%): [03:38:15] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [03:39:15] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:39:25] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1857 [03:45:25] MySQL slave on z-dat-s3-a is OK: Uptime: 911150 Threads: 21 Questions: 884625212 Slow queries: 128501 Opens: 13120158 Flush tables: 2 Open tables: 16384 Queries per second avg: 970.888 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1773 [04:01:27] @replag [04:01:29] Chris_G: s1-sec: 1h 9m 1s [+0.01 s/s]; s2-pri: 16s [+0.00 s/s]; s3-rr: 27s [-0.00 s/s]; s3-user: 27s [-0.00 s/s]; s7-rr: 1h 20m 13s [+0.02 s/s]; s7-user: 1h 20m 13s [+0.02 s/s] [04:08:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4002 [04:09:25] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4049.000000 [04:11:25] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4347 [04:13:25] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.803711/2.00, alarm hl:mem_free=175.000000M/250M [04:13:55] SMF on turnera.esi is UNKNOWN: Invalid host name turnera.esi [04:16:25] SMTP on rosemary is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:18:05] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3596 [04:18:24] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3506.000000 [04:18:57] siebrand: poke [04:19:26] SMTP on rosemary is OK: SMTP OK - 7.772 sec. response time [04:26:01] is it me or has SSHing just died? [04:31:14] Hi, the toolserver seems to be unresponsive. :( [04:31:14] nightshade/willow/wolfsbane are unreachable [04:31:51] The topic is truncated. [04:31:54] I am getting Cannot connect to LDAP server errrors [04:32:03] Apparently not your fault, though. [04:32:57] @replag [04:32:57] anyone have an admins cell number? [04:33:08] Jeff_G: bots down [04:34:49] You could probably find River's number, but she's mostly gone. [04:34:50] Aren't they all in Europe? They'd presumably be sleeping currently [04:35:03] DaB. and most of the WMDE folks are in Germany, yeah. [04:35:06] Not sure where nosy is. [04:35:12] See, that's the problem :( [04:35:22] And Reedy will be asleep (not that'd he'd be useful). [04:35:51] Betacommand: E-mail the mailing list? [04:35:58] Joan: I think she is in Germany too [04:36:09] Seems likely. [04:36:44] Joan: in the process [04:37:11] I have an active SSH session into nightshade, but it's now hanging on "ls". [04:39:00] probably nfs went tits up [04:39:00] Joan: Just got your email :) [04:39:14] *John Sorry [04:39:19] or ldap [04:39:44] Hi woosters. [04:39:52] hi [04:40:12] wassup joan? [04:40:24] toolserver died [04:40:27] nothing aparently [04:40:40] http://lists.wikimedia.org/pipermail/toolserver-l/2011-December/004639.html [04:40:46] stwalkerster: Sorry, I loled :P [04:40:49] O.O [04:41:00] woosters: I'm ready for the holidays. :-) [04:41:02] You? [04:41:17] I just logged into Nightshade [04:41:19] yea, me too! [04:41:34] amd now, everything is coming up [04:41:38] Odd. [04:41:43] Seems like it was NFS-related. [04:41:44] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66554 MB (6% inode=99%): [04:41:49] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [04:41:50] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:41:50] SMF on turnera.esi is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:42:02] let me see if i could contact some of those guys [04:42:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [04:43:20] Cluster on turnera.esi is CRITICAL: check nfs-hasp, nfs-home Online, [04:43:40] NTP on turnera.esi is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.007869 secs [04:43:51] NTP on damiana.esi is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.007503 secs [04:44:01] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.302246/2.00, alarm hl:mem_free=141.000000M/250M [04:44:10] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, [04:46:03] John * [Toolserver-l] Toolserver outage, [04:46:39] It appears that the replag graphing quit at about 04:11:30 UTC. [04:47:01] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [04:56:01] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1933.000000 [04:56:10] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1941.000000 [04:56:23] 3(commented) [OSM-2] ST_Transform doesn't work <10https://jira.toolserver.org/browse/OSM-2> (Kai Krueger) [04:56:40] NTP on turnera.esi is OK: NTP OK: Offset -0.012172 secs [04:56:50] NTP on damiana.esi is OK: NTP OK: Offset -0.01068 secs [04:58:50] s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1957.000000 [05:01:01] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1943.000000 [05:01:10] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1923.000000 [05:01:10] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1956.000000 [05:02:01] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1976.000000 [05:24:01] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3614.000000 [05:24:10] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3622.000000 [05:26:50] s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3636.000000 [05:29:00] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3623.000000 [05:29:10] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3604.000000 [05:29:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3636.000000 [05:30:10] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3656.000000 [05:30:10] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 604.000000 [05:32:00] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3704 [05:32:10] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3825 [05:32:10] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3722 [05:32:11] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3783 [05:32:19] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3730 [05:32:31] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2987 [05:32:51] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2301 [05:32:51] s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3600.000000 [05:33:01] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3574.000000 [05:33:01] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3522 [05:33:10] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3546 [05:34:10] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3501.000000 [05:34:20] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3477 [05:34:50] MySQL slave on z-dat-s4-a is OK: Uptime: 2101855 Threads: 11 Questions: 92349985 Slow queries: 21075 Opens: 14925 Flush tables: 1 Open tables: 402 Queries per second avg: 43.937 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1529 [05:36:30] MySQL slave on z-dat-s6-a is OK: Uptime: 2184736 Threads: 10 Questions: 383832554 Slow queries: 148134 Opens: 4275358 Flush tables: 2 Open tables: 2794 Queries per second avg: 175.688 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1359 [05:39:11] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3558 [05:39:11] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3588 [05:41:01] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 205589 MB (3% inode=26%): [05:41:30] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:41:40] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66426 MB (6% inode=99%): [05:41:50] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:41:50] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:41:50] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [05:41:50] s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1764.000000 [05:42:00] MySQL slave on daphne is OK: Uptime: 2097871 Threads: 28 Questions: 1889012423 Slow queries: 241585 Opens: 9378493 Flush tables: 3 Open tables: 15021 Queries per second avg: 900.442 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1764 [05:42:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:44:00] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1657.000000 [05:44:10] MySQL slave on thyme is OK: Uptime: 3051356 Threads: 15 Questions: 857225621 Slow queries: 417395 Opens: 3358812 Flush tables: 1 Open tables: 2881 Queries per second avg: 280.932 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1608 [05:50:10] MySQL slave on z-dat-s7-a is OK: Uptime: 2185558 Threads: 13 Questions: 466511047 Slow queries: 61566 Opens: 5212949 Flush tables: 1 Open tables: 6578 Queries per second avg: 213.451 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1783 [05:58:10] MySQL slave on z-dat-s3-a is OK: Uptime: 919116 Threads: 26 Questions: 891139440 Slow queries: 129105 Opens: 13205687 Flush tables: 2 Open tables: 16384 Queries per second avg: 969.561 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1708 [06:09:03] Tanvir Rahman * Re: [Toolserver-l] Toolserver outage, [06:09:45] Tanvir: so, i should hold off ringing germany? [06:10:03] Sure. :) [06:10:21] Still it's not working for you Jeremyb? [06:10:43] Tanvir: i have approximately nothing to check. i don't have an account [06:11:08] Ah, independent guy you are! :) [06:11:30] * Tanvir goes idling. [06:11:35] lazing* [06:12:11] enjoy [06:13:09] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1699.000000 [06:13:20] MySQL slave on rosemary is OK: Uptime: 3672454 Threads: 32 Questions: 1062726445 Slow queries: 438051 Opens: 5544 Flush tables: 1 Open tables: 837 Queries per second avg: 289.377 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1654 [06:29:01] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7223.000000 [06:29:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7235.000000 [06:31:01] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6341.000000 [06:33:00] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1988.000000 [06:34:01] s5 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 784.000000 [06:37:00] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3065.000000 [06:38:10] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3554.000000 [06:40:00] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 764.000000 [06:41:00] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 205241 MB (3% inode=26%): [06:41:10] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1616.000000 [06:41:31] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:41:40] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66321 MB (6% inode=99%): [06:41:50] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:41:50] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:41:50] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:11:30] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 83388 MB (8% inode=99%): [07:12:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [07:41:31] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:41:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67154 MB (6% inode=99%): [07:41:52] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:41:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:42:00] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 204784 MB (3% inode=26%): [07:42:10] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:07:20] 3(updated) [ACCAPP-425] Run a bot that adds and removes protection templates on Wikipedia. <10https://jira.toolserver.org/browse/ACCAPP-425> (Andrew Wang) [08:13:00] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.153809/2.00, alarm hl:mem_free=134.000000M/250M [08:14:01] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [08:41:40] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:42:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:42:01] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:42:01] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 211275 MB (3% inode=27%): [08:42:19] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:42:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [08:42:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66528 MB (6% inode=99%): [09:13:20] 3(commented) [VVV-25] sulutil giving a faulty contrib count <10https://jira.toolserver.org/browse/VVV-25> (Manish Goregaokar) [09:33:20] 3(created) [ACC-240] Can the list handle spoofs?; ACC: E-Mail; Improvement <10https://jira.toolserver.org/browse/ACC-240> (Manish Goregaokar) [09:39:20] 3(resolved) [CHTWO-41] Typo on attention warning <10https://jira.toolserver.org/browse/CHTWO-41> (Jan Luca) [09:41:20] 3(commented) [CHTWO-42] Warning warnings are most of the time unnecessary <10https://jira.toolserver.org/browse/CHTWO-42> (Jan Luca) [09:41:49] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:42:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:42:12] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 220266 MB (4% inode=28%): [09:42:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:42:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [09:42:30] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [09:43:01] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66934 MB (6% inode=99%): [09:43:19] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.197266/1.00, alarm hl:mem_free=15017.000000M/100M [09:45:25] 3(updated) [CHTWO-40] header lines not replaced <10https://jira.toolserver.org/browse/CHTWO-40> (Jan Luca) [09:45:26] 3(updated) [CHTWO-38] I am never able to use this tool to transfer images either from Wikipedia, or Wikisource <10https://jira.toolserver.org/browse/CHTWO-38> (Jan Luca) [09:47:26] 3(resolved) [CHTWO-36] Attention : The upload function is only available to some users during the test period! <10https://jira.toolserver.org/browse/CHTWO-36> (Jan Luca) [09:51:20] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [09:56:43] @replag [09:56:43] jem-: s2-pri: 10s [-] [10:02:19] Load avg. on willow is WARNING: WARNING - load average: 16.64, 14.75, 11.87 [10:03:13] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.780762/1.50, alarm hl:mem_free=573.000000M/100M [10:03:20] Load avg. on willow is OK: OK - load average: 12.72, 13.92, 11.75 [10:06:11] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [10:41:50] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:42:23] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 220103 MB (4% inode=28%): [10:42:23] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:42:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:42:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [10:42:40] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [10:43:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66820 MB (6% inode=99%): [11:41:50] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:42:32] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 228924 MB (4% inode=28%): [11:42:33] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:42:33] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:42:59] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [11:44:00] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66704 MB (6% inode=99%): [12:12:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [12:42:10] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:42:32] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 228685 MB (4% inode=28%): [12:42:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:42:40] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:43:20] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [12:44:10] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66186 MB (6% inode=99%): [12:44:40] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.120117/1.00, alarm hl:mem_free=14814.000000M/100M [12:49:40] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [13:00:21] 3(commented) [ACCAPP-413] I want to operate my global bot "GedawyBot" on toolserver in order to add and update interwikis <10https://jira.toolserver.org/browse/ACCAPP-413> (Mohamed ElGedawy) [13:12:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [13:20:53] i'm having problems on a bash script execution, i get: /bin/sh: 1: not found ... /opt/local/bin/cronsub[50]: bash: cannot open [13:28:03] Marlen Caemmerer * Re: [Toolserver-l] Toolserver outage, [13:39:37] hello all [13:42:20] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:42:42] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 228471 MB (4% inode=28%): [13:42:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:42:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:43:24] DaBPunkt: i'm having a problem with starting a sh script in cronie. i have this: http://pastebin.com/8Ahmw56k [13:43:50] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [13:44:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66466 MB (6% inode=99%): [13:44:28] Alchimista: remove the space bewteen the "!" and the "/" [13:45:44] DaBPunkt: i'll try that way. peraps the doc's should be updated: https://wiki.toolserver.org/view/Job_scheduling [13:46:09] mm [13:46:42] maybe I'm wrong, let me test it [13:48:21] mm, looks like this is not the problem [13:48:31] I will investigate, wait a moment [13:48:37] ok [13:53:07] Alchimista: on which host is the cron? [13:53:12] submit? [13:53:46] i'm not sure, from willows i use "cronie -e" [13:53:54] ok, then willow [13:59:02] Alchimista: Are you sure that you want "screen -r" and not "screen -S"? [13:59:37] DaBPunkt: i already have a screen created has alph [14:00:59] sorry, I killed that [14:01:39] nop, if there is a better way to do it, i'm ready to learn. so i've changed the .sh to use -S [14:04:00] the normal way is to send the output of a daemon to a log-file. A screen is normaly only used to let a interactive-programm in the background [14:04:12] +run [14:05:17] so to do that, it's just removing the screen line? [14:06:11] yes, and send the output to a file [14:08:24] but still with the bash problem :s [14:09:15] it stiil fails? [14:10:49] DaBPunkt: seems to be running now! [14:13:03] I stoped it again. Let's wait for the next timeslot [14:13:24] ok [14:18:00] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 11.233 second response time [14:21:00] Alchimista: I have to leave for a momemnt, cu [14:27:12] DaBPunkt: i must go now, but it is working again, thanks [14:42:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [14:42:40] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:42:51] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 228246 MB (4% inode=28%): [14:42:51] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:42:59] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:44:29] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66344 MB (6% inode=99%): [15:42:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [15:42:52] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:42:52] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 235764 MB (4% inode=29%): [15:43:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:43:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:44:29] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66140 MB (6% inode=99%): [16:06:11] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.166016/1.00, alarm hl:mem_free=14844.000000M/100M [16:13:10] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.827637/2.00, alarm hl:mem_free=242.000000M/250M [16:14:10] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [16:23:11] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [16:42:51] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:43:00] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 233757 MB (4% inode=29%): [16:43:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:44:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:44:41] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66933 MB (6% inode=99%): [16:45:10] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.131836/1.00, alarm hl:mem_free=14877.000000M/100M [16:48:10] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [17:12:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [17:19:20] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.066406/1.00, alarm hl:mem_free=14973.000000M/100M [17:25:20] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [17:39:00] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 15.958 second response time [17:43:10] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 231917 MB (4% inode=29%): [17:43:10] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:43:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:44:20] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:44:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66856 MB (6% inode=99%): [17:44:59] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:00:00] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 18.071 second response time [18:01:10] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:12:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:14:59] confused about server structure... i requested that a python module be installed, and it was apparently installed on the nightshade server. if I SSH into the nightshade server, python can find the module. [18:15:58] however, I have some python CGI scripts which are running web tools, and they can't find the module [18:16:07] does the module need to be installed on a different server [18:16:09] ? [18:43:09] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 230556 MB (4% inode=29%): [18:43:20] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:43:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:44:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:45:41] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66727 MB (6% inode=99%): [18:49:09] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.545410/1.50, alarm hl:mem_free=1406.000000M/100M [18:50:10] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [18:50:10] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:53:24] 3(commented) [RIVER-22] Request for installation of Babel python module <10https://jira.toolserver.org/browse/RIVER-22> (Snottywong) [19:11:31] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 81530 MB (8% inode=99%): [19:42:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [19:43:14] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 228862 MB (4% inode=28%): [19:43:32] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:43:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:44:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:46:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66571 MB (6% inode=99%): [19:50:32] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [20:09:20] 3(created) [MNT-1163] Restarted mysql on adenia (sql); Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1163> (DaB.) [20:22:20] 3(resolved) [MNT-1163] Restarted mysql on adenia (sql) <10https://jira.toolserver.org/browse/MNT-1163> (DaB.) [20:42:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [20:43:22] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 227069 MB (4% inode=28%): [20:43:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:43:42] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:45:31] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:47:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65483 MB (6% inode=99%): [20:50:42] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [21:43:43] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:43:43] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:44:20] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 225113 MB (4% inode=28%): [21:44:42] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.253906/1.00, alarm hl:mem_free=14646.000000M/100M [21:45:31] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:47:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66274 MB (6% inode=99%): [21:50:42] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [21:50:51] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [22:12:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [22:12:52] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.071777/2.00, alarm hl:mem_free=236.000000M/250M [22:13:52] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [22:43:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:43:52] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:44:32] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 221515 MB (4% inode=28%): [22:45:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:48:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66131 MB (6% inode=99%): [22:50:52] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [23:06:09] nacht ts [23:32:02] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.837402/2.00, alarm hl:mem_free=204.000000M/250M [23:33:01] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [23:42:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [23:44:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:44:13] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:45:31] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 209119 MB (3% inode=26%): [23:46:31] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:48:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66940 MB (6% inode=99%): [23:51:13] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds