[00:09:53] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:11:52] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:12:42] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [00:13:42] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [00:17:53] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:27:31] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53559 MB (5% inode=99%): [01:10:51] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:12:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:12:51] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [01:13:42] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [01:18:52] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:27:41] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53461 MB (5% inode=99%): [02:10:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:12:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:12:52] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [02:13:42] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [02:18:51] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:27:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54311 MB (5% inode=99%): [03:11:01] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:12:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:12:22] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:12:42] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:12:43] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:12:52] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:12:53] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:12:53] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [03:12:53] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:13:03] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:13:12] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:13:22] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:13:22] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:13:32] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:13:42] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [03:14:02] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [03:14:11] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [03:14:12] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [03:14:21] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 3199 MB (99% inode=99%): [03:14:22] SMTP on z-dat-s3-a is OK: SMTP OK - 7.843 sec. response time [03:14:22] MySQL slave on z-dat-s3-a is OK: Uptime: 3501291 Threads: 18 Questions: 3805650394 Slow queries: 311061 Opens: 48432620 Flush tables: 2 Open tables: 16384 Queries per second avg: 1086.927 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 249 [03:14:22] MySQL on z-dat-s3-a is OK: Uptime: 3501291 Threads: 18 Questions: 3805650395 Slow queries: 311061 Opens: 48432620 Flush tables: 2 Open tables: 16384 Queries per second avg: 1086.927 [03:14:22] SMTP on z-dat-s4-a is OK: SMTP OK - 7.661 sec. response time [03:14:22] MySQL slave on z-dat-s7-a is OK: Uptime: 2273728 Threads: 4 Questions: 522186671 Slow queries: 92346 Opens: 6227209 Flush tables: 1 Open tables: 3770 Queries per second avg: 229.661 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 244 [03:14:31] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:14:32] SMTP on z-dat-s7-a is OK: SMTP OK - 0.042 sec. response time [03:14:32] SMTP on z-dat-s6-a is OK: SMTP OK - 0.024 sec. response time [03:14:43] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:14:43] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:14:43] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 3527 MB (99% inode=99%): [03:14:43] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:14:52] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:18:52] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:28:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54225 MB (5% inode=99%): [04:11:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:12:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:12:53] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [04:13:41] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [04:18:53] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:29:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54146 MB (5% inode=99%): [05:04:02] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.491211/1.75, alarm hl:np_load_avg=0.560059/2.00, alarm hl:mem_free=263.000000M/300M [05:06:01] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [05:11:13] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:12:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:13:02] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [05:13:42] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [05:19:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:25:13] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:25:42] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:25:52] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:26:02] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:26:14] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:26:14] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:26:31] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [05:26:42] MySQL slave on z-dat-s7-a is OK: Uptime: 2281660 Threads: 7 Questions: 522935525 Slow queries: 92578 Opens: 6227823 Flush tables: 1 Open tables: 3884 Queries per second avg: 229.190 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [05:26:43] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:26:52] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:29:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53968 MB (5% inode=99%): [06:11:13] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:12:14] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:13:11] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [06:14:41] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [06:20:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:29:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53906 MB (5% inode=99%): [07:12:13] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:13:13] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [07:13:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:14:42] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [07:20:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:29:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53839 MB (5% inode=99%): [07:41:52] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:52] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:42:02] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:42:22] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:42:22] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:42:31] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:42:32] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:42:41] SMTP on z-dat-s6-a is OK: SMTP OK - 0.002 sec. response time [07:42:42] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [07:42:52] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [07:42:52] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 3530 MB (99% inode=99%): [07:43:02] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 3512 MB (99% inode=99%): [07:43:22] SMTP on z-dat-s3-a is OK: SMTP OK - 0.012 sec. response time [07:43:22] SMTP on z-dat-s4-a is OK: SMTP OK - 0.042 sec. response time [08:08:42] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 41978 MB (4% inode=99%): [08:13:12] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [08:13:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:14:12] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:14:52] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [08:20:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:30:41] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53756 MB (5% inode=99%): [09:13:13] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [09:13:13] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:14:12] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:14:53] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [09:20:13] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:30:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53694 MB (5% inode=99%): [09:53:43] @replag [09:53:44] Quentinv57: s2-pri: 11m 21s [+0.01 s/s]; s2/s5-pri-c: 15m 7s [+0.01 s/s]; s3-rr: error; s3-user: error; s4-user: 51s [+0.00 s/s]; s5-user: 15m 13s [+0.01 s/s]; s6-rr: error; s6-user: error [09:53:45] Quentinv57: s7-rr: error; s7-user: error [09:53:50] hmmm [10:13:11] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [10:13:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:15:11] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:15:52] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [10:21:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:31:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53619 MB (5% inode=99%): [10:42:01] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:42:02] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:02] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:32] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:42:52] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:42:52] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:13:21] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [11:13:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:15:20] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:15:53] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [11:21:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:31:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53536 MB (5% inode=99%): [12:13:21] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [12:13:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:14:03] Merlijn van Deen * [Toolserver-l] toolserver interwiki project (was: interwiki.py) [12:15:20] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:15:53] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [12:21:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:31:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 50624 MB (5% inode=99%): [12:40:21] [[Special:Log/newusers]] create 10 * Arisdp * (New user account) [12:44:23] 3(created) [ACCAPP-440] Wikimedia Hackathon; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-440> (Mahmoud Hashemi) [12:46:21] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.081055/1.00, alarm hl:np_load_long=0.898438/1.50, alarm hl:mem_free=24188.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.081055/1.10, alarm hl:np_load_long=0.898438/1.75, alarm hl:mem_free=24188.000000M/300M [12:47:20] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [12:49:29] is there something like memcached on the TS? [12:49:49] ah, on amaranth [12:50:37] not exactly what I was looking for, though. Hm. [13:13:20] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [13:13:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:15:20] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:15:53] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [13:22:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:31:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 20156 MB (2% inode=99%): [13:55:01] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:02] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:02] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:55:02] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:02] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:02] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:23] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:55:23] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:55:32] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:52] SMTP on z-dat-s7-a is OK: SMTP OK - 0.003 sec. response time [13:55:52] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [13:55:53] SMTP on z-dat-s6-a is OK: SMTP OK - 0.003 sec. response time [13:55:53] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [13:55:53] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [13:55:53] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 3534 MB (99% inode=99%): [13:55:53] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [13:56:01] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 3535 MB (99% inode=99%): [13:56:21] SMTP on hyacinth is OK: SMTP OK - 0.004 sec. response time [14:13:22] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [14:13:22] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:15:22] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:16:02] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [14:22:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:28:02] DaB. * Re: [Toolserver-l] interwiki.py [14:32:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53252 MB (5% inode=99%): [15:14:22] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:14:22] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [15:16:22] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:17:02] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [15:22:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:32:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54077 MB (5% inode=99%): [15:55:03] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:12] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:12] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:12] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:12] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:12] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:32] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:32] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:32] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:32] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:55:41] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:01] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:12] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 3385 MB (99% inode=99%): [15:56:12] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:56:12] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:56:12] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:56:12] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:56:13] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:56:13] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [15:56:13] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [15:56:14] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 3300 MB (99% inode=99%): [15:56:21] MySQL slave on z-dat-s3-a is OK: Uptime: 3547006 Threads: 15 Questions: 3846363649 Slow queries: 313462 Opens: 48691442 Flush tables: 2 Open tables: 16384 Queries per second avg: 1084.397 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 185 [15:56:22] MySQL on z-dat-s3-a is OK: Uptime: 3547006 Threads: 15 Questions: 3846363674 Slow queries: 313462 Opens: 48691442 Flush tables: 2 Open tables: 16384 Queries per second avg: 1084.397 [15:56:22] SMTP on hyacinth is OK: SMTP OK - 0.002 sec. response time [15:56:22] SMTP on z-dat-s4-a is OK: SMTP OK - 0.013 sec. response time [15:56:22] SMTP on z-dat-s3-a is OK: SMTP OK - 0.007 sec. response time [15:56:32] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [15:56:53] SMTP on z-dat-s7-a is OK: SMTP OK - 0.016 sec. response time [16:13:21] 3(created) [ACCAPP-441] SVN, Database access and storage; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-441> [16:14:32] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [16:14:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:16:32] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:18:02] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [16:22:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:32:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53955 MB (5% inode=99%): [16:48:21] Sun Grid Engine execd on nightshade is WARNING: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=1.536133/1.50, alarm hl:np_load_long=0.926269/1.75, alarm hl:mem_free=3523.000000M/250M [16:49:22] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [17:05:22] 3(commented) [MNT-1173] Add the index on the hstore columns on osm_mapnik database again, that were forgotten during the re-import <10https://jira.toolserver.org/browse/MNT-1173> (Kai Krueger) [17:15:32] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [17:15:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:16:31] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:18:11] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [17:22:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:32:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53836 MB (5% inode=99%): [17:34:02] Platonides * Re: [Toolserver-l] toolserver interwiki project (was: interwiki.py) [17:53:24] Platonides: thanks. it doesn't seem to work for user pages though [17:55:11] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:55:12] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:12] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:21] Environment on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:55:42] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [17:56:02] Environment on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [17:56:02] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [17:56:02] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [17:59:12] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 16.842 second response time [18:04:11] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:11:05] !replah [18:11:06] !replag [18:11:12] @replag [18:11:12] valhallasw: s2-pri: 1m 5s [-0.02 s/s]; s3-rr: 43s [+0.00 s/s]; s3-user: 43s [+0.00 s/s] [18:11:55] @replag [18:11:56] valhallasw: s2-pri: 1m 9s [+0.09 s/s] [18:13:11] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 19.393 second response time [18:16:31] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [18:16:31] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:17:31] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:18:12] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [18:22:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:32:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53727 MB (5% inode=99%): [18:45:11] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:16:32] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [19:16:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:17:32] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:19:11] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [19:22:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:24:32] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1939.000000 [19:32:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53429 MB (5% inode=99%): [19:45:11] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:49:32] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.647461/1.00, alarm hl:np_load_long=0.779297/1.50, alarm hl:mem_free=24523.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.647461/1.10, alarm hl:np_load_long=0.779297/1.75, alarm hl:mem_free=24523.000000M/300M [19:51:31] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [19:54:22] 3(created) [ACCAPP-442] Operate a bot and editor tools; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-442> (Stephen LaPorte) [20:08:32] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3621.000000 [20:08:51] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 40775 MB (4% inode=99%): [20:16:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:16:32] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [20:17:32] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:19:11] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [20:20:30] zzz =_= [20:22:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:32:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53035 MB (5% inode=99%): [20:35:02] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1855.000000 [20:45:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [20:48:22] 3(assigned) [ACCAPP-440] Wikimedia Hackathon <10https://jira.toolserver.org/browse/ACCAPP-440> (Daniel Kinzler) [20:55:10] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:55:11] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:55:11] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:55:21] Environment on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:55:42] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:56:02] Environment on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [20:56:02] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [20:56:02] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [21:06:22] 3(assigned) [ACCAPP-442] Operate a bot and editor tools <10https://jira.toolserver.org/browse/ACCAPP-442> (Daniel Kinzler) [21:08:23] 3(created) [ACCAPP-443] Wikimedia Hackathon; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-443> (Kurt Rose) [21:09:32] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6431.000000 [21:13:02] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1800.000000 [21:13:20] 3(commented) [ACCAPP-443] Wikimedia Hackathon <10https://jira.toolserver.org/browse/ACCAPP-443> (Kurt Rose) [21:16:32] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [21:16:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:17:31] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:18:22] 3(updated) [ACCAPP-440] Wikimedia Hackathon <10https://jira.toolserver.org/browse/ACCAPP-440> (Mahmoud Hashemi) [21:19:11] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [21:22:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:33:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52697 MB (5% inode=99%): [21:36:14] @replag [21:36:15] Daniel_WMDE: s2/s5-pri-c: 2h 5m 7s [+0.16 s/s]; s3-rr: 21s [-0.00 s/s]; s3-user: 21s [-0.00 s/s] [21:39:14] Daniel_WMDE: How is SF? [21:39:29] very busy, working on 5 things at once :) [21:40:00] http://www.mediawiki.org/wiki/San_Francisco_Hackathon_January_2012/Teams#Automatic_categorization is a bit like what we've been doing at Commons [21:40:28] Altough I think you've never implemented the coordinates based categorization right? [21:42:05] Daniel_WMDE: Crossbreading etherpad with MediaWiki would be cool thing to hack up ;-) [21:46:11] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [22:02:31] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.482422/1.75, alarm hl:np_load_avg=0.441406/2.00, alarm hl:mem_free=269.000000M/300M [22:03:32] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [22:09:32] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8376.000000 [22:10:01] @replag [22:10:01] DaBPunkt: s1-pri: 6m 23s [-0.00 s/s]; s1-sec: 12s [-0.01 s/s]; s1-sec-c: 3m 14s [+0.00 s/s]; s2/s5-pri-c: 2h 19m 37s [+0.43 s/s]; s3-rr: 1m 28s [+0.03 s/s]; s3-user: 1m 28s [+0.03 s/s]; s4-rr: 3m 14s [+0.00 s/s]; s4-user: 28s [-0.00 s/s] [22:16:31] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [22:16:31] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:18:32] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:19:12] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [22:23:20] 3(updated) [ACCAPP-442] Operate a bot and editor tools <10https://jira.toolserver.org/browse/ACCAPP-442> (Stephen LaPorte) [22:23:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:34:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52508 MB (5% inode=99%): [22:46:11] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [22:58:20] 3(created) [ERWIN-24] Protocol-relative URLs; erwin85's tools; Minor Bug <10https://jira.toolserver.org/browse/ERWIN-24> (Marco Aurelio) [23:00:56] @replag [23:00:56] DaBPunkt: s1-sec: 12s [-]; s1-sec-c: 22m 56s [+0.39 s/s]; s2/s5-pri-c: 2h 42m 20s [+0.45 s/s]; s4-rr: 22m 56s [+0.39 s/s]; s4-user: 9m 2s [+0.17 s/s] [23:01:49] nacht ts [23:09:31] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10179.000000 [23:17:31] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [23:17:31] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:19:01] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1838.000000 [23:19:12] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [23:19:41] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:23:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:34:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53294 MB (5% inode=99%): [23:46:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds