[00:00:41] <Alchimista>	 Yetanotherx: 403: User account expired on ts
[00:01:37] <Yetanotherx>	 Really.
[00:01:43] <Yetanotherx>	 I didn't realize.
[00:01:55] <Yetanotherx>	 I suppose I could have known through the dozens of emails and talk page notifications.
[00:02:02] <Yetanotherx>	 end sarcasm.
[00:02:35] <Joan>	 You read those e-mails, but not the one from the Toolserver about your account expiring?
[00:02:44] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 16.86, 14.57, 11.91  
[00:03:20] <Yetanotherx>	 I read the ones from the Toolserver.
[00:03:44] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 14.38, 14.31, 11.99  
[00:04:49] <Joan>	 And then let your account expire?
[00:07:33] <Alchimista>	 Yetanotherx: that happened to me too :P just fill a jira request for reactivation
[00:07:42] <Alchimista>	 without the tool, wp sucks
[00:08:13] <Alchimista>	 fortunelly, there is a clone: http://toolserver.org/~Tparis/pcount/index.php
[00:10:20] <Joan>	 http://toolserver.org/~tparis/pcount/index.php?name=%3C%21--&lang=en&wiki=wikipedia
[00:10:28] <Joan>	 That tool isn't sanitizing its output.
[00:10:35] <Joan>	 You should fix that.
[00:10:46] <Joan>	 Or TParis should, I guess.
[00:11:14] <Joan>	 Interestingly, </script> seems to be sanitized to <script>.
[00:11:44] <AMadman[busy]>	 Ack.
[00:13:46] <Joan>	 Heh, it just removes all /?
[00:15:08] <Joan>	 https://en.wikipedia.org/wiki/User_talk:TParis#XSS_vulnerability_in_one_of_your_tools
[00:19:24] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[00:19:33] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27939.000000  
[00:19:55] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[00:26:14] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[00:26:24] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[00:26:43] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[00:29:14] <tsnag>	 ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL  
[00:29:14] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[00:30:34] <tsnag>	 s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1910.000000  
[00:31:54] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[00:34:04] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[00:34:44] <tsnag>	 s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1902.000000  
[00:35:27] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[00:41:44] <tsnag>	 s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1627.000000  
[01:00:44] <tsnag>	 s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1710.000000  
[01:06:27] <msgbot>	 3(updated) [DRTRIGON-114] Support for named groups in regex <10https://jira.toolserver.org/browse/DRTRIGON-114>  (drtrigon)
[01:08:24] <msgbot>	 3(updated) [DRTRIGON-114] Support for named groups in regexs <10https://jira.toolserver.org/browse/DRTRIGON-114>  (drtrigon)
[01:13:24] <msgbot>	 3(closed) [DRTRIGON-116] Possiblity to supress error output to pages for subster <10https://jira.toolserver.org/browse/DRTRIGON-116>  (drtrigon)
[01:19:24] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[01:19:44] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31549.000000  
[01:19:53] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[01:26:14] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[01:26:24] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[01:26:44] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[01:29:14] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[01:29:14] <tsnag>	 ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL  
[01:32:04] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[01:34:04] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[01:36:24] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[02:19:34] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[02:19:44] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35153.000000  
[02:19:54] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[02:26:14] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[02:26:23] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[02:26:44] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[02:29:14] <tsnag>	 ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL  
[02:29:14] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[02:32:14] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[02:34:14] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[02:36:34] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[02:55:24] <msgbot>	 3(created) [ACCAPP-456] 04.04; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-456>  (J.Scipt )
[02:58:24] <msgbot>	 3(commented) [ACCAPP-456] 04.04 <10https://jira.toolserver.org/browse/ACCAPP-456>  (J.Scipt )
[03:02:54] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 16.24, 14.17, 12.86  
[03:05:02] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 14.64, 14.49, 13.16  
[03:17:29] <ceradon>	 Do I need to use my public or private keys?
[03:17:39] <AMadman>	 To log in?
[03:17:44] <AMadman>	 You'd use your private key.
[03:18:30] <ceradon>	 DaB asked me for my key when I requested a toolserver account which one will I need to use?
[03:19:12] <ceradon>	 https://jira.toolserver.org/browse/ACCAPP-453
[03:19:55] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[03:19:55] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38761.000000  
[03:19:55] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[03:21:32] <AMadman>	 Send him the public one.
[03:21:47] <AMadman>	 The premise of the private key is that you're the only one who has it. :p
[03:22:07] <AMadman>	 So the server will have your public key; you'll have your private key, they fit together, and that's how you authenticate.
[03:23:01] <ceradon>	 Did I close my request as unresolved?
[03:23:04] <ceradon>	 https://jira.toolserver.org/browse/ACCAPP-453
[03:23:11] <ceradon>	 resolved?
[03:23:18] <ceradon>	 *
[03:24:00] <AMadman>	 Looks like you did.
[03:24:14] <ceradon>	 ohh
[03:24:21] <ceradon>	 can I reopen it
[03:24:35] <AMadman>	 Yep. Just click Re-Open Issue (I think it may be in the Workflow drop-down).
[03:24:47] <AMadman>	 DaB should be the one to close it ultimately; he's the assignee.
[03:26:23] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[03:26:52] <ceradon>	 can't find it
[03:26:54] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[03:26:54] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[03:27:41] <ceradon>	 ^what is that bot doing?
[03:28:16] <AMadman>	 Reporting on issues with servers and services.
[03:28:56] <AMadman>	 tsnag both because nag is what it does and it's short for Nagios (a network monitor).
[03:29:10] <AMadman>	 And you may not be able to re-open it, depending on the queue; DaB might have to do it.
[03:29:23] <tsnag>	 ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL  
[03:29:23] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[03:33:03] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 16.64, 16.86, 15.20  
[03:33:13] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[03:34:23] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[03:36:54] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[03:39:02] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 14.37, 15.00, 14.85  
[04:19:58] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[04:20:49] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42421.000000  
[04:20:49] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[04:26:29] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[04:27:00] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[04:27:50] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[04:29:29] <tsnag>	 ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL  
[04:29:29] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[04:31:08] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 21.95, 16.84, 14.54  
[04:33:30] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[04:34:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[04:35:23] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 12.45, 14.58, 14.16  
[04:36:54] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[05:02:23] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 17.19, 15.79, 14.41  
[05:04:23] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 13.95, 14.81, 14.20  
[05:18:53] <tsnag>	 /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 7829 MB (0% inode=99%):  
[05:20:53] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[05:20:53] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[05:20:53] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46023.000000  
[05:26:33] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[05:27:23] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 15.69, 13.90, 13.22  
[05:27:53] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[05:27:53] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[05:28:23] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 14.05, 13.71, 13.20  
[05:29:33] <tsnag>	 ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL  
[05:29:33] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[05:32:23] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 13.68, 15.02, 13.91  
[05:33:33] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[05:35:33] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[05:36:54] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[05:43:29] <ceradon>	 are there any good perl programmers aroung?
[05:43:54] <ceradon>	 around*
[05:51:26] <msgbot>	 3(commented) [ACCAPP-453] I plan to use the toolserver for programming of my bot. <10https://jira.toolserver.org/browse/ACCAPP-453>  (Symon Klyde)
[06:02:23] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 21.23, 18.23, 15.27  
[06:14:23] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 13.85, 14.88, 14.89  
[06:20:54] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[06:20:54] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[06:20:54] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49623.000000  
[06:22:23] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 17.42, 15.69, 15.16  
[06:26:33] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[06:27:54] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[06:27:54] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[06:29:22] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 13.86, 14.85, 14.99  
[06:29:34] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[06:29:34] <tsnag>	 ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL  
[06:33:33] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[06:35:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[06:37:53] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[06:56:23] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 17.05, 16.04, 15.74  
[07:16:23] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 13.27, 13.82, 14.91  
[07:21:53] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[07:21:53] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[07:21:53] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53284.000000  
[07:26:34] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[07:27:53] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[07:27:54] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[07:28:23] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 16.24, 14.82, 14.56  
[07:29:34] <tsnag>	 ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL  
[07:29:34] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[07:33:34] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[07:35:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[07:37:53] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[07:50:33] <tsnag>	 /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 31322 MB (7% inode=99%):  
[07:52:33] <tsnag>	 /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23828 MB (5% inode=99%):  
[07:59:23] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 12.28, 13.25, 14.93  
[08:02:23] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 16.90, 15.89, 15.74  
[08:06:33] <tsnag>	 /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 90591 MB (22% inode=99%):  
[08:09:22] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 12.43, 14.04, 14.95  
[08:21:53] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[08:21:54] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56884.000000  
[08:21:54] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[08:26:33] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[08:27:54] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[08:27:54] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[08:29:43] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[08:29:43] <tsnag>	 ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL  
[08:33:33] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[08:35:43] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[08:38:04] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[08:55:10] <nosy>	 apmon: looks like we will be in the data center an hour later
[08:55:15] <nosy>	 sorry
[08:55:32] <nosy>	 @replag all
[08:55:32] <tsbot>	 nosy: s1-pri: 1s [-]; s1-sec: 1s [-]; s1-sec-c: 1s [-0.01 s/s]; s2-pri: 16h 21m 46s [+1.00 s/s]; s2/s5-pri-c: 1s [-0.08 s/s]; s3-rr: 26s [-0.00 s/s]; s3-user: 26s [-0.00 s/s]; s4-rr: 1s [-0.01 s/s]
[08:55:33] <tsbot>	 nosy: s4-user: 1s [-0.00 s/s]; s5-rr: 4s [-0.00 s/s]; s5-user: 4s [-0.01 s/s]; s6-rr: 0s [-0.00 s/s]; s6-user: 0s [-0.00 s/s]; s7-rr: 6s [-0.13 s/s]; s7-user: 6s [-0.13 s/s]
[09:03:23] <msgbot>	 3(commented) [MNT-1192] s3 and s7 slaves were stopped <10https://jira.toolserver.org/browse/MNT-1192>  (Marlen Caemmerer)
[09:03:36] <msgbot>	 3(reopened) [MNT-1192] s3 and s7 slaves were stopped <10https://jira.toolserver.org/browse/MNT-1192>  (Marlen Caemmerer)
[09:22:04] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[09:22:04] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57507.000000  
[09:22:04] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[09:25:25] <msgbot>	 3(commented) [MAGNUS-295] Image Siblings <10https://jira.toolserver.org/browse/MAGNUS-295>  (reza)
[09:26:43] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[09:28:03] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[09:28:13] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[09:29:43] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[09:29:43] <tsnag>	 ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL  
[09:32:23] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 17.16, 15.27, 14.16  
[09:33:33] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[09:34:23] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 14.00, 14.76, 14.11  
[09:35:53] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[09:38:03] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[09:42:23] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 14.98, 15.11, 14.49  
[10:10:04] <reba>	 José Emilio Mori Recio * Re: [Toolserver-l] What happened to pagecounts stats?
[10:12:23] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 17.49, 17.55, 16.46  
[10:22:04] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[10:22:04] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61107.000000  
[10:22:04] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[10:26:43] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[10:28:03] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[10:28:13] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[10:28:13] <DaB|DC>	 Hello all
[10:28:26] <DaB|DC>	 We starting today's maintaince now
[10:28:57] <jongleur>	 thanks for the work you do.
[10:28:57] <DaB|DC>	 apmon: are you stilll online?
[10:29:24] * jongleur  is working on a local copy for now during the maintenance work going on.

[10:29:53] <tsnag>	 ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL  
[10:29:53] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[10:30:32] <apmon>	 Yes, just about
[10:31:12] <DaB|DC>	 apmon: ok. We move ptolmy now
[10:31:22] <apmon>	 OK
[10:33:43] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[10:35:54] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[10:37:52] <tsnag>	 FMA on ptolemy is CRITICAL: ERROR - unexpected output from snmpwalk  
[10:38:02] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[10:38:13] <tsnag>	 NTP on ptolemy is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[10:38:23] <tsnag>	 SMF on ptolemy is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[10:38:33] <tsnag>	 SMTP on ptolemy is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[10:41:24] <msgbot>	 3(created) [TS-1295] Ortelius error light blinking; Toolserver; Minor Bug <10https://jira.toolserver.org/browse/TS-1295>  (Marlen Caemmerer)
[10:41:30] <msgbot>	 3(created) [TS-1296] ts-array3 is beeping - seems to be an error.; Toolserver; Bug <10https://jira.toolserver.org/browse/TS-1296>  (Marlen Caemmerer)
[10:41:32] <msgbot>	 3(assigned) [TS-1296] ts-array3 is beeping - seems to be an error. <10https://jira.toolserver.org/browse/TS-1296>  (Marlen Caemmerer)
[10:43:23] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 13.34, 14.32, 14.89  
[10:48:37] <wikirc_>	 [[User:DrTrigon]] ! 10https://wiki.toolserver.org/w/index.php?diff=6673&oldid=6491&rcid=8807 * DrTrigon * (+126) (+ filter.py)
[10:50:53] <tsnag>	 ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/17:UP:1 UP: OK  
[10:51:13] <tsnag>	 ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL  
[10:51:24] <msgbot>	 3(created) [MAGNUS-298] Commonshelper rtl; Magnus' tools; Support request <10https://jira.toolserver.org/browse/MAGNUS-298>  (reza)
[10:52:43] <tsnag>	 ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL  
[11:02:22] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 19.41, 17.14, 15.47  
[11:06:23] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 12.29, 14.93, 14.97  
[11:07:53] <tsnag>	 FMA on ptolemy is OK: OK  
[11:10:03] <tsnag>	 NTP on ptolemy is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.000799 secs  
[11:10:43] <tsnag>	 PostgreSQL on ptolemy is CRITICAL: CRITICAL - no connection to osm_mapnik (could not connect to server: Connection refused  
[11:16:21] <wikirc_>	 [[Special:Log/newusers]] create 10 * B.vd.velden *  (New user account)
[11:16:43] <tsnag>	 ethernet 0/1/19 on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/19:DOWN:1 UP: OK  
[11:20:33] <tsnag>	 PostgreSQL on ptolemy is OK: OK - database osm_mapnik (0 sec.)  
[11:21:43] <tsnag>	 ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL  
[11:22:03] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[11:22:04] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64707.000000  
[11:22:04] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[11:22:18] <nosy>	 @replag all
[11:22:19] <tsbot>	 nosy: s1-pri: 0s [-0.00 s/s]; s1-sec: 0s [-0.00 s/s]; s1-sec-c: 0s [-0.00 s/s]; s2-pri: 17h 58m 47s [+0.66 s/s]; s2/s5-pri-c: 0s [-0.00 s/s]; s3-rr: 50s [+0.00 s/s]; s3-user: 50s [+0.00 s/s]; s4-rr: 0s [-0.00 s/s]
[11:22:20] <tsbot>	 nosy: s4-user: 0s [-0.00 s/s]; s5-rr: 0s [-0.00 s/s]; s5-user: 0s [-0.00 s/s]; s6-rr: 1s [+0.00 s/s]; s6-user: 1s [+0.00 s/s]; s7-rr: 5s [-0.00 s/s]; s7-user: 5s [-0.00 s/s]
[11:22:23] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 15.09, 14.43, 14.26  
[11:22:29] <nosy>	 @replag all
[11:22:29] <tsbot>	 nosy: s1-pri: 1s [+0.09 s/s]; s1-sec: 1s [+0.09 s/s]; s1-sec-c: 1s [+0.09 s/s]; s2-pri: 17h 58m 58s [+1.04 s/s]; s2/s5-pri-c: 1s [+0.09 s/s]; s3-rr: 1m 1s [+1.04 s/s]; s3-user: 1m 1s [+1.04 s/s]; s4-rr: 1s [+0.09 s/s]
[11:22:30] <tsbot>	 nosy: s4-user: 1s [+0.09 s/s]; s5-rr: 2s [+0.19 s/s]; s5-user: 2s [+0.19 s/s]; s6-rr: 2s [+0.09 s/s]; s6-user: 2s [+0.09 s/s]; s7-rr: 2s [-0.28 s/s]; s7-user: 2s [-0.28 s/s]
[11:22:33] <wikirc_>	 [[Admin:Datacenter-Tasks]] ! 10https://wiki.toolserver.org/w/index.php?diff=6674&oldid=6672&rcid=8809 * 85.17.198.101 * (+7) ()
[11:22:39] <nosy>	 @replag all
[11:22:39] <tsbot>	 nosy: s1-pri: 1s [-]; s1-sec: 1s [-]; s1-sec-c: 1s [-]; s2-pri: 17h 59m 8s [+0.99 s/s]; s2/s5-pri-c: 1s [-]; s3-rr: 1m 11s [+0.99 s/s]; s3-user: 1m 11s [+0.99 s/s]; s4-rr: 1s [-]
[11:22:40] <tsbot>	 nosy: s4-user: 1s [-]; s5-rr: 5s [+0.30 s/s]; s5-user: 5s [+0.30 s/s]; s6-rr: 3s [+0.10 s/s]; s6-user: 3s [+0.10 s/s]; s7-rr: 3s [+0.10 s/s]; s7-user: 3s [+0.10 s/s]
[11:24:03] <tsnag>	 NTP on ptolemy is OK: NTP OK: Offset -0.001599 secs  
[11:24:57] <wikirc_>	 [[Admin:Datacenter-Tasks]] ! 10https://wiki.toolserver.org/w/index.php?diff=6675&oldid=6674&rcid=8810 * 85.17.198.101 * (+7) ()
[11:26:24] <msgbot>	 3(assigned) [MNT-1192] s3 and s7 slaves were stopped <10https://jira.toolserver.org/browse/MNT-1192>  (Marlen Caemmerer)
[11:26:31] <msgbot>	 3(commented) [MNT-1192] s3 and s7 slaves were stopped <10https://jira.toolserver.org/browse/MNT-1192>  (Marlen Caemmerer)
[11:26:43] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[11:28:03] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[11:28:13] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[11:29:53] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[11:29:53] <|Danny_B|>	 hmm, what's happening to nightshade now?
[11:30:08] <|Danny_B|>	 i can't ssh to it
[11:31:05] <wikirc_>	 [[Scs-oe10]]  10https://wiki.toolserver.org/w/index.php?diff=6676&oldid=6560&rcid=8811 * Nosy * (+30) ()
[11:32:57] <wikirc_>	 [[Scs-oe10]]  10https://wiki.toolserver.org/w/index.php?diff=6677&oldid=6676&rcid=8812 * Nosy * (+23) ()
[11:33:44] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[11:34:48] <|Danny_B|>	 ah, obviously there is an issue
[11:35:13] <|Danny_B|>	 nosy: around? pls see ^^
[11:36:53] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[11:37:32] <tsnag>	 SMTP on ptolemy is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[11:38:03] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[11:38:33] <tsnag>	 SMTP on ptolemy is OK: SMTP OK - 3.437 sec. response time  
[11:38:53] <tsnag>	 SMF on ptolemy is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[11:39:12] <wikirc_>	 [[Scs-oe10]]  10https://wiki.toolserver.org/w/index.php?diff=6678&oldid=6677&rcid=8813 * Nosy * (+982) ()
[11:40:36] <nosy>	 Danny_B: they are repairing the hardware
[11:40:38] <Merlissimo>	 |Danny_B|: have you read mails or topic?
[11:41:44] <Quentinv57>	 |Danny_B|, I guess there are some maintenace done now
[11:42:43] <tsnag>	 ethernet 0/1/2 [nightshade] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/2:DOWN: 1 int NOK : CRITICAL  
[11:42:52] <tsnag>	 ethernet 0/1/13 [nightshade.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/13:DOWN: 1 int NOK : CRITICAL  
[11:43:27] <msgbot>	 3(created) [TS-1297] Adjust status.toolserver.org; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1297>  (Marlen Caemmerer)
[11:44:32] <tsnag>	 /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 39735 MB (4% inode=99%):  
[11:45:55] <|Danny_B|>	 well, some console warning would be nice anyway, so one could end his programs correctly
[11:46:23] <|Danny_B|>	 hardware maintenance *today* is not much timewise descriptive :-(
[11:47:53] <tsnag>	 ethernet 0/1/13 [nightshade.mgmt] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/13:UP:1 UP: OK  
[11:48:27] <Merlissimo>	 |Danny_B|: sge will move your jobs automatically to willow
[11:48:44] <tsnag>	 ethernet 0/1/2 [nightshade] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/2:UP:1 UP: OK  
[11:51:13] <tsnag>	 ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL  
[11:51:42] <wikirc_>	 [[Admin:Datacenter-Tasks]] ! 10https://wiki.toolserver.org/w/index.php?diff=6679&oldid=6675&rcid=8814 * 85.17.198.101 * (+7) ()
[11:51:43] <tsnag>	 SSH on nightshade.mgmt is OK: SSH OK - OpenSSH_3.9p1 (protocol 2.0)  
[11:56:33] <nosy>	 turnera will get a maintenance next as it seems
[11:56:33] <nosy>	 will most likely be offline 
[12:10:07] <DarkoNeko>	 zzz =_=
[12:16:50] <mmovchin>	 hello
[12:17:05] <mmovchin>	 just a question: Toolserver doesn't support Ruby on Rails, right?
[12:20:32] <Merlissimo>	 nosy: there is no /etc/nologin on willow (as i would expect after shutdown+delay)
[12:21:41] <nosy>	 Merlissimo: willow was not rebooted
[12:21:52] <tsnag>	 ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL  
[12:22:13] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[12:22:14] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53577.000000  
[12:22:14] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[12:22:14] <Merlissimo>	 nosy: "The system willow will be shut down in 1 minute"
[12:22:14] <nosy>	 but someone is doing it now
[12:22:48] <Merlissimo>	 now there is the nologin there
[12:26:53] <tsnag>	 ethernet 0/1/1 [willow] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/1:DOWN: 1 int NOK : CRITICAL  
[12:26:53] <tsnag>	 ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:DOWN: 1 int NOK : CRITICAL  
[12:28:14] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[12:28:14] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[12:29:53] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[12:33:53] <tsnag>	 ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:DOWN: 1 int NOK : CRITICAL  
[12:33:53] <tsnag>	 ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/16:DOWN: 1 int NOK : CRITICAL  
[12:33:54] <tsnag>	 ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/8:DOWN: 1 int NOK : CRITICAL  
[12:35:53] <tsnag>	 ethernet 0/1/1 [willow] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/1:UP:1 UP: OK  
[12:35:53] <tsnag>	 ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/14:UP:1 UP: OK  
[12:36:53] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[12:38:13] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[12:38:53] <tsnag>	 ethernet 0/1/2 [nightshade] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/2:DOWN: 1 int NOK : CRITICAL  
[12:38:53] <tsnag>	 ethernet 0/1/13 [nightshade.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/13:DOWN: 1 int NOK : CRITICAL  
[12:38:54] <tsnag>	 SMF on ptolemy is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[12:39:53] <tsnag>	 Sun Grid Engine execd on willow is CRITICAL: all.q@willow in unknown state: longrun@willow in unknown state  
[12:39:53] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[12:51:13] <tsnag>	 ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL  
[12:51:54] <tsnag>	 ethernet 0/1/2 [nightshade] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/2:UP:1 UP: OK  
[12:51:54] <tsnag>	 ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/16:UP:1 UP: OK  
[12:52:04] <jem->	 nosy: I guess the jobs queues aren't still ready...?
[12:53:53] <tsnag>	 ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/18:UP:1 UP: OK  
[12:53:54] <tsnag>	 ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/8:UP:1 UP: OK  
[12:53:54] <tsnag>	 ethernet 0/1/13 [nightshade.mgmt] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/13:UP:1 UP: OK  
[12:54:34] <wikirc_>	 [[Scs-oe10]]  10https://wiki.toolserver.org/w/index.php?diff=6680&oldid=6678&rcid=8815 * Nosy * (+0) (/* Management via Console */ )
[12:55:28] <Merlissimo>	 jem-: those on willow and nightshade are down. so there is no longrun queue, but job running thess than 1 week can run on other servers
[12:57:53] <Merlissimo>	 jem-: should i delete your job? (you have deletet it, so its not requeued, and deletion process is pending)
[13:01:48] <tsnag>	 Cluster on turnera.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[13:02:08] <tsnag>	 SMTP on turnera.esi is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[13:02:18] <tsnag>	 Cluster on damiana.esi is CRITICAL: turnera FAILED, damiana:nge1-turnera:nge1 faulted, damiana:nge0-turnera:nge0 faulted, vote turnera Offline, check nfs-hasp, nfs-home Online,  
[13:02:19] <tsnag>	 NTP on turnera.esi is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[13:02:20] <tsnag>	 SSH on turnera.esi is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[13:02:29] <tsnag>	 Free Memory on turnera.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[13:02:29] <tsnag>	 ts-array5 on turnera.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[13:02:29] <tsnag>	 DiskSuite on turnera.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[13:02:29] <tsnag>	 PING on turnera.esi is CRITICAL: CRITICAL - Plugin timed out after 10 seconds  
[13:02:39] <tsnag>	 /tmp on turnera.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[13:02:49] <tsnag>	 Environment on turnera.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[13:03:59] <tsnag>	 Free Memory on damiana.esi is CRITICAL: CRITICAL - 1.9% (80252 kB) free!  
[13:13:53] <jem->	 Merlissimo: thanks :), although currently qstat is not even responding
[13:14:18] <jem->	 I guess we'll have to wait
[13:15:13] <Merlissimo>	 jem-: there is one job on willow you tried to delete and one job running on ortelius (vardata)
[13:15:50] <jem->	 Ok, you can kill both of them, Merlissimo 
[13:15:59] <jem->	 (Well, delete) :)
[13:16:27] <Merlissimo>	 the job on ortelius is running normally, but ofcourse homedir is gone
[13:16:47] <jem->	 Then it is no use
[13:17:50] <tsnag>	 Environment on turnera.esi is OK: ok:  temperature ok fan ok voltage ok chassis ok  
[13:18:07] <tsnag>	 NTP on turnera.esi is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.007014 secs  
[13:18:08] <jem->	 Because my jobs need the configuration files of the homedir
[13:18:39] <jem->	 "NFS server ha-nfs.esi ok" ... back to normal?
[13:19:44] <jem->	 Nope :(
[13:22:10] <tsnag>	 ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL  
[13:22:20] <tsnag>	 MySQL on ha-sql.esi is CRITICAL: Cant connect to MySQL server on ha-sql.esi (146)  
[13:22:50] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[13:23:10] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[13:23:10] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57240.000000  
[13:23:39] <tsnag>	 Cluster on damiana.esi is OK: CLUSTER OK !  
[13:25:21] <tsnag>	 MySQL on ha-sql.esi is OK: Uptime: 7  Threads: 2  Questions: 60  Slow queries: 0  Opens: 20  Flush tables: 1  Open tables: 13  Queries per second avg: 8.571  
[13:26:12] <tsnag>	 Cluster on turnera.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp,  
[13:26:40] <tsnag>	 Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp,  
[13:28:31] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[13:29:12] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default offline:  svc:/system/cluster/scsymon-srv:default  
[13:29:40] <tsnag>	 NTP on ptolemy is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown  
[13:30:30] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[13:31:19] <wikirc_>	 [[Admin:Datacenter-Tasks]]  10https://wiki.toolserver.org/w/index.php?diff=6681&oldid=6679&rcid=8816 * DaB * (+7) ()
[13:32:21] <tsnag>	 NTP on turnera.esi is OK: NTP OK: Offset -0.004044 secs  
[13:35:00] <tsnag>	 SMF on ptolemy is OK: OK - all services online  
[13:36:12] <tsnag>	 SMF on willow is OK: OK - all services online  
[13:36:21] <tsnag>	 Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK  
[13:36:53] <Quentinv57>	 nosy, hello, TS is still down ?
[13:37:11] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[13:37:13] <nosy>	 Quentinv57: nightshade is still installing
[13:37:21] <tsnag>	 SMTP on ptolemy is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[13:37:23] <nosy>	 Merlissimo:
[13:37:37] <nosy>	 some error messages arrived
[13:37:51] <nosy>	 i had turned on the sge service on willow again
[13:37:52] <Quentinv57>	 nosy, if I need to code some tools, may I use ortelius to submit my files ?
[13:38:01] <nosy>	 Quentinv57: thats fine
[13:38:06] <Quentinv57>	 thanks :)
[13:38:07] <nosy>	 we will not touch tat
[13:38:12] <tsnag>	 SMTP on ptolemy is OK: SMTP OK - 1.233 sec. response time  
[13:38:12] <nosy>	 that
[13:39:11] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[13:40:05] <Merlissimo>	 nosy: ?
[13:40:24] <nosy>	 Merlissimo: the sge service was in maintenance state via svc
[13:40:34] <nosy>	 i reenabled it to be sure everything is running
[13:40:39] <nosy>	 was that a mistake?
[13:40:40] <tsnag>	 NTP on ptolemy is OK: NTP OK: Offset -0.007125 secs  
[13:41:04] <Merlissimo>	 nosy: nsi = ldap error
[13:41:52] <nosy>	 Merlissimo: that is where those errors come from?
[13:42:18] <nosy>	 i got heaps of mail from failed jobs on willow
[13:42:21] <Merlissimo>	 yes, login failied because of password check, because ldap was not reachable
[13:42:40] <nosy>	 ah ok fine
[13:42:45] <Merlissimo>	 4 job failed, 44 were sucessful
[13:42:57] <Merlissimo>	 i cleared error state of thoese jobs
[13:43:14] <nosy>	 Shepherd pe_hostfile:
[13:43:15] <nosy>	 willow.toolserver.org 1 longrun@willow.toolserver.org UNDEFINED
[13:43:20] <nosy>	 thats the end of the mails
[13:43:22] <Merlissimo>	 on nightshade execd isn't still running
[13:44:07] <nosy>	 Merlissimo: does this relate?
[13:44:12] <Merlissimo>	 no
[13:44:17] <nosy>	 its about 42 error mails from willow sge
[13:44:35] <nosy>	 nightshade will take a while to recover
[13:44:46] <nosy>	 we simply exchanged the drac management card
[13:45:14] <nosy>	 probably it only reinstalled because he drac was set to tftp boot 
[13:45:26] <nosy>	 but now we have to go through this
[13:45:40] <tsnag>	 Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp,  
[13:45:41] <Merlissimo>	 the mass of the mail were the old jobs that were kill during shutdown and shepared cleared them
[13:46:00] <tsnag>	 Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp,  
[13:46:31] <nosy>	 good. this explains also the different job numbers
[13:49:40] <nosy>	 Merlissimo: so there seems to be no more error that i do not know of
[13:50:48] <Merlissimo>	 sge is completly usable - only two queue instances missing on nightshade, but this doen't hurt
[13:51:50] <tsnag>	 ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL  
[13:55:31] <tsnag>	 SMF on hemlock is CRITICAL: ERROR - maintenance:  svc:/network/dhcp-server:default  
[13:56:21] <msgbot>	 3(created) [TS-1298] Reenable DHCPD on hemlock; Toolserver; Minor Task <10https://jira.toolserver.org/browse/TS-1298>  (Marlen Caemmerer)
[14:00:32] <tsnag>	 ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:DOWN: 1 int NOK : CRITICAL  
[14:17:21] <tsnag>	 Load avg. on daphne is WARNING: WARNING - load average: 16.49, 11.19, 6.62  
[14:18:31] <tsnag>	 ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/18:UP:1 UP: OK  
[14:19:43] <Merlissimo>	 @replag
[14:19:43] <tsbot>	 Merlissimo: s2-pri: 16h 50m 35s [+1.00 s/s]; s3-rr: 1m 14s [+0.21 s/s]; s3-user: 1m 14s [+0.21 s/s]
[14:20:22] <tsnag>	 Load avg. on daphne is OK: OK - load average: 8.00, 11.01, 7.47  
[14:22:23] <tsnag>	 ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL  
[14:22:51] <tsnag>	 MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 60281  
[14:23:23] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60170.000000  
[14:23:23] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[14:25:55] <Quentinv57>	 Merlissimo, s2 is definitely out :D
[14:28:41] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[14:29:23] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[14:30:41] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[14:38:10] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[14:39:23] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[14:51:21] <msgbot>	 3(updated) [ACCAPP-449] Account Approval for the toolserver withing the RENDER project <10https://jira.toolserver.org/browse/ACCAPP-449>  (Gerrit Holz)
[14:51:51] <tsnag>	 ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL  
[14:55:41] <tsnag>	 SMF on hemlock is CRITICAL: ERROR - maintenance:  svc:/network/dhcp-server:default  
[15:04:50] <apmon>	 Is the maintenance still ongoing?
[15:22:34] <mmovchin>	  just a question: Toolserver doesn't support Ruby on Rails, right?
[15:22:52] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[15:23:21] <tsnag>	 ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL  
[15:23:31] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[15:24:20] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47491.000000  
[15:28:42] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[15:30:21] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[15:30:42] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[15:34:20] <apmon>	 does anyone know when nosy or dab will be back?
[15:38:10] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[15:39:31] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[15:52:00] <tsnag>	 ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL  
[15:55:41] <tsnag>	 SMF on hemlock is CRITICAL: ERROR - maintenance:  svc:/network/dhcp-server:default  
[16:13:21] <msgbot>	 3(closed) [DRTRIGON-106] Translations (TW) for subster.py <10https://jira.toolserver.org/browse/DRTRIGON-106>  (drtrigon)
[16:23:01] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[16:23:32] <tsnag>	 ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL  
[16:23:32] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[16:24:32] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51096.000000  
[16:29:41] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[16:30:32] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[16:30:42] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[16:35:51] <wikirc_>	 [[Special:Log/newusers]] create 10 * Davy-62 *  (New user account)
[16:39:10] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[16:39:31] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[16:52:00] <tsnag>	 ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL  
[16:55:52] <tsnag>	 SMF on hemlock is CRITICAL: ERROR - maintenance:  svc:/network/dhcp-server:default  
[17:19:41] <tsnag>	 /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 13629 MB (1% inode=99%):  
[17:23:01] <tsnag>	 MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds)  
[17:23:32] <tsnag>	 ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL  
[17:23:41] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[17:24:32] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54699.000000  
[17:29:52] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[17:30:32] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[17:30:51] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[17:39:10] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[17:39:41] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[17:52:01] <tsnag>	 ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL  
[17:55:51] <tsnag>	 SMF on hemlock is CRITICAL: ERROR - maintenance:  svc:/network/dhcp-server:default  
[18:23:13] <tsnag>	 MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57914  
[18:24:31] <tsnag>	 ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL  
[18:24:42] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[18:25:31] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56672.000000  
[18:30:00] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[18:30:31] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[18:31:00] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[18:39:21] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[18:40:41] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[18:52:11] <tsnag>	 ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL  
[18:55:51] <tsnag>	 SMF on hemlock is CRITICAL: ERROR - maintenance:  svc:/network/dhcp-server:default  
[19:24:12] <tsnag>	 MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18155  
[19:24:41] <tsnag>	 ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL  
[19:24:41] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[19:26:31] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17253.000000  
[19:30:01] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[19:30:41] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[19:31:01] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[19:39:20] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[19:40:40] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[19:52:12] <tsnag>	 ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL  
[19:56:01] <tsnag>	 SMF on hemlock is CRITICAL: ERROR - maintenance:  svc:/network/dhcp-server:default  
[20:05:21] <wikirc_>	 [[Special:Log/newusers]] create 10 * Savadkoohi *  (New user account)
[20:08:10] <tsnag>	 MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3296  
[20:08:40] <tsnag>	 s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2954.000000  
[20:10:41] <tsnag>	 s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1798.000000  
[20:11:10] <tsnag>	 MySQL slave on daphne is OK: Uptime: 96908  Threads: 11  Questions: 83100112  Slow queries: 11248  Opens: 558175  Flush tables: 1  Open tables: 10745  Queries per second avg: 857.515 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1419  
[20:24:40] <tsnag>	 ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL  
[20:24:40] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[20:25:30] <tsnag>	 MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1935  
[20:25:51] <tsnag>	 s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1950.000000  
[20:27:30] <tsnag>	 MySQL slave on thyme is OK: Uptime: 107867  Threads: 2  Questions: 55244030  Slow queries: 14639  Opens: 6650  Flush tables: 1  Open tables: 2418  Queries per second avg: 512.149 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1750  
[20:27:50] <tsnag>	 s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1713.000000  
[20:30:01] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[20:30:41] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[20:31:01] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[20:35:56] <Gnumarcoo>	 any problem with SSH login?
[20:38:26] <AMadman>	 There was for me earlier.
[20:38:28] * AMadman  checks.

[20:38:53] <AMadman>	 Oh. Wait. It says in the topic hardware-maintenance today, status down. :p
[20:38:59] <AMadman>	 There was an e-mail about it on the listserv.
[20:39:22] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[20:40:41] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[20:47:43] <apmon>	 I am still having problems to get back into ptolemy. The home directories aren't mounted anymore
[20:48:01] <AMadman>	 You can assume everything's down.
[20:48:07] <AMadman>	 There's no point in trying to fool with things.
[20:49:17] <DaBPunkt>	 re
[20:49:53] <apmon>	 Hi DaBPunkt
[20:50:10] <AMadman>	 Hey DaB.
[20:51:34] <apmon>	 DaBPunkt: Are things supposed to be back to normal, or is the maintenance still going on?
[20:52:12] <DaBPunkt>	 apmon: nightshade is down and there are some minor open chnages, but the hardware-maintaince is done, yes
[20:52:22] <tsnag>	 ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL  
[20:52:38] <apmon>	 DaBPunkt: Ptolemy can't mount the home directories
[20:53:29] <apmon>	 And possibly because of that apache appears to not be working
[20:53:41] <DaBPunkt>	 let me take a look
[20:53:46] <apmon>	 thanks
[20:56:02] <tsnag>	 SMF on hemlock is CRITICAL: ERROR - maintenance:  svc:/network/dhcp-server:default  
[20:56:04] <DaBPunkt>	 apmon: mounted manualy
[20:58:47] <apmon>	 still having troubles sshing into ptolemy. It hangs on the attempt to connect
[21:03:13] <DaBPunkt>	 apmon: from where do you try to login?
[21:08:58] <DaBPunkt>	 apmon: ping
[21:15:20] <msgbot>	 3(created) [MNT-1194] Added ptolemy's new MGMT-IP-adress into DNS; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1194>  (DaB.)
[21:15:23] <msgbot>	 3(resolved) [MNT-1194] Added ptolemy's new MGMT-IP-adress into DNS <10https://jira.toolserver.org/browse/MNT-1194>  (DaB.)
[21:20:32] <apmon>	 DaBPunkt: I try to log on from willow
[21:21:25] <DaBPunkt>	 apmon: →query
[21:21:39] <apmon>	 DaBPunkt: After a very long time, ssh succeeds, but the home directory is still not visible
[21:21:54] <apmon>	 Presumably the long time is the timeout to trying to read the home directory
[21:23:21] <apmon>	 DaBPunkt: Sorry, what do you mean by "→query"?
[21:23:43] <DaBPunkt>	 apmon: that I have question for you in a query
[21:24:03] <DaBPunkt>	 there should be a open tab in your client
[21:24:24] <DaBPunkt>	 IRC-client
[21:24:38] <apmon>	 sorry, don't see the extra client window
[21:24:41] <tsnag>	 ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL  
[21:24:41] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[21:30:01] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[21:30:41] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[21:31:02] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[21:39:22] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[21:40:51] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[21:41:20] <msgbot>	 3(created) [MNT-1195] ptolemy's mounts the wrong /home; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1195>  (DaB.)
[21:43:20] <msgbot>	 3(resolved) [MNT-1195] ptolemy's mounts the wrong /home <10https://jira.toolserver.org/browse/MNT-1195>  (DaB.)
[21:43:48] <Platonides>	 DaBPunkt, what is expected to work?
[21:43:57] <Platonides>	 you mention above that nightshade is down
[21:44:17] <Platonides>	 login still points to nightshade
[21:44:22] <Platonides>	 which refuses the connection
[21:44:23] <DaBPunkt>	 yes, looks like I disconnected the disc or the controller by accident
[21:44:26] <Platonides>	 willow asks password...
[21:44:45] <DaBPunkt>	 willow ask for a pasword?
[21:44:46] <Platonides>	 I could try to login to the other servers, but I don't think I should
[21:44:50] <Platonides>	 yes
[21:45:10] <DaBPunkt>	 mm, works here
[21:45:20] <msgbot>	 3(created) [MNT-1196] ptolemy is not longer accessable by ipv6; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1196>  (DaB.)
[21:46:03] <Platonides>	 not here...
[21:46:15] <Platonides>	 ssh -v willow.toolserver.org / debug1: Offering RSA public key: MediaWiki / debug1: Authentications that can continue: publickey,password,keyboard-interactive,hostbased / debug1: Next authentication method: keyboard-interactive
[21:46:23] <ceradon>	 DaBPunkt, Ping.
[21:46:31] <DaBPunkt>	 ceradon: yes?
[21:46:48] <ceradon>	 I PM'd you, dont you see it?
[21:46:49] <Merlissimo>	 login on willow works for me
[21:46:57] <DaBPunkt>	 Platonides: there are several users logined AFAIS. Did it work today or yesterday for you?
[21:47:23] <Platonides>	 I think I didn't try yesterday
[21:47:25] <DaBPunkt>	 ceradon: I saw it, but I can do only I thing at the same tzime (I am a old, serial modell)
[21:47:32] <Platonides>	 I can happily login into wolfsbane or ortelius, though
[21:47:50] <DaBPunkt>	 Platonides: mm, ok. THat's strange then
[21:49:49] <Platonides>	 nevermind
[21:50:03] <Platonides>	 it wasn't using the right username
[21:50:26] <Platonides>	 no config entry at .ssh/config for willow
[21:50:47] <DaBPunkt>	 ah Ok :-)
[21:51:07] <Platonides>	 sorry for the noise
[21:51:46] <DaBPunkt>	 Platonides: no problem. It is solved that easily everything, it would be great :-)
[21:52:02] <DaBPunkt>	 everything → everytime
[21:53:03] <Platonides>	 hehe
[21:53:21] <tsnag>	 ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL  
[21:54:00] <Vito>	 hello guys, some bot on the ts are not working
[21:54:54] <Vito>	 *s
[21:55:17] <Vito>	 oh status:down
[21:55:20] <msgbot>	 3(created) [MNT-1197] login.toolserver.org should point to willow; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1197>  (DaB.)
[21:55:25] <Platonides>	 Vito, there was a maintenance today
[21:55:33] <Vito>	 I see Platonides, sorry :D
[21:55:38] <Platonides>	 you should poke the maintainers to restart them
[21:56:08] <ToAruShiroiNeko>	 o hai DaBPunkt
[21:56:12] <tsnag>	 SMF on hemlock is CRITICAL: ERROR - maintenance:  svc:/network/dhcp-server:default  
[21:56:14] <ToAruShiroiNeko>	 you available?
[21:56:16] <Merlissimo>	 Vito: depends on the toolserver user. if he/she is using submit.toolserver.org for cron and sge as suggested all is fine.
[21:56:22] <ToAruShiroiNeko>	 I'd like you to run  those queries a bit
[21:56:45] <Vito>	 Platonides: I see
[21:57:05] <Vito>	 Merlissimo: it's our welcoming bot, filnik's account I think
[21:57:08] <Vito>	 lemme see
[21:57:20] <msgbot>	 3(resolved) [MNT-1197] login.toolserver.org should point to willow <10https://jira.toolserver.org/browse/MNT-1197>  (DaB.)
[21:58:39] <DaBPunkt>	 ToAruShiroiNeko: in a moment
[22:02:55] <Merlissimo>	 job "welcomewp" only run at Jan  5th on sge
[22:09:16] <DaBPunkt>	 ToAruShiroiNeko: ok, the query runs now. Talk to be back tomorrow evening
[22:12:21] <msgbot>	 3(created) [MNT-1198] nightshade down because of missing disc; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1198>  (DaB.)
[22:13:20] <ToAruShiroiNeko>	 DaBPunkt ok whats the query going to retrieve?
[22:13:23] <wikirc_>	 [[Admin:Datacenter-Tasks]]  10https://wiki.toolserver.org/w/index.php?diff=6682&oldid=6681&rcid=8819 * Dab * (+47) ()
[22:14:23] <DaBPunkt>	 ToAruShiroiNeko: you told me that you need a update of http://toolserver.org/~dab/queries/ToAruShiroiNeko.2.txt, didn't you?
[22:14:49] <ToAruShiroiNeko>	 if possible I want to have file breakdowns of deleted and current files seperately
[22:14:57] <ToAruShiroiNeko>	 would that make such a distinciton?
[22:15:13] <ToAruShiroiNeko>	 what fraction of current/deleted images are jpgs is a relevant question
[22:19:09] <ToAruShiroiNeko>	 also I would want the break down of something similar for en.wikipedia articles if possible
[22:19:23] <ToAruShiroiNeko>	 would this be doable?
[22:20:04] <Jyothis>	 DaBPunkt: is nightshade down?
[22:20:10] <DaBPunkt>	 yes
[22:20:29] <Jyothis>	 Ok, thanks. 
[22:24:41] <tsnag>	 ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL  
[22:24:50] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[22:27:18] <wikirc_>	 [[Admin:Service processors]]  10https://wiki.toolserver.org/w/index.php?diff=6683&oldid=6671&rcid=8820 * Dab * (+335) (+damiana and turnera)
[22:27:59] <DaBPunkt>	 ToAruShiroiNeko: yes, but not tonight
[22:30:21] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[22:30:51] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[22:31:11] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[22:31:55] <ToAruShiroiNeko>	 DaBPunkt ok
[22:31:57] <ToAruShiroiNeko>	 thats fine too
[22:34:38] <|Danny_B|>	 DaBPunkt: any eta for having nightshade up?
[22:34:48] <DaBPunkt>	 few weeks
[22:35:24] <Saibo>	 umpf..  what? :D
[22:35:48] <|Danny_B|>	 huh
[22:36:12] <Saibo>	 Rotatebot runs on nightshade (IIRC) - that means it is down for weeks now?
[22:36:31] <|Danny_B|>	 so should i use willow now for all my screens?
[22:38:34] <DaBPunkt>	 yes. Or convert to SGE.
[22:39:25] <Saibo>	 DaBPunkt: was that at danny or at me?
[22:39:52] * |Danny_B|  needs to lookup about sge

[22:40:21] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[22:40:50] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[22:41:39] <DaBPunkt>	 Saibo: |Danny_B|
[22:42:21] <msgbot>	 3(created) [MNT-1199] Configured damiana's and tunera's MGMT-IPs and put them into DNS; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1199>  (DaB.)
[22:47:28] <|Danny_B|>	 i'm confused by the documentation :-/
[22:47:49] <|Danny_B|>	 DaBPunkt: will you be around when wikimedia conference will take a place?
[22:48:15] <DaBPunkt>	 whcih conference?
[22:48:35] <|Danny_B|>	 chapters
[22:49:21] <DaBPunkt>	 |Danny_B|: I am not in the board of any chapters, so no
[22:50:08] <Saibo>	 http://status.toolserver.org/ shows "Internal Server Error"  but  topic here says "up" and Rotatebot is not running - DaBPunkt, could you give me a hint? ;-)  I just want to make some kind of announcement how long it takes until normal operation.
[22:53:36] <DaBPunkt>	 Saibo: status.toolserver.org is still hosted by River, so I have no controll about that server. And I have no idea why a specific bot is not running, ask the ts-user who runs it
[22:53:50] <Saibo>	 meh.. 
[22:54:03] <Saibo>	 yes.. fine
[22:54:12] <Saibo>	 crappy information politics ;)
[22:54:21] <tsnag>	 ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL  
[22:54:35] <Saibo>	 thank you anyway - not your fault probably :D
[22:56:20] <|Danny_B|>	 DaBPunkt: i know you are not, i was just wondering if you would be around helping out or just simply visiting...
[22:56:50] <DaBPunkt>	 |Danny_B|: ah ok. No, sorry :-)
[22:57:12] <tsnag>	 SMF on hemlock is CRITICAL: ERROR - maintenance:  svc:/network/dhcp-server:default  
[23:02:18] <wikirc_>	 [[Special:Log/newusers]] create 10 * Mbalsalama *  (New user account)
[23:24:52] <tsnag>	 ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL  
[23:24:52] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[23:30:22] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[23:30:51] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[23:31:11] <tsnag>	 ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL  
[23:39:45] <wikirc_>	 [[Special:Log/newusers]] create 10 * Ignazio *  (New user account)
[23:40:21] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[23:40:51] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[23:45:31] <tsnag>	 /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 39256 MB (4% inode=99%):  
[23:47:26] <DaBPunkt>	 @replag
[23:47:26] <tsbot>	 DaBPunkt: s3-rr: 34s [+0.00 s/s]; s3-user: 34s [+0.00 s/s]; s6-rr: 31s [-]; s6-user: 31s [-]
[23:54:21] <tsnag>	 ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL  
[23:57:21] <tsnag>	 SMF on hemlock is CRITICAL: ERROR - maintenance:  svc:/network/dhcp-server:default