[00:00:20] http://kpaste.net/ee3eb1 [00:01:11] Merlissimo: Ah, neat. :-) [00:01:40] This file is confusing. [00:02:11] I think you've got weird line breaks in there. [00:02:27] Load avg. on willow is WARNING: WARNING - load average: 19.80, 18.03, 14.76 [00:02:44] Now I'm doubting myself about cron syntax. I think each command has to be a single line. [00:02:46] I saw that. I removed them and resubmitted it and still get the error. [00:02:56] A comment, a command to cron, or a blank line. [00:02:58] Fairly sure. [00:03:10] cyberpower678: Paste the new .cronie file. :-) [00:03:17] And new error, if any. [00:03:33] Help me help you. &c. [00:05:25] http://kpaste.net/5d70 [00:05:27] Same error [00:06:54] I removed more of the linebreaks and executed the command again. [00:07:09] cyberpower678@willow:~$ cronie /home/cyberpower678/.crontab [00:07:13] Result was: [00:07:22] cyberpower678@willow:~$ [00:08:33] cyberpower678: gratulations. You made it :-) [00:08:59] I just verified it by executing cronie -l and it's displaying my cron. Yay. [00:09:14] I verified too before .) [00:09:44] Joan: thanks for your endurance [00:10:11] DaBPunkt: Danke DaP. Thank you Joan and Merlissimo. [00:13:05] Ok. So now I am getting this: /opt/local/bin/cronsub[51]: /home/cyberpower678/bots/chu.php^M: cannot open [00:13:30] This is coming from Cron Daemon [00:14:15] HELP!!!! [00:14:50] cyberpower678: is the php file executable? [00:15:18] What do you mean? It's readable. [00:15:29] Notepad can read it. [00:16:11] it's .php [00:16:16] What's the ^M? [00:16:26] a wrong line-break [00:16:36] That would be the problem, right? [00:16:42] cyberpower678: how long to you expect that these scripts run? [00:16:53] I don't know. It's definetely not in the cron. [00:17:01] The scripts run forever. [00:17:15] the first char of the file is a unicode-detect-chat (feff). i guess that is the problem [00:17:51] cyberpower678: Do you have Notepad++? [00:18:05] Use that to edit your files on Windows. [00:18:06] Just Notepad [00:18:27] Load avg. on willow is OK: OK - load average: 13.83, 14.79, 14.93 [00:19:48] cyberpower678: Right. So you open the file in Notepad++ and convert to a file without a BOM. [00:20:04] Just hilight the contents of the file and use the menu. [00:20:17] Merlissimo: by the way, the qcronie that you've sended me other day is giving a problem: [00:20:26] Unable to initialize environment because of error: Please set the environment variable SGE_ROOT. exiting [00:21:44] Ok I'll try that. [00:26:57] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [00:31:06] Joan: What file exactly do I convert it to? [00:43:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 39280 MB (4% inode=99%): [00:47:28] Load avg. on willow is WARNING: WARNING - load average: 19.12, 16.50, 14.93 [00:48:27] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:48:48] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:49:08] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [00:49:29] cyberpower678: You want UTF-8 without BOM. [00:50:07] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [00:50:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:50:27] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:50:36] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12038.000000 [00:50:56] Thanks, I think I got. It seems to be booting. [00:51:28] Load avg. on willow is OK: OK - load average: 14.19, 14.93, 14.62 [00:51:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:51:57] Whee. [00:52:08] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [00:54:21] Notepad++ is pretty nice as a general text editor. [01:01:27] Load avg. on willow is WARNING: WARNING - load average: 23.50, 18.16, 15.59 [01:07:19] nacht ts [01:11:33] cyberpower678: Hello [01:11:42] @replag [01:11:42] SigmaWP: s1-rr-a-c: 3h 7m 28s [+0.24 s/s]; s2-user: 4d 16h 29m 26s [+0.89 s/s]; s3-rr-a: 2m 48s [+0.02 s/s]; s3-user: 2m 48s [+0.02 s/s]; s5-rr-a: 11s [+0.00 s/s]; s5-user: 11s [+0.00 s/s] [01:18:27] Load avg. on willow is OK: OK - load average: 10.02, 12.93, 14.86 [01:26:53] 3(created) [GEOHACK-22] Nokia link badly needs revising !; GeoHack; Bug <10https://jira.toolserver.org/browse/GEOHACK-22> (Jidanni) [01:27:07] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [01:31:08] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [01:32:56] 3(commented) [GEOHACK-22] Nokia link badly needs revising ! <10https://jira.toolserver.org/browse/GEOHACK-22> (Jidanni) [01:48:40] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:49:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:49:10] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [01:50:09] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [01:50:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:50:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:50:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10911.000000 [01:52:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:52:10] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [01:57:09] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39042 MB (9% inode=99%): [01:57:40] Load avg. on willow is WARNING: WARNING - load average: 15.28, 14.50, 13.76 [01:58:40] Load avg. on willow is OK: OK - load average: 14.66, 14.41, 13.77 [02:01:40] Load avg. on willow is WARNING: WARNING - load average: 19.88, 17.41, 15.08 [02:02:19] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24279 MB (5% inode=99%): [02:03:20] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 48604 MB (11% inode=99%): [02:25:49] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3469.000000 [02:27:09] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [02:30:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.18, 19.91, 17.70 [02:31:19] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [02:31:39] Load avg. on willow is WARNING: WARNING - load average: 22.37, 19.25, 17.60 [02:31:49] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1720.000000 [02:48:40] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:49:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:49:20] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [02:50:20] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [02:50:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:50:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:52:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:52:09] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [03:00:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.43, 19.28, 17.29 [03:01:39] Load avg. on willow is WARNING: WARNING - load average: 23.23, 19.02, 17.32 [03:23:40] Load avg. on willow is OK: OK - load average: 12.34, 13.16, 14.94 [03:27:09] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [03:32:20] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [03:48:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:49:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:49:20] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [03:50:19] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [03:50:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:50:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:52:19] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37121 MB (9% inode=99%): [03:53:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:53:09] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [03:55:20] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 61273 MB (15% inode=99%): [04:00:20] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 35553 MB (8% inode=99%): [04:00:39] Load avg. on willow is WARNING: WARNING - load average: 25.65, 17.86, 15.57 [04:28:19] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [04:32:21] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [04:34:49] @replag [04:34:49] matthewrbowker: s1-rr-a-c: 13s [-0.92 s/s]; s2-user: 4d 18h 37m 3s [+0.63 s/s]; s2-user-c: 13s [+0.00 s/s]; s3-rr-a: 1m 17s [-0.01 s/s]; s3-user: 1m 17s [-0.01 s/s]; s4-rr-a: 13s [+0.00 s/s]; s4-user: 13s [+0.00 s/s]; s5-rr-a: 11s [-] [04:34:50] matthewrbowker: s5-user: 11s [-]; s5-user-c: 13s [+0.00 s/s]; s7-rr-a: 3h 6m 10s [+0.21 s/s]; s7-user: 3h 6m 10s [+0.21 s/s] [04:38:50] Load avg. on willow is OK: OK - load average: 12.35, 13.92, 14.90 [04:45:50] Load avg. on willow is WARNING: WARNING - load average: 17.83, 15.80, 15.28 [04:47:50] Load avg. on willow is OK: OK - load average: 12.23, 14.44, 14.84 [04:49:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:49:01] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:49:20] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [04:50:19] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [04:50:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:50:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:52:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:53:09] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [04:57:55] 3(created) [TS-1319] PEAR package requests; Toolserver: General/Unknown; Trivial Task <10https://jira.toolserver.org/browse/TS-1319> (madman) [05:02:50] Load avg. on willow is WARNING: WARNING - load average: 19.23, 17.98, 15.77 [05:12:53] 3(created) [TS-1320] Update Sun Studio; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1320> (Marlen Caemmerer) [05:14:57] 3(commented) [MNT-1192] s3 and s7 slaves were stopped <10https://jira.toolserver.org/browse/MNT-1192> (Marlen Caemmerer) [05:28:19] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [05:33:19] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [05:33:19] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 34303 MB (8% inode=99%): [05:36:20] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22134 MB (5% inode=99%): [05:43:19] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 28885 MB (7% inode=99%): [05:49:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:49:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:49:19] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [05:50:19] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [05:50:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:51:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:53:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:53:09] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [06:03:48] Load avg. on willow is WARNING: WARNING - load average: 16.82, 17.95, 18.30 [06:28:28] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [06:33:19] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [06:49:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:50:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:50:19] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [06:50:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:51:18] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [06:52:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:53:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:53:12] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [07:00:49] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.25, 21.01, 18.30 [07:01:49] Load avg. on willow is WARNING: WARNING - load average: 26.20, 21.25, 18.57 [07:06:21] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36027 MB (8% inode=99%): [07:10:19] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22808 MB (5% inode=99%): [07:18:20] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39404 MB (9% inode=99%): [07:28:40] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [07:33:20] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [07:49:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:50:19] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [07:50:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:50:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:50:49] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.68, 22.36, 20.28 [07:51:19] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [07:52:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:53:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:53:21] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [07:53:50] Load avg. on willow is WARNING: WARNING - load average: 18.29, 20.63, 19.98 [08:00:49] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.96, 21.00, 20.09 [08:11:20] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36386 MB (8% inode=99%): [08:15:21] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 21927 MB (5% inode=99%): [08:25:50] Load avg. on willow is WARNING: WARNING - load average: 18.96, 18.40, 18.93 [08:29:38] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [08:33:21] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [08:35:21] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 137034 MB (34% inode=99%): [08:49:20] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:50:20] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [08:50:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:50:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:51:20] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [08:52:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:53:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:53:20] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [09:00:59] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.71, 21.77, 20.22 [09:01:50] Load avg. on willow is WARNING: WARNING - load average: 18.83, 20.60, 19.89 [09:03:20] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 35387 MB (8% inode=99%): [09:06:21] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22635 MB (5% inode=99%): [09:12:20] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 53030 MB (13% inode=99%): [09:29:39] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [09:33:21] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [09:49:21] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:50:21] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [09:50:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:50:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:51:20] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [09:52:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:53:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:53:29] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [09:55:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.61, 22.23, 20.34 [09:56:20] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 33118 MB (8% inode=99%): [09:58:49] Load avg. on willow is WARNING: WARNING - load average: 16.18, 19.75, 19.73 [09:59:20] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22276 MB (5% inode=99%): [10:00:09] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1930 [10:00:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.93, 20.91, 20.12 [10:01:09] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1926 [10:01:20] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 113375 MB (28% inode=99%): [10:03:09] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:05:59] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:16:09] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:25:59] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:26:00] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:26:50] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:28:29] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3615 [10:28:49] Load avg. on willow is WARNING: WARNING - load average: 16.98, 17.61, 19.98 [10:28:59] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:29:39] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [10:30:09] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3634 [10:30:29] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:49] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.02, 19.36, 20.31 [10:31:20] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:31:50] Load avg. on willow is WARNING: WARNING - load average: 16.48, 18.15, 19.82 [10:33:00] Environment on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:39] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [10:33:50] Environment on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [10:36:34] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:37:09] Environment on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:38:00] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:38:00] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:49] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:42:39] wiki.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 16 seconds [10:45:00] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:00] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:09] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [10:45:20] SMF on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:20] /tmp on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:20] Load avg. on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:20] Load avg. on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:20] Load avg. on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:20] /tmp on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:20] SMF on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:30] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:30] /sql on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:30] / on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:30] Load avg. on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:30] / on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:30] /tmp on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:30] SMF on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:31] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:39] SMF on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:39] SMF on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:50] /tmp on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:50] / on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:50] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:50] / on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:50] /tmp on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:51] Load avg. on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:51] / on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:45:59] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:46:20] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:46:30] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [10:46:31] MySQL slave on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [10:47:00] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [10:47:09] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [10:47:09] MySQL on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [10:47:30] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:50] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 131 MB (99% inode=99%): [10:47:50] / on z-dat-s3-a is OK: DISK OK - free space: / 11694 MB (39% inode=87%): [10:47:50] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 214209 MB (22% inode=99%): [10:47:50] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 131 MB (99% inode=99%): [10:47:50] / on z-dat-s6-a is OK: DISK OK - free space: / 11694 MB (39% inode=87%): [10:47:51] SMF on hyacinth is OK: OK - all services online [10:47:51] / on z-dat-s7-a is OK: DISK OK - free space: / 11694 MB (39% inode=87%): [10:48:09] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 136 MB (99% inode=99%): [10:48:10] SMF on z-dat-s7-a is OK: OK - all services online [10:48:20] Load avg. on hyacinth is OK: OK - load average: 0.01, 0.21, 0.90 [10:48:20] Load avg. on z-dat-s7-a is OK: OK - load average: 0.01, 0.21, 0.90 [10:48:20] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 142 MB (99% inode=99%): [10:48:20] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:48:20] SMF on z-dat-s6-a is OK: OK - all services online [10:48:30] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 101658 MB (25% inode=99%): [10:48:30] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 120282 MB (29% inode=99%): [10:48:30] wiki.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 349 bytes in 0.708 second response time [10:48:30] / on z-dat-s4-a is OK: DISK OK - free space: / 11694 MB (39% inode=87%): [10:48:30] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 214207 MB (22% inode=99%): [10:48:30] SMF on z-dat-s3-a is OK: OK - all services online [10:48:30] Load avg. on z-dat-s3-a is OK: OK - load average: 0.01, 0.20, 0.88 [10:48:31] Load avg. on z-dat-s6-a is OK: OK - load average: 0.01, 0.20, 0.88 [10:48:31] /tmp on hyacinth is OK: DISK OK - free space: /tmp 136 MB (100% inode=99%): [10:48:32] / on hyacinth is OK: DISK OK - free space: / 11694 MB (39% inode=87%): [10:48:39] MySQL on z-dat-s3-a is OK: Uptime: 91099 Threads: 93 Questions: 114646653 Slow queries: 5974 Opens: 939235 Flush tables: 1 Open tables: 16384 Queries per second avg: 1258.484 [10:48:39] SMF on z-dat-s4-a is CRITICAL: ERROR - offline: svc:/application/ts/mysql-51:default [10:48:50] Load avg. on z-dat-s4-a is OK: OK - load average: 0.19, 0.23, 0.88 [10:48:50] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:48:59] MySQL on z-dat-s7-a is OK: Uptime: 523541 Threads: 100 Questions: 106779600 Slow queries: 15155 Opens: 717934 Flush tables: 1 Open tables: 3520 Queries per second avg: 203.956 [10:49:09] SMTP on z-dat-s4-a is OK: SMTP OK - 0.081 sec. response time [10:49:20] SMTP on hyacinth is OK: SMTP OK - 0.004 sec. response time [10:49:30] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:49:30] MySQL on z-dat-s4-a is OK: Uptime: 40 Threads: 1 Questions: 5 Slow queries: 0 Opens: 18 Flush tables: 1 Open tables: 9 Queries per second avg: 0.125 [10:49:39] SMF on z-dat-s4-a is OK: OK - all services online [10:49:59] MySQL slave on z-dat-s4-a is OK: Uptime: 68 Threads: 3 Questions: 2157 Slow queries: 0 Opens: 60 Flush tables: 1 Open tables: 51 Queries per second avg: 31.720 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 310 [10:50:20] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 172.000000 [10:50:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:50:50] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:51:09] MySQL on z-dat-s6-a is OK: Uptime: 147 Threads: 4 Questions: 65 Slow queries: 0 Opens: 47 Flush tables: 1 Open tables: 38 Queries per second avg: 0.442 [10:51:19] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [10:51:20] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [10:53:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:53:31] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [10:53:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:01:50] Load avg. on willow is WARNING: WARNING - load average: 20.34, 19.20, 19.07 [11:16:18] /sql on thyme is WARNING: DISK WARNING - free space: /sql 165788 MB (17% inode=99%): [11:19:13] !replag [11:19:22] @replag [11:20:19] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 96147 MB (9% inode=99%): [11:21:19] /sql on thyme is WARNING: DISK WARNING - free space: /sql 132983 MB (13% inode=99%): [11:21:30] kevin_brown: s1-user: 3m 11s [+0.57 s/s]; s2-user: 4d 23h 30m 20s [+1.00 s/s]; s3-rr-a: 1h 51m 14s [+1.00 s/s]; s3-user: 1h 51m 14s [+1.00 s/s]; s4-rr-a: 10m 54s [-0.33 s/s]; s4-user: 10m 54s [-0.33 s/s]; s6-rr-a: 1h 25m 50s [+0.76 s/s]; s6-user: 1h 25m 50s [+0.76 s/s] [11:21:31] kevin_brown: s7-rr-a: 9h 50m 44s [+1.01 s/s]; s7-user: 9h 51m 49s [+1.00 s/s] [11:29:00] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7215 [11:29:50] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [11:30:09] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5550 [11:34:09] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [11:41:52] 3(resolved) [MAGNUS-300] Replacing {{NowCommons}} by {{NowCommons (MtC drive)}} <10https://jira.toolserver.org/browse/MAGNUS-300> (Magnus Manske) [11:41:54] 3(closed) [CHTWO-43] Replacement of {{NowCommons}} by {{NowCommons (MtC drive)}} in March <10https://jira.toolserver.org/browse/CHTWO-43> (Magnus Manske) [11:44:18] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.956055/1.00, alarm hl:np_load_long=0.596680/1.50, alarm hl:mem_free=22320.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.956055/1.10, alarm hl:np_load_long=0.596680/1.50, alarm hl:mem_free=22320.000000M/300M, alarm hl:available=1/0 [11:45:19] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [11:50:19] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.125000/1.00, alarm hl:np_load_long=0.691406/1.50, alarm hl:mem_free=22268.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.125000/1.10, alarm hl:np_load_long=0.691406/1.50, alarm hl:mem_free=22268.000000M/300M, alarm hl:available=1/0 [11:50:29] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:51:29] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [11:51:29] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [11:51:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:51:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:53:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:53:40] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [11:53:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:55:39] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37589 MB (9% inode=99%): [12:00:39] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22747 MB (5% inode=99%): [12:01:59] Load avg. on willow is WARNING: WARNING - load average: 22.25, 19.57, 18.33 [12:07:38] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 96672 MB (23% inode=99%): [12:08:59] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.21, 22.89, 20.25 [12:22:00] Load avg. on willow is WARNING: WARNING - load average: 16.70, 19.12, 19.92 [12:24:25] Hello all [12:24:27] @replag [12:24:32] DaBPunkt: s1-rr-a-c: 46s [+0.00 s/s]; s2-user: 5d 30m 35s [+0.93 s/s]; s3-rr-a: 2h 51m 0s [+0.92 s/s]; s3-user: 2h 51m 0s [+0.92 s/s]; s6-rr-a: 2h 15m 56s [+0.77 s/s]; s6-user: 2h 15m 56s [+0.77 s/s]; s7-rr-a: 10h 55m 48s [+1.00 s/s]; s7-user: 10h 55m 53s [+1.00 s/s] [12:28:59] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10440 [12:29:50] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [12:30:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8247 [12:34:08] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [12:37:39] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 35579 MB (8% inode=99%): [12:38:43] eswiki is mising rev_sha1, I will add it [12:40:58] oh. nosy started it already. Stoped mine [12:41:39] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 21460 MB (5% inode=99%): [12:44:19] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 38872 MB (3% inode=99%): [12:48:40] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 80881 MB (20% inode=99%): [12:50:30] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:51:29] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [12:51:29] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [12:51:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:51:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:53:39] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [12:54:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:54:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:57:40] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36777 MB (9% inode=99%): [13:22:59] Load avg. on willow is WARNING: WARNING - load average: 12.77, 15.00, 16.70 [13:28:59] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12426 [13:30:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4063 [13:30:49] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [13:33:20] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3549 [13:34:09] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [13:43:14] Hello DaBPunkt [13:50:30] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:51:29] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [13:51:29] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [13:51:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:51:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:53:39] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [13:53:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:54:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:00:58] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.23, 22.82, 19.84 [14:03:00] Load avg. on willow is WARNING: WARNING - load average: 20.17, 21.84, 19.88 [14:04:59] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.58, 23.19, 20.57 [14:05:50] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 34998 MB (8% inode=99%): [14:08:41] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 62687 MB (15% inode=99%): [14:16:40] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36332 MB (8% inode=99%): [14:19:23] MySQL slave on z-dat-s6-a is OK: Uptime: 12631 Threads: 10 Questions: 853353 Slow queries: 2307 Opens: 583 Flush tables: 1 Open tables: 490 Queries per second avg: 67.560 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1721 [14:25:21] Load avg. on willow is WARNING: WARNING - load average: 15.69, 15.25, 17.29 [14:29:22] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12202 [14:30:51] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [14:34:35] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [14:43:54] 3(created) [ACCAPP-465] Request for new account; Account Approval; Minor New Account <10https://jira.toolserver.org/browse/ACCAPP-465> (Luca Rabinelli) [14:50:32] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:51:32] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [14:51:33] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [14:51:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:52:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:53:52] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [14:54:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:54:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:55:21] Load avg. on willow is CRITICAL: CRITICAL - load average: 35.21, 24.46, 20.00 [14:58:20] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3474 [15:02:22] MySQL slave on z-dat-s3-a is OK: Uptime: 106326 Threads: 15 Questions: 122161607 Slow queries: 17597 Opens: 972309 Flush tables: 1 Open tables: 16384 Queries per second avg: 1148.934 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1799 [15:09:22] Load avg. on willow is WARNING: WARNING - load average: 14.07, 17.82, 19.57 [15:30:52] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [15:35:31] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [15:50:41] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:51:42] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [15:51:42] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [15:51:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:52:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:53:52] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [15:54:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:55:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:09:22] Load avg. on willow is WARNING: WARNING - load average: 15.82, 16.52, 16.43 [16:30:52] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [16:36:31] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [16:47:02] [[Special:Log/newusers]] create 10 * Angel * (New user account) [16:50:51] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:51:42] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [16:51:42] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [16:51:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:52:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:54:00] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [16:54:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:55:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:08:52] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39898 MB (9% inode=99%): [17:10:20] Load avg. on willow is WARNING: WARNING - load average: 24.49, 17.73, 16.77 [17:10:52] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 72755 MB (17% inode=99%): [17:27:55] 3(created) [MNT-1206] added knissen to mmp RENDER; Maintenance: multimaintainer; Minor work <10https://jira.toolserver.org/browse/MNT-1206> (Daniel Kinzler) [17:27:57] 3(assigned) [MNT-1206] added knissen to mmp RENDER <10https://jira.toolserver.org/browse/MNT-1206> (Daniel Kinzler) [17:27:59] 3(resolved) [MNT-1206] added knissen to mmp RENDER <10https://jira.toolserver.org/browse/MNT-1206> (Daniel Kinzler) [17:30:51] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [17:31:52] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42617 MB (10% inode=99%): [17:36:30] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [17:39:52] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 58251 MB (14% inode=99%): [17:41:05] Toto Azéro * [Toolserver-l] MMP interwiki-bot [17:45:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38431 MB (9% inode=99%): [17:50:52] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:51:51] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [17:51:51] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [17:51:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:52:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:54:00] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [17:54:21] Load avg. on willow is OK: OK - load average: 13.07, 14.30, 14.97 [17:55:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:55:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:02:22] Load avg. on willow is WARNING: WARNING - load average: 19.20, 16.32, 15.28 [18:25:47] hmm, S2 replication seems to hardly running, replag still going up? [18:27:54] Akoopal: possibly also schema update? [18:28:07] !replag [18:28:10] @replag [18:28:11] valhallasw: s2-user: 5d 3h 43m 3s [+0.53 s/s]; s3-rr-a: 49s [-0.47 s/s]; s3-user: 49s [-0.47 s/s]; s7-rr-a: 16h 59m 31s [+1.00 s/s]; s7-user: 16h 59m 31s [+1.00 s/s] [18:28:26] it's only going up at 0.5 s/s, so it's not completely broken [18:29:25] yeah, the graph now is a bit flat, but still going up [18:31:01] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [18:36:21] Load avg. on willow is OK: OK - load average: 12.58, 14.04, 14.89 [18:36:30] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [18:50:51] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:52:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:52:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:52:52] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [18:52:52] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [18:54:01] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [18:56:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:56:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:57:21] Load avg. on willow is WARNING: WARNING - load average: 14.65, 15.27, 14.96 [18:58:21] Load avg. on willow is OK: OK - load average: 14.18, 14.99, 14.88 [19:31:01] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [19:36:41] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [19:42:22] Load avg. on willow is WARNING: WARNING - load average: 16.16, 15.19, 14.81 [19:42:38] [[Special:Log/newusers]] create 10 * Aviyal * (New user account) [19:44:22] Load avg. on willow is OK: OK - load average: 13.27, 14.65, 14.67 [19:45:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36758 MB (9% inode=99%): [19:49:52] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23906 MB (5% inode=99%): [19:51:01] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:52:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:52:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:52:52] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [19:52:52] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [19:52:52] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40931 MB (10% inode=99%): [19:54:11] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [19:56:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:56:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:04:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 127217 MB (20% inode=99%): [20:23:51] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36484 MB (8% inode=99%): [20:27:51] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24054 MB (5% inode=99%): [20:30:36] @replag [20:30:37] DaBPunkt: s2-user: 5d 4h 45m 9s [+0.51 s/s]; s3-rr-a: 30s [-0.00 s/s]; s3-user: 30s [-0.00 s/s]; s6-rr-a: 13s [-0.28 s/s]; s6-user: 13s [-0.28 s/s]; s7-rr-a: 19h 1m 53s [+1.00 s/s]; s7-user: 19h 1m 53s [+1.00 s/s] [20:31:11] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [20:37:30] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [20:51:12] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:52:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:53:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:53:51] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [20:53:51] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [20:54:11] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [20:56:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:57:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:59:29] hello nosy :) [20:59:37] hello DaBPunkt [20:59:40] * matthewrbowker waves to nosy [20:59:58] i am here only for a short while i wanted to talk to mark regarding the data center [21:00:11] hello matthewrbowker [21:00:12] have fun [21:00:31] nosy: did you see sebastians mail? [21:00:34] DaBPunkt: do you know which hardware the move from mark would regard? i have no idea [21:00:38] yes thats why [21:01:03] regard? [21:01:16] weisst du welche server er meint [21:01:28] meint er die ollen dell-kisten die er wegwerfen wollte? [21:01:29] nosy: die dells [21:01:33] ah ok [21:01:45] die nehmen wir als spielkisten bis sie auseinanderfallen? [21:02:45] wie viele eigentlich? [21:03:09] gna....schema update for s7 is still ongoing [21:03:26] nosy: yes, I watched it the hole day… [21:03:31] nosy: nach möglichkeit 3 [21:03:42] now arwiki is the next for alter table [21:04:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126933 MB (20% inode=99%): [21:04:18] i wonder why no automatic alter table from the master came through here [21:05:09] nosy: für den Switch: Mark (und ich denke, dass er recht hat) meinte, dass wir den billigsten Switch nehmen solen, den wir finden [21:05:22] DaBPunkt: right [21:05:28] baumarktswitch [21:05:40] (für die MGMT täte es soagr ein alter 10er ;)) [21:05:52] nosy: naja, ins Rack muss er schon passen [21:06:01] 10 mb/s koennen die nics alle afaik ;) [21:06:13] stimmt...sonst kriegt mark plack [21:06:19] jepp [21:08:10] bei ebay gibt's welche (neu) für 30-50€ [21:09:18] DaBPunkt: ich glaub wir brauchen vlans wg verschiedener netze [21:09:32] wir haben in oe10 noch genau einen freien port [21:09:40] dann brauchen wir sowieso nen switch [21:09:43] die MGMTs sidn alle in selbem VLAN [21:10:16] nosy: der neue switch soll für die MGMT-ports sein, die dann den normalen switch verlassen können [21:10:25] achso [21:10:38] nur für management-ports? dann reicht ja ein vlan für alle [21:10:43] ja [21:10:45] Do you guys need an old switch? [21:10:51] oki das kriegt man auch unmanaged konfiguriert [21:10:59] multichill: would be great for next week [21:11:05] Specs? [21:11:12] 220V [21:11:13] any ethernet switch [21:11:15] 19" [21:11:30] 100 Mbit or do you need Gigabit? [21:11:31] just for sandkasten-benutzung [21:11:40] 100 mb would be fine [21:11:52] but if you have gb [21:12:09] do you need a recepy or any paper for this? [21:12:34] something like a  donation receipt? [21:12:35] Nah [21:12:40] ok cool :) [21:12:57] I'm out of the office tomorrow, but I can check next week [21:13:18] It's going to be an oldie because the not so olds are still spare or just for management [21:13:19] yeah yeah [21:13:46] doesnt matter as long it still goes online and has some functioning ports [21:13:56] why did you take them out of the rack? [21:16:05] mm, we need the switch in the datacenter when Mark is there next week. nosy: Can you ask him on which day he is there? [21:16:22] Load avg. on willow is WARNING: WARNING - load average: 15.23, 15.92, 15.99 [21:16:30] DaBPunkt: i think so - got the question list anyway. [21:16:31] Various reasons. Replaced, end of project, not needed etc. [21:16:44] oh ok i see [21:18:01] DaBPunkt: do you have todos for mark? i remember yarrow and the cables for the san [21:18:41] nosy: complete reset of the DRAC of yarrow is my only point [21:18:51] I have a bunch of spare Cisco 2950's and 2960's [21:18:52] (I locked myself out :() [21:19:13] ok went through my list and i also have nothing else then the cables and the reset for yarrow [21:19:19] multichill: wow cool [21:20:06] nosy: if he has time (and 1 of us is online): He can daisy-chain the Array below hemlock's array (we spoke about that) [21:20:21] And a shitload of Dell switches, but I don't want to make the toolserver setup worse than it already is ;-) [21:20:58] multichill: 2960 would be great, 2950 would also be very nice and dell would still be fine [21:21:05] we have our zoo here anyway ;) [21:21:23] Mark would probably kill me if he sees a Dell in one of his racks :P [21:21:39] cisco would be a bit better for me since i am used to this and never had dell only hp [21:21:43] lol [21:22:01] I do have an hp [21:22:26] I think it's a 48 port 1G [21:22:55] oh wow...1 hu? [21:23:06] multichill: I don't think we can miss mark being a couple of years in jail :-) [21:23:13] I think that's oversized for our pourpos [21:23:37] Yes, everything is rack mountable. [21:23:40] DaBPunkt: I dont mind having good spare hardware in the rack [21:23:41] Akoopal: *grin* I was thinking of multichill as the guy who is selling stuff that "fell off the truck" ;-_ [21:23:47] nosy: true [21:24:07] valhallasw: :-) [21:24:19] Nah, I just prevent people from taking old crap back in production. [21:24:35] right. thank you multichill: [21:24:41] by taking it in production somewhere else :-) [21:25:06] It's not that old, but not really HA ;-) [21:25:18] Btw, was that test you Akoopal ? [21:25:27] multichill: nope [21:25:39] toolserver is only a bit production [21:25:43] its a funny mix [21:26:35] Most of the cost is actually in hours, not buying hardware [21:31:40] status.toolserver.org wil be away for 3 minutes, I have to reboot the server [21:32:11] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [21:32:36] * matthewrbowker wishes he could help, but he lives in the states :/ [21:33:11] nosy: we realy have to have a talk about the SAN-switch and the ports on it. The nagios-warning beginn to getting on my nerves [21:33:48] hehe :) its only a message that it is configured as offline in nagios but is online in reality [21:33:57] so hang on i will adjust that [21:35:21] Load avg. on willow is OK: OK - load average: 12.65, 13.85, 14.93 [21:36:32] hey Sebaso [21:36:42] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:36:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:37:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126775 MB (20% inode=99%): [21:37:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:38:18] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [21:39:00] nosy: and what's connected to number 7? [21:39:02] yay...s7 is replicating again... [21:39:20] should be daphne [21:39:53] nosy: can you document that in the wiki? The ports should not switch/change that often [21:40:11] short fun...now kowiki is next [21:40:26] that's a small one [21:40:33] eswiki was far the biggest [21:40:45] DaBPunkt: might be that others will follow [21:41:02] and then probably we still can get a alter table from the master... [21:41:22] ok...all we can do is wait...:) ill do that [21:41:24] nosy: I think I will be "procative" on this and just alter all of them now [21:41:40] I just waited for eswiki to finish [21:41:44] DaBPunkt: would you? [21:41:49] eswiki is through [21:42:02] now kowiki is on and i had arwiki [21:42:08] and fawiki [21:47:40] Load avg. on willow is WARNING: WARNING - load average: 17.38, 15.51, 14.92 [21:56:07] DaBPunkt: people from the render mmp have stated that the databases in general did not work today until 1 pm [21:56:18] do you have this impression too? [21:58:23] I doubt that all databases were down and nobody told us [22:00:13] DaBPunkt: we would have gotten warnings for this [22:00:24] i asked how the error exactly looked [22:01:24] DaBPunkt: kowiki finished [22:01:33] next would be ukwiki [22:01:43] stop for a moment please [22:02:00] ok [22:02:27] ok, double-adding is no problem (throws an error). I will do them all in a row now. [22:02:32] DaBPunkt: ein zitat einer kollegin: "warten kann ich immernoch am besten" :) [22:02:43] gute idee [22:05:53] ich hab cassia auch gleich mitangestoßen [22:06:17] für enwiki werden wir wohl downtime machen müssen, wenn's soweit ist [22:07:40] naja fuer enwiki haben wir immerhin 2 hosts [22:07:46] ja [22:08:54] DaBPunkt: den fehler von daniel konntet ihr gestern nicht reproduzieren auf daphne - oder? [22:09:04] nein, konnte er nicht [22:09:10] schade [22:15:12] gn8 ts [22:36:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:36:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:36:19] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [22:36:19] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [22:36:39] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:37:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126469 MB (20% inode=99%): [22:37:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:37:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:38:18] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [22:47:40] Load avg. on willow is WARNING: WARNING - load average: 13.98, 14.86, 15.25 [22:51:39] Load avg. on willow is OK: OK - load average: 12.49, 14.23, 14.92 [22:51:53] 3(created) [LUXO-24] Global block link for IPs at ~luxo/contributions/contributions.php; Luxo's tools; Minor Improvement <10https://jira.toolserver.org/browse/LUXO-24> (Marco Aurelio) [22:56:53] 3(updated) [LUXO-24] Global block link for IPs at ~luxo/contributions/contributions.php <10https://jira.toolserver.org/browse/LUXO-24> (DaB.) [22:57:39] Load avg. on willow is WARNING: WARNING - load average: 15.91, 15.27, 15.13 [22:58:54] 3(deleted) [LUXO-24] Global block link for IPs at ~luxo/contributions/contributions.php (DaB.) [23:01:35] nacht ts [23:01:43] DaBPunkt: 'night [23:05:54] 3(created) [LUXO-25] Global block link for IPs at ~luxo/contributions/contributions.php; Luxo's tools; Minor Improvement <10https://jira.toolserver.org/browse/LUXO-25> (Marco Aurelio) [23:05:56] 3(updated) [LUXO-25] Global block link for IPs at ~luxo/contributions/contributions.php <10https://jira.toolserver.org/browse/LUXO-25> (Marco Aurelio) [23:10:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.76, 22.29, 18.57 [23:30:38] Load avg. on willow is WARNING: WARNING - load average: 22.17, 18.94, 18.55 [23:36:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:36:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:36:19] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [23:36:20] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [23:37:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126276 MB (20% inode=99%): [23:37:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:37:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:37:49] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:38:19] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds)