[00:02:44] Load avg. on willow is WARNING: WARNING - load average: 15.12, 16.45, 14.40 [00:03:45] Load avg. on willow is OK: OK - load average: 10.62, 14.93, 13.99 [00:06:53] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [00:07:03] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:08:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:09:25] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.210938/1.10, alarm hl:np_load_long=0.747070/1.55, alarm hl:mem_free=10570.000000M/500M, alarm hl:tmp_free=13073M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.210938/1.00, alarm hl:np_load_long=0.747070/1.50, alarm hl:mem_free=10570.000000M/600M, alarm hl:tmp_free [00:10:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:11:25] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [00:11:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:15:53] SSH on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:16:45] SSH on adenia is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [00:24:25] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:44:14] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 83835 [00:51:55] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 299645 MB (5% inode=33%): [00:51:55] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:02:55] Load avg. on willow is WARNING: WARNING - load average: 14.07, 16.83, 14.44 [01:02:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.740723/1.95, alarm hl:tmp_free=29305M/100M, alarm hl:np_load_avg=2.098633/2.0, alarm hl:mem_free=1200.000000M/350M, alarm hl:available=1/0 [01:03:54] Sun Grid Engine execd on willow is OK: testqueue@willow OK: medium-sol@willow OK: longrun-sol@willow OK [01:04:55] Load avg. on willow is OK: OK - load average: 8.70, 14.18, 13.75 [01:07:03] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [01:08:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:08:54] Load avg. on willow is WARNING: WARNING - load average: 12.25, 16.24, 14.95 [01:08:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.985351/1.95, alarm hl:tmp_free=29270M/100M, alarm hl:np_load_avg=2.150879/2.0, alarm hl:mem_free=1358.000000M/350M, alarm hl:available=1/0 [01:10:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:11:44] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:25:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:45:14] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 85637 [01:51:55] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 299913 MB (5% inode=33%): [02:02:55] Load avg. on willow is WARNING: WARNING - load average: 13.73, 16.73, 15.02 [02:03:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.730957/1.95, alarm hl:tmp_free=29180M/100M, alarm hl:np_load_avg=2.095215/2.0, alarm hl:mem_free=2012.000000M/350M, alarm hl:available=1/0 [02:04:04] Sun Grid Engine execd on willow is OK: testqueue@willow OK: medium-sol@willow OK: longrun-sol@willow OK [02:04:55] Load avg. on willow is OK: OK - load average: 8.91, 13.98, 14.20 [02:07:03] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [02:08:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:11:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:11:55] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:14:55] Load avg. on willow is WARNING: WARNING - load average: 12.34, 16.44, 15.49 [02:15:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.556152/1.95, alarm hl:tmp_free=29167M/100M, alarm hl:np_load_avg=2.059082/2.0, alarm hl:mem_free=968.000000M/350M, alarm hl:available=1/0 [02:20:40] [[Special:Log/newusers]] create 10 * Gertyzer45 * (New user account) [02:25:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:45:14] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86892 [02:51:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.368164/1.95, alarm hl:tmp_free=29111M/100M, alarm hl:np_load_avg=2.556152/2.0, alarm hl:mem_free=1148.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.368164/2.3, alarm hl:np_load_long=2.405762/2.5, alarm hl:cpu=85.900000/98, alarm hl:mem_free=1148.000000M/200M, [02:51:54] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 298048 MB (5% inode=33%): [03:00:04] Sun Grid Engine execd on willow is OK: testqueue@willow OK: medium-sol@willow OK: longrun-sol@willow OK [03:03:04] Sun Grid Engine execd on willow is WARNING: testqueue@willow exceedes load threshold: alarm hl:np_load_avg=3.056152/2.75: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.901855/1.95, alarm hl:tmp_free=29093M/100M, alarm hl:np_load_avg=3.056152/2.0, alarm hl:mem_free=506.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.901855/2.3, alarm hl:np_load_long=2 [03:07:05] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [03:08:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:11:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:12:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:16:05] Sun Grid Engine execd on willow is OK: testqueue@willow OK: medium-sol@willow OK: longrun-sol@willow OK [03:25:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:27:05] Load avg. on willow is WARNING: WARNING - load average: 12.25, 15.05, 16.37 [03:31:05] Load avg. on willow is CRITICAL: CRITICAL - load average: 47.30, 23.59, 19.00 [03:32:05] Load avg. on willow is WARNING: WARNING - load average: 25.14, 21.79, 18.67 [03:33:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.942383/1.95, alarm hl:tmp_free=29042M/100M, alarm hl:np_load_avg=2.472656/2.0, alarm hl:mem_free=1411.000000M/350M, alarm hl:available=1/0 [03:36:05] Sun Grid Engine execd on willow is OK: testqueue@willow OK: medium-sol@willow OK: longrun-sol@willow OK [03:42:04] Load avg. on willow is OK: OK - load average: 8.24, 11.83, 14.89 [03:45:23] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86805 [03:52:05] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 297980 MB (5% inode=33%): [04:04:05] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 2052777s failure: longrun-sol@willow in error state: QERROR as result of job 2052777s failure [04:08:14] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [04:08:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:11:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:12:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:12:34] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.649902/1.10, alarm hl:np_load_long=0.551269/1.55, alarm hl:mem_free=336.000000M/500M, alarm hl:tmp_free=12461M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.649902/1.00, alarm hl:np_load_long=0.551269/1.50, alarm hl:mem_free=336.000000M/600M, alarm hl:tmp_free= [04:13:05] Load avg. on willow is WARNING: WARNING - load average: 29.62, 22.20, 18.30 [04:17:33] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [04:18:14] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:18:14] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:19:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:19:13] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 2052777s failure: longrun-sol@willow in error state: QERROR as result of job 2052777s failure [04:23:05] Load avg. on willow is OK: OK - load average: 6.80, 11.69, 14.64 [04:24:14] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:25:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:42:44] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.527832/1.10, alarm hl:np_load_long=0.568359/1.55, alarm hl:mem_free=499.000000M/500M, alarm hl:tmp_free=12421M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.527832/1.00, alarm hl:np_load_long=0.568359/1.50, alarm hl:mem_free=499.000000M/600M, alarm hl:tmp_free= [04:44:15] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 2052777s failure: longrun-sol@willow in error state: QERROR as result of job 2052777s failure [04:45:24] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86980 [04:47:44] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [04:53:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 297922 MB (5% inode=33%): [05:01:14] wolfsbane and ortelius only server on port 80, right ? [05:01:16] which server does https ? [05:03:14] Load avg. on willow is WARNING: WARNING - load average: 11.69, 15.42, 13.67 [05:04:15] Load avg. on willow is OK: OK - load average: 8.80, 13.93, 13.25 [05:05:15] [[Tool considerations]] 10https://wiki.toolserver.org/w/index.php?diff=7251&oldid=7021&rcid=9675 * Krinkle * (+1) (/* Internationalization */ ) [05:05:36] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7246 of [[02Wiki server assignments10]] patrolled ) [05:05:41] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7248 of [[02Interwiki bot MMP planning10]] patrolled ) [05:07:31] [[Willow]] N 10https://wiki.toolserver.org/w/index.php?oldid=7252&rcid=9676 * Krinkle * (+25) (Redirected page to [[Admin:Willow]]) [05:07:40] [[User:91.198.174.202]] N 10https://wiki.toolserver.org/w/index.php?oldid=7253&rcid=9677 * Krinkle * (+111) (Created page with "{{PAGENAME}} is the IP address of [[willow]] at the Wikimedia Toolserver cluster. == See also == * [[Servers]]") [05:08:14] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [05:08:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:08:59] [[Special:Log/delete]] delete 10 * Krinkle * (deleted "[[02User:91.198.174.20210]]": Redundant page) [05:09:10] [[User talk:91.198.174.202]] N 10https://wiki.toolserver.org/w/index.php?oldid=7254&rcid=9679 * Krinkle * (+95) (Created page with "{{PAGENAME}} is the IP address of [[willow]] at the Wikimedia Toolserver cluster ([[Servers]]).") [05:11:27] [[Category:Pages for speedy deletion]] 10https://wiki.toolserver.org/w/index.php?diff=7255&oldid=4356&rcid=9680 * Krinkle * (-6) () [05:11:34] [[Category:Wiki]] N 10https://wiki.toolserver.org/w/index.php?oldid=7256&rcid=9681 * Krinkle * (+23) (Created page with "[[Category:Categories]]") [05:12:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:12:41] [[Category:User pages of Toolserver IP-addresses]] N 10https://wiki.toolserver.org/w/index.php?oldid=7257&rcid=9682 * Krinkle * (+17) (Created page with "[[Category:Wiki]]") [05:14:20] [[Template:Tshost]] N 10https://wiki.toolserver.org/w/index.php?oldid=7258&rcid=9683 * Krinkle * (+274) (Created page with "{{Notice| {{PAGENAME}}0.0.0.0 is the IP address of '''[[{{{1|example}}}]]''' at the Wikimedia Toolserver cluster ([[Servers]]). ...") [05:14:30] [[User talk:91.198.174.202]] 10https://wiki.toolserver.org/w/index.php?diff=7259&oldid=7254&rcid=9684 * Krinkle * (-78) (+{{tshost|willow}}) [05:15:10] [[User talk:91.198.174.194]] N 10https://wiki.toolserver.org/w/index.php?oldid=7260&rcid=9685 * Krinkle * (+18) (+{{tshost|hemlock}}) [05:15:27] [[Template:Tshost]] 10https://wiki.toolserver.org/w/index.php?diff=7261&oldid=7258&rcid=9686 * Krinkle * (+22) () [05:16:12] [[User talk:91.198.174.201]] N 10https://wiki.toolserver.org/w/index.php?oldid=7262&rcid=9687 * Krinkle * (+21) (+{{tshost|nightshade}}) [05:16:37] [[User talk:91.198.174.210]] N 10https://wiki.toolserver.org/w/index.php?oldid=7263&rcid=9688 * Krinkle * (+20) (+{{tshost|wolfsbane}}) [05:16:54] [[User talk:91.198.174.211]] N 10https://wiki.toolserver.org/w/index.php?oldid=7264&rcid=9689 * Krinkle * (+19) (+{{tswiki|ortelius}}) [05:17:00] [[User talk:91.198.174.211]] 10https://wiki.toolserver.org/w/index.php?diff=7265&oldid=7264&rcid=9690 * Krinkle * (+0) (+{{tshost|ortelius}} ) [05:19:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:20:44] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.467774/1.10, alarm hl:np_load_long=0.479492/1.55, alarm hl:mem_free=135.000000M/500M, alarm hl:tmp_free=12400M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.467774/1.00, alarm hl:np_load_long=0.479492/1.50, alarm hl:mem_free=135.000000M/600M, alarm hl:tmp_free= [05:25:44] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:27:44] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [05:30:44] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.400879/1.10, alarm hl:np_load_long=0.451660/1.55, alarm hl:mem_free=182.000000M/500M, alarm hl:tmp_free=12380M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.400879/1.00, alarm hl:np_load_long=0.451660/1.50, alarm hl:mem_free=182.000000M/600M, alarm hl:tmp_free= [05:44:14] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 2052777s failure: longrun-sol@willow in error state: QERROR as result of job 2052777s failure [05:45:35] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86530 [05:53:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 297834 MB (5% inode=33%): [05:58:44] 3(commented) [TS-1369] SSL certificate problem, bad/outdated HTTPS CA <10https://jira.toolserver.org/browse/TS-1369> (Krinkle) [06:01:42] Krinkle-away: scripts* [06:02:31] you could of course just ship your own copy of the CA cert with your script [06:03:15] Load avg. on willow is WARNING: WARNING - load average: 13.17, 17.20, 15.29 [06:08:15] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [06:08:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:11:45] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.125976/1.10, alarm hl:np_load_long=0.752930/1.55, alarm hl:mem_free=10748.000000M/500M, alarm hl:tmp_free=14604M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.125976/1.00, alarm hl:np_load_long=0.752930/1.50, alarm hl:mem_free=10748.000000M/600M, alarm hl:tmp_free [06:12:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:12:44] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [06:15:52] Krinkle-away: poke [06:16:11] Betacommand: pong [06:16:51] can you let the guy looking for ns page title lists know that I can get him that list [06:17:12] ? [06:17:46] Krinkle-away: never mind got the nicks confused [06:18:46] seeing both you an [[User:Killiondude]]'s nicks pop up at this hour Im not seeing straight [06:19:14] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.72, 21.86, 18.31 [06:19:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:20:14] Load avg. on willow is WARNING: WARNING - load average: 19.88, 20.04, 17.88 [06:25:14] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.91, 22.94, 19.30 [06:25:44] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:44:24] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 2052777s failure: longrun-sol@willow in error state: QERROR as result of job 2052777s failure [06:46:34] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86486 [06:53:14] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 297714 MB (5% inode=33%): [07:04:34] Free Memory on damiana is WARNING: WARNING - 6.9% (290460 kB) free! [07:07:04] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:08:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:09:19] Load avg. on willow is WARNING: WARNING - load average: 15.21, 16.82, 17.34 [07:09:19] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [07:11:44] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [07:12:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:13:14] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.27, 22.09, 19.09 [07:14:14] Load avg. on willow is WARNING: WARNING - load average: 19.36, 20.29, 18.65 [07:19:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:26:44] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:27:14] Load avg. on willow is OK: OK - load average: 8.91, 11.42, 14.80 [07:44:24] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 2052777s failure: longrun-sol@willow in error state: QERROR as result of job 2052777s failure [07:46:33] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86808 [07:53:14] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 297263 MB (5% inode=33%): [07:57:04] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129207 MB (10% inode=99%): [07:58:04] /sql on cassia is OK: DISK OK - free space: /sql 133801 MB (11% inode=99%): [08:03:14] Load avg. on willow is WARNING: WARNING - load average: 13.00, 16.32, 14.19 [08:04:14] Load avg. on willow is OK: OK - load average: 10.27, 14.94, 13.84 [08:04:33] Free Memory on damiana is WARNING: WARNING - 6.0% (250928 kB) free! [08:09:48] Toolserver is down for me, but status.toolserver.org says it's ok [08:09:59] Toolserver also down for me [08:10:17] dispenser@willow:~$ screen \n getpwuid() can't identify your account! [08:13:20] Is there an op in here to deal with the flooding? [08:13:28] If not I'll get one from freenode [08:15:48] Free Memory on damiana is OK: OK - 65.8% (2755500 kB) free. [08:15:48] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [08:16:40] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:16:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:20:13] hmm [08:20:18] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:21:17] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:21:58] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [08:22:49] mrmist: had enough? [08:23:05] Pine: just deciding on what to do about it [08:25:45] Hopefully his client won't reconnect now until he tells it to [08:27:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:44:38] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [08:46:38] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86847 [08:54:07] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 297169 MB (5% inode=33%): [09:16:48] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [09:17:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:17:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:20:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:26:58] Load avg. on willow is WARNING: WARNING - load average: 13.14, 15.47, 13.30 [09:27:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:27:59] Load avg. on willow is OK: OK - load average: 9.46, 14.12, 12.96 [09:44:48] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [09:46:49] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87104 [09:48:43] Hi, anybody can say me what if does some server exist with Python 2.7.2 to run pywikipedia ? [09:54:07] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 297035 MB (5% inode=33%): [10:13:27] [[Special:Log/newusers]] create 10 * Klz43a1 * (New user account) [10:16:57] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [10:17:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:17:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:20:27] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:27:17] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:45:07] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [10:47:48] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86705 [10:54:08] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 296863 MB (5% inode=33%): [11:04:09] [[Special:Log/newusers]] create 10 * Astropiloto * (New user account) [11:17:07] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [11:18:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:18:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:20:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:27:17] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:45:09] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [11:47:48] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86908 [11:54:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 296711 MB (5% inode=33%): [12:06:58] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:17:18] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [12:18:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:18:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:20:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:27:19] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:45:28] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [12:47:58] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87726 [12:55:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 296554 MB (5% inode=33%): [12:56:28] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:59:09] Someone probably wants to fix the motd: "Next general maintenance window: Wed, 14.03, 19:00-23-59 UTC." ;-) [13:17:18] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [13:18:17] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:18:20] [[Special:Log/newusers]] create 10 * Tribeshack64 * (New user account) [13:18:48] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:20:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:27:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:45:27] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [13:48:18] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87789 [13:55:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 296407 MB (5% inode=33%): [14:17:18] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [14:18:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:18:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:20:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:27:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:33:50] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.149414/1.10, alarm hl:np_load_long=0.745117/1.55, alarm hl:mem_free=11007.000000M/500M, alarm hl:tmp_free=13917M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.149414/1.00, alarm hl:np_load_long=0.745117/1.50, alarm hl:mem_free=11007.000000M/600M, alarm hl:tmp_free [14:34:50] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [14:45:38] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [14:48:18] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87752 [14:55:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 296253 MB (5% inode=33%): [15:02:56] hi, I'd like to run a bot to guess potential interwikis and add it to the article then invoke interwiki.py to confirm it and add more interwikis if my guess is correct, or remove my bad attempt. does this use of interwiki bot violates the rule? [15:12:49] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.769531/1.10, alarm hl:np_load_long=0.922852/1.55, alarm hl:mem_free=11007.000000M/500M, alarm hl:tmp_free=13858M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.769531/1.00, alarm hl:np_load_long=0.922852/1.50, alarm hl:mem_free=11007.000000M/600M, alarm hl:tmp_free [15:16:49] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [15:17:28] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [15:18:28] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:18:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:21:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:28:48] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:45:48] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [15:48:18] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87662 [15:55:28] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 296131 MB (5% inode=33%): [16:17:38] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [16:18:38] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:19:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:21:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:28:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:45:48] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [16:48:28] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88364 [16:55:29] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 295989 MB (5% inode=33%): [17:00:38] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:01:37] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [17:02:22] have you seen "couldn't set locale correctly" on toolserver? [17:06:23] liangent: in what resepect? [17:06:58] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:07:05] Betacommand: I have mediawiki code on ts [17:07:24] and when I try to wfShellExec() something [17:07:29] it warns like this [17:07:49] Betacommand: more precisely, in the passthru( $cmd, $retval ); line [17:15:59] seems I can ssh to yarrow now but no home? [17:17:49] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [17:18:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:19:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:22:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:26:30] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [17:29:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:37:59] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.864258/1.10, alarm hl:np_load_long=1.163086/1.55, alarm hl:mem_free=11398.000000M/500M, alarm hl:tmp_free=13706M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.864258/1.00, alarm hl:np_load_long=1.163086/1.50, alarm hl:mem_free=11398.000000M/600M, alarm hl:tmp_free [17:41:58] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [17:45:58] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [17:48:38] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88392 [17:55:38] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 295842 MB (5% inode=33%): [17:56:01] Does anyone have any idea why my stored procedures disappear sometimes from mysql db [18:17:58] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [18:18:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:19:38] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:22:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:29:18] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:45:59] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [18:48:48] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88605 [18:55:48] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 295698 MB (5% inode=33%): [19:17:59] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [19:19:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:19:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:21:58] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:22:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:24:49] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1836 [19:24:49] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1836.000000 [19:29:28] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:46:08] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [19:46:28] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [19:48:48] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 89903 [19:50:48] MySQL slave on thyme is OK: Uptime: 317162 Threads: 10 Questions: 118431401 Slow queries: 33221 Opens: 21346 Flush tables: 1 Open tables: 2760 Queries per second avg: 373.409 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1778 [19:50:58] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1770.000000 [19:52:46] 3(updated) [ET-48] Newsletter ready for delivery <10https://jira.toolserver.org/browse/ET-48> (Keith Dorey) [19:55:48] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 295553 MB (5% inode=33%): [20:18:08] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [20:19:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:19:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:22:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:29:28] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:30:18] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:30:28] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:31:18] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [20:31:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:38:42] 3(resolved) [JARRY-36] Image existence checker program fails due to image in black list with not English characters <10https://jira.toolserver.org/browse/JARRY-36> (Jarry1250) [20:38:44] 3(closed) [JARRY-36] Image existence checker program fails due to image in black list with not English characters <10https://jira.toolserver.org/browse/JARRY-36> (Jarry1250) [20:48:48] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 91238 [20:55:48] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 295519 MB (5% inode=33%): [21:08:49] Question, does the PHP web interface on the toolserver have CPU/Mem limits? [21:09:28] [[CommonsDelinker]] 10https://wiki.toolserver.org/w/index.php?diff=7266&oldid=6071&rcid=9694 * Multichill * (-83) (getent group | grep delink) [21:13:18] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.482422/1.10, alarm hl:np_load_long=0.820312/1.55, alarm hl:mem_free=11076.000000M/500M, alarm hl:tmp_free=13419M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.482422/1.00, alarm hl:np_load_long=0.820312/1.50, alarm hl:mem_free=11076.000000M/600M, alarm hl:tmp_free [21:14:18] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [21:18:18] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [21:19:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:19:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:29:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:31:17] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [21:31:37] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:45:17] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.319336/1.10, alarm hl:np_load_long=0.683594/1.55, alarm hl:mem_free=11003.000000M/500M, alarm hl:tmp_free=13383M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.319336/1.00, alarm hl:np_load_long=0.683594/1.50, alarm hl:mem_free=11003.000000M/600M, alarm hl:tmp_free [21:45:48] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1855 [21:46:08] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1859.000000 [21:46:18] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [21:49:48] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 91050 [21:55:57] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 295512 MB (5% inode=33%): [22:18:18] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [22:19:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:20:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:30:38] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:31:17] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [22:31:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:40:40] tsnag is getting overly verbose lately [22:41:12] to the point where I would suggest to have a separate channel for it [22:41:31] Then people will ignore it, not that you can do much about it anyways. [22:41:42] yeah, most of it is useless for me [22:41:49] Sun Grid Engine execd on ortelius is OK [22:41:53] great to know! [22:42:13] Is if you want to run jobs [22:42:31] how about just notifying in case it is not OK [22:42:58] and what is that supposed to mean: [22:43:13] never mind [22:43:36] it is basically a wall of CRITICAL errors [22:43:45] numbs you down [22:44:06] if any real issues come up, nobody will notice in any case [22:44:34] I may as well filter everything from tsnag to /dev/null [22:45:58] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2352 [22:46:18] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2355.000000 [22:49:58] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 90956 [22:55:58] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 294742 MB (5% inode=33%): [23:03:18] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.241211/1.10, alarm hl:np_load_long=0.880860/1.55, alarm hl:mem_free=10433.000000M/500M, alarm hl:tmp_free=13208M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.241211/1.00, alarm hl:np_load_long=0.880860/1.50, alarm hl:mem_free=10433.000000M/600M, alarm hl:tmp_free [23:04:30] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [23:18:28] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [23:19:28] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:20:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:30:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:31:28] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: longrun-sol@willow in error state [23:31:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:32:19] Load avg. on willow is WARNING: WARNING - load average: 17.54, 16.63, 13.58 [23:34:18] Load avg. on willow is OK: OK - load average: 8.64, 13.47, 12.75 [23:45:28] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.231445/1.10, alarm hl:np_load_long=0.822265/1.55, alarm hl:mem_free=11313.000000M/500M, alarm hl:tmp_free=13116M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.231445/1.00, alarm hl:np_load_long=0.822265/1.50, alarm hl:mem_free=11313.000000M/600M, alarm hl:tmp_free [23:45:58] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2647 [23:46:17] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2646.000000 [23:47:28] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [23:49:58] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 90385 [23:55:58] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 294732 MB (5% inode=33%):