[00:03:08] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.820312/1.95, alarm hl:tmp_free=11394M/100M, alarm hl:np_load_avg=1.625000/2.0, alarm hl:mem_free=208.000000M/350M, alarm hl:available=1/0 [00:06:58] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 265967 MB (5% inode=30%): [00:13:48] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 11361 MB (10% inode=98%): [00:13:48] / on willow is CRITICAL: DISK CRITICAL - free space: / 11361 MB (10% inode=98%): [00:28:37] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1832 [00:29:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:29:09] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1831.000000 [00:29:59] [[Special:Log/newusers]] create 10 * Nakaptony * (New user account) [00:31:58] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [00:32:37] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:43:10] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [00:44:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:44:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:46:10] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.861328/1.95, alarm hl:tmp_free=11267M/100M, alarm hl:np_load_avg=1.697754/2.0, alarm hl:mem_free=165.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.861328/2.3, alarm hl:np_load_long=1.672363/2.5, alarm hl:cpu=98.200000/98, alarm hl:mem_free=165.000000M/200M, al [00:54:10] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [00:59:09] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 126242 [01:03:01] Load avg. on willow is WARNING: WARNING - load average: 14.92, 16.17, 14.94 [01:07:01] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 267037 MB (5% inode=30%): [01:14:00] / on willow is CRITICAL: DISK CRITICAL - free space: / 11182 MB (10% inode=98%): [01:14:01] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 11182 MB (10% inode=98%): [01:15:44] [[Special:Log/delete]] delete 10 * Dispenser * (deleted "[[02User:FixToolbox10]]": Vandalism) [01:16:09] [[Special:Log/block]] block 10 * Dispenser * (blocked [[02User:FixToolbox10]] with an expiry time of infinite (account creation disabled): Spamming links to external sites) [01:16:10] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.478027/1.95, alarm hl:tmp_free=11179M/100M, alarm hl:np_load_avg=2.281250/2.0, alarm hl:mem_free=203.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.478027/2.3, alarm hl:np_load_long=2.082520/2.5, alarm hl:cpu=99.400000/98, alarm hl:mem_free=203.000000M/200M, al [01:17:44] 3(commented) [MAGNUS-323] Catscan not picking up templates from talk pages <10https://jira.toolserver.org/browse/MAGNUS-323> (Allen Watkins Smith, Ph.D.) [01:24:10] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [01:25:00] Load avg. on willow is OK: OK - load average: 10.85, 13.18, 14.82 [01:28:10] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.579101/1.95, alarm hl:tmp_free=11147M/100M, alarm hl:np_load_avg=1.694336/2.0, alarm hl:mem_free=266.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.579101/2.3, alarm hl:np_load_long=1.841309/2.5, alarm hl:cpu=99.200000/98, alarm hl:mem_free=266.000000M/200M, al [01:29:00] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3252 [01:29:09] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [01:29:10] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3263.000000 [01:29:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:32:10] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [01:32:50] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:34:09] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [01:37:00] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3647 [01:37:17] [[User:Nakaptony]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7274&rcid=9711 * Nakaptony * (+403) (Created page with "Payday financial loans on-line offer you an easy way to get cash in an emergency. The funds advance business tends to make it really convenient for folks to get fast cash when ...") [01:44:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:44:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:59:09] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 127891 [02:07:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 266939 MB (5% inode=30%): [02:13:00] Load avg. on willow is WARNING: WARNING - load average: 15.84, 15.46, 14.52 [02:14:10] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 11016 MB (10% inode=98%): [02:15:00] / on willow is CRITICAL: DISK CRITICAL - free space: / 11015 MB (10% inode=98%): [02:17:10] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.009277/1.95, alarm hl:tmp_free=11012M/100M, alarm hl:np_load_avg=2.047363/2.0, alarm hl:mem_free=541.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.009277/2.3, alarm hl:np_load_long=1.890625/2.5, alarm hl:cpu=100.000000/98, alarm hl:mem_free=541.000000M/200M, a [02:24:01] Load avg. on willow is OK: OK - load average: 12.26, 14.46, 14.74 [02:29:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:32:10] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [02:33:07] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:33:44] Load avg. on willow is WARNING: WARNING - load average: 13.78, 15.58, 15.26 [02:34:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4845.000000 [02:37:14] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4906 [02:44:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:44:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:59:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 128693 [03:00:24] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:01:04] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [03:02:15] Load avg. on willow is WARNING: WARNING - load average: 20.55, 18.66, 16.09 [03:06:44] 3(updated) [VVV-9] closed projects at sul util <10https://jira.toolserver.org/browse/VVV-9> (Amie Suzanne Norton) [03:06:46] 3(updated) [VVV-11] Error on SULUTIL <10https://jira.toolserver.org/browse/VVV-11> (Amie Suzanne Norton) [03:06:48] 3(updated) [VVV-29] Total editcount is wrong <10https://jira.toolserver.org/browse/VVV-29> (Amie Suzanne Norton) [03:06:49] 3(updated) [VVV-30] Malformed table on SUL page off account creation interface with this error "Notice: Undefined variable: color in /home/vvv/public_html/sulutil.php on line 181" <10https://jira.toolserver.org/browse/VVV-30> (Amie Suzanne Norton) [03:06:52] 3(updated) [VVV-15] Please add a line with the "Total databasecount", "Total projectcount", or "Total wikis" to the bottom of sulutil.php <10https://jira.toolserver.org/browse/VVV-15> (Amie Suzanne Norton) [03:06:53] 3(updated) [VVV-8] sulutil.php doesn't find hidden/locked status properly <10https://jira.toolserver.org/browse/VVV-8> (Amie Suzanne Norton) [03:06:56] 3(updated) [VVV-18] YAEC miscalculates edit count percentages <10https://jira.toolserver.org/browse/VVV-18> (Amie Suzanne Norton) [03:06:57] 3(updated) [VVV-2] Provide output before the query is done <10https://jira.toolserver.org/browse/VVV-2> (Amie Suzanne Norton) [03:06:58] 3(updated) [VVV-13] sulutil should trim username input <10https://jira.toolserver.org/browse/VVV-13> (Amie Suzanne Norton) [03:06:59] 3(updated) [VVV-27] Yet Another Edit Counter: Non-ASCII usernames are corrupted <10https://jira.toolserver.org/browse/VVV-27> (Amie Suzanne Norton) [03:06:59] 3(updated) [VVV-21] Unicode/utf-8 issues <10https://jira.toolserver.org/browse/VVV-21> (Amie Suzanne Norton) [03:07:00] 3(updated) [VVV-22] ~vvv/sulutil.php is not working for sinhala names (Unicode/utf-8 issues) <10https://jira.toolserver.org/browse/VVV-22> (Amie Suzanne Norton) [03:07:03] 3(updated) [VVV-4] Please make a similar tool for patrolling <10https://jira.toolserver.org/browse/VVV-4> (Amie Suzanne Norton) [03:07:04] 3(updated) [VVV-20] sulutil.php does not handle some UTF-8 characters in usernames properly <10https://jira.toolserver.org/browse/VVV-20> (Amie Suzanne Norton) [03:07:05] 3(updated) [VVV-16] Please correct the "bug tracker" link on http://toolserver.org/~vvv/ to point to http://jira.toolserver.org/browse/VVV instead of http://jira.ts.wikimedia.org/browse/VVV <10https://jira.toolserver.org/browse/VVV-16> (Amie Suzanne Norton) [03:07:06] 3(updated) [VVV-25] sulutil giving a faulty contrib count <10https://jira.toolserver.org/browse/VVV-25> (Amie Suzanne Norton) [03:07:06] 3(updated) [VVV-24] sulutil: Make link to the "official tool" [[:m:Special:CentralAuth]] more visible (workaround) <10https://jira.toolserver.org/browse/VVV-24> (Amie Suzanne Norton) [03:07:08] 3(updated) [VVV-14] Link home wiki in sulutil <10https://jira.toolserver.org/browse/VVV-14> (Amie Suzanne Norton) [03:07:10] 3(updated) [VVV-6] sulutil.php claims to be XHTML1 compliant, but is not <10https://jira.toolserver.org/browse/VVV-6> (Amie Suzanne Norton) [03:07:11] 3(updated) [VVV-28] Error messages everywhere <10https://jira.toolserver.org/browse/VVV-28> (Amie Suzanne Norton) [03:07:14] 3(updated) [VVV-23] Yet Another Edit Counter fails to load edit count <10https://jira.toolserver.org/browse/VVV-23> (Amie Suzanne Norton) [03:07:24] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 265137 MB (4% inode=30%): [03:07:24] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:09:42] 3(work started) [VVV-14] Link home wiki in sulutil <10https://jira.toolserver.org/browse/VVV-14> (Amie Suzanne Norton) [03:14:24] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 10654 MB (10% inode=98%): [03:15:13] / on willow is CRITICAL: DISK CRITICAL - free space: / 10641 MB (10% inode=98%): [03:17:25] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.030273/1.95, alarm hl:tmp_free=10626M/100M, alarm hl:np_load_avg=2.165527/2.0, alarm hl:mem_free=418.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.030273/2.3, alarm hl:np_load_long=2.080078/2.5, alarm hl:cpu=99.200000/98, alarm hl:mem_free=418.000000M/200M, al [03:18:09] @replag [03:18:09] Magog_the_Ogre: s1-rr-a: 1h 46m 41s [+0.33 s/s]; s1-user: 1h 46m 41s [+0.33 s/s]; s2-user: 44s [-]; s2-user-c: 43s [+0.00 s/s]; s3-rr-a: 57s [+0.00 s/s]; s3-user: 57s [+0.00 s/s]; s4-rr-a: 24s [-]; s4-user: 24s [-] [03:18:11] Magog_the_Ogre: s5-rr-a: 1m 25s [-0.00 s/s]; s5-user: 1m 25s [-0.00 s/s]; s5-user-c: 43s [+0.00 s/s]; s6-rr-a: 1d 11h 48m 54s [+0.47 s/s]; s6-user: 1d 11h 48m 54s [+0.47 s/s] [03:26:14] /sql on cassia is WARNING: DISK WARNING - free space: /sql 126301 MB (10% inode=99%): [03:29:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:32:24] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [03:32:34] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [03:34:06] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:34:14] Load avg. on willow is OK: OK - load average: 10.41, 13.41, 14.93 [03:35:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.583496/1.95, alarm hl:tmp_free=10383M/100M, alarm hl:np_load_avg=1.654297/2.0, alarm hl:mem_free=206.000000M/350M, alarm hl:available=1/0 [03:35:44] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6911.000000 [03:37:14] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6969 [03:44:26] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:44:34] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:59:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 129872 [04:02:24] Load avg. on willow is WARNING: WARNING - load average: 15.44, 15.27, 14.07 [04:03:26] Load avg. on willow is OK: OK - load average: 13.34, 14.68, 13.93 [04:06:25] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.188476/1.10, alarm hl:np_load_long=0.833008/1.55, alarm hl:mem_free=9290.000000M/500M, alarm hl:tmp_free=15006M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.188476/1.00, alarm hl:np_load_long=0.833008/1.50, alarm hl:mem_free=9290.000000M/600M, alarm hl:tmp_free=1 [04:07:04] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:07:25] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 265115 MB (4% inode=30%): [04:12:26] Load avg. on willow is WARNING: WARNING - load average: 21.12, 17.66, 15.36 [04:14:25] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 9825 MB (9% inode=97%): [04:15:24] / on willow is CRITICAL: DISK CRITICAL - free space: / 9810 MB (9% inode=97%): [04:16:24] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:27:25] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.245117/1.10, alarm hl:np_load_long=0.939453/1.55, alarm hl:mem_free=8599.000000M/500M, alarm hl:tmp_free=15004M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.245117/1.00, alarm hl:np_load_long=0.939453/1.50, alarm hl:mem_free=8599.000000M/600M, alarm hl:tmp_free=1 [04:29:25] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:29:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:32:26] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [04:33:34] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [04:34:14] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:35:24] Load avg. on willow is OK: OK - load average: 13.65, 14.00, 14.88 [04:35:44] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8319.000000 [04:36:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.579590/1.95, alarm hl:tmp_free=9481M/100M, alarm hl:np_load_avg=1.712891/2.0, alarm hl:mem_free=129.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.579590/2.3, alarm hl:np_load_long=1.842285/2.5, alarm hl:cpu=92.800000/98, alarm hl:mem_free=129.000000M/200M, ala [04:37:25] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8329 [04:44:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:44:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:46:24] Load avg. on willow is WARNING: WARNING - load average: 17.76, 16.05, 15.10 [04:47:34] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [04:48:24] Load avg. on willow is OK: OK - load average: 12.96, 14.66, 14.68 [05:00:24] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 130754 [05:02:24] Load avg. on willow is WARNING: WARNING - load average: 16.92, 15.94, 15.00 [05:07:24] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 264989 MB (4% inode=30%): [05:15:25] / on willow is CRITICAL: DISK CRITICAL - free space: / 8837 MB (8% inode=97%): [05:15:25] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 8837 MB (8% inode=97%): [05:15:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.131836/1.95, alarm hl:tmp_free=8843M/100M, alarm hl:np_load_avg=1.927734/2.0, alarm hl:mem_free=197.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.131836/2.3, alarm hl:np_load_long=1.909668/2.5, alarm hl:cpu=97.400000/98, alarm hl:mem_free=197.000000M/200M, ala [05:18:35] Load avg. on willow is OK: OK - load average: 14.38, 14.63, 14.96 [05:22:34] Load avg. on willow is WARNING: WARNING - load average: 15.35, 15.82, 15.41 [05:22:44] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [05:27:34] Load avg. on willow is OK: OK - load average: 12.08, 14.20, 14.88 [05:29:33] [[Special:Log/delete]] delete 10 * Matthewrbowker * (deleted "[[02User:Nakaptony10]]": Spam) [05:29:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:30:20] [[Special:Log/block]] block 10 * Matthewrbowker * (blocked [[02User:Nakaptony10]] with an expiry time of infinite (account creation disabled): Spamming links to external sites) [05:30:45] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.441406/1.95, alarm hl:tmp_free=8586M/100M, alarm hl:np_load_avg=1.918945/2.0, alarm hl:mem_free=198.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.441406/2.3, alarm hl:np_load_long=1.890625/2.5, alarm hl:cpu=95.300000/98, alarm hl:mem_free=198.000000M/200M, ala [05:31:43] [[Special:Log/delete]] delete 10 * Matthewrbowker * (deleted "[[02User:Maverickxx787610]]": Spam) [05:31:44] [[Special:Log/block]] block 10 * Matthewrbowker * (blocked [[02User:Maverickxx787610]] with an expiry time of infinite (account creation disabled): Spamming links to external sites) [05:32:44] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [05:35:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:36:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9218.000000 [05:37:24] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9172 [05:44:45] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:44:45] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:47:24] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:50:34] Load avg. on willow is WARNING: WARNING - load average: 24.21, 16.10, 14.30 [05:52:04] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:56:54] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [05:59:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.792480/1.95, alarm hl:tmp_free=8118M/100M, alarm hl:np_load_avg=1.920899/2.0, alarm hl:mem_free=271.000000M/350M, alarm hl:available=1/0 [06:00:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 130778 [06:00:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.21, 19.32, 16.39 [06:01:33] Load avg. on willow is WARNING: WARNING - load average: 22.47, 18.89, 16.43 [06:07:34] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 265258 MB (4% inode=30%): [06:15:33] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 7813 MB (7% inode=97%): [06:15:44] / on willow is CRITICAL: DISK CRITICAL - free space: / 7813 MB (7% inode=97%): [06:20:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 41.83, 26.27, 20.69 [06:25:34] Load avg. on willow is WARNING: WARNING - load average: 15.36, 20.38, 19.88 [06:29:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:32:54] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.258789/1.10, alarm hl:np_load_long=0.635742/1.55, alarm hl:mem_free=9516.000000M/500M, alarm hl:tmp_free=14859M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.258789/1.00, alarm hl:np_load_long=0.635742/1.50, alarm hl:mem_free=9516.000000M/600M, alarm hl:tmp_free=1 [06:32:54] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [06:35:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:36:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8641.000000 [06:38:24] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8624 [06:41:19] [[GeoHack]] 10https://wiki.toolserver.org/w/index.php?diff=7275&oldid=7201&rcid=9716 * Dispenser * (+17) (/* params */ deprecating region:) [06:43:34] Load avg. on willow is OK: OK - load average: 11.62, 12.25, 14.87 [06:44:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:44:55] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:47:34] Load avg. on willow is WARNING: WARNING - load average: 15.59, 15.06, 15.47 [06:48:54] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [06:48:54] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [06:49:34] Load avg. on willow is OK: OK - load average: 10.70, 13.50, 14.84 [06:52:53] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.112305/1.10, alarm hl:np_load_long=1.012695/1.55, alarm hl:mem_free=9802.000000M/500M, alarm hl:tmp_free=14836M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.112305/1.00, alarm hl:np_load_long=1.012695/1.50, alarm hl:mem_free=9802.000000M/600M, alarm hl:tmp_free=1 [06:54:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.582520/1.95, alarm hl:tmp_free=7023M/100M, alarm hl:np_load_avg=1.587402/2.0, alarm hl:mem_free=193.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.582520/2.3, alarm hl:np_load_long=1.761230/2.5, alarm hl:cpu=97.400000/98, alarm hl:mem_free=193.000000M/200M, ala [06:56:55] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [06:57:54] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [07:01:24] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 131585 [07:07:34] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 262772 MB (4% inode=30%): [07:13:34] Load avg. on willow is WARNING: WARNING - load average: 15.38, 15.66, 15.31 [07:15:43] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 6593 MB (6% inode=96%): [07:15:44] / on willow is CRITICAL: DISK CRITICAL - free space: / 6593 MB (6% inode=96%): [07:17:33] Load avg. on willow is OK: OK - load average: 11.51, 13.97, 14.73 [07:24:24] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [07:26:44] /tmp on willow is WARNING: DISK WARNING - free space: / 15674 MB (14% inode=98%): [07:26:44] / on willow is WARNING: DISK WARNING - free space: / 15674 MB (14% inode=98%): [07:27:33] Load avg. on willow is WARNING: WARNING - load average: 14.38, 15.02, 14.92 [07:28:34] Load avg. on willow is OK: OK - load average: 12.67, 14.48, 14.74 [07:29:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:31:45] /tmp on willow is OK: DISK OK - free space: / 23195 MB (22% inode=99%): [07:31:45] / on willow is OK: DISK OK - free space: / 23195 MB (22% inode=99%): [07:32:54] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [07:35:16] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:36:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7576.000000 [07:38:24] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7405 [07:38:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.288574/1.95, alarm hl:tmp_free=32130M/100M, alarm hl:np_load_avg=1.676758/2.0, alarm hl:mem_free=189.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.288574/2.3, alarm hl:np_load_long=1.812012/2.5, alarm hl:cpu=91.000000/98, alarm hl:mem_free=189.000000M/200M, al [07:41:24] MySQL on z-dat-s6-a is OK: Uptime: 1139 Threads: 3 Questions: 78 Slow queries: 0 Opens: 27 Flush tables: 1 Open tables: 18 Queries per second avg: 0.68 [07:45:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:45:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:47:23] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:57:54] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [08:01:33] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 133466 [08:01:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.916992/1.95, alarm hl:tmp_free=51678M/100M, alarm hl:np_load_avg=1.731445/2.0, alarm hl:mem_free=238.000000M/350M, alarm hl:available=1/0 [08:07:04] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:07:34] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 274264 MB (5% inode=31%): [08:07:54] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [08:08:42] 3(created) [MNT-1242] s6 relication running slow since some days; Maintenance; Planned work - user impact <10https://jira.toolserver.org/browse/MNT-1242> (Marlen Caemmerer) [08:24:13] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3567.000000 [08:24:24] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3549 [08:28:54] s4 replag on cassia is CRITICAL: (Service Check Timed Out) [08:29:55] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:30:44] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 310.000000 [08:32:34] Load avg. on willow is WARNING: WARNING - load average: 16.15, 15.03, 13.31 [08:32:54] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [08:33:34] Load avg. on willow is OK: OK - load average: 14.51, 14.78, 13.33 [08:36:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:43:14] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [08:43:34] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2679.000000 [08:45:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:45:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:01:34] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 134204 [09:02:24] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1775.000000 [09:02:24] MySQL slave on thyme is OK: Uptime: 537458 Threads: 6 Questions: 184666686 Slow queries: 72757 Opens: 35823 Flush tables: 1 Open tables: 2862 Queries per second avg: 343.592 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1778 [09:07:34] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 250874 MB (4% inode=29%): [09:07:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.512695/1.95, alarm hl:tmp_free=50375M/100M, alarm hl:np_load_avg=1.484375/2.0, alarm hl:mem_free=245.000000M/350M, alarm hl:available=1/0 [09:16:04] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [09:29:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:33:54] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [09:37:14] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:38:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.165527/1.95, alarm hl:tmp_free=49748M/100M, alarm hl:np_load_avg=1.277344/2.0, alarm hl:mem_free=242.000000M/350M, alarm hl:available=1/0 [09:39:04] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [09:45:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:46:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:53:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.941406/1.95, alarm hl:tmp_free=49284M/100M, alarm hl:np_load_avg=1.058594/2.0, alarm hl:mem_free=249.000000M/350M, alarm hl:available=1/0 [10:01:34] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 134874 [10:07:35] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 231818 MB (4% inode=28%): [10:19:24] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1941 [10:19:24] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1940.000000 [10:29:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:33:59] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [10:37:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:46:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:47:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:53:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.189453/1.95, alarm hl:tmp_free=47972M/100M, alarm hl:np_load_avg=1.212891/2.0, alarm hl:mem_free=245.000000M/350M, alarm hl:available=1/0 [10:56:03] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [11:02:32] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 135319 [11:03:02] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.216797/1.10, alarm hl:np_load_long=0.688477/1.55, alarm hl:mem_free=10785.000000M/500M, alarm hl:tmp_free=14475M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.216797/1.00, alarm hl:np_load_long=0.688477/1.50, alarm hl:mem_free=10785.000000M/600M, alarm hl:tmp_free [11:04:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:07:22] MySQL slave on thyme is OK: Uptime: 544959 Threads: 4 Questions: 186989517 Slow queries: 73196 Opens: 36349 Flush tables: 1 Open tables: 2818 Queries per second avg: 343.125 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1759 [11:07:22] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1754.000000 [11:08:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.452637/1.95, alarm hl:tmp_free=47679M/100M, alarm hl:np_load_avg=1.397949/2.0, alarm hl:mem_free=241.000000M/350M, alarm hl:available=1/0 [11:08:32] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 229776 MB (4% inode=27%): [11:12:32] Load avg. on willow is WARNING: WARNING - load average: 15.91, 14.17, 12.03 [11:14:12] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [11:14:32] Load avg. on willow is OK: OK - load average: 14.26, 14.23, 12.32 [11:15:02] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.813476/1.10, alarm hl:np_load_long=0.867188/1.55, alarm hl:mem_free=10743.000000M/500M, alarm hl:tmp_free=14459M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.813476/1.00, alarm hl:np_load_long=0.867188/1.50, alarm hl:mem_free=10743.000000M/600M, alarm hl:tmp_free [11:24:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:30:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:34:02] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [11:37:40] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:44:19] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.160645/1.95, alarm hl:tmp_free=47003M/100M, alarm hl:np_load_avg=1.209473/2.0, alarm hl:mem_free=292.000000M/350M, alarm hl:available=1/0 [11:45:59] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.171875/1.10, alarm hl:np_load_long=1.645508/1.55, alarm hl:mem_free=10761.000000M/500M, alarm hl:tmp_free=14405M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.171875/1.00, alarm hl:np_load_long=1.645508/1.50, alarm hl:mem_free=10761.000000M/600M, alarm hl:tmp_free [11:46:19] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [11:46:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:47:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:53:59] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:58:19] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.048340/1.95, alarm hl:tmp_free=46848M/100M, alarm hl:np_load_avg=1.082520/2.0, alarm hl:mem_free=322.000000M/350M, alarm hl:available=1/0 [12:02:40] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 135408 [12:03:59] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.199219/1.10, alarm hl:np_load_long=1.213867/1.55, alarm hl:mem_free=9664.000000M/500M, alarm hl:tmp_free=14366M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.199219/1.00, alarm hl:np_load_long=1.213867/1.50, alarm hl:mem_free=9664.000000M/600M, alarm hl:tmp_free=1 [12:05:59] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:08:39] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 228691 MB (4% inode=27%): [12:11:42] 3(updated) [VVV-13] sulutil should trim username input <10https://jira.toolserver.org/browse/VVV-13> (Mike.lifeguard) [12:11:43] 3(updated) [VVV-13] sulutil should trim username input <10https://jira.toolserver.org/browse/VVV-13> (Mike.lifeguard) [12:11:45] 3(updated) [VVV-8] sulutil.php doesn't find hidden/locked status properly <10https://jira.toolserver.org/browse/VVV-8> (Mike.lifeguard) [12:11:47] 3(updated) [VVV-2] Provide output before the query is done <10https://jira.toolserver.org/browse/VVV-2> (Mike.lifeguard) [12:11:49] 3(updated) [VVV-4] Please make a similar tool for patrolling <10https://jira.toolserver.org/browse/VVV-4> (Mike.lifeguard) [12:12:01] <^Mike> Can we block the spammer amiesoso from jira? [12:14:33] <^Mike> and is there a way to undo the spammer's changes all in one shot? [12:17:44] 3(updated) [VVV-15] Please add a line with the "Total databasecount", "Total projectcount", or "Total wikis" to the bottom of sulutil.php <10https://jira.toolserver.org/browse/VVV-15> (Mike.lifeguard) [12:17:47] 3(updated) [VVV-9] closed projects at sul util <10https://jira.toolserver.org/browse/VVV-9> (Mike.lifeguard) [12:19:44] 3(updated) [VVV-30] Malformed table on SUL page off account creation interface with this error "Notice: Undefined variable: color in /home/vvv/public_html/sulutil.php on line 181" <10https://jira.toolserver.org/browse/VVV-30> (Mike.lifeguard) [12:19:45] 3(updated) [VVV-29] Total editcount is wrong <10https://jira.toolserver.org/browse/VVV-29> (Mike.lifeguard) [12:19:45] 3(updated) [VVV-11] Error on SULUTIL <10https://jira.toolserver.org/browse/VVV-11> (Mike.lifeguard) [12:19:49] 3(updated) [VVV-29] Total editcount is wrong <10https://jira.toolserver.org/browse/VVV-29> (Mike.lifeguard) [12:21:44] 3(updated) [VVV-18] YAEC miscalculates edit count percentages <10https://jira.toolserver.org/browse/VVV-18> (Mike.lifeguard) [12:21:45] 3(updated) [VVV-22] ~vvv/sulutil.php is not working for sinhala names (Unicode/utf-8 issues) <10https://jira.toolserver.org/browse/VVV-22> (Mike.lifeguard) [12:21:46] 3(updated) [VVV-22] ~vvv/sulutil.php is not working for sinhala names (Unicode/utf-8 issues) <10https://jira.toolserver.org/browse/VVV-22> (Mike.lifeguard) [12:21:48] 3(updated) [VVV-20] sulutil.php does not handle some UTF-8 characters in usernames properly <10https://jira.toolserver.org/browse/VVV-20> (Mike.lifeguard) [12:23:44] 3(updated) [VVV-21] Unicode/utf-8 issues <10https://jira.toolserver.org/browse/VVV-21> (Mike.lifeguard) [12:23:46] 3(updated) [VVV-28] Error messages everywhere <10https://jira.toolserver.org/browse/VVV-28> (Mike.lifeguard) [12:23:46] 3(updated) [VVV-25] sulutil giving a faulty contrib count <10https://jira.toolserver.org/browse/VVV-25> (Mike.lifeguard) [12:23:47] 3(updated) [VVV-23] Yet Another Edit Counter fails to load edit count <10https://jira.toolserver.org/browse/VVV-23> (Mike.lifeguard) [12:25:44] 3(updated) [VVV-14] Link home wiki in sulutil <10https://jira.toolserver.org/browse/VVV-14> (Mike.lifeguard) [12:25:45] 3(updated) [VVV-6] sulutil.php claims to be XHTML1 compliant, but is not <10https://jira.toolserver.org/browse/VVV-6> (Mike.lifeguard) [12:27:43] 3(updated) [VVV-16] Please correct the "bug tracker" link on http://toolserver.org/~vvv/ to point to http://jira.toolserver.org/browse/VVV instead of http://jira.ts.wikimedia.org/browse/VVV <10https://jira.toolserver.org/browse/VVV-16> (Mike.lifeguard) [12:27:46] 3(updated) [VVV-24] sulutil: Make link to the "official tool" [[:m:Special:CentralAuth]] more visible (workaround) <10https://jira.toolserver.org/browse/VVV-24> (Mike.lifeguard) [12:27:48] 3(updated) [VVV-27] Yet Another Edit Counter: Non-ASCII usernames are corrupted <10https://jira.toolserver.org/browse/VVV-27> (Mike.lifeguard) [12:30:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:34:08] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [12:38:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:46:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:47:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:03:39] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 136065 [13:08:39] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 245561 MB (4% inode=29%): [13:10:19] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.903809/1.95, alarm hl:tmp_free=45999M/100M, alarm hl:np_load_avg=1.028320/2.0, alarm hl:mem_free=219.000000M/350M, alarm hl:available=1/0 [13:16:19] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [13:30:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:30:26] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.986816/1.95, alarm hl:tmp_free=45446M/100M, alarm hl:np_load_avg=0.960938/2.0, alarm hl:mem_free=196.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.986816/2.3, alarm hl:np_load_long=1.018066/2.5, alarm hl:cpu=77.400000/98, alarm hl:mem_free=196.000000M/200M, al [13:34:09] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [13:38:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:47:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:47:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:03:51] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 135945 [14:08:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 242842 MB (4% inode=28%): [14:30:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:34:09] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [14:38:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:47:19] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:47:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:04:00] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 136217 [15:08:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 240647 MB (4% inode=28%): [15:26:59] /sql on cassia is WARNING: DISK WARNING - free space: /sql 125393 MB (10% inode=99%): [15:30:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:31:47] 3(created) [ACCAPP-512] RENDER project tool development; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-512> (Andriy Rodchenko) [15:34:20] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [15:37:29] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:39:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:42:09] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [15:47:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:48:19] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:04:07] [[Bots]] ! 10https://wiki.toolserver.org/w/index.php?diff=7276&oldid=5861&rcid=9717 * Mgedawy * (+23) (Correction) [16:05:00] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 137430 [16:09:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 239997 MB (4% inode=28%): [16:13:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.828125/1.95, alarm hl:tmp_free=45018M/100M, alarm hl:np_load_avg=0.834961/2.0, alarm hl:mem_free=319.000000M/350M, alarm hl:available=1/0 [16:14:28] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [16:17:29] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:30:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:34:29] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [16:39:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:47:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:48:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:01:42] 3(commented) [ET-48] Newsletter ready for delivery <10https://jira.toolserver.org/browse/ET-48> (Keith Dorey) [17:05:00] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 138472 [17:10:01] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 236998 MB (4% inode=28%): [17:12:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.748047/1.95, alarm hl:tmp_free=44927M/100M, alarm hl:np_load_avg=0.738769/2.0, alarm hl:mem_free=284.000000M/350M, alarm hl:available=1/0 [17:13:31] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [17:30:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:34:32] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [17:39:40] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:47:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:48:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:04:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.917481/1.95, alarm hl:tmp_free=44710M/100M, alarm hl:np_load_avg=1.018555/2.0, alarm hl:mem_free=212.000000M/350M, alarm hl:available=1/0 [18:05:01] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 138995 [18:06:31] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [18:10:01] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 234310 MB (4% inode=28%): [18:12:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.034668/1.95, alarm hl:tmp_free=44637M/100M, alarm hl:np_load_avg=1.052734/2.0, alarm hl:mem_free=297.000000M/350M, alarm hl:available=1/0 [18:30:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:34:32] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [18:39:40] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:47:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:48:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:02:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.284668/1.95, alarm hl:tmp_free=44408M/100M, alarm hl:np_load_avg=1.212891/2.0, alarm hl:mem_free=278.000000M/350M, alarm hl:available=1/0 [19:03:32] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [19:05:02] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 139524 [19:06:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.874023/1.95, alarm hl:tmp_free=44378M/100M, alarm hl:np_load_avg=1.034180/2.0, alarm hl:mem_free=341.000000M/350M, alarm hl:available=1/0 [19:10:11] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 231502 MB (4% inode=27%): [19:30:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:35:31] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [19:37:38] hello all [19:39:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:47:41] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.835449/1.95, alarm hl:tmp_free=44277M/100M, alarm hl:np_load_avg=0.756836/2.0, alarm hl:mem_free=333.000000M/350M, alarm hl:available=1/0 [19:47:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:48:42] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:50:41] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [20:05:01] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 141265 [20:05:25] @replag [20:05:25] DaBPunkt: s6-rr-a: 1d 15h 14m 27s [+0.20 s/s]; s6-user: 1d 15h 14m 27s [+0.20 s/s] [20:10:10] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 228777 MB (4% inode=27%): [20:12:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.929199/1.95, alarm hl:tmp_free=44231M/100M, alarm hl:np_load_avg=0.803223/2.0, alarm hl:mem_free=225.000000M/350M, alarm hl:available=1/0 [20:13:42] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [20:20:11] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:21:51] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [20:30:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:35:41] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [20:36:42] 3(created) [ACCAPP-513] Request for new account for Oxford affiliated project for analysing Wikipedia editing communities across different wiki languages; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-513> (Ahmed Medhat) [20:39:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:40:11] /sql on thyme is WARNING: DISK WARNING - free space: /sql 191848 MB (19% inode=99%): [20:44:41] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.930664/1.95, alarm hl:tmp_free=44148M/100M, alarm hl:np_load_avg=0.927246/2.0, alarm hl:mem_free=242.000000M/350M, alarm hl:available=1/0 [20:46:41] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [20:48:03] DaBPunkt: i got this error: [20:48:04] *** libc thread failure: cannot allocate thread hash table [20:48:06] /sge62/bin/sol-amd64/qcronsub: fork: Not enough space [20:48:11] /sql on thyme is OK: DISK OK - free space: /sql 203357 MB (21% inode=99%): [20:48:15] qcronsub problems again? [20:48:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:49:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:58:38] Alchimista: no, it's a memory-problem. Do you know from which box it came? [21:00:50] DaBPunkt: how can i see wich box? [21:01:18] Alchimista: if you got the error by mail, the box-name is somewhere in the header [21:01:48] hawthorn ? [21:02:21] Load avg. on willow is WARNING: WARNING - load average: 18.09, 14.90, 10.64 [21:02:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.075195/1.95, alarm hl:tmp_free=44023M/100M, alarm hl:np_load_avg=1.834473/2.0, alarm hl:mem_free=1288.000000M/350M, alarm hl:available=1/0 [21:02:49] Alchimista:mm, hawthorn is special, because it is also the mail-server… [21:03:21] Load avg. on willow is OK: OK - load average: 11.48, 13.56, 10.44 [21:03:27] DaBPunkt: the header is: Cron followed by the cron command [21:03:35] ah yes [21:03:53] and submited @submit [21:04:39] DaBPunkt: two days ago i got: /sge62/default/common/settings.sh: fork: Not enough space [21:05:11] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 142803 [21:07:30] I freed some memory on hawthorn now; that's all O can do at the moment [21:09:01] thanks [21:09:34] DaBPunkt: how can i check/control how many memory my scripts are using? [21:10:21] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 224335 MB (4% inode=27%): [21:12:52] Alchimista: "man ps". Something like "ps -ef -o,user,comm,rss|grep alchimista" should do [21:31:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:36:15] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [21:40:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:49:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:49:37] @replag [21:49:37] DaBPunkt: s2-user: 22m 41s [+0.02 s/s]; s2-user-c: 3m 37s [+0.00 s/s]; s5-user-c: 3m 38s [+0.00 s/s]; s6-rr-a: 1d 15h 45m 10s [+0.29 s/s]; s6-user: 1d 15h 45m 10s [+0.29 s/s] [21:50:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:02:56] nacht ts [22:05:13] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 143392 [22:10:32] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 221721 MB (4% inode=27%): [22:32:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:36:16] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [22:40:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:49:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:50:15] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:05:16] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 145460 [23:05:35] /sql on thyme is WARNING: DISK WARNING - free space: /sql 189348 MB (19% inode=99%): [23:10:16] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1888 [23:10:55] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 219262 MB (4% inode=26%): [23:11:05] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1920.000000 [23:18:34] /sql on thyme is OK: DISK OK - free space: /sql 213086 MB (22% inode=99%): [23:32:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:34:07] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1779.000000 [23:34:17] MySQL slave on thyme is OK: Uptime: 589766 Threads: 11 Questions: 202224993 Slow queries: 80084 Opens: 38420 Flush tables: 1 Open tables: 2833 Queries per second avg: 342.890 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1774 [23:34:43] 3(resolved) [ET-47] AdminStatsBot dormant <10https://jira.toolserver.org/browse/ET-47> (JamesR) [23:36:17] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [23:36:43] 3(work started) [ET-48] Newsletter ready for delivery <10https://jira.toolserver.org/browse/ET-48> (JamesR) [23:38:43] 3(resolved) [ET-46] April Newsletter <10https://jira.toolserver.org/browse/ET-46> (JamesR) [23:38:44] 3(resolved) [ET-45] February newsletter delivery <10https://jira.toolserver.org/browse/ET-45> (JamesR) [23:38:46] 3(commented) [ET-48] Newsletter ready for delivery <10https://jira.toolserver.org/browse/ET-48> (JamesR) [23:40:37] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:49:18] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:50:18] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:59:44] 3(resolved) [ET-48] Newsletter ready for delivery <10https://jira.toolserver.org/browse/ET-48> (JamesR)