[00:03:43] 3(created) [TS-1384] Add wikimania2013 wiki to database; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1384> (Thehelpfulone) [00:05:17] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 147608 [00:10:57] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 214127 MB (4% inode=26%): [00:14:44] 3(updated) [TS-1384] Add wikimania2013 wiki to database <10https://jira.toolserver.org/browse/TS-1384> (Jesse Plamondon-Willard) [00:27:17] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.808106/1.95, alarm hl:tmp_free=43400M/100M, alarm hl:np_load_avg=0.888184/2.0, alarm hl:mem_free=255.000000M/350M, alarm hl:available=1/0 [00:30:27] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [00:30:37] /sql on thyme is WARNING: DISK WARNING - free space: /sql 192191 MB (20% inode=99%): [00:32:27] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:33:07] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1913.000000 [00:33:22] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1918 [00:33:27] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.779785/1.95, alarm hl:tmp_free=43381M/100M, alarm hl:np_load_avg=0.911133/2.0, alarm hl:mem_free=226.000000M/350M, alarm hl:available=1/0 [00:36:27] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [00:40:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:49:27] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:51:17] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:52:57] /sql on thyme is OK: DISK OK - free space: /sql 239415 MB (25% inode=99%): [01:05:18] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 150089 [01:09:07] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3619.000000 [01:09:29] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3627 [01:09:35] @replag [01:09:36] Betacommand: s1-rr-a: 1h 51s [-0.03 s/s]; s1-user: 1h 51s [-0.03 s/s]; s2-user: 31s [-0.11 s/s]; s2-user-c: 12s [-0.02 s/s]; s3-rr-a: 1m 54s [+0.00 s/s]; s3-user: 1m 54s [+0.00 s/s]; s4-rr-a: 12s [-0.00 s/s]; s4-user: 12s [-0.00 s/s] [01:09:37] Betacommand: s5-user-c: 12s [-0.02 s/s]; s6-rr-a: 1d 17h 44m 10s [+0.60 s/s]; s6-user: 1d 17h 44m 10s [+0.60 s/s] [01:11:00] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 213898 MB (4% inode=26%): [01:14:42] 3(created) [TS-1385] S6 replag issues; Toolserver: Databases; Bug <10https://jira.toolserver.org/browse/TS-1385> (Betacommand) [01:26:43] 3(assigned) [TS-1225] Roger Mbiama <10https://jira.toolserver.org/browse/TS-1225> (Betacommand) [01:28:43] 3(resolved) [TS-1225] Roger Mbiama <10https://jira.toolserver.org/browse/TS-1225> (Betacommand) [01:28:46] 3(assigned) [TS-1116] Why indexes of mysql tables are not set? <10https://jira.toolserver.org/browse/TS-1116> (Betacommand) [01:30:44] 3(resolved) [TS-1116] Why indexes of mysql tables are not set? <10https://jira.toolserver.org/browse/TS-1116> (Betacommand) [01:30:47] 3(assigned) [TS-866] Account expired and can not login <10https://jira.toolserver.org/browse/TS-866> (Betacommand) [01:30:48] 3(assigned) [TS-866] Account expired and can not login <10https://jira.toolserver.org/browse/TS-866> (Betacommand) [01:30:52] 3(assigned) [TS-932] Handle step 2 of the interwiki-bot-situation <10https://jira.toolserver.org/browse/TS-932> (Betacommand) [01:30:54] 3(assigned) [TS-494] Amalgamate all interwiki-bots <10https://jira.toolserver.org/browse/TS-494> (Betacommand) [01:31:04] 3(resolved) [TS-932] Handle step 2 of the interwiki-bot-situation <10https://jira.toolserver.org/browse/TS-932> (Betacommand) [01:32:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:32:45] 3(resolved) [TS-494] Amalgamate all interwiki-bots <10https://jira.toolserver.org/browse/TS-494> (Betacommand) [01:36:39] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [01:41:08] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:49:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:51:40] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:05:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 151851 [02:07:29] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:09:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4299.000000 [02:10:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4287 [02:11:00] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 212789 MB (4% inode=26%): [02:33:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:34:59] /sql on thyme is WARNING: DISK WARNING - free space: /sql 196580 MB (20% inode=99%): [02:36:49] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [02:41:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:48:59] /sql on thyme is OK: DISK OK - free space: /sql 223974 MB (23% inode=99%): [02:49:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:51:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:52:09] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:05:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 153924 [03:07:29] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:09:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4661.000000 [03:10:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4672 [03:11:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 210972 MB (3% inode=26%): [03:24:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.289551/1.95, alarm hl:tmp_free=43071M/100M, alarm hl:np_load_avg=1.046387/2.0, alarm hl:mem_free=253.000000M/350M, alarm hl:available=1/0 [03:26:49] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [03:27:49] /sql on cassia is WARNING: DISK WARNING - free space: /sql 126528 MB (10% inode=99%): [03:29:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.288574/1.95, alarm hl:tmp_free=43061M/100M, alarm hl:np_load_avg=1.225586/2.0, alarm hl:mem_free=282.000000M/350M, alarm hl:available=1/0 [03:33:51] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:36:59] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [03:41:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:45:59] /sql on thyme is WARNING: DISK WARNING - free space: /sql 194187 MB (20% inode=99%): [03:49:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:51:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:02:00] /sql on thyme is OK: DISK OK - free space: /sql 237080 MB (24% inode=99%): [04:05:39] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 155063 [04:09:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5647.000000 [04:10:39] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5641 [04:11:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 210941 MB (3% inode=26%): [04:26:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.180664/1.95, alarm hl:tmp_free=42973M/100M, alarm hl:np_load_avg=1.043945/2.0, alarm hl:mem_free=278.000000M/350M, alarm hl:available=1/0 [04:26:59] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [04:33:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:37:09] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [04:41:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:49:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:52:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:05:39] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 155685 [05:09:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5500.000000 [05:10:40] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5405 [05:12:09] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 210863 MB (3% inode=26%): [05:33:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:37:09] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [05:41:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:45:49] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3588 [05:46:09] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3565.000000 [05:46:59] /sql on cassia is OK: DISK OK - free space: /sql 130887 MB (11% inode=99%): [05:49:59] /sql on cassia is WARNING: DISK WARNING - free space: /sql 126275 MB (10% inode=99%): [05:49:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:52:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:05:50] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 156199 [06:10:08] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1773.000000 [06:10:50] MySQL slave on thyme is OK: Uptime: 613566 Threads: 3 Questions: 214070015 Slow queries: 83785 Opens: 39623 Flush tables: 1 Open tables: 2868 Queries per second avg: 348.894 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1728 [06:12:12] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 210482 MB (3% inode=26%): [06:34:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:37:10] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [06:41:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:46:10] /sql on thyme is WARNING: DISK WARNING - free space: /sql 198529 MB (20% inode=99%): [06:50:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:52:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:53:09] /sql on thyme is OK: DISK OK - free space: /sql 230904 MB (24% inode=99%): [07:06:01] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 157016 [07:12:09] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 210383 MB (3% inode=26%): [07:34:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:37:09] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [07:41:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:50:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:52:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:06:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38597 MB (9% inode=99%): [08:06:49] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 158132 [08:12:20] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 210253 MB (3% inode=26%): [08:15:28] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 53398 MB (13% inode=99%): [08:24:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40265 MB (9% inode=99%): [08:34:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:34:30] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22848 MB (5% inode=99%): [08:37:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [08:41:38] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:44:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39585 MB (9% inode=99%): [08:47:44] 3(commented) [MAGNUS-323] Catscan not picking up templates from talk pages <10https://jira.toolserver.org/browse/MAGNUS-323> (Magnus Manske) [08:50:30] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:52:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:56:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 77635 MB (19% inode=99%): [09:06:59] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 159207 [09:12:29] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 210083 MB (3% inode=26%): [09:34:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:37:41] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [09:41:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:42:44] 3(created) [ACC-249] Account expiration; ACC: Logging; Support request <10https://jira.toolserver.org/browse/ACC-249> (phe) [09:50:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:52:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:06:59] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 159856 [10:12:40] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 209919 MB (3% inode=26%): [10:15:25] Betacommand: around? [10:34:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:37:40] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [10:41:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:50:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:52:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:59:40] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.061524/1.00, alarm hl:np_load_long=0.550781/1.50, alarm hl:mem_free=11713.000000M/600M, alarm hl:tmp_free=14405M/100M, alarm hl:available=1/0 [11:00:39] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:04:39] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.055664/1.00, alarm hl:np_load_long=0.678711/1.50, alarm hl:mem_free=11608.000000M/600M, alarm hl:tmp_free=14394M/100M, alarm hl:available=1/0 [11:07:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 160316 [11:12:40] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 209744 MB (3% inode=26%): [11:34:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:37:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.444824/1.95, alarm hl:tmp_free=41740M/100M, alarm hl:np_load_avg=0.498535/2.0, alarm hl:mem_free=300.000000M/350M, alarm hl:available=1/0 [11:37:49] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [11:38:49] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [11:42:00] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:42:48] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.438476/1.95, alarm hl:tmp_free=41730M/100M, alarm hl:np_load_avg=0.471680/2.0, alarm hl:mem_free=341.000000M/350M, alarm hl:available=1/0 [11:43:46] 3(assigned) [TS-1384] Add wikimania2013 wiki to database <10https://jira.toolserver.org/browse/TS-1384> (Marlen Caemmerer) [11:47:51] [[Special:Log/newusers]] create 10 * Abyes * (New user account) [11:49:43] 3(resolved) [MAGNUS-323] Catscan not picking up templates from talk pages <10https://jira.toolserver.org/browse/MAGNUS-323> (Magnus Manske) [11:50:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:52:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:07:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 161042 [12:08:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.880859/1.95, alarm hl:tmp_free=41670M/100M, alarm hl:np_load_avg=0.791504/2.0, alarm hl:mem_free=233.000000M/350M, alarm hl:available=1/0 [12:08:59] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [12:12:49] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 209587 MB (3% inode=25%): [12:12:59] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.663574/1.95, alarm hl:tmp_free=41657M/100M, alarm hl:np_load_avg=0.740723/2.0, alarm hl:mem_free=260.000000M/350M, alarm hl:available=1/0 [12:24:56] [[User:Abyes]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7277&rcid=9719 * Abyes * (+149) (Created page with "Alan Taylor is a keen enthusiast of Abs Routines. My Webpage: [http://abyes.net/ab-workout-ab-scissors-step-by-step-instructions/ stomach workouts]") [12:33:20] [[User:Abyes]] ! 10https://wiki.toolserver.org/w/index.php?diff=7278&oldid=7277&rcid=9720 * Phe * (-149) (blanked, spam) [12:34:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:37:43] 3(moved) [TS-1386] Account expiration <10https://jira.toolserver.org/browse/TS-1386> (Simon Walker) [12:38:00] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [12:42:08] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:50:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:52:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:07:18] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 162603 [13:12:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 209434 MB (3% inode=25%): [13:14:22] Danny_B|backup: yeah [13:35:06] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:38:00] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [13:42:50] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:51:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:52:43] 3(commented) [MAGNUS-316] link to ESIS library in cas.php to be changed <10https://jira.toolserver.org/browse/MAGNUS-316> (Magnus Manske) [13:53:01] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:07:40] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 163777 [14:13:00] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 237399 MB (4% inode=28%): [14:22:30] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:27:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.554199/1.95, alarm hl:tmp_free=41345M/100M, alarm hl:np_load_avg=0.583008/2.0, alarm hl:mem_free=324.000000M/350M, alarm hl:available=1/0 [14:27:19] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [14:28:02] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [14:35:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:38:00] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [14:43:00] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:51:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:53:00] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:06:27] hello all [15:06:56] hi DaBPunkt [15:07:40] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 164433 [15:13:00] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 237182 MB (4% inode=28%): [15:35:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:38:20] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [15:38:42] 3(updated) [VVV-15] Please add a line with the "Total databasecount", "Total projectcount", or "Total wikis" to the bottom of sulutil.php <10https://jira.toolserver.org/browse/VVV-15> (Jeff G.) [15:43:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:44:09] what's the progress of yarrow and other linux boxes? [15:49:45] fine [15:51:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:51:20] * Danny_B|backup wonders why people run same expensive programs in multiple instances [15:51:33] it eats lot of resources :-/ [15:52:56] shouldn't that be limited somehow? [15:53:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:07:40] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 165475 [16:13:10] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 225025 MB (4% inode=27%): [16:13:20] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.678711/1.95, alarm hl:tmp_free=41138M/100M, alarm hl:np_load_avg=0.720215/2.0, alarm hl:mem_free=301.000000M/350M, alarm hl:available=1/0 [16:15:20] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [16:23:40] Danny_B|backup: depends on what you mean, I run 4 of the same bot with different configurations [16:24:03] that's different [16:24:19] but i saw somebody running exactly same command [16:25:21] Danny_B|backup: let them know the odds are they dont realize what they are doing [16:27:10] if you run top with displaying full command line and order it by time, you'll see what i mean [16:35:35] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:38:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [16:43:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:51:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:53:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:07:46] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 166081 [17:14:05] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 233753 MB (4% inode=28%): [17:35:36] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:38:34] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [17:43:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:50:05] /sql on cassia is WARNING: DISK WARNING - free space: /sql 124752 MB (10% inode=99%): [17:51:35] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:53:45] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:58:22] [[Special:Log/newusers]] create 10 * Couponexpert1212 * (New user account) [18:08:45] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 166353 [18:09:35] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.516601/1.10, alarm hl:np_load_long=0.645508/1.55, alarm hl:mem_free=10428.000000M/500M, alarm hl:tmp_free=13688M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.516601/1.00, alarm hl:np_load_long=0.645508/1.50, alarm hl:mem_free=10428.000000M/600M, alarm hl:tmp_free [18:10:35] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [18:12:35] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.787109/1.95, alarm hl:tmp_free=40784M/100M, alarm hl:np_load_avg=0.747559/2.0, alarm hl:mem_free=304.000000M/350M, alarm hl:available=1/0 [18:13:35] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [18:14:15] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 233630 MB (4% inode=28%): [18:22:35] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.486328/1.95, alarm hl:tmp_free=40743M/100M, alarm hl:np_load_avg=0.547852/2.0, alarm hl:mem_free=290.000000M/350M, alarm hl:available=1/0 [18:35:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:38:45] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [18:43:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:51:45] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:53:45] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:09:04] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 166971 [19:14:22] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 233491 MB (4% inode=28%): [19:36:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:38:56] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [19:43:55] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:52:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:54:04] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:09:55] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 168192 [20:14:55] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 232246 MB (4% inode=28%): [20:36:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:37:31] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:39:11] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [20:42:11] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:42:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.738769/1.95, alarm hl:tmp_free=40205M/100M, alarm hl:np_load_avg=0.690430/2.0, alarm hl:mem_free=281.000000M/350M, alarm hl:available=1/0 [20:43:10] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [20:44:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:52:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:54:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:10:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 168831 [21:12:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.896484/1.95, alarm hl:tmp_free=40000M/100M, alarm hl:np_load_avg=0.873047/2.0, alarm hl:mem_free=216.000000M/350M, alarm hl:available=1/0 [21:13:10] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [21:15:10] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 229509 MB (4% inode=27%): [21:34:14] @replag [21:34:14] Matthew_: s1-rr-a: 9m 44s [-0.04 s/s]; s1-user: 9m 44s [-0.04 s/s]; s3-rr-a: 13s [-0.00 s/s]; s3-user: 13s [-0.00 s/s]; s6-rr-a: 1d 22h 55m 44s [+0.25 s/s]; s6-user: 1d 22h 55m 44s [+0.25 s/s] [21:36:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:36:40] [[Special:Log/newusers]] create 10 * Portika * (New user account) [21:39:11] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [21:43:29] [[User:Portika]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7279&rcid=9723 * Portika * (+172) (Created page with "Well what can I say about me? I am enthusiastic about many things really (to much for you to mention). My Site: [http://website-submission-seo.blogspot.com SEO software]") [21:43:46] [[Special:Log/delete]] delete 10 * Matthewrbowker * (deleted "[[02User:Abyes10]]": Spam) [21:43:58] [[Special:Log/block]] block 10 * Matthewrbowker * (blocked [[02User:Abyes10]] with an expiry time of infinite (account creation disabled): Spamming links to external sites) [21:44:18] [[Special:Log/delete]] delete 10 * Matthewrbowker * (deleted "[[02User:Portika10]]": Spam) [21:44:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:46:45] [[Special:Log/block]] block 10 * Matthewrbowker * (blocked [[02User:Portika10]] with an expiry time of infinite (account creation disabled): Spamming links to external sites) [21:52:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:53:02] Lars Aronsson * [Toolserver-l] Toolserver user phe account expired [21:54:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:05:12] pffff [22:06:02] K. Peachey * Re: [Toolserver-l] Toolserver user phe account expired [22:07:02] phil.el * Re: [Toolserver-l] [Wikisource-l] Toolserver user phe account expired [22:08:02] DaB. * Re: [Toolserver-l] Toolserver user phe account expired [22:10:11] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 168967 [22:13:02] DaB. * Re: [Toolserver-l] [Wikisource-l] Toolserver user phe account expired [22:13:34] thanks Danny_B|backup [22:14:57] that's not me ;-) [22:15:11] but DaBPunkt who's not here atm [22:15:25] oops :) [22:16:10] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 229347 MB (4% inode=27%): [22:17:34] phe: no prob, you're not the first nor the latest ;-) [22:22:40] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:37:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:39:11] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [22:44:30] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:53:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:55:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:10:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 168495 [23:12:12] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [23:17:10] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 229249 MB (4% inode=27%): [23:37:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:39:19] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [23:44:40] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:53:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:55:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default