[00:00:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:02:14] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:03:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:03:34] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:10:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:13:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:16:24] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [00:24:25] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 303297 MB (5% inode=33%): [00:38:23] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:46:04] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.552734/1.10, alarm hl:np_load_long=0.797852/1.55, alarm hl:mem_free=12376.000000M/500M, alarm hl:tmp_free=13798M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.552734/1.00, alarm hl:np_load_long=0.797852/1.50, alarm hl:mem_free=12376.000000M/600M, alarm hl:tmp_free [00:47:04] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [01:00:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:03:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:08:34] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:10:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:11:04] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.041992/1.00, alarm hl:np_load_long=0.690430/1.50, alarm hl:mem_free=12398.000000M/600M, alarm hl:tmp_free=13764M/100M, alarm hl:available=1/0 [01:12:04] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [01:13:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:16:34] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [01:25:26] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 303221 MB (5% inode=33%): [01:33:05] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.105469/1.10, alarm hl:np_load_long=0.763672/1.55, alarm hl:mem_free=12289.000000M/500M, alarm hl:tmp_free=13736M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.105469/1.00, alarm hl:np_load_long=0.763672/1.50, alarm hl:mem_free=12289.000000M/600M, alarm hl:tmp_free [01:34:04] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [02:00:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:04:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:10:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:13:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.207031/1.10, alarm hl:np_load_long=0.853515/1.55, alarm hl:mem_free=12117.000000M/500M, alarm hl:tmp_free=13729M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.207031/1.00, alarm hl:np_load_long=0.853515/1.50, alarm hl:mem_free=12117.000000M/600M, alarm hl:tmp_free [02:13:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:15:04] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [02:16:33] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [02:26:24] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 301803 MB (5% inode=33%): [03:00:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:04:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:10:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:13:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.212891/1.10, alarm hl:np_load_long=0.841797/1.55, alarm hl:mem_free=11726.000000M/500M, alarm hl:tmp_free=15019M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.212891/1.00, alarm hl:np_load_long=0.841797/1.50, alarm hl:mem_free=11726.000000M/600M, alarm hl:tmp_free [03:13:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:14:04] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [03:16:33] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [03:26:24] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 300963 MB (5% inode=33%): [03:33:04] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.481445/1.10, alarm hl:np_load_long=0.845703/1.55, alarm hl:mem_free=11697.000000M/500M, alarm hl:tmp_free=15020M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.481445/1.00, alarm hl:np_load_long=0.845703/1.50, alarm hl:mem_free=11697.000000M/600M, alarm hl:tmp_free [03:45:15] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [03:45:34] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:45:44] MySQL slave on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [03:49:14] MySQL on thyme is OK: Uptime: 265 Threads: 15 Questions: 2197 Slow queries: 1 Opens: 112 Flush tables: 1 Open tables: 102 Queries per second avg: 8.290 [03:49:34] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 356.000000 [03:49:44] MySQL slave on thyme is OK: Uptime: 297 Threads: 6 Questions: 9147 Slow queries: 12 Opens: 163 Flush tables: 1 Open tables: 153 Queries per second avg: 30.797 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 343 [03:59:47] 3(created) [TS-1380] Crontab for MMT cvn on willow is not running; Toolserver; Bug <10https://jira.toolserver.org/browse/TS-1380> (Krinkle) [04:00:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:04:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:10:04] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.430664/1.10, alarm hl:np_load_long=1.163086/1.55, alarm hl:mem_free=11549.000000M/500M, alarm hl:tmp_free=14976M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.430664/1.00, alarm hl:np_load_long=1.163086/1.50, alarm hl:mem_free=11549.000000M/600M, alarm hl:tmp_free [04:10:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:13:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:16:34] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [04:18:04] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:21:04] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.043945/1.00, alarm hl:np_load_long=1.284180/1.50, alarm hl:mem_free=11715.000000M/600M, alarm hl:tmp_free=14985M/100M, alarm hl:available=1/0 [04:22:04] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:26:23] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 286227 MB (5% inode=32%): [04:37:14] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:38:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:50:14] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:00:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:04:04] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.475586/1.10, alarm hl:np_load_long=0.883789/1.55, alarm hl:mem_free=11016.000000M/500M, alarm hl:tmp_free=14950M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.475586/1.00, alarm hl:np_load_long=0.883789/1.50, alarm hl:mem_free=11016.000000M/600M, alarm hl:tmp_free [05:09:04] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [05:10:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:10:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:12:14] SMF on willow is UNKNOWN: NRPE: Call to fork() failed [05:13:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:13:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:16:34] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [05:23:34] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:26:23] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 285877 MB (5% inode=32%): [05:43:15] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:44:34] Load avg. on willow is WARNING: WARNING - load average: 15.10, 13.81, 11.56 [05:44:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.976562/1.95, alarm hl:tmp_free=41241M/100M, alarm hl:np_load_avg=1.741211/2.0, alarm hl:mem_free=977.000000M/350M, alarm hl:available=1/0 [05:45:34] Load avg. on willow is OK: OK - load average: 10.50, 12.75, 11.32 [05:45:34] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:56:33] Load avg. on willow is WARNING: WARNING - load average: 17.77, 12.96, 11.55 [05:56:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.250488/1.95, alarm hl:tmp_free=41142M/100M, alarm hl:np_load_avg=1.620117/2.0, alarm hl:mem_free=964.000000M/350M, alarm hl:available=1/0 [05:58:37] * CoalBalls pokes Dispenser [06:00:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:00:35] Oh no! [06:00:42] * Dispenser pokes CoalBalls [06:01:31] Dispenser: Do you have a function somewhere that can match nested templates-and-parameters-and-other-stuff? [06:02:23] Its been on the todo list. Actually I think tim1357 posted something IIRC WP:BOTR [06:05:54] OK [06:06:06] What are you trying to do? [06:06:23] Match something like {{hello|world={{foo}}}} [06:06:32] Match for what purpose? [06:06:37] or {{hello|world={unbalanced}} [06:06:43] For extracting templates and parameters [06:06:49] I have a function to strip templates. [06:06:59] The unbalanced thing can't really be caught... [06:07:46] When you learn that parameters names can be dynamic, you really give up hope of parsing. [06:10:13] I see [06:10:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:11:07] Dispenser: Isn't TemplateTiger doing template parsing or something? [06:11:29] It only has like 150 k entries [06:11:35] Right. [06:11:40] But I think he's looking to re-use code. [06:12:00] https://de.wikipedia.org/wiki/Wikipedia:WikiProjekt_Vorlagenauswertung/en [06:12:25] https://en.wikipedia.org/wiki/Wikipedia:Database_reports/Articles_containing_invalid_template_parameters/Configuration [06:12:29] That was my attempt at parsing. [06:13:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:14:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:16:34] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [06:26:23] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 285489 MB (5% inode=32%): [06:34:08] Dispenser: But do you know what sort of logic would go into making such a template parser? [06:34:20] I'm thinking tokenize, but I've only used it once before [06:34:32] (and have no idea if it is the right solution or not) [06:37:35] * CoalBalls pokes Dispenser and points up [06:38:20] According to mediawiki.org the new-preprocess converts wikitext to an XML tree [06:40:14] Tree parsing is complicated by {{ and {{{ aren't easily distinguished [06:43:04] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.401367/1.10, alarm hl:np_load_long=0.804688/1.55, alarm hl:mem_free=11816.000000M/500M, alarm hl:tmp_free=14799M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.401367/1.00, alarm hl:np_load_long=0.804688/1.50, alarm hl:mem_free=11816.000000M/600M, alarm hl:tmp_free [06:46:04] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [06:46:57] well actually }} and }}} [06:53:04] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.577149/1.10, alarm hl:np_load_long=0.923828/1.55, alarm hl:mem_free=12414.000000M/500M, alarm hl:tmp_free=14793M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.577149/1.00, alarm hl:np_load_long=0.923828/1.50, alarm hl:mem_free=12414.000000M/600M, alarm hl:tmp_free [06:53:26] Yeah, this is impossible [06:53:49] Apparently {{{bla{h}}} is a valid parameter [06:58:26] No it isn't. [06:58:42] Parameter names have to be legal titles in MediaWiki. [06:58:45] Minus one or two characters. [06:58:50] Yes it is, check the WP:SANDBOX and Template:X3 [07:00:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:00:26] Parameter names can't contain "{". [07:00:41] "{" isn't a legal title character. [07:01:46] Hmm. [07:01:47] No. [07:01:50] You're right. [07:01:52] wtf [07:02:02] God that's atrocious. [07:06:03] Well, no chance of making a foolproof template parser now [07:06:18] * CoalBalls grumbles [07:06:22] Good bye, all [07:10:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:13:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:14:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:16:34] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [07:22:55] [[Special:Log/newusers]] create 10 * Picardo * (New user account) [07:26:23] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 285368 MB (5% inode=32%): [07:33:34] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:38:34] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:00:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:10:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:13:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:14:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:16:33] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [08:26:24] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 285255 MB (5% inode=32%): [08:32:55] [[User:Picardo]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7228&rcid=9637 * Picardo * (+502) (Created page with "Monster-Trucks sind gewiss der Traum für etliche Actionfans. Jedoch ein Monstertruck selbst steuern muss schon lange kein unerfüllter Wunsch mehr bleiben. Mit dem Erlebnis-Gut...") [08:38:34] Load avg. on willow is WARNING: WARNING - load average: 15.66, 15.21, 12.87 [08:38:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.172852/1.95, alarm hl:tmp_free=40025M/100M, alarm hl:np_load_avg=1.937500/2.0, alarm hl:mem_free=712.000000M/350M, alarm hl:available=1/0 [08:39:33] Load avg. on willow is OK: OK - load average: 12.48, 14.34, 12.71 [08:39:34] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:44:33] Load avg. on willow is WARNING: WARNING - load average: 16.46, 17.60, 14.65 [08:44:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.226074/1.95, alarm hl:tmp_free=40009M/100M, alarm hl:np_load_avg=2.236328/2.0, alarm hl:mem_free=296.000000M/350M, alarm hl:available=1/0 [09:00:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:08:47] 3(resolved) [TS-1284] chown <10https://jira.toolserver.org/browse/TS-1284> (Marlen Caemmerer) [09:10:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:13:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:15:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:15:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.268555/1.95, alarm hl:tmp_free=39921M/100M, alarm hl:np_load_avg=2.400879/2.0, alarm hl:mem_free=404.000000M/350M, alarm hl:available=1/0 [09:16:34] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [09:26:24] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 285227 MB (5% inode=32%): [09:52:43] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [09:56:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.838379/1.95, alarm hl:tmp_free=39809M/100M, alarm hl:np_load_avg=2.069336/2.0, alarm hl:mem_free=363.000000M/350M, alarm hl:available=1/0 [09:57:34] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:00:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:10:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:13:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:15:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:16:44] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [10:19:43] 3(resolved) [TS-1111] Please re-activate my TS account <10https://jira.toolserver.org/browse/TS-1111> (Marlen Caemmerer) [10:26:35] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 285101 MB (5% inode=32%): [10:31:43] 3(created) [MNT-1239] Clean-up old osm-titles on hemlock; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1239> (DaB.) [10:33:43] 3(work started) [MNT-1239] Clean-up old osm-titles on hemlock <10https://jira.toolserver.org/browse/MNT-1239> (DaB.) [10:42:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.913574/1.95, alarm hl:tmp_free=39699M/100M, alarm hl:np_load_avg=1.953125/2.0, alarm hl:mem_free=247.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.913574/2.3, alarm hl:np_load_long=1.988770/2.5, alarm hl:cpu=78.500000/98, alarm hl:mem_free=247.000000M/200M, al [10:44:44] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:50:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.257812/1.95, alarm hl:tmp_free=39676M/100M, alarm hl:np_load_avg=1.978027/2.0, alarm hl:mem_free=377.000000M/350M, alarm hl:available=1/0 [10:51:44] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:00:44] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:04:04] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.166992/1.10, alarm hl:np_load_long=0.683594/1.55, alarm hl:mem_free=11824.000000M/500M, alarm hl:tmp_free=14516M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.166992/1.00, alarm hl:np_load_long=0.683594/1.50, alarm hl:mem_free=11824.000000M/600M, alarm hl:tmp_free [11:07:04] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:10:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:13:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=4.010742/1.95, alarm hl:tmp_free=39610M/100M, alarm hl:np_load_avg=2.673828/2.0, alarm hl:mem_free=328.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=4.010742/2.3, alarm hl:np_load_long=2.213379/2.5, alarm hl:cpu=81.700000/98, alarm hl:mem_free=328.000000M/200M, al [11:13:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:15:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:16:49] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [11:20:49] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:26:51] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 298365 MB (5% inode=33%): [11:26:51] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.282227/1.95, alarm hl:tmp_free=39573M/100M, alarm hl:np_load_avg=2.221680/2.0, alarm hl:mem_free=242.000000M/350M, alarm hl:available=1/0 [11:30:09] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.321289/1.10, alarm hl:np_load_long=0.899414/1.55, alarm hl:mem_free=12135.000000M/500M, alarm hl:tmp_free=14489M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.321289/1.00, alarm hl:np_load_long=0.899414/1.50, alarm hl:mem_free=12135.000000M/600M, alarm hl:tmp_free [11:34:09] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:38:09] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.448242/1.10, alarm hl:np_load_long=1.022461/1.55, alarm hl:mem_free=12043.000000M/500M, alarm hl:tmp_free=14487M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.448242/1.00, alarm hl:np_load_long=1.022461/1.50, alarm hl:mem_free=12043.000000M/600M, alarm hl:tmp_free [11:38:49] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:00:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:10:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:12:09] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.198242/1.10, alarm hl:np_load_long=1.231445/1.55, alarm hl:mem_free=12143.000000M/500M, alarm hl:tmp_free=14422M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.198242/1.00, alarm hl:np_load_long=1.231445/1.50, alarm hl:mem_free=12143.000000M/600M, alarm hl:tmp_free [12:12:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.850586/1.95, alarm hl:tmp_free=39437M/100M, alarm hl:np_load_avg=2.246582/2.0, alarm hl:mem_free=153.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.850586/2.3, alarm hl:np_load_long=1.964844/2.5, alarm hl:cpu=79.000000/98, alarm hl:mem_free=153.000000M/200M, al [12:13:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:15:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:15:49] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:16:50] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [12:20:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.667969/1.95, alarm hl:tmp_free=39411M/100M, alarm hl:np_load_avg=1.867676/2.0, alarm hl:mem_free=283.000000M/350M, alarm hl:available=1/0 [12:21:49] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:27:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 301578 MB (5% inode=33%): [12:30:09] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:35:09] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.194336/1.10, alarm hl:np_load_long=1.245117/1.55, alarm hl:mem_free=12142.000000M/500M, alarm hl:tmp_free=14421M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.194336/1.00, alarm hl:np_load_long=1.245117/1.50, alarm hl:mem_free=12142.000000M/600M, alarm hl:tmp_free [12:50:09] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:00:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:01:46] 3(assigned) [TS-1380] Crontab for MMT cvn on willow is not running <10https://jira.toolserver.org/browse/TS-1380> (Marlen Caemmerer) [13:08:49] Load avg. on willow is WARNING: WARNING - load average: 13.06, 15.63, 14.93 [13:09:49] Load avg. on willow is OK: OK - load average: 10.23, 14.40, 14.54 [13:11:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:12:09] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.657227/1.10, alarm hl:np_load_long=1.682617/1.55, alarm hl:mem_free=12180.000000M/500M, alarm hl:tmp_free=14337M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.657227/1.00, alarm hl:np_load_long=1.682617/1.50, alarm hl:mem_free=12180.000000M/600M, alarm hl:tmp_free [13:14:49] Load avg. on willow is WARNING: WARNING - load average: 13.66, 17.13, 15.85 [13:14:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:15:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:17:49] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [13:27:49] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 304027 MB (5% inode=33%): [13:30:08] Dab can you take a look at https://jira.toolserver.org/browse/TS-1379 [13:30:52] 3(commented) [TS-1380] Crontab for MMT cvn on willow is not running <10https://jira.toolserver.org/browse/TS-1380> (Marlen Caemmerer) [13:36:44] 3(commented) [TS-1380] Crontab for MMT cvn on willow is not running <10https://jira.toolserver.org/browse/TS-1380> (Marlen Caemmerer) [13:39:44] 3(commented) [TS-1380] Crontab for MMT cvn on willow is not running <10https://jira.toolserver.org/browse/TS-1380> (Marlen Caemmerer) [13:46:09] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:53:09] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.372070/1.10, alarm hl:np_load_long=1.342774/1.55, alarm hl:mem_free=11698.000000M/500M, alarm hl:tmp_free=14307M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.372070/1.00, alarm hl:np_load_long=1.342774/1.50, alarm hl:mem_free=11698.000000M/600M, alarm hl:tmp_free [13:54:08] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [14:00:39] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:01:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:01:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:11:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:14:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:18:49] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [14:20:59] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.048828/1.95, alarm hl:tmp_free=39043M/100M, alarm hl:np_load_avg=3.239746/2.0, alarm hl:mem_free=454.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.048828/2.3, alarm hl:np_load_long=2.952637/2.5, alarm hl:cpu=74.900000/98, alarm hl:mem_free=454.000000M/200M, al [14:21:52] [[Special:Log/newusers]] create 10 * Florida surrogate * (New user account) [14:22:35] [[User:Florida surrogate]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7229&rcid=9639 * Florida surrogate * (+1273) (Created page with "Issues and Concerns about Surrogacy Surrogacy is fast becoming one of the most popular options for getting your own child. For intended parents and childless couples who are wan...") [14:26:30] Joan / Betacommand ^ [14:28:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 306027 MB (5% inode=33%): [14:31:49] Load avg. on willow is CRITICAL: CRITICAL - load average: 38.38, 30.72, 25.73 [14:33:38] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:54:58] SSH on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:58] SSH on adenia is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [14:58:38] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:59:29] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [14:59:49] Load avg. on willow is WARNING: WARNING - load average: 11.05, 16.33, 19.89 [15:00:49] Load avg. on willow is CRITICAL: CRITICAL - load average: 64.49, 28.39, 23.81 [15:01:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:01:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:02:20] [[Special:Log/block]] block 10 * Betacommand * (blocked [[02User:Florida surrogate10]] with an expiry time of infinite (account creation disabled): Spamming links to external sites) [15:02:47] [[Special:Log/delete]] delete 10 * Betacommand * (deleted "[[02User:Florida surrogate10]]": content was: "Issues and Concerns about Surrogacy Surrogacy is fast becoming one of the most popular options for getting your own child. For intended paren..." (and the only contributor was "[[Special:Contributions/Florida surrogate|Florida surrogate]]")) [15:11:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:14:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:18:59] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [15:20:59] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.294434/1.95, alarm hl:tmp_free=38839M/100M, alarm hl:np_load_avg=3.362793/2.0, alarm hl:mem_free=546.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.294434/2.3, alarm hl:np_load_long=3.046387/2.5, alarm hl:cpu=71.600000/98, alarm hl:mem_free=546.000000M/200M, al [15:28:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 308187 MB (5% inode=34%): [15:33:39] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:43:19] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:01:10] Load avg. on willow is CRITICAL: CRITICAL - load average: 59.05, 32.91, 26.64 [16:01:30] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:01:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:06:49] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:07:19] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=7.386230/1.95, alarm hl:tmp_free=38689M/100M, alarm hl:np_load_avg=4.374023/2.0, alarm hl:mem_free=854.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=7.386230/2.3, alarm hl:np_load_long=3.528809/2.5, alarm hl:cpu=80.400000/98, alarm hl:mem_free=854.000000M/200M, al [16:12:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:15:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:16:47] 3(created) [ACCAPP-510] Requesting Tool-Server account as team member of the RENDER project in close cooperation with Wikimedia Deutschland (Angelika Adam, Kai Nissen); Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-510> [16:18:19] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.217774/1.10, alarm hl:np_load_long=0.790039/1.55, alarm hl:mem_free=10784.000000M/500M, alarm hl:tmp_free=14123M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.217774/1.00, alarm hl:np_load_long=0.790039/1.50, alarm hl:mem_free=10784.000000M/600M, alarm hl:tmp_free [16:19:19] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [16:19:19] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [16:29:08] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 310066 MB (5% inode=34%): [16:30:19] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:31:22] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=9.151367/1.95, alarm hl:tmp_free=38619M/100M, alarm hl:np_load_avg=4.718262/2.0, alarm hl:mem_free=804.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=9.151367/2.3, alarm hl:np_load_long=3.879395/2.5, alarm hl:cpu=78.200000/98, alarm hl:mem_free=804.000000M/200M, al [16:33:20] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.029297/1.00, alarm hl:np_load_long=0.954102/1.50, alarm hl:mem_free=10595.000000M/600M, alarm hl:tmp_free=14100M/100M, alarm hl:available=1/0 [16:48:25] do we have a copy of files on wikipedia or commons? [17:00:19] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:00:29] Sun Grid Engine execd on willow is UNKNOWN: NRPE: Call to fork() failed [17:01:20] Load avg. on willow is CRITICAL: CRITICAL - load average: 70.46, 37.94, 32.68 [17:01:20] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=8.967773/1.95, alarm hl:tmp_free=38521M/100M, alarm hl:np_load_avg=4.705566/2.0, alarm hl:mem_free=643.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=8.967773/2.3, alarm hl:np_load_long=4.069336/2.5, alarm hl:cpu=93.800000/98, alarm hl:mem_free=643.000000M/200M, al [17:01:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:01:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:06:30] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:12:19] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:16:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:19:20] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [17:26:20] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=4.437988/1.95, alarm hl:tmp_free=38445M/100M, alarm hl:np_load_avg=4.116211/2.0, alarm hl:mem_free=524.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=4.437988/2.3, alarm hl:np_load_long=3.883301/2.5, alarm hl:cpu=62.900000/98, alarm hl:mem_free=524.000000M/200M, al [17:29:25] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 310922 MB (5% inode=34%): [17:30:20] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:30:29] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:31:20] Load avg. on willow is CRITICAL: CRITICAL - load average: 60.12, 33.12, 30.54 [17:31:21] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=6.984375/1.95, alarm hl:tmp_free=38432M/100M, alarm hl:np_load_avg=3.899414/2.0, alarm hl:mem_free=972.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=6.984375/2.3, alarm hl:np_load_long=3.733887/2.5, alarm hl:cpu=69.000000/98, alarm hl:mem_free=972.000000M/200M, al [17:36:21] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:38:38] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:43:29] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [17:56:20] Load avg. on willow is CRITICAL: CRITICAL - load average: 36.77, 33.62, 30.64 [18:00:22] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:00:29] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:01:19] Load avg. on willow is CRITICAL: CRITICAL - load average: 65.75, 36.05, 31.29 [18:01:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=7.944824/1.95, alarm hl:tmp_free=38269M/100M, alarm hl:np_load_avg=4.403320/2.0, alarm hl:mem_free=792.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=7.944824/2.3, alarm hl:np_load_long=3.875000/2.5, alarm hl:cpu=98.100000/98, alarm hl:mem_free=792.000000M/200M, al [18:01:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:01:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:06:20] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:12:22] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:12:29] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:13:19] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.252930/1.10, alarm hl:np_load_long=1.356445/1.55, alarm hl:mem_free=11002.000000M/500M, alarm hl:tmp_free=13936M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.252930/1.00, alarm hl:np_load_long=1.356445/1.50, alarm hl:mem_free=11002.000000M/600M, alarm hl:tmp_free [18:16:20] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [18:16:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:19:20] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [18:26:19] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.55, 30.94, 29.83 [18:30:20] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 313074 MB (5% inode=34%): [18:32:19] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=5.988770/1.95, alarm hl:tmp_free=38008M/100M, alarm hl:np_load_avg=4.839355/2.0, alarm hl:mem_free=663.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=5.988770/2.3, alarm hl:np_load_long=4.071289/2.5, alarm hl:cpu=64.900000/98, alarm hl:mem_free=663.000000M/200M, al [18:36:38] Sun Grid Engine execd on willow is UNKNOWN: NRPE: Unable to read output [18:37:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=7.618164/1.95, alarm hl:tmp_free=37972M/100M, alarm hl:np_load_avg=4.987793/2.0, alarm hl:mem_free=590.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=7.618164/2.3, alarm hl:np_load_long=4.239258/2.5, alarm hl:cpu=82.000000/98, alarm hl:mem_free=590.000000M/200M, al [18:42:49] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:48:29] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:49:28] Load avg. on willow is CRITICAL: CRITICAL - load average: 49.23, 33.11, 31.99 [19:01:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:01:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:08:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=4.814941/1.95, alarm hl:tmp_free=37768M/100M, alarm hl:np_load_avg=5.301758/2.0, alarm hl:mem_free=1494.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=4.814941/2.3, alarm hl:np_load_long=4.930176/2.5, alarm hl:cpu=49.100000/98, alarm hl:mem_free=1494.000000M/200M, [19:12:59] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:13:19] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:13:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=7.717285/1.95, alarm hl:tmp_free=37754M/100M, alarm hl:np_load_avg=5.508301/2.0, alarm hl:mem_free=1046.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=7.717285/2.3, alarm hl:np_load_long=5.017578/2.5, alarm hl:cpu=83.600000/98, alarm hl:mem_free=1046.000000M/200M, [19:17:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:20:19] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [19:24:29] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:24:29] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:25:29] Load avg. on willow is CRITICAL: CRITICAL - load average: 52.12, 37.23, 37.11 [19:30:19] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 313467 MB (5% inode=34%): [19:36:52] is the plan still to start disabling interwiki scripts on the 15th? [19:38:29] carl-m: yes [19:38:33] :) [19:39:21] maybe that will help some with willow. A second ago, I could't run ps because it couldn't fork a new process :) [19:39:38] carl-m: it will help drastically [19:40:01] o hai Betacommand [19:40:34] I do like http://toolserver.org/~betacommand/reports/stats/mediawiki_p.txt but you hide global sysops and stewards completely [19:40:58] I would like to see them, just distinguished somehow [19:41:06] ToAruShiroiNeko: Ill look into figuring out how to do that [19:42:39] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:53:26] the other likely culprit for the fork problems is mgedawy, who seems to have something that runs 150+ processes simultaneously every few seconds [19:56:05] 150? :o [19:56:09] Environment IPMI on willow is CRITICAL: Connection refused by host [19:56:27] Firebolt: well, here's 500 :) http://pastebin.com/HTLAQsNW [19:56:29] / on willow is CRITICAL: Connection refused by host [19:56:29] /tmp on willow is CRITICAL: Connection refused by host [19:56:29] Cluster on willow is CRITICAL: Connection refused by host [19:56:54] wow [19:56:59] what is he even running [20:00:31] Firebolt: I think it is happening every minute on the minute, so something from cron most likely [20:01:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:02:09] SMF on willow is CRITICAL: Connection refused by host [20:02:29] Load avg. on willow is CRITICAL: Connection refused by host [20:06:02] Betacommand it would be interesting to see if the local community is more active than the global users :) [20:06:52] carl-m, yeah, but what would he be running every minute? [20:08:28] bunches of stuff into cronsub; no way to tell from ps what the scripts are doing [20:09:39] Sun Grid Engine execd on willow is CRITICAL: Connection refused by host [20:10:29] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1906 [20:13:04] Hersfold Wikipedia * [Toolserver-l] Out of memory/segfaults? [20:13:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:14:29] / on willow is OK: DISK OK - free space: / 37579 MB (35% inode=99%): [20:14:29] /tmp on willow is OK: DISK OK - free space: / 37579 MB (35% inode=99%): [20:14:29] Cluster on willow is OK: CLUSTER OK ! [20:14:39] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=4.652832/1.95, alarm hl:tmp_free=37582M/100M, alarm hl:np_load_avg=4.850586/2.0, alarm hl:mem_free=2070.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=4.652832/2.3, alarm hl:np_load_long=4.740234/2.5, alarm hl:cpu=67.500000/98, alarm hl:mem_free=2070.000000M/200M, [20:15:19] Environment IPMI on willow is OK: ok: temperature ok fan ok voltage ok chassis ok [20:16:03] Carl (CBM) * Re: [Toolserver-l] Out of memory/segfaults? [20:17:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:18:09] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:18:50] maybe he just misunderstood the crontab format :) 589 procs just now [20:19:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:20:29] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [20:24:40] Load avg. on willow is UNKNOWN: NRPE: Unable to read output [20:24:50] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:25:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 69.04, 49.35, 42.09 [20:25:39] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=8.839355/1.95, alarm hl:tmp_free=37551M/100M, alarm hl:np_load_avg=6.140137/2.0, alarm hl:mem_free=2132.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=8.839355/2.3, alarm hl:np_load_long=5.243652/2.5, alarm hl:cpu=100.000000/98, alarm hl:mem_free=2132.000000M/200M, [20:30:29] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 314602 MB (5% inode=34%): [20:36:19] SMF on willow is UNKNOWN: NRPE: Unable to read output [20:55:03] DaB. * Re: [Toolserver-l] Out of memory/segfaults? [21:01:03] hello all [21:01:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:02:06] Dispenser: I have bad news: The only avaiable backup of home does not include your file for unknown reasons – I'm sorry [21:03:35] Ok, I'll tell JaGa that he'll have to fix/regenerate those results [21:04:00] ok, tnx [21:04:34] DaBPunkt: did you see the discussion above about process counts on willow? [21:05:09] carl-m: on the ts-mailinglist? [21:05:18] in this channel [21:06:01] carl-m: I do not spy on the channel when my nick is away ;) [21:06:36] The ts log bot hasn't been running presumably because of the discussed segfaults [21:08:50] DaBPunkt: in short, user mgedawy starts about 200 processes every minute from crontab; sometimes there are 500 procs on the account, as in http://pastebin.com/HTLAQsNW [21:10:29] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2364 [21:11:27] I see [21:12:49] Load avg. on willow is UNKNOWN: NRPE: Call to fork() failed [21:13:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:13:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 52.63, 42.62, 41.59 [21:15:04] that's nothing new. there's a group of users that runs >100 processes with totals mem usage close to 1GB. [21:15:53] johang: the issue here is that he runs 200 procs simultaneously, once per minute. that's not typical for bot operators, it's either bad implementation or lack of experience [21:16:08] mgedawy, beria and darafsh all have >100 procs [21:16:56] carl-m: certainly abusing the available resources [21:17:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:17:48] * Betacommand notes that he is only at 34 [21:17:50] the difference with darafsh is that he has a constant load of 134. mgedawy forks 200 (or more) then waits 60 seconds and does it again, so the overhead of starting all the processes is worse [21:17:52] i can imagine why they have so many procs.. iw's bots [21:18:11] DaBPunkt: mgedawy is using sge, but he seems to call qcronsub or cronsub hundret times _every minute_. can you check his crontab on willow? [21:18:26] Merlissimo: disabled them all [21:18:49] Alchimista: yes mgedawy is running recentchages watching interwiki pwd bot [21:18:50] the load average dropped from 40 to 5 [21:19:27] 3GB free mem. I can breath again! :) [21:19:56] Merlissimo: it was easy to gess why someone would run so many procs :P [21:20:33] Alchimista: you can easily check his processtree by using ptree [21:20:36] Merlissimo: or DaBPunkt can you take care of https://jira.toolserver.org/browse/TS-1379 it should be a real easy fix [21:20:39] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [21:21:49] Betacommand: in a minute [21:21:55] DaBPunkt: thanks [21:23:18] Betacommand: i am not a toolserver admin [21:23:58] Merlissimo: for some reason I thought you where :P [21:25:41] Betacommand: i only have sge manager privilege, nothing else [21:28:40] Load avg. on willow is WARNING: WARNING - load average: 4.97, 6.93, 19.30 [21:30:40] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 315077 MB (5% inode=34%): [21:32:02] DaB. * [Toolserver-announce] Fairness on the toolserver [21:32:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:32:45] 3(assigned) [TS-1379] user_former_groups database view <10https://jira.toolserver.org/browse/TS-1379> (DaB.) [21:36:43] Load avg. on willow is OK: OK - load average: 4.77, 7.15, 14.70 [21:44:45] 3(resolved) [TS-1379] user_former_groups database view <10https://jira.toolserver.org/browse/TS-1379> (DaB.) [21:50:03] Hersfold Wikipedia * Re: [Toolserver-l] Fairness on the toolserver [21:52:55] DaBPunkt: how can i find out # of running processes and other stuff you mentioned? you did not name anybody and i want to be sure that i'm not of those the mail applies to [21:54:19] Danny_B|backup: ps -ef|grep danny_b - but you are perfectly within the limits, no worry :) [21:56:27] thanks [21:57:58] Danny_B|backup: there is also a difference between "I started 200 processes by mistake, because I forgot a sleep in a loop" and "let's see how I can get more resources for me" [22:02:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:11:24] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3061 [22:13:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:17:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:20:43] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [22:22:09] nacht ts [22:31:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 318237 MB (5% inode=34%): [22:32:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:39:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 318368 MB (6% inode=34%): [22:42:47] Merlissimo: How can we check how much resources are our tools/scripts consumming? [22:43:23] on a single server? [22:43:59] yes [22:44:16] prstat -t [22:44:32] shows an overview of all users [22:45:01] [[User:Picardo]] ! 10https://wiki.toolserver.org/w/index.php?diff=7230&oldid=7228&rcid=9642 * Hercule * (-502) (spam) [22:45:46] maybe "prstat -a -U alchimista" is more useful [22:45:50] Alchimista: ^^ [22:45:53] [[Dien lanh vinh]] ! 10https://wiki.toolserver.org/w/index.php?diff=7231&oldid=7224&rcid=9643 * Hercule * (-150) (spam) [22:46:29] Merlissimo: prstat was what i couldn't remember. i had a cheatset for those kind of things, to check what and how my tools where on ts, but lost it [22:47:14] prstat should be used on solaris and top on linux [22:48:09] by the way, i'm using "qcronsub -N albethi -l h_rt=INFINITY -l virtual_free=20M -r y -b y -wd" on submit@ for my irc bots, it's fine, or can be improved? [22:50:32] that is perfect [22:51:15] how often do you call it? [22:51:46] every 10 min [22:56:20] Alchimista: your job aleph is using 94M, but was submitted using by virtual_free=20M [22:57:15] so it may get killed if memory on that server is low [22:57:40] but shouldn't he get sticked to 20M? [22:58:35] virtual_free is only a soft limit. [22:59:49] [[User talk:The Queen (QueenLady)]] ! 10https://wiki.toolserver.org/w/index.php?diff=7232&oldid=7199&rcid=9644 * 90.12.4.115 * (-4464) (Blanked the page) [23:00:04] if you want a hard limit you can add -l h_vmem=20M [23:00:23] [[User:Elineone]] ! 10https://wiki.toolserver.org/w/index.php?diff=7233&oldid=7205&rcid=9645 * 90.12.4.115 * (-2248) (spam) [23:00:54] [[Dienlanhnghean]] ! 10https://wiki.toolserver.org/w/index.php?diff=7234&oldid=7203&rcid=9646 * 90.12.4.115 * (-150) (spam) [23:02:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:10:43] 3(commented) [TS-1380] Crontab for MMT cvn on willow is not running <10https://jira.toolserver.org/browse/TS-1380> (Krinkle) [23:12:23] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3301 [23:13:42] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7234 of [[02Dienlanhnghean10]] patrolled ) [23:13:42] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:13:55] [[Special:Log/move]] move_redir 10 * Krinkle * (moved [[02Dienlanhnghean10]] to [[User talk:Sgolldmi]] over redirect) [23:14:08] [[Special:Log/delete]] delete 10 * Krinkle * (deleted "[[02Dienlanhnghean10]]": non-use: spam) [23:14:18] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7232 of [[02User talk:The Queen (QueenLady)10]] patrolled ) [23:14:41] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7233 of [[02User:Elineone10]] patrolled ) [23:14:43] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7231 of [[02Dien lanh vinh10]] patrolled ) [23:14:46] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7230 of [[02User:Picardo10]] patrolled ) [23:14:55] [[Special:Log/delete]] delete 10 * Krinkle * (deleted "[[02User:Picardo10]]": spam) [23:15:07] [[Special:Log/delete]] delete 10 * Krinkle * (deleted "[[02Dien lanh vinh10]]": spam) [23:17:20] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7227 of [[02Category:Documentation10]] patrolled ) [23:17:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:18:30] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7226 of [[02Database access10]] patrolled ) [23:20:42] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [23:22:49] [[Database access]] 10https://wiki.toolserver.org/w/index.php?diff=7237&oldid=7226&rcid=9651 * Krinkle * (+294) (/* namespacename */ ) [23:25:08] [[MediaWiki:Deletereason-dropdown]] 10https://wiki.toolserver.org/w/index.php?diff=7238&oldid=3706&rcid=9652 * Krinkle * (+39) () [23:25:15] [[Special:Log/delete]] delete 10 * Krinkle * (deleted "[[02User talk:Sgolldmiss10]]": Broken or incorrect redirect) [23:25:34] [[MediaWiki:Deletereason-dropdown]] 10https://wiki.toolserver.org/w/index.php?diff=7239&oldid=7238&rcid=9654 * Krinkle * (+8) () [23:26:16] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7219 of [[02Category:Documentation10]] patrolled ) [23:33:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:33:24] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3605 [23:35:43] [[Special:Log/move]] move_redir 10 * Krinkle * (moved [[02Điện lạnh nghệ an10]] to [[User talk:Sgolldmis]] over redirect) [23:35:51] [[Special:Log/delete]] delete 10 * Krinkle * (deleted "[[02Điện lạnh nghệ an10]]": Broken or incorrect redirect) [23:37:26] [[Special:Log/delete]] delete 10 * Krinkle * (deleted "[[02User talk:Sgolldmis10]]": Spam) [23:37:49] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7198 of [[02Wiki server assignments10]] patrolled ) [23:38:07] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7190 of [[02User talk:Mgedawy10]] patrolled ) [23:38:10] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7189 of [[02User:Mgedawy10]] patrolled ) [23:39:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 324504 MB (6% inode=35%): [23:39:48] [[Special:Log/delete]] delete 10 * Krinkle * (deleted "[[02User talk:The Queen (QueenLady)10]]") [23:59:03] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [23:59:23] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds)