[00:14:42] 3(created) [ACCAPP-534] Bot to: scrub VisualEditor test pages for mediawiki.org; to test out code changes; and for similar purposes.; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-534> (James D. Forrester) [00:26:34] @replag [00:26:35] Betacommand: s1-rr-a: 14s [-]; s1-user: 24s [+0.00 s/s]; s2-user: 6d 9h 38m 29s [-2.01 s/s]; s7-rr-a: 11s [-]; s7-user: 11s [-] [00:39:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 79249 MB (6% inode=99%): [00:40:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 357413 MB (6% inode=36%): [00:40:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:40:32] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:41:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:42:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:51:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.197266/1.95, alarm hl:tmp_free=36856M/100M, alarm hl:np_load_avg=1.101074/2.0, alarm hl:mem_free=281.000000M/350M, alarm hl:available=1/0 [00:53:32] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [00:59:39] beta it seems the issue is fixed :) [00:59:43] next issue? [00:59:54] ToAruShiroiNeko: its the weekend [01:00:18] friday :) [01:00:32] or fryday in my case [01:00:43] I am so not looking forward to my Machine learning exam on tuesday [01:23:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.936035/1.95, alarm hl:tmp_free=36796M/100M, alarm hl:np_load_avg=1.098633/2.0, alarm hl:mem_free=184.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.936035/2.3, alarm hl:np_load_long=1.106445/2.5, alarm hl:cpu=74.400000/98, alarm hl:mem_free=184.000000M/200M, al [01:27:02] MZMcBride * Re: [Toolserver-l] Dude, where is my logfile? [01:40:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 348188 MB (6% inode=36%): [01:40:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:40:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:40:49] Brooke: /facepalm with your last email [01:42:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:42:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:54:03] Betacommand is that a valid command? :p [01:54:27] ToAruShiroiNeko: ?? [02:02:32] /sql on thyme is WARNING: DISK WARNING - free space: /sql 193181 MB (20% inode=99%): [02:40:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 346298 MB (6% inode=36%): [02:40:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.074707/1.95, alarm hl:tmp_free=36666M/100M, alarm hl:np_load_avg=0.972168/2.0, alarm hl:mem_free=154.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.074707/2.3, alarm hl:np_load_long=0.975586/2.5, alarm hl:cpu=81.500000/98, alarm hl:mem_free=154.000000M/200M, al [02:41:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:41:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:42:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:42:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:43:30] I mean /facepalm [02:48:32] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [02:49:07] no [03:39:41] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:40:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 346238 MB (6% inode=36%): [03:41:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:41:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:42:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:42:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:59:41] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:10:32] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2198 MB (19% inode=99%): [04:21:30] /tmp on ortelius is OK: DISK OK - free space: /tmp 3073 MB (24% inode=99%): [04:28:31] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2045 MB (17% inode=99%): [04:38:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.745606/1.95, alarm hl:tmp_free=36458M/100M, alarm hl:np_load_avg=0.734375/2.0, alarm hl:mem_free=222.000000M/350M, alarm hl:available=1/0 [04:40:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 348930 MB (6% inode=36%): [04:41:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:41:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:42:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:42:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:46:41] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [04:53:03] Sumana Harihareswara * Re: [Toolserver-l] Upcoming hackathon for experts AND newbies: Washington, DC, USA July 10-11 [04:56:31] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2190 MB (18% inode=99%): [05:00:31] /tmp on ortelius is OK: DISK OK - free space: /tmp 3726 MB (28% inode=99%): [05:18:20] [[Special:Log/newusers]] create 10 * Mypengu4 * (New user account) [05:40:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 346190 MB (6% inode=36%): [05:41:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:41:40] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:43:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:43:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:55:43] 3(created) [MAGNUS-329] show redirects; Magnus' tools: FIST; Improvement <10https://jira.toolserver.org/browse/MAGNUS-329> (Traveler100) [06:40:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 346050 MB (6% inode=36%): [06:41:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:42:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:43:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:43:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:09:02] K. Peachey * Re: [Toolserver-l] Dude, where is my logfile? [07:40:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 345981 MB (6% inode=36%): [07:42:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:43:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:44:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:44:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:40:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 345922 MB (6% inode=36%): [08:42:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:43:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:45:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:45:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:56:43] [[Special:Log/newusers]] create 10 * DustyPlg * (New user account) [09:39:02] Yishen Miao * [Toolserver-l] Running weblinkchecker.py on Toolserver [09:40:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 345840 MB (6% inode=36%): [09:42:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:43:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:43:47] 3(commented) [TS-1413] cassia: connection errors because of "MySQL server has gone away" and s5 replication has stopped <10https://jira.toolserver.org/browse/TS-1413> (Marlen Caemmerer) [09:45:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:45:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:59:07] Mmm .. possible corruption on enwiki_p :( Anyone got a sec to double-check I'm not missing something please? Article titles 'Netaji_Subhas_Chandra_Bose_International_Airport' and 'Netaji_Subhash_Chandra_Bose_International_Airport' in namespace 0 [10:14:52] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.844238/1.95, alarm hl:tmp_free=35829M/100M, alarm hl:np_load_avg=0.877441/2.0, alarm hl:mem_free=219.000000M/350M, alarm hl:available=1/0 [10:16:52] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [10:20:49] 3(created) [TS-1414] Possible corruption of enwiki_p circa 6AM 20th June 2011; Toolserver: Databases; Bug <10https://jira.toolserver.org/browse/TS-1414> (Topbanana) [10:22:44] 3(updated) [TS-1414] Possible corruption of enwiki_p circa 6AM 20th June 2012 <10https://jira.toolserver.org/browse/TS-1414> (Topbanana) [10:40:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 345282 MB (6% inode=36%): [10:42:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:43:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:46:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:46:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:50:47] 3(commented) [MNT-668] Add missing quotas <10https://jira.toolserver.org/browse/MNT-668> (Marlen Caemmerer) [10:50:58] @replag [10:50:58] Alchimista: s2-user: 5d 19h 32m 57s [-0.15 s/s] [10:54:43] 3(commented) [MNT-1252] /home is nearly full <10https://jira.toolserver.org/browse/MNT-1252> (Marlen Caemmerer) [11:07:02] Marlen Caemmerer * Re: [Toolserver-l] Dude, where is my logfile? [11:13:43] 3(updated) [DRTRIGON-105] Support for multiple BS template configurations <10https://jira.toolserver.org/browse/DRTRIGON-105> (drtrigon) [11:13:46] 3(commented) [DRTRIGON-105] Support for multiple BS template configurations <10https://jira.toolserver.org/browse/DRTRIGON-105> (drtrigon) [11:29:48] 3(assigned) [TS-1414] Possible corruption of enwiki_p circa 6AM 20th June 2012 <10https://jira.toolserver.org/browse/TS-1414> (Marlen Caemmerer) [11:31:45] 3(created) [TS-1415] Please set LDAP email address for user "the_wub"; Toolserver: Accounts; Task <10https://jira.toolserver.org/browse/TS-1415> (Peter Coombe) [11:31:45] 3(commented) [TS-1414] Possible corruption of enwiki_p circa 6AM 20th June 2012 <10https://jira.toolserver.org/browse/TS-1414> (Marlen Caemmerer) [11:32:36] @replag [11:32:36] Akoopal: s2-user: 5d 19h 19m 58s [-0.31 s/s]; s3-rr-a: 1m 28s [+0.00 s/s]; s3-user: 1m 28s [+0.00 s/s] [11:36:52] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40496 MB (9% inode=99%): [11:40:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 345366 MB (6% inode=36%): [11:42:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:43:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:47:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:47:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:52:51] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 63887 MB (15% inode=99%): [12:01:47] 3(commented) [TS-1414] Possible corruption of enwiki_p circa 6AM 20th June 2012 <10https://jira.toolserver.org/browse/TS-1414> (Marlen Caemmerer) [12:39:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 77414 MB (6% inode=99%): [12:40:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 345158 MB (6% inode=36%): [12:40:43] 3(closed) [DRTRIGON-105] Support for multiple BS template configurations <10https://jira.toolserver.org/browse/DRTRIGON-105> (drtrigon) [12:40:44] 3(closed) [DRTRIGON-114] Support for named groups in regexs <10https://jira.toolserver.org/browse/DRTRIGON-114> (drtrigon) [12:43:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:43:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:44:51] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40354 MB (9% inode=99%): [12:47:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:47:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:00:51] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 63918 MB (15% inode=99%): [13:26:47] 3(commented) [TS-1360] Install OpenCV libraries and python bindings <10https://jira.toolserver.org/browse/TS-1360> (DaB.) [13:30:43] 3(created) [TS-1416] Expose ar_sha1; Toolserver: Databases; Task <10https://jira.toolserver.org/browse/TS-1416> (Liangent) [13:35:11] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:UP: 1 int NOK : CRITICAL [13:35:40] DaBPunkt: did you fix the cronie reboot yet? [13:36:33] Danny_B|backup: no, sorry. Is there anything I should start by hand after the reboot? [13:36:55] i'll start myself, i just wanted to know if i should be around [13:37:19] well, when we'll be back to linux, @reboot should work again anyway, right? [13:41:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 345034 MB (6% inode=36%): [13:42:34] Danny_B|backup: yes [13:43:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:43:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:44:50] * Danny_B|backup is slowly preparing to reboot, saving all the stuff etc... [13:46:20] sorry, we didn't know before that mark is in the datacenter today [13:46:45] and on the plus-side, with more memorx willow should run a lot better [13:47:42] 3(closed) [DRTRIGON-119] More/better SGE integration and interaction in bot_control <10https://jira.toolserver.org/browse/DRTRIGON-119> (drtrigon) [13:48:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:48:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:02:52] /sql on thyme is WARNING: DISK WARNING - free space: /sql 188648 MB (19% inode=99%): [14:05:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.744141/1.95, alarm hl:tmp_free=35367M/100M, alarm hl:np_load_avg=0.769531/2.0, alarm hl:mem_free=719.000000M/350M, alarm hl:available=0/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.744141/2.3, alarm hl:np_load_long=0.762207/2.5, alarm hl:cpu=74.900000/98, alarm hl:mem_free=719.000000M/200M, al [14:07:51] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [14:14:45] 3(created) [TS-1417] Monitoring what has become slow; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1417> (Marlen Caemmerer) [14:18:52] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/16:DOWN: 1 int NOK : CRITICAL [14:19:50] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:DOWN: 1 int NOK : CRITICAL [14:23:22] DaBPunkt: how much time i have? [14:29:52] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [14:34:29] Danny_B|backup: We do not know exactly [14:35:10] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:UP: 1 int NOK : CRITICAL [14:35:10] ethernet 0/1/19 [ptolemy.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:DOWN: 1 int NOK : CRITICAL [14:40:51] ethernet 0/1/23 [rosemary.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/23:DOWN: 1 int NOK : CRITICAL [14:41:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 344957 MB (6% inode=36%): [14:42:10] ethernet 0/1/20 [hemlock.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/20:DOWN: 1 int NOK : CRITICAL [14:42:52] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/17:DOWN: 1 int NOK : CRITICAL [14:43:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:43:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:46:51] ethernet 0/1/13 [nightshade.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/13:DOWN: 1 int NOK : CRITICAL [14:49:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:49:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:51:00] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70842 MB (5% inode=99%): [14:51:20] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:51:20] ethernet 0/1/19 [ptolemy.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:51:20] ethernet 0/1/20 [hemlock.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:52:01] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:52:01] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:52:01] ethernet 0/1/13 [nightshade.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:52:01] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:52:02] ethernet 0/1/23 [rosemary.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:52:02] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:56:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71414 MB (6% inode=99%): [14:57:01] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70496 MB (5% inode=99%): [14:57:02] [[Admin:Switches]] 10https://wiki.toolserver.org/w/index.php?diff=7409&oldid=6319&rcid=10014 * Nosy * (+245) () [14:57:20] ethernet 0/1/6 [asw-oe16:0/1/23 LACP secondary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:57:20] ethernet 0/1/2 [nightshade] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:57:21] ethernet 0/1/10 [hawthorn] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:57:21] ethernet 0/1/1 [willow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:57:21] ethernet 0/1/7 [hemlock] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:57:21] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:57:27] [[Admin:Switches]] 10https://wiki.toolserver.org/w/index.php?diff=7410&oldid=7409&rcid=10015 * Nosy * (+6) () [14:58:00] ethernet 0/1/3 [rosemary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:58:00] ethernet 0/1/15 [ts-array3.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:58:01] ethernet 0/1/22 [hyacinth] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:58:01] ethernet 0/1/18 [scs-oe10-esams] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:58:01] ethernet 0/1/21 [ptolemy] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:58:01] ethernet 0/1/9 [ortelius] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:58:01] ethernet 0/1/4 [wolfsbane] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:58:02] ethernet 0/1/12 [csw1-esams:1/24] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:58:02] ethernet 0/1/5 [asw-oe16:0/1/22 LACP primary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [14:58:55] is reboot done now? [15:00:18] liangent: no [15:02:44] 3(created) [TS-1418] Broken fans in asw-oe10; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1418> (Marlen Caemmerer) [15:13:30] will shutdown willow now [15:16:07] DaBPunkt: Lol [15:16:42] multichill: paordon? [15:16:43] i like false alarms [15:17:10] "False Alarm: The system willow will not be brought down." [15:17:16] "The system willow will be shut down in 1 minute" [15:18:50] DaBPunkt: nosy : since there is no @reboot working, please let me know when it's back up, thx [15:23:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 76883 MB (6% inode=99%): [15:24:48] DaBPunkt: Was the new memory also placed? [15:26:03] topic may be updated [15:28:24] willow is back [15:28:34] with more memory [15:28:55] Danny_B|webgate: pong [15:29:01] woots \o/ [15:29:38] yaba daba dooo! [15:30:22] i hope to finish turnera/damiana today too [15:32:05] hmm, it's stucked somehow [15:32:17] yes since turnera went away [15:32:22] gets more memory too [15:32:40] sorry i cannot tell you before because i never know when someone goes to the datacenter [15:32:43] nosy: do not shutdown both ha-nodes at the same time ;-) [15:33:16] DaBPunkt: we might be lucky damiana does not boot itself because of too few ram when turnera is away ;) [15:33:32] if you are fast enough ;) [15:33:44] lets see what mark does [15:34:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70990 MB (5% inode=99%): [15:34:40] wtf [15:34:55] are you all using irc on willow? [15:35:27] Tsss, using the Toolserver as a bouncer... [15:36:11] the best was one user using emacs on willow for this purpose consuming 600 mb of our rare memory [15:37:34] i see it :) [15:37:35] Yes, that sounds like emacs :P [15:38:08] /sql on cassia is WARNING: DISK WARNING - free space: /sql 72253 MB (6% inode=99%): [15:39:17] there will be a little hang in a minute when i boot damiana [15:40:49] NFS server ha-nfs.esi not responding still trying [15:49:17] Is it a problem if willow is asking me for my password when I try to ssh? [15:52:06] Firebolt: /home is not available now so your key is not available [15:52:12] ah [15:54:26] seems it should work now [15:54:40] turnera has an interfaces that flapps (goes online/offline) [15:55:06] nosy: that's the multicast-direct-interface betwen the ha-nodes [15:55:28] it needs a moment to become stable [15:55:31] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:55:32] ethernet 0/1/23 [rosemary.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:55:34] ok great [15:55:41] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:55:41] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:56:03] seems to be back again [15:56:09] \o/ [15:56:14] nosy: if mark is still at the servers: he should push the black network-cables in [15:56:31] on damiana and turnera? [15:56:47] ja. Die waren nach dem letzten Umbau etwas locker [15:58:01] ethernet 0/1/6 [asw-oe16:0/1/23 LACP secondary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:01] ethernet 0/1/1 [willow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:01] ethernet 0/1/10 [hawthorn] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:01] ethernet 0/1/2 [nightshade] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:01] ethernet 0/1/7 [hemlock] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:11] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:21] ethernet 0/1/9 [ortelius] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:21] ethernet 0/1/12 [csw1-esams:1/24] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:21] ethernet 0/1/21 [ptolemy] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:21] ethernet 0/1/22 [hyacinth] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:34] we should disable these nagios-checks for first… [15:58:42] ethernet 0/1/15 [ts-array3.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:42] ethernet 0/1/3 [rosemary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:42] ethernet 0/1/4 [wolfsbane] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:52] ethernet 0/1/5 [asw-oe16:0/1/22 LACP primary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:52] ethernet 0/1/18 [scs-oe10-esams] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:58] DaBPunkt: asked mark about the link stuff [16:02:20] ok mark recovered the link it seems [16:05:43] 3(created) [ACCAPP-535] Request key reset; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-535> (DeltaQuad) [16:07:44] 3(moved) [TS-1419] Request key reset <10https://jira.toolserver.org/browse/TS-1419> (DaB.) [16:13:45] 3(created) [TS-1420] Really properly installed the replacement disk into turnera?; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1420> (Marlen Caemmerer) [16:54:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:55:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:55:01] ethernet 0/1/19 [ptolemy.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:55:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 333046 MB (6% inode=35%): [16:55:11] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:55:11] ethernet 0/1/20 [hemlock.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:55:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:55:21] ethernet 0/1/13 [nightshade.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:55:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:55:42] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:55:42] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:55:42] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:56:32] ethernet 0/1/23 [rosemary.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:56:33] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:58:21] ethernet 0/1/12 [csw1-esams:1/24] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:58:22] ethernet 0/1/9 [ortelius] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:58:22] ethernet 0/1/21 [ptolemy] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:58:22] ethernet 0/1/22 [hyacinth] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:58:42] ethernet 0/1/15 [ts-array3.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:58:42] ethernet 0/1/3 [rosemary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:58:42] ethernet 0/1/4 [wolfsbane] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:59:01] ethernet 0/1/6 [asw-oe16:0/1/23 LACP secondary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:59:01] ethernet 0/1/1 [willow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:59:01] ethernet 0/1/10 [hawthorn] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:59:02] ethernet 0/1/2 [nightshade] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:59:02] ethernet 0/1/7 [hemlock] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:59:11] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:59:51] ethernet 0/1/5 [asw-oe16:0/1/22 LACP primary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [16:59:52] ethernet 0/1/18 [scs-oe10-esams] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:54:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:55:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:55:01] ethernet 0/1/19 [ptolemy.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:55:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 344646 MB (6% inode=36%): [17:55:11] ethernet 0/1/20 [hemlock.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:55:11] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:55:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:55:21] ethernet 0/1/13 [nightshade.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:55:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:56:31] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:56:31] ethernet 0/1/23 [rosemary.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:56:41] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:56:41] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:56:41] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:58:41] ethernet 0/1/15 [ts-array3.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:58:41] ethernet 0/1/3 [rosemary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:58:41] ethernet 0/1/4 [wolfsbane] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:59:02] ethernet 0/1/6 [asw-oe16:0/1/23 LACP secondary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:59:11] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:59:21] ethernet 0/1/9 [ortelius] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:59:21] ethernet 0/1/21 [ptolemy] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:59:21] ethernet 0/1/12 [csw1-esams:1/24] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [17:59:21] ethernet 0/1/22 [hyacinth] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:00:01] ethernet 0/1/1 [willow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:00:01] ethernet 0/1/10 [hawthorn] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:00:01] ethernet 0/1/7 [hemlock] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:00:01] ethernet 0/1/2 [nightshade] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:00:52] ethernet 0/1/5 [asw-oe16:0/1/22 LACP primary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:00:52] ethernet 0/1/18 [scs-oe10-esams] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:54:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:55:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:55:01] ethernet 0/1/19 [ptolemy.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:55:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 341944 MB (6% inode=35%): [18:55:11] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:55:11] ethernet 0/1/20 [hemlock.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:55:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:55:20] ethernet 0/1/13 [nightshade.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:55:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:56:35] ethernet 0/1/23 [rosemary.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:56:35] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:56:35] [[Special:Log/newusers]] create 10 * AnnMillere54 * (New user account) [18:56:41] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:56:41] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:56:41] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:59:01] ethernet 0/1/6 [asw-oe16:0/1/23 LACP secondary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:59:12] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:59:21] ethernet 0/1/12 [csw1-esams:1/24] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:59:21] ethernet 0/1/9 [ortelius] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:59:21] ethernet 0/1/21 [ptolemy] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:59:21] ethernet 0/1/22 [hyacinth] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:59:42] ethernet 0/1/15 [ts-array3.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:59:42] ethernet 0/1/3 [rosemary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [18:59:42] ethernet 0/1/4 [wolfsbane] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:00:51] ethernet 0/1/18 [scs-oe10-esams] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:00:51] ethernet 0/1/5 [asw-oe16:0/1/22 LACP primary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:01:01] ethernet 0/1/1 [willow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:01:01] ethernet 0/1/10 [hawthorn] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:01:01] ethernet 0/1/7 [hemlock] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:01:01] ethernet 0/1/2 [nightshade] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:46:01] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:50:31] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [19:54:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:55:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:55:01] ethernet 0/1/19 [ptolemy.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:55:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 351130 MB (6% inode=36%): [19:55:20] ethernet 0/1/13 [nightshade.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:55:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:56:11] ethernet 0/1/20 [hemlock.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:56:11] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:56:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:56:41] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:56:41] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:56:41] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:57:30] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:57:31] ethernet 0/1/23 [rosemary.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:59:11] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:59:21] ethernet 0/1/9 [ortelius] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:59:22] ethernet 0/1/12 [csw1-esams:1/24] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:59:22] ethernet 0/1/21 [ptolemy] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:59:22] ethernet 0/1/22 [hyacinth] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:59:42] ethernet 0/1/15 [ts-array3.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:59:42] ethernet 0/1/3 [rosemary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [19:59:42] ethernet 0/1/4 [wolfsbane] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:00:01] ethernet 0/1/6 [asw-oe16:0/1/23 LACP secondary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:00:51] ethernet 0/1/18 [scs-oe10-esams] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:00:51] ethernet 0/1/5 [asw-oe16:0/1/22 LACP primary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:01:02] ethernet 0/1/1 [willow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:01:02] ethernet 0/1/10 [hawthorn] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:01:02] ethernet 0/1/7 [hemlock] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:01:02] ethernet 0/1/2 [nightshade] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:05:11] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [20:05:21] MySQL slave on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [20:05:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:06:11] MySQL on thyme is OK: Uptime: 314 Threads: 10 Questions: 268 Slow queries: 0 Opens: 98 Flush tables: 1 Open tables: 88 Queries per second avg: 0.853 [20:06:21] MySQL slave on thyme is OK: Uptime: 321 Threads: 14 Questions: 867 Slow queries: 9 Opens: 159 Flush tables: 1 Open tables: 149 Queries per second avg: 2.700 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 708 [20:06:42] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 715.000000 [20:38:07] Hey folks [20:38:19] PHP sessions are fucked up I can't maintain a session in any of my tools [20:38:40] I use login, sets session, then after the post it redirects and that page redirects me back to login because I'm no longer logged in accordin to the session [20:38:48] the PHPSESSID is different after every POST [20:39:18] this has worked fine for years, startedbreaking up since today. [20:39:23] haven't changed anything [20:47:41] Krinkle: they rebooted servers [20:48:08] so it's possible that there is some problem in session storing now [20:54:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:55:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:55:01] ethernet 0/1/19 [ptolemy.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:55:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 342570 MB (6% inode=35%): [20:55:20] ethernet 0/1/13 [nightshade.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:55:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:56:12] ethernet 0/1/20 [hemlock.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:56:12] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:56:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:56:41] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:56:41] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:57:32] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:57:32] ethernet 0/1/23 [rosemary.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:57:42] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:59:21] ethernet 0/1/9 [ortelius] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:59:21] ethernet 0/1/12 [csw1-esams:1/24] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:59:21] ethernet 0/1/21 [ptolemy] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:59:42] ethernet 0/1/3 [rosemary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [20:59:42] ethernet 0/1/4 [wolfsbane] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:00:01] ethernet 0/1/6 [asw-oe16:0/1/23 LACP secondary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:00:11] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:00:21] ethernet 0/1/22 [hyacinth] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:00:21] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39872 MB (9% inode=99%): [21:00:41] ethernet 0/1/15 [ts-array3.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:00:50] ethernet 0/1/18 [scs-oe10-esams] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:00:51] ethernet 0/1/5 [asw-oe16:0/1/22 LACP primary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:02:01] ethernet 0/1/1 [willow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:02:01] ethernet 0/1/10 [hawthorn] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:02:01] ethernet 0/1/7 [hemlock] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:02:01] ethernet 0/1/2 [nightshade] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:14:13] @replag nlwiki [21:14:14] valhallasw: s2-user: 5d 6h 29m 3s [-]; s3-rr-a: 34s [-]; s3-user: 34s [-] [21:14:22] @help [21:14:22] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.8.2.4 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [21:14:22] valhallasw: (help [] []) -- This command gives a useful description of what does. is only necessary if the command is in more than one plugin. [21:14:29] @help replag [21:14:30] Unknown command type @commands for a list of all commands I know [21:14:30] valhallasw: replag Show current replag [21:15:21] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 63581 MB (15% inode=99%): [21:17:56] @replag [21:17:56] DaBPunkt: s2-user: 5d 6h 16m 32s [-3.38 s/s] [21:23:56] DaBPunkt: was willow rebooted? [21:24:23] Betacommand: yes. Mark was in the datacenter today and he added the memory [21:24:28] Ah [21:24:31] I will wirte a short email later [21:24:41] was wondering why my screens where down :) [21:25:13] DaBPunkt: read krinkle at 22:38 pls [21:26:20] Danny_B|backup: doesn't sound like a problem on our side [21:30:37] DaBPunkt: how much RAM was added? [21:30:57] new 8gb for willow [21:34:20] the userland-servers have now all 16GB memory. The HA-nodes have (GB now [21:34:29] ( → 8 [21:42:02] DaB. * [Toolserver-announce] Reboot of willow today [21:54:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:55:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:55:01] ethernet 0/1/19 [ptolemy.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:55:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 342408 MB (6% inode=35%): [21:55:20] ethernet 0/1/13 [nightshade.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:55:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:56:42] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:56:42] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:57:11] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:57:11] ethernet 0/1/20 [hemlock.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:57:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:57:42] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:58:31] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:58:31] ethernet 0/1/23 [rosemary.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:59:20] ethernet 0/1/9 [ortelius] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:59:21] ethernet 0/1/21 [ptolemy] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [21:59:21] ethernet 0/1/12 [csw1-esams:1/24] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:00:01] ethernet 0/1/6 [asw-oe16:0/1/23 LACP secondary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:00:11] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:00:21] ethernet 0/1/22 [hyacinth] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:00:41] ethernet 0/1/15 [ts-array3.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:00:41] ethernet 0/1/3 [rosemary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:00:41] ethernet 0/1/4 [wolfsbane] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:00:51] ethernet 0/1/5 [asw-oe16:0/1/22 LACP primary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:00:51] ethernet 0/1/18 [scs-oe10-esams] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:02:01] ethernet 0/1/1 [willow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:02:01] ethernet 0/1/10 [hawthorn] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:02:01] ethernet 0/1/7 [hemlock] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:03:01] ethernet 0/1/2 [nightshade] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:54:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:55:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:55:01] ethernet 0/1/19 [ptolemy.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:55:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 342301 MB (6% inode=35%): [22:55:21] ethernet 0/1/13 [nightshade.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:55:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:56:41] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:56:41] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:57:10] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:57:11] ethernet 0/1/20 [hemlock.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:57:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:58:31] ethernet 0/1/23 [rosemary.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:58:31] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:58:41] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:59:21] ethernet 0/1/12 [csw1-esams:1/24] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:59:21] ethernet 0/1/21 [ptolemy] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [22:59:21] ethernet 0/1/9 [ortelius] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:00:01] ethernet 0/1/6 [asw-oe16:0/1/23 LACP secondary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:00:11] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:00:21] ethernet 0/1/22 [hyacinth] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:00:42] ethernet 0/1/15 [ts-array3.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:00:42] ethernet 0/1/3 [rosemary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:00:42] ethernet 0/1/4 [wolfsbane] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:01:51] ethernet 0/1/18 [scs-oe10-esams] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:01:51] ethernet 0/1/5 [asw-oe16:0/1/22 LACP primary] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:02:01] ethernet 0/1/1 [willow] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:02:01] ethernet 0/1/10 [hawthorn] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:02:01] ethernet 0/1/7 [hemlock] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:03:01] ethernet 0/1/2 [nightshade] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:21:43] nacht ts [23:39:48] 3(created) [TS-1422] PHP sessions broken; Toolserver; Blocker Bug <10https://jira.toolserver.org/browse/TS-1422> (Krinkle) [23:41:38] Toolserver exploding, horses flying from the sky. everything is broken. the end of the world. it was nice knowing you. [23:41:44] 3(commented) [TS-1423] Can't create files in /home <10https://jira.toolserver.org/browse/TS-1423> (Krinkle) [23:41:44] 3(created) [TS-1423] Can't create files in /home; Toolserver; Blocker Bug <10https://jira.toolserver.org/browse/TS-1423> (Krinkle) [23:41:50] Block limit reached on /home [23:41:53] wut? [23:43:02] http://status.toolserver.org/ looks fine [23:49:02] Krinkle: you have been set quota obviously [23:49:17] It didn't say /home/krinkle [23:49:18] Krinkle: use /mnt/user-store/ to store some of your stuff [23:49:26] aha [23:49:27] oki [23:49:41] it is also affecting random other multi-maintainer projects that I work on, and I deleted several files bigger than the one I am trying to upload [23:50:02] Hazard-SJ * Re: [Toolserver-l] Running weblinkchecker.py on Toolserver [23:50:04] Krinkle: nosy set up quota for everybody (see the mail post) [23:50:08] but I wouldn't be surprised if that is the case, I have no idea [23:50:15] either way, if that is it, then the error handling is terrible [23:50:20] so i assume this is the cause [23:50:23] (i.e. creating a file, but blocking chmod/contents) [23:50:31] see those 2 bugs I opened [23:50:48] php session is related [23:50:58] I know, those are stored in home [23:51:01] php sessions are stored in home [23:51:03] yup [23:53:25] Krinkle: what does your quota -v return? [23:53:47] :D [23:53:52] /home 3.2G 500.0M 1000.0M NOT STARTED 250025 0 0 [23:54:06] Disk quotas for krinkle (uid 8108): [23:54:06] Filesystem usage quota limit timeleft files quota limit timeleft [23:54:12] /home 3.2G 500.0M 1000.0M NOT STARTED 250025 0 0 [23:54:42] there we go [23:54:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:54:51] Danny_B|backup: but that doesn't' explain the problem [23:55:02] ethernet 0/1/19 [ptolemy.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:55:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:55:06] if you are over quota, you can't write in your home [23:55:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 340862 MB (6% inode=35%): [23:55:22] ethernet 0/1/13 [nightshade.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:55:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:55:29] Danny_B|backup: and, according to the mail it is not being applied yet until after the weekend [23:55:39] once i went over and it destroyed my config files [23:56:13] I clean up regularly, I need the space I'm using and I know I get get the quota if I need it (up to 5GB without questions). The quota is not active yet. [23:56:18] Krinkle: try to move 2,7g to usrstore and then try to create something in your home. just to be sure... [23:56:42] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:56:43] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:56:53] I can't. Almost all is in active use. Or would be re-created if missing [23:56:59] I don't have time for this nonsense. [23:57:02] :/ [23:57:12] ethernet 0/1/20 [hemlock.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:57:12] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:57:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:57:17] not your fault, but I can't focus on this now. [23:58:02] who is nosy on irc? [23:58:04] nothing else than quotas has been set up today so far i remember, so i would blame that. but i'm not admin nor linux pro [23:58:05] what it was set now? [23:58:06] wtf [23:58:20] DaB said we'd have the weekend at least to make request [23:58:21] nosy is nosy here [23:58:30] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:58:31] ethernet 0/1/23 [rosemary.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:58:34] I didn't even get the chance to ask for the quota, just bang like that, frozen and screw it [23:58:36] Thanks! [23:58:43] argh [23:58:52] ethernet 0/1/17 [zedler.mgmt] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:59:21] ethernet 0/1/21 [ptolemy] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:59:21] ethernet 0/1/12 [csw1-esams:1/24] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:59:22] ethernet 0/1/9 [ortelius] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:59:28] "A default quota for each user without a quota was set tonight. " [23:59:38] Krinkle: ^^