[00:03:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [00:12:32] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.010254/2.00, alarm hl:mem_free=129.000000M/250M [00:18:33] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:34:24] 3(commented) [TS-1251] operating my global bot "GedawyBot" on toolserver in order to add and update interwikis <10https://jira.toolserver.org/browse/TS-1251> (Mohamed ElGedawy) [00:41:25] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:42:05] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:44:05] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:46:05] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62454 MB (6% inode=99%): [00:49:05] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 183255 MB (3% inode=24%): [01:03:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:13:35] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.982422/2.00, alarm hl:mem_free=225.000000M/250M [01:15:34] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:18:34] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.044922/2.00, alarm hl:mem_free=210.000000M/250M [01:41:24] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:42:05] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:44:04] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:44:11] hi [01:46:05] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62371 MB (6% inode=99%): [01:49:06] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 182425 MB (3% inode=24%): [02:03:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [02:31:34] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.289062/1.00, alarm hl:mem_free=13388.000000M/100M [02:34:21] do we have an ETA for cluster sql restart? [02:34:35] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [02:34:45] the RESTART RUNNING is now more than 12 hours old [02:41:35] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:42:05] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:44:05] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:46:05] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62301 MB (6% inode=99%): [02:49:05] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 182268 MB (3% inode=24%): [02:55:35] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:55:35] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:55:35] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:55:35] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:55:35] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:56:05] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:56:34] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:56:34] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:56:34] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:56:34] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:56:34] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:56:34] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:56:53] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [02:57:06] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 132477 MB (33% inode=99%): [02:57:06] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 3391 MB (99% inode=99%): [02:57:06] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 3482 MB (99% inode=99%): [02:57:06] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 3473 MB (99% inode=99%): [02:57:06] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 3473 MB (99% inode=99%): [02:57:25] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:57:25] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:57:25] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:57:25] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:57:25] SMTP on z-dat-s6-a is OK: SMTP OK - 0.003 sec. response time [02:57:25] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:03:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [03:05:03] [[Special:Log/newusers]] create 10 * Ske * (New user account) [03:13:35] Sun Grid Engine execd on nightshade is WARNING: longrun@nightshade exceedes load threshold: alarm hl:np_load_long=1.022461/2.00, alarm hl:mem_free=166.000000M/250M [03:14:35] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [03:19:35] Sun Grid Engine execd on nightshade is WARNING: longrun@nightshade exceedes load threshold: alarm hl:np_load_long=1.130371/2.00, alarm hl:mem_free=142.000000M/250M [03:41:36] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:42:06] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:44:06] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:46:05] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62232 MB (6% inode=99%): [03:49:05] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 181957 MB (3% inode=24%): [04:03:36] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.734375/1.50, alarm hl:mem_free=304.000000M/100M [04:03:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [04:06:35] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [04:41:35] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:42:04] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:45:05] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:46:05] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62164 MB (6% inode=99%): [04:49:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 181913 MB (3% inode=24%): [05:03:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:41:36] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:42:05] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:45:05] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:46:05] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62986 MB (6% inode=99%): [05:49:05] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 181866 MB (3% inode=24%): [06:03:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [06:41:35] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:42:04] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:45:05] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:46:05] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62950 MB (6% inode=99%): [06:48:24] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 18.330 second response time [06:49:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 181636 MB (3% inode=24%): [06:50:15] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 274 bytes in 10.830 second response time [07:03:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [07:20:24] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 69099 MB (7% inode=99%): [07:42:05] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:42:34] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:45:04] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:46:04] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62879 MB (6% inode=99%): [07:49:05] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 181135 MB (3% inode=24%): [08:03:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [08:42:15] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:42:35] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:45:15] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:47:04] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62812 MB (6% inode=99%): [08:50:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 180568 MB (3% inode=24%): [08:52:45] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.512695/1.50, alarm hl:mem_free=513.000000M/100M [08:53:44] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [09:03:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [09:08:45] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.632324/1.50, alarm hl:mem_free=533.000000M/100M [09:16:45] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [09:20:30] [[User:Dcoetzee]] ! 10https://wiki.toolserver.org/w/index.php?diff=6494&oldid=4493&rcid=8533 * Dcoetzee * (+144) () [09:26:18] [[Permalinks]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6495&rcid=8534 * Dcoetzee * (+1149) (Create) [09:35:06] [[Duplication Detector]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6496&rcid=8535 * Dcoetzee * (+659) (Create) [09:42:16] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:42:45] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:45:15] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:47:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62734 MB (6% inode=99%): [09:50:15] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 180091 MB (3% inode=24%): [10:03:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [10:24:14] Load avg. on hyacinth is WARNING: WARNING - load average: 16.77, 11.08, 6.38 [10:28:14] Load avg. on hyacinth is OK: OK - load average: 13.79, 13.58, 8.61 [10:42:14] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:42:45] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:45:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:47:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62679 MB (6% inode=99%): [10:49:14] Load avg. on hyacinth is WARNING: WARNING - load average: 16.92, 13.66, 10.69 [10:50:15] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 179585 MB (3% inode=23%): [11:03:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [11:12:44] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.969727/2.00, alarm hl:mem_free=175.000000M/250M [11:13:45] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [11:16:44] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.007812/2.00, alarm hl:mem_free=177.000000M/250M [11:28:24] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:28:24] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:44] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:28:44] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:28:44] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:28:44] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:53] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:53] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:53] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:54] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:55] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:55] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:04] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [11:29:15] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 131587 MB (32% inode=99%): [11:29:15] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 3676 MB (99% inode=99%): [11:29:15] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 3676 MB (99% inode=99%): [11:29:15] SMTP on z-dat-s7-a is OK: SMTP OK - 0.283 sec. response time [11:29:34] SMTP on z-dat-s6-a is OK: SMTP OK - 0.003 sec. response time [11:29:45] SMTP on z-dat-s4-a is OK: SMTP OK - 0.005 sec. response time [11:29:45] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:29:45] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:29:45] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:29:45] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:29:45] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:30:15] Load avg. on hyacinth is OK: OK - load average: 8.82, 11.27, 14.95 [11:32:45] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.183594/1.00, alarm hl:mem_free=10443.000000M/100M [11:38:45] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [11:40:45] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.867188/2.00, alarm hl:mem_free=203.000000M/250M [11:41:44] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [11:42:14] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:42:45] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:45:15] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:45:44] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.912109/2.00, alarm hl:mem_free=230.000000M/250M [11:47:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62604 MB (6% inode=99%): [11:50:15] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 179007 MB (3% inode=23%): [12:03:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [12:23:45] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.053711/1.00, alarm hl:mem_free=13368.000000M/100M [12:38:54] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [12:42:25] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:43:44] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:46:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:46:44] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.021484/1.00, alarm hl:mem_free=13519.000000M/100M [12:48:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 30912 MB (3% inode=99%): [12:48:20] 3(created) [DBQ-172] List of all bot scripts in all wmf wikis (or at least something similiar); Database Queries; Task <10https://jira.toolserver.org/browse/DBQ-172> (Nickanc) [12:48:44] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [12:51:15] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 179399 MB (3% inode=23%): [13:02:45] Load avg. on willow is WARNING: WARNING - load average: 16.21, 14.66, 10.59 [13:02:54] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.879883/1.50, alarm hl:mem_free=277.000000M/100M [13:03:45] Load avg. on willow is OK: OK - load average: 9.76, 13.11, 10.30 [13:03:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [13:06:54] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [13:12:45] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.165039/2.00, alarm hl:mem_free=216.000000M/250M [13:13:21] 3(commented) [MAGNUS-272] File Upload Bot (Magnus Manske) sets wrong rotate template when importing flickr <10https://jira.toolserver.org/browse/MAGNUS-272> (Saibo) [13:19:21] 3(resolved) [DBQ-172] List of all bot scripts in all wmf wikis (or at least something similiar) <10https://jira.toolserver.org/browse/DBQ-172> (Hoo man) [13:21:22] 3(commented) [DBQ-172] List of all bot scripts in all wmf wikis (or at least something similiar) <10https://jira.toolserver.org/browse/DBQ-172> (Nickanc) [13:31:22] 3(resolved) [TS-1251] operating my global bot "GedawyBot" on toolserver in order to add and update interwikis <10https://jira.toolserver.org/browse/TS-1251> (DaB.) [13:31:23] 3(commented) [DBQ-172] List of all bot scripts in all wmf wikis (or at least something similiar) <10https://jira.toolserver.org/browse/DBQ-172> (Hoo man) [13:42:45] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.059570/2.00, alarm hl:mem_free=211.000000M/250M [13:43:25] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:44:45] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:46:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:48:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62442 MB (6% inode=99%): [13:49:54] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [13:51:14] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 177685 MB (3% inode=23%): [13:54:36] @replag [13:54:37] DaBPunkt: No replag found. Use "@replag all" to see all lags. [13:54:40] :) [14:03:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [14:43:25] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:44:45] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:46:24] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:48:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62346 MB (6% inode=99%): [14:51:15] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 177028 MB (3% inode=23%): [15:03:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [15:12:55] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.803223/2.00, alarm hl:mem_free=210.000000M/250M [15:13:55] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [15:44:24] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:44:54] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:46:24] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:48:25] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62250 MB (6% inode=99%): [15:48:25] [[Getting help]] 10https://wiki.toolserver.org/w/index.php?diff=6497&oldid=4888&rcid=8536 * Dab * (-342) (updated) [15:51:24] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 176417 MB (3% inode=23%): [16:03:56] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [16:10:47] [[Getting help]] 10https://wiki.toolserver.org/w/index.php?diff=6498&oldid=6497&rcid=8537 * Dab * (+36) () [16:11:46] [[Getting help]] M 10https://wiki.toolserver.org/w/index.php?diff=6499&oldid=6498&rcid=8538 * Dab * (+21) (/* Visitors */ ) [16:13:58] [[Getting help]] 10https://wiki.toolserver.org/w/index.php?diff=6500&oldid=6499&rcid=8539 * Dab * (+37) (/* Visitors */ ) [16:14:24] [[Getting help]] M 10https://wiki.toolserver.org/w/index.php?diff=6501&oldid=6500&rcid=8540 * Dab * (+3) (/* Visitors */ ) [16:44:25] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:45:55] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:46:24] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:48:24] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62131 MB (6% inode=99%): [16:51:24] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 175934 MB (3% inode=23%): [17:04:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [17:42:39] [[List of Wikimedia bots]] ! 10https://wiki.toolserver.org/w/index.php?diff=6502&oldid=6490&rcid=8541 * Ghaly * (+116) (/* {{#language:arz}} ) [17:44:30] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:46:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:46:29] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:49:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62039 MB (6% inode=99%): [17:52:19] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 175588 MB (3% inode=23%): [18:33:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:47:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:49:29] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 61946 MB (6% inode=99%): [18:52:29] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 175280 MB (3% inode=23%): [19:28:37] cu all in 2012 [19:29:09] and dab leaves just as all the IRC bots got nuked :( [19:29:36] well at least the willow ones [19:30:59] the bots are partying [19:45:39] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:46:16] [[Special:Log/newusers]] create 10 * Darkdowell * (New user account) [19:46:40] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:47:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:50:29] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62788 MB (6% inode=99%): [19:52:29] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 174605 MB (3% inode=23%): [20:33:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [20:36:50] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:36:59] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:37:28] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:37:39] SMTP on hyacinth is OK: SMTP OK - 0.097 sec. response time [20:45:40] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:46:39] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:47:20] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:50:40] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62711 MB (6% inode=99%): [20:52:40] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 173866 MB (3% inode=23%): [21:33:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [21:39:07] If anyone sees a security alert that someone tried to login with "root", that was my bad. Habbit. [21:46:00] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:47:00] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:48:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:50:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62639 MB (6% inode=99%): [21:52:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 173196 MB (3% inode=23%): [22:47:00] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:47:00] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:48:19] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:50:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62548 MB (6% inode=99%): [22:53:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 172665 MB (3% inode=23%): [23:03:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [23:47:00] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:47:00] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:48:29] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:50:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62447 MB (6% inode=99%): [23:54:00] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 170816 MB (3% inode=23%):