[00:04:39] [[Special:Log/newusers]] create 10 * Nope453 * (New user account) [00:43:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:43:28] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:43:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 341211 MB (6% inode=35%): [00:44:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:45:51] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [00:45:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:43:08] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:43:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:43:30] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 341246 MB (6% inode=35%): [01:44:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:45:20] @replag [01:45:21] Brooke: s2-user: 22s [+0.00 s/s]; s3-rr-a: 43s [+0.00 s/s]; s3-user: 43s [+0.00 s/s] [01:46:12] Toolserver database corruption known? [01:46:29] Dispenser: You around? [01:46:41] yup [01:46:50] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [01:46:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:46:51] Dispenser: mysql> select * from page where page_namespace = 0 and page_title = 'Netaji_Subhas_Chandra_Bose_International_Airport'; [01:46:56] on enwiki_p / thyme [01:48:45] 3(created) [TS-1431] thyme's enwiki_p appears to be corrupt; Toolserver; Bug <10https://jira.toolserver.org/browse/TS-1431> (MZMcBride) [02:00:27] Brooke: thyme has one less article than rosemary. How do you manage to find them? [02:03:33] https://en.wikipedia.org/wiki/Wikipedia:Database_reports/Broken_redirects [02:03:37] And the talk page. [02:04:04] mysql> select * from page where page_namespace = 0 and page_title = 'Flores_warty_pig'; [02:04:07] Empty set (0.00 sec) [02:04:08] https://en.wikipedia.org/wiki/Flores_warty_pig [02:04:11] Gay. [02:05:59] /sql on thyme is WARNING: DISK WARNING - free space: /sql 187730 MB (19% inode=99%): [02:07:44] 3(commented) [TS-1428] cron jobs not running intermittently? <10https://jira.toolserver.org/browse/TS-1428> (MZMcBride) [02:18:59] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.278320/1.10, alarm hl:np_load_long=0.398438/1.55, alarm hl:mem_free=465.000000M/500M, alarm hl:tmp_free=13394M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.278320/1.00, alarm hl:np_load_long=0.398438/1.50, alarm hl:mem_free=465.000000M/600M, alarm hl:tmp_free= [02:28:00] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [02:35:00] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.333984/1.10, alarm hl:np_load_long=0.337402/1.55, alarm hl:mem_free=415.000000M/500M, alarm hl:tmp_free=13376M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.333984/1.00, alarm hl:np_load_long=0.337402/1.50, alarm hl:mem_free=415.000000M/600M, alarm hl:tmp_free= [02:43:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:43:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:43:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 339207 MB (6% inode=35%): [02:44:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:46:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:46:50] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [02:48:00] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [02:54:08] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71077 MB (6% inode=99%): [02:57:17] It appears that site_stats increments with a ss_total_edits++; method, I've made a short script to compare clusters [02:58:50] We (apparently) only have redundancies for enwiki and commonswiki, and both are corrupt [03:00:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70470 MB (5% inode=99%): [03:01:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 72158 MB (6% inode=99%): [03:10:28] Are we missing 60,000 edits on commons? [03:20:48] I think I've figured out a way to automatically determine corruption [03:21:08] detect* [03:26:08] Detection is usually the easy part. Have you figured out a way to prevent it or fix it quickly? ;-) [03:28:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71307 MB (6% inode=99%): [03:38:10] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70981 MB (5% inode=99%): [03:41:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71964 MB (6% inode=99%): [03:43:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:43:10] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70986 MB (5% inode=99%): [03:44:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:44:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:44:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 338996 MB (6% inode=35%): [03:46:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:46:50] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [04:07:33] If its so easy why haven't you automated it? [04:15:10] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71050 MB (6% inode=99%): [04:20:10] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70774 MB (5% inode=99%): [04:30:13] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=7416&oldid=7402&rcid=10031 * 91.198.174.202 * (+1) (updated page) [04:39:10] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71300 MB (6% inode=99%): [04:42:34] I've getting conflicting data. Rosemary site_stats is messed up while thyme are correct. And our dewiki ss_total_edits is 9 edit higher than the API. [04:44:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:44:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:45:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:45:30] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 338343 MB (6% inode=35%): [04:47:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:47:51] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [04:52:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70988 MB (5% inode=99%): [05:45:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:45:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:46:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:46:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 337982 MB (6% inode=35%): [05:47:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:48:50] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [06:21:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 72289 MB (6% inode=99%): [06:22:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 69972 MB (5% inode=99%): [06:45:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:46:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:47:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:47:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 337484 MB (6% inode=35%): [06:47:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:48:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [07:00:08] /sql on cassia is WARNING: DISK WARNING - free space: /sql 74244 MB (6% inode=99%): [07:01:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 69976 MB (5% inode=99%): [07:45:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:46:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:47:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:47:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 337279 MB (6% inode=35%): [07:47:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:48:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [08:23:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71997 MB (6% inode=99%): [08:24:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 69370 MB (5% inode=99%): [08:46:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:46:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:47:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:48:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:48:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 336953 MB (6% inode=35%): [08:48:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [08:56:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71494 MB (6% inode=99%): [08:57:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 69701 MB (5% inode=99%): [09:22:43] 3(created) [ACCAPP-537] Public key; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-537> [09:25:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71127 MB (6% inode=99%): [09:26:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70042 MB (5% inode=99%): [09:30:19] Load avg. on adenia is WARNING: WARNING - load average: 17.20, 15.71, 11.99 [09:43:03] Marlen Caemmerer * [Toolserver-l] What is slow [09:44:19] Load avg. on adenia is OK: OK - load average: 7.45, 14.36, 14.34 [09:47:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:47:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:48:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:48:30] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 336613 MB (6% inode=35%): [09:48:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:49:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [10:23:10] /sql on cassia is WARNING: DISK WARNING - free space: /sql 73700 MB (6% inode=99%): [10:24:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 69426 MB (5% inode=99%): [10:27:36] [[Special:Log/newusers]] create 10 * YasBot * (New user account) [10:34:44] 3(resolved) [TS-1428] cron jobs not running intermittently? <10https://jira.toolserver.org/browse/TS-1428> (DaB.) [10:47:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:47:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:48:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:49:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 336156 MB (6% inode=35%): [10:49:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:49:48] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [11:47:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:47:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:48:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:49:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:49:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 335705 MB (6% inode=35%): [11:49:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [11:50:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 72422 MB (6% inode=99%): [11:51:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 68808 MB (5% inode=99%): [12:09:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71485 MB (6% inode=99%): [12:10:10] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70184 MB (5% inode=99%): [12:47:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:47:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:48:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:48:49] 3(assigned) [TS-1432] adenia is sometimes really slow <10https://jira.toolserver.org/browse/TS-1432> (Marlen Caemmerer) [12:48:51] 3(created) [TS-1432] adenia is sometimes really slow; Toolserver; Bug <10https://jira.toolserver.org/browse/TS-1432> (Marlen Caemmerer) [12:49:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:49:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 334941 MB (6% inode=35%): [12:49:50] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [12:53:50] 3(resolved) [TS-1423] Can't create files in /home <10https://jira.toolserver.org/browse/TS-1423> (Marlen Caemmerer) [13:08:44] 3(commented) [TS-1418] Broken fans in asw-oe10 <10https://jira.toolserver.org/browse/TS-1418> (Marlen Caemmerer) [13:11:49] 3(commented) [TS-1401] New SSH keypair <10https://jira.toolserver.org/browse/TS-1401> (Marlen Caemmerer) [13:11:51] 3(resolved) [TS-1375] Hard disk failure in turnera <10https://jira.toolserver.org/browse/TS-1375> (Marlen Caemmerer) [13:13:45] 3(assigned) [TS-1431] thyme's enwiki_p appears to be corrupt <10https://jira.toolserver.org/browse/TS-1431> (Marlen Caemmerer) [13:15:43] 3(commented) [TS-1431] thyme's enwiki_p appears to be corrupt <10https://jira.toolserver.org/browse/TS-1431> (Marlen Caemmerer) [13:18:11] Is Marlen Caemmerer in here? [13:18:56] Can someone update about thyme's enwiki_p? [13:19:10] I apparently filed a duplicate JIRA ticket because the status hadn't been updated. [13:30:44] 3(resolved) [TS-1420] Really properly installed the replacement disk into turnera? <10https://jira.toolserver.org/browse/TS-1420> (Marlen Caemmerer) [13:34:47] 3(commented) [TS-1425] Raise quota 'cvn' <10https://jira.toolserver.org/browse/TS-1425> (Marlen Caemmerer) [13:40:43] 3(commented) [TS-1401] New SSH keypair <10https://jira.toolserver.org/browse/TS-1401> (inductiveload) [13:45:44] 3(commented) [TS-1425] Raise quota 'cvn' <10https://jira.toolserver.org/browse/TS-1425> (Krinkle) [13:47:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:47:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:48:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:49:28] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:49:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 333473 MB (6% inode=35%): [13:49:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [14:05:59] /sql on thyme is WARNING: DISK WARNING - free space: /sql 188333 MB (19% inode=99%): [14:12:29] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 2202922s failure: longrun-sol@willow in error state: QERROR as result of job 2202922s failure [14:34:29] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [14:47:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:47:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:48:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:49:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:49:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 333193 MB (6% inode=35%): [14:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [15:19:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 72146 MB (6% inode=99%): [15:30:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70932 MB (5% inode=99%): [15:33:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71113 MB (6% inode=99%): [15:47:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:47:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70730 MB (5% inode=99%): [15:47:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:48:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:49:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:49:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 331685 MB (6% inode=35%): [15:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [15:51:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71091 MB (6% inode=99%): [16:03:03] Jeremy Baron * Re: [Toolserver-l] What is slow [16:06:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70990 MB (5% inode=99%): [16:43:49] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37132 MB (9% inode=99%): [16:47:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:47:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:49:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:49:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 331788 MB (6% inode=35%): [16:49:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [16:52:50] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24356 MB (5% inode=99%): [17:00:10] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71029 MB (6% inode=99%): [17:01:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 69848 MB (5% inode=99%): [17:07:50] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 48922 MB (12% inode=99%): [17:14:49] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 33965 MB (8% inode=99%): [17:27:49] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 68129 MB (16% inode=99%): [17:47:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:47:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:49:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:49:30] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 331684 MB (6% inode=35%): [17:49:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [17:51:45] Brooke: Maybe file a JIRA inquire about nosy's MIA on IRC [18:05:11] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71135 MB (6% inode=99%): [18:06:08] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70269 MB (5% inode=99%): [18:09:44] 3(commented) [TS-1431] thyme's enwiki_p appears to be corrupt <10https://jira.toolserver.org/browse/TS-1431> (MZMcBride) [18:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:49:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:50:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:50:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 330625 MB (6% inode=35%): [18:50:48] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [19:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:50:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:50:30] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 330545 MB (6% inode=35%): [19:50:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [19:54:20] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:59:19] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:04:14] hi there [20:30:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71034 MB (6% inode=99%): [20:31:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 71002 MB (5% inode=99%): [20:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:50:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [20:51:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:51:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 330332 MB (6% inode=35%): [21:27:50] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37622 MB (9% inode=99%): [21:39:49] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24272 MB (5% inode=99%): [21:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:50:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:50:50] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [21:51:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:51:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 330252 MB (6% inode=35%): [21:57:45] 3(commented) [TS-1425] Raise quota 'cvn' <10https://jira.toolserver.org/browse/TS-1425> (Marlen Caemmerer) [21:58:49] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 44456 MB (10% inode=99%): [22:17:50] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 67344 MB (16% inode=99%): [22:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:48:47] 3(commented) [TS-1425] Raise quota 'cvn' <10https://jira.toolserver.org/browse/TS-1425> (Krinkle) [22:50:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:50:48] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [22:51:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:51:30] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 330132 MB (6% inode=35%): [22:59:20] nacht ts [23:18:39] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1837 [23:19:09] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1838.000000 [23:21:39] MySQL slave on thyme is OK: Uptime: 444045 Threads: 14 Questions: 190944072 Slow queries: 61917 Opens: 37118 Flush tables: 1 Open tables: 2969 Queries per second avg: 430.10 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1797 [23:22:09] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1783.000000 [23:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:50:09] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1833.000000 [23:50:39] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1824 [23:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [23:50:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:51:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:52:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 328753 MB (6% inode=34%): [23:56:39] MySQL slave on thyme is OK: Uptime: 446146 Threads: 12 Questions: 191448421 Slow queries: 62361 Opens: 37215 Flush tables: 1 Open tables: 2976 Queries per second avg: 429.116 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1799 [23:57:09] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1779.000000