[00:06:38] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37269 MB (9% inode=99%): [00:08:39] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 64853 MB (16% inode=99%): [00:16:39] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38245 MB (9% inode=99%): [00:30:12] [[Special:Log/newusers]] create 10 * وائل بسيوني * (New user account) [00:30:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.05, 20.96, 18.60 [00:31:39] Load avg. on willow is WARNING: WARNING - load average: 24.77, 20.76, 18.68 [00:35:52] 3(commented) [MNT-1192] s3 and s7 slaves were stopped <10https://jira.toolserver.org/browse/MNT-1192> (DaB.) [00:36:19] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [00:36:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:36:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:36:30] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [00:37:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:37:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:37:49] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:37:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126097 MB (20% inode=99%): [00:38:20] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 82464 [00:42:52] 3(commented) [MNT-1192] s3 and s7 slaves were stopped <10https://jira.toolserver.org/browse/MNT-1192> (DaB.) [00:44:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 38180 MB (3% inode=99%): [01:00:38] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.64, 21.45, 18.62 [01:01:39] Load avg. on willow is WARNING: WARNING - load average: 26.22, 21.61, 18.86 [01:15:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.87, 21.52, 20.16 [01:36:20] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [01:36:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:36:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:37:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:37:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:37:29] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [01:37:38] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36834 MB (9% inode=99%): [01:37:50] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:38:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125943 MB (20% inode=99%): [01:39:19] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50269 [01:41:39] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23516 MB (5% inode=99%): [01:45:39] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38273 MB (9% inode=99%): [01:50:39] Load avg. on willow is WARNING: WARNING - load average: 22.18, 20.44, 19.81 [02:00:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 56.23, 29.82, 23.02 [02:36:20] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [02:36:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:36:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:37:50] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:38:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:38:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:38:29] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [02:39:20] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35782 [02:39:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125862 MB (20% inode=99%): [03:00:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.22, 23.72, 22.17 [03:10:39] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38944 MB (9% inode=99%): [03:17:39] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23666 MB (5% inode=99%): [03:18:39] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 85698 MB (21% inode=99%): [03:30:39] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37565 MB (9% inode=99%): [03:36:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:36:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:36:39] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 48855 MB (12% inode=99%): [03:37:20] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [03:38:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:38:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:38:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:38:28] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [03:39:20] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23334 [03:39:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125795 MB (20% inode=99%): [03:45:00] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.012695/1.00, alarm hl:np_load_long=0.649414/1.50, alarm hl:mem_free=21284.000000M/300M, alarm hl:available=1/0 [03:46:01] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [03:51:49] Load avg. on willow is WARNING: WARNING - load average: 15.38, 17.89, 19.63 [04:06:02] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.217774/1.00, alarm hl:np_load_long=0.885742/1.50, alarm hl:mem_free=22209.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.217774/1.10, alarm hl:np_load_long=0.885742/1.50, alarm hl:mem_free=22209.000000M/300M, alarm hl:available=1/0 [04:06:49] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.46, 21.65, 20.00 [04:12:39] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40130 MB (9% inode=99%): [04:14:39] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 63383 MB (15% inode=99%): [04:36:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:36:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:37:19] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [04:38:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:38:39] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [04:38:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:38:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:39:40] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2982 [04:40:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125722 MB (20% inode=99%): [04:41:40] MySQL slave on z-dat-s7-a is OK: Uptime: 587901 Threads: 7 Questions: 113863191 Slow queries: 26020 Opens: 718589 Flush tables: 1 Open tables: 3993 Queries per second avg: 193.677 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1349 [04:54:01] Load avg. on willow is WARNING: WARNING - load average: 17.85, 19.27, 19.91 [05:01:01] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.61, 20.83, 20.20 [05:29:00] Load avg. on willow is WARNING: WARNING - load average: 17.20, 19.14, 19.96 [05:31:00] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.61, 20.91, 20.52 [05:34:59] Load avg. on willow is WARNING: WARNING - load average: 17.42, 19.11, 19.86 [05:36:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:36:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:37:19] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [05:38:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:38:40] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [05:38:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:38:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:40:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125450 MB (20% inode=99%): [05:49:11] can someone tell me how to get job scheduling to work on the toolserver? However I had it set up before isn't working anymore :( [05:49:44] Did you read ? [05:50:12] What commands are you running and where and what happens when you run them? You know how to get help on IRC... [05:50:13] yes, it was throroughly confusing [05:50:36] Yeah, I complained about that when River wrote it. [05:50:45] I think she at least added a simplified section. [05:51:08] "Quick start," yeah. [05:52:31] The quick start is just about how to convert a job from cron (or possibly cronie) to the job scheduler, but I don't even have a working cronjob any more [05:53:08] Well, if you pastebin what you're actually running (commands and such) and what's happening, someone might be able to help. [05:53:21] if I want to just set up a cronjob am I supposed to use "crontab -e" or "cronie -e" or "cronietab -e" or what? [05:53:35] I just want to run 1 comand, nothing fancy [05:53:37] cronie is Vixie cron. [05:53:39] Oh. [05:53:46] Just use cronie, then. [05:53:50] cronie -e [05:53:53] thanks [05:54:01] that's all I need to know [05:54:14] Heh. [05:54:22] too bad it doesn't say that at the top of the docuementation page in bold letters [05:54:31] maybe I'll be bold :) [05:54:55] Well, cronie was hacked into Solaris because Solaris' cron doesn't support */foo syntax. [05:55:05] And some other minor features were missing, I think. [05:55:21] But River was big into job scheduling and SGE and shit, so she tried to make it seem like that's what everyone should use. [05:55:30] Even though it's about two or three more times complicated. [05:55:52] Whatever I use always breaks after a few months :( [05:55:57] In thoery, it would help with load balancing and such. [05:56:10] Well, whether you use cron or cronie, you should put your jobs in a text file and sync that. [05:56:30] Using "cronie ~/path/to/crontab.txt" or whatever. [05:56:53] That at least means you'll have a backup of the text file if your crontab gets deleted (for whatever reason). [05:56:56] :-) [05:57:02] good idea [05:57:38] I think my favorite example of instability and the Toolserver is . [05:58:57] :) [05:59:01] that seem about right [05:59:04] seems [06:01:31] [[Job scheduling]] ! 10https://wiki.toolserver.org/w/index.php?diff=6722&oldid=6719&rcid=8882 * 208.87.217.46 * (+4) (/* Converting an existing cron job to use the scheduler */ link) [06:04:26] [[Cronie]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6723&rcid=8883 * Kaldari * (+18) (#REDIRECT [[Cron]]) [06:07:00] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.05, 27.30, 24.16 [06:10:08] wasn't me :) [06:36:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:37:00] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:37:18] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [06:38:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:38:39] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [06:38:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:39:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125345 MB (20% inode=99%): [07:03:40] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 35555 MB (8% inode=99%): [07:06:39] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22523 MB (5% inode=99%): [07:07:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.21, 25.82, 25.18 [07:21:39] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 32060 MB (7% inode=99%): [07:36:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:37:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:37:19] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [07:38:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:38:40] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [07:38:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:39:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124267 MB (20% inode=99%): [08:08:01] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.25, 22.09, 22.62 [08:16:25] @replag all [08:16:25] Anselm_WMDE: s1-rr-a: 2s [-0.00 s/s]; s1-rr-a-c: 3s [-0.00 s/s]; s1-user: 2s [-0.00 s/s]; s2-user: 5d 7h 14m 47s [+0.21 s/s]; s2-user-c: 3s [-0.00 s/s]; s3-rr-a: 38s [+0.00 s/s]; s3-user: 38s [+0.00 s/s]; s4-rr-a: 3s [-0.01 s/s] [08:16:26] Anselm_WMDE: s4-user: 3s [-0.01 s/s]; s5-rr-a: 0s [-0.00 s/s]; s5-user: 0s [-0.00 s/s]; s5-user-c: 3s [-0.00 s/s]; s6-rr-a: 4s [-0.00 s/s]; s6-user: 4s [-0.00 s/s]; s7-rr-a: 5s [-1.62 s/s]; s7-user: 5s [-1.62 s/s] [08:36:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:37:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:37:19] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [08:38:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:38:40] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) [08:38:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:39:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:40:10] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124322 MB (20% inode=99%): [09:08:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:09:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:26:34] Load avg. on willow is WARNING: WARNING - load average: 17.57, 18.50, 19.98 [09:27:05] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39803 MB (9% inode=99%): [09:28:04] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 67388 MB (16% inode=99%): [09:30:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.89, 21.50, 20.63 [09:35:35] Load avg. on willow is WARNING: WARNING - load average: 18.04, 19.27, 19.89 [09:49:04] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36459 MB (8% inode=99%): [09:50:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.55, 21.26, 20.00 [09:51:26] [[User talk:Sahim]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6724&rcid=8884 * Nemobis * (+324) (Created page with "==Bot== Hello, I've seen that you're running bots on fa projects apparently continuously. Please add your bot to the [[List of Wikimedia bots]]. Moreover, you might be interested...") [09:51:34] Load avg. on willow is WARNING: WARNING - load average: 21.19, 20.77, 19.91 [09:53:04] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23703 MB (5% inode=99%): [09:54:04] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 53344 MB (13% inode=99%): [10:00:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.14, 22.25, 20.46 [10:08:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:08:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124243 MB (20% inode=99%): [10:08:44] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:08:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:09:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:21:33] Load avg. on willow is WARNING: WARNING - load average: 17.38, 19.14, 19.98 [10:27:52] 3(updated) [ACCAPP-400] Requesting access to tool server to calculate statistics for a sociology research project <10https://jira.toolserver.org/browse/ACCAPP-400> (Ahmed Medhat Mohamed) [11:08:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:08:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124231 MB (20% inode=99%): [11:08:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:09:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:09:54] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:11:03] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38240 MB (9% inode=99%): [11:16:04] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22591 MB (5% inode=99%): [11:17:04] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 46021 MB (11% inode=99%): [11:21:34] Load avg. on willow is WARNING: WARNING - load average: 18.21, 18.56, 18.57 [12:00:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.67, 20.47, 18.83 [12:01:34] Load avg. on willow is WARNING: WARNING - load average: 24.01, 20.34, 18.88 [12:02:55] [[Special:Log/newusers]] create 10 * Jm34harvey * (New user account) [12:08:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:08:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123948 MB (20% inode=99%): [12:08:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:09:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:10:43] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:30:35] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.64, 21.26, 19.16 [12:31:34] Load avg. on willow is WARNING: WARNING - load average: 23.10, 20.72, 19.09 [12:44:54] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 37928 MB (3% inode=99%): [12:49:54] Hello all [12:49:57] @replag [12:49:58] DaBPunkt: s2-user: 4d 21h 8m 1s [-2.22 s/s]; s3-rr-a: 28s [-0.00 s/s]; s3-user: 28s [-0.00 s/s] [12:50:56] [[Category:Communication]] ! 10https://wiki.toolserver.org/w/index.php?diff=6725&oldid=6566&rcid=8886 * 115.87.97.76 * (+247) () [12:51:15] [[Category:Communication]] ! 10https://wiki.toolserver.org/w/index.php?diff=6726&oldid=6725&rcid=8887 * 115.87.97.76 * (-247) (Undo revision 6725 by [[Special:Contributions/115.87.97.76|115.87.97.76]] ([[User talk:115.87.97.76|talk]])) [12:55:52] 3(created) [MNT-1207] Import commonswiki on daphne; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1207> (DaB.) [12:57:53] 3(commented) [MNT-1192] s3 and s7 slaves were stopped <10https://jira.toolserver.org/browse/MNT-1192> (DaB.) [12:59:55] 3(created) [MNT-1208] Switch z-dat-s6-a back to r/w; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1208> (DaB.) [13:01:52] 3(resolved) [MNT-1208] Switch z-dat-s6-a back to r/w <10https://jira.toolserver.org/browse/MNT-1208> (DaB.) [13:08:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:08:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123758 MB (20% inode=99%): [13:08:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:09:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:10:44] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:32:34] Load avg. on willow is WARNING: WARNING - load average: 21.68, 21.80, 19.80 [13:50:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.43, 21.71, 20.34 [13:53:35] Load avg. on willow is WARNING: WARNING - load average: 17.51, 19.98, 19.91 [13:55:35] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.26, 20.91, 20.20 [14:04:14] [[Special:Log/newusers]] create 10 * Mehrajmir13 * (New user account) [14:08:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123391 MB (20% inode=99%): [14:08:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:09:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:09:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:10:53] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:21:03] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40381 MB (9% inode=99%): [14:22:34] Load avg. on willow is WARNING: WARNING - load average: 16.98, 18.45, 19.22 [14:27:04] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 100645 MB (24% inode=99%): [14:30:35] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.53, 21.83, 19.99 [14:31:34] Load avg. on willow is WARNING: WARNING - load average: 24.71, 21.45, 19.97 [14:45:24] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:46:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:51:20] DaBPunkt: cronie on willow has issues: [14:51:54] couldn't set locale correctly [14:51:55] /bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8) [14:54:07] re [14:54:28] Danny_B|backup: did you changed anything? [14:57:39] nope, i just copied the one from nightshade to willow [14:57:53] no jobs run :-/ [14:59:41] Danny_B|backup: try "en_US.UTF-8" [15:02:30] will see in few mins [15:08:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123200 MB (20% inode=99%): [15:08:55] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:09:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:10:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.94, 22.28, 20.21 [15:11:45] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:46:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:49:44] Load avg. on willow is WARNING: WARNING - load average: 17.73, 18.74, 19.96 [15:50:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.79, 19.48, 20.14 [15:51:44] Load avg. on willow is WARNING: WARNING - load average: 16.77, 18.59, 19.79 [16:08:55] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:09:44] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 114565 MB (18% inode=99%): [16:10:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:11:53] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:30:44] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.64, 22.79, 20.60 [16:37:44] Load avg. on willow is WARNING: WARNING - load average: 17.77, 19.70, 19.94 [16:40:44] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.39, 20.16, 20.05 [16:46:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:09:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:09:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122141 MB (20% inode=99%): [17:09:46] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.75, 23.16, 22.55 [17:10:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:11:54] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:25:54] 3(commented) [ACCAPP-441] SVN, Database access and storage <10https://jira.toolserver.org/browse/ACCAPP-441> [17:46:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:48:45] Load avg. on willow is WARNING: WARNING - load average: 17.67, 18.91, 19.99 [17:54:44] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.246094/1.00, alarm hl:np_load_long=0.917969/1.50, alarm hl:mem_free=21242.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.246094/1.10, alarm hl:np_load_long=0.917969/1.50, alarm hl:mem_free=21242.000000M/300M, alarm hl:available=1/0 [17:55:44] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.86, 20.08, 20.03 [17:55:45] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [17:59:44] Load avg. on willow is WARNING: WARNING - load average: 19.76, 19.90, 19.98 [18:09:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:09:44] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 118913 MB (19% inode=99%): [18:10:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:12:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:19:44] Load avg. on willow is CRITICAL: CRITICAL - load average: 17.72, 19.39, 20.09 [18:22:45] Load avg. on willow is WARNING: WARNING - load average: 18.12, 19.19, 19.91 [18:39:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.10, 20.71, 20.10 [18:47:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:03:46] Load avg. on willow is WARNING: WARNING - load average: 8.97, 15.88, 19.62 [19:09:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:09:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123336 MB (20% inode=99%): [19:10:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:12:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:47:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:04:46] Load avg. on willow is WARNING: WARNING - load average: 13.24, 16.16, 17.80 [20:09:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:10:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:10:44] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122942 MB (20% inode=99%): [20:12:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:20:14] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:21:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:25:15] [[User talk:Sahim]] ! 10https://wiki.toolserver.org/w/index.php?diff=6727&oldid=6724&rcid=8889 * Sahim * (+81) () [20:25:39] [[List of Wikimedia bots]] ! 10https://wiki.toolserver.org/w/index.php?diff=6728&oldid=6716&rcid=8890 * Sahim * (+79) (/* {{#language:fa}} */ ) [20:28:35] [[List of Wikimedia bots]] ! 10https://wiki.toolserver.org/w/index.php?diff=6729&oldid=6728&rcid=8891 * Sahim * (+97) (/* {{#language:ar}} */ ) [20:28:44] Load avg. on willow is OK: OK - load average: 12.84, 13.44, 14.89 [20:29:34] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:30:52] 3(commented) [TS-1309] daphne s2/s5 corrupted and offline <10https://jira.toolserver.org/browse/TS-1309> (Maciej Jaros) [20:32:47] Load avg. on willow is WARNING: WARNING - load average: 14.54, 15.62, 15.57 [20:32:49] [[List of Wikimedia bots]] ! 10https://wiki.toolserver.org/w/index.php?diff=6730&oldid=6729&rcid=8892 * Sahim * (+82) (/* {{#language:en}} */ ) [20:34:54] 3(commented) [TS-1309] daphne s2/s5 corrupted and offline <10https://jira.toolserver.org/browse/TS-1309> (DaB.) [20:35:04] 3(resolved) [TS-1309] daphne s2/s5 corrupted and offline <10https://jira.toolserver.org/browse/TS-1309> (DaB.) [20:44:55] Load avg. on willow is OK: OK - load average: 13.32, 14.34, 14.93 [20:46:22] [[List of Wikimedia bots]] ! 10https://wiki.toolserver.org/w/index.php?diff=6731&oldid=6730&rcid=8893 * Sahim * (+113) (/* {{#language:en}} */ ) [20:47:22] [[List of Wikimedia bots]] ! 10https://wiki.toolserver.org/w/index.php?diff=6732&oldid=6731&rcid=8894 * Sahim * (+91) (/* {{#language:fr}} */ ) [20:47:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:47:53] @replag [20:47:53] Sebaso_WMDE: s1-rr-a-c: 11m 22s [+0.02 s/s]; s2-user: 4d 5h 21m 5s [-1.98 s/s]; s3-rr-a: 24s [-0.00 s/s]; s3-user: 24s [-0.00 s/s] [20:48:57] [[List of Wikimedia bots]] ! 10https://wiki.toolserver.org/w/index.php?diff=6733&oldid=6732&rcid=8895 * Sahim * (+84) (/* {{#language:it}} */ ) [20:51:10] [[List of Wikimedia bots]] ! 10https://wiki.toolserver.org/w/index.php?diff=6734&oldid=6733&rcid=8896 * Sahim * (+103) (/* {{#language:fa}} */ ) [20:51:59] [[List of Wikimedia bots]] ! 10https://wiki.toolserver.org/w/index.php?diff=6735&oldid=6734&rcid=8897 * Sahim * (+84) (/* {{#language:it}} */ ) [20:52:48] [[List of Wikimedia bots]] !M 10https://wiki.toolserver.org/w/index.php?diff=6736&oldid=6735&rcid=8898 * Sahim * (+84) (/* {{#language:fr}} */ ) [21:10:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:10:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122565 MB (20% inode=99%): [21:12:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:15:55] Load avg. on willow is WARNING: WARNING - load average: 13.59, 14.61, 15.19 [21:17:55] Load avg. on willow is OK: OK - load average: 11.98, 13.82, 14.83 [21:21:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:27:55] Load avg. on willow is WARNING: WARNING - load average: 15.15, 15.24, 15.14 [21:29:35] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:40:24] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1814.000000 [21:45:24] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1703.000000 [21:48:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:10:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:11:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122307 MB (20% inode=99%): [22:12:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:18:24] Environment on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:20:14] Environment on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [22:21:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:27:55] Load avg. on willow is WARNING: WARNING - load average: 15.12, 15.89, 16.16 [22:29:34] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:39:25] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [22:49:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:02:35] @replag [23:02:35] Akoopal: s2-user: 4d 3h 2m 38s [-1.03 s/s]; s3-rr-a: 18s [-0.00 s/s]; s3-user: 18s [-0.00 s/s]; s6-rr-a: 14s [+0.00 s/s]; s6-user: 14s [+0.00 s/s] [23:09:04] Nickanc Wikipedia * Re: [Toolserver-l] MMP interwiki-bot [23:10:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:11:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122003 MB (20% inode=99%): [23:12:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:19:03] DaB. * Re: [Toolserver-l] MMP interwiki-bot [23:21:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:28:54] Load avg. on willow is WARNING: WARNING - load average: 17.31, 18.48, 18.46 [23:48:36] nacht ts [23:49:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output