[01:06:09] (03PS1) 10BryanDavis: [WIP] Move beta scap source directory off of NFS [operations/puppet] - 10https://gerrit.wikimedia.org/r/127399 [02:13:13] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3080 MB (3% inode=99%): [02:19:13] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3746 MB (3% inode=99%): [02:29:35] !log LocalisationUpdate completed (1.23wmf22) at 2014-04-19 02:29:33+00:00 [02:29:42] Logged the message, Master [02:41:04] !log LocalisationUpdate completed (1.24wmf1) at 2014-04-19 02:41:02+00:00 [02:41:09] Logged the message, Master [02:56:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (202701) [03:00:13] RECOVERY - Disk space on virt0 is OK: DISK OK [03:27:57] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 19 03:27:52 UTC 2014 (duration 27m 51s) [03:28:03] Logged the message, Master [04:01:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [04:09:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200534) [04:12:27] ... [04:16:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [04:21:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (202599) [04:22:16] legoktm: "..." re check_job_queue? it does that, the 199,999 number was somewhat random and a fixed number isn't useful anyways. [04:22:32] spikes happen [04:22:50] greg-g: well it was more for the "the following wikis... : , " [04:23:03] legoktm: heh, touche [04:37:00] greg-g is asking legoktm to touch someone [04:37:08] you need an accent on the e :P [04:41:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [04:43:06] Jasper_Deng: lazy american typer :) [04:43:17] <^d> typist? [04:44:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200813) [04:44:52] ^d: whatever :P [04:45:22] * ^d wants someone to merge his puppet change so that job queue check will suck less soon. [04:46:53] <^d> I also want some juice and goldfish crackers. [04:46:56] <^d> Those are easier to get [04:46:57] * ^d fetches [04:49:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [04:52:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (202798) [05:00:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [05:01:03] (03PS2) 10Ori.livneh: Clean up job queue length check [operations/puppet] - 10https://gerrit.wikimedia.org/r/126148 (owner: 10Chad) [05:03:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200818) [05:04:25] (03CR) 10Ori.livneh: [C: 032] Clean up job queue length check [operations/puppet] - 10https://gerrit.wikimedia.org/r/126148 (owner: 10Chad) [05:05:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [05:08:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (201514) [05:16:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [05:19:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200439) [05:35:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [05:43:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (202245) [05:44:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [05:57:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (204552) [06:00:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [06:18:14] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200754) [06:20:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [06:28:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200069) [06:33:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [06:43:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200916) [06:52:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [06:56:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200347) [06:58:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [07:02:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200691) [07:07:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [08:37:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200272) [08:38:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [11:01:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (201542) [11:09:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [13:37:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200113) [13:38:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [13:41:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (202445) [13:44:14] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [13:50:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200491) [13:57:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [14:05:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (201418) [14:13:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [14:39:53] PROBLEM - Host wtp1018 is DOWN: PING CRITICAL - Packet loss = 100% [14:43:36] (03PS1) 10John F. Lewis: Add beta.m to Wikiversity DNS [operations/dns] - 10https://gerrit.wikimedia.org/r/127434 [15:09:16] (03CR) 10Prtksxna: TextExtracts: Add classes and elements to the exclusion list (033 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/126226 (owner: 10Prtksxna) [15:09:29] (03PS3) 10Prtksxna: TextExtracts: Add classes and elements to the exclusion list [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/126226 [15:47:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (203777) [15:53:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [15:56:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (201586) [16:29:20] Hi! today, when I try to test the mchnery server using an SMTP connection today, I get the error relay not permitted. Is it a new addition or some problem with my configs ? [17:53:46] (03PS1) 10John F. Lewis: Add checkuser(-log) permissions by default [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/127443 [18:18:05] (03CR) 10Hoo man: [C: 04-1] "pending extensions/CheckUser change" (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/127443 (owner: 10John F. Lewis) [18:18:28] hoo :p [18:19:11] JohnLewis: If there's a nit to pick, I'll do it :P [18:19:23] Sorry for the style complaining... [18:21:33] (03PS2) 10John F. Lewis: Add checkuser(-log) permissions by default [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/127443 [22:47:09] (03PS1) 10QChris: Add cp301[34] to mobile caches [operations/puppet] - 10https://gerrit.wikimedia.org/r/127456 [23:01:05] (03CR) 10QChris: "According to SAL, bblack added the machines in pybal, hence" [operations/puppet] - 10https://gerrit.wikimedia.org/r/127456 (owner: 10QChris)