[00:09:38] (03CR) 10Ori.livneh: [C: 032] Parametrize Graphite's DOCUMENTATION_URL option & set it to [[wikitech:Graphite]] [operations/puppet] - 10https://gerrit.wikimedia.org/r/100194 (owner: 10Ori.livneh) [00:40:51] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [02:04:26] !log LocalisationUpdate completed (1.23wmf5) at Sun Dec 8 02:04:26 UTC 2013 [02:04:42] Logged the message, Master [02:08:15] !log LocalisationUpdate completed (1.23wmf6) at Sun Dec 8 02:08:14 UTC 2013 [02:08:28] Logged the message, Master [02:19:11] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Dec 8 02:19:11 UTC 2013 [02:19:28] Logged the message, Master [02:31:46] PROBLEM - Puppet freshness on sq80 is CRITICAL: Last successful Puppet run was Sat 07 Dec 2013 11:31:16 PM UTC [04:34:53] (03CR) 10BryanDavis: [C: 031] "Nrpe itself has a timeout for each check that is set in /etc/nagios/nrpe.cfg. The value set there on tin is 60 seconds. Looking at the pup" [operations/puppet] - 10https://gerrit.wikimedia.org/r/99410 (owner: 10Hashar) [04:35:37] (03PS2) 10Ori.livneh: nrpe: let us specify timeout of check_nrpe [operations/puppet] - 10https://gerrit.wikimedia.org/r/99410 (owner: 10Hashar) [04:36:04] (03CR) 10Ori.livneh: [C: 032 V: 032] nrpe: let us specify timeout of check_nrpe [operations/puppet] - 10https://gerrit.wikimedia.org/r/99410 (owner: 10Hashar) [05:31:55] PROBLEM - Puppet freshness on sq80 is CRITICAL: Last successful Puppet run was Sat 07 Dec 2013 11:31:16 PM UTC [06:34:33] ori-l: I hope you took advantage of your temporary daftness to place a redirect from the title you expected to the "right" one :) [08:32:31] PROBLEM - Puppet freshness on sq80 is CRITICAL: Last successful Puppet run was Sat 07 Dec 2013 11:31:16 PM UTC [08:56:58] (03PS2) 10M4tx: Update favicon piece.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100161 [08:59:06] (03CR) 10M4tx: "The problem is that the 16x16 version is based on different SVG file than 32x32 (what is nonsense for me, but ok...). I already uploaded a" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100161 (owner: 10M4tx) [10:22:17] (03CR) 10Odder: [C: 031] "Feel free to fix such obvious nonsenses in the future :-) The icon submitted in this patch looks OK, thanks!" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100161 (owner: 10M4tx) [11:33:21] PROBLEM - Puppet freshness on sq80 is CRITICAL: Last successful Puppet run was Sat 07 Dec 2013 11:31:16 PM UTC [14:34:02] PROBLEM - Puppet freshness on sq80 is CRITICAL: Last successful Puppet run was Sat 07 Dec 2013 11:31:16 PM UTC [14:42:35] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [15:03:05] PROBLEM - MySQL Replication Heartbeat on db73 is CRITICAL: CRIT replication delay 311 seconds [15:03:15] PROBLEM - MySQL Slave Delay on db73 is CRITICAL: CRIT replication delay 320 seconds [15:51:21] PROBLEM - MySQL Slave Delay on db73 is CRITICAL: CRIT replication delay 302 seconds [15:52:02] PROBLEM - MySQL Replication Heartbeat on db73 is CRITICAL: CRIT replication delay 312 seconds [15:54:02] RECOVERY - MySQL Replication Heartbeat on db73 is OK: OK replication delay 131 seconds [15:54:21] RECOVERY - MySQL Slave Delay on db73 is OK: OK replication delay 145 seconds [16:02:56] (03PS1) 10Gerrit Patch Uploader: Bug: 58183 Updated with new favicon, wikiquote.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100212 [16:03:00] (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100212 (owner: 10Gerrit Patch Uploader) [17:12:39] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [17:34:29] PROBLEM - Puppet freshness on sq80 is CRITICAL: Last successful Puppet run was Sat 07 Dec 2013 11:31:16 PM UTC [17:38:10] (03CR) 10Odder: [C: 031] "Looks nice, thanks!" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100212 (owner: 10Gerrit Patch Uploader) [18:10:58] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:13:58] RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical [18:37:02] PROBLEM - MySQL Replication Heartbeat on db73 is CRITICAL: CRIT replication delay 306 seconds [18:37:22] PROBLEM - MySQL Slave Delay on db73 is CRITICAL: CRIT replication delay 321 seconds [18:45:42] PROBLEM - Host ms-be1006 is DOWN: PING CRITICAL - Packet loss = 100% [18:48:59] (03Abandoned) 10Umherirrender: enable Echo on all beta.wmflabs.org-wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95450 (owner: 10Umherirrender) [19:06:23] RECOVERY - MySQL Slave Delay on db73 is OK: OK replication delay 145 seconds [19:07:03] RECOVERY - MySQL Replication Heartbeat on db73 is OK: OK replication delay 108 seconds [19:49:59] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (210281) [19:55:59] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [20:19:53] (03PS1) 10Andrew Bogott: Labs: Record puppet run status in instance metadata [operations/puppet] - 10https://gerrit.wikimedia.org/r/100221 [20:22:58] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (214936) [20:25:25] (03PS2) 10Andrew Bogott: Labs: Record puppet run status in instance metadata [operations/puppet] - 10https://gerrit.wikimedia.org/r/100221 [20:29:25] (03CR) 10Andrew Bogott: [C: 032] Labs: Record puppet run status in instance metadata [operations/puppet] - 10https://gerrit.wikimedia.org/r/100221 (owner: 10Andrew Bogott) [20:35:15] PROBLEM - Puppet freshness on sq80 is CRITICAL: Last successful Puppet run was Sat 07 Dec 2013 11:31:16 PM UTC [20:47:00] (03PS1) 10Gerrit Patch Uploader: Bug: 58187 Fixed internals.ico favicon. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100326 [20:47:05] (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100326 (owner: 10Gerrit Patch Uploader) [21:19:56] (03PS1) 10Ori.livneh: Use glib's GHashTable instead of persistent BDB [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/100329 [21:25:41] mutante: mind looking up an RT ticket for me? [21:25:43] (03CR) 10Ori.livneh: "The production UDP profiling stream is duplicated to port 3812 on tungsten.eqiad.wmnet, so you can test this there. The files are checked " [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/100329 (owner: 10Ori.livneh) [21:26:06] or perhaps jeremyb [21:26:25] sure [21:27:50] #5531 jeremyb [21:28:16] !log disabling OSPFv3 on eqiad-esams [21:28:32] Logged the message, Master [21:30:10] twkozlowski: yana replied the same day as you last asked on the ticket. you got a copy of her reply? [21:30:37] it says it sent you a copy [21:30:38] that she placed an order with Mark Monitor? [21:30:46] yes [21:30:58] that's the latest there is on the ticket [21:31:05] still it doesn't explain why some of the domains are free, and why some of them have been squatted. [21:31:37] * jeremyb wonders if twkozlowski should mail yana about that? :) [21:32:09] I'm in a legal action against the WMF, perhaps not. [21:32:28] srsly?? [21:32:36] srsly wut [21:37:05] twkozlowski: AFAIK she said she only can't talk with us about the commuity logo, not everything [21:37:24] I see no issues in replying to that ticket [21:37:38] btw "legal action against" is an overstatement IMHO [21:38:27] these are her words, Nemo_bis [21:38:49] but I hope I will be able to communicate with her freely soon enough. [21:39:08] sure, so quit the jokes my dear [21:39:29] oh, the Milanese lack of sense of humour... [21:50:22] tsk [22:06:07] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [23:36:15] PROBLEM - Puppet freshness on sq80 is CRITICAL: Last successful Puppet run was Sat 07 Dec 2013 11:31:16 PM UTC