[00:01:14] !log reedy updated /a/common to {{Gerrit|I54a2246d7}}: db1060 to full steam [00:01:18] (03PS1) 10Reedy: Remove $wgCategoryTreeDynamicTag = true [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107008 [00:01:21] Logged the message, Master [00:03:04] (03CR) 10Reedy: [C: 032] Remove $wgCategoryTreeDynamicTag = true [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107008 (owner: 10Reedy) [00:03:12] (03Merged) 10jenkins-bot: Remove $wgCategoryTreeDynamicTag = true [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107008 (owner: 10Reedy) [00:04:05] !log reedy synchronized wmf-config/ '//gerrit.wikimedia.org/r/107008' [00:04:13] Logged the message, Master [00:38:05] (03CR) 10Andrew Bogott: "Is this about bug 59662? I think that was resolved a while ago." [operations/puppet] - 10https://gerrit.wikimedia.org/r/106907 (owner: 10Stwalkerster) [00:42:22] (03CR) 10Stwalkerster: "It's related, yes. I thought that bug was fixed , and for about 95% of cases the existing fix is suitable. It wasn't until today that I re" [operations/puppet] - 10https://gerrit.wikimedia.org/r/106907 (owner: 10Stwalkerster) [02:09:57] (03PS1) 10Tim Landscheidt: WIP: Puppetize aliases and NAT for LabsDB replica servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/107010 [02:10:32] (03CR) 10jenkins-bot: [V: 04-1] WIP: Puppetize aliases and NAT for LabsDB replica servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/107010 (owner: 10Tim Landscheidt) [02:12:44] !log LocalisationUpdate completed (1.23wmf9) at Sun Jan 12 02:12:44 UTC 2014 [02:12:51] Logged the message, Master [02:23:36] !log LocalisationUpdate completed (1.23wmf10) at Sun Jan 12 02:23:36 UTC 2014 [02:23:42] Logged the message, Master [02:42:39] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Jan 12 02:42:38 UTC 2014 [02:42:44] Logged the message, Master [06:06:57] (03PS1) 10MZMcBride: Scale back deployment of UniversalLanguageSelector (ULS) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107014 [07:54:18] PROBLEM - MySQL Processlist on db1006 is CRITICAL: CRIT 1 unauthenticated, 0 locked, 0 copy to table, 67 statistics [07:55:18] RECOVERY - MySQL Processlist on db1006 is OK: OK 1 unauthenticated, 0 locked, 0 copy to table, 1 statistics [08:39:42] (03CR) 1001tonythomas: [C: 031] Changed date format in l10nupdate-1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/106892 (owner: 10Tinaj1234) [09:02:11] morning paravoid i haven't buged you in some days. i have started to reduce our webserver pp's and need some advice/guidence, can you help please? [09:20:06] (03CR) 10Nemo bis: "Example output:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/106892 (owner: 10Tinaj1234) [09:34:59] (03CR) 10Nikerabbit: [C: 04-2] "Yay what a great idea. Let's try it out and break everything." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107014 (owner: 10MZMcBride) [09:38:13] (03CR) 10Peachey88: "> More seriously, it has already been pointed why this setting cannot be changed." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107014 (owner: 10MZMcBride) [09:42:49] Nikerabbit, seriously though - is there a way to fully disable ULS for me? [09:42:49] (03CR) 10MZMcBride: "Nike: I'm not sure what you mean. I'm fairly sure that Wikimedia wikis operated for many years prior to this setting being enabled by defa" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107014 (owner: 10MZMcBride) [09:43:34] Like, COMPLETELY?:) [09:44:42] (03CR) 10Ori.livneh: [C: 04-2] "> Please feel free to elaborate on such a comment." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107014 (owner: 10MZMcBride) [09:46:45] MaxSem: officially no, you can find some gadgets around the wikis most likely [09:47:50] cause its fonts are absolutely awful [09:48:08] serif font for cyrillic languages, really? [09:48:16] seriously though, my happy sunday is ruined, please don't mention ULS to me [09:48:23] I don't want to discuss it now [09:48:29] Nikerabbit, ULS ULS ULS ULS [09:48:41] it ruins my every day;) [10:04:42] MaxSem: that wasn't nice [10:15:10] (03CR) 10Nemo bis: [C: 04-1] "1) On this patch: doesn't work due to dependencies." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107014 (owner: 10MZMcBride) [10:26:52] MaxSem: don't you think [13:51:12] (03PS1) 10Matanya: identd:lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/107032 [14:23:37] (03PS1) 10Matanya: gridengine: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107035 [14:26:45] (03CR) 10Nemo bis: "Added Leslie to reviewers, in case stats like these (exim deferred delivery/queued messages?) can help monitor weirdnesses like the messag" [operations/puppet] - 10https://gerrit.wikimedia.org/r/101117 (owner: 10Nemo bis) [14:57:52] (03PS1) 10Matanya: generic: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107037 [14:59:41] hmm, why in site.pp we have for sodium: enable_mail_relay => "secondary", while nowhere can a primary be seen? [15:00:31] Nemo_bis: mchanry is primary [15:04:19] mchenry* [15:06:26] that's what I know as well, but where is it defined [17:15:28] PROBLEM - Solr on zinc is CRITICAL: Average request time is 1000.0105 (gt 1000) [17:26:05] (03CR) 10MZMcBride: "> 1) On this patch: doesn't work due to dependencies." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107014 (owner: 10MZMcBride) [17:59:18] PROBLEM - MySQL Processlist on db1006 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 92 statistics [18:00:18] RECOVERY - MySQL Processlist on db1006 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 25 statistics [18:32:36] (03CR) 10Guido.iaquinti: [C: 04-1] "modules/generic/manifests/upstart_job.pp" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/107037 (owner: 10Matanya) [18:49:52] (03CR) 10Guido.iaquinti: [C: 031] gridengine: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107035 (owner: 10Matanya) [18:51:14] (03CR) 10Guido.iaquinti: [C: 031] identd:lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/107032 (owner: 10Matanya) [18:55:18] PROBLEM - MySQL Processlist on db1006 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 82 statistics [18:56:18] RECOVERY - MySQL Processlist on db1006 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 2 statistics [19:14:12] (03PS2) 10Tim Landscheidt: WIP: Puppetize aliases and NAT for LabsDB replica servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/107010 [19:15:16] (03CR) 10jenkins-bot: [V: 04-1] WIP: Puppetize aliases and NAT for LabsDB replica servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/107010 (owner: 10Tim Landscheidt) [19:25:34] Gerrit seems to be hanging (intermittenly) again. [19:28:57] (03CR) 10Tim Landscheidt: [C: 04-1] WIP: Puppetize aliases and NAT for LabsDB replica servers (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/107010 (owner: 10Tim Landscheidt) [19:36:53] (03PS2) 10Matanya: generic: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107037 [19:39:49] (03CR) 10Guido.iaquinti: [C: 031] generic: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107037 (owner: 10Matanya) [19:41:32] (03CR) 10Guido.iaquinti: "Maybe it could be good also if we try to understand why the previous build passed" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107037 (owner: 10Matanya) [19:46:03] (03CR) 10Tim Landscheidt: "@Guido.iaquinti: Could you file a bug for that, please?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107037 (owner: 10Matanya) [19:51:32] !log gerrit lockup on ytterbium - dozens of gerrit procs busy-waiting on futex [19:51:37] Logged the message, Master [19:53:33] !log gerrit intermittently unresponsive; restarting. [19:53:39] Logged the message, Master [19:53:58] Who named ytterbium? [19:54:31] dunno [19:56:17] chemicists? [20:00:43] could be [20:08:51] (03CR) 10Guido.iaquinti: "@Tim Landscheidt: https://bugzilla.wikimedia.org/show_bug.cgi?id=59980" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107037 (owner: 10Matanya) [20:11:21] (03CR) 10Guido.iaquinti: [C: 031] generic: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107037 (owner: 10Matanya) [21:12:47] ori: gerrit was limping all day long [21:12:57] yesterday too [21:13:10] yes [21:13:17] really bad preformance [21:13:45] the cpu usage jump in spiks, i looked at ganglia [21:27:37] ori: do you have time/will to review some of my patches? [21:29:38] matanya: probably not until later tonight PST (in 10 hours or so). i looked over the latest version of your SVN patch. it looks good. i need to find a time to merge it where i (or someone else) can look at the machine and verify the merge. [21:29:51] you can add me as a reviewer to the other ones; i don't think you have. [21:30:25] no, i haven't i didn't know if you are willing to do it, now i know :) [21:31:09] yeah happy to [21:31:36] thanks, added you to all [21:32:46] i started working on reforming the webserver stuff, and spoke to andrewbogott, but i think it would be better doing it in pair mode [22:20:17] (03PS1) 10Tim Landscheidt: WIP: Tools: Puppetize LabsDB aliases and /etc/hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/107103 [22:20:52] (03CR) 10jenkins-bot: [V: 04-1] WIP: Tools: Puppetize LabsDB aliases and /etc/hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/107103 (owner: 10Tim Landscheidt) [22:28:28] PROBLEM - Varnish HTCP daemon on cp1055 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:28:28] PROBLEM - Varnish HTTP text-backend on cp1055 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:28:28] PROBLEM - Varnish traffic logger on cp1055 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:54:18] RECOVERY - Varnish HTCP daemon on cp1055 is OK: PROCS OK: 1 process with UID = 111 (vhtcpd), args vhtcpd [22:54:18] RECOVERY - Varnish HTTP text-backend on cp1055 is OK: HTTP OK: HTTP/1.1 200 OK - 189 bytes in 0.005 second response time [22:54:18] RECOVERY - Varnish traffic logger on cp1055 is OK: PROCS OK: 2 processes with command name varnishncsa [23:36:11] (03PS1) 10Nemo bis: [Planet] Update "voices" feed URL: old one gaves error 500 [operations/puppet] - 10https://gerrit.wikimedia.org/r/107105 [23:41:35] (03PS1) 10Nemo bis: [Planet] Add Virginia Gentilini to Italian Planet [operations/puppet] - 10https://gerrit.wikimedia.org/r/107106 [23:43:28] (03PS2) 10Nemo bis: [Planet] Update "voices" feed URL: old one gaves error 500 [operations/puppet] - 10https://gerrit.wikimedia.org/r/107105