[00:02:05] AzaToth: seems to be fine now? [00:02:17] reseted it [00:02:34] (03PS2) 10MarkTraceur: WIP Add MMV feature flags for beta and pilot sites [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117376 [00:07:37] testing again YuviPanda [00:07:46] AzaToth: ok! [00:07:55] added a !_.isEmpty() before _.has() [00:15:30] YuviPanda: seems to work [00:15:37] AzaToth: woo! [00:15:53] AzaToth: want me to +2? [00:16:14] w8 [00:16:19] gonna do a test [00:16:25] AzaToth: ok [00:16:33] please join #wikimedia-qa [00:16:36] AzaToth: I'm going to go off now. feel free to just leave a commit in the local repo? [00:16:48] ok [00:16:50] AzaToth: and feel free to self-merge too [00:16:54] can't [00:16:54] AzaToth: after testing [00:17:09] I don't have permission to +2 or merge grrrit [00:17:10] AzaToth: oh? you don'thave +2? [00:17:16] AzaToth: let me see if i can fix that now [00:17:17] * YuviPanda fiddles [00:17:31] (03PS1) 10Tim Landscheidt: puppet::self::master: Specify IP range for eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/118030 [00:19:21] AzaToth: grr, I can't figure that out :( [00:19:30] AzaToth: leave them on tools, I'll +2 when I wake up. [00:19:37] AzaToth: and figure out how to give you +2 tomorrow! [00:19:42] AzaToth: thanks for fixing the issues :) [00:20:00] night! [00:21:12] nait [00:24:13] (03PS1) 10AzaToth: testing grrrit; please ignore [operations/puppet] - 10https://gerrit.wikimedia.org/r/118031 [00:24:41] argh, failed push [00:25:05] (03Abandoned) 10AzaToth: testing grrrit; please ignore [operations/puppet] - 10https://gerrit.wikimedia.org/r/118031 (owner: 10AzaToth) [00:25:44] forgot I can't make new branches [00:27:05] ah, good [00:27:31] <^d> Why can't people make branches? [00:27:38] at #wikimedia-betacluster: (PS1) AzaToth: testing grrrit; please ignore [operations/apache-config] (betacluster) - https://gerrit.wikimedia.org/r/118035 [00:27:47] ^d: I assume it's a permission [00:28:00] can make topics, but not new branches [00:28:02] <^d> Meh, should be granted on most things :\ [00:28:08] :/ [00:28:17] erm #wikimedia-qa [00:28:20] There's a wikimedia-betacluster channel [00:28:25] wth... [00:28:46] (03PS1) 10Tim Landscheidt: Tools: Rename references to local-admin to tools.admin [operations/puppet] - 10https://gerrit.wikimedia.org/r/118036 [00:28:46] hoo: I mixed it up in my head [00:29:20] hoo: I meant #wikimedia-qa [00:30:00] ok, I'm not in that one either [00:30:35] (03CR) 10Tim Landscheidt: "This is apparently the reason why Puppet is failing for tools-master.eqiad.wmflabs at the moment." [operations/puppet] - 10https://gerrit.wikimedia.org/r/118036 (owner: 10Tim Landscheidt) [00:31:04] (03PS1) 10AzaToth: Testing grrrit; please ignore [operations/apache-config] - 10https://gerrit.wikimedia.org/r/118038 [00:31:08] good [00:31:19] works now fine [00:31:44] commits to branch "betacluster" goes to #wikimedia-qa, all else goes whereever they should, for example here [00:32:32] hoo: so for now on, you will be totally oblivious what happens on betacluster [00:34:04] AzaToth: mh... I'm not going to idle in yet another channel [00:34:16] hoo: blame hashar [00:34:22] sadly I still have other things to do than idle in a billion IRC channels [00:35:01] hoo: long or short scale? [00:35:25] sorry? [00:35:38] hoo: billion in long or short scale? [00:36:20] AzaToth: Don't really get that question... but when I started MediaWiki Tech. stuff, I was in one, maybe two channels... now there's more ... [00:36:55] hoo: https://en.wikipedia.org/wiki/Long_and_short_scales [00:37:29] oh, that one :P [00:37:41] ツ [00:37:58] I'm in 20 odd channels [00:37:58] I don't think that matters much :P [00:38:05] (03PS1) 10Tim Landscheidt: Tools: Allow access for administrators from bastions [operations/puppet] - 10https://gerrit.wikimedia.org/r/118039 [00:38:27] I know you where hyperboling [00:38:34] :D [00:42:08] scfc_de: Want to review https://gerrit.wikimedia.org/r/113755 ? :) [00:44:55] (03Abandoned) 10AzaToth: Testing grrrit; please ignore [operations/apache-config] - 10https://gerrit.wikimedia.org/r/118038 (owner: 10AzaToth) [00:47:49] hoo: It's on my list, but I have some backlog re sql from last summer (!), so I need to process that first :-). [00:48:42] ok... I can relate to that, my review backlog also is quite insane :) [00:49:03] On labs, I'm seeing errors from ntpd every 30 s in syslog: "bind(22) AF_INET6 fe80::f816:3eff:fece:f4c%2#123 flags 0x11 failed: Cannot assign requested address", "unable to create socket on eth0 (14945) for fe80::f816:3eff:fece:f4c#123", "failed to init interface for address fe80::f816:3eff:fece:f4c". [00:49:22] Is that something important, or does someone have a clue how to silence that? [01:01:13] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [01:08:24] Found https://bugzilla.wikimedia.org/show_bug.cgi?id=60166 for ntpd issue. [01:09:43] PROBLEM - MySQL Slave Delay on db1007 is CRITICAL: CRIT replication delay 333 seconds [01:09:43] PROBLEM - MySQL Replication Heartbeat on db1007 is CRITICAL: CRIT replication delay 333 seconds [02:09:13] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [02:12:33] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200374) [02:12:35] !log LocalisationUpdate completed (1.23wmf16) at 2014-03-11 02:12:35+00:00 [02:12:51] Logged the message, Master [02:15:33] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [02:22:40] !log LocalisationUpdate completed (1.23wmf17) at 2014-03-11 02:22:39+00:00 [02:22:48] Logged the message, Master [02:42:52] (03CR) 10Andrew Bogott: [C: 032] puppet::self::master: Specify IP range for eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/118030 (owner: 10Tim Landscheidt) [02:48:26] * sDrewth looks up and down for csteipp [02:57:14] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Mar 11 02:57:11 UTC 2014 (duration 57m 10s) [02:57:24] Logged the message, Master [03:10:33] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (203624) [03:27:33] PROBLEM - MySQL Idle Transactions on db1018 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:28:24] RECOVERY - MySQL Idle Transactions on db1018 is OK: OK longest blocking idle transaction sleeps for 0 seconds [03:35:43] RECOVERY - MySQL Slave Delay on db1007 is OK: OK replication delay 79 seconds [03:35:43] RECOVERY - MySQL Replication Heartbeat on db1007 is OK: OK replication delay 78 seconds [03:49:33] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [03:54:33] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200127) [03:58:33] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [04:02:13] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [05:03:33] (03CR) 10ArielGlenn: [C: 032] convert oooold snapshot manifest to module [operations/puppet] - 10https://gerrit.wikimedia.org/r/117901 (owner: 10ArielGlenn) [05:10:13] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [06:25:38] (03PS1) 10Physikerwelt: Change home directory of vagrant user [operations/puppet] - 10https://gerrit.wikimedia.org/r/118053 [06:26:52] (03CR) 10jenkins-bot: [V: 04-1] Change home directory of vagrant user [operations/puppet] - 10https://gerrit.wikimedia.org/r/118053 (owner: 10Physikerwelt) [06:28:33] (03PS2) 10Physikerwelt: Change home directory of vagrant user [operations/puppet] - 10https://gerrit.wikimedia.org/r/118053 [06:30:52] (03PS3) 10Physikerwelt: Change home directory of vagrant user [operations/puppet] - 10https://gerrit.wikimedia.org/r/118053 [06:31:22] (03PS4) 10Physikerwelt: Change home directory of vagrant user [operations/puppet] - 10https://gerrit.wikimedia.org/r/118053 [06:33:30] (03PS5) 10Physikerwelt: Change home directory of vagrant user [operations/puppet] - 10https://gerrit.wikimedia.org/r/118053 [06:40:49] (03CR) 10Physikerwelt: "That might work. But I have no idea how to test that." [operations/puppet] - 10https://gerrit.wikimedia.org/r/118053 (owner: 10Physikerwelt) [07:03:13] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [07:15:04] (03CR) 10Physikerwelt: "one might also have to replace" [operations/puppet] - 10https://gerrit.wikimedia.org/r/118053 (owner: 10Physikerwelt) [07:18:51] (03CR) 10Alexandros Kosiaris: [C: 032] base: remove lookupvar and replace with top scope @ var [operations/puppet] - 10https://gerrit.wikimedia.org/r/117922 (owner: 10Matanya) [07:23:43] (03CR) 10Alexandros Kosiaris: [C: 04-1] ganglia_new: remove lookupvar and replace with top scope @ var and fix scoping (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/112889 (owner: 10Matanya) [07:25:12] (03PS2) 10Alexandros Kosiaris: openstack: remove var and replace with top scope ::var [operations/puppet] - 10https://gerrit.wikimedia.org/r/116976 (owner: 10Matanya) [07:26:38] (03CR) 10Alexandros Kosiaris: [C: 032] openstack: remove var and replace with top scope ::var [operations/puppet] - 10https://gerrit.wikimedia.org/r/116976 (owner: 10Matanya) [07:28:20] (03PS3) 10Matanya: ganglia_new: remove lookupvar and replace with top scope @ var and fix scoping [operations/puppet] - 10https://gerrit.wikimedia.org/r/112889 [08:11:13] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [08:48:07] matanya: what's the RT number for "Global_JobQueue_length ganglia graph empty since January"? [08:48:54] (03CR) 10Nemo bis: [C: 04-1] "Needs rebase" [operations/puppet] - 10https://gerrit.wikimedia.org/r/113639 (owner: 10Matanya) [08:51:05] Nemo_bis, 6771 [08:51:14] thanks [08:52:29] (03PS4) 10Matanya: remove darrell shell account [operations/puppet] - 10https://gerrit.wikimedia.org/r/113639 [09:01:42] andre__: is there a ticket for moving noc.wikimedia.org from fenari too? [09:02:49] Nemo_bis, 6862 [09:03:53] ok [09:06:19] (03PS3) 10Nemo bis: Use $wgTranslatePageTranslationULS [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/115879 [09:09:45] (03PS1) 10Nemo bis: Move Global_JobQueue_length ganglia graph to terbium in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/118058 [09:10:40] (03CR) 10Nemo bis: "Moving noc.wm.o to an eqiad host is RT 6862, should I make a patch for that too? I sent Idf779971a7b83a1efbde647edffbdaaf0429b8a8 to test " [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 (owner: 10Nemo bis) [09:17:45] (03CR) 10Matanya: Add cron job to run characterEditStats.php on multilingual wikis weekly (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 (owner: 10Nemo bis) [09:20:37] (03PS8) 10Physikerwelt: added basic hbase support [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 [09:22:33] (03Abandoned) 10Physikerwelt: Add Mathoid module (TeX -> MathML / SVG conversion web service) [operations/puppet] - 10https://gerrit.wikimedia.org/r/90733 (owner: 10Physikerwelt) [09:38:12] (03PS1) 10Andrew Bogott: Specify region-appropriate dns resolver. [operations/puppet] - 10https://gerrit.wikimedia.org/r/118060 [09:39:21] YuviPanda: ^ [09:42:24] (03CR) 10Yuvipanda: [C: 04-1] "Perhaps not have a default value for the resolver in the class, and always specify it in the role?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/118060 (owner: 10Andrew Bogott) [09:44:50] (03PS2) 10Andrew Bogott: Specify region-appropriate dns resolver. [operations/puppet] - 10https://gerrit.wikimedia.org/r/118060 [09:44:53] YuviPanda, better? [09:45:33] andrewbogott: yeah, looks good, provided it tests fine :) [09:47:07] (03CR) 10Andrew Bogott: [C: 032] Specify region-appropriate dns resolver. [operations/puppet] - 10https://gerrit.wikimedia.org/r/118060 (owner: 10Andrew Bogott) [09:51:37] (03CR) 10Alexandros Kosiaris: [C: 032] Move Global_JobQueue_length ganglia graph to terbium in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/118058 (owner: 10Nemo bis) [10:04:13] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [10:05:58] akosiaris_away, will ^^^ also mean that existing cronjobs on hume will get disabled? [10:32:52] MaxSem: no. but hume is to be disabled due to the tampa => eqiad migration anyway [10:33:19] yep, already discovered myself that it's still spilling fatals [10:33:39] want me to disable that ? [10:33:55] would it be a crime to just sudo -u mwdeploy crontab -e it? I have privs to do it myself [10:34:36] like a felony ? or a misdemeanor ? [10:35:04] feel free to do it, just log it please [10:35:12] at least like a "you'll be slapped for it":P [10:35:22] ok [10:37:02] !log Manually disabled old broken job queue cronjobs on hume [10:37:21] Logged the message, Master [10:45:36] (03PS3) 10Nemo bis: Add cron job to run characterEditStats.php on multilingual wikis weekly [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 [10:46:28] (03CR) 10Nemo bis: "Thanks for the review, I learnt to follow https://wikitech.wikimedia.org/wiki/Puppet_coding instead of copying code around mine. :)" (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 (owner: 10Nemo bis) [10:48:27] (03CR) 10Nemo bis: "It works! Someone close the RT ticket please. done Nemo_bis [10:52:38] and i'm glad my work is useful for you - re puppet coding :) [10:53:43] :) [10:53:57] Weird, now the old data disappeared and the recent data appeared, on hume https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Miscellaneous%20pmtpa&h=hume.wikimedia.org&r=year&z=default&jr=&js=&st=1394535162&v=1&m=Global%20JobQueue%20length&z=large [11:12:13] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [11:13:07] (03CR) 10Addshore: "Well if they already exist then removing them imo would be bad as they may be used by people." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/113972 (owner: 10Thiemo Mättig (WMDE)) [11:43:16] paravoid: [11:43:29] paravoid: I have approved you on phabricator instance [11:44:08] hashar: i would love to see that too :) <-- hint :P [11:44:28] matanya: not sure who can register at http://fab.wmflabs.org [11:44:35] can'y [11:44:36] *t [11:44:37] matanya: I think you need a @wikimedia.org email [11:45:04] :-( [11:45:04] gerrit is upset now [11:45:17] matanya: mail Chad Horohoe, he can probably add you [11:45:37] getent passwd l10nupdate [11:45:40] wrong window [11:46:56] how hashar it seems it did work [11:47:11] *oh [11:47:22] waiting for admin approval [11:55:55] matanya: approved [11:56:03] thanks [11:57:21] nap + commuting, will be back later in the afternoon [11:57:22] looks good [11:57:32] see ya [13:05:13] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [13:14:43] re [13:17:56] * Reedy rebots hashar [13:24:20] Reedy: hi ! ! do you have any clue how /usr/local/bin/sync-common is installed on the box ? Where does it come from ? [13:37:32] hashar: do you need an answer more specific than "puppet, probably"? [13:37:46] yeah will have to grep I guess : [13:37:47] D [13:37:53] with age I am getting lazier and lazier [13:43:31] I was waiting for my shell to be useable again ;) [13:44:19] hashar: I think it should just be a symlink now created by puppet [13:44:25] to /srv/scap/bin/sync-common' [13:45:37] And that's mediawiki/tools/scap.git or something [13:46:06] ahhh yeah /srv/scap [13:46:15] it is probably not fetched on the beta instance I am recreating [13:47:45] modules/mediawiki/manifests/sync.pp [13:47:52] mediawiki::sync [13:48:26] (03PS4) 10Nemo bis: Add cron job to run characterEditStats.php on multilingual wikis weekly [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 [13:48:39] (03CR) 10jenkins-bot: [V: 04-1] Add cron job to run characterEditStats.php on multilingual wikis weekly [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 (owner: 10Nemo bis) [13:49:11] (03PS5) 10Nemo bis: Add cron job to run characterEditStats.php on multilingual wikis weekly [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 [13:49:23] (03CR) 10jenkins-bot: [V: 04-1] Add cron job to run characterEditStats.php on multilingual wikis weekly [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 (owner: 10Nemo bis) [13:53:59] (03PS2) 10Reedy: Remove remnants of old CA RC2UDP config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117667 [13:54:03] (03CR) 10Reedy: [C: 032] Remove remnants of old CA RC2UDP config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117667 (owner: 10Reedy) [13:54:12] (03Merged) 10jenkins-bot: Remove remnants of old CA RC2UDP config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117667 (owner: 10Reedy) [13:54:31] (03PS6) 10Nemo bis: Add cron job to run characterEditStats.php on multilingual wikis weekly [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 [13:54:59] !log reedy synchronized wmf-config/CommonSettings.php 'I208d51b5db031d35518453e2b9de096f7f53f7a0' [13:55:07] Logged the message, Master [13:55:22] (03PS2) 10Reedy: Add logo for legalteamwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117377 (owner: 10Jalexander) [13:55:27] (03CR) 10Reedy: [C: 032] Add logo for legalteamwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117377 (owner: 10Jalexander) [13:55:39] (03PS7) 10Nemo bis: Add cron job to run characterEditStats.php on multilingual wikis weekly [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 [13:55:55] (03Merged) 10jenkins-bot: Add logo for legalteamwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117377 (owner: 10Jalexander) [13:56:06] (03CR) 10Nemo bis: "PS4 move to terbium, then rebase." [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 (owner: 10Nemo bis) [13:56:12] (03PS2) 10Reedy: Set wmgBabelCategoryNames for Chinese Wikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114970 (owner: 10Odder) [13:56:16] (03CR) 10Reedy: [C: 032] Set wmgBabelCategoryNames for Chinese Wikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114970 (owner: 10Odder) [13:56:24] (03Merged) 10jenkins-bot: Set wmgBabelCategoryNames for Chinese Wikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114970 (owner: 10Odder) [13:56:59] (03PS3) 10Reedy: Enable VisualEditor by default on French Wikiversity [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117238 (owner: 10Jforrester) [13:57:04] (03CR) 10Reedy: [C: 032] Enable VisualEditor by default on French Wikiversity [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117238 (owner: 10Jforrester) [13:57:11] (03Merged) 10jenkins-bot: Enable VisualEditor by default on French Wikiversity [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117238 (owner: 10Jforrester) [13:57:29] (03PS2) 10Reedy: 'Markbotedits' user right for rollbackers on shwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117807 (owner: 10Odder) [13:57:31] (03CR) 10jenkins-bot: [V: 04-1] 'Markbotedits' user right for rollbackers on shwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117807 (owner: 10Odder) [13:57:33] (03CR) 10Reedy: [C: 032] 'Markbotedits' user right for rollbackers on shwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117807 (owner: 10Odder) [13:57:42] (03Merged) 10jenkins-bot: 'Markbotedits' user right for rollbackers on shwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117807 (owner: 10Odder) [13:58:25] (03PS2) 10Reedy: Add Wikimedia CH domain to wgCopyUploadsDomains [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116263 (owner: 10Odder) [13:58:36] (03CR) 10Reedy: [C: 032] Add Wikimedia CH domain to wgCopyUploadsDomains [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116263 (owner: 10Odder) [13:58:43] (03Merged) 10jenkins-bot: Add Wikimedia CH domain to wgCopyUploadsDomains [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116263 (owner: 10Odder) [13:59:25] (03PS2) 10Reedy: Allow more upload file types for sewikimedia sysops [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116057 (owner: 10TTO) [13:59:29] (03CR) 10Reedy: [C: 032] Allow more upload file types for sewikimedia sysops [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116057 (owner: 10TTO) [13:59:37] (03Merged) 10jenkins-bot: Allow more upload file types for sewikimedia sysops [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116057 (owner: 10TTO) [13:59:54] (03PS2) 10Reedy: localize wmgBabelCategoryNames and wmgBabelMainCategory for oswiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/115181 (owner: 10Ricordisamoa) [13:59:55] Reedy: if you're interested and you give me time I can rebase the commits depending on https://gerrit.wikimedia.org/r/115879 too [13:59:58] (03CR) 10Reedy: [C: 032] localize wmgBabelCategoryNames and wmgBabelMainCategory for oswiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/115181 (owner: 10Ricordisamoa) [14:00:06] (03Merged) 10jenkins-bot: localize wmgBabelCategoryNames and wmgBabelMainCategory for oswiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/115181 (owner: 10Ricordisamoa) [14:00:34] Are they complex rebases? [14:00:57] (03PS2) 10Reedy: Crats should not add users to import on frwiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/115152 (owner: 10Odder) [14:01:03] (03CR) 10Reedy: [C: 032] Crats should not add users to import on frwiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/115152 (owner: 10Odder) [14:01:10] (03Merged) 10jenkins-bot: Crats should not add users to import on frwiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/115152 (owner: 10Odder) [14:01:48] (03CR) 10Reedy: [C: 04-1] "Conflicts, needs rebase :(" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116059 (owner: 10TTO) [14:02:51] !log reedy synchronized database lists files: [14:02:59] Logged the message, Master [14:03:16] !log reedy synchronized wmf-config/ [14:03:26] Logged the message, Master [14:04:40] (03PS1) 10Cmjohnson: Adding dhcpd entries for db1061-63 [operations/puppet] - 10https://gerrit.wikimedia.org/r/118070 [14:06:30] (03CR) 10Cmjohnson: [C: 032] Adding dhcpd entries for db1061-63 [operations/puppet] - 10https://gerrit.wikimedia.org/r/118070 (owner: 10Cmjohnson) [14:13:13] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [14:15:24] (03PS1) 10Hashar: Tweak l10nupdate user/group creations for beta cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/118071 [14:19:22] (03CR) 10Ottomata: [C: 031] Tweak l10nupdate user/group creations for beta cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/118071 (owner: 10Hashar) [14:21:15] (03PS1) 10Hashar: mediawiki::sync one file{} statement per file [operations/puppet] - 10https://gerrit.wikimedia.org/r/118076 [14:26:47] (03CR) 10Matanya: Tweak l10nupdate user/group creations for beta cluster (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/118071 (owner: 10Hashar) [14:27:53] (03CR) 10Matanya: mediawiki::sync one file{} statement per file (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/118076 (owner: 10Hashar) [14:28:56] matanya: not sure what you mean by "break resources into multiple lines" at https://gerrit.wikimedia.org/r/#/c/118071/1/modules/mediawiki/manifests/users/l10nupdate.pp,unified [14:29:43] name => blah [14:29:50] home => blah [14:30:01] etc, and not a one liner [14:30:08] ah [14:30:16] yeah should do that [14:34:04] (03CR) 10Hashar: "There is apparently no need to quote special keywords." (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/118076 (owner: 10Hashar) [14:35:53] (03CR) 10Matanya: mediawiki::sync one file{} statement per file (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/118076 (owner: 10Hashar) [14:36:14] (03CR) 10Hashar: Tweak l10nupdate user/group creations for beta cluster (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/118071 (owner: 10Hashar) [14:36:19] (03PS2) 10Hashar: Tweak l10nupdate user/group creations for beta cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/118071 [14:36:50] hashar: grep -r "ensure" [14:37:03] matanya: yeah you told me about it a few days ago on a different statement [14:37:09] i have? [14:37:17] can't remember which change [14:37:18] * matanya has a memory leak [14:37:23] but I did comply and quoted the value [14:37:46] this time I decided to verify whether we should really quote them and its not needed [14:38:00] in vim that shows the strings and keyword in different colors which is handy [14:38:35] i agree [14:38:55] but then we need to not quote all quoted ensure's [14:45:12] ideally :] [14:46:02] and i enforced the other way :/ [14:47:27] (03CR) 10Addshore: [C: 031] bugzilla: remove the new from motd, old is dead [operations/puppet] - 10https://gerrit.wikimedia.org/r/117669 (owner: 10Matanya) [14:57:37] (03CR) 10Ottomata: [C: 031] "This is looking good, thanks! If you confirm that you've tested this and it works as it should, I think we can merge." [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 (owner: 10Physikerwelt) [14:59:17] (03CR) 10Ottomata: [C: 031] "Wow cool, so simple!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117670 (owner: 10Matanya) [15:00:00] ottomata: it won't work though :) [15:00:25] not until we add the ferm class to the hosts [15:00:32] k [16:06:03] PROBLEM - Host ps1-b5-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [16:06:13] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [16:08:45] !log attempting to fix ps1-b5 and ps1-b6 [16:08:58] Logged the message, Master [16:17:59] (03CR) 10Dzahn: [C: 04-1] "this is for Chase to amend to and try out gerrit while creating his own account" [operations/puppet] - 10https://gerrit.wikimedia.org/r/118016 (owner: 10Dzahn) [16:19:57] (03CR) 10Dzahn: [C: 032] bugzilla: remove the new from motd, old is dead [operations/puppet] - 10https://gerrit.wikimedia.org/r/117669 (owner: 10Matanya) [16:23:26] (03CR) 10Dzahn: [C: 032] remove darrell shell account [operations/puppet] - 10https://gerrit.wikimedia.org/r/113639 (owner: 10Matanya) [16:27:32] (03CR) 10Dzahn: "can you just add Vibha Bamba to this maybe? is there still a controversy?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117659 (owner: 10Nemo bis) [16:29:12] (03CR) 10Nemo bis: "There isn't any controversy, just MZ being overzealous. :) I follow Talleyrand's advice. I'd add Vibha if she was on gerrit; she isn't, he" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117659 (owner: 10Nemo bis) [16:34:55] (03CR) 10Dzahn: "i pasted this URL in the -design channel" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117659 (owner: 10Nemo bis) [16:40:33] RECOVERY - Host ps1-b5-eqiad is UP: PING OK - Packet loss = 0%, RTA = 2.62 ms [16:45:46] (03PS9) 10Ottomata: Initial 2.0.0-1 debian release [operations/debs/archiva] (debian) - 10https://gerrit.wikimedia.org/r/115323 [16:46:17] (03PS6) 10BBlack: Overwrite stale ZeroOpts= (blank) cookies if HTTPS zero-rated. [operations/puppet] - 10https://gerrit.wikimedia.org/r/117661 (owner: 10Dr0ptp4kt) [16:46:44] (03CR) 10Faidon Liambotis: "OK, let's talk some data. On our (sampled randomly 1:1000) logs from March 1st-11th, we have 9988 lines for wikidata.org URLs that do not " [operations/apache-config] - 10https://gerrit.wikimedia.org/r/113972 (owner: 10Thiemo Mättig (WMDE)) [16:50:40] (03PS10) 10Ottomata: Initial 2.0.0-1 debian release [operations/debs/archiva] (debian) - 10https://gerrit.wikimedia.org/r/115323 [16:52:27] (03CR) 10Ottomata: "Ok, I made the init script a little bit smarter. Note that this is the template that dh_make provides, and I've only changed a little bi" [operations/debs/archiva] (debian) - 10https://gerrit.wikimedia.org/r/115323 (owner: 10Ottomata) [16:52:54] (03CR) 10Ottomata: Initial 2.0.0-1 debian release (031 comment) [operations/debs/archiva] (debian) - 10https://gerrit.wikimedia.org/r/115323 (owner: 10Ottomata) [16:57:35] robh: i need to take down mw1201-1203 from A6 and 1208-10 from B6 [16:57:40] going to put then in row D [16:58:21] (we moved this conversation in here, heh ;) [16:58:36] So yea.... i dunno if just changing the network config and moving is ideal [16:58:37] or reinstalling [16:58:51] could try the ip thing first and if it works, awesome [16:59:10] oh, i forgot to ask [16:59:13] why are they moving? [16:59:58] our power utilization on those 2 racks is too high [17:00:12] keeps alarming [17:00:31] i see temp alartms [17:00:33] what other alarms? [17:00:44] or is this non email alert? [17:00:55] https://icinga.wikimedia.org/icinga/ [17:01:14] non-email [17:01:33] its not over [17:01:35] its unbalanced [17:01:39] no way to balance and keep in rack? [17:01:45] rack is full [17:01:49] i have to move servers [17:01:58] what i mean is [17:02:04] can we balance the phases without removing? [17:02:10] cuz we are below the 8.6kW ceiling [17:02:17] without removing servers that is [17:02:36] ideally we load each rack to the fuill 8.6 and balace, but i realize it may not always be possible since we have limited power outlets per pdu. [17:03:28] i don't really see what I can move..all 3 phases on b6 jump over the threshold [17:03:49] same with A6. I think the most logical thing to do is move a few [17:03:51] i dont see any over threshhold on icinga [17:03:55] i see only they are out of balance [17:03:56] not over. [17:04:09] logging directly into the pdu [17:04:21] yea, tower b is over, but not tower a [17:04:29] so the towers are not balanced [17:04:43] (maybe all the network gear is on one only, or some server pdu isnt in both?) [17:04:56] its just odd that tower a y and z phase are 10.3 and 10.2 [17:05:07] but the tower b y and z are 12.6 and 12.8 [17:05:26] so yea, its over loaded on those two phases, but not on the identical redundant tower, so its unbalacned imo [17:05:31] but dunno if it can be fixed. [17:05:35] it is odd...but most of them our that way. I don't think the psu are drawing equal power [17:05:49] ^ the servers [17:05:49] (03PS4) 10Rush: add account for Chase and add to admins::root add key RT #7004 [operations/puppet] - 10https://gerrit.wikimedia.org/r/118016 (owner: 10Dzahn) [17:06:18] that bugs me, somethign is off [17:06:45] moving the servers is a bandaid, which may be the final thign we have to do [17:07:25] we can wait...I can email my contact at servertech and see if they have any suggestions [17:07:30] another reason i want all per outlet control [17:07:37] so can see draw per outlet. [17:07:49] that would be nice [17:08:08] let's pause this and I will do some more digging [17:08:25] but yea, if we end up having to move in row then sounds fine what you wanted to do [17:08:43] !ls [17:08:46] oops :) [17:08:59] sorry, out of row [17:09:02] heh [17:09:59] robh: one reason could be we are using y cables in A6 [17:10:22] ohh, maybe yea [17:10:49] meh, if you have to mvoe em, feel free [17:10:58] just do them two at a time or so and should be fine. [17:11:06] (and admin log it) [17:11:16] and dont do it during a deployment window [17:11:36] its annoying for them to have apaches going on and offline during those [17:11:58] but i forgot the y cables and they arent being used normally [17:12:04] and even normally they suck and arent a great idea. [17:12:10] yeah...probably going to do it first thing tomorrow around 8:30..should be least impact [17:12:48] https://wikitech.wikimedia.org/wiki/Deployments [17:13:17] you should be fine [17:13:38] yep..no conflicts [17:13:51] the dns stuff may be wonky, hence my suggestion of two at a time, heh [17:14:13] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [17:14:33] (03CR) 10BBlack: [C: 032 V: 032] Overwrite stale ZeroOpts= (blank) cookies if HTTPS zero-rated. [operations/puppet] - 10https://gerrit.wikimedia.org/r/117661 (owner: 10Dr0ptp4kt) [17:14:54] meh, forget my suggestion [17:15:00] i keep thinking you are moving more than 3 [17:15:01] 3 i' [17:15:05] i'd do all at once. [17:15:11] all three are in the api pool [17:15:38] so if you look at the api cluster, its weights are a bit off [17:15:48] and the newer machines are pulling a bit more than their fair share imo [17:15:58] BBlack, thx. ^ yurik, see merged patch above several messages. that paves the way for the ZRMA JS/CSS patch [17:16:00] but there are idle api machines, so taking all three down seems totally legit to me. [17:16:16] http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&s=by+name&c=API%2520application%2520servers%2520eqiad&tab=m&vn=&hide-hf=false [17:16:34] cmjohnson1: when you moved and reconfigured the analytics node system ips [17:16:43] they were already claling into puppet and continued without issue? [17:16:54] (i think its all hostname, not ip based, so sounds right to me,just checkin) [17:17:07] (03PS1) 10Odder: Add Meta to $wgImportSources for Serbian Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/118099 [17:17:19] that you would have to ask ottomata [17:19:53] (03PS5) 10Dzahn: add account for Chase and add to admins::root add key RT #7004 [operations/puppet] - 10https://gerrit.wikimedia.org/r/118016 [17:22:24] (03PS1) 10Hashar: beta: disable memcached accross datacenters [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/118100 [17:23:00] (03CR) 10Hashar: [C: 032] beta: disable memcached accross datacenters [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/118100 (owner: 10Hashar) [17:23:07] (03Merged) 10jenkins-bot: beta: disable memcached accross datacenters [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/118100 (owner: 10Hashar) [17:24:43] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [17:25:23] https://graphite.wikimedia.org/render/?title=HTTP%205xx%20Responses%20-8hours&from=-8hours&width=1024&height=500&until=now&areaMode=none&hideLegend=false&lineWidth=2&lineMode=connected&target=color%28cactiStyle%28alias%28reqstats.5xx,%225xx%20resp/min%22%29%29,%22blue%22%29 [17:25:28] ^ ? [17:27:30] (03CR) 10Ragesoss: [C: 031] Add Meta to $wgImportSources for Serbian Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/118099 (owner: 10Odder) [17:31:21] !log shut down srv284-srv301 (sdtpa row B, B5) [17:31:29] Logged the message, Master [17:32:14] test [17:32:22] chasemp: welcome [17:32:28] hi chasemp [17:32:30] bblack: I can't find any spikes in fatals or exceptions that match that 5xx graph. [17:32:43] thanks [17:32:47] mutante: thanks for the merges [17:32:52] hola chasemp [17:34:07] for posterity: rush <- chasemp, rush -> chasemp [17:34:20] chasemp: ello there [17:34:34] * greg-g is Greg from the standing desk ;) [17:35:09] \o/ for new opsen! [17:35:52] so now there are chasemp and jgage, yay for ops team revival :) [17:39:07] were are the new opsen?????????? [17:39:16] * hashar celebrates [17:41:21] 13:36 < chasemp!~chasemp@2620:62:c000:149:7479:bf7d:2ba3:ffb4 [Remote host closed the connection] [17:41:24] :) [17:42:55] jgage: did you have a look at rt #6999 ? [17:44:04] .... [17:47:23] * hashar waves [17:47:57] matanya, looking at it now [17:50:06] i have not written any ferm rules yet [17:50:10] another thing to learn :) [17:50:48] jgage: i told ottomata i'll handle this [17:51:04]