[00:03:46] (03PS1) 10Reedy: Update wikivoyage logo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85145 [00:05:14] (03CR) 10Reedy: [C: 032] Update wikivoyage logo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85145 (owner: 10Reedy) [00:05:37] (03Merged) 10jenkins-bot: Update wikivoyage logo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85145 (owner: 10Reedy) [00:06:27] !log reedy synchronized images/sul/wikivoyage.png [00:06:30] Logged the message, Master [00:09:46] (03CR) 10Akosiaris: "The change is more of an argument to try and figure out what kind of service we want to provide to users. Allowing both is the bad choice " [operations/puppet] - 10https://gerrit.wikimedia.org/r/84873 (owner: 10Akosiaris) [00:17:33] Logged the message, Master [00:18:02] heya akosiaris, i want to install the consumer.properties and producer.properties files in the kafka deb under different names [00:18:03] like [00:18:08] consumer.properties.example [00:18:12] or something [00:18:41] currently i'm installing them with debian/kafka.install [00:18:48] but I can't rename files with that [00:18:54] I could use your makefile [00:19:01] and use cp instead of $(INSTALL) [00:24:57] (03PS1) 10Ryan Lane: Add missing config for test/testrepo [operations/puppet] - 10https://gerrit.wikimedia.org/r/85147 [00:26:04] (03CR) 10Ryan Lane: [C: 032] Add missing config for test/testrepo [operations/puppet] - 10https://gerrit.wikimedia.org/r/85147 (owner: 10Ryan Lane) [00:27:24] (03PS1) 10Ottomata: Refactoring kafka::server and adding kafka::mirror support. [operations/puppet/kafka] - 10https://gerrit.wikimedia.org/r/85148 [00:29:50] (03PS2) 10Ottomata: Refactoring kafka::server and adding kafka::mirror support. [operations/puppet/kafka] - 10https://gerrit.wikimedia.org/r/85148 [00:38:44] PROBLEM - Puppet freshness on sq45 is CRITICAL: No successful Puppet run in the last 10 hours [00:44:26] (03CR) 10CSteipp: "https, with no user authentication, provides:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84873 (owner: 10Akosiaris) [00:46:33] !log rebooting sq45 [00:46:37] Logged the message, Master [00:50:14] PROBLEM - Host sq45 is DOWN: PING CRITICAL - Packet loss = 100% [01:39:09] (03PS1) 10Reedy: Expand MWMultiversion test cases [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85162 [01:43:57] (03PS2) 10Reedy: Expand MWMultiversion test cases [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85162 [01:45:31] (03CR) 10Reedy: [C: 032] Expand MWMultiversion test cases [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85162 (owner: 10Reedy) [01:45:48] (03Merged) 10jenkins-bot: Expand MWMultiversion test cases [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85162 (owner: 10Reedy) [01:57:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [01:58:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [02:12:46] !log LocalisationUpdate completed (1.22wmf17) at Fri Sep 20 02:12:46 UTC 2013 [02:12:51] Logged the message, Master [02:24:44] !log LocalisationUpdate completed (1.22wmf18) at Fri Sep 20 02:24:44 UTC 2013 [02:24:48] Logged the message, Master [02:44:52] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 20 02:44:51 UTC 2013 [02:44:55] Logged the message, Master [02:54:47] (03PS1) 10Reedy: WIP don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 [02:55:39] (03PS1) 10Reedy: Add tearDown to make sure singleton is destroyed [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85166 [02:56:44] TimStarling: was MW_SECURE_HOST for singer rather than the current setup? [02:56:52] (used in multiversion) [02:57:01] MWVersion.php: # MW_SECURE_HOST set from secure gateway? [02:57:01] MWVersion.php: $secure = getenv( 'MW_SECURE_HOST' ); [02:57:01] MWVersion.php: $host = $secure ? $secure : $_SERVER['HTTP_HOST']; [03:01:16] (03PS2) 10Reedy: Add tearDown to make sure singleton is destroyed [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85166 [03:01:34] (03CR) 10Reedy: [C: 032] Add tearDown to make sure singleton is destroyed [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85166 (owner: 10Reedy) [03:01:56] (03Merged) 10jenkins-bot: Add tearDown to make sure singleton is destroyed [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85166 (owner: 10Reedy) [03:03:03] Hell, the $host at the end isn't used anywhere else [03:03:38] (03PS1) 10Reedy: Remove unused MW_SECURE_HOST check [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85167 [03:16:41] (03PS1) 10Reedy: Add testcase for pa_uswikimedia cause it breaks stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85169 [03:16:59] (03CR) 10Reedy: [C: 032] Add testcase for pa_uswikimedia cause it breaks stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85169 (owner: 10Reedy) [03:17:12] (03Merged) 10jenkins-bot: Add testcase for pa_uswikimedia cause it breaks stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85169 (owner: 10Reedy) [03:19:04] (03PS2) 10Reedy: WIP don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 [03:19:18] (03CR) 10jenkins-bot: [V: 04-1] WIP don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 (owner: 10Reedy) [03:47:34] (03PS3) 10Reedy: WIP don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 [03:54:21] (03PS4) 10Reedy: WIP don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 [03:56:52] Do we have any other weird exceptions? [03:58:31] 8 "projects", plus wikimedia, mediawiki, wikisource.org [04:06:17] oh, wikidata [04:09:50] global job queue length is 2.9M [04:10:01] (03PS1) 10Reedy: Fix wikidata.org domain [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85171 [04:10:18] refreshlinks? [04:10:43] (03CR) 10Reedy: [C: 032] Fix wikidata.org domain [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85171 (owner: 10Reedy) [04:10:52] (03Merged) 10jenkins-bot: Fix wikidata.org domain [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85171 (owner: 10Reedy) [04:11:06] mostly parsoid [04:11:14] which wikis? [04:11:16] ParsoidCacheUpdateJob: 2413680 queued; 611 claimed (12 active, 599 abandoned) [04:11:21] on enwiki alone [04:16:59] good to see wikitech is up to date as usual [04:21:59] Reedy: Fancy a pint later? YuviPanda, MaxSem and I are painting the town red. [04:23:54] (03PS5) 10Reedy: WIP don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 [04:24:04] (03CR) 10jenkins-bot: [V: 04-1] WIP don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 (owner: 10Reedy) [04:26:48] (03PS6) 10Reedy: WIP don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 [04:26:49] (03PS1) 10Reedy: Update tests to remove docroot setting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85172 [04:39:57] the job runners do look to be slacking a bit [06:20:32] Reedy: we still have less high prio jobs than we did in March [06:21:09] it would be useful to, say, double those so that at least emails arrive on time :) [06:21:27] dunno what other stuff is in there but at least emails must not be too DB intensive I hope? [06:26:41] (03PS1) 10Reedy: Remove 404.html symlinks, 404s are handled by w/404.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85175 [06:29:41] The config is open... [06:30:10] "Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request." [06:30:21] 404 whilst trying to handle the 404 [06:31:19] !log reedy synchronized docroot/bits/ [06:31:23] Logged the message, Master [06:31:41] Hmm, that's slightly better [06:34:19] I'm guessing php isn't enabled for bits usage [06:39:10] (03PS1) 10Reedy: Enable PHP on w/404.php on bits [operations/apache-config] - 10https://gerrit.wikimedia.org/r/85177 [06:39:14] (03CR) 10jenkins-bot: [V: 04-1] Enable PHP on w/404.php on bits [operations/apache-config] - 10https://gerrit.wikimedia.org/r/85177 (owner: 10Reedy) [06:40:21] (03PS2) 10Reedy: Enable PHP on w/404.php on bits [operations/apache-config] - 10https://gerrit.wikimedia.org/r/85177 [06:40:23] (03PS1) 10Reedy: Add w symlink for 404.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85178 [06:40:45] (03CR) 10Reedy: [C: 032] Add w symlink for 404.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85178 (owner: 10Reedy) [06:40:54] (03Merged) 10jenkins-bot: Add w symlink for 404.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85178 (owner: 10Reedy) [09:50:03] PROBLEM - Puppet freshness on hume is CRITICAL: No successful Puppet run in the last 10 hours [09:55:03] PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 10 hours [11:32:18] (03PS1) 10Petr Onderka: Better error reporting for GCC [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/85189 [12:03:35] (03PS1) 10Nemo bis: Fix misc::maintenance::updatequerypages duplicate cronjobs [operations/puppet] - 10https://gerrit.wikimedia.org/r/85192 [12:04:49] (03CR) 10jenkins-bot: [V: 04-1] Fix misc::maintenance::updatequerypages duplicate cronjobs [operations/puppet] - 10https://gerrit.wikimedia.org/r/85192 (owner: 10Nemo bis) [12:05:09] (03PS2) 10Petr Onderka: Better error reporting for GCC [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/85189 [12:05:42] (03CR) 10Petr Onderka: [C: 032 V: 032] Better error reporting for GCC [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/85189 (owner: 10Petr Onderka) [12:07:03] hmm [12:07:11] (03PS2) 10Nemo bis: Fix misc::maintenance::updatequerypages duplicate cronjobs [operations/puppet] - 10https://gerrit.wikimedia.org/r/85192 [12:07:17] (03CR) 10jenkins-bot: [V: 04-1] Fix misc::maintenance::updatequerypages duplicate cronjobs [operations/puppet] - 10https://gerrit.wikimedia.org/r/85192 (owner: 10Nemo bis) [12:07:17] bbl [12:07:24] ah that was lquick ^^ [12:16:35] (03PS1) 10Petr Onderka: Fixed exception when reading near end of stream [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/85196 [12:21:38] (03CR) 10Petr Onderka: [C: 032 V: 032] Fixed exception when reading near end of stream [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/85196 (owner: 10Petr Onderka) [13:05:06] (03PS3) 10Nemo bis: Fix misc::maintenance::updatequerypages duplicate cronjobs [operations/puppet] - 10https://gerrit.wikimedia.org/r/85192 [13:11:08] (03CR) 10ArielGlenn: [C: 031] Fix misc::maintenance::updatequerypages duplicate cronjobs [operations/puppet] - 10https://gerrit.wikimedia.org/r/85192 (owner: 10Nemo bis) [13:39:55] !log Jenkins: PHPUnit jobs for MediaWiki extensions had an issue for the last couple days. They were being fetched in an incorrect place :/ Fix is https://gerrit.wikimedia.org/r/85202 and got deployed [13:39:58] Logged the message, Master [13:45:14] hey mark; any news regarding the varnish and incorrect tagging of zero traffic? [14:12:15] (03PS7) 10Hashar: Jenkins validation (please ignore) [operations/debs/pybal] - 10https://gerrit.wikimedia.org/r/84932 [14:17:35] (03PS8) 10Hashar: Jenkins validation (please ignore) [operations/debs/pybal] - 10https://gerrit.wikimedia.org/r/84932 [14:24:37] (03PS1) 10Jgreen: begin puppetizing clamav for otrs [operations/puppet] - 10https://gerrit.wikimedia.org/r/85209 [14:25:56] (03CR) 10Jgreen: [C: 032 V: 031] begin puppetizing clamav for otrs [operations/puppet] - 10https://gerrit.wikimedia.org/r/85209 (owner: 10Jgreen) [14:34:18] (03PS1) 10Jgreen: puppetize clamd.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/85211 [14:34:35] (03CR) 10jenkins-bot: [V: 04-1] puppetize clamd.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/85211 (owner: 10Jgreen) [14:42:02] (03PS2) 10Jgreen: puppetize clamd.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/85211 [14:43:26] (03CR) 10Jgreen: [C: 032 V: 031] puppetize clamd.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/85211 (owner: 10Jgreen) [14:49:36] (03PS1) 10Jgreen: oops, checked in clamd.conf in the wrong dir [operations/puppet] - 10https://gerrit.wikimedia.org/r/85214 [14:51:55] (03CR) 10Jgreen: [C: 032 V: 031] oops, checked in clamd.conf in the wrong dir [operations/puppet] - 10https://gerrit.wikimedia.org/r/85214 (owner: 10Jgreen) [15:12:04] (03PS1) 10Jgreen: rename clamd service to clamav-daemon [operations/puppet] - 10https://gerrit.wikimedia.org/r/85215 [15:14:31] (03CR) 10Jgreen: [C: 032 V: 031] rename clamd service to clamav-daemon [operations/puppet] - 10https://gerrit.wikimedia.org/r/85215 (owner: 10Jgreen) [15:17:46] (03CR) 10Ottomata: "Sounds like git-deploy is the way to go here. Nik, sorry for having advised you wrong." [operations/puppet] - 10https://gerrit.wikimedia.org/r/82673 (owner: 10Manybubbles) [15:23:33] heya dr0ptp4kt, did you still want to check up on real varnish logs with more headers? or did you guys solve your thing all the way? [15:33:10] (03PS1) 10Ottomata: Not including consumer.properties and producer.properties in /etc/kafka. [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/85219 [15:34:40] manybubbles: if you are ever pointed to some git-deploy doc, I am willing to lear as well :-) [15:34:58] manybubbles: the workaround I found is to add my files in a git repository and do the git pull manually on the server to deploy the changes :( [15:36:15] (03CR) 10Ottomata: "(1 comment)" [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/85219 (owner: 10Ottomata) [15:58:30] !log gerrit: deleting reference refs/for/master from pywikibot/core.git , confuses some git clients [15:58:34] Logged the message, Master [16:36:11] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [16:37:51] lies, damned lies [16:38:20] ... [16:39:19] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:43:00] Nemo_bis: this graph says 0 jobs? https://ganglia.wikimedia.org/latest/graph.php?r=year&z=xlarge&c=Miscellaneous+pmtpa&h=hume.wikimedia.org&jr=&js=&v=2899505&m=Global_JobQueue_length [16:44:58] I see nothing obvious here https://wikitech.wikimedia.org/wiki/Server_admin_log [16:48:59] greg-g: hm? to me does say 0, but preceded by a 3 and 5 more zeros https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Miscellaneous%20pmtpa&h=hume.wikimedia.org&v=823574&m=Global_JobQueue_length&r=hour&z=default&jr=&js=&st=1365625056&z=large [16:51:13] sigh, +1M last week [16:52:08] I wonder what those 0.1M bursts followed by oscillations are [16:53:01] greg-g: can we know the job queue length per job type? [16:53:50] and that is just tampa? [16:53:55] aude: Tim said it was 2.0 out of 2.5M parsoid jobs on en.wiki [16:54:04] there is no such a thing as tampa job queue [16:55:01] hmmm, makes sense [16:55:06] it's just run on that server [16:55:47] yeah, it was just the server with a MW install able to run maintenance reports like that one [16:55:52] ok [17:12:03] (03PS1) 10Aude: add commons site link group section for test wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85230 [17:12:53] (03CR) 10Aude: [C: 04-1] "depends on repopulating sites table on testwikidata, updating WikimediaMessages and Wikibase branches on wmf18 and running localisation up" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85230 (owner: 10Aude) [17:16:55] (03CR) 10Aude: "see https://gerrit.wikimedia.org/r/#/c/85230/ for submodule updates" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85230 (owner: 10Aude) [17:19:17] Nemo_bis, apergos, what was the puppet error that https://gerrit.wikimedia.org/r/85192 fixes? [17:20:03] springle: puppet is just refusing to run on hume [17:20:25] because a cron with same name is declared multiple times [17:20:37] this was discussed on #wikimedia-tech [17:21:30] Ariel is afk [17:21:58] springle: [17:22:25] the cron jobs added by the previous change don't vary by name [17:22:41] so puppet tries to declare the same resource twice and bails [17:23:06] Error 400 on SERVER: Duplicate definition: Cron[cron-updatequerypages-lonelypages-s1] is already defined in file /etc/puppet/manifests/misc/maintenance.pp at line 487; cannot redefine at /etc/puppet/manifests/misc/maintenance.pp:487 on node hume.wikimedia.org [17:23:36] this is because the previous patch didn't actually do what was intended (run each corn job on its specific date only on s1) [17:23:46] ah ok [17:24:20] the new code looks like it should in fact run the new cron jobs only on s1 on the given dates (but please look it over, I wont' be here to babysit when the jobs actually run so I didn't want to +2 it and stick you with it) [17:25:07] apergos: thank you [17:25:49] yw [17:26:46] I've been on a 'keep puppet running on all hosts' kick, hoping to make it a habit [17:32:17] (03Abandoned) 10Akosiaris: Reduce retry timeout for etherpad frontend apache [operations/puppet] - 10https://gerrit.wikimedia.org/r/84882 (owner: 10Akosiaris) [17:36:08] <^d> Ryan_Lane: Do you remember what the purpose of installing git-svn on the gerrit box was? [17:36:20] (03PS1) 10Akosiaris: Retry timeout to 15sec for etherpad frontend [operations/puppet] - 10https://gerrit.wikimedia.org/r/85236 [17:37:28] !log Created Echo tables on officewiki [17:37:32] Logged the message, Master [17:38:00] (03PS4) 10Springle: Fix misc::maintenance::updatequerypages duplicate cronjobs [operations/puppet] - 10https://gerrit.wikimedia.org/r/85192 (owner: 10Nemo bis) [17:38:45] (03CR) 10Springle: [C: 032] Fix misc::maintenance::updatequerypages duplicate cronjobs [operations/puppet] - 10https://gerrit.wikimedia.org/r/85192 (owner: 10Nemo bis) [17:39:08] (03PS1) 10Reedy: Add echo to officewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85237 [17:39:26] (03CR) 10Akosiaris: [C: 032] Retry timeout to 15sec for etherpad frontend [operations/puppet] - 10https://gerrit.wikimedia.org/r/85236 (owner: 10Akosiaris) [17:39:37] !log reedy synchronized database lists files: [17:39:41] Logged the message, Master [17:40:08] (03CR) 10Reedy: [C: 032] Add echo to officewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85237 (owner: 10Reedy) [17:40:21] (03Merged) 10jenkins-bot: Add echo to officewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85237 (owner: 10Reedy) [17:41:40] !log reedy synchronized wmf-config/ 'touch' [17:41:44] Logged the message, Master [17:41:58] Can someone merge and deploy https://gerrit.wikimedia.org/r/#/c/85177/ to fix the 404 handler on bits so it doesn't 404? [17:44:15] PROBLEM - MySQL Processlist on db1051 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 64 statistics [17:44:33] Well, I lie slightly [17:44:45] I stopped it 404-ing by adding the w symlink so it can find 404.php [17:44:50] It just won't execute PHP [17:47:15] RECOVERY - MySQL Processlist on db1051 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 5 statistics [17:47:44] (03PS1) 10Chad: Cirrus on all closed wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85239 [17:47:45] RECOVERY - Puppet freshness on hume is OK: puppet ran at Fri Sep 20 17:47:38 UTC 2013 [17:49:44] ^d: umm, yeah [17:49:49] ^d: you asked for it ;) [17:49:56] to do the svn -> git migration [17:50:06] <^d> Why on earth did I need that. [17:50:08] <^d> I was stupid. [17:50:12] hahaha [17:51:33] (03PS1) 10Chad: Remove stupid package I don't need [operations/puppet] - 10https://gerrit.wikimedia.org/r/85240 [17:53:25] PROBLEM - MySQL Processlist on db1043 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 49 statistics [17:53:25] RECOVERY - Puppet freshness on terbium is OK: puppet ran at Fri Sep 20 17:53:23 UTC 2013 [17:56:25] PROBLEM - MySQL Processlist on db1043 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 33 statistics [17:59:01] (03PS1) 10Springle: adjust processlist warn/crit thresholds [operations/puppet] - 10https://gerrit.wikimedia.org/r/85243 [17:59:25] RECOVERY - MySQL Processlist on db1043 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 16 statistics [18:00:13] (03CR) 10Springle: [C: 032] adjust processlist warn/crit thresholds [operations/puppet] - 10https://gerrit.wikimedia.org/r/85243 (owner: 10Springle) [18:04:00] (03PS1) 10Akosiaris: Update debian/changelog [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/85246 [18:04:01] (03PS1) 10Akosiaris: Remove scala 2.8 annotations [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/85247 [18:04:30] (03CR) 10Akosiaris: [C: 032 V: 032] Update debian/changelog [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/85246 (owner: 10Akosiaris) [18:04:42] (03CR) 10Akosiaris: [C: 032 V: 032] Remove scala 2.8 annotations [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/85247 (owner: 10Akosiaris) [18:05:35] ottomata: drdee: ^ This fixes kafka not building with the latest patches. I also updated version number so if you want new kafka packages on apt tell me [18:06:25] PROBLEM - MySQL Processlist on db1043 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 43 statistics [18:06:29] awesoome, danke [18:06:40] before you build a new one [18:06:44] let's check / merge this [18:06:45] https://gerrit.wikimedia.org/r/#/c/85219/ [18:06:55] check my comment, i'm not really sure of the best way to do what I want [18:11:06] (03PS4) 10BBlack: netmapper data sync stuff [operations/puppet] - 10https://gerrit.wikimedia.org/r/85120 [18:12:25] PROBLEM - MySQL Processlist on db1043 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 37 statistics [18:12:36] ottomata: i see what you mean... you absolutely want the .example suffix in those 2 files? [18:14:21] (03CR) 10BBlack: [C: 032] netmapper data sync stuff [operations/puppet] - 10https://gerrit.wikimedia.org/r/85120 (owner: 10BBlack) [18:14:25] RECOVERY - MySQL Processlist on db1043 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 14 statistics [18:16:09] well, no akosiaris, not necessarily [18:16:33] the main problem is kinda that there are multipel consumer .property files, and their names will probably be arbitrary [18:16:35] for example [18:16:44] consumer.eqiad.properties, consumer.esams.properties [18:16:45] etc. [18:16:47] but [18:16:56] if the .deb installs one as consumer.properties [18:17:09] the init script will read it in [18:17:27] alternativelly, i guess i could ensure => absent on that file in puppet [18:17:58] i just don't want to one day provision a kafka mirror instance using this .deb and puppet [18:18:11] and have the extra manual step of removing a file so it can actually work [18:19:24] and, since the shipped consumer and producer files almost certainly will not work for mirror. they both point at the kafka cluster defaults, which means if mirror starts up and someone is actually running the defaults, like in a test env, mirror will create an infinite loop of logs [18:19:34] consume and produce back to the same brokers [18:21:15] (03PS1) 10Lcarr: adding subnet sandbox1-b-eqiad and dickson.freenode.net [operations/dns] - 10https://gerrit.wikimedia.org/r/85250 [18:23:27] PROBLEM - Host mw1085 is DOWN: PING CRITICAL - Packet loss = 100% [18:24:37] RECOVERY - Host mw1085 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [18:24:38] (03PS1) 10BBlack: apparently class-require does not do what I wish it did [operations/puppet] - 10https://gerrit.wikimedia.org/r/85251 [18:24:40] (03CR) 10jenkins-bot: [V: 04-1] apparently class-require does not do what I wish it did [operations/puppet] - 10https://gerrit.wikimedia.org/r/85251 (owner: 10BBlack) [18:27:27] PROBLEM - Apache HTTP on mw1085 is CRITICAL: Connection refused [18:27:31] (03Abandoned) 10BBlack: apparently class-require does not do what I wish it did [operations/puppet] - 10https://gerrit.wikimedia.org/r/85251 (owner: 10BBlack) [18:27:47] (03PS1) 10BBlack: apparently class-require does not do what I wish it did [operations/puppet] - 10https://gerrit.wikimedia.org/r/85253 [18:28:15] jenkins-- [18:29:00] (03PS6) 10Ryan Lane: Simplify git-deploy configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/83046 [18:29:59] (03CR) 10BBlack: [C: 032] apparently class-require does not do what I wish it did [operations/puppet] - 10https://gerrit.wikimedia.org/r/85253 (owner: 10BBlack) [18:30:02] greg-g: Do we do Friday LDs? What's the policy there? [18:30:38] RoanKattouw: no Friday deploys :) [18:30:44] That's what I thought :) [18:30:53] * MaxSem deploys greg-g [18:30:55] In that case, can I sign up for an LD on Monday? [18:30:59] MatmaRex: :P [18:31:01] anomie: I'm tempted to add rate-limiting for null edits [18:31:03] I have some VE page corruption fixes [18:31:08] RoanKattouw: yessir, what's the issue? [18:31:13] ah [18:31:23] (03CR) 10Akosiaris: "(1 comment)" [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/85219 (owner: 10Ottomata) [18:31:23] cool, yeah, plz do add to wikitech deploy page [18:31:27] RECOVERY - Apache HTTP on mw1085 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.092 second response time [18:31:34] https://en.wikipedia.org/w/index.php?title=Matt_Chandler_%28pastor%29&curid=37214846&diff=573675359&oldid=571858677 (search for page corruption? I hold greg-g, you deploy! [18:32:54] grrrit-wm: aren't tabbing mistakes fun? :D [18:33:04] AaronSchulz: People would still do it, just slower. [18:33:26] RoanKattouw: eek [18:33:34] RoanKattouw: how often is it happening? [18:34:00] <^d> manybubbles: Sooo, I thought about it for a long time. Hacked a couple of alternatives. All ended up being worse than my dislike of LSB. [18:34:02] Let me look at data for the past 24h [18:34:07] <^d> So I merged your LSB approach :) [18:34:10] My guess is not too often because the page has to be in a very specific state [18:34:19] ^d: I saw that. [18:34:21] * greg-g nods [18:34:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [18:34:38] ^d: I think I broke something with that phrase rescore stuff. I'm kind of mired in it. [18:34:42] but cool! [18:35:03] <^d> Mmk. Oh, and we've scheduled all "closed" wikis to migrate on wednesday. [18:35:25] <^d> Nobody uses them much so user impact will be almost non-existent, but will allow us to index 126 more wikis :) [18:35:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 14.184 second response time [18:35:54] lol [18:37:06] greg-g: Today (UTC, so the past 18 hours) there were 16 edits flagged for possible corruption issues on enwiki, 13-14 of them exhibit the reference corruption bug [18:37:33] So it's corrupting ~20 pages per day, which is annoying but not massive [18:37:48] urgh [18:37:52] Like, it would take me a couple hours to fix all of them over the weekend [18:37:52] I wish you would have said 5 [18:38:11] RoanKattouw: what's the gerrit change that fixes this? [18:38:44] ^d: yay! [18:38:51] greg-g: https://gerrit.wikimedia.org/r/#/c/85144/ [18:39:00] anomie: well slow can be handled [18:39:14] (03PS2) 10Reedy: wmgHTTPSBlacklistCountries changes for CN [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84781 [18:39:27] whoa, bigger than I though [18:39:28] t [18:39:31] Yeah [18:39:36] (03CR) 10Reedy: [C: 032] wmgHTTPSBlacklistCountries changes for CN [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84781 (owner: 10Reedy) [18:39:43] More than half is in the test directory but still [18:40:08] at least you added/edited tests! [18:40:15] Yeah [18:40:33] is this live on beta cluster? [18:40:36] I even added a failing test case in PS1 but for some reason Jenkins V+2ed even though the tests were failing for me locally :S [18:40:38] Probably [18:40:47] It was merged into master at 5am this morning so it should be live there [18:40:49] I haven't checked [18:40:51] * greg-g nods [18:40:52] (03CR) 10Reedy: [C: 031] Cirrus on all closed wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85239 (owner: 10Chad) [18:40:55] (03Merged) 10jenkins-bot: wmgHTTPSBlacklistCountries changes for CN [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84781 (owner: 10Reedy) [18:41:14] (03PS3) 10Reedy: Language Template fixup definition for UploadWizard. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/39026 (owner: 10Lupo) [18:41:25] I need to do a bit of testing with that change as well [18:41:30] * greg-g nods [18:41:40] k, well, I'll assume monday for now [18:41:43] The unit tests pretty much prove that we won't corrupt things but I'm paranoid [18:41:46] Yeah sounds reasonable [18:41:55] yeah, thanks RoanKattouw [18:41:56] If it was a one-liner and it was corrupting more pages .... but it's not and it's not [18:42:01] right [18:42:02] :) [18:43:09] (03Abandoned) 10Reedy: Delete all the superfluous wikimania docroots [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/62565 (owner: 10Reedy) [18:43:14] (03PS3) 10Reedy: Change name of cywikisource to "Wicidestun" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84221 (owner: 10TTO) [18:43:20] (03CR) 10Reedy: [C: 032] Change name of cywikisource to "Wicidestun" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84221 (owner: 10TTO) [18:43:21] anomie: huh, ApiPurge already calls pingLimiter() thought not PurgeAction [18:43:47] (03Merged) 10jenkins-bot: Change name of cywikisource to "Wicidestun" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84221 (owner: 10TTO) [18:44:20] (03PS2) 10Reedy: (bug 54287) Change logo for crwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85110 (owner: 10Yuvipanda) [18:44:26] (03CR) 10Reedy: [C: 032] (bug 54287) Change logo for crwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85110 (owner: 10Yuvipanda) [18:44:55] (03Merged) 10jenkins-bot: (bug 54287) Change logo for crwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85110 (owner: 10Yuvipanda) [18:45:20] well purge alone doesn't do links updates (other than randomly for cascade protection)...I guess that makes sense [18:45:25] (03PS2) 10Reedy: Remove unused MW_SECURE_HOST check [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85167 [18:45:30] (03CR) 10Reedy: [C: 032] Remove unused MW_SECURE_HOST check [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85167 (owner: 10Reedy) [18:45:50] (03Merged) 10jenkins-bot: Remove unused MW_SECURE_HOST check [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85167 (owner: 10Reedy) [18:46:03] merge spamming :) [18:46:51] (03PS2) 10Reedy: Remove 404.html symlinks, 404s are handled by w/404.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85175 [18:46:57] (03CR) 10Reedy: [C: 032] Remove 404.html symlinks, 404s are handled by w/404.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85175 (owner: 10Reedy) [18:47:08] seems $wgUser->pingLimiter() gets hit for null edits to, maybe it's the 'noratelimit' right [18:47:32] (03PS1) 10coren: Rename manganese -> dickson and prep for debian [operations/puppet] - 10https://gerrit.wikimedia.org/r/85256 [18:47:37] andrewbogott: is domain necessary for lookup of public ips? [18:47:44] it's an extra query [18:47:52] (03Merged) 10jenkins-bot: Remove 404.html symlinks, 404s are handled by w/404.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85175 (owner: 10Reedy) [18:48:05] It's not needed for the lookup but it's used elsewhere in the code… lemme see [18:48:59] LeslieCarr: https://gerrit.wikimedia.org/r/#/c/85256/1 [18:49:30] !log reedy synchronized wmf-config/ [18:49:34] Logged the message, Master [18:49:40] (03PS2) 10Lcarr: adding subnet sandbox1-b-eqiad and dickson.freenode.net fixed to include ipv6 [operations/dns] - 10https://gerrit.wikimedia.org/r/85250 [18:50:35] (03PS2) 10coren: Rename manganese -> dickson and prep for debian [operations/puppet] - 10https://gerrit.wikimedia.org/r/85256 [18:51:00] !log reedy synchronized docroot and w [18:51:04] Logged the message, Master [18:51:30] nope, no 'noratelimit' rights [18:52:16] !log reedy synchronized multiversion/ [18:52:20] Logged the message, Master [18:52:28] Ryan_Lane, deleteHost does $this->getDomain()->updateSOA(); (an example, I feel like there was another place that motivated me to add that...) [18:52:46] ah. right. [18:53:38] Yeah, and this->domain happens a few other places for public hosts. [18:53:40] AaronSchulz: Is the problem that no rate limits are set on 'edit', except for ips and "newbies"? [18:53:49] We could maybe do the lookup when we need it rather than ahead of time... [18:54:23] * Ryan_Lane nods [18:54:40] (03PS3) 10coren: Rename manganese -> dickson and prep for debian [operations/puppet] - 10https://gerrit.wikimedia.org/r/85256 [18:55:18] (03CR) 10Reedy: "Is this still needed?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/39026 (owner: 10Lupo) [18:55:24] anomie: probably [18:56:09] (03CR) 10Lcarr: [C: 04-1] "(2 comments)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/85256 (owner: 10coren) [18:56:33] * AaronSchulz checked the config [18:57:28] anomie: that should definitely be limited somehow [18:57:53] which probably means limiting edits and not just using 'purge' and configuring that...lest people do mass edits adding and removing a newline or something to get around it [18:57:56] akosiaris: fyi I just pushed a couple more recent 0.8 branch changes into our repo [18:58:08] arghhhh [18:58:09] why ? [18:58:12] (03PS4) 10coren: Rename manganese -> dickson and prep for debian [operations/puppet] - 10https://gerrit.wikimedia.org/r/85256 [18:58:28] haha, because i'm trying to stay up to date with 0.8 so it is easier for us when they finally mark it as stable [18:58:35] there were only 2 changes [18:58:37] one was irrelevant [18:58:41] the other was for consumers [18:58:47] Well if it does not compile again you are the one cleaning up :P [18:58:48] looks ok i think :) [18:58:50] haha ok [18:59:16] (03PS1) 10Kaldari: Slightly more concise language for wgRightsText [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85259 [18:59:18] AaronSchulz: OTOH, mass edits adding/removing a newline are likely to be noticed by the community and blocked. Null edits are invisible to the community. [18:59:25] so, i just tried running install like you said, it seems to work [18:59:32] for some reason I though install wouldn't let you rename files [18:59:38] but I guess that is just dh_install [18:59:40] not install install [18:59:45] anomie: hopefully, I guess it's worth a try [19:00:04] i think your suggested change is good, what else did you want to test [19:00:06] akosiaris: ^? [19:00:17] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [19:00:40] Avoiding naming these files directly and relying on the MIRROR_CONFFILES variable you created [19:00:49] It will make it a bit more robust [19:00:49] oh aye [19:00:59] i could just make a var for each file [19:01:02] there are only 2 files [19:01:26] I am doing a target that expands that one so probably better [19:01:32] i am testing now [19:01:38] ok [19:03:27] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:04:24] anomie: are you doing that or should I? [19:04:36] AaronSchulz: Doing what? [19:04:52] (so, no, I'm not doing whatever it is) [19:05:03] anomie: rate limiting link purges [19:05:14] * AaronSchulz would make some patches then [19:05:22] *will...can't type today [19:06:23] (03PS1) 10coren: Add dickson to DNS (new freenode server) [operations/dns] - 10https://gerrit.wikimedia.org/r/85261 [19:07:56] (03CR) 10Ryan Lane: [C: 032] Add dickson to DNS (new freenode server) [operations/dns] - 10https://gerrit.wikimedia.org/r/85261 (owner: 10coren) [19:09:43] (03PS1) 10Chad: Officewiki gets Cirrus [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85262 [19:13:05] (03CR) 10Chad: [C: 032] Officewiki gets Cirrus [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85262 (owner: 10Chad) [19:13:16] (03Merged) 10jenkins-bot: Officewiki gets Cirrus [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85262 (owner: 10Chad) [19:14:00] !log demon synchronized wmf-config/InitialiseSettings.php 'Cirrus on officewiki' [19:14:04] Logged the message, Master [19:15:53] (03PS1) 10Hashar: contint: puppet class to setup browsertests slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/85264 [19:18:37] (03CR) 10Lcarr: [C: 032] Rename manganese -> dickson and prep for debian [operations/puppet] - 10https://gerrit.wikimedia.org/r/85256 (owner: 10coren) [19:19:59] (03PS1) 10Chad: Turn officewiki Cirrus back off, stupid TMH [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85266 [19:20:24] (03CR) 10Chad: [C: 032] Turn officewiki Cirrus back off, stupid TMH [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85266 (owner: 10Chad) [19:20:28] !log demon synchronized wmf-config/InitialiseSettings.php [19:20:31] Logged the message, Master [19:23:48] <^demon> manybubbles: fyi, I filed bug 54394. It's not our fault, but keeps us from using Cirrus on private wikis (eg officewiki) just yet. [19:24:04] ^d: cool [19:30:38] (03PS2) 10Hashar: contint: puppet class to setup browsertests slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/85264 [19:32:17] (03CR) 10Hashar: "Did it manually on integration-selenium-driver.pmtpa.wmflabs (it is not a puppet self instance, so I have did this change manually :])." [operations/puppet] - 10https://gerrit.wikimedia.org/r/85264 (owner: 10Hashar) [20:08:18] heyaaa LeslieCarr, I'm messing with FoxyProxy + ssh SOCKS proxy for analytics hadoop nodes [20:08:20] and it mostly works [20:09:38] i think i need someone to verify what I'm seeing. [20:09:51] foxyproxy works for the datanode edge case [20:10:04] but only if I enable it for all urls. [20:10:13] the datanode pattern should be no different than the namenode (or others) [20:10:25] but it doesn't seem to work, and i'm not sure why [20:10:47] it does use the proxy, at least, I have output from ssh -v when I request a datanode [20:11:14] but it seems the proxy can't actually find the datanode to proxy to, unless foxyproxy is set to proxy ALL urls through it [20:11:20] which is totally whack! [20:13:43]