[00:00:39] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [00:00:59] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [00:01:10] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [00:01:39] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [00:02:10] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [01:01:49] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 04 Sep 2014 00:21:29 UTC [02:02:40] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Puppet has 1 failures [02:11:09] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3616 MB (3% inode=99%): [02:19:40] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [02:20:55] !log LocalisationUpdate completed (1.24wmf15) at 2014-09-08 02:19:52+00:00 [02:21:03] Logged the message, Master [02:22:09] (03PS1) 10Ori.livneh: wmflib: add documentation, standardize code conventions [puppet] - 10https://gerrit.wikimedia.org/r/159007 [02:33:40] !log LocalisationUpdate completed (1.24wmf19) at 2014-09-08 02:32:37+00:00 [02:33:46] Logged the message, Master [02:45:50] !log LocalisationUpdate completed (1.24wmf20) at 2014-09-08 02:44:47+00:00 [02:45:56] Logged the message, Master [03:01:09] RECOVERY - Disk space on virt0 is OK: DISK OK [03:02:49] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 04 Sep 2014 00:21:29 UTC [03:37:20] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Sep 8 03:36:13 UTC 2014 (duration 36m 12s) [03:37:26] Logged the message, Master [03:39:29] (03CR) 10Ori.livneh: [C: 032] "docs / whitespace only change" [puppet] - 10https://gerrit.wikimedia.org/r/159007 (owner: 10Ori.livneh) [03:59:20] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Puppet has 1 failures [04:18:19] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [04:56:30] (03PS1) 10Springle: move enwiki api traffic to db1051/db1066 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159014 [04:57:25] (03CR) 10Springle: [C: 032] move enwiki api traffic to db1051/db1066 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159014 (owner: 10Springle) [04:57:29] (03Merged) 10jenkins-bot: move enwiki api traffic to db1051/db1066 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159014 (owner: 10Springle) [04:58:51] !log springle Synchronized wmf-config/db-eqiad.php: move enwiki api traffic to db1051/db1066 (duration: 00m 09s) [04:58:56] Logged the message, Master [05:03:49] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 04 Sep 2014 00:21:29 UTC [06:28:39] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: Puppet has 3 failures [06:28:49] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:50] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:59] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:09] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 2 failures [06:45:10] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:45:39] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:46:00] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:46:49] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:46:49] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:04:49] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 04 Sep 2014 00:21:29 UTC [07:51:58] (03Abandoned) 10Legoktm: Add centralauth.dblist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/145743 (https://bugzilla.wikimedia.org/67910) (owner: 10Legoktm) [08:49:51] (03CR) 10Alexandros Kosiaris: [C: 04-2] "I revisited this. While this change can be cleanly rebased, it will not carry all the changes that have been done in the torrus manifests." [puppet] - 10https://gerrit.wikimedia.org/r/108498 (owner: 10Matanya) [09:05:49] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 04 Sep 2014 00:21:29 UTC [09:20:50] (03PS1) 10Giuseppe Lavagetto: varnish: do not mark X-analytics header on tier-2 on bits [puppet] - 10https://gerrit.wikimedia.org/r/159026 [09:27:17] (03PS3) 10Giuseppe Lavagetto: Add shell helper for the Puppet catalog compiler [puppet] - 10https://gerrit.wikimedia.org/r/158435 (owner: 10Ori.livneh) [09:27:33] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Add shell helper for the Puppet catalog compiler [puppet] - 10https://gerrit.wikimedia.org/r/158435 (owner: 10Ori.livneh) [09:34:19] (03PS2) 10Giuseppe Lavagetto: varnish: do not mark X-analytics header on tier-2 on bits [puppet] - 10https://gerrit.wikimedia.org/r/159026 [09:34:44] (03CR) 10Alexandros Kosiaris: [C: 032] ssl_ciphersuite - change Header add to Header set [puppet] - 10https://gerrit.wikimedia.org/r/155016 (owner: 10Chmarkine) [09:34:49] (03PS2) 10Alexandros Kosiaris: ssl_ciphersuite - change Header add to Header set [puppet] - 10https://gerrit.wikimedia.org/r/155016 (owner: 10Chmarkine) [09:34:57] (03CR) 10Giuseppe Lavagetto: [C: 032] "verified with pcc:" [puppet] - 10https://gerrit.wikimedia.org/r/159026 (owner: 10Giuseppe Lavagetto) [09:35:40] (03CR) 10Alexandros Kosiaris: [C: 032] ssl_ciphersuite - change Header add to Header set [puppet] - 10https://gerrit.wikimedia.org/r/155016 (owner: 10Chmarkine) [09:36:13] _joe_: I merged yours as well [09:36:53] <_joe_> akosiaris: thanks I was waiting to merge the varnish change for doing one puppet-merge [09:36:56] * _joe_ lazy [09:37:10] (03PS3) 10Giuseppe Lavagetto: varnish: do not mark X-analytics header on tier-2 on bits [puppet] - 10https://gerrit.wikimedia.org/r/159026 [09:37:16] (03CR) 10Giuseppe Lavagetto: [V: 032] varnish: do not mark X-analytics header on tier-2 on bits [puppet] - 10https://gerrit.wikimedia.org/r/159026 (owner: 10Giuseppe Lavagetto) [09:37:48] yeah, I do that too often [10:01:26] (03PS2) 10Filippo Giunchedi: image scalers: bump workers limits [puppet] - 10https://gerrit.wikimedia.org/r/157678 [10:01:35] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] image scalers: bump workers limits [puppet] - 10https://gerrit.wikimedia.org/r/157678 (owner: 10Filippo Giunchedi) [10:04:14] (03PS5) 10Giuseppe Lavagetto: beta: use HHVM everywhere, get rid of mod_php [puppet] - 10https://gerrit.wikimedia.org/r/158602 [10:04:31] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] beta: use HHVM everywhere, get rid of mod_php [puppet] - 10https://gerrit.wikimedia.org/r/158602 (owner: 10Giuseppe Lavagetto) [10:25:05] (03CR) 10Filippo Giunchedi: [C: 031] "out of curiosity what else will need change?" [puppet] - 10https://gerrit.wikimedia.org/r/158317 (owner: 10Ori.livneh) [10:29:19] (03PS2) 10Giuseppe Lavagetto: Update path references in Apache configs for /srv/mediawiki [puppet] - 10https://gerrit.wikimedia.org/r/158407 (owner: 10Ori.livneh) [10:29:51] <_joe_> godog: ^^ :) [10:30:06] <_joe_> and this has to go live first I guess [10:30:14] <_joe_> so, working on it [10:30:19] (03CR) 10Filippo Giunchedi: "LGTM generally, some minor comments, also could you add context to the commit message on what the original problem was?" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/158633 (owner: 10JanZerebecki) [10:31:04] _joe_: ack, I was expecting a lot more stuff to be relying on those paths, if it is just that even better [10:33:00] lunch, bbl [10:34:10] (03CR) 10JanZerebecki: Puppetize icinga log file permission fix. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/158633 (owner: 10JanZerebecki) [10:34:44] (03PS3) 10Giuseppe Lavagetto: Update path references in Apache configs for /srv/mediawiki [puppet] - 10https://gerrit.wikimedia.org/r/158407 (owner: 10Ori.livneh) [10:42:26] <_joe_> !log disabling puppet on all appservers while updating apache config. [10:42:31] Logged the message, Master [10:43:56] (03CR) 10Giuseppe Lavagetto: [C: 032] Update path references in Apache configs for /srv/mediawiki [puppet] - 10https://gerrit.wikimedia.org/r/158407 (owner: 10Ori.livneh) [10:48:59] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: Puppet has 1 failures [10:55:18] <_joe_> !log re-enabled puppet, the change results in a no-op as expected [10:55:23] Logged the message, Master [11:05:58] (03PS1) 10Springle: depool db1073 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159033 [11:05:59] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [11:06:26] (03CR) 10Springle: [C: 032] depool db1073 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159033 (owner: 10Springle) [11:06:34] (03Merged) 10jenkins-bot: depool db1073 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159033 (owner: 10Springle) [11:06:49] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 04 Sep 2014 00:21:29 UTC [11:07:06] !log springle Synchronized wmf-config/db-eqiad.php: depool db1073wq (duration: 00m 09s) [11:07:11] Logged the message, Master [11:07:56] springle: I like that hostname :D [11:09:47] oh [11:09:49] heh [11:09:55] guess my editor [11:10:03] nano, of course :D [11:12:53] <_joe_> lol [11:18:30] (03CR) 10Mark Bergsma: "I don't understand. How will this correctly mark bits objects delivered on tier 2 then?" [puppet] - 10https://gerrit.wikimedia.org/r/159026 (owner: 10Giuseppe Lavagetto) [11:25:00] (03PS1) 10Giuseppe Lavagetto: HAT: turn off mod_php [puppet] - 10https://gerrit.wikimedia.org/r/159037 [12:00:14] (03PS3) 10Alexandros Kosiaris: module/role class for servermon [puppet] - 10https://gerrit.wikimedia.org/r/153412 [12:00:53] (03CR) 10jenkins-bot: [V: 04-1] module/role class for servermon [puppet] - 10https://gerrit.wikimedia.org/r/153412 (owner: 10Alexandros Kosiaris) [12:04:55] (03PS4) 10Alexandros Kosiaris: module/role class for servermon [puppet] - 10https://gerrit.wikimedia.org/r/153412 [12:05:32] (03CR) 10jenkins-bot: [V: 04-1] module/role class for servermon [puppet] - 10https://gerrit.wikimedia.org/r/153412 (owner: 10Alexandros Kosiaris) [12:07:41] pep8 crap... [12:09:04] (03PS5) 10Alexandros Kosiaris: module/role class for servermon [puppet] - 10https://gerrit.wikimedia.org/r/153412 [12:09:43] (03CR) 10jenkins-bot: [V: 04-1] module/role class for servermon [puppet] - 10https://gerrit.wikimedia.org/r/153412 (owner: 10Alexandros Kosiaris) [12:20:25] (03PS6) 10Alexandros Kosiaris: module/role class for servermon [puppet] - 10https://gerrit.wikimedia.org/r/153412 [12:20:27] (03PS1) 10Springle: repool db1073, depool db1072 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159039 [12:22:54] (03CR) 10Springle: [C: 032] repool db1073, depool db1072 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159039 (owner: 10Springle) [12:22:58] (03Merged) 10jenkins-bot: repool db1073, depool db1072 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159039 (owner: 10Springle) [12:23:47] !log springle Synchronized wmf-config/db-eqiad.php: repool db1073, depool db1072 (duration: 00m 06s) [12:23:50] Logged the message, Master [12:39:51] (03CR) 10Filippo Giunchedi: [C: 031] "LGTM, just fix the commit message with the rationale for the fixes so context isn't lost" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/158633 (owner: 10JanZerebecki) [12:42:14] (03PS7) 10Alexandros Kosiaris: module/role class for servermon [puppet] - 10https://gerrit.wikimedia.org/r/153412 [12:43:24] (03PS1) 10Alexandros Kosiaris: Introduce servermon.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/159041 [12:45:41] (03CR) 10Filippo Giunchedi: [C: 031] HAT: turn off mod_php [puppet] - 10https://gerrit.wikimedia.org/r/159037 (owner: 10Giuseppe Lavagetto) [12:55:56] (03PS4) 10Filippo Giunchedi: puppetize icinga tmpfs mount in prod [puppet] - 10https://gerrit.wikimedia.org/r/158555 (owner: 10Dzahn) [12:56:03] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] puppetize icinga tmpfs mount in prod [puppet] - 10https://gerrit.wikimedia.org/r/158555 (owner: 10Dzahn) [13:07:49] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 04 Sep 2014 00:21:29 UTC [13:11:31] (03PS8) 10Alexandros Kosiaris: module/role class for servermon [puppet] - 10https://gerrit.wikimedia.org/r/153412 [13:19:53] (03PS1) 10Alexandros Kosiaris: Assign recursive dns roles to acamar/achernar [puppet] - 10https://gerrit.wikimedia.org/r/159045 [13:25:56] (03PS1) 10Filippo Giunchedi: check_graphite: fix accepted "from" ranges [puppet] - 10https://gerrit.wikimedia.org/r/159046 [13:28:43] (03CR) 10Giuseppe Lavagetto: [C: 031] check_graphite: fix accepted "from" ranges [puppet] - 10https://gerrit.wikimedia.org/r/159046 (owner: 10Filippo Giunchedi) [13:29:39] (03Abandoned) 10Giuseppe Lavagetto: beta: use HHVM for all requests [puppet] - 10https://gerrit.wikimedia.org/r/157823 (owner: 10Giuseppe Lavagetto) [13:37:17] (03CR) 10Ottomata: [WIP] Adding caching headers for wikimetrics public directory (032 comments) [puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/158819 (owner: 10Nuria) [13:42:56] (03CR) 10Alexandros Kosiaris: [C: 032] gerrit: qualify vars (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/158071 (owner: 10Matanya) [13:44:03] (03CR) 10Alexandros Kosiaris: [C: 032] protoproxy: qualify vars [puppet] - 10https://gerrit.wikimedia.org/r/157856 (owner: 10Matanya) [13:45:07] (03CR) 10Alexandros Kosiaris: [C: 032] Assign recursive dns roles to acamar/achernar [puppet] - 10https://gerrit.wikimedia.org/r/159045 (owner: 10Alexandros Kosiaris) [13:52:40] PROBLEM - puppet last run on acamar is CRITICAL: CRITICAL: Epic puppet fail [13:52:47] <_joe_> acamar [13:56:19] PROBLEM - puppet last run on achernar is CRITICAL: CRITICAL: Epic puppet fail [14:08:53] yes that is me [14:18:06] <_joe_> akosiaris: yes I was trying to think how hard will it be for me to pronounce [14:18:11] <_joe_> :p [14:39:30] (03PS1) 10Springle: repool db1072 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159053 [14:40:22] (03CR) 10Springle: [C: 032] repool db1072 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159053 (owner: 10Springle) [14:40:27] (03Merged) 10jenkins-bot: repool db1072 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159053 (owner: 10Springle) [14:41:17] !log springle Synchronized wmf-config/db-eqiad.php: repool db1072 (duration: 00m 09s) [14:41:23] Logged the message, Master [14:48:00] (03PS2) 10Nuria: Adding caching headers for wikimetrics public directory [puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/158819 (https://bugzilla.wikimedia.org/68445) [14:48:28] anomie: I'm happy to do SWAT today - I have a few patches in it [14:48:51] manybubbles: I had assumed that you would (: [14:49:08] * aude panic.... makign my patch [14:49:19] can you review the https://gerrit.wikimedia.org/r/#/c/158786/ and https://gerrit.wikimedia.org/r/#/c/158785/ ? [14:49:30] they are one line changes to things that use the api [14:49:33] anomie: ^^ [14:49:50] chasemp: re: phabricator - analytics doesn't need their stuff migrated from the labs instance [14:50:08] (03PS1) 10Chad: CirrusSearch: svwiki to primary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159054 [14:50:15] manybubbles: +2ed, cherry picks look correct [14:50:47] (03PS1) 10Alexandros Kosiaris: Revert "Assign recursive dns roles to acamar/achernar" [puppet] - 10https://gerrit.wikimedia.org/r/159055 [14:51:02] milimetric: what projects are those? jsut analytics? [14:51:11] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Revert "Assign recursive dns roles to acamar/achernar" [puppet] - 10https://gerrit.wikimedia.org/r/159055 (owner: 10Alexandros Kosiaris) [14:51:34] chasemp: yes, "Analytics-EEVS" [14:51:43] thanks dude! [14:51:48] danke you! [14:51:51] or sir however you prefer [14:54:26] anomie: sweet. [14:55:19] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [14:55:29] (03CR) 10Alexandros Kosiaris: [C: 031] "The idea is fine and me likes :-). Filippo does have a point though." [puppet] - 10https://gerrit.wikimedia.org/r/158317 (owner: 10Ori.livneh) [14:55:48] (03CR) 10Manybubbles: [C: 031] "Can deploy in an hour during our window. Or after I'm done with SWAT - same difference. Also did performance tests and it looks just fin" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159054 (owner: 10Chad) [14:57:18] thedj: I'm going to do your wmf19 backport for wikilove soon [14:57:46] (03CR) 10Ottomata: [C: 032 V: 032] Adding caching headers for wikimetrics public directory [puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/158819 (https://bugzilla.wikimedia.org/68445) (owner: 10Nuria) [14:57:49] manybubbles: k [15:00:04] manybubbles, anomie, ^d, marktraceur: Respected human, time to deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140908T1500). Please do the needful. [15:00:26] building the submodule update for it now [15:02:06] * aude waits for jenkins [15:02:20] I am going to be taking neon down in a few minutes [15:02:33] Hey, sorry I'm late. [15:08:48] (03CR) 10Manybubbles: [C: 032] Lower throttle for Cirrus template update jobs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/158401 (owner: 10Manybubbles) [15:08:49] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 04 Sep 2014 00:21:29 UTC [15:09:04] (03CR) 10Manybubbles: [C: 032] Separate high traffic Elasticsearch shards [mediawiki-config] - 10https://gerrit.wikimedia.org/r/158566 (owner: 10Manybubbles) [15:09:08] (03Merged) 10jenkins-bot: Lower throttle for Cirrus template update jobs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/158401 (owner: 10Manybubbles) [15:09:13] (03Merged) 10jenkins-bot: Separate high traffic Elasticsearch shards [mediawiki-config] - 10https://gerrit.wikimedia.org/r/158566 (owner: 10Manybubbles) [15:09:18] (03CR) 10Manybubbles: [C: 032] Switch all wikis to Cirrus' weighted all fields [mediawiki-config] - 10https://gerrit.wikimedia.org/r/158420 (owner: 10Manybubbles) [15:09:46] (03Merged) 10jenkins-bot: Switch all wikis to Cirrus' weighted all fields [mediawiki-config] - 10https://gerrit.wikimedia.org/r/158420 (owner: 10Manybubbles) [15:10:14] !log shutting down neon for memory upgrade [15:10:20] Logged the message, Master [15:10:24] !log manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT update some cirrus settings (duration: 00m 04s) [15:10:29] Logged the message, Master [15:10:39] !log manybubbles Synchronized wmf-config: SWAT finish updating Cirrus settings (duration: 00m 05s) [15:10:40] RECOVERY - puppet last run on acamar is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [15:10:45] Logged the message, Master [15:12:24] manybubbles: uploading patches [15:14:06] thedj: you are synced [15:14:28] bd808: error in sync-dir https://gist.github.com/nik9000/2437dedfd93ba43928f5 [15:15:14] thedj: I did wmf19 first (backward, oops) can you verify it? [15:16:40] (03PS3) 10Alexandros Kosiaris: WIP: assign mathoid production hosts [puppet] - 10https://gerrit.wikimedia.org/r/156576 (https://bugzilla.wikimedia.org/69990) (owner: 10Physikerwelt) [15:16:42] James_F: going to do liquidthreads soon [15:16:58] Thanks. [15:17:10] manybubbles: That's the irc proxy being down/unreachable I think. It's trying to talk to neon.wikimedia.org:9200 [15:17:13] manybubbles: when you are ready for us, https://gerrit.wikimedia.org/r/#/c/159064/ and https://gerrit.wikimedia.org/r/#/c/159066/ [15:17:40] aude: awesome - can you add it to the page? [15:17:44] I tend to read that [15:17:51] bd808: k. I thought so given no irc ping [15:18:01] done [15:18:02] anomie: can you verify the fix for wikilove on wmf19? [15:18:14] I'm not getting a response from thedj at the moment [15:18:19] * anomie will look [15:18:24] thanks! [15:19:25] Bah. Tin is trying to connect to neon over ipv6 again. I thought I remembered that being fixed/changed a few months ago. [15:19:42] manybubbles: Works [15:20:05] !log manybubbles Synchronized wmf-config: SWAT another cirrus setting update (duration: 00m 04s) [15:20:09] anomie: thanks [15:20:11] Logged the message, Master [15:20:44] anomie and thedj: merging wmf20's wikilove fix now. will deploy on merge [15:21:02] Anyone remember anything more about that? tin -> neon via ipv6 not working and a fix to make tin not try? [15:21:10] * bd808 goes to look in emails [15:21:44] ^d: we're using the all fields everywhere. yay [15:21:51] finally [15:21:51] <^d> :) [15:22:03] (03PS4) 10Alexandros Kosiaris: Introducing Service Cluster A, hosting mathoid [puppet] - 10https://gerrit.wikimedia.org/r/156576 (https://bugzilla.wikimedia.org/69990) (owner: 10Physikerwelt) [15:22:31] ^d: and now I can remove the option not to use them! [15:22:35] that'll be fun [15:22:44] <^d> Less complicated at least :) [15:22:50] ^d: oh yeah! [15:23:02] and, also, ! I have a patch I'm working on to make them a bit more efficient [15:23:21] there is a slight loss in precision I think but it'll be worth it I think [15:23:31] right now they take up a lot of room [15:23:50] and our disk io problem seems to be caused by doing too much io on pulling in the positional information for them [15:25:45] !log manybubbles Synchronized php-1.24wmf20/extensions/WikiLove/: (no message) (duration: 00m 05s) [15:25:49] Logged the message, Master [15:25:53] bd808: you fixed it! [15:25:59] also, [15:26:08] anomie and thedj: wikilove fix is live on wmf20 [15:26:50] manybubbles: Maybe I tricked the hosts cache on tin? My raw telnet test is still failing [15:27:03] bd808: dunno [15:27:09] but I got logging [15:27:39] !log sync logging was down so it missed some syncing I just did. [15:27:44] Logged the message, Master [15:28:13] !log this is the missing log: [15:28:15] !log 15:13:53 Synchronized php-1.24wmf19/extensions/WikiLove/: SWAT fix for WikiLove (duration: 00m 04s) [15:28:18] Logged the message, Master [15:28:22] Logged the message, Master [15:32:01] !log manybubbles Synchronized php-1.24wmf20/extensions/LiquidThreads/: SWAT update liquidthreads to fix some missing images (duration: 00m 04s) [15:32:05] Logged the message, Master [15:32:23] James_F: liquidthreads is done [15:32:32] * James_F checks. [15:32:46] Yup, confirmed. Thanks! [15:33:41] sweet [15:34:42] aude: just merging your submodule update [15:34:48] ok [15:35:13] PSA: there is a #openstack-operators freenode channel which I just joined, if anyone is interested [15:36:20] Is bits broken again? [15:36:27] bd808: connection problems from tin to neon... again?! [15:36:44] hoo: Apparently :( [15:36:55] (03CR) 10Chad: "I'll do it myself in ~24 minutes during the window :)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159054 (owner: 10Chad) [15:36:56] https://bits.wikimedia.org/en.wikipedia.org/load.php?debug=true&lang=en&modules=ext.visualEditor.viewPageTarget.noscript&only=styles&skin=vector&* [15:37:36] 404 file not found, unless you change the URL slightly [15:38:39] we had the very same problem before... that's weird [15:38:39] hoo bd808 lining up with recent neon maint perhaps? [15:40:14] https://gerrit.wikimedia.org/r/134277 [15:40:18] possible [15:40:22] https://en.wikipedia.org/w/index.php?title=Special:Random&debug=true - unstyled every time I try it [15:40:37] lots of 404s [15:41:01] WFM [15:41:04] :( [15:41:10] * aude gets nostalgia [15:41:16] ���� [15:41:22] whoops [15:41:51] but only sometimes [15:42:17] https://en.wikipedia.org/wiki/Varicella_vaccine?debug=true is quite odd [15:42:38] !log manybubbles Synchronized php-1.24wmf20/extensions/Wikidata/: SWAT update wikidata to fix add links widget (duration: 00m 06s) [15:42:43] Logged the message, Master [15:42:49] http://snag.gy/rxSNR.jpg [15:42:55] aude: synced for you [15:42:57] wmf20 [15:43:00] ok [15:43:07] let's figure out bits [15:43:24] * aude verifies on test.wikidata / wiipedia [15:46:13] aude: I can wait to merge your wmf19 backport until the bits thing is calmed down [15:46:18] http://snag.gy/nf8W8.jpg on mw1210 [15:46:27] manybubbles: please do, although looks fine on test [15:46:29] aude: that's because it passes the debug=true to the stylesheet rel links. We're debugging that now, but it's all about the bits cache/servers somehow and debug=true requests [15:46:34] cool [15:46:38] ok [15:46:54] [15:47:03] hmmm [15:47:04] (^ from the content served by your Varicella link) [15:48:14] bd808: could it be this: [15:48:15] ferm::rule { 'tcpircbot_allowed': [15:48:15] rule => 'proto tcp dport 9200 { saddr (10.64.21.123 10.64.0.196 208.80.152.165 127.0.0.1) ACCEPT; }', [15:49:31] maybe someone per hand edited iptables and it's gone after reboot [15:53:18] hoo: That looks like a likely culprit. It is acting like the ipv6 packets are silently dropped. [15:53:24] yep [15:53:31] some op would need to check that [15:53:37] or you can just submit a puppet change [15:53:40] * bd808 shakes fist at ferm [15:57:07] aude: your final SWAT sync is going to bleed into Cirrus's window - which is cool with us - we don't need the whole hour [15:57:16] :/ [15:57:44] * ^d missed scrollback [15:57:45] <^d> I can wait a bit for you guys to finish up. [16:00:04] manybubbles, ^d: Respected human, time to deploy Search (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140908T1600). Please do the needful. [16:00:04] RobH: Dear anthropoid, the time has come. Please deploy Ops (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140908T1600). [16:00:22] <^d> I'm not anthropoid today! [16:01:16] ? [16:01:33] ahhh, the icinga upgrade [16:01:35] cmjohnson1: ^ [16:01:47] seems the depoloyment calendar is reminding us of the time [16:01:49] <^d> It's on the deploy calendar :) [16:01:50] <^d> Yep [16:01:52] <^d> Jouncebot [16:02:01] thats the first time jouncebot has pinged me, ever. [16:02:11] I've lived a charmed life. [16:03:05] I googled jounce, I think I know less than I knew before [16:03:19] robh: that was done 1 hour ago [16:03:23] at 1500UTC [16:03:24] oh, awesome [16:03:35] (03PS9) 10Alexandros Kosiaris: module/role class for servermon [puppet] - 10https://gerrit.wikimedia.org/r/153412 [16:03:36] so the calendar was a bit wonky then, no worries [16:03:38] <^d> chasemp: https://github.com/wikimedia/wikimedia-bots-jouncebot I think [16:04:03] I meant where did the name 'jounce' come from :) [16:04:51] cmjohnson1: well you did awesome enough that its not noticable now ;] [16:04:58] <^d> chasemp: Not a clue [16:05:21] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 15:59:17 UTC [16:05:32] robh ^ [16:06:18] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 15:59:17 UTC [16:07:18] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 15:59:17 UTC [16:08:19] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 15:59:17 UTC [16:09:18] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 15:59:17 UTC [16:10:08] bleh [16:10:15] i jinxed it, sorry [16:10:18] RECOVERY - Puppet freshness on neon is OK: puppet ran at Mon Sep 8 16:10:15 UTC 2014 [16:10:19] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 15:59:17 UTC [16:10:30] ^d: should we wait? [16:10:58] <^d> I dunno. We can. [16:11:19] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:10:15 UTC [16:12:18] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:10:15 UTC [16:12:37] ^d: k. mind if I go away for a bit? I'll be back in about an hour? I imagine the deploy will be boring. [16:12:59] <^d> Yeah, go ahead. [16:13:00] * aude in no hurry [16:13:06] sweet [16:13:18] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:10:15 UTC [16:14:16] <^d> aude: tl;dr for me: what're you still waiting on? [16:14:19] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:10:15 UTC [16:14:49] ^d https://gerrit.wikimedia.org/r/#/c/159064/ [16:15:04] if bits is having problems, maybe it's best to wait until that is resolved? [16:15:18] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:10:15 UTC [16:15:33] <^d> Hmm, ok [16:16:18] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:10:15 UTC [16:16:19] <^d> What bits problems are we having? I saw zh-yue in scrollback? [16:16:27] <^d> zh-yue wfm logged in & out. [16:16:52] (03CR) 10BBlack: Move all geoip-based resolution to DYNA (032 comments) [dns] - 10https://gerrit.wikimedia.org/r/158382 (owner: 10BBlack) [16:16:57] (03PS1) 10Giuseppe Lavagetto: varnish: mangle request data for uncacheable objects [puppet] - 10https://gerrit.wikimedia.org/r/159080 [16:17:19] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:10:15 UTC [16:17:57] (03CR) 10Chad: [C: 032] CirrusSearch: svwiki to primary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159054 (owner: 10Chad) [16:17:59] (03PS2) 10Giuseppe Lavagetto: varnish: mangle request data for uncacheable objects [puppet] - 10https://gerrit.wikimedia.org/r/159080 [16:18:17] (03Merged) 10jenkins-bot: CirrusSearch: svwiki to primary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159054 (owner: 10Chad) [16:18:19] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:10:15 UTC [16:18:57] ^d: with debug=true [16:19:00] !log demon Synchronized wmf-config/InitialiseSettings.php: svwiki: Cirrus as primary (duration: 00m 04s) [16:19:05] <^d> aude: Ew. [16:19:05] Logged the message, Master [16:19:22] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:10:15 UTC [16:19:53] https://en.wikipedia.org/wiki/Berlin?debug=true [16:19:58] RECOVERY - Puppet freshness on neon is OK: puppet ran at Mon Sep 8 16:19:55 UTC 2014 [16:20:19] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:21:18] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:22:31] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:23:28] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:24:14] (03PS1) 10Alexandros Kosiaris: tcpircbot: add tin's IPv6 in ferm rules [puppet] - 10https://gerrit.wikimedia.org/r/159081 [16:24:19] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:25:18] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:25:48] (03CR) 10Ori.livneh: [C: 031] "LGTM apart from what Filippo pointed out" [puppet] - 10https://gerrit.wikimedia.org/r/156303 (owner: 10Giuseppe Lavagetto) [16:26:19] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:27:19] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:28:19] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:29:19] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:30:13] (03PS2) 10Alexandros Kosiaris: tcpircbot: add tin's IPv6 in ferm rules [puppet] - 10https://gerrit.wikimedia.org/r/159081 [16:30:19] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:31:19] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:32:28] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:33:28] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:34:05] (03PS1) 10Ori.livneh: Remove plural-form alias for require_package() [puppet] - 10https://gerrit.wikimedia.org/r/159083 [16:34:28] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:35:28] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:35:44] (03PS3) 10Alexandros Kosiaris: tcpircbot: add tin's IPv6 in ferm rules [puppet] - 10https://gerrit.wikimedia.org/r/159081 [16:36:27] is gitblit on neon? [16:36:28] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:36:49] * aude assume so [16:37:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:38:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:38:56] <^d> aude: No, antimony. [16:39:28] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:19:55 UTC [16:39:48] RECOVERY - Puppet freshness on neon is OK: puppet ran at Mon Sep 8 16:39:43 UTC 2014 [16:40:04] well, it's broken and been broken [16:40:21] <^d> It's always broken. [16:40:27] heh, not always [16:40:33] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:39:43 UTC [16:40:34] <^d> gitblit sucks. [16:40:39] for jenkins we do curl 'https://git.wikimedia.org/zip/?r=mediawiki/core.git&format=gz&h=master' [16:40:48] maybe we should get from github again [16:41:02] <^d> dafuq? [16:41:10] therefore jenkins says 'no' to all my patches [16:41:17] <^d> That /zip/ thing is the worst thing gitblit does. [16:41:24] :( [16:41:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:39:43 UTC [16:41:30] it usually works okay [16:41:31] <^d> It's why we rewrote extension dist. [16:41:41] so... use github? [16:41:43] hey Coren: I created last week this ops-request ticket https://rt.wikimedia.org/Ticket/Display.html?id=8283 but now RT tells me I don’t have permissions to view it :( [16:41:44] yes [16:41:48] re: github [16:41:49] i think we did use github [16:41:56] <^d> For on-the-fly zip generation of repos? Yes. [16:42:05] <^d> Github's better at that than we'll ever be, and for good reason. [16:42:06] i forget why we changed it... there must have been some problem [16:42:10] heh [16:42:19] aude, hoo: thanks for jumping on the memcached issue so quickly btw, and sorry if i was e-mailing the wrong list (wikidata-l) [16:42:28] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:39:43 UTC [16:42:29] legoktm mentioned wikidata-tech would have been more appropriate [16:42:31] I'm not even on wikidata-l :P [16:42:34] yep [16:42:56] i'll use that in the future [16:43:05] ori: if you have spare time, can you look at curl 'https://git.wikimedia.org/zip/?r=mediawiki/core.git&format=gz&h=master' [16:43:08] gah [16:43:14] https://gerrit.wikimedia.org/r/#/c/158879/ [16:43:38] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:39:43 UTC [16:43:39] is that sane or what do you think is a good way for us to update part of parser output (e.g. extension data) [16:43:47] without having to reparse everything [16:43:49] <^d> Quick, everybody curl mediawiki/core from gitblit! [16:43:50] <^d> :) [16:44:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:39:43 UTC [16:45:00] aude: when i was looking at the raw memcached keys, i noticed the key had a timestamp that was weeks/months old. how often does the data change? [16:45:21] for sites list, only when we get a new wiki [16:45:28] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:39:43 UTC [16:45:39] aude: can't it live in an array in operations-config then? [16:46:15] maybe cdb? [16:46:28] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:39:43 UTC [16:46:30] why cdb? it can really just be a plain array [16:46:36] (03PS3) 10Giuseppe Lavagetto: varnish: mangle request data for uncacheable objects [puppet] - 10https://gerrit.wikimedia.org/r/159080 [16:46:38] maybe [16:46:59] we don't want to maintain it by hand [16:47:03] a plain array would certainly be faster than fetching a byte string across a network link from memcached and decoding it into a plain array [16:47:23] not sure cdb is warranted yet [16:47:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:39:43 UTC [16:48:09] * bd808 crosses self against more cdb files [16:48:15] heh [16:48:28] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:39:43 UTC [16:48:35] could be json, maybe [16:48:39] the general point of my question is: how do we make sure we avoid this in the future? at a glance, your patch seems like it fixes the current issue, but doesn't it leave open the possibility that some other innocent code will quietly reintroduced huge memcached gets into every request? [16:48:41] something autogenerated [16:49:00] yeah, that would work too [16:49:11] you could add it to the scripts Reedy uses to cut a new version [16:49:20] (03CR) 10BBlack: [C: 031] varnish: mangle request data for uncacheable objects [puppet] - 10https://gerrit.wikimedia.org/r/159080 (owner: 10Giuseppe Lavagetto) [16:49:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:39:43 UTC [16:49:31] sure [16:49:33] (03CR) 10Alexandros Kosiaris: [C: 032] tcpircbot: add tin's IPv6 in ferm rules [puppet] - 10https://gerrit.wikimedia.org/r/159081 (owner: 10Alexandros Kosiaris) [16:49:55] <^d> aude, ori: mediawiki/tools/release.git : make-wmf-branch [16:49:59] for https://gerrit.wikimedia.org/r/#/c/158879/ the issue is sidebarcache [16:50:23] and we want per-page thing added (links to related projects, e.g. wikivoyage associated with the page / wikidata item) [16:50:30] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:39:43 UTC [16:50:36] aude: it looks fine; i really don't know enough about wikibase internals to give it a very detailed review. [16:50:41] ok :/ [16:50:55] i can help you verify it if you like [16:51:07] we wanted to put it inparser output instead of doing lookup when viewing [16:51:20] ohh [16:51:25] but it's very big, no? [16:51:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:39:43 UTC [16:51:29] no [16:52:19] RECOVERY - Puppet freshness on neon is OK: puppet ran at Mon Sep 8 16:52:12 UTC 2014 [16:52:31] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:52:12 UTC [16:52:45] so, when we deploy code to put it in parser output, then the feature breaks for all non-purged pages [16:52:53] might be ok to have people purge [16:53:27] but would be nice to be able to invalidate / update small part of parser cache in more granular way [16:53:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:52:12 UTC [16:53:51] might help with other things we do also [16:54:04] if you have any ideas..... [16:54:31] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:52:12 UTC [16:55:30] hmm [16:55:38] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:52:12 UTC [16:56:07] * ori pats logmsgbot. [16:56:10] i might need to send mail to wikitech [16:56:20] +1 [16:56:30] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:52:12 UTC [16:57:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:52:12 UTC [16:58:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:52:12 UTC [16:59:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:52:12 UTC [16:59:49] RECOVERY - Puppet freshness on neon is OK: puppet ran at Mon Sep 8 16:59:36 UTC 2014 [17:00:05] (03CR) 10Ori.livneh: [C: 031] HAT: turn off mod_php [puppet] - 10https://gerrit.wikimedia.org/r/159037 (owner: 10Giuseppe Lavagetto) [17:00:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:59:36 UTC [17:01:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:59:36 UTC [17:02:30] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:59:36 UTC [17:03:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:59:36 UTC [17:04:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:59:36 UTC [17:04:46] <_joe_> can someone silence neon? [17:05:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:59:36 UTC [17:06:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:59:36 UTC [17:06:32] (03CR) 10Alexandros Kosiaris: pybal: qualify vars (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/158086 (owner: 10Matanya) [17:06:40] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Epic puppet fail [17:07:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:59:36 UTC [17:07:34] (03PS4) 10Giuseppe Lavagetto: varnish: mangle request data for uncacheable objects [puppet] - 10https://gerrit.wikimedia.org/r/159080 [17:08:22] (03CR) 10Giuseppe Lavagetto: [C: 032] varnish: mangle request data for uncacheable objects [puppet] - 10https://gerrit.wikimedia.org/r/159080 (owner: 10Giuseppe Lavagetto) [17:08:28] (03Abandoned) 10Matanya: mobile: replace iptables with ferm rule [puppet] - 10https://gerrit.wikimedia.org/r/117673 (owner: 10Matanya) [17:08:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:59:36 UTC [17:09:28] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 04 Sep 2014 00:21:29 UTC [17:09:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:59:36 UTC [17:10:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:59:36 UTC [17:11:30] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:59:36 UTC [17:12:04] (03PS9) 10ArielGlenn: data retention audit script for logs, /root and /home dirs [software] - 10https://gerrit.wikimedia.org/r/141473 [17:12:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:59:36 UTC [17:12:45] ACKNOWLEDGEMENT - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 16:59:36 UTC Giuseppe Lavagetto STOP IT! [17:14:11] yeah.. let's kill this puppet freshness check this week.. it has no reason to exist anymore [17:14:39] RECOVERY - Puppet freshness on neon is OK: puppet ran at Mon Sep 8 17:14:33 UTC 2014 [17:15:30] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 17:14:33 UTC [17:16:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 17:14:33 UTC [17:16:39] are the bits issues resolved? [17:16:45] <_joe_> OMFG MAKE THAT STOP [17:16:49] looks better to me [17:16:57] <_joe_> aude: they will in the next 20 mins [17:17:00] ok [17:17:10] <_joe_> but yes, they should be resolved [17:17:28] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: Puppet has 1 failures [17:17:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 17:14:33 UTC [17:18:29] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Mon 08 Sep 2014 17:14:33 UTC [17:19:00] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 1 failures [17:19:07] !log disabled notifications for puppet freshness on neon [17:19:08] there [17:19:09] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 1 failures [17:19:12] Logged the message, Master [17:19:37] Coren, are you actually on RT? if so, wanna check this out? [17:19:37] https://rt.wikimedia.org/Ticket/Display.html?id=8283 [17:19:59] <_joe_> varnishes are my error btw [17:20:04] an ack just lasts until the next change, but if it's flapping like that, disabling notifications entirely and the irc bot wont be told anymore [17:21:45] (03PS1) 10Giuseppe Lavagetto: bits: declare mangle_request on tier1 only [puppet] - 10https://gerrit.wikimedia.org/r/159087 [17:22:07] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] bits: declare mangle_request on tier1 only [puppet] - 10https://gerrit.wikimedia.org/r/159087 (owner: 10Giuseppe Lavagetto) [17:23:09] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: Puppet has 1 failures [17:25:08] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Puppet has 1 failures [17:25:28] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:26:48] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [17:30:25] (03PS1) 10Legoktm: Enable $wgContentHandlerUseDB on mediawikiwiki, testwiki, & test2wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159089 (https://bugzilla.wikimedia.org/49193) [17:32:28] ^d: _joe_ should be ok to deploy https://gerrit.wikimedia.org/r/#/c/159064/ ? [17:37:09] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:37:18] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [17:42:09] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [17:43:18] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:46:08] ^d: around? [17:46:48] <^d> I'm trying to outsmart eswiki's abusefilter right now :p [17:46:53] ok :) [17:48:17] <^d> aude: +2'd. [17:48:25] yay [17:48:37] i can sync if that's easier [17:49:09] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: Puppet has 1 failures [17:49:13] <^d> I'm already on tin, no big deal. [17:49:22] k [17:51:59] aude: did you get deployed? sorry that took so long [17:52:12] doing now [17:52:19] great [17:57:16] manybubbles: sorry for not responding, my boss pulled me away for an emergency release on our software [17:57:50] thedj: fun fun - its not problem. anomie was able to verify. if he hadn't I'd have muddled through :) [17:57:56] thanks for the fix [17:59:01] well anomie made the fix, i just rolled the backport [17:59:55] of ffs [18:00:02] fram is at it again [18:01:28] !log demon Synchronized php-1.24wmf19/extensions/Wikidata: Updating Wikidata to f1d2110 (duration: 00m 09s) [18:01:32] <^d> aude: ^ [18:01:32] Logged the message, Master [18:01:33] yay [18:01:53] beta thing looks good [18:02:21] * aude check the widget [18:05:21] hoo: can you check the widget? [18:06:03] in debug mode, it works [18:06:24] works in private browsing, non debug [18:07:18] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:08:03] ok, it works [18:15:49] <_joe_> win 23 [18:19:33] aude: Still need me to check? [18:19:51] ottomata: I was last week. I think it's apergos this week. [18:20:08] yes indeed [18:20:14] I forgot to change it over [18:21:49] apergos, when you have time: https://rt.wikimedia.org/Ticket/Display.html?id=8283 [18:46:58] PROBLEM - Ubuntu mirror in sync with upstream on carbon is CRITICAL: /srv/ubuntu/project/trace/carbon.wikimedia.org is over 12 hours old. [18:47:49] RECOVERY - Ubuntu mirror in sync with upstream on carbon is OK: /srv/ubuntu/project/trace/carbon.wikimedia.org is over 0 hours old. [18:48:29] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Puppet has 1 failures [18:53:37] _joe_, aude: Was the bits issue fixed by https://gerrit.wikimedia.org/r/#/c/159087 then ? [18:54:00] <_joe_> Krenair: yes, it should have [18:54:12] <_joe_> and the preceding change [18:54:24] <_joe_> the preceding one solved the issue [18:54:51] <_joe_> https://gerrit.wikimedia.org/r/#/c/159080/ [18:54:59] <_joe_> Krenair: still seeing issues? [18:55:15] nope, looks ok now [18:55:23] <_joe_> ok :) [18:55:25] thanks [19:06:28] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [19:10:28] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 04 Sep 2014 00:21:29 UTC [19:30:49] (03PS1) 10Ori.livneh: Small tweaks for pcc script [puppet] - 10https://gerrit.wikimedia.org/r/159121 [19:33:53] (03PS2) 10Ori.livneh: Remove plural-form alias for require_package() [puppet] - 10https://gerrit.wikimedia.org/r/159083 [19:34:41] (03CR) 10Ori.livneh: [C: 032] Remove plural-form alias for require_package() [puppet] - 10https://gerrit.wikimedia.org/r/159083 (owner: 10Ori.livneh) [19:39:09] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: Epic puppet fail [19:42:57] <_joe_> ori: used pcc today, it's really really nice [19:43:18] _joe_: thanks! :) [19:44:08] <_joe_> it's funny how a tool that was thought to be used from the cli (via vagrant) turned into a jenkins job, and we wrote a cli script to interact with it :P [19:46:51] yes, we just need a web interface for the cli script now [19:50:57] ori: pcc stands for puppet compiler c? [19:51:02] catalog ? [19:57:48] puppet catalog compiler [19:58:11] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:14:54] !log deployed Parsoid ce108cb5 [20:14:58] Logged the message, Master [20:23:50] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: Puppet has 1 failures [20:36:53] (03CR) 10Ori.livneh: [C: 032] "simple & tested" [puppet] - 10https://gerrit.wikimedia.org/r/159121 (owner: 10Ori.livneh) [20:37:39] ori: all the work with wmflib man, great stuff thank you [20:38:06] chasemp: :) have you seen https://wikitech.wikimedia.org/wiki/Wmflib ? [20:38:59] whaaaat awesome [20:41:58] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [20:42:40] PROBLEM - puppet last run on tarin is CRITICAL: CRITICAL: Epic puppet fail [20:47:13] (03PS1) 10RobH: adding pending deployment ganglia group and setting it to default [puppet] - 10https://gerrit.wikimedia.org/r/159167 [20:47:49] mutante: ^ added you as reviewer ;] [20:47:55] (03CR) 10jenkins-bot: [V: 04-1] adding pending deployment ganglia group and setting it to default [puppet] - 10https://gerrit.wikimedia.org/r/159167 (owner: 10RobH) [20:48:02] aww [20:48:06] gerrit hated it. [20:50:48] (03PS2) 10RobH: adding pending deployment ganglia group and setting it to default [puppet] - 10https://gerrit.wikimedia.org/r/159167 [20:51:25] (03CR) 10jenkins-bot: [V: 04-1] adding pending deployment ganglia group and setting it to default [puppet] - 10https://gerrit.wikimedia.org/r/159167 (owner: 10RobH) [20:51:50] grrr [20:52:49] it complains it needs to be } not }, when the line that line that ended the array before my change ended in }, [20:53:18] 20:50:58 Error: Could not parse for environment production: Syntax error at ','; expected '}' at /srv/ssd/jenkins-slave/workspace/operations-puppet-pplint-HEAD/modules/ganglia_new/manifests/configuration.pp:107 [20:53:22] https://gerrit.wikimedia.org/r/#/c/159167/2/modules/ganglia_new/manifests/configuration.pp [20:54:03] oh wait, damn it [20:54:05] im wrong. [20:54:24] i close the entry on the line before by mistake. [20:54:54] (03PS3) 10RobH: adding pending deployment ganglia group and setting it to default [puppet] - 10https://gerrit.wikimedia.org/r/159167 [20:57:09] (03PS4) 10RobH: adding pending deployment ganglia group and setting it to default [puppet] - 10https://gerrit.wikimedia.org/r/159167 [20:57:26] mutante: ok, now that im four patchsets in for what should have been 2. [20:57:34] feel free to look it over when you have a moment, its not urgent [21:00:49] (03PS8) 10Ori.livneh: Add Grafana module & role [puppet] - 10https://gerrit.wikimedia.org/r/133274 [21:01:08] ^ chasemp oldie but goodie :) finally got around to updating it. operations/software/grafana is up-to-date now, too. [21:01:22] (03PS1) 10BryanDavis: Fix doc header for ensure_link [puppet] - 10https://gerrit.wikimedia.org/r/159174 [21:01:25] actually I saw that today [21:01:30] good test data in labs now too [21:01:31] (03CR) 10jenkins-bot: [V: 04-1] Add Grafana module & role [puppet] - 10https://gerrit.wikimedia.org/r/133274 (owner: 10Ori.livneh) [21:01:38] RECOVERY - puppet last run on tarin is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [21:02:20] ACKNOWLEDGEMENT - RAID on nickel is CRITICAL: CRITICAL: Active: 1, Working: 1, Failed: 1, Spare: 0 daniel_zahn RT: 8252 Disk fail [21:03:02] ori: Can you make wmflib a submodule so we can use it directly in MediaWiki-Vagrant? [21:11:28] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 04 Sep 2014 00:21:29 UTC [21:15:17] bd808: yes, good idea [21:15:34] (03PS9) 10Ori.livneh: Add Grafana module & role [puppet] - 10https://gerrit.wikimedia.org/r/133274 [21:19:17] (03PS1) 10Nuria: Adding gzip compression to several file types [puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/159181 [21:20:17] (03PS2) 10Nuria: Adding gzip compression for several file types [puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/159181 [21:34:00] (03CR) 10Dzahn: [C: 032] "Hosts where compilation is identical:" [puppet] - 10https://gerrit.wikimedia.org/r/153763 (owner: 10Hashar) [21:34:09] (03PS1) 10RobH: setting ip address for db2001-db2042 in codfw [dns] - 10https://gerrit.wikimedia.org/r/159188 [21:39:13] (03CR) 10RobH: [C: 031] "I am 99% certain what I am doing here is right. However, as it will set the standard that the rest of the internal IP address hosts follo" [dns] - 10https://gerrit.wikimedia.org/r/159188 (owner: 10RobH) [21:40:35] (03CR) 10Dzahn: [C: 032] check_graphite: fix accepted "from" ranges [puppet] - 10https://gerrit.wikimedia.org/r/159046 (owner: 10Filippo Giunchedi) [22:03:09] PROBLEM - MySQL Processlist on db1054 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 87 statistics [22:04:08] RECOVERY - MySQL Processlist on db1054 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 1 statistics [22:08:09] (03PS1) 10RobH: setting install params for db2001-2031 [puppet] - 10https://gerrit.wikimedia.org/r/159200 [22:27:22] OuKB: starting early? [22:30:20] PROBLEM - puppet last run on antimony is CRITICAL: CRITICAL: Puppet has 1 failures [22:30:54] legoktm, submodule commits are needed by window start [22:31:03] oh [22:31:10] do you want me to bump the submodules then? [22:31:27] would appreciate this, yes [22:32:19] RECOVERY - puppet last run on antimony is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [22:41:30] OuKB: https://gerrit.wikimedia.org/r/159212 and https://gerrit.wikimedia.org/r/159213 [22:43:38] (03CR) 10Dzahn: [C: 032] "ran in compiler, job 322" [puppet] - 10https://gerrit.wikimedia.org/r/149890 (owner: 10ArielGlenn) [22:51:18] OuKB: are you deploying SWAT today? i've added a note on the deployment page but the Flow+Echo bump to wmf20 requires a full scap for two new i18n messages [22:51:21] http://people.wikimedia.org/~dzahn/ [22:51:29] ^ in case you have stuff on fenari [22:51:34] you can now move that over [22:51:48] this if for people who used a public_html on fenari [22:52:05] mutante, to tin? [22:52:13] OuKB: terbium [22:52:20] uh [22:52:38] why terbium, it's a host for running maint scripts [22:52:39] deployers should have that option [22:53:27] i guess it's more "why not", terbium wasnt doing that much and having another server just for this is a waste [22:53:42] but i didnt decide either [22:54:15] it's a place where deployers already have shell [22:55:59] i like how it's not a bastion host :) [23:00:05] RoanKattouw, ^d, marktraceur, MaxSem, ebernhardson: Dear anthropoid, the time has come. Please deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140908T2300). [23:00:17] grmbl [23:00:22] me I guess [23:00:37] Don't sound so enthusiastic [23:00:37] i can deploy if you really dont want to :) [23:00:49] can i still add mw config changes?:) [23:01:00] ebernhardson, thanks! [23:01:05] OuKB: np [23:01:30] legoktm: are you still prepping submodule bumps or should i do that now? [23:01:56] (03CR) 10EBernhardson: [C: 032] Enable $wgContentHandlerUseDB on mediawikiwiki, testwiki, & test2wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159089 (https://bugzilla.wikimedia.org/49193) (owner: 10Legoktm) [23:02:05] (03Merged) 10jenkins-bot: Enable $wgContentHandlerUseDB on mediawikiwiki, testwiki, & test2wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/159089 (https://bugzilla.wikimedia.org/49193) (owner: 10Legoktm) [23:02:57] !log ebernhardson Synchronized wmf-config/InitialiseSettings.php: gerrit:159089 Enable $wgContentHandlerUseDB on mediawikiwiki, testwiki, & test2wiki (duration: 00m 05s) [23:03:02] Logged the message, Master [23:04:28] (03CR) 10Ori.livneh: [C: 032] "Other changes: https://gerrit.wikimedia.org/r/#/c/158407/ , https://gerrit.wikimedia.org/r/#/c/157485/" [puppet] - 10https://gerrit.wikimedia.org/r/158317 (owner: 10Ori.livneh) [23:12:28] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 04 Sep 2014 00:21:29 UTC [23:13:23] who owns zero portal? that are 'new commits' in tin:/a/common/php-1.24wmf20/extensions/ZeroPortal [23:13:42] try yurikR [23:13:56] yurikR: around? [23:13:58] yep [23:14:19] yurikR: are "new commits" expected in tin:/a/common/php-1.24wmf20/extnesions/ZeroPortal? about to scap out SWAT changes [23:14:32] it hasn't been synced? [23:14:46] zp should be identical in 19&20 [23:14:52] it may or may not have been synced, new commits means the submodule hash doesn't match the hash in the core repository [23:14:55] with possible i18n exceptions [23:15:06] and yes it looks like wmf19 is same [23:15:16] than should be fine [23:15:18] it might just be that you guys didn't submit a submodule update to mediawiki/core? ok [23:15:32] i think RoanKattouw_away was swating it last [23:15:37] but should be fine [23:15:51] !log ebernhardson Started scap: SWAT deploy updates to Flow, Echo and Thanks [23:15:54] both 19&20 can be on master [23:15:55] Logged the message, Master [23:16:45] yurikR: ok awsome, thanks! [23:17:26] np, any time [23:21:03] ebernhardson: I already uploaded the submodule bumps [23:21:05] oh, you found them :) [23:31:09] PROBLEM - puppet last run on ssl1009 is CRITICAL: CRITICAL: Puppet has 1 failures [23:31:38] !log scap failed to connect to mw1070. Repeated message: rsync: failed to connect to mw1070.eqiad.wmnet (10.64.16.50): Connection refused (111) [23:31:42] Logged the message, Master [23:32:34] mutante: ok back to right channnel :) scap is at 80% and closing in on the finish. What should i do about 1070? [23:33:58] ebernhardson: since i see it's enabled, i'll take it out of the loadbalancer rotation [23:34:54] !log disabled mw1070 in pybal because it refused sync [23:35:00] Logged the message, Master [23:35:19] looks if there is some hardware ticket for it [23:36:27] ofc stalled out at 99% with left: 1 :) [23:36:52] can you skip it? [23:37:02] it used to be hitting enter [23:37:13] a couple times :p [23:37:14] doesn't look like it, hitting enter hasn't done anything [23:37:25] hit it a good 10 times :) [23:38:04] does it really get stuck by this stuff? i expect it must timeout [23:38:11] unless there was some deployment change [23:38:43] afaik it doesn't usually, but its just siting at: scap-rebuild-cdbs: 99% (ok: 228; fail: 0; left: 1) [23:39:15] ah, rebuilding cdb's , that's interwiki, isnt it [23:39:30] and localisation cache [23:39:41] ori: how do you just skip one? [23:39:47] hmm, actually something else might have happened. Scrolling up at the end of the many many repeated errors about mw1070i have this: [23:39:51] !log ebernhardson Finished scap: SWAT deploy updates to Flow, Echo and Thanks (duration: 24m 00s) [23:39:54] sync-common: 100% (ok: 0; fail: 229; left: 0) [23:39:56] heh [23:39:56] :) [23:39:57] 23:27:27 229 apaches had sync errors [23:39:57] Logged the message, Master [23:40:11] i think this happened before [23:40:18] i asked how to skip, and then it finished,hehe [23:40:23] ebernhardson: try running it again [23:40:44] !log ebernhardson Started scap: Repeat SWAT scap deployment due to possible sync-common failure [23:40:48] Logged the message, Master [23:41:03] ori: ok [23:42:01] there used to be ways to just sync a single one [23:42:22] mutante: there is, but one swat patch has a pair of new i18n messages [23:42:43] mutante: (and i18n requires full scap) [23:44:27] i see [23:46:00] :( [23:49:18] RECOVERY - puppet last run on ssl1009 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [23:53:07] (03PS1) 10Krinkle: contint: Ensure nodejs-legacy is installed [puppet] - 10https://gerrit.wikimedia.org/r/159226 [23:54:18] ori, mutante: Its stalled out at 'sync-proxies: 75% (ok: 3; fail: 0; left: 1)' for the last ~7 minutes. I notice /etc/dsh/groups/scap-proxies contains mw1070, could it be stalled due to disable from pybal mutante did above? [23:54:42] i would think it should time out anyways though [23:56:07] is mw1070 a scap proxy? [23:56:27] ori: i dunno, but it s listed in /etc/dsh/groups/scap-proxies, which i think is how scap gets its list? [23:56:33] i'm not super familiar with how it works internally [23:56:34] yeah [23:56:51] let's see what it's doing [23:56:53] rsyncd is running there but refused connection earlier [23:57:48] PROBLEM - puppet last run on cp1040 is CRITICAL: CRITICAL: Puppet has 1 failures [23:58:51] ori: ori ok it finally went to 100% (ok: 4; fail: 0; left: 0) [23:58:53] heh, you restarted rsync ? try again [23:58:55] hah [23:59:13] want me to enable it again? [23:59:27] it'd be good to know what happened, exactly :/ [23:59:28] only reason i did that was to avoid service from a server that isn't in sync [23:59:39] serving [23:59:53] !log restarted rsync on mw1070 to unblock scap [23:59:59] Logged the message, Master