[00:00:04] RoanKattouw ostriches Krenair: Respected human, time to deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20151208T0000). Please do the needful. [00:00:06] RoanKattouw aude bblack Krinkle jdlrobson: A patch you scheduled for Evening SWAT (Max 8 patches) is about to be deployed. Please be available during the process. [00:00:59] * aude here [00:01:03] \o [00:02:12] (03CR) 10Catrope: [C: 032] Enable Flow user opt-in Beta Feature on two more wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/251523 (https://phabricator.wikimedia.org/T117991) (owner: 10Jforrester) [00:03:24] (Roan is taking SWAT.) [00:04:06] (03Merged) 10jenkins-bot: Enable Flow user opt-in Beta Feature on two more wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/251523 (https://phabricator.wikimedia.org/T117991) (owner: 10Jforrester) [00:07:23] (03PS4) 10BryanDavis: Enable Cards and RelatedArticles so it rides the train [mediawiki-config] - 10https://gerrit.wikimedia.org/r/257434 (https://phabricator.wikimedia.org/T116676) (owner: 10Jdlrobson) [00:07:31] !log catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow opt-in beta feature on bswiki and urwiki (duration: 00m 28s) [00:07:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:10:02] 6operations, 6WMF-NDA-Requests: Give Neil Quinn access to WMF-NDA - https://phabricator.wikimedia.org/T119122#1860865 (10Neil_P._Quinn_WMF) @Aklapper, no, I haven't. I assume I do that by adding the project? [00:11:47] Config change done and scripts run [00:11:56] Apparently we had one prior opt-in on bswiki and two on urwiki [00:12:15] RoanKattouw: shoot i have one more config change that is supposed to be in swat.. [00:12:20] Go for it [00:12:34] As in, put it on the wiki page and ping me when done [00:12:34] 6operations, 6WMF-NDA-Requests: Give Neil Quinn access to WMF-NDA - https://phabricator.wikimedia.org/T119122#1860883 (10Dzahn) >>! In T119122#1860326, @Aklapper wrote: > Have you contacted Operations? No specific reason for Operations here. It's about having the permissiosn in phabricator. [00:14:16] (03PS1) 10Jdlrobson: Disable beta optin experiment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/257503 (https://phabricator.wikimedia.org/T114038) [00:14:35] Oh RoanKattouw actually ignore me. [00:14:52] Seems like we might have disabled it - so we may be okay. [00:16:35] RoanKattouw: we'll stick to the plan :) [00:17:08] (03Abandoned) 10Jdlrobson: Disable beta optin experiment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/257503 (https://phabricator.wikimedia.org/T114038) (owner: 10Jdlrobson) [00:18:30] RECOVERY - Outgoing network saturation on labstore1003 is OK: OK: Less than 10.00% above the threshold [75000000.0] [00:19:52] 6operations, 6WMF-NDA-Requests: Give Neil Quinn access to WMF-NDA - https://phabricator.wikimedia.org/T119122#1860957 (10Dzahn) That said, the process above is for volunteers. I'm not sure we have anything for employees. [00:22:38] (03CR) 10Jdlrobson: [C: 031] Enable Cards and RelatedArticles so it rides the train [mediawiki-config] - 10https://gerrit.wikimedia.org/r/257434 (https://phabricator.wikimedia.org/T116676) (owner: 10Jdlrobson) [00:24:20] RoanKattouw: https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20151208T0000 [00:25:13] (03CR) 10Catrope: [C: 032] Enable data access for beta meta-wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/256707 (owner: 10Aude) [00:25:54] (03Merged) 10jenkins-bot: Enable data access for beta meta-wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/256707 (owner: 10Aude) [00:26:14] (03CR) 10Catrope: [C: 032] Revert "wgHTCPRouting: use separate address for upload" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/256730 (owner: 10BBlack) [00:26:55] (03Merged) 10jenkins-bot: Revert "wgHTCPRouting: use separate address for upload" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/256730 (owner: 10BBlack) [00:27:31] RoanKattouw: o/ [00:27:34] (03CR) 10Catrope: [C: 032] Remove Browse experimental config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252660 (https://phabricator.wikimedia.org/T113686) (owner: 10Phuedx) [00:27:45] Hey Krinkle [00:28:18] (03Merged) 10jenkins-bot: Remove Browse experimental config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252660 (https://phabricator.wikimedia.org/T113686) (owner: 10Phuedx) [00:29:41] (03CR) 10Catrope: [C: 032] Enable banners extension on mobile web beta only (enwiki) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/251544 (https://phabricator.wikimedia.org/T101108) (owner: 10Jdlrobson) [00:30:03] 6operations, 10DBA, 6WMF-Legal, 7HTTPS, 5Patch-For-Review: dbtree loads third party resources (from jquery.com and google.com) - https://phabricator.wikimedia.org/T96499#1861019 (10Dzahn) [00:30:21] (03Merged) 10jenkins-bot: Enable banners extension on mobile web beta only (enwiki) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/251544 (https://phabricator.wikimedia.org/T101108) (owner: 10Jdlrobson) [00:31:48] !log catrope@tin Synchronized wmf-config/squid.php: SWAT: stop using separate address for upload in wgHTCPRouting (duration: 00m 29s) [00:31:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:34:36] !log catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: enable banners on mobile web beta on enwiki; remove MobileFrontend Browse config (duration: 00m 27s) [00:34:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:37:01] !log catrope@tin Synchronized php-1.27.0-wmf.7/resources/ResourcesOOUI.php: SWAT: connect OOjs UI to MW's l10n system (duration: 00m 27s) [00:37:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:37:44] !log catrope@tin Synchronized php-1.27.0-wmf.7/resources/src/oojs-ui-local.js: SWAT: connect OOjs UI to MW's l10n system (duration: 00m 28s) [00:37:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:38:22] !log catrope@tin Synchronized php-1.27.0-wmf.7/extensions/MwEmbedSupport/: SWAT (duration: 00m 28s) [00:38:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:39:12] !log catrope@tin Synchronized php-1.27.0-wmf.7/extensions/MobileFrontend/: SWAT (duration: 00m 28s) [00:39:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:40:56] Krinkle jdlrobson bblack aude: That's SWAT all done [00:41:06] RoanKattouw: okay testing. Thanks! [00:41:23] RoanKattouw: thanks [00:42:02] (03PS2) 10GWicke: Update RESTBase configs for RESTBase v0.9.1 [puppet] - 10https://gerrit.wikimedia.org/r/257408 [00:42:04] it will take a few minutes for the changes to appear on beta, but confirm we didn't accidentally enable our stuff on the real meta-wiki :) [00:42:15] RoanKattouw: confirmed. fixed. [00:48:55] 6operations, 10Wikimedia-Mailing-lists, 6Wiktionary: wiktionary-l: assign new moderators - https://phabricator.wikimedia.org/T110969#1591088 (10Dzahn) [00:52:40] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: puppet fail [00:54:04] 6operations: Check incoming requests to secure.wm.o - https://phabricator.wikimedia.org/T119274#1861105 (10Reedy) This was filed for T93531 [00:59:05] !log disabling puppet in production restbase cluster in preparation for testing https://gerrit.wikimedia.org/r/#/c/257408/ in staging [00:59:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:59:49] PROBLEM - Kafka Broker Replica Max Lag on kafka1022 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [5000000.0] [01:00:07] 6operations: Remove secure.wikimedia.org - https://phabricator.wikimedia.org/T120790#1861112 (10Reedy) 3NEW [01:00:23] 6operations: Remove secure.wikimedia.org - https://phabricator.wikimedia.org/T120790#1861119 (10Reedy) [01:00:25] 6operations: Check incoming requests to secure.wm.o - https://phabricator.wikimedia.org/T119274#1861120 (10Reedy) [01:00:25] (03PS1) 10Ori.livneh: keystone: migrate to redis::instance [puppet] - 10https://gerrit.wikimedia.org/r/257509 [01:00:49] PROBLEM - Kafka Broker Replica Max Lag on kafka1020 is CRITICAL: CRITICAL: 71.43% of data above the critical threshold [5000000.0] [01:00:51] 6operations: Remove secure.wikimedia.org - https://phabricator.wikimedia.org/T120790#1861112 (10Reedy) [01:00:53] 6operations, 10SEO: secure.wikimedia.org entries still showing up in Google search results - https://phabricator.wikimedia.org/T93531#1861128 (10Reedy) [01:01:21] (03CR) 10jenkins-bot: [V: 04-1] keystone: migrate to redis::instance [puppet] - 10https://gerrit.wikimedia.org/r/257509 (owner: 10Ori.livneh) [01:02:18] (03CR) 10GWicke: "I have now disabled puppet on all restbase nodes, so that we can test this in staging without affecting production." [puppet] - 10https://gerrit.wikimedia.org/r/257408 (owner: 10GWicke) [01:03:02] (03PS1) 10Reedy: Remove secure.wikimedia.org from apacheo [puppet] - 10https://gerrit.wikimedia.org/r/257510 (https://phabricator.wikimedia.org/T120790) [01:03:30] RoanKattouw: i noticed a follow up needed with one of the patches - guessing i'll have to wait till tomorrows swat window for that right? [01:04:21] (03CR) 10Ori.livneh: [C: 032] "My understanding from chatting with Gabriel is that this patch is a dependency for a VisualEditor feature which is slotted for deployment " [puppet] - 10https://gerrit.wikimedia.org/r/257408 (owner: 10GWicke) [01:04:43] (03PS1) 10Jdlrobson: Minerva should also be blacklisted [mediawiki-config] - 10https://gerrit.wikimedia.org/r/257511 (https://phabricator.wikimedia.org/T101108) [01:04:56] (03CR) 10Andrew Bogott: "We used redis replication when we migrated from pmtpa to eqiad -- now that Labs is single datacenter this feature isn't used (and the upst" [puppet] - 10https://gerrit.wikimedia.org/r/257509 (owner: 10Ori.livneh) [01:05:52] (03CR) 10Ori.livneh: "Andrew, gotcha. Would you prefer that I decom the redis instance altogether?" [puppet] - 10https://gerrit.wikimedia.org/r/257509 (owner: 10Ori.livneh) [01:05:55] jdlrobson: https://gerrit.wikimedia.org/r/#/c/257511/ is trivial enough that I'll deploy it right now if you want [01:06:07] if you could that would save me an early start tomorrow :) [01:06:14] and i would me much appreciative :) [01:07:14] (03PS4) 10Ori.livneh: gerrit: Map /tools/hooks/commit-msg to /r/tools/hooks/commit-msg [puppet] - 10https://gerrit.wikimedia.org/r/257396 (owner: 10Bartosz Dziewoński) [01:07:20] (03CR) 10Ori.livneh: [C: 032 V: 032] gerrit: Map /tools/hooks/commit-msg to /r/tools/hooks/commit-msg [puppet] - 10https://gerrit.wikimedia.org/r/257396 (owner: 10Bartosz Dziewoński) [01:07:47] (03CR) 10Andrew Bogott: "Yes, probably, so that that code doesn't remain in wait to trap another hopeful developer :)" [puppet] - 10https://gerrit.wikimedia.org/r/257509 (owner: 10Ori.livneh) [01:08:17] (03PS2) 10Reedy: Remove secure.wikimedia.org from apacheo [puppet] - 10https://gerrit.wikimedia.org/r/257510 (https://phabricator.wikimedia.org/T120790) [01:08:35] RoanKattouw: ^ [01:09:07] (03CR) 10Chad: [C: 04-1] "My bookmarks in my browser will break! :(" [puppet] - 10https://gerrit.wikimedia.org/r/257510 (https://phabricator.wikimedia.org/T120790) (owner: 10Reedy) [01:09:18] * Reedy kicks ostriches [01:09:51] PROBLEM - Kafka Broker Replica Max Lag on kafka1022 is CRITICAL: CRITICAL: 83.33% of data above the critical threshold [5000000.0] [01:09:59] (03CR) 10Catrope: [C: 032] Minerva should also be blacklisted [mediawiki-config] - 10https://gerrit.wikimedia.org/r/257511 (https://phabricator.wikimedia.org/T101108) (owner: 10Jdlrobson) [01:10:18] (03PS1) 10Reedy: Remove secure.wikimedia.org from DNS [dns] - 10https://gerrit.wikimedia.org/r/257513 (https://phabricator.wikimedia.org/T120790) [01:10:25] Reedy: You shouldn't kick ostriches. They're fast and very much taller than you! [01:10:27] (03Merged) 10jenkins-bot: Minerva should also be blacklisted [mediawiki-config] - 10https://gerrit.wikimedia.org/r/257511 (https://phabricator.wikimedia.org/T101108) (owner: 10Jdlrobson) [01:10:50] * Reedy wonders what apacheo is [01:10:57] Also a good question! [01:11:16] (03CR) 10Jforrester: "Cool URLs don't change… but secure.wm.org was never cool" [puppet] - 10https://gerrit.wikimedia.org/r/257510 (https://phabricator.wikimedia.org/T120790) (owner: 10Reedy) [01:11:28] (03PS2) 10Ori.livneh: role::labs::openstack::keystone::server: decom redis instance [puppet] - 10https://gerrit.wikimedia.org/r/257509 [01:11:34] andrewbogott: ^ [01:11:47] James_F: But if you were never cool...how do you know what cool was?! [01:11:54] secure.wm.o could've been too cool for you! [01:11:54] :) [01:12:07] ostriches: I worked for the author of the essay. Does that make me cool? [01:12:17] James_F: For the record, I never got invited to those sorts of parties growing up either :( [01:12:17] If it's good enough for a US Defence Contractor... [01:12:22] * James_F grins. [01:13:02] (03PS3) 10Reedy: Remove secure.wikimedia.org from apache [puppet] - 10https://gerrit.wikimedia.org/r/257510 (https://phabricator.wikimedia.org/T120790) [01:13:51] PROBLEM - Kafka Broker Replica Max Lag on kafka1022 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [5000000.0] [01:14:32] (03CR) 10Andrew Bogott: [C: 031] "thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/257509 (owner: 10Ori.livneh) [01:15:12] Thanks RoanKattouw - should this patch be live now? [01:15:49] Oh sorry [01:15:50] PROBLEM - Restbase endpoints health on xenon is CRITICAL: Generic error: Generic connection error: HTTPConnectionPool(host=127.0.0.1, port=7231): Max retries exceeded with url: /en.wikipedia.org/v1/?spec (Caused by ProtocolError(Connection aborted., error(111, Connection refused))) [01:15:54] I was waiting for it to merge without noticing it already had [01:15:56] Deploying now [01:16:21] PROBLEM - Restbase root url on xenon is CRITICAL: Connection refused [01:16:35] this is me testing ^^ [01:16:47] !log catrope@tin Synchronized wmf-config/InitialiseSettings.php: Also blacklist Minerva for WPB (duration: 00m 28s) [01:16:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [01:17:05] jdlrobson: Done [01:17:08] thanks RoanKattouw ! [01:17:13] andrewbogott: am finishing it up, https://etherpad.wikimedia.org/p/labs-incident-timeline [01:17:20] andrewbogott: finding exact times for the last bits [01:17:25] andrewbogott: but that's the ordering [01:17:44] ACKNOWLEDGEMENT - Restbase endpoints health on xenon is CRITICAL: Generic error: Generic connection error: HTTPConnectionPool(host=127.0.0.1, port=7231): Max retries exceeded with url: /en.wikipedia.org/v1/?spec (Caused by ProtocolError(Connection aborted., error(111, Connection refused))) gwicke Config testing. [01:17:44] ACKNOWLEDGEMENT - Restbase root url on xenon is CRITICAL: Connection refused gwicke Config testing. [01:17:54] YuviPanda: thanks, will look [01:18:41] RECOVERY - Kafka Broker Replica Max Lag on kafka1020 is OK: OK: Less than 1.00% above the threshold [1000000.0] [01:19:39] (03PS1) 10Ori.livneh: Fix-up for I81452131b: rewrite into absolute URL [puppet] - 10https://gerrit.wikimedia.org/r/257515 [01:19:52] (03CR) 10Ori.livneh: [C: 032 V: 032] Fix-up for I81452131b: rewrite into absolute URL [puppet] - 10https://gerrit.wikimedia.org/r/257515 (owner: 10Ori.livneh) [01:23:02] andrewbogott: don't have exact timelnies for the three restarts but put in close enough numbers [01:23:13] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:23:13] 6operations, 5Patch-For-Review: Remove secure.wikimedia.org - https://phabricator.wikimedia.org/T120790#1861150 (10Legoktm) Uhh, lets not break links please? :/ [01:23:15] RECOVERY - Kafka Broker Replica Max Lag on kafka1022 is OK: OK: Less than 1.00% above the threshold [1000000.0] [01:23:16] (03PS3) 10Ori.livneh: role::labs::openstack::keystone::server: decom redis instance [puppet] - 10https://gerrit.wikimedia.org/r/257509 [01:23:18] (03CR) 10Ori.livneh: [C: 032 V: 032] "catalog compiler: https://puppet-compiler.wmflabs.org/1456/" [puppet] - 10https://gerrit.wikimedia.org/r/257509 (owner: 10Ori.livneh) [01:24:04] thanks [01:25:13] 6operations, 5Patch-For-Review: Remove secure.wikimedia.org - https://phabricator.wikimedia.org/T120790#1861152 (10Reedy) What links? Where? See T119274 for amount of incoming links [01:32:43] (03PS1) 10Ori.livneh: ircyall: migrate to redis::instance [puppet] - 10https://gerrit.wikimedia.org/r/257517 [01:33:33] YuviPanda: earlier today (19:30 ish UTC) I definitely verified that new instances still weren’t working, due to failure to communicate with designate [01:33:49] do you know for a fact that new instances were working correctly at any point between ‘everything back to normal’ and then? [01:34:18] andrewbogott: no, I had looked at instances in tools project and in contincloud that were in 'scheduling' that became active [01:34:20] and I logged into one [01:34:23] and i was like ok' [01:34:27] (03CR) 10Ori.livneh: [C: 032] ircyall: migrate to redis::instance [puppet] - 10https://gerrit.wikimedia.org/r/257517 (owner: 10Ori.livneh) [01:35:11] hmmmmm [01:40:41] PROBLEM - puppet last run on db2052 is CRITICAL: CRITICAL: Puppet has 1 failures [01:41:03] YuviPanda: so, the timing is so close… it seems like the puppet run at 10:25 /must/ have caused the dns outage. [01:41:13] Do you know how long puppet had been broken before then? [01:42:19] andrewbogott: no, but I think I saw it broke about the time gabriel reported the breakage... [01:42:55] When was that? [01:43:13] yesterday around noon SF time [01:43:16] There were probably alerts, surely there’s an icinga log of things someplace [01:43:20] 22:55 - https://phabricator.wikimedia.org/T120586 gets filed [01:43:24] so the previous day [01:43:51] ok... [01:44:19] I must’ve been bit by a radioactive ellipsis a few weeks ago [01:45:59] (03PS1) 10Ori.livneh: wikimetrics: migrate to redis::instance [puppet] - 10https://gerrit.wikimedia.org/r/257521 [01:48:43] (03PS1) 10Chad: Gerrit: redirect old gitweb project urls to Diffusion instead of Gitblit [puppet] - 10https://gerrit.wikimedia.org/r/257523 [01:49:17] (03CR) 10Ori.livneh: "> is it in the cards to keep using it after the migration is done? -- nope!" [debs/bloomd] - 10https://gerrit.wikimedia.org/r/257167 (https://phabricator.wikimedia.org/T120544) (owner: 10Ori.livneh) [01:50:06] (03CR) 10Ori.livneh: [C: 032 V: 032] Imported Upstream version 0.7.4 [debs/bloomd] - 10https://gerrit.wikimedia.org/r/257167 (https://phabricator.wikimedia.org/T120544) (owner: 10Ori.livneh) [01:56:41] btw, YuviPanda just because reverse dns is dumb for that IP I’d still expect everything to work [01:56:49] you can access that system and such right? [01:57:14] andrewbogott: no [01:57:36] andrewbogott: cna't ssh [01:57:37] is it getting the wrong ip? [01:57:38] *can't [01:58:28] andrewbogott: dig is giving me an IP, let me verify it with wikitech [01:59:08] andrewbogott: um, the IP is correct [01:59:12] andrewbogott: but instance isn't reachable [01:59:36] andrewbogott: I wonder if both of those instances have the same IP [01:59:54] andrewbogott: oh, hmm, only one of the instances is inaccessible (tools-worker-05) [01:59:57] another one is accessible... [02:00:00] wat [02:00:22] console access for tools-worker-05 says [02:00:24] tools-worker-05 login: [02:00:41] except I can't ssh in [02:00:43] * YuviPanda is very nervous [02:04:11] YuviPanda: root@labservices1001:/home/andrew/latestdesignate.txt has a dump of everything designate should know about (forward dns, not reverse) [02:04:52] latestdesignatereverse.txt has the reverse entries [02:05:35] RECOVERY - puppet last run on db2052 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [02:08:34] andrewbogott: hmm so there are two records there [02:08:37] andrewbogott: with the same IP [02:08:53] andrewbogott: well, *4* records [02:09:01] ok, for starters, I think this is a bug but not the cause of the overlap: https://phabricator.wikimedia.org/T120792 [02:09:10] There should be two... [02:09:15] but, yes, 4 is bad [02:09:18] so I can delete the bad ones by hand [02:09:35] PROBLEM - puppet last run on mw2104 is CRITICAL: CRITICAL: puppet fail [02:09:37] I haven’t looked at the dates yet — which came first? [02:11:15] YuviPanda: ok, so the CI dns entry leaked [02:11:28] then tools reused the IP, duplicating things [02:11:32] that doesn’t upset/shock me a ton [02:11:44] since CI names are monotonic, I’ll just purge old ones [02:12:29] ok [02:17:42] (03CR) 10Ori.livneh: [C: 032] "Won't apply immediately because wikimetrics is using a self-hosted puppetmaster setup, but when it is merged the configured redis instance" [puppet] - 10https://gerrit.wikimedia.org/r/257521 (owner: 10Ori.livneh) [02:18:52] (03PS1) 10Ori.livneh: Fixup for I6fe4b05b922 [puppet] - 10https://gerrit.wikimedia.org/r/257529 [02:19:05] (03CR) 10Ori.livneh: [C: 032 V: 032] Fixup for I6fe4b05b922 [puppet] - 10https://gerrit.wikimedia.org/r/257529 (owner: 10Ori.livneh) [02:21:35] ori: don't do the tools instance, needs announcements [02:21:45] YuviPanda: yes of course not [02:21:59] i thought of helping with the patch tho [02:22:04] ori: +1 [02:22:08] ori: that'd be lovely :D [02:22:08] where is it actually running? [02:22:12] * YuviPanda is still in several roles [02:22:17] ori: tools-redis-01 and -02 [02:22:21] one is master one is slave [02:22:25] * ori nods [02:22:44] ori: nobody is hitting the slave, so you can practice with that if need be [02:23:15] nah, let's coordinate tomorrow. I asked because I have a good workflow now for diffing the config file against the default config that ships with the package [02:23:27] ori: ah, cool [02:23:45] ori: btw, I thought I should show you https://phabricator.wikimedia.org/T120697 [02:24:08] (I don't expect it to get deployed anytime soon, but will end up building it anyway :D) [02:24:15] yeah, i've suggested that before [02:24:23] good idea [02:24:34] once all the current stuff gets stabilized [02:24:46] ori: also, https://github.com/oreillymedia/thebe [02:25:18] !log mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 09m 56s) [02:25:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:26:02] ori: I suppose I can easily make it so that it can render it either by shelling out or by calling a service [02:26:13] * YuviPanda shall make this a fun side project at some point [02:26:30] I suspect I should show this to stephen at some point too [02:27:26] https://gist.github.com/atdt/4a334fd1716a8db111ce [02:28:16] RECOVERY - Restbase root url on xenon is OK: HTTP OK: HTTP/1.1 200 - 15171 bytes in 0.028 second response time [02:28:42] ori: is that the diff? [02:28:47] yep [02:29:10] yeah, so maxmemory and the renames are important [02:29:17] and so is maxmemory-policy [02:30:05] RECOVERY - Restbase endpoints health on xenon is OK: All endpoints are healthy [02:32:51] andrewbogott: so... should it all be ok now? [02:33:10] I still can't reach tools-worker-05 [02:33:16] Deleting these records turns out to be incredibly hard [02:34:03] ouch [02:34:08] ok... [02:34:19] andrewbogott: the instance is expendable if needed, fwiw. [02:34:23] both of them [02:34:26] designate api v1 won’t delete ‘managed’ records [02:34:33] and v2 will except it doesn’t do anything, just returns success [02:34:40] fun [02:34:47] nah, that IP will just stay there waiting to bite us if we don’t clean these up [02:34:49] *thos [02:34:50] *those [02:34:57] ouch [02:34:59] ok [02:35:31] Hey YuviPanda, do you have a howto move a bot into a container? I'd move csbot and use the opportunity to note if anything is missing or unclear in the doc? [02:35:45] RECOVERY - puppet last run on mw2104 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [02:35:54] Coren: no, I think we shouldn't move more bots for a few more weeks. [02:36:09] Coren: PAWS is putting real users into the system, showing up bugs that I'm fixing. [02:36:29] (03PS1) 10Ori.livneh: toollabs: migrate to redis::instance [puppet] - 10https://gerrit.wikimedia.org/r/257534 [02:36:33] Coren: I'll write up more docs once PAWS is all good [02:36:38] YuviPanda: ^ (for later, don't worry about it now) [02:36:56] Hm, okay. Csbot makes a good perl test case if nothing else. [02:36:58] thanks ori <3 [02:37:08] \o [02:37:23] Coren: yeah. http://imgur.com/gallery/Yodcq [02:38:05] Coren: we'll probably end up just using debian inside the containers [02:38:05] (03CR) 10jenkins-bot: [V: 04-1] toollabs: migrate to redis::instance [puppet] - 10https://gerrit.wikimedia.org/r/257534 (owner: 10Ori.livneh) [02:39:00] YuviPanda: A little more heavy, but more general and flexible. Only this or also one or two specialized ones like pywikibot? [02:39:04] (03PS2) 10Ori.livneh: toollabs: migrate to redis::instance [puppet] - 10https://gerrit.wikimedia.org/r/257534 [02:39:29] Coren: so I think we'll end up with a 'base' debian container [02:39:35] Coren: and all other ones will be based off it [02:39:42] YuviPanda: Sounds sane. [02:39:52] Coren: the current paws one is based off jessie as well [02:40:12] YuviPanda: Wave in my direction when you want me to throw csbot at it. It's very self-contained so a pretty good test. [02:40:44] Coren: are you able to test it locally? [02:41:01] Coren: also does it run continuously or on a cron? [02:41:46] YuviPanda: It's continuous, and I see no reason I couldn't test it locally - it was developed that way and ran on my own colo box for a long time. [02:42:04] Coren: ok, so I'd suggest putting it into a Docker container and seeing how it does locally [02:42:19] * Coren nods. [02:42:23] Coren: then I can review it and see how to adapt it, and you can also get some experience writing Dockerfiiles [02:42:31] Sounds good! [02:42:33] Coren: once a Docker container is ready, it's about 5 mins to actually deploy it [02:43:01] Coren: does it need DB access? [02:43:15] YuviPanda: Nope; it does API only. [02:43:39] Coren: awesome. so play around with a dockerfile (inherit from debian:jessie!) and get it to run locally? [02:43:47] kk! [02:44:11] Coren: https://github.com/yuvipanda/paws/blob/master/singleuser/Dockerfile is the PAWS pywikibot one, if that's of any help [02:44:21] Coren: poke me if you run into anything! happy/excited to help [02:44:36] It probably will. Are we planning to have a method by which custom debs can be added though? [02:44:36] PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0] [02:45:18] Obviously I can do so locally. [02:45:26] Coren: custom debs as in? [02:45:30] custom built debs? [02:45:42] or just debs you want to apt-get install? [02:45:44] PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0] [02:45:48] Yeah; my bot needs a perl module that requires compilation. [02:46:19] Coren: yeah, so you can just do RUN apt-get install --yes [02:46:34] Coren: PAWS too requires python modules that require compilation [02:46:48] Ah, so it's a good example for that too. [02:46:52] Coren: yup! [02:47:26] PROBLEM - Mobile HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0] [02:47:57] Goodie. That's my volunteer hat, but I'll be doing this during downtime. [02:48:03] Coren: <3 cool! [02:50:37] RECOVERY - Esams HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [02:53:25] RECOVERY - Mobile HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [02:53:35] RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [03:04:10] (03PS1) 10Chad: gitblit: use mw/vendor as the icinga check instead of mw/core [puppet] - 10https://gerrit.wikimedia.org/r/257538 [03:04:38] ori: That'll probably actually make a difference so we don't have to hit mw/core every minute-ish. [03:12:26] !log Keeping puppet on restbase production cluster disabled until Marko & Filippo can deploy tomorrow morning. PLEASE DO NOT ENABLE! [03:12:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [04:00:03] !log l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec 8 04:00:02 UTC 2015 (duration 1h 34m 44s) [04:00:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [04:25:42] (03CR) 10GWicke: "Updated Filippo, Mark & James per mail. Staging testing has gone well. Puppet is still disabled in preparation for a deploy during Europea" [puppet] - 10https://gerrit.wikimedia.org/r/257408 (owner: 10GWicke) [04:52:56] PROBLEM - cassandra CQL 10.64.32.178:9042 on restbase1008 is CRITICAL: Connection refused [04:55:36] ACKNOWLEDGEMENT - cassandra CQL 10.64.32.178:9042 on restbase1008 is CRITICAL: Connection refused gwicke Finished decommission in preparation for a conversion to a multi-instance setup. [05:12:25] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: Puppet has 1 failures [05:14:45] PROBLEM - Kafka Broker Replica Max Lag on kafka1020 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [5000000.0] [05:16:46] PROBLEM - puppet last run on es2003 is CRITICAL: CRITICAL: puppet fail [05:26:34] RECOVERY - Kafka Broker Replica Max Lag on kafka1020 is OK: OK: Less than 1.00% above the threshold [1000000.0] [05:36:06] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [05:42:34] RECOVERY - puppet last run on es2003 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [05:45:56] !log restbase1008: disable cassandra with `systemctl mask cassandra` [05:46:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [05:49:12] gwicke: I didn't know about 'systemctl mask'. That's handy. [05:50:05] PROBLEM - cassandra service on restbase1008 is CRITICAL: CRITICAL - Expecting active but unit cassandra is failed [05:50:20] yeah, that's the "do-as-I-say" variant [05:50:49] "puppet-I-really-don't-want-you-to-start-this" [05:52:26] Heh, yes. [05:53:29] downside is that it breaks puppet [05:54:17] or at least makes puppet's brokenness more apparent [05:55:01] it is the world that is broken, puppet is correct a priori [05:55:48] heh, yeah [06:03:01] 6operations: Update to Cassandra 2.1.12 - https://phabricator.wikimedia.org/T120803#1861397 (10GWicke) 3NEW [06:03:37] 6operations, 10RESTBase: Update to Cassandra 2.1.12 - https://phabricator.wikimedia.org/T120803#1861404 (10GWicke) [06:30:26] PROBLEM - puppet last run on cp3048 is CRITICAL: CRITICAL: puppet fail [06:30:35] PROBLEM - puppet last run on mw2145 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:56] PROBLEM - puppet last run on mc2007 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:16] PROBLEM - puppet last run on db2055 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:25] PROBLEM - puppet last run on mw2023 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:25] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:14] PROBLEM - puppet last run on mw2126 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:15] PROBLEM - puppet last run on mw2018 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:15] PROBLEM - puppet last run on mw2045 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:15] PROBLEM - puppet last run on mw2016 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:26] PROBLEM - puppet last run on mw2158 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:34] PROBLEM - puppet last run on mw2207 is CRITICAL: CRITICAL: Puppet has 1 failures [06:56:35] RECOVERY - puppet last run on mc2007 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:56:54] RECOVERY - puppet last run on db2055 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:56:55] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:56:56] RECOVERY - puppet last run on mw2023 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:57:05] RECOVERY - puppet last run on mw2207 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:57:45] RECOVERY - puppet last run on mw2126 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:46] RECOVERY - puppet last run on mw2018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:46] RECOVERY - puppet last run on mw2045 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:57:46] RECOVERY - puppet last run on mw2016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:56] RECOVERY - puppet last run on mw2158 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:05] RECOVERY - puppet last run on cp3048 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:05] RECOVERY - puppet last run on mw2145 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:25] (03CR) 10Alexandros Kosiaris: [C: 032] gitblit: use mw/vendor as the icinga check instead of mw/core [puppet] - 10https://gerrit.wikimedia.org/r/257538 (owner: 10Chad) [07:42:07] (03CR) 10Alexandros Kosiaris: [C: 032] Update hieradata for trebuchet module [puppet] - 10https://gerrit.wikimedia.org/r/256261 (https://phabricator.wikimedia.org/T119988) (owner: 10Thcipriani) [07:42:13] (03PS2) 10Alexandros Kosiaris: Update hieradata for trebuchet module [puppet] - 10https://gerrit.wikimedia.org/r/256261 (https://phabricator.wikimedia.org/T119988) (owner: 10Thcipriani) [07:45:10] (03CR) 10Alexandros Kosiaris: [C: 032] AQS: Configure Cassandra for AQS in BetaCluster [puppet] - 10https://gerrit.wikimedia.org/r/257406 (https://phabricator.wikimedia.org/T116206) (owner: 10Mobrovac) [07:45:16] (03PS4) 10Alexandros Kosiaris: AQS: Configure Cassandra for AQS in BetaCluster [puppet] - 10https://gerrit.wikimedia.org/r/257406 (https://phabricator.wikimedia.org/T116206) (owner: 10Mobrovac) [08:03:57] 6operations, 10Wikimedia-Etherpad: Disable old Etherpad installation after migrating content to Etherpad Lite installion - https://phabricator.wikimedia.org/T47312#1861489 (10akosiaris) [08:46:31] 6operations, 10RESTBase: Update to Cassandra 2.1.12 - https://phabricator.wikimedia.org/T120803#1861535 (10GWicke) After testing 2.1.12 on cerium for a while I gradually proceeded to roll it out to the eqiad staging hosts, followed by restbase1007. That looked good after an hour, with significantly less compac... [09:11:27] mobrovac: I'm going to start with the restbase1008 reimage, unless you want to look at the rb deploy first? [09:15:13] !log reimage restbase1008 [09:15:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [09:19:54] kk godog [09:20:33] godog: i'd like to do the rb deploy after lunch [09:20:50] mobrovac: yup, works for me [09:20:55] cool [09:21:24] PROBLEM - Kafka Broker Replica Max Lag on kafka1012 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [5000000.0] [09:21:40] (03CR) 10Filippo Giunchedi: [C: 04-1] "thanks Ori, still -1 for debian/rules and $(SCONS) but looks good generally" [debs/bloomd] - 10https://gerrit.wikimedia.org/r/257168 (owner: 10Ori.livneh) [09:21:44] PROBLEM - Kafka Broker Replica Max Lag on kafka1022 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [5000000.0] [09:37:28] RECOVERY - Kafka Broker Replica Max Lag on kafka1012 is OK: OK: Less than 1.00% above the threshold [1000000.0] [09:40:29] (03PS1) 10ArielGlenn: puppetize dumps monitor as a service [puppet] - 10https://gerrit.wikimedia.org/r/257560 (https://phabricator.wikimedia.org/T110888) [09:42:18] RECOVERY - Kafka Broker Replica Max Lag on kafka1022 is OK: OK: Less than 1.00% above the threshold [1000000.0] [09:48:35] !log gallium: removing bunch of :i386 packages which were used for Android build but got removed ( https://gerrit.wikimedia.org/r/#/c/183790/3/modules/androidsdk/manifests/dependencies.pp,unified ) [09:48:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [09:53:07] (03PS1) 10Filippo Giunchedi: cassandra: add restbase1008-a instance [puppet] - 10https://gerrit.wikimedia.org/r/257562 [09:55:05] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] cassandra: add restbase1008-a instance [puppet] - 10https://gerrit.wikimedia.org/r/257562 (owner: 10Filippo Giunchedi) [09:56:13] !log gallium: upgraded python-diamond and java. Restarting Jenkins [09:56:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [10:04:36] !log start cassandra on restbase1008, bootstrapping [10:04:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [10:07:18] PROBLEM - Restbase endpoints health on restbase1008 is CRITICAL: Generic error: Generic connection error: HTTPConnectionPool(host=127.0.0.1, port=7231): Max retries exceeded with url: /en.wikipedia.org/v1/?spec (Caused by ProtocolError(Connection aborted., error(111, Connection refused))) [10:07:38] PROBLEM - Restbase root url on restbase1008 is CRITICAL: Connection refused [10:08:08] PROBLEM - cassandra-a CQL 10.64.32.187:9042 on restbase1008 is CRITICAL: Connection refused [10:08:37] ACKNOWLEDGEMENT - cassandra-a CQL 10.64.32.187:9042 on restbase1008 is CRITICAL: Connection refused Filippo Giunchedi bootstrapping [10:08:53] mobrovac: I've depooled 1008 from pybal, let me know when I can repool it [10:11:54] 6operations, 10RESTBase, 10RESTBase-Cassandra, 5Patch-For-Review: Test multiple Cassandra instances per hardware node - https://phabricator.wikimedia.org/T95253#1861667 (10fgiunchedi) restbase1008-a bootstrapping ``` Receiving 818 files, 410.89 GB total. Already received 23 files, 608.11 MB total... [10:15:31] (03PS1) 10Hashar: contint: monitor Jenkins has a ZMQ publisher [puppet] - 10https://gerrit.wikimedia.org/r/257568 (https://phabricator.wikimedia.org/T120669) [10:16:19] (03CR) 10Hashar: [C: 031 V: 031] "Low timeout cause it should always respond quite fast and there is no need to have nrpe idle for 10 seconds if zmq is dead." [puppet] - 10https://gerrit.wikimedia.org/r/257568 (https://phabricator.wikimedia.org/T120669) (owner: 10Hashar) [10:24:01] Maybe it could be useful to add Cenarium to the gerrit whitelist, he makes useful changes: https://gerrit.wikimedia.org/r/#/c/257559/ and is sysop at enwiki, so what do you think? [10:25:53] 6operations, 10RESTBase-Cassandra: Update to Cassandra 2.1.12 - https://phabricator.wikimedia.org/T120803#1861693 (10fgiunchedi) I don't see anything regarding this unscheduled upgrade in SAL, why is that? [10:56:23] 6operations, 6Analytics-Backlog, 6Discovery, 6WMDE-Analytics-Engineering, and 3 others: Add firewall exception to get to wdqs*:8888 from analytics cluster - https://phabricator.wikimedia.org/T120010#1861764 (10akosiaris) 5Open>3Resolved a:3akosiaris ACLs updated. Just tested it from `stat1003` and it... [11:02:54] !log upgrade diamond to 3.5-5 in codfw [11:02:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [11:06:10] (03CR) 10Addshore: [C: 04-1] "Per little chat with daniel this should be the size as recorded in the db, also what is displayed on LongPages." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/257318 (owner: 10JanZerebecki) [11:14:38] 6operations, 6Analytics-Backlog, 6Discovery, 6WMDE-Analytics-Engineering, and 3 others: Add firewall exception to get to wdqs*:8888 from analytics cluster - https://phabricator.wikimedia.org/T120010#1861805 (10Addshore) Many thanks! :) [11:39:59] 7Puppet, 6operations, 6Labs, 10Labs-Infrastructure: ldap-yaml-enc.py fails with host_info['puppetClass'] --> KeyError: 'puppetClass' - https://phabricator.wikimedia.org/T120817#1861849 (10hashar) 3NEW [12:04:48] PROBLEM - Kafka Broker Replica Max Lag on kafka1012 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [5000000.0] [12:18:38] RECOVERY - Kafka Broker Replica Max Lag on kafka1012 is OK: OK: Less than 1.00% above the threshold [1000000.0] [12:39:44] (03PS3) 10Alexandros Kosiaris: Update WikimediaTemplates to support 5.0.1 [software/otrs] - 10https://gerrit.wikimedia.org/r/248916 [12:39:46] (03PS1) 10Alexandros Kosiaris: Drop the WikimediaEnableMultilines packages [software/otrs] - 10https://gerrit.wikimedia.org/r/257587 [13:00:10] 6operations, 10Dumps-Generation, 10hardware-requests: determine hardware needs for dumps in eqiad (boxes out of warranty, capacity planning) - https://phabricator.wikimedia.org/T118154#1861962 (10mark) >>! In T118154#1792791, @ArielGlenn wrote: > Initial thoughts: we have dataset1001 with ms1001 as a fallbac... [13:02:28] 6operations, 10Traffic, 5Patch-For-Review, 7Pybal: pybal fails to detect dead servers under production lb IPs for port 80 - https://phabricator.wikimedia.org/T113151#1861965 (10mark) >>! In T113151#1858288, @Joe wrote: > So testing this with a single apache in a pool, I can see that AcceptFilter http none... [13:02:39] godog: i'd be ready to do a deploy of RB now [13:03:06] godog: what's the status of RB on rb1008? it's got the tin version of the code? [13:25:44] hi, i've got a renaming problem: [13:25:50] https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress/Archives_cantonales_jurassiennes [13:26:24] task was started Nov 26th - hasn't completed correctly [13:27:11] initial logged error was [90e25e28] 2015-11-26 09:54:29: Fataler Ausnahmefehler des Typs „JobQueueError“ [13:28:43] i can't restart it myself, can a database admin help? [13:30:11] (03PS2) 10coren: Add stable.toolserver.org to legacy redirects [puppet] - 10https://gerrit.wikimedia.org/r/257425 (https://phabricator.wikimedia.org/T120526) [13:30:31] MBq: that is in prod right ? [13:31:11] MBq: If noone is here, file a task at phabricator ;) (I don't know if a DB admin is here) [13:31:26] hahar: ? [13:31:49] akosiaris: utils/new_wmf_service.py is awesome. [13:31:51] hashar: ? [13:32:08] hashar: yeah it is [13:32:10]