[00:21:46] (03PS7) 10Smalyshev: Create wikidata ntriples dump from ttl dump [puppet] - 10https://gerrit.wikimedia.org/r/447922 (https://phabricator.wikimedia.org/T144103) [00:26:43] (03PS2) 10Dzahn: openstack::network: update private IP of relic.toolserver instance [puppet] - 10https://gerrit.wikimedia.org/r/455737 [00:28:59] (03PS3) 10Dzahn: openstack::network: update private IP of relic.toolserver instance [puppet] - 10https://gerrit.wikimedia.org/r/455737 (https://phabricator.wikimedia.org/T204564) [00:30:01] (03CR) 10Andrew Bogott: "Let's move this to eqiad1-r while we're at it? I'll check in with you tomorrow and do that if that's OK with you." [puppet] - 10https://gerrit.wikimedia.org/r/455737 (https://phabricator.wikimedia.org/T204564) (owner: 10Dzahn) [00:30:12] (03CR) 10Andrew Bogott: "(that will change the IP again, alas)" [puppet] - 10https://gerrit.wikimedia.org/r/455737 (https://phabricator.wikimedia.org/T204564) (owner: 10Dzahn) [00:31:04] (03CR) 10Dzahn: [C: 032] "checked exim and Apache config on new instance, looks good. checked DNS setup in Horizon, looks good and like this should do the needful. " [puppet] - 10https://gerrit.wikimedia.org/r/455737 (https://phabricator.wikimedia.org/T204564) (owner: 10Dzahn) [00:31:12] oh. i havent submitted yet [00:32:31] (03CR) 10Dzahn: [C: 031] "oh, re: Andrew. will do, just said to Paladox "lets do it separate" when he pointed it out. i didn't want to mix 2 things. i haven't submi" [puppet] - 10https://gerrit.wikimedia.org/r/455737 (https://phabricator.wikimedia.org/T204564) (owner: 10Dzahn) [00:33:37] (03CR) 10Dzahn: [C: 031] "ok, thanks, Andrew, waiting until tomorrow" [puppet] - 10https://gerrit.wikimedia.org/r/455737 (https://phabricator.wikimedia.org/T204564) (owner: 10Dzahn) [00:43:09] 10Operations, 10media-storage, 10User-fgiunchedi: Running swiftrepl is not puppetized - https://phabricator.wikimedia.org/T162123 (10Dzahn) Afaict we are not going to have to rely on swiftrepl anymore soon. Is that right? In that case, would this ticket be declined? [00:48:14] 10Operations: reinstall RT server with private IP and stretch - https://phabricator.wikimedia.org/T180641 (10Dzahn) p:05Low>03Normal [00:49:10] 10Operations, 10Domains, 10Traffic, 10WMF-Legal, 10Patch-For-Review: Move wikimedia.ee under WM-EE - https://phabricator.wikimedia.org/T204056 (10Dzahn) a:05Dzahn>03tramm [05:33:27] (03CR) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert foundation.w.o to use vhost (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/461394 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [05:41:57] (03PS1) 10Marostegui: db-eqiad.php: Depool pc1005 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461858 [05:46:00] (03CR) 10ArielGlenn: [C: 032] Create wikidata ntriples dump from ttl dump [puppet] - 10https://gerrit.wikimedia.org/r/447922 (https://phabricator.wikimedia.org/T144103) (owner: 10Smalyshev) [05:56:10] (03PS1) 10Smalyshev: Add lexemes dump as separate dump [puppet] - 10https://gerrit.wikimedia.org/r/461862 (https://phabricator.wikimedia.org/T202830) [05:56:50] (03CR) 10jerkins-bot: [V: 04-1] Add lexemes dump as separate dump [puppet] - 10https://gerrit.wikimedia.org/r/461862 (https://phabricator.wikimedia.org/T202830) (owner: 10Smalyshev) [05:58:44] (03PS2) 10Smalyshev: Add lexemes dump as separate dump [puppet] - 10https://gerrit.wikimedia.org/r/461862 (https://phabricator.wikimedia.org/T202830) [05:59:14] (03CR) 10jerkins-bot: [V: 04-1] Add lexemes dump as separate dump [puppet] - 10https://gerrit.wikimedia.org/r/461862 (https://phabricator.wikimedia.org/T202830) (owner: 10Smalyshev) [06:00:07] (03PS3) 10Smalyshev: Add lexemes dump as separate dump [puppet] - 10https://gerrit.wikimedia.org/r/461862 (https://phabricator.wikimedia.org/T202830) [06:06:36] (03CR) 10Smalyshev: Add lexemes dump as separate dump (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/461862 (https://phabricator.wikimedia.org/T202830) (owner: 10Smalyshev) [06:29:45] PROBLEM - puppet last run on mw1300 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/profile.d/mysql-ps1.sh] [06:48:17] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool pc1005 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461858 (owner: 10Marostegui) [06:50:02] (03Merged) 10jenkins-bot: db-eqiad.php: Depool pc1005 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461858 (owner: 10Marostegui) [06:51:04] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool pc1005 (duration: 00m 51s) [06:51:06] !log Stop MySQL on pc1005 for kernel upgrade [06:51:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:51:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:54:47] (03CR) 10jenkins-bot: db-eqiad.php: Depool pc1005 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461858 (owner: 10Marostegui) [06:55:44] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool pc1005" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461864 [06:56:05] (03PS33) 10Mathew.onipe: Add elasticsearch_cluster module. [software/spicerack] - 10https://gerrit.wikimedia.org/r/456322 (https://phabricator.wikimedia.org/T202885) [06:56:07] (03PS4) 10Mathew.onipe: Added force shard allocation to elasticsearch_cluster [software/spicerack] - 10https://gerrit.wikimedia.org/r/461267 (https://phabricator.wikimedia.org/T202885) [06:57:03] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool pc1005" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461864 (owner: 10Marostegui) [06:57:08] (03PS12) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert chapterwiki [puppet] - 10https://gerrit.wikimedia.org/r/451258 (https://phabricator.wikimedia.org/T196968) [06:57:10] (03PS3) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert foundation.w.o to use vhost [puppet] - 10https://gerrit.wikimedia.org/r/461394 (https://phabricator.wikimedia.org/T196968) [06:57:12] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert login.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461866 (https://phabricator.wikimedia.org/T196968) [06:57:14] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::vhost: Allow if using canonical name or not [puppet] - 10https://gerrit.wikimedia.org/r/461867 [06:57:20] (03CR) 10jerkins-bot: [V: 04-1] Add elasticsearch_cluster module. [software/spicerack] - 10https://gerrit.wikimedia.org/r/456322 (https://phabricator.wikimedia.org/T202885) (owner: 10Mathew.onipe) [06:57:22] (03CR) 10jerkins-bot: [V: 04-1] Added force shard allocation to elasticsearch_cluster [software/spicerack] - 10https://gerrit.wikimedia.org/r/461267 (https://phabricator.wikimedia.org/T202885) (owner: 10Mathew.onipe) [06:59:28] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool pc1005" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461864 (owner: 10Marostegui) [07:00:14] RECOVERY - puppet last run on mw1300 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [07:00:33] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool pc1005 (duration: 00m 50s) [07:00:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:04:15] (03PS13) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert chapterwiki [puppet] - 10https://gerrit.wikimedia.org/r/451258 (https://phabricator.wikimedia.org/T196968) [07:08:58] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool pc1005" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461864 (owner: 10Marostegui) [07:12:19] (03PS34) 10Mathew.onipe: Add elasticsearch_cluster module. [software/spicerack] - 10https://gerrit.wikimedia.org/r/456322 (https://phabricator.wikimedia.org/T202885) [07:12:21] (03PS5) 10Mathew.onipe: Added force shard allocation to elasticsearch_cluster [software/spicerack] - 10https://gerrit.wikimedia.org/r/461267 (https://phabricator.wikimedia.org/T202885) [07:13:13] (03CR) 10Mathew.onipe: Add elasticsearch_cluster module. (0311 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/456322 (https://phabricator.wikimedia.org/T202885) (owner: 10Mathew.onipe) [07:13:36] (03CR) 10jerkins-bot: [V: 04-1] Add elasticsearch_cluster module. [software/spicerack] - 10https://gerrit.wikimedia.org/r/456322 (https://phabricator.wikimedia.org/T202885) (owner: 10Mathew.onipe) [07:15:30] (03CR) 10jerkins-bot: [V: 04-1] Added force shard allocation to elasticsearch_cluster [software/spicerack] - 10https://gerrit.wikimedia.org/r/461267 (https://phabricator.wikimedia.org/T202885) (owner: 10Mathew.onipe) [07:17:38] !log uploaded debdeploy 0.0.99.6 to apt.wikimedia.org [07:17:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:18:22] (03CR) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert chapterwiki (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/451258 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [07:46:25] (03CR) 10Filippo Giunchedi: [C: 032] Define Haproxy Prometheus jobs [puppet] - 10https://gerrit.wikimedia.org/r/461724 (https://phabricator.wikimedia.org/T187765) (owner: 10Gilles) [07:46:33] (03PS2) 10Filippo Giunchedi: Define Haproxy Prometheus jobs [puppet] - 10https://gerrit.wikimedia.org/r/461724 (https://phabricator.wikimedia.org/T187765) (owner: 10Gilles) [07:56:32] 10Operations, 10MediaWiki-API, 10Availability, 10HHVM, and 6 others: HHVM request timeouts not working; support lowering the API request timeout per request - https://phabricator.wikimedia.org/T97192 (10Joe) I think this ticket can be closed thanks to @tstarling's work. [08:06:37] (03CR) 10Muehlenhoff: "This looks good to go!" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/451258 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [08:06:44] (03CR) 10Muehlenhoff: [C: 031] mediawiki::web::prod_sites: convert chapterwiki [puppet] - 10https://gerrit.wikimedia.org/r/451258 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [08:08:53] (03CR) 10Marostegui: mariadb: First version of mariadb backup fresness alert (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/461665 (https://phabricator.wikimedia.org/T203969) (owner: 10Jcrespo) [08:10:57] 10Operations, 10DBA, 10Epic, 10Patch-For-Review: DB meta task for next DC failover issues - https://phabricator.wikimedia.org/T189107 (10Marostegui) [08:11:10] 10Operations, 10DBA, 10Epic, 10Patch-For-Review: DB meta task for next DC failover issues - https://phabricator.wikimedia.org/T189107 (10Marostegui) T203565 and T186188 are no longer tasks that are blocked/depend on the DC failover - removing them as subtasks [08:12:00] 10Operations, 10DBA, 10Patch-For-Review: mysql user and group should be a system user/group - https://phabricator.wikimedia.org/T100501 (10Marostegui) [08:12:02] 10Operations, 10DBA, 10Epic, 10Patch-For-Review: DB meta task for next DC failover issues - https://phabricator.wikimedia.org/T189107 (10Marostegui) [08:25:55] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki::web::prod_sites: convert chapterwiki [puppet] - 10https://gerrit.wikimedia.org/r/451258 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [08:26:05] (03PS14) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert chapterwiki [puppet] - 10https://gerrit.wikimedia.org/r/451258 (https://phabricator.wikimedia.org/T196968) [08:39:01] (03PS1) 10Volans: Rename clustermgmt to cumin [dns] - 10https://gerrit.wikimedia.org/r/461919 (https://phabricator.wikimedia.org/T201346) [08:41:13] (03CR) 10Muehlenhoff: [C: 031] Rename clustermgmt to cumin [dns] - 10https://gerrit.wikimedia.org/r/461919 (https://phabricator.wikimedia.org/T201346) (owner: 10Volans) [08:41:35] <_joe_> meh [08:41:40] <_joe_> I preferred clustermgmt [08:42:07] _joe_: comment in the task(s), but cumin2001 is already in production [08:42:14] <_joe_> I know [08:42:18] ;) [08:44:06] (03PS7) 10Gehel: elasticsearch: disable the default elasticsearch unit [puppet] - 10https://gerrit.wikimedia.org/r/458464 (https://phabricator.wikimedia.org/T198351) [08:46:18] (03CR) 10Gehel: [C: 032] elasticsearch: disable the default elasticsearch unit [puppet] - 10https://gerrit.wikimedia.org/r/458464 (https://phabricator.wikimedia.org/T198351) (owner: 10Gehel) [09:08:48] 10Operations, 10media-storage, 10User-fgiunchedi: Running swiftrepl is not puppetized - https://phabricator.wikimedia.org/T162123 (10fgiunchedi) That's correct, let's leave it open until {T204245} is fixed [09:10:21] (03PS1) 10Arturo Borrero Gonzalez: icinga: wikitech_static: check Special:RecentChanges instead of the SAL [puppet] - 10https://gerrit.wikimedia.org/r/461924 (https://phabricator.wikimedia.org/T204840) [09:11:21] (03CR) 10Arturo Borrero Gonzalez: [C: 032] icinga: wikitech_static: check Special:RecentChanges instead of the SAL [puppet] - 10https://gerrit.wikimedia.org/r/461924 (https://phabricator.wikimedia.org/T204840) (owner: 10Arturo Borrero Gonzalez) [09:29:43] 10Operations, 10ops-eqiad, 10procurement: DB eqiad refresh: 13 DB hosts - https://phabricator.wikimedia.org/T205068 (10Marostegui) [09:36:03] (03PS1) 10Gehel: elasticsearch: load jvm options from instance specific config dir [puppet] - 10https://gerrit.wikimedia.org/r/461926 (https://phabricator.wikimedia.org/T198351) [09:36:34] 10Operations, 10ops-eqiad, 10Cloud-VPS, 10Patch-For-Review, 10cloud-services-team (Kanban): rack/setup/install cloudstore1008 & cloudstore1009 - https://phabricator.wikimedia.org/T193655 (10aborrero) hey @RobH @Cmjohnson what is the status of this? Could we move forward with OS installation? [09:45:53] (03CR) 10DCausse: [C: 04-1] elasticsearch: load jvm options from instance specific config dir (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/461926 (https://phabricator.wikimedia.org/T198351) (owner: 10Gehel) [09:47:04] (03PS2) 10Gehel: elasticsearch: load jvm options from instance specific config dir [puppet] - 10https://gerrit.wikimedia.org/r/461926 (https://phabricator.wikimedia.org/T198351) [09:47:38] (03CR) 10DCausse: [C: 031] elasticsearch: load jvm options from instance specific config dir [puppet] - 10https://gerrit.wikimedia.org/r/461926 (https://phabricator.wikimedia.org/T198351) (owner: 10Gehel) [09:48:01] (03PS3) 10Gehel: elasticsearch: load jvm options from instance specific config dir [puppet] - 10https://gerrit.wikimedia.org/r/461926 (https://phabricator.wikimedia.org/T198351) [09:57:53] (03PS1) 10Banyek: wmf-pt-kill: Initial commit for wmf-pt-kill [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 [09:59:12] (03CR) 10Gehel: "And a few more nitpicks!" (035 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/456322 (https://phabricator.wikimedia.org/T202885) (owner: 10Mathew.onipe) [10:01:39] (03PS2) 10Banyek: wmf-pt-kill: Initial commit for wmf-pt-kill [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 [10:02:20] (03CR) 10Marostegui: wmf-pt-kill: Initial commit for wmf-pt-kill (031 comment) [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 (owner: 10Banyek) [10:03:54] (03CR) 10Gehel: [C: 032] elasticsearch: load jvm options from instance specific config dir (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/461926 (https://phabricator.wikimedia.org/T198351) (owner: 10Gehel) [10:06:28] (03PS3) 10Banyek: wmf-pt-kill: Initial commit for wmf-pt-kill [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 [10:10:41] (03PS4) 10Banyek: wmf-pt-kill: Initial commit for wmf-pt-kill [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 [10:11:49] (03PS5) 10Banyek: wmf-pt-kill: Initial commit for wmf-pt-kill [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 [10:21:20] (03CR) 10Muehlenhoff: wmf-pt-kill: Initial commit for wmf-pt-kill (0311 comments) [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 (owner: 10Banyek) [10:25:53] (03PS6) 10Banyek: wmf-pt-kill: Initial commit for wmf-pt-kill [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 [10:26:57] (03CR) 10Marostegui: wmf-pt-kill: Initial commit for wmf-pt-kill (031 comment) [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 (owner: 10Banyek) [10:38:09] (03CR) 10Banyek: "marostegui: I want to aim now for the minimum working state after I have the way of building the package - I'll move forward for patching " (031 comment) [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 (owner: 10Banyek) [10:42:48] (03CR) 10Muehlenhoff: wmf-pt-kill: Initial commit for wmf-pt-kill (033 comments) [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 (owner: 10Banyek) [10:43:25] (03CR) 10Marostegui: wmf-pt-kill: Initial commit for wmf-pt-kill (031 comment) [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 (owner: 10Banyek) [10:43:57] PROBLEM - Debian mirror in sync with upstream on sodium is CRITICAL: /srv/mirrors/debian is over 14 hours old. [10:45:58] (03PS7) 10Banyek: wmf-pt-kill: Initial commit for wmf-pt-kill [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 [10:48:24] (03CR) 10Marostegui: wmf-pt-kill: Initial commit for wmf-pt-kill (031 comment) [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 (owner: 10Banyek) [10:50:34] (03CR) 10Volans: [C: 032] Rename clustermgmt to cumin [dns] - 10https://gerrit.wikimedia.org/r/461919 (https://phabricator.wikimedia.org/T201346) (owner: 10Volans) [10:53:08] (03CR) 10Marostegui: wmf-pt-kill: Initial commit for wmf-pt-kill (031 comment) [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 (owner: 10Banyek) [10:56:44] banyek: marostegui: I am adding a CI job to build wmf-pt-kill package :) [10:57:00] as a non voting job, so it is not going to vote Verified-1 for now [10:59:42] 10Operations, 10Wikimedia-Mailing-lists, 10Patch-For-Review: Mailman issues a "403 Forbidden" error when subscribing to a list - https://phabricator.wikimedia.org/T195750 (10D3r1ck01) This issue still persists in some IP Addresses from Cameroon I think. I've gotten folks poke about the 403 error. Thanks! [11:00:12] (03CR) 10Hashar: "recheck" [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 (owner: 10Banyek) [11:09:05] (03PS8) 10Banyek: wmf-pt-kill: Initial commit for wmf-pt-kill [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 [11:11:46] (03CR) 10Muehlenhoff: wmf-pt-kill: Initial commit for wmf-pt-kill (031 comment) [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 (owner: 10Banyek) [11:13:36] (03PS9) 10Banyek: wmf-pt-kill: Initial commit for wmf-pt-kill [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 [11:16:16] (03CR) 10Muehlenhoff: [C: 031] "Looks good :-)" [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 (owner: 10Banyek) [11:20:54] 10Operations, 10Domains, 10Traffic, 10WMF-Legal, 10Patch-For-Review: Move wikimedia.ee under WM-EE - https://phabricator.wikimedia.org/T204056 (10tramm) Our preferred hosting provider doesn't have separate nameserver service for domains not hosted in their system. But as I see domain wikimedia.ee is mana... [11:21:54] (03CR) 10Gehel: [C: 04-1] "A few comments inline" (039 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/461267 (https://phabricator.wikimedia.org/T202885) (owner: 10Mathew.onipe) [11:24:04] Is there a way to see what IP resolves to what node? I'm looking at this but I can't find the nodes I'm looking for. e.g. 10.68.18.83 [11:24:15] https://github.com/wikimedia/operations-dns/blob/master/templates/10.in-addr.arpa [11:28:27] (03PS10) 10Banyek: wmf-pt-kill: Initial commit for wmf-pt-kill [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 [11:29:10] (03CR) 10Banyek: "I added the wmf-pt-kill_2.2.20.orig.tar.xz file as well" [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 (owner: 10Banyek) [11:29:25] akosiaris: Hey, can you take a look at it ^ [11:30:23] (03CR) 10Banyek: [C: 032] wmf-pt-kill: Initial commit for wmf-pt-kill [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461928 (owner: 10Banyek) [11:31:35] Amir1: that's a WMCS IP: tools-exec-1439.tools.eqiad.wmflabs [11:32:46] volans: Oh thanks, is there I can take a look a the list of them? [11:33:09] Amir1: I just did a dig -x from a labs instance [11:33:34] I see [11:33:41] That's good idea [11:38:07] PROBLEM - Logstash rate of ingestion percent change compared to yesterday on einsteinium is CRITICAL: 131.1 ge 130 https://grafana.wikimedia.org/dashboard/db/logstash?orgId=1&panelId=2&fullscreen [11:42:56] 10Operations, 10DNS, 10Domains, 10Traffic: redirect wikipedia.gr to el.wikipedia.org - https://phabricator.wikimedia.org/T205077 (10jijiki) [11:56:06] Amir1, you want a list of labs instances and their IPs? [11:56:45] Krenair: for labs, now I use a cidr range, now I'm looking for mw nodes in prd [11:56:49] *prod [11:57:21] ok [11:57:45] note all the labs networking is in the process of being changed [12:02:51] (03PS1) 10Hashar: git buildpackage configuration [debs/wmf-pt-kill] - 10https://gerrit.wikimedia.org/r/461941 [12:07:35] jouncebot: next [12:07:35] In 70 hour(s) and 22 minute(s): Wikimedia Portals Update (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180924T1030) [12:08:31] (03PS1) 10Marostegui: db-eqiad.php: Depool pc1006 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461943 [12:10:01] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool pc1006 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461943 (owner: 10Marostegui) [12:11:18] (03Merged) 10jenkins-bot: db-eqiad.php: Depool pc1006 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461943 (owner: 10Marostegui) [12:13:08] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool pc1006 (duration: 00m 50s) [12:13:10] !log Stop MySQL on pc1006 for kernel upgrade [12:13:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:13:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:17:27] (03CR) 10jenkins-bot: db-eqiad.php: Depool pc1006 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461943 (owner: 10Marostegui) [12:18:50] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool pc1006" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461945 [12:21:44] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool pc1006" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461945 (owner: 10Marostegui) [12:22:59] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool pc1006" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461945 (owner: 10Marostegui) [12:24:28] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool pc1006 (duration: 00m 49s) [12:24:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:31:23] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool pc1006" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461945 (owner: 10Marostegui) [12:31:35] !log adding wmf-pt-kill_2.2.20-1+wmf1 package for strech [12:31:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:33:09] \o/ [12:42:06] 10Operations, 10DNS, 10Domains, 10Traffic, 10User-jijiki: redirect wikipedia.gr to el.wikipedia.org - https://phabricator.wikimedia.org/T205077 (10jijiki) a:03jijiki [12:43:03] Is it fine if I do an unscheduled deploy of ores? [12:43:19] https://phabricator.wikimedia.org/T205087 [12:43:44] !log adding wmf-pt-kill_2.2.20-1+wmf1 package for jessie [12:43:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:44:23] banyek: I wonder why a package for jessie? [12:45:30] We still have a few jessie hosts - I know we are getting rid of them, but a few left, so it won't hurt to have the package there as well [12:46:05] banyek: Yeah, probably useful for toolsdb (labsdb1004 and labsdb1005) [12:50:02] I'm going to deploy for https://phabricator.wikimedia.org/T205087 [12:50:14] (03PS4) 10Bstorm: puppetdb: Don't try to install tuning.conf until dir/package exists [puppet] - 10https://gerrit.wikimedia.org/r/435677 (owner: 10Alex Monk) [12:51:58] (03CR) 10Bstorm: [C: 032] puppetdb: Don't try to install tuning.conf until dir/package exists [puppet] - 10https://gerrit.wikimedia.org/r/435677 (owner: 10Alex Monk) [12:57:48] (03PS2) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert login.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461866 (https://phabricator.wikimedia.org/T196968) [13:01:51] 10Operations, 10Goal: Migrate the hardware inventory from Racktables to Netbox - https://phabricator.wikimedia.org/T199083 (10Volans) [13:01:53] !log wiping netbox DB to re-import it cleanly from racktables - T199083 [13:02:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:02:02] T199083: Migrate the hardware inventory from Racktables to Netbox - https://phabricator.wikimedia.org/T199083 [13:02:42] 10Operations, 10Community-Tech, 10MediaWiki-Parser, 10Traffic: Show SVGs in wiki language if available - https://phabricator.wikimedia.org/T205040 (10Niharika) [13:04:45] 10Puppet, 10Shinken: Puppet broken on shinken-01 by requirement of missing package - https://phabricator.wikimedia.org/T205091 (10Krenair) [13:04:49] 10Operations, 10Community-Tech, 10MediaWiki-Parser, 10Traffic: Show SVGs in wiki language if available - https://phabricator.wikimedia.org/T205040 (10Niharika) @MaxSem @Mooeypoo If this ticket is a lot of work (sounds like it), it might be better to split it in smaller chunks. [13:04:53] (03CR) 10Alex Monk: "Caused T205091" [puppet] - 10https://gerrit.wikimedia.org/r/370103 (https://phabricator.wikimedia.org/T83992) (owner: 10Ayounsi) [13:08:38] (03CR) 10Giuseppe Lavagetto: "https://puppet-compiler.wmflabs.org/compiler1002/12534/mw1261.eqiad.wmnet/ the changes seem particularly innocuous. We might still decide " [puppet] - 10https://gerrit.wikimedia.org/r/461866 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [13:08:59] (03PS2) 10Giuseppe Lavagetto: mediawiki::web::vhost: Allow if using canonical name or not [puppet] - 10https://gerrit.wikimedia.org/r/461867 [13:11:23] !log ladsgroup@deploy1001 Started deploy [ores/deploy@7b987a7]: Pass the real IP (T205087) [13:11:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:11:30] ^ deploying ores, UBN [13:11:31] T205087: ORES locks on varnish IPs and not the client IP - https://phabricator.wikimedia.org/T205087 [13:11:49] 10Puppet, 10Shinken: Puppet broken on shinken-01 by requirement of missing package - https://phabricator.wikimedia.org/T205091 (10Krenair) p:05Triage>03High [13:18:04] canary looked fine [13:18:13] beta was also tested, it was working fine [13:20:53] (03PS13) 10Bstorm: wiki replicas - prepare for refactored actor storage [puppet] - 10https://gerrit.wikimedia.org/r/431823 (https://phabricator.wikimedia.org/T195747) [13:21:28] (03PS6) 10Petar.petkovic: Remove unused default source language config for CX [mediawiki-config] - 10https://gerrit.wikimedia.org/r/460492 [13:23:27] (03CR) 10Giuseppe Lavagetto: "https://puppet-compiler.wmflabs.org/compiler1002/12535/mw1261.eqiad.wmnet/ only real diffs are the ones of the parent change." [puppet] - 10https://gerrit.wikimedia.org/r/461867 (owner: 10Giuseppe Lavagetto) [13:27:48] (03CR) 10Muehlenhoff: [C: 031] mediawiki::web::prod_sites: convert login.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461866 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [13:35:33] !log ladsgroup@deploy1001 Finished deploy [ores/deploy@7b987a7]: Pass the real IP (T205087) (duration: 24m 10s) [13:35:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:35:41] T205087: ORES locks on varnish IPs and not the client IP - https://phabricator.wikimedia.org/T205087 [13:40:30] 10Operations, 10hardware-requests: Refresh (leased) restbase2001-2006 - https://phabricator.wikimedia.org/T205092 (10mark) [13:40:49] 10Operations, 10hardware-requests: Refresh (leased) restbase2001-2006 - https://phabricator.wikimedia.org/T205092 (10mark) p:05Triage>03High [13:44:41] 10Operations, 10Puppet, 10User-Banyek: Debian package or files managed my puppet for pt-kill-wmf - https://phabricator.wikimedia.org/T203674 (10Banyek) The debian package is done, it's repository can be found on https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/wmf-pt-kill I need the cloud peo... [13:47:07] (03PS1) 10Alex Monk: Labs monitoring: Authorise new shinken host [puppet] - 10https://gerrit.wikimedia.org/r/461957 (https://phabricator.wikimedia.org/T204562) [13:48:34] (03CR) 10Alexandros Kosiaris: [C: 031] mediawiki: improve siteinfo checks [software/spicerack] - 10https://gerrit.wikimedia.org/r/460114 (https://phabricator.wikimedia.org/T199079) (owner: 10Volans) [13:50:24] (03CR) 10Volans: [C: 032] mediawiki: improve siteinfo checks [software/spicerack] - 10https://gerrit.wikimedia.org/r/460114 (https://phabricator.wikimedia.org/T199079) (owner: 10Volans) [13:51:02] (03CR) 10Volans: [C: 04-2] "Making sure it doesn't get merged by error before the switchback to eqiad" [software/spicerack] - 10https://gerrit.wikimedia.org/r/458115 (https://phabricator.wikimedia.org/T199079) (owner: 10Volans) [13:52:21] (03CR) 10jerkins-bot: [V: 04-1] mediawiki: improve siteinfo checks [software/spicerack] - 10https://gerrit.wikimedia.org/r/460114 (https://phabricator.wikimedia.org/T199079) (owner: 10Volans) [13:52:32] wut? [13:52:49] * volans checking [13:59:28] PROBLEM - MediaWiki memcached error rate on graphite1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [5000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [14:00:44] (03CR) 10Muehlenhoff: "The puppet catalogue fails to compile in PCC:" [puppet] - 10https://gerrit.wikimedia.org/r/461394 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [14:01:33] (03PS1) 10Alex Monk: shinken: Don't try to exec shinkengen until shinken is installed [puppet] - 10https://gerrit.wikimedia.org/r/461962 [14:01:46] RECOVERY - MediaWiki memcached error rate on graphite1001 is OK: OK: Less than 40.00% above the threshold [1000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [14:02:16] 10Puppet, 10Shinken: Puppet broken on shinken-01 by requirement of missing package - https://phabricator.wikimedia.org/T205091 (10Krenair) Depending on how long {T204562} takes we might not need to do anything here [14:04:34] (03CR) 10Muehlenhoff: "Also getting a failing catalogue in PCC:" [puppet] - 10https://gerrit.wikimedia.org/r/452635 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [14:07:57] (03CR) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert simple wikis in remnant.conf (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/452323 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [14:08:13] (03PS26) 10Banyek: Labs: Config template generation for pt-kill [puppet] - 10https://gerrit.wikimedia.org/r/458810 (https://phabricator.wikimedia.org/T183983) [14:08:52] (03CR) 10jerkins-bot: [V: 04-1] Labs: Config template generation for pt-kill [puppet] - 10https://gerrit.wikimedia.org/r/458810 (https://phabricator.wikimedia.org/T183983) (owner: 10Banyek) [14:10:30] (03PS27) 10Banyek: Labs: Config template generation for pt-kill [puppet] - 10https://gerrit.wikimedia.org/r/458810 (https://phabricator.wikimedia.org/T183983) [14:13:20] (03PS1) 10Andrew Bogott: Region-migrate: Deactivate the image right after we use it [puppet] - 10https://gerrit.wikimedia.org/r/461963 [14:13:26] 10Operations, 10Puppet, 10User-Banyek: Debian package or files managed my puppet for pt-kill-wmf - https://phabricator.wikimedia.org/T203674 (10Marostegui) You probably want to add @bstorm and @andrew to review that Gerrit patch. [14:13:55] (03PS28) 10Banyek: wikireplicas: Config template generation for wmf-pt-kill [puppet] - 10https://gerrit.wikimedia.org/r/458810 (https://phabricator.wikimedia.org/T183983) [14:15:31] (03CR) 10Alex Monk: [C: 031] Region-migrate: Deactivate the image right after we use it [puppet] - 10https://gerrit.wikimedia.org/r/461963 (owner: 10Andrew Bogott) [14:17:01] (03CR) 10Andrew Bogott: [C: 032] Region-migrate: Deactivate the image right after we use it [puppet] - 10https://gerrit.wikimedia.org/r/461963 (owner: 10Andrew Bogott) [14:18:05] 10Operations, 10monitoring: improve reqstats error alerts - https://phabricator.wikimedia.org/T98450 (10fgiunchedi) 05Open>03Invalid No longer valid, nowadays we monitor nginx/varnish availability through Prometheus, have setup alerts on low availability and have dashboards such as https://grafana.wikimedi... [14:21:19] (03CR) 10Muehlenhoff: [C: 031] "Actually, the image need to be extended so that libssl1.1 is also upgraded (as nodejs needs a more recent version from the component, whic" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/461627 (https://phabricator.wikimedia.org/T201611) (owner: 10Alexandros Kosiaris) [14:27:57] (03CR) 10Ottomata: [C: 031] Add IPv6 interface to an-master100[1,2] [puppet] - 10https://gerrit.wikimedia.org/r/461729 (https://phabricator.wikimedia.org/T201939) (owner: 10Elukey) [14:28:21] (03PS2) 10Elukey: Add IPv6 interface to an-master100[1,2] [puppet] - 10https://gerrit.wikimedia.org/r/461729 (https://phabricator.wikimedia.org/T201939) [14:30:04] (03CR) 10Elukey: [C: 032] Add IPv6 interface to an-master100[1,2] [puppet] - 10https://gerrit.wikimedia.org/r/461729 (https://phabricator.wikimedia.org/T201939) (owner: 10Elukey) [14:41:32] (03PS1) 10Elukey: Add IPv6 PTRs for an-master100[1,2] [dns] - 10https://gerrit.wikimedia.org/r/461969 (https://phabricator.wikimedia.org/T201939) [14:45:27] RECOVERY - Debian mirror in sync with upstream on sodium is OK: /srv/mirrors/debian is over 0 hours old. [14:46:04] (03CR) 10Elukey: [C: 032] Add IPv6 PTRs for an-master100[1,2] [dns] - 10https://gerrit.wikimedia.org/r/461969 (https://phabricator.wikimedia.org/T201939) (owner: 10Elukey) [15:11:12] (03PS3) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert login.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461866 (https://phabricator.wikimedia.org/T196968) [15:11:14] (03PS3) 10Giuseppe Lavagetto: mediawiki::web::vhost: Allow if using canonical name or not [puppet] - 10https://gerrit.wikimedia.org/r/461867 [15:11:16] (03PS4) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert foundation.w.o to use vhost [puppet] - 10https://gerrit.wikimedia.org/r/461394 (https://phabricator.wikimedia.org/T196968) [15:11:18] (03PS5) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert simple wikis in remnant.conf [puppet] - 10https://gerrit.wikimedia.org/r/452323 (https://phabricator.wikimedia.org/T196968) [15:11:20] (03PS3) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert usability wiki [puppet] - 10https://gerrit.wikimedia.org/r/452635 (https://phabricator.wikimedia.org/T196968) [15:11:22] (03PS3) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: migrate wikispecies [puppet] - 10https://gerrit.wikimedia.org/r/452636 (https://phabricator.wikimedia.org/T196968) [15:11:24] (03PS2) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert commons.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461395 (https://phabricator.wikimedia.org/T196968) [15:11:26] (03PS2) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert meta.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461396 (https://phabricator.wikimedia.org/T196968) [15:11:28] (03PS2) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikisource.org [puppet] - 10https://gerrit.wikimedia.org/r/461397 (https://phabricator.wikimedia.org/T196968) [15:11:30] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikivoyage.org [puppet] - 10https://gerrit.wikimedia.org/r/461976 (https://phabricator.wikimedia.org/T196968) [15:16:01] !log route ns2 to multatuli.wikimedia.org on cr1/2-eqiad [15:16:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:18:37] 10Operations, 10ops-eqiad, 10Cloud-VPS, 10Patch-For-Review, 10cloud-services-team (Kanban): rack/setup/install cloudvirt102[34] - https://phabricator.wikimedia.org/T199125 (10Andrew) Can I get an update on this? Are we blocked waiting for Dell to ship us RAID controllers? [15:21:06] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:27:03] (03PS1) 10Ayounsi: Only install python3-snimpy on debian >= jessie [puppet] - 10https://gerrit.wikimedia.org/r/461978 (https://phabricator.wikimedia.org/T205091) [15:28:24] (03CR) 10Alex Monk: [C: 031] Only install python3-snimpy on debian >= jessie [puppet] - 10https://gerrit.wikimedia.org/r/461978 (https://phabricator.wikimedia.org/T205091) (owner: 10Ayounsi) [15:28:28] (03CR) 10jerkins-bot: [V: 04-1] Only install python3-snimpy on debian >= jessie [puppet] - 10https://gerrit.wikimedia.org/r/461978 (https://phabricator.wikimedia.org/T205091) (owner: 10Ayounsi) [15:29:21] (03PS1) 10Elukey: Swap analytics100[1,2] with an-master100[1,2]$ [puppet] - 10https://gerrit.wikimedia.org/r/461979 (https://phabricator.wikimedia.org/T203635) [15:29:38] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] "Which version do we need ? I 've built the image today and we currently have" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/461627 (https://phabricator.wikimedia.org/T201611) (owner: 10Alexandros Kosiaris) [15:30:19] (03CR) 10jerkins-bot: [V: 04-1] Swap analytics100[1,2] with an-master100[1,2]$ [puppet] - 10https://gerrit.wikimedia.org/r/461979 (https://phabricator.wikimedia.org/T203635) (owner: 10Elukey) [15:30:30] (03PS2) 10Ayounsi: Only install python3-snimpy on debian >= jessie [puppet] - 10https://gerrit.wikimedia.org/r/461978 (https://phabricator.wikimedia.org/T205091) [15:31:03] 10Operations, 10Citoid, 10Services, 10Patch-For-Review, and 3 others: Deploy translation-server-v2 - https://phabricator.wikimedia.org/T201611 (10akosiaris) @thcipriani, @dduvall, nodejs 10 image built and uploaded. It's the "slim" variant, I was wondering if it is enough or if we also require the "devel"... [15:31:17] (03PS2) 10Elukey: Swap analytics100[1,2] with an-master100[1,2]$ [puppet] - 10https://gerrit.wikimedia.org/r/461979 (https://phabricator.wikimedia.org/T203635) [15:31:49] (03PS3) 10Elukey: Swap analytics100[1,2] with an-master100[1,2] [puppet] - 10https://gerrit.wikimedia.org/r/461979 (https://phabricator.wikimedia.org/T203635) [15:35:51] (03CR) 10Ayounsi: "https://puppet-compiler.wmflabs.org/compiler1002/12539/" [puppet] - 10https://gerrit.wikimedia.org/r/461978 (https://phabricator.wikimedia.org/T205091) (owner: 10Ayounsi) [15:37:27] 10Operations, 10ops-eqiad: install additonal SSDs maps100[1-4] - https://phabricator.wikimedia.org/T205110 (10Cmjohnson) [15:37:34] 10Operations, 10ops-eqiad: install additonal SSDs maps100[1-4] - https://phabricator.wikimedia.org/T205110 (10Cmjohnson) 05Open>03Resolved Done [15:39:43] !log upgrading gdnsd to 2.99.42 on authdns2001 [15:39:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:41:14] 10Operations, 10Scap, 10Patch-For-Review: Intermittent git-fat failure during deploy - https://phabricator.wikimedia.org/T202100 (10thcipriani) >>! In T202100#4543309, @Ottomata wrote: > Done, worked everywhere except: > > ``` > The following hosts were unreachable: > cloudservices1004.wikimedia.org > ```... [15:42:46] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:48:47] 10Operations, 10Fundraising-Backlog, 10Traffic, 10fundraising-tech-ops: SSL cert for links.email.wikimedia.org - https://phabricator.wikimedia.org/T188561 (10CCogdill_WMF) [15:51:39] (03PS1) 10Alex Monk: shinken: Only overwrite init.d script on trusty [puppet] - 10https://gerrit.wikimedia.org/r/461982 (https://phabricator.wikimedia.org/T204562) [15:57:14] (03PS4) 10Elukey: Swap analytics100[1,2] with an-master100[1,2] [puppet] - 10https://gerrit.wikimedia.org/r/461979 (https://phabricator.wikimedia.org/T203635) [15:58:04] (03CR) 10Dzahn: [C: 031] "agree with the workaround and not making this wait for a shinken upgrade. i also think it's starting to become more and more of a pain tha" [puppet] - 10https://gerrit.wikimedia.org/r/461978 (https://phabricator.wikimedia.org/T205091) (owner: 10Ayounsi) [16:00:43] (03CR) 10Alex Monk: [C: 031] "we're working on it" [puppet] - 10https://gerrit.wikimedia.org/r/461978 (https://phabricator.wikimedia.org/T205091) (owner: 10Ayounsi) [16:02:37] (03CR) 10Ayounsi: [C: 032] Only install python3-snimpy on debian >= jessie [puppet] - 10https://gerrit.wikimedia.org/r/461978 (https://phabricator.wikimedia.org/T205091) (owner: 10Ayounsi) [16:07:05] 10Operations, 10Citoid, 10Services, 10Patch-For-Review, and 3 others: Deploy translation-server-v2 - https://phabricator.wikimedia.org/T201611 (10thcipriani) >>! In T201611#4606083, @akosiaris wrote: > @thcipriani, @dduvall, nodejs 10 image built and uploaded. It's the "slim" variant, I was wondering if it... [16:10:40] 10Puppet, 10Shinken, 10Patch-For-Review: Puppet broken on shinken-01 by requirement of missing package - https://phabricator.wikimedia.org/T205091 (10Krenair) 05Open>03Resolved a:03ayounsi [16:12:23] (03CR) 10Dzahn: [C: 032] "just merging this because i know shinken is on trusty, so this won't affect anything currently used" [puppet] - 10https://gerrit.wikimedia.org/r/461982 (https://phabricator.wikimedia.org/T204562) (owner: 10Alex Monk) [16:12:49] (03CR) 10Dzahn: [C: 031] "well, i didn't notice the parent patches, so +1 then" [puppet] - 10https://gerrit.wikimedia.org/r/461982 (https://phabricator.wikimedia.org/T204562) (owner: 10Alex Monk) [16:15:06] mutante, gonna rebase that on top of the production branch because it'd be nice to get puppet re-enabled on the new instance [16:15:30] Krenair: sure, ok :) [16:15:40] (03PS2) 10Alex Monk: shinken: Only overwrite init.d script on trusty [puppet] - 10https://gerrit.wikimedia.org/r/461982 (https://phabricator.wikimedia.org/T204562) [16:16:00] I didn't really need to stack them it was just convenient [16:16:03] (03CR) 10Dzahn: [C: 032] shinken: Only overwrite init.d script on trusty [puppet] - 10https://gerrit.wikimedia.org/r/461982 (https://phabricator.wikimedia.org/T204562) (owner: 10Alex Monk) [16:17:52] (03CR) 10Dzahn: [C: 032] shinken: Don't try to exec shinkengen until shinken is installed [puppet] - 10https://gerrit.wikimedia.org/r/461962 (owner: 10Alex Monk) [16:18:02] (03CR) 10Dzahn: [C: 031] shinken: Don't try to exec shinkengen until shinken is installed [puppet] - 10https://gerrit.wikimedia.org/r/461962 (owner: 10Alex Monk) [16:18:11] that one isn't so important [16:18:25] one of the one-off kind of errors [16:18:46] ok, let's keep it at this for now then. i'd step back from the one above that. the "authorize / network::constants" part and leave that to wmcs [16:18:58] yes [16:19:13] (03CR) 10Andrew Bogott: [C: 031] "I don't really understand the context here but I certainly have no objection." [puppet] - 10https://gerrit.wikimedia.org/r/460467 (https://phabricator.wikimedia.org/T193848) (owner: 10Krinkle) [16:19:23] that's the bit that lets it through ferm everywhere IIRC :) [16:22:24] (03CR) 10Krinkle: "Context:" [puppet] - 10https://gerrit.wikimedia.org/r/460467 (https://phabricator.wikimedia.org/T193848) (owner: 10Krinkle) [16:23:12] (03CR) 10Andrew Bogott: [C: 031] "Thank you for explaining! That definitely sounds worth the trouble." [puppet] - 10https://gerrit.wikimedia.org/r/460467 (https://phabricator.wikimedia.org/T193848) (owner: 10Krinkle) [16:23:47] (03PS2) 10Andrew Bogott: openstack: Add redirect for /view/ on Wikitech [puppet] - 10https://gerrit.wikimedia.org/r/460467 (https://phabricator.wikimedia.org/T193848) (owner: 10Krinkle) [16:24:17] (03CR) 10Alex Monk: [C: 031] openstack: Add redirect for /view/ on Wikitech [puppet] - 10https://gerrit.wikimedia.org/r/460467 (https://phabricator.wikimedia.org/T193848) (owner: 10Krinkle) [16:24:47] (03CR) 10Andrew Bogott: [C: 032] openstack: Add redirect for /view/ on Wikitech [puppet] - 10https://gerrit.wikimedia.org/r/460467 (https://phabricator.wikimedia.org/T193848) (owner: 10Krinkle) [16:26:43] Krinkle: that change is applied now — did it help? [16:27:08] andrewbogott: confirmed. Thanks ! [16:27:12] https://wikitech.wikimedia.org/view/Main_Page redir lgtm [16:27:14] cool [16:29:36] !log puppetmaster: running mcrouter_generate_certs to add an mcrouter cert for mwmaint1002 (T201343) https://wikitech.wikimedia.org/wiki/Mcrouter#Generate_certs_for_a_new_host [16:29:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:29:44] T201343: rack/setup/install mwmaint1002.eqiad.wmnet - https://phabricator.wikimedia.org/T201343 [16:30:55] (03PS1) 10Elukey: Guard analytics cron jobs to ease the hw refresh of an1003 [puppet] - 10https://gerrit.wikimedia.org/r/461988 (https://phabricator.wikimedia.org/T203635) [16:32:54] !log puppetmaster1001: mcrouter_generate_certs --generate [16:32:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:33:44] !log puppetmaster1001: cergen --base-path /srv/private/modules/secret/secrets/mcrouter/ --generate /etc/cergen/mcrouter.manifests.d [16:33:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:37:33] !log route ns2-v6 to multatuli.wikimedia.org on cr1/2-esams [16:37:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:37:47] PROBLEM - Varnish traffic drop between 30min ago and now at eqsin on einsteinium is CRITICAL: 58.22 le 60 https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [16:38:48] (03PS1) 10Alex Monk: shinken: Fix dependency on jessie [puppet] - 10https://gerrit.wikimedia.org/r/461991 [16:39:20] (03CR) 10Alex Monk: "oops, I3b4b0b22" [puppet] - 10https://gerrit.wikimedia.org/r/461982 (https://phabricator.wikimedia.org/T204562) (owner: 10Alex Monk) [16:39:25] (03CR) 10jerkins-bot: [V: 04-1] shinken: Fix dependency on jessie [puppet] - 10https://gerrit.wikimedia.org/r/461991 (owner: 10Alex Monk) [16:39:57] RECOVERY - Varnish traffic drop between 30min ago and now at eqsin on einsteinium is OK: (C)60 le (W)70 le 79.48 https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [16:40:13] (03PS2) 10Alex Monk: shinken: Fix dependency on jessie [puppet] - 10https://gerrit.wikimedia.org/r/461991 [16:41:10] (03CR) 10jerkins-bot: [V: 04-1] shinken: Fix dependency on jessie [puppet] - 10https://gerrit.wikimedia.org/r/461991 (owner: 10Alex Monk) [16:42:11] (03PS3) 10Alex Monk: shinken: Fix dependency on jessie [puppet] - 10https://gerrit.wikimedia.org/r/461991 [16:42:26] mutante, ^ (sorry) [16:42:51] (03PS1) 10Andrew Bogott: Horizon: move toolserver-legacy to eqiad1 [puppet] - 10https://gerrit.wikimedia.org/r/461992 [16:43:11] (03CR) 10Dzahn: [C: 032] shinken: Fix dependency on jessie [puppet] - 10https://gerrit.wikimedia.org/r/461991 (owner: 10Alex Monk) [16:44:24] (03PS2) 10Andrew Bogott: Horizon: move toolserver-legacy to eqiad1 [puppet] - 10https://gerrit.wikimedia.org/r/461992 [16:45:36] (03CR) 10Andrew Bogott: [C: 032] Horizon: move toolserver-legacy to eqiad1 [puppet] - 10https://gerrit.wikimedia.org/r/461992 (owner: 10Andrew Bogott) [16:46:28] 10Operations, 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash, 10Release-Engineering-Team (Watching / External): logstash-beta.wmflab throws multiple "Error: Could not locate that visualization" - https://phabricator.wikimedia.org/T204845 (10greg) @fgiunchedi fyi, you got assigned to this task because... [16:46:47] Krenair: no problem. you can try again [16:46:55] (running puppet i mean) [16:47:11] yeah [16:47:39] now it seems something else in our puppetisation somewhere has broken it :) [16:47:42] will get there in the end [16:47:48] that's normal :) [16:52:58] (03PS4) 10Dzahn: openstack::network: update IPs of relic.toolserver [puppet] - 10https://gerrit.wikimedia.org/r/455737 (https://phabricator.wikimedia.org/T204564) [16:56:29] (03PS1) 10Elukey: Replace analytics1003's occurrences with an-coord1001 [puppet] - 10https://gerrit.wikimedia.org/r/461997 (https://phabricator.wikimedia.org/T203635) [17:01:20] yeah we're gonna need to vary the shinken config a bit between versions :/ [17:01:27] (probably) [17:12:06] (03PS1) 10BryanDavis: cloud: update MediaWiki-Vagrant container start actions [puppet] - 10https://gerrit.wikimedia.org/r/462000 [17:20:28] (03PS1) 10Dzahn: toolserver_legacy: enable ssl module for httpd [puppet] - 10https://gerrit.wikimedia.org/r/462004 (https://phabricator.wikimedia.org/T204564) [17:21:41] (03CR) 10Dzahn: [C: 032] "don't know how this worked on the existing trusty instance, we did not just add the ssl config part in Apache or anything" [puppet] - 10https://gerrit.wikimedia.org/r/462004 (https://phabricator.wikimedia.org/T204564) (owner: 10Dzahn) [17:25:55] (03CR) 10Andrew Bogott: [C: 031] openstack::network: update IPs of relic.toolserver [puppet] - 10https://gerrit.wikimedia.org/r/455737 (https://phabricator.wikimedia.org/T204564) (owner: 10Dzahn) [17:26:02] (03PS5) 10Andrew Bogott: openstack::network: update IPs of relic.toolserver [puppet] - 10https://gerrit.wikimedia.org/r/455737 (https://phabricator.wikimedia.org/T204564) (owner: 10Dzahn) [17:37:58] (03CR) 10Dzahn: [C: 032] openstack::network: update IPs of relic.toolserver [puppet] - 10https://gerrit.wikimedia.org/r/455737 (https://phabricator.wikimedia.org/T204564) (owner: 10Dzahn) [17:38:26] PROBLEM - Logstash rate of ingestion percent change compared to yesterday on einsteinium is CRITICAL: 131.6 ge 130 https://grafana.wikimedia.org/dashboard/db/logstash?orgId=1&panelId=2&fullscreen [17:43:43] (03PS1) 10Andrew Bogott: Update toolserver.org IP [dns] - 10https://gerrit.wikimedia.org/r/462005 [17:44:02] hmm, not sure if that logstash rate of ingetsion alert is actionable [17:45:37] (03CR) 10Dzahn: [C: 031] "host 185.15.56.23" [dns] - 10https://gerrit.wikimedia.org/r/462005 (owner: 10Andrew Bogott) [17:45:50] (03CR) 10Andrew Bogott: [C: 032] Update toolserver.org IP [dns] - 10https://gerrit.wikimedia.org/r/462005 (owner: 10Andrew Bogott) [17:46:22] looks like there is an increase in events from gelf input https://grafana.wikimedia.org/dashboard/db/logstash?orgId=1&from=1536947122108&to=1537551922108&panelId=9&fullscreen [17:47:24] (03CR) 10Mathew.onipe: Added force shard allocation to elasticsearch_cluster (034 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/461267 (https://phabricator.wikimedia.org/T202885) (owner: 10Mathew.onipe) [17:48:45] 10Operations, 10DNS, 10Traffic, 10WMF-Communications, and 4 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776 (10bd808) [17:49:27] seeing lots of this from scb hosts https://www.irccloud.com/pastebin/fSaulB0A/ [17:50:50] mobrovac, Pchelolo --^ [17:51:08] herron: all scb hosts or only a subset? [17:51:30] looks to be evenly distributed across all [17:51:59] https://logstash.wikimedia.org/goto/3086543f63e7f054db3abd0d05bf8748 [17:52:08] scb2XXX [17:55:54] no idea if if it is transient or not, might be worth to know Marko or Petr's opinion [17:56:24] 10Operations, 10CirrusSearch, 10Discovery, 10Elasticsearch, 10Discovery-Search (Current work): Resolve elasticsearch shard size alert by doing an in place reindex - https://phabricator.wikimedia.org/T204362 (10Mathew.onipe) [17:58:09] !log toolserver.org and subdomains (wiki.toolserver, status.toolserver, stable.toolserver) legacy URLs have been switched to new stretch backend, away from trusty [17:58:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:59:14] !log *.toolserver.org also moved from eqiad to eqiad-r region in cloud vps, which gave it new IP addresses [17:59:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:02:27] can somebody invite me to #mediawiki_security ? It seems my cloak is not allows that [18:03:33] ^ robh [18:06:44] !log uploaded 2.99.1729-beta-1+wmf1 to stretch-wikimedia [18:06:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:08:12] !log upgrade multatuli to gdnsd-2.99.1729-beta [18:08:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:19:46] !log deploy patch related to T197279 T204825 [18:19:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:54:28] (03PS35) 10Mathew.onipe: Add elasticsearch_cluster module [software/spicerack] - 10https://gerrit.wikimedia.org/r/456322 (https://phabricator.wikimedia.org/T202885) [18:54:29] (03PS6) 10Mathew.onipe: elasticsearch_cluster: adding new features [software/spicerack] - 10https://gerrit.wikimedia.org/r/461267 (https://phabricator.wikimedia.org/T202885) [18:54:56] (03CR) 10Mathew.onipe: Add elasticsearch_cluster module (035 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/456322 (https://phabricator.wikimedia.org/T202885) (owner: 10Mathew.onipe) [19:04:41] (03PS1) 10Framawiki: Set wgRestrictDisplayTitle = false for pflwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462011 (https://phabricator.wikimedia.org/T205055) [19:04:43] (03PS1) 10Dzahn: network: add 172.16.0.0/21 (labs-instances2-b-eqiad) to external_networks [puppet] - 10https://gerrit.wikimedia.org/r/462012 [19:05:25] (03PS2) 10Dzahn: network: add 172.16.0.0/21 (labs-instances2-b-eqiad) to external_networks [puppet] - 10https://gerrit.wikimedia.org/r/462012 [19:05:52] (03CR) 10jerkins-bot: [V: 04-1] network: add 172.16.0.0/21 (labs-instances2-b-eqiad) to external_networks [puppet] - 10https://gerrit.wikimedia.org/r/462012 (owner: 10Dzahn) [19:07:07] (03CR) 10Dzahn: "https://wikitech.wikimedia.org/wiki/IP_and_AS_allocations" [puppet] - 10https://gerrit.wikimedia.org/r/462012 (owner: 10Dzahn) [19:07:51] (03PS3) 10Dzahn: network: add 172.16.0.0/21 (labs-instances2-b-eqiad) to external_networks [puppet] - 10https://gerrit.wikimedia.org/r/462012 (https://phabricator.wikimedia.org/T204564) [19:08:39] (03CR) 10Paladox: [C: 031] network: add 172.16.0.0/21 (labs-instances2-b-eqiad) to external_networks [puppet] - 10https://gerrit.wikimedia.org/r/462012 (https://phabricator.wikimedia.org/T204564) (owner: 10Dzahn) [19:10:14] !log shutdown cr3/4-ulsfo (not in prod) [19:10:14] (03PS4) 10Dzahn: network: add 172.16.0.0/21 (labs-instances2-b-eqiad) to external_networks [puppet] - 10https://gerrit.wikimedia.org/r/462012 (https://phabricator.wikimedia.org/T204564) [19:10:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:10:56] (03CR) 10jerkins-bot: [V: 04-1] network: add 172.16.0.0/21 (labs-instances2-b-eqiad) to external_networks [puppet] - 10https://gerrit.wikimedia.org/r/462012 (https://phabricator.wikimedia.org/T204564) (owner: 10Dzahn) [19:15:47] (03CR) 10Herron: [C: 031] "One minor comment inline, otherwise looks good to me! Afaict the only references to external_networks are ntp and all_networks, and I'd sa" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462012 (https://phabricator.wikimedia.org/T204564) (owner: 10Dzahn) [19:23:16] (03PS1) 10Framawiki: Throttle for October 11 event [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462014 (https://phabricator.wikimedia.org/T204829) [19:24:49] (03PS2) 10Framawiki: Throttle for October 11 event [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462014 (https://phabricator.wikimedia.org/T204829) [19:38:26] RECOVERY - Logstash rate of ingestion percent change compared to yesterday on einsteinium is OK: (C)130 ge (W)110 ge 93.07 https://grafana.wikimedia.org/dashboard/db/logstash?orgId=1&panelId=2&fullscreen [19:46:11] (03PS1) 10Andrew Bogott: Revert "Horizon: move toolserver-legacy to eqiad1" [puppet] - 10https://gerrit.wikimedia.org/r/462016 [19:46:22] (03PS2) 10Andrew Bogott: Revert "Horizon: move toolserver-legacy to eqiad1" [puppet] - 10https://gerrit.wikimedia.org/r/462016 [19:47:43] (03CR) 10Andrew Bogott: [C: 032] Revert "Horizon: move toolserver-legacy to eqiad1" [puppet] - 10https://gerrit.wikimedia.org/r/462016 (owner: 10Andrew Bogott) [19:49:27] (03CR) 10Dzahn: [C: 04-2] "per IRC" [puppet] - 10https://gerrit.wikimedia.org/r/462012 (https://phabricator.wikimedia.org/T204564) (owner: 10Dzahn) [20:06:32] (03Abandoned) 10Dzahn: network: add 172.16.0.0/21 (labs-instances2-b-eqiad) to external_networks [puppet] - 10https://gerrit.wikimedia.org/r/462012 (https://phabricator.wikimedia.org/T204564) (owner: 10Dzahn) [20:10:18] 10Operations, 10Cloud-Services, 10Mail: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785 (10bd808) @Andrew & @harej: The bump of this ticket after figuring out that the eqiad-r region needs a new set of mail hosts means that we **do not** want to delete the //project-smtp// project. [20:14:21] 10Operations, 10Cloud-Services, 10Mail: Create a Cloud VPS SMTP smarthost - https://phabricator.wikimedia.org/T41785 (10bd808) [20:14:23] (03PS1) 10Thcipriani: Beta: maintenance: skip mediawiki::state function [puppet] - 10https://gerrit.wikimedia.org/r/462019 (https://phabricator.wikimedia.org/T125976) [20:14:27] (03PS1) 10Thcipriani: Beta: maintenance: no openldap management [puppet] - 10https://gerrit.wikimedia.org/r/462020 (https://phabricator.wikimedia.org/T125976) [20:15:18] 10Operations, 10Cloud-Services, 10Mail: Create a Cloud VPS SMTP smarthost - https://phabricator.wikimedia.org/T41785 (10herron) p:05Normal>03High [20:19:00] 10Operations, 10Beta-Cluster-Infrastructure, 10Wikidata, 10wikidata-tech-focus, and 3 others: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976 (10thcipriani) I've got a stretch instance called `deployment-mwmaint01` running in beta with `role::mediawiki_mainten... [20:25:42] 10Operations, 10Cloud-Services, 10Mail: Create a Cloud VPS SMTP smarthost - https://phabricator.wikimedia.org/T41785 (10herron) So in terms of IP addressing we will need to assign the cloud outbound smtp instance a public IP (or static NAT) address with matching forward and reverse DNS. Will a floating IP a... [20:36:11] 10Operations, 10Cloud-Services, 10Mail: Create a Cloud VPS SMTP smarthost - https://phabricator.wikimedia.org/T41785 (10Andrew) [20:46:58] (03PS1) 10Andrew Bogott: Horizon: move project-smtp to eqiad1r [puppet] - 10https://gerrit.wikimedia.org/r/462021 [20:47:45] (03CR) 10Andrew Bogott: [C: 032] Horizon: move project-smtp to eqiad1r [puppet] - 10https://gerrit.wikimedia.org/r/462021 (owner: 10Andrew Bogott) [20:47:57] (03PS1) 10Dzahn: Revert "Update toolserver.org IP" [dns] - 10https://gerrit.wikimedia.org/r/462022 [20:49:04] (03CR) 10Dzahn: [C: 032] "we can't use this yet until we have separate mail relays for cloud instances in eqiad-r" [dns] - 10https://gerrit.wikimedia.org/r/462022 (owner: 10Dzahn) [20:55:29] (03PS1) 10Dzahn: openstack::network: update private IP of relic.toolserver [puppet] - 10https://gerrit.wikimedia.org/r/462023 (https://phabricator.wikimedia.org/T204564) [20:56:51] (03PS1) 10Cwhite: monitoring: set mode on host and service configs [puppet] - 10https://gerrit.wikimedia.org/r/462024 (https://phabricator.wikimedia.org/T202782) [20:57:42] (03CR) 10jerkins-bot: [V: 04-1] monitoring: set mode on host and service configs [puppet] - 10https://gerrit.wikimedia.org/r/462024 (https://phabricator.wikimedia.org/T202782) (owner: 10Cwhite) [21:04:34] (03PS2) 10Cwhite: monitoring: set mode on host and service configs [puppet] - 10https://gerrit.wikimedia.org/r/462024 (https://phabricator.wikimedia.org/T202782) [21:13:57] 10Operations, 10Mail: Mail relays needed for VMs in eqiad1 - https://phabricator.wikimedia.org/T205158 (10Framawiki) [21:34:27] PROBLEM - HHVM rendering on mw1256 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:26] RECOVERY - HHVM rendering on mw1256 is OK: HTTP OK: HTTP/1.1 200 OK - 79101 bytes in 0.192 second response time [21:41:36] 10Operations, 10Beta-Cluster-Infrastructure, 10Wikidata, 10wikidata-tech-focus, and 3 others: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976 (10MarcoAurelio) Note that not all scripts running on production can be run on beta (for example, the lack of CheckUse... [21:43:39] (03CR) 10Dzahn: [C: 032] openstack::network: update private IP of relic.toolserver [puppet] - 10https://gerrit.wikimedia.org/r/462023 (https://phabricator.wikimedia.org/T204564) (owner: 10Dzahn) [21:43:46] (03CR) 10Volans: "It's not clear to me what is the issue you're trying to solve here. Could you add some additional context please?" [puppet] - 10https://gerrit.wikimedia.org/r/462024 (https://phabricator.wikimedia.org/T202782) (owner: 10Cwhite) [21:56:36] (03CR) 10Cwhite: "> It's not clear to me what is the issue you're trying to solve here." [puppet] - 10https://gerrit.wikimedia.org/r/462024 (https://phabricator.wikimedia.org/T202782) (owner: 10Cwhite) [22:01:44] 10Operations, 10Beta-Cluster-Infrastructure, 10Wikidata, 10wikidata-tech-focus, and 3 others: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976 (10Dzahn) Currently there is just one general switch to either enable all crons (scripts) or disable them all, and it'... [22:12:19] !log how to generate mcrouter certs, needed when you add any new mw server: https://wikitech.wikimedia.org/wiki/Mcrouter#Generate_certs_for_a_new_host [22:12:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:13:32] !log mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (201343) [22:13:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:13:38] !log mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (T201343) [22:13:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:13:46] T201343: rack/setup/install mwmaint1002.eqiad.wmnet - https://phabricator.wikimedia.org/T201343 [22:14:50] 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install mwmaint1002.eqiad.wmnet - https://phabricator.wikimedia.org/T201343 (10Dzahn) 05stalled>03Open [22:29:21] (03CR) 10Volans: "> Patch Set 2:" [puppet] - 10https://gerrit.wikimedia.org/r/462024 (https://phabricator.wikimedia.org/T202782) (owner: 10Cwhite) [22:29:36] (03CR) 10Gehel: [C: 031] "LGTM, let's see what volans says." [software/spicerack] - 10https://gerrit.wikimedia.org/r/456322 (https://phabricator.wikimedia.org/T202885) (owner: 10Mathew.onipe) [22:33:56] (03CR) 10Dzahn: [C: 032] "mwmaint1002 can now be used as LDAP client server. tested ldapsearch, modify-ldap-group, ldaplist" [puppet] - 10https://gerrit.wikimedia.org/r/461487 (https://phabricator.wikimedia.org/T201343) (owner: 10Dzahn) [22:34:14] (03PS2) 10Dzahn: site/admin: replace mwmaint1001 with 1002 in comments [puppet] - 10https://gerrit.wikimedia.org/r/461487 (https://phabricator.wikimedia.org/T201343) [22:38:44] (03PS2) 10Dzahn: scap/dsh: add mwmaint1002 to mw scap hosts [puppet] - 10https://gerrit.wikimedia.org/r/461489 (https://phabricator.wikimedia.org/T201343) [22:40:46] !log mwmaint1002 - scap pull [22:40:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:46:17] (03CR) 10Dzahn: [C: 032] "we have /srv/mediawiki populated now" [puppet] - 10https://gerrit.wikimedia.org/r/461489 (https://phabricator.wikimedia.org/T201343) (owner: 10Dzahn) [23:13:43] (03CR) 10Cwhite: "> > Patch Set 2:" [puppet] - 10https://gerrit.wikimedia.org/r/462024 (https://phabricator.wikimedia.org/T202782) (owner: 10Cwhite) [23:17:36] 10Puppet, 10Cloud-VPS: cloudvps: puppet project trusty deprecation - https://phabricator.wikimedia.org/T204558 (10Dzahn) 05Open>03Resolved [23:25:02] (03PS2) 10Elukey: [WIP] Replace analytics1003's occurrences with an-coord1001 [puppet] - 10https://gerrit.wikimedia.org/r/461997 (https://phabricator.wikimedia.org/T203635) [23:26:22] 10Operations, 10Cloud-VPS: cloudvps: toolserver-legacy project trusty deprecation - https://phabricator.wikimedia.org/T204564 (10Dzahn) [23:32:12] 10Operations, 10Scap, 10Patch-For-Review: Intermittent git-fat failure during deploy - https://phabricator.wikimedia.org/T202100 (10Dzahn) ``` [cloudservices1004:~] $ dpkg -l git-fat .. ii git-fat 0.1.3-2~jessie1 all Manage large files with git, without checking ``` [23:33:59] 10Operations, 10Scap, 10Patch-For-Review: Intermittent git-fat failure during deploy - https://phabricator.wikimedia.org/T202100 (10Dzahn) 05Open>03Resolved a:03Ottomata [23:41:16] (03PS1) 10Dzahn: cache::text: replace (commented) mwmaint1001 with mwmaint1002 [puppet] - 10https://gerrit.wikimedia.org/r/462036 (https://phabricator.wikimedia.org/T201343) [23:47:15] !log mwmaint1002 - rsyncing home dirs over from mwmaint1001 which synced them from terbium. still large old files from terbium here, like a NASA video :) (T201343) [23:47:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:47:23] T201343: rack/setup/install mwmaint1002.eqiad.wmnet - https://phabricator.wikimedia.org/T201343 [23:52:57] 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install mwmaint1002.eqiad.wmnet - https://phabricator.wikimedia.org/T201343 (10Dzahn) p:05Normal>03High